How To Deploy Python Selenium Script in AWS Lambda in 2022
Vložit
- čas přidán 6. 09. 2024
- 🌐 NodeMaven Proxy Provider: go.nodemaven.c...
💥 Special Bonus: Use "Michael" at checkout for an extra +2GB of bandwidth.
🤖 2captcha Captcha Solving Service: bit.ly/2captch...
In this video, I am showing you how to create a serverless Python Selenium script and deploy successfully your Python Selenium script in AWS Lambda. We are also going to utilize AWS lambda layers to provide a convenient way to package libraries and other dependencies that you can use with your Lambda functions. Using layers reduces the size of uploaded deployment archives and makes it faster to deploy your code. A layer is a . zip file archive that can contain additional code or data.
Selenium is an open-source tool that automates web browsers. It provides a single interface that lets you write test scripts in programming languages like Ruby, Java, NodeJS, PHP, Perl, Python, and C#, among others.
AWS Lambda is serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers
Playlist: • Python Selenium Tutorial
Code: github.com/mic...
Commands: pastebin.com/j...
Join our Discord: / discord
Undetectable ChromeDriver: pypi.org/proje...
Gecko Driver: github.com/moz...
Chrome Driver: chromedriver.c...
Download Visual Studio Code: code.visualstu...
Download Python: www.python.org...
Selenium Library: pypi.org/proje...
Donate
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
PayPal: support@websidev.com
Bitcoin Wallet: bc1q05j8gcnq4mzvgj603cxdc8xxck4jgnu2ljsrt4
Ethereum Wallet: 0x5e7BD4f473f153d400b39D593A55D68Ce80F8a2e
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: websidev.com
Linkedin: / michael-kitas-638aa4209
Instagram: / michael_kitas
Github: github.com/mic...
Business Email: support@websidev.com
Tags:
- Python Selenium Tutorial
- Full Course Selenium
- Python Selenium
- Web Scraping Full Course
- Python Selenium Web Scraping Full Course
#selenium #aws #python
Hello
I've followed your video carefully and am pretty sure I didn't miss any detail or do anything wrong. However, I got the error:
"errorMessage": "Unable to import module 'main': No module named 'selenium'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
I really hope you can help me with this, I've turn the internet upside down, but so far nothing works
1. Did you add the selenium layer?
2. Did you connect the layer to the lambda function?
3. Did you install specifically selenium 3.8.0 and with python 3.7 version with the correct path command which I have in the description?
Check those 3, as I faced the same issue before recording the video, and the same checklist applied to me as well.
@@MichaelKitas Nevermind, thank you so much, now it's working
@@minhthieu2466 well what was the issue
@@miclo_ssx he didn't add the selenium layer
@@minhthieu2466
Hey Michael, thanks for the absolutely awesome tutorial.
Just one suggestion -
In the pastebin script referenced in video description, the 1st command to download selenium dependency incorrectly mentions the installation path as selenium/python/lib.
It should be python/lib as you showed in the video.
Thanks
Hi Michael, can you make another video on selenium using container implementation to build lambda? It would be a life saver! Thanks in advance
Great suggestion, will make sure to add it to the list
@@MichaelKitas Yes please!!!! Thanks
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
Hi thanks for the video, this was something that frustrated me in may and dropping it all together. There are plenty of sites with the information, BUT the most important element: this ONLY works with Python 3.7 and Selenium 3.8.
Following the security mindset, everything back in may was trying to use the latest version =)
Agreed with this. Wish it is compatible with all the python and selenium version.
Hi Michael,
Thank you so much for such a good video. Recently AWS lambda announced that lambda will not be maintaining Python3.7 anymore . Need to move my code to python 3.11. Not sure how to do this. Can you please make a video to use selenium chrome driver with python 3.11 in lambda? Or can you suggest me a way to solve this?
Please reply Michael
Many requested this, I've added it to the to-do list, will try to make a video asap
when will be ready this video?@@MichaelKitas
please@@MichaelKitas
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
Can't thank you enough for this, I am gonna write a blog and add this video there.
Thank you a lot for this 🙂
Hello bro, Thanks for ur great work , but I got a question
AWS Lambda end of support for Python 3.7
so it's not supported anymore at least for layers, do u have another way to get around with this , I would really appreciate this alot!
Thanks in advance !
Yes, making an updated video this week. Hint, check: AWS SAM CLI
Thanks so much, unfortunately python 3.7 isn't supported any longer. An updated video would be awesome!
Very soon, working on it, found a solution 👍
@@MichaelKitas How did you get past the error - Error setting up WebDriver: Message: disconnected: unable to connect to renderer? I am using a Docker image.
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
@@MichaelKitas damn, thanks for following it up!
Thank you very much!
what an incredible tutorial, priceless information very hard to find over the internet.
Really awesome tutorial, even I was able to get this up and running!
SUPERB !! i just need to pay attention to the zip path structure, thanks a lot!
Glad it was helpful!
I used the same versions but instead I download chromedriver and headless-chromium from s3 to my tmp files (because of the limit size of layers). I got this error: Message: 'chromedriver' executable may have wrong permissions. Do you know how to fix this?
Check this out: stackoverflow.com/questions/49787327/selenium-on-mac-message-chromedriver-executable-may-have-wrong-permissions
hey. im facing the exact same issue. were u able to fix it?
Hi Michael, in a few comments you talk about an updated video that doesn't rely on a deprecated version of Python. If this video exists, can you please provide a link here
i've been working with python 3.9 and i am struggle with this error : "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127"
Does anyone have any idea how to fix it? I have followed step by step very carefully
same issue with me as well. Strugling with this from past few days and tried almost all the options available on internet. But still can't figure out. Did you got the solution??
AWS gives us the option to set up specific runtime for each lambda function. Use the same version as I said, and there shouldn't be an issue.
@@khushindarshan156 It is an version issue, it will only run on 3.7, any higher version will give 127 error
@@VIKASSAINI007 there's a way to make it work with 3.8 or later?, aws ends support for 3.7, so it can't be created anyways
Hello, thanks for sharing. I tested it and it worked perfectly as in the video, but I have a question. I couldn't get it to work with newer versions of the libraries. Why does this happen? Today I have several codes implemented with the latest versions and it would be a lot of work to adapt the old versions. Tks
Many requested this, I've added it to the to-do list, will try to make a video asap
How did you manage to make headless-chromium turn from 102,6 MB uncompressed to 5,1 MB uncompressed at the 9:50 mark? It's like the number just changes without any explanation. How are you able to upload headless-chromium and chromedriver without exceeding the Layer size limit of 50MB? The exact same file uncompressed on my system is 100MB as well. I'd prefer not to have to use S3
Are you using Mac? That’s how much it is compressed using the Mac’s compression (Under 50mb). Using another software would for sure give different results.
thank you so much.
It was my big problem. you saved my time.!
Thank you for your work. I want to run Selenium4.8. and Request I have a urllib3 library conflict. what could be the problem?
Can you send screenshot (send url of screenshot) of the error?
Thanks for sharing this, incredibly helpful!
Hi, Which selenium version is compatible with Python 3.9 lambda version??
Hello Michael and everyone, does anyone have where I can find compatible versions for headless chrome and selenium that supports Python3.8?
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
Hello, thanks for the video, but but when i am adding the layer of chormdrive there, i got a error 'Layers consume more than the available size of 262144000 bytes'
I have the same issue!! The chromedriver ZIP is a larger file than the ones from the other layers.
@@alexkeila I have managed to solve that by using the exact version of selenium , however , the one run on lambda does not have the same result as I run on my local machine
You need to use the exact version of selenium as @user-zy7qo3ch1s mentioned.
Hi, I have this error: "errorMessage": "2024-03-12T04:26:45.989Z 394e1df8-6683-4e3b-aa9c-da1864b9bba9 Task timed out after 20.01 seconds". I followed all steps but this error appears.
Maybe increase timeout? Probably doesnt have enough time to complete the script
This is amazing, thank you!
using the commands provided, it downloads the chromedriver and the headless chromium but it doesnt open, it always says that the file format is not supported or is broken
Going to make an updated video today, stay tuned.
@@MichaelKitas Hey Michael, when can we expect the updated video?
Thank you very much! It works!
You're welcome!
aws lambda gonna end support for Python 3.7. Also it seems like the current setup we got cannot be migrated to python 3.11. I tried to migrate but chromedriver couldnt start. How should we go from here, Michael?
@MichaelKitas
Let me know if you find a way to work the code in python 3.11
Many requested this, I've added it to the to-do list, will try to make a video asap
@@MichaelKitas Still waiting, I guess docker container is the only option
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
I get the message below when putting it to test:
{
"errorMessage": "'module' object is not callable",
"errorType": "TypeError",
"stackTrace": [
" File \"/var/runtime/bootstrap.py\", line 127, in handle_event_request
response = request_handler(event, lambda_context)
"
]
}
can you help me?
Can you send full code?
Ok, so how can I know which version of Python, Selenium and Chromedriver works with Lambda?
I showed on the video the versions I used
@@MichaelKitas Yes, and only that version works for me. But how do I know when AWS supports newer versions and which combination of python, selenium and chromedriver? I don't want to operate with 3.7 forever.
@@ghoulflier1016 Not sure, I don't think there is a specific combination each time
@@MichaelKitas Thank you anyways for your great videos. They are such a big help for me to get started with Selenium!
Hello Michael ,
How to Making Captcha Breaking Script Work in AWS Lambda
Same way you do it locally, I have made a video for bypassing captcha, check it out here: czcams.com/video/5puz9Mb2d_c/video.html
Very informative.
Do you know if there is a way to configure your AWS lambda account, to tell Amazon a max dollar amount you want to spend, and then if you accidentally exceed that amount with your resources, they either immediately notify you, or just simply shut down your site? I'd prefer it works that way.
I'd hate to get a big surprise bill from amazon, because I didn't know my code was very inefficient, and the code was spending too much time during a function invocation.
Yes, check how to do that here: aws.amazon.com/getting-started/hands-on/control-your-costs-free-tier-budgets/
@@MichaelKitas Thank you
Hi I want to use undetected-chromedriver in my AWS lambda, which version is compliant with selenium 3.8.0?
I think latest version works: pypi.org/project/undetected-chromedriver/3.4.7/
It says Python3.6 and newer, so latest works
Did you get it to run with Undetected-chromedriver? I'm also trying to run in Lambda but I can't make it work at all
@@pedrodellazzari528 unfortunately no.
Do you know about this error "errorMessage": "Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127
",?
Python version must be 3.7 and selenium 3.8.0
Yes, first of all did you add chromedriver as a layer?
I was also getting the same error when I was using latest selenium and chromedriver version.
Hi, I keep getting the error "no chrome binary at /var/taskheadless-chromium
". I added the selenium layer, connected it to the lambda function and installed the specific versions you had. Nothing is working to fix it.
Did you add correct executable path? It’s searching in the wrong place
Message: Service /opt/chromedriver unexpectedly exited. Status code was: 127
:(
I had the same issue, and did not find anything that could solve it. Apparently a new tutorial is on the way!
@@zozoland9057 Hey bro, did you find the new tutorial video?
It works for me.
does anyone knows if there is a way to do it in python3.9 or python3.8? i did not found a solution without the use of a 5xxmb approx docker image
My concern is because python3.7 is "deprecated" since june 2023, and if aws lambda does not support it anymore, we are done...
Looking for a solution and will make a video for newer version
@@user-st1qr6et5h So if i want to do it now, Can't i do it with python 3.7 ? Please elaborate more . And if I have to use Python3.9/3.10 then What versions of chromedriver and headless chromium should i use?
Just uploaded a new video which supports any python version: czcams.com/video/j_qOVqQDBWw/video.html
what is the name of vscode theme you are using ?
Bearded Theme - Arc
Hey Mike! This can work with 'playwright'?
Not sure, will add a series for playwright in the future, and try it
Hello. Thank you for so necessary video. How could you upload the file more than 10Mb in the layer? (chromedriver.zip 44.6Mb) System should had to make you to upload through S3. Is not it?
Yes, you can use S3, max is 50mb if not using s3
Is it possible to teach how to create a website or blog traffic bot that uses VPS and sends requests from Virtual Private server 🙏🙏
Very specific, but I might do that in the future, stay tuned!
Does downloading the chromedriver on windows follows the same command?
You can download chromedriver from the official website: chromedriver.chromium.org/downloads
@@MichaelKitas for windows, do we also need to download the headless chromium like you did on the video or just a chrome driver downloaded from the site you mentioned above is okay?if yes, where do i download the headless chromium?
I have done exactly the same as shown in the video . But when i run I get this response .-> {
"errorMessage": "Syntax error in module 'main': invalid syntax (selenium_manager.py, line 46)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File \"/opt/python/lib/python3.7/site-packages/selenium/webdriver/common/selenium_manager.py\" Line 46
if (path := os.getenv(\"SE_MANAGER_PATH\")) is not None:
"
]
}
Going to make an updated video today, stay tuned.