Mastering Browser Automation with Python and Selenium
Vložit
- čas přidán 14. 08. 2020
- In today's episode of learn Python by doing we explore browser automation using Selenium and control the Google chrome browser to automatically login with Python. This is part 1 of 2 scraping video, make sure to subscribe if you haven't already
This is amazing! Thank you for explaining it in a way that makes it easier to understand! Keep the great content coming!
Very clear video! Selenium seems to be the number one choice for web scraping but there are very few modern tutorial videos on this. The web driver manager also takes away a huge headache I used to have with the web driver.
Thank you Adam! So far I must say your tutorials rank among the best in the subject matter. At this rate you will hit 2k plus subscribers by year-end 2020. Congratulations!
Thank you!!
I started leaning python a few weeks ago and used selenium, it took me hours to understand what you explained in less than 15’, great job!!
Thank you! I like to create a mix of learning by doing and not hiding my errors, it's the best way to learn!
This video help me so much, I'm newbie on python and dont speak english very well, but I can understand everthing and make my web scrapping!
You are amazing MAN. Thanks for your efforts.
You are an incredible instructor!!! I've been learning the tools to scrape for the last couple of years and this is the best instruction yet. Would love to see a video on how and where you store the large data sets that you scrape.
Hey good thinking! I'll add it to my list, with so many different options maybe I'll share a different method each week 🤙
Congrats. You deserve it. You’re doing great work.
Thanks Jeff, I really appreciate it.
great video again. looking forward to part two.
Thanks Christoph!
It helped to crack my project milestone. Thanks a lot. love your support.
Great to hear Anup!
you are awesome and really just love your positivity. I just hope everyone around us has a person like you to teach.
ycoolest guy in youtube.
Really liked your video, well explained, I paid for a "Excel automation with Python course" and it was just xpath scraping (1 line of code) and basic openpyxl, which pandas can do better. Your video is free and much better, keep up the hard-work!
Really easy to follow video. I really appreciate the help!
Glad to hear it!
You made that very clear and easy to understand... thank you. Five stars from me! *****
This is awesome, Adam! Thanks for providing these incredible resources. Do you use any apps to help you memorise the elements of the selenium package (such as Anki?) Also congrats on reaching 1k subscribers and completely blowing your goal out of the water.
Thanks John! I'm blown away by how quickly we reached 1k 😃 I haven't been using any apps but checking out Anki now.
THANK YOU!!!
This is sooooo helpful
Amazing stuff. Thanks a lot for making this video.
Glad you liked it!
Awesome man....i really learnt something frm u
Happy to help
this is amazing i wish i watched this years ago
More to come! Working on some longer videos to go through data collection, cleanup and analysis/outputs. Stay tuned!
@@MakeDataUsefulSounds exciting, can't wait 💻
For anyone having issues at the find_element step, I had to slightly modify my code to this, where you have to input the type (class name, id, etc.) and value differently:
driver.find_element(by="id", value="connect_button")
Thank You for sharing this, couldn't find this anywhere!!
thank you bro
Thanks for contributing to the community @RetroPandroo much appreciated!
SO SHIFT ENTER ON THAT!!! thank you!
I’ d been messing with bs4 straight getting a “419 Page Expired” error in the soup… Looking forward to trying starting out w/ Selenium, thanks this is great 🙏👌
Selenium is always a great help, appears as a standard browser to the site
Great video thanks, you made it easy to understand
Glad it helped!
thansk u man, this help me weary mhch THANK U LUV FROM CHENNAI
just awesome
Thanks Siddhant!
Thanx bro thanx alot
Always welcome
Thank you this is amazing and i would like to know more about python i am a new to this and i really wanna know more
How's the python learning going Alhamdi? Still keeping at it?
thanks bro
No problem! I hope you enjoyed the video
Also, I'm sure you are already familiar with that, but you shouldn't use time to wait for elements to wait, you should use the selenium wait features instead. On a slow internet that code could break.
Thanks byjoseh! 100% agree, I'll make sure to use the selenium wait in future videos. I appreciate the feedback!
well done
Thank you!
@@MakeDataUseful chrome driver manager really helped me
first i had to mess with the driver and other things and when new update comes out i had to badly mess with it but now it manages for me easily
You can import plugins using Selenium. I use ublock to block ads
Awesome!! Thanks for the tip! Myself and the community appreciates it.
Hey just checking, where'd you go? Would love more of this.
Hey! Thanks for checking in, new content dropping soon 🙂
Subbed.
Thank you!
@@MakeDataUseful Thank you for such an entertaining and informative tutorials and videos. I really enjoyed your reactions when you mess up.
Thank you for a very great video. How to handle https ssl certificate with this python code?
Yey ... good job Adam. Do you have any tips on getting accepted on upwork ?
Hi Velvet, it is becoming harder and harder to get approved these days. My advice would be to select every subcategory of skill that applies to you and if you have got some past projects you are happy to share host them on GitHub as a bit of a portfolio. Let me know how you go!
@@MakeDataUseful Thanks Adam ... i'm working on my portfolio now ... I've read that they also prefer a personal mail over gmail or other mail services .
can u make it live? coz it gives some unknown error. And thank u for video.
As a final capstone project, I am attempting to create a project on Jupyter Notebook that will retrieve live market data(e.g share indexes,currency values,etc). Is selenium the best for this in comparison to scraper or Bs4
Hey Dylan, I would recommend checking out what API's are available and simplify the project and just use python requests. I seem to recall there even being a python package you can pip install that collects data direct from Yahoo finance. Let me know how you go!
@@MakeDataUseful Thanks for the suggestion, ive made use of pandas_datareader to collect data from the yahoo website. Sill having some issues with regards to converting the data in a more readable graph with matplotlib, but making progress none the less. Thanks once again
Hey how can i login multiple accounts from a csv file?
I get SyntaxError: invalid syntax when I get to driver.get(url)
Can you help me im having problem on blocking a url from loading on the background how can i do that using python and selenium?
Sure can, what's the URL?
Dear Your content is awesome!... I am using google colab on google chrome browser. I encounters error in the line code: driver = webdriver.Chrome(ChromeDriverManager().install()). error says "ValueError: Could not get version for Chrome with this command: google-chrome --version || google-chrome-stable --version". I installed & imported required packages and dependancies such as "!pip install selenium
!pip install webdriver-manager". Dear, your technical help is required to overcome this issue!!!!!!
Hi! May I ask how to click a login button that is in style="display:none;"? Thank you!
Does it have an ID? Finding the element by the ID might be the way to go in this example
@@MakeDataUseful I still get an error but thank you for your reply. I tried your tutorial on other websites and it worked. I didn't know it doesn't need countless codes to log in to a website with python. Your video is really helpful! 😊
@@rbmz1712 how the hell a button to login not displayed
What to do if website is having capitcha?!
How do you log into a website where access is denied from selenium?
Hey, i recently have been working on my project for gmail login automation using selenium , but whenever i run the code gmail wont allow me sign and and will say - "This browser or app may not be secure. Learn more
Try using a different browser. If you’re already using a supported browser, you can refresh your screen and try again to sign in." please help bro much appreciated :)
You have to disable the tow step verification in your google account
How could I overcome CAPTCHA in login for site I am doing?
It's a tough one Nilesh, Captcha is design to stop automated access. One work around I have come up with is moving to a semi automated script that stops and waits for user input to solve the captcha. Not ideal but sometimes it's all we can do. If you are happy to share the site let me know and I can have a look.
Hi! How do I write code that, if my username is incorrect, will try to sign me in with a different username? Can you help me please?
If (exist this element that show incorrect) :
Input another username
Step one is to purposely get the username wrong, then inspect the element that appears to indicate "hey that username is incorrect" then run an if statement that looks for that element and does something different. Interesting question, I hope your not hacking Csimu :P
@@MakeDataUseful yeah he seems sus, but it takes a billion times to do traditional bruteforce :/
When I open chrome using driver it gets closed automatically.
That's very strange, is there any specific error?
where can i get code
I've transcribed it to a gist, although there's something to be said for trying to transcribe it yourself. Muscle memory, making mistakes, etc. gist.github.com/quadespresso/864b3157785a7d14ac2147a601d9e63e
Adam, feel free to remove my comment or whatever.
Thanks for taking the time to transcribe Jim, really appreciate it and 100% agree with benefit of writing the code each time.
@@jimcarroll9738 ya I too agree but sometime it is more better to see from somewhere else
It was giving error when importing driver manager but I had installed it
Can u do this on drive
I had just started coding and not much into it so I dont know how to use git
@@vaibhav3852 you need to do `pip install webdriver-manager selenium` *before* starting Jupyter. If that alone is giving you troubles, then you've got something misconfigured with your Python setup. Trying to troubleshoot that in CZcams comments will prove to be challenging for everyone involved. Recommend you try to search that error online to see what you get back. Failing that, take the question over to a forum such as Stack Overflow.
Hi,
How can i contact you are you have any email on which i can contact you ?
I need litle help on the relationship with this video (about scrapping web site with class inside class).
Thanks in advance!
Best regards
Thank you for a very great video. How to handle https ssl certificate with this python code?