By this time my code consists of: 25% Stack overflow scripts, 10% my own work, 40% people sharing ideas on CZcams and 25% pure luck (no one know how or why the code works but it does!). Thank you so much for this video!
Dude, This is like the most underrated Python Programming channel I have ever seen. subscribers = 0 while True: subscribers+=100 print("Nice Content"*99999)
Very nice! Just keep in mind that different sites use different tokens to login with. You may need to play around in the inspector for a bit until you find it.
For those who cannot login access to the url, there's a new item under 'Form Data', which is 'csrfToken'. You'll need to include it to the 'login_data' using the same method searching for 'form_build_id'. Code for 'csrfToken': login_data['csrfToken'] = soup.find('input', attrs={'name': 'csrfToken'})['value']
Hello Michelle, were you able to login to the same site after card token addition? I am getting 200 response but the username and password fields are empty
After hours of searching for a way to edit online forms through a python script. (I'm a noob at requests) This video gave me all the tools I needed in ~12 mins. Wish I had found this earlier...
Thank you very much indeed, this video has helped me doing exactlly what I wanted, and with 0 errors. Now that I understand it thanks to you, I can keep doing my work. You're great :D
This is fantastic work. Thank you! Could you please do a tutorial on the network part. How exactly do you spot so easily which "pop up" does what in our requests? This would be awesome. Appreciate it. Best regards
Different websites could send different packets across the network and under different names. This can make it a little confusing to try and find the right sign-in request over many different websites. However, one thing you can look for is the timing of the packets. If you just watch the network for a few minutes, you can see if there are any constant requests, like ping/latency checks. As soon as you press "Login" you will probably see a few initial requests, and then a surge of more requests. Usually the login request is in the first few requests following your login, the others are requests sent to load the new page (icons, images, etc.) Hopefully this helps a bit (and it's not too late), feel free to correct me if I messed up anywhere. Best of luck
For those who get headers errors, try to send all the headers for the login, in my case that was necesary. Also you can just get the CSRF token from the cookies with this: csfrtoken = r.cookies['csrftoken'] My question is, now, that I am logged in, how can I get the response from a specific URL of the page? Thanks for the great video!
@@IndianPythonista Yes! I did it! I have the Response of the sites now. Now I am trying to get the hls URLS: videos: {"serverA": {"id": "serverA", "hls": "movin-b.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=m3u8-aapl,audio-only=false)", "dash": "movin-b.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=mpd-time-csf)"}, "serverB": {"id": "serverB", "hls": "movin.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=m3u8-aapl,audio-only=false)", "dash": "movin.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=mpd-time-csf)"}, "serverC": {"id": "serverC", "hls": "mdstrm.com/video/5ae7e0dee883414be4a13216.m3u8"}} How can I get them? They are inside a script. Thanks for the answer!
@@iamsebs So from what i understand you get something like this? {"pubnub": { "credentials": { "noleave": true, "ssl": true, "subscribe_key": "sub-c-c9437e20-7978-11e8-9f59-fec9626a7085" }, "preamble": "" }, "production": true, "navbar": { .... if you have the whole html response you''ll have to use beautiful soup with find or find_all like he did to get the input. First of all. Is it one item or a list that returns(if you use find its an item. find_all is a list). i dont mean the videos. i mean the "dictionary". First select an item like so resp[0] lets say that resp[0] contains the tag with the the dictionary inside. j = Use resp[0].text.strip() this will store the data in str format. import json jdata = json.loads(j) Now jdata is a dict and you cann access every value
Great content! I have a problem though: I'm trying to login as you suggested but in my case, when I do the inspection of the website in Network/Headers I don't find anything about "Form Data" I don't find anything about password or username. Is it another kind of login or am I doing something wrong? Thank you in advance.
Great video. Very helpful for getting started. However what do I do if I get redirected from my login page. For example say I log in at www.example.com, and press sign in, and I get redirected to www.example.com/account How do I scrape the page from there
Thank You Nikhil for the video. I am facing an issue while using this code to login into a microsoft page, specifically Sharepoint page which is a kind of company's internal site. Also, when I inspect on the sharepoint homepage to look for user-agent and form_build_id, I can find the user-agent, but cannot find the form_build_id information. There is no section of "Form Data" under the Network when I inspect. Is it because the sharepoint login is "Connected to Windows"? Can you please help me out here?
hey and is there anyway to parse javascript values ? cuz there is a needed header that i need to send on the post method , how can i get this header value ( knowing that its calculated in a js FILE ) from what i have seen in inspector
Thanks for the video ! :) I have an additional question. I wish to get the redirected url, that is in Web Inspector under the name "location". Yet I cannot see it anywhere in the soup. From where can I get it ?
I want to login into facebook as a sample. And when I click the "login" button I always find the "www.facebook.com" network process but Its an "get" request with the code "200". I cant seem to find a post request with the categorie "Form Data" with password and username in it (etc.)
Hey, i have a python script that fetches attendance for which i use selenium for logging in and everything works pretty fine, but my question is that after ruuning the script it automatically opens up the browser and does the login and then clicks on the specified link to fetch the attendence !...............What to do if i want to do all of this behind the scene without the browser actually been showing up?
my webpage is not even showing me the form data, when I login with the inspect page beside. Instead I get the username and password under request payload. how do I get the form data?
um can someone help me out here? I copied the video word for word but the error 'TypeError: 'NoneType' object is not callable' is showing up for this line: login_data['form_build_id']=soup.find('input',attrs={'name':'form_build_id'})['value'] I have no fricken idea as to why this is happening.
If anyone is struggling trying to get this to work with a Cloudflare protected page, simply use the module cfscrape. It works exactly the same way the requests module does, the only difference is that instead of requests.Session you would use cfscrape.CloudflareScraper(). To find out more about this module check out: pypi.org/project/cfscrape/
really appreciate your effort and it works like magic with me to arrange for login, but the problem is when I try to navigate to other link in the website rather than login page, Check the downloaded page using get url and find that I'm not signed in , even when I tried to use it using within the same session , could you please help in this part
I don't think any website will use get method for login. In get request, any data has to be passed through url parameters and putting your username and password in the url is the craziest thing one can think of!
Hey man I tried this but the website I'm trying has 2 POST requests. One to login, and one to validate with a unique token. The problem is when I try to parse the URL it's using to 'validate' it says error parsing body. So I can get the unique token value. Any tips?
By this time my code consists of: 25% Stack overflow scripts, 10% my own work, 40% people sharing ideas on CZcams and 25% pure luck (no one know how or why the code works but it does!). Thank you so much for this video!
LOL
Relatable 😃
Dude, This is like the most underrated Python Programming channel I have ever seen.
subscribers = 0
while True:
subscribers+=100
print("Nice Content"*99999)
LOL
it will never print "Nice Content"*99999 since you have an infinite loop
@@simthegreat5690 loool
Very nice!
Just keep in mind that different sites use different tokens to login with. You may need to play around in the inspector for a bit until you find it.
I usualy dont watch indian tutorials but man you are a legend thank you so much❤❤
You're the best Python teacher I've followed by far. I am new to Python and I'm learning fast because of your videos.
Super cool, I was struggling to login into an ASP login, with your neat steps and some tricks I am able to get in now :)
BestT 12 mins of Educational video ever!!!
Thank you so much! This helped me a lot, with this explanation I was able to submit in multi page forms. Greetings from Brasil! ⭐⭐
bas ek chiz kami hai aur vo hai subscribers ki bhai kya content hai aap ka maza aagaya.
hey u got a new subscriber now.
For those who cannot login access to the url, there's a new item under 'Form Data', which is 'csrfToken'. You'll need to include it to the 'login_data' using the same method searching for 'form_build_id'.
Code for 'csrfToken':
login_data['csrfToken'] = soup.find('input', attrs={'name': 'csrfToken'})['value']
Hello Michelle, were you able to login to the same site after card token addition? I am getting 200 response but the username and password fields are empty
@@ashwinis.chavan4598 Hi, I suppose you need to have an account of the website beforehand. Then, you shall use your password and username in the code.
Thank you ! Now I get it with the options like headers and login_data ... and also where I get the login_data dictionary values ! Thanks man ! :)
After hours of searching for a way to edit online forms through a python script. (I'm a noob at requests) This video gave me all the tools I needed in ~12 mins. Wish I had found this earlier...
great video, very helpful to scrape website protected with login
I followed the instructions and it really worked
Before start watching, I want to say thank you !
Thank you very much indeed, this video has helped me doing exactlly what I wanted, and with 0 errors. Now that I understand it thanks to you, I can keep doing my work. You're great :D
This is fantastic work. Thank you! Could you please do a tutorial on the network part. How exactly do you spot so easily which "pop up" does what in our requests? This would be awesome. Appreciate it.
Best regards
Different websites could send different packets across the network and under different names.
This can make it a little confusing to try and find the right sign-in request over many different websites.
However, one thing you can look for is the timing of the packets.
If you just watch the network for a few minutes, you can see if there are any constant requests, like ping/latency checks.
As soon as you press "Login" you will probably see a few initial requests, and then a surge of more requests.
Usually the login request is in the first few requests following your login, the others are requests sent to load the new page (icons, images, etc.)
Hopefully this helps a bit (and it's not too late), feel free to correct me if I messed up anywhere. Best of luck
Finally found what i was looking for ...thank you for high quality content
Great video. I was trying to do something similar in java without using selenium. This helps.
Still using Java at this modern age for web scraping?
i cannot find the form_build... how can we solve that? or what can we do? help please!
def Bravo():
print("Great video! You're very smart")
Bravo()
very clear instructions and explanation. great job and thanks for the tutorial
The video that I was looking for. Thanks.😊
Hey cool tutorial. One question, the website i am trying to do this on has an encrypted password in the form data. How can we go about doing it?
did u try passing the encrypted password as the value for password key?
Exactly what I needed, thanks for the video!
what should i do if there is nothing called 'form_build_id' and there is no value?
skip and watch another video
Check post request
@K H check if your site have something like a token, in the request headerss.
For those who get headers errors, try to send all the headers for the login, in my case that was necesary.
Also you can just get the CSRF token from the cookies with this: csfrtoken = r.cookies['csrftoken']
My question is, now, that I am logged in, how can I get the response from a specific URL of the page?
Thanks for the great video!
Once logged in, you can send request to any url of the website using the same session object.
@@IndianPythonista Yes! I did it! I have the Response of the sites now. Now I am trying to get the hls URLS:
videos: {"serverA": {"id": "serverA", "hls": "movin-b.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=m3u8-aapl,audio-only=false)", "dash": "movin-b.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=mpd-time-csf)"}, "serverB": {"id": "serverB", "hls": "movin.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=m3u8-aapl,audio-only=false)", "dash": "movin.platzi.com/080e9d74-4dbe-4de8-baa5-1fe3ae97f8a9/5ae7e0dee883414be4a13216.ism/manifest(format=mpd-time-csf)"}, "serverC": {"id": "serverC", "hls": "mdstrm.com/video/5ae7e0dee883414be4a13216.m3u8"}}
How can I get them? They are inside a script. Thanks for the answer!
@@iamsebs So from what i understand you get something like this?
{"pubnub": {
"credentials": {
"noleave": true,
"ssl": true,
"subscribe_key": "sub-c-c9437e20-7978-11e8-9f59-fec9626a7085"
},
"preamble": ""
},
"production": true,
"navbar": {
....
if you have the whole html response you''ll have to use beautiful soup with find or find_all like he did to get the input.
First of all. Is it one item or a list that returns(if you use find its an item. find_all is a list). i dont mean the videos. i mean the "dictionary". First select an item like so
resp[0]
lets say that resp[0] contains the tag with the the dictionary inside.
j = Use resp[0].text.strip() this will store the data in str format.
import json
jdata = json.loads(j)
Now jdata is a dict and you cann access every value
You're the best!!!
Great content! I have a problem though: I'm trying to login as you suggested but in my case, when I do the inspection of the website in Network/Headers I don't find anything about "Form Data" I don't find anything about password or username. Is it another kind of login or am I doing something wrong? Thank you in advance.
Logon cannot happen without sending a request. See if you have not applied any filter for type of request.
@@IndianPythonista hey i face the same issue. i cannot find Form Data after i logged in the test website. can you help me with it ?
THANK YOU !! U saved my life!
Thank you. This helped a lot!
Awesome video! Keep up the great work guy!
Thank you! It really helped me with my poject
Thank you! Useful stuff :)
Great video, I learned alot! Keep up the good work!
Bro I have m3u8 which is encrypted
I have key
But I don't know how to decrypt the file ?
When I run the code, it just prints out thelogin page, why is this happpening
Great video. Very helpful for getting started. However what do I do if I get redirected from my login page. For example say I log in at www.example.com, and press sign in, and I get redirected to www.example.com/account
How do I scrape the page from there
Really great video. Thank you pretty much.
Highly talented Indian !! Good job bro
I still need help with Mongo dB
Please if you can help
DNS error
Very good. Thanks!
Super useful and well explained - thanks!
Thank you so much it is very helpful ...nikhil
Great explanation , .. got a question... why you have used bs4 ? is request.get has any disadvantage?
requests.get is used to fetch the content from the website. bs4 helps you to parse that content.
Thanks for tutorial, helped my project!
You're the best!!!
Thank you very much
amazing video, it looks very simple, thanks man
Very helpful thank you
Thank You Nikhil for the video.
I am facing an issue while using this code to login into a microsoft page, specifically Sharepoint page which is a kind of company's internal site.
Also, when I inspect on the sharepoint homepage to look for user-agent and form_build_id, I can find the user-agent, but cannot find the form_build_id information. There is no section of "Form Data" under the Network when I inspect. Is it because the sharepoint login is "Connected to Windows"? Can you please help me out here?
If I wanted to make a post request, do I need to main website or the request url link in the dev tools?
Thanks for the efforts
Help please. In your line number 19 in git code its showing "TypeError: 'NoneType' object is not subscriptable."
did you find the resolution ? i am getting same error
OMG! Thank you soooo much!!!!
I'm getting crazy, you are incomprehensible because of your pronunciation
I do not think you understand English very well, I'm getting crazy.
learn it yourself then
I tried doing it with Reddit, and maybe I got a good callback, but their HTML is gigantic, BeautifulSoup doesn't parse it all
Great content, sir. Thank you!
Thank you for nice explanation.
Very helpful. Thank you!
What's the Linux that you use? It looks super clean and organized. I'd be glad if you can share the theme name aswell. Thanks for the tutorial.
Its adapta gtk + paper icon theme with docky.
Great, thanks a lot from Morocco
Thanks for this tutorial, good work.
Very helpful video thanks
There is csfr token also now a day how to get that for login_data pls tell
I can not find form build id in my login form data headers or anything seems like it with a different header name. What can I do about it?
Great tutorial, thanks a mil man!
hey and is there anyway to parse javascript values ? cuz there is a needed header that i need to send on the post method , how can i get this header value ( knowing that its calculated in a js FILE ) from what i have seen in inspector
Dude you're awesome! Ty
Class explanation..
Pure Gold !!!
Nice, thanks a lot
THANK YOU FOR THIS VIDEO!
Thank you! Great Video!
Thanks you so much sir
Thanks for the video ! :)
I have an additional question. I wish to get the redirected url, that is in Web Inspector under the name "location". Yet I cannot see it anywhere in the soup. From where can I get it ?
And what if in my html there are more then one form_key (because there are more actions (login, register, newsletter)? The key are in the same
Dude, thank you!
I want to login into facebook as a sample. And when I click the "login" button I always find the "www.facebook.com" network process but Its an "get" request with the code "200". I cant seem to find a post request with the categorie "Form Data" with password and username in it (etc.)
Thank you for this content. In my Chrome Inspector is no Form Data shown. How can I see this?
Did you find it? Same problem for me too
Hey, i have a python script that fetches attendance for which i use selenium for logging in and everything works pretty fine, but my question is that after ruuning the script it automatically opens up the browser and does the login and then clicks on the specified link to fetch the attendence !...............What to do if i want to do all of this behind the scene without the browser actually been showing up?
Thank you so much bro
Awesome man!!!
Awesome, thank you so much!
Very useful video; thank you (y)
Bhai thank you I made my code better on my website as I was not know about it that it can be happen
my webpage is not even showing me the form data, when I login with the inspect page beside. Instead I get the username and password under request payload. how do I get the form data?
same problem here
were you able to figure out?
good video and explanation! thanks!
wow bro i learned good thing today
Nice video. There is something I did not get. At 2:02 you used a utility (right side) to capture the data packets from the network. What was that?
Right click and select inspect. You will get the debugger. Then select network tab.
@@IndianPythonista Thanks buddy.
um can someone help me out here?
I copied the video word for word but the error 'TypeError: 'NoneType' object is not callable' is showing up
for this line:
login_data['form_build_id']=soup.find('input',attrs={'name':'form_build_id'})['value']
I have no fricken idea as to why this is happening.
The website that i want to try to scrap, don't have form data area. So i can't get formbuild id name etc. Can u help about it ?
If anyone is struggling trying to get this to work with a Cloudflare protected page, simply use the module cfscrape. It works exactly the same way the requests module does, the only difference is that instead of requests.Session you would use cfscrape.CloudflareScraper(). To find out more about this module check out: pypi.org/project/cfscrape/
Very useful. Thanks
really appreciate your effort and it works like magic with me to arrange for login, but the problem is when I try to navigate to other link in the website rather than login page, Check the downloaded page using get url and find that I'm not signed in , even when I tried to use it using within the same session , could you please help in this part
thanks i tried this with 3 web & successfully logged in
but few website using GET method for login, can you guide me how i can login there.
I don't think any website will use get method for login. In get request, any data has to be passed through url parameters and putting your username and password in the url is the craziest thing one can think of!
@@Luffy-um8jo So where are you stuck?
@@IndianPythonista bro i am making account checker
I have the same question--> Request Method: GET
i want to login to a website using requests but the login form is an can you explain me how?
Useful stuff.......
When I tried to do this to Amazon, the website gives an error message saying that I need to enable cookies. Do you have any idea what's going on?
You can try to send a request to amazon.com login page and save the cookies obtained in response and pass them in the login request.
@@IndianPythonista ??? The tutorial is seriously lacking without going through that step by step.
Looking forward to meet you @PyDataConf2018
Sure! :)
Hey man I tried this but the website I'm trying has 2 POST requests. One to login, and one to validate with a unique token. The problem is when I try to parse the URL it's using to 'validate' it says error parsing body. So I can get the unique token value. Any tips?
any idea how to do it with the form data when there is only the username, the password, and rd , _xsrf ? nevermind got it. awesome tutorial!
how did you get it?