Tutorial helped me download 1300+ product images in 15 minutes, awesome! Thank you. Would have taken ages otherwise. Edit: I (Mac user) ran in to the issue of the "URL using bad/illegal format" at first, i think it was because i copied the URL's from a Excel sheet to .txt file. After that i copied the URL's in to a new Visual Studio Code file, and then copied the url's from VSC to the .txt file. Now the shellscript would run. Not sure why it would work after those steps, but hope it helps someone out having the same issue.
That's what I like to hear. Always happy when I make someone more productive. About the error, it's the line endings. Happens when the text file has Windows line ends, carriage return + line feed. Macs are weird... legacy Mac stuff uses carriage return only, but now based on Unix since OSX, the underlying system uses line feed only. Kind of a mess. On a Mac best way to make sure it works perfect is use TextEdit in plain text mode. Or use terminal and pico. Then text files work in shell scripts on Mac always. Thanks for watching.
does not work, files are not in the folder. gives this error "curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log. curl: (6) Could not resolve host: pause"
Hi , I have list of psd files( Photshops files) in a folder. i need to replace the text of each psd file with the data present in excel and save it.. Can you please help
I have another script that might help. It had a different purpose originally, but I think what it does might work for your situation. Have a look at the video, then download and see if it works for you. czcams.com/video/NH9bYHnlUUk/video.html
I'm having the issue that every image is just titled "image", so they are just all overwriting each other. Anyway to get past that? Otherwise would have worked beautifully!
I'm not sure if it's something the server is doing or the choice of precise command when accessing the particular server. There might be a way around it with tweaks to the command syntax, but I'm not sure the details. If I'm not mistaken another user posted a comment about it. They had the same problem but found a solution. Look through the comments.
@@wc7 Having this as my code in the download.sh did the trick! counter=1 while read -r url; do curl -O "$url" && mv "$(basename "$url")" "image_$((counter++)).jpg" done < list.txt
Do the files have extensions? If so, what is the extension? If not, try adding one, .jpg for example. If that doesn't work try .png. What is telling you "unsupported"?
@@bharamappabandak9226 I don't do Whatsup or others like it. For me email works fine for decades now. Get me on my website and I'll reply via email: www.marspremedia.com/contact/
Nothing special for different file types. If a URL works in a browser, it should work in a batch/script file also. Check one of the URLs. Paste in a browser. What happens?
thanks for the tutorial...as a mathematician I am trying to cope my current knowledge with computer programing and data analysis so which computer languages do you recommend me to learn?
I'm not a mathematician so I couldn't say the best computer languages for that. Personally I love JavaScript but I imagine there are better languages for dealing with math. I also like C and flavors of it (C++, C#).
Mine isn't working. It gives a single file output named "export," and it contains text like "Temporary Redirect. The document has moved here." Do you have any ideas regarding this issue?
Hello, thank you for the tutorial. For some reason, when I attempt to run my .bat file I am told, "The revocation function was unable to check revocation for the certificate." Do you know a fix for this? Thank you in advance.
Has to do with the server SSL certificate check. Search web for the error message there are many results. If possible use http non-secure if possible instead of https. Not really anything different can be done in batch file that I am aware of. It's a restriction at the server hosting the files.
Hello, and thank you for the tutorial. Can be downloaded with the original image extensions because images have different types of extensions (not fixed extensions).
Images download with whatever extension they have. If it's not the right extension, there isn't any way for a script or other method to know or change that.
Thanks for this. I had a list of 200 jpg files in Jotform from a survey we've recently done. The script worked great, worked out some problems with " marks... but all the JPG files are there, but zero bytes. Any thoughts/solution, please? Phil
In the end, I used this command and it worked. xargs -n1 curl -LO --remote-name < list.txt The -LO did something smart to follow redirects, apparently. I'm not a coder...
Doesn't work with me. I do everything as shown and 2-3 minutes after I start download.bat it automatically closes without downloading anything. My list is 800+ URLs. Could that be the problem?
The number of URLs is likely OK. Probably something else is wrong. Add to the batch file "pause" command at the end, so the screen stays open to show any error messages. Sometimes the error could be the server and there's nothing different you can do. Some servers block this kind of download.
Hi, thanks a lot. But when I try on my Mac I get this: curl: (23) Failed writing received data to disk/application curl: Remote file name has no length! My OS is Sonoma 14.0
Hi, thanks a lot for this tutorial, I did manage to get the files downloaded, however, I got incomplete files and corrupted. The files are only 5kb in size, while I knew such files are over 150kb in size. Is there a solution to this issue. Thank you.
Sounds like a bad URL. What happens if you paste the same URL in a browser address bar? Also some URLs the server prevents downloading in this manner. Nothing you can do about that.
Hi, thanks a lot. I need to do two more things: 1) rename the saved files with a number (the number of the url's lines in the txt file should be fine) Better but more complicated solution: 2) if possible, grab urls from a list of html files (each html file has one url, always on he line 299. All html files are in the same folder and have a number as name). Maybe i could convert html files in txt files, but i have no idea how to say in DOS commands "take the url on the 299 line of file "1” and download naming the file as "video_1, take the url on the 299 line of file "2" and download naming the file as "video_2", and so on. Can you help me? Ps: if option 2 is viable, i have to say that line 299 has other code written before the url i have to grab! Thanks a lot
The technique used in the video isn't going to work on HTML files like you describe. You would need to manipulate the HTML first to make text files with the lines wanted (many ways to do this beyond scope of this video). About adding a number to the output name, I don't know of a way in this context. The -O command of CURL makes the saved file named the same as the remote file. There's no way (that I know) to load this into a variable so an incrementing number could be added. But if adding a number is all that is needed, there are plenty of tools to do that. Personally I use "Better File Rename" for this and many other daily batch file re-naming needs: www.publicspace.net/windows/BetterFileRename/
Most likely yes. You probably need to be logged in to access Google drive. Might work if a browser is logged in, but not for sure. Likely the batch file/shell script would have to authenticate with Google. That's beyond the scope of this solution.
Hi, Thank you for this video, but this is NOT working for me idk I have a win 11 PC, It did download the files, but all are unsupported extesnion is .jpg, but all the files have size of 1KB Please check
Likely it's the host not anything you're doing. They probably don't want users downloading their assets directly. There's nothing different I could do to make it work.
I don't know, never tried. If you have URLs to the files, it should work. If not, the answer is no, it won't work. The solution uses URLs to get files.
Works fine Windows 11. Make sure run on local drive. DOS commands in batch file won't work on network share. Add "pause" to end of batch file so window doesn't close then can see errors.
I am sorry but this is another useless tip. I have to download a several thousands of files. Legal files dont worry. I need some trick without urls copying, or some smart trick to extract a few thousands of urls which are hided under a few thousands of little download icons. DO You know how to do it?
⚠🚫⁉ William Campbell -----> My image URLs are from Facebook. It looks like my list is downloading but I end up with a single "photo.php" file in my directory. FB apparently uses .php in the image's full URL. Any idea how to fix this?! Thanks and great video!
The URLs are not to image assets rather to the the Facebook photo viewer PHP app, and then each URL provides a request (part after ?, and more after &) for the particular image. The only way to download photos this way is to click each URL then the three dots to choose "download". There's no way I know of to automate with a batch or shell script. Anyone else with experience can chime in.
Very helpful William you are a great , you made it so quickly and its tutorial is very very very helpful, Anyway, I appreciate your response a lot today.I just looked it up just now . I'll download and give it a try Anyway I can personally contact you on your e-mail Thank you
Tutorial helped me download 1300+ product images in 15 minutes, awesome! Thank you. Would have taken ages otherwise.
Edit: I (Mac user) ran in to the issue of the "URL using bad/illegal format" at first, i think it was because i copied the URL's from a Excel sheet to .txt file.
After that i copied the URL's in to a new Visual Studio Code file, and then copied the url's from VSC to the .txt file. Now the shellscript would run.
Not sure why it would work after those steps, but hope it helps someone out having the same issue.
That's what I like to hear. Always happy when I make someone more productive. About the error, it's the line endings. Happens when the text file has Windows line ends, carriage return + line feed. Macs are weird... legacy Mac stuff uses carriage return only, but now based on Unix since OSX, the underlying system uses line feed only. Kind of a mess. On a Mac best way to make sure it works perfect is use TextEdit in plain text mode. Or use terminal and pico. Then text files work in shell scripts on Mac always. Thanks for watching.
@@wc7 how to download csv file from any website automatically on scheduled time
Thank you so much for your help William Campbell. Really like your detailed explanation and clarity, anyone could understand it.
Worked, saved me a lot of time. Thank you William
Excellent sir, it worked for me too. Thank you William.
Thank you! it saved so much of my time.
Thank you very much! Huge time saver!
You sir are a humble genius! Thanks
Thank you very much from Holland!
William I cannot thank you enough !!! But I will try: many many many many ..... ...... many many thanks !!! Jeff
Great to hear. That makes effort worthwhile. Thanks for watching.
does not work, files are not in the folder. gives this error "curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log. curl: (6) Could not resolve host: pause"
Hi , I have list of psd files( Photshops files) in a folder. i need to replace the text of each psd file with the data present in excel and save it.. Can you please help
I have another script that might help. It had a different purpose originally, but I think what it does might work for your situation. Have a look at the video, then download and see if it works for you. czcams.com/video/NH9bYHnlUUk/video.html
I'm having the issue that every image is just titled "image", so they are just all overwriting each other. Anyway to get past that? Otherwise would have worked beautifully!
I'm not sure if it's something the server is doing or the choice of precise command when accessing the particular server. There might be a way around it with tweaks to the command syntax, but I'm not sure the details. If I'm not mistaken another user posted a comment about it. They had the same problem but found a solution. Look through the comments.
@@wc7 Thank you I will take a look through the comments then! Very appreciated
@@wc7 Having this as my code in the download.sh did the trick! counter=1
while read -r url; do
curl -O "$url" && mv "$(basename "$url")" "image_$((counter++)).jpg"
done < list.txt
@@neutralarchive137 found a solution for this? thanks
i like it, Thanks
Thanks for watching.
Huge time saver thank you
thank you sir.... gr8.... thanks again
Thank you very much!!!
You're welcome. Thanks for watching.
It's help me save a ton of man-hour, thank you!
Great to hear. Thanks for watching.
It works but I can't open the image it says unsupported 😢 pls help
Do the files have extensions? If so, what is the extension? If not, try adding one, .jpg for example. If that doesn't work try .png. What is telling you "unsupported"?
Thank you, you save my life
Good information ℹ️👍🏻👍🏻
Thanks
You save me! Thank you!
👌👌
Thanks.
@@wc7give me your whatsup number sir
@@bharamappabandak9226 I don't do Whatsup or others like it. For me email works fine for decades now. Get me on my website and I'll reply via email: www.marspremedia.com/contact/
amazing tutorial.
thank you so so much! :)
I have a long list of pdf files. I tried to follow these instructions and didn't get any pdf files. Is there a different code for pdf file types?
Nothing special for different file types. If a URL works in a browser, it should work in a batch/script file also. Check one of the URLs. Paste in a browser. What happens?
Its not working for me. On double clicking the bat file, it blink and disappear and nothing happened.
thanks for the tutorial...as a mathematician I am trying to cope my current knowledge with computer programing and data analysis so which computer languages do you recommend me to learn?
I'm not a mathematician so I couldn't say the best computer languages for that. Personally I love JavaScript but I imagine there are better languages for dealing with math. I also like C and flavors of it (C++, C#).
Mine isn't working. It gives a single file output named "export," and it contains text like "Temporary Redirect. The document has moved here." Do you have any ideas regarding this issue?
It's the server's response. Nothing you can do about it at this end that I'm aware of.
Hello, thank you for the tutorial. For some reason, when I attempt to run my .bat file I am told, "The revocation function was unable to check revocation for the certificate." Do you know a fix for this? Thank you in advance.
Has to do with the server SSL certificate check. Search web for the error message there are many results. If possible use http non-secure if possible instead of https. Not really anything different can be done in batch file that I am aware of. It's a restriction at the server hosting the files.
Thanks a lot.
thanks so much
Hello, and thank you for the tutorial. Can be downloaded with the original image extensions because images have different types of extensions (not fixed extensions).
Images download with whatever extension they have. If it's not the right extension, there isn't any way for a script or other method to know or change that.
It helps a lot, thank you🙂
Thanks for this. I had a list of 200 jpg files in Jotform from a survey we've recently done. The script worked great, worked out some problems with " marks... but all the JPG files are there, but zero bytes. Any thoughts/solution, please? Phil
In the end, I used this command and it worked.
xargs -n1 curl -LO --remote-name < list.txt
The -LO did something smart to follow redirects, apparently. I'm not a coder...
thanks a lot
Can I use this method to download videos instead of images?
I don't know. Try and find out. It may try to play them, not sure.
Doesn't work with me. I do everything as shown and 2-3 minutes after I start download.bat it automatically closes without downloading anything. My list is 800+ URLs. Could that be the problem?
The number of URLs is likely OK. Probably something else is wrong. Add to the batch file "pause" command at the end, so the screen stays open to show any error messages. Sometimes the error could be the server and there's nothing different you can do. Some servers block this kind of download.
Hi, thanks a lot. But when I try on my Mac I get this:
curl: (23) Failed writing received data to disk/application
curl: Remote file name has no length!
My OS is Sonoma 14.0
Check that -O argument is the capital letter O ("oh") not a zero.
thanks worked
Hi, thanks a lot for this tutorial, I did manage to get the files downloaded, however, I got incomplete files and corrupted. The files are only 5kb in size, while I knew such files are over 150kb in size. Is there a solution to this issue. Thank you.
I have no answer for corrupt downloads. Problems like that are beyond this solution.
@@wc7 noted on this. Thank you Sir 😊
curl: Failed to extract a sensible file name from the URL to use for storage!
curl: (3) URL using bad/illegal format or missing URL
Sounds like a bad URL. What happens if you paste the same URL in a browser address bar? Also some URLs the server prevents downloading in this manner. Nothing you can do about that.
Hi, thanks a lot. I need to do two more things:
1) rename the saved files with a number (the number of the url's lines in the txt file should be fine)
Better but more complicated solution:
2) if possible, grab urls from a list of html files (each html file has one url, always on he line 299. All html files are in the same folder and have a number as name). Maybe i could convert html files in txt files, but i have no idea how to say in DOS commands "take the url on the 299 line of file "1” and download naming the file as "video_1, take the url on the 299 line of file "2" and download naming the file as "video_2", and so on. Can you help me?
Ps: if option 2 is viable, i have to say that line 299 has other code written before the url i have to grab!
Thanks a lot
The technique used in the video isn't going to work on HTML files like you describe. You would need to manipulate the HTML first to make text files with the lines wanted (many ways to do this beyond scope of this video). About adding a number to the output name, I don't know of a way in this context. The -O command of CURL makes the saved file named the same as the remote file. There's no way (that I know) to load this into a variable so an incrementing number could be added. But if adding a number is all that is needed, there are plenty of tools to do that. Personally I use "Better File Rename" for this and many other daily batch file re-naming needs: www.publicspace.net/windows/BetterFileRename/
I get an warning message "File name too long" and it didn't download most files. Is there a way for this?
Use a shorter path to where the files are saved.
Am using Windows, it works but all my files are having 1 KB as file size, please help
It's the host and/or the URL. The host is sending back the asset at the request in the URL.
I failed to batch download from list of google drive files, why is that? is it because google drive prevent batch downloading like this?
Most likely yes. You probably need to be logged in to access Google drive. Might work if a browser is logged in, but not for sure. Likely the batch file/shell script would have to authenticate with Google. That's beyond the scope of this solution.
@@wc7 ok thankyou for the reply
Anyone know what would be needed if I wanted to change the names of the files which currently have the URL as names?
Rename the files after downloading them? Try Better Finder Rename (or Better File Rename for Windows)
Hi, Thank you for this video, but this is NOT working for me idk
I have a win 11 PC, It did download the files, but all are unsupported
extesnion is .jpg, but all the files have size of 1KB
Please check
Likely it's the host not anything you're doing. They probably don't want users downloading their assets directly. There's nothing different I could do to make it work.
Hi, will it work for music files from youtube music
I don't know, never tried. If you have URLs to the files, it should work. If not, the answer is no, it won't work. The solution uses URLs to get files.
How to use it to download audio files ? Kindly help
Same steps should work for any file type as long as the host has file extensions on them.
@@wc7 Thank you sir!
don"t work windows 11 here
Works fine Windows 11. Make sure run on local drive. DOS commands in batch file won't work on network share. Add "pause" to end of batch file so window doesn't close then can see errors.
I am sorry but this is another useless tip. I have to download a several thousands of files. Legal files dont worry. I need some trick without urls copying, or some smart trick to extract a few thousands of urls which are hided under a few thousands of little download icons. DO You know how to do it?
No I do not know how to do it. Search for "web scraping". That's what you want and it's beyond this topic of simple URL download.
⚠🚫⁉ William Campbell -----> My image URLs are from Facebook. It looks like my list is downloading but I end up with a single "photo.php" file in my directory. FB apparently uses .php in the image's full URL. Any idea how to fix this?! Thanks and great video!
The URLs are not to image assets rather to the the Facebook photo viewer PHP app, and then each URL provides a request (part after ?, and more after &) for the particular image. The only way to download photos this way is to click each URL then the three dots to choose "download". There's no way I know of to automate with a batch or shell script. Anyone else with experience can chime in.
Very helpful William you are a great , you made it so quickly and its tutorial is very very very helpful, Anyway, I appreciate your response a lot today.I just looked it up just now . I'll download and give it a try
Anyway I can personally contact you on your e-mail
Thank you
Glad you like it. Thanks for watching.
for /f "tokens=*" %%a in (list.txt) do curl -O %%a