How to copy large amounts of files in Windows
Vložit
- čas přidán 21. 01. 2022
- Robocopy is one of those commands that is often known about, but mis-used. Let's go over all the common options and do a little speed test.
Website Guide: www.christitus.com/robocopy .
►► Digital Downloads ➜ www.cttstore.com
►► Reddit ➜ / christitustech
►► Titus Tech Talk ➜ / titustechtalk
►► Twitch ➜ / christitustech - Věda a technologie
It should be noted and I missed it in the video. If you are copying TONs of very small 1-100kb files you can completely turn off logging with /NFL /NDL /NJH /NJS - This will 10-100x the speed, but at the cost of logging.
NFL = No File List
NDL = No Directory List
NJH = No Job Header
NJS = No Job Summary.
You can copy 100,000s to millions of small files extremely fast, but will have no summary or log.
is syncToy using robocopy?
IS galaxies still a game???
@@TinS0lder SWG Legends EMU is alive and well they just released city in the clouds
@@ChrisTitusTech can we robocopy it to a local drive and play it there? 😁
hey Chris, how's the performance of this vs Rsync?
Been using Windows for decades and never knew about robocopy, thanks.
This is so very much appreciated. I just want to copy everything on an older smaller drive to a new larger one... but NOT lose any of the dates in any way. Sounds simple... been hard. I will test my method first before I go "BIG".
Been using this tool, especially the mirror + copy all attributes to make backups and matching with logfiles to verify all the file copying ever since it was a downloadable tool for Windows XP. A really great commandline tool that really should have a GUI wrapper in the modern age for the less tech savvy.
You can double click the icon in the title bar to close a window.
Just thought I'd throw that out there since Chris was right clicking the title bar of a window with the controls off-screen, and choosing close.
Thank you!!! I had a NAS hard drive die - despite having multiple backups, the VMDK file on the drive was the ONLY corrupted spot on the disk! I loathed COPY and PASTE to another computer/drive, and this has saved my bacon!! Thank you!!!! 👏👏
Sounds like you also could have benefited from GRC Spin Rite, Roadkil's Unstoppable Copier, maybe even Test Disk/Photorec.
This was so helpful. Unfortunately 2 weeks late for me. I ran into issue doing a multi terrabyte copy for my NAS migration. I got around it by zipin gup the folders and unziping them to the new destination. Robocopy would have been much smarter and faster. This was awesome! Thanks so much for sharing.
With that much data you might want to look at something like imaging with Macrium Reflect free. That is a lot of data and if you don't have the switches right can be a difficult copy. Using Macrium Reflect might be a better option.
😂
The one thing I miss in Robocopy is what rsync have, the --backup-dir option so that "deleted" files are actually moved to another location.
I love this channel. I have learnt so many things from this channel ❤
I was using shared drives to access my NAS and copy files around. I recently installed Synology Drive to synchronize my laptop to my NAS. So far, so good. The only thing missing is a way to synchronize my Android phone to my NAS. I currently have to dedicate space on my laptop and use Syncthing to sync my phone to the laptop, then the laptop to the NAS.
My "big" file structure is about 300 GB (so far) of music albums stored as FLAC. There are all sorts of challenges with long filenames for classical music. Some people have overloaded the filename field with too much information. I shorten filenames when I rip the CDs. There are alternate character sets used for some albums, such as Blue Oyster Cult. These played havoc with my old Debian-based NAS.
I wish there was a way to put all my music in a database AND be able to play it back. The file system Music > Artist > Albums > Tracks is cumbersome at best. I actually added two folders to the structure to keep software from failing. I use Music > (A-M and N-Z) > Artist > Albums > Tracks. I may have to spit things again.
Sorry i might have missed it but how did you solve the max character limit in the path issue? Had this many times and only solved with mapped drives which as you mentioned is not a great solution.
YO keep these vids coming Chris!!
This just reminds me of how we used to copy files from drive to drive back in the day using 3.11 in DOS :)
xcopy? By any chance.
No such problems. RS232 to another computer. Then reset connection, restart machines (for the IBM), copy or cp depending on the DOS or Unix machine you access. What I miss though are the cassetes in Amstrad. And the time to go out to flirt chicks.
I switched to robocopy long ago. xcopy is good, but inferior
Ty for this helpful information; I always learn something new from your videos.
Nicely explained. Many thanks for sharing!
good vid. Thanks. Fan of your deep dive presentations vs the glossy overview.
Awesome tutorial - I am sure many will benefit, i will for sure.
Hello chris. Is There a way to use this method to copy files from my android phone to pc. I tried the method in vid but it cannot get the android folder path.
Dude, this is insanely faster than normal method. Thank you so much :D
Thanks for the great video! I didnt find a switch that would skip over unmodified files already in the destination folder. Or does Robocopy skips over by default?
Chris, Would this be faster and safer for copying 100’s of GBs of video from a Camera SD card to my SSD?
Thank you for explaining this tool. I will use it now. 👍
I dunno I would rather stick to Teracopy since it does all of what you say but wit a gui. Most probably would want to avoid command line for something like this and anything more important there is backup software better suited.
Teracopy has always failed for me on large copy jobs. I've had it simply skip files and not tell me and has caused me problems. Either that, or it simply crashes. It's a POS.
@@justsaynotoboomers I dunno I've copied hundereds of gigabytes of files across multiple computers and not once did it fail and warned me on corruption even resumed after accidently disconnecting an ethernet once. Sounds to me nothing more than user error on your part
If anyone uses lazy slang like dunno and wit when conversing with others, especially when already writing more than a few words, then anything they have to say is immediately invalidated. Have you even looked at r/DataHoarder? Its chock full of people complaining about how badly Teracopy has been failing on them, only within the last few years, even after years of flawless use. Perhaps you should not be so quick to accuse of others of 'user error' when you obviously don't know the situation, and can't even communicate intelligently.
@@othername2428hey man, people can talk however they want, it doesn’t invalidate their point. I agree with you on Teracopy being issue prone, but you can be right without putting people down and judging their choices. I’m on your side I’m just trying to get you to communicate empathetically. r/DataHoarders is always welcome at my house 😂
@@pcmv6832whew! That’s good, maybe have a backup plan or some other tool to verify it worked as well as it said it did if you continue to use it in the future. A tool is only as good/bad as our expectations of the work we assume it did.
Hi Chris. I need to simply copy, or migrate data from an old file server(2008 R2) to a new one(2019). I just need to move, or mirror the data from the old to new, retaining all NTFS permissions. The old partition is called "DATA", and I just need to dump all files, folders, sub-folders into the new server's "DATA" partition. Seems like there's so many options. What switches should I use for that? Again, basically just a mirror. Thanks!
there is any way to exclude some format? Let's say I wanna copy the contents of a folder, EXCEPT .mp3 files. How should I do it?
Hi, what is the difference between xcopy and Robocopy? I use xcopy a lot, it's faster than in the GUI, and it controls the files?
Thankx for breaking it down.. great tutorial for beginners
This works great. Small issue: when the transfer was complete, the only way I can access the folder is to type in the path in explorer, the folder I transfer everything to disappears for some reason once robocopy has completed the transfer. and it’s not hidden. I just created a shortcut to get to it. Just curious if there’s a way around this?
How does this compare with teracopy? Anything I’m missing as the average user?
Great video. Makes it a lot clearer!
i do have credentials for my (\\) folder , which syntax to be used ?
I've been using this since Server 2003R2 days. Once small caveat to the video is that there are times when you may want to turn off security all together. Like if you're copying user home folders from an old file server to a new one on a different domain prior to a user migration. I also might add that using robocopy in a scheduled task with a service account works well if you want a copy of something to go somewhere else to be backed up. No not as a backup itself, but a copy to be backed up there too. It's a powerful tool, and one that so many admins don't use or forget about using. DFS has gotten better, but it still has quirks (child/forest file copies are garbage), where robocopy will still succeed every time.
Indeed. Just remember it cannot copy any open locked files. Not geared for the same purpose but TSCOPY is an excellent tool that can copy any locked file including the hive files.
So the reason my copy n paste keeps crashing my pc or aborts midway is because i go over some limit and it cant handle that? I have been frustrated as hell doing 500 latge files one by one because it keeps crashing and blue screen and what have you...
Very good video. Thank you. Can I also access my internal phone storage and copy videos from tis location usin robocopy. It shows up as MTP-device and doesn't have any drive letter assigned? Thanks very much in advance!
if you copy to one hard disk what's the point of using multiple threads? also in large files will it cause file fragmentation?
Very informative! Thanks.
Robocopy rocks. Have been using it since it was a part of the Windows 2000 Resource Kit. Was pleased to see it promoted into default install of Windows. One useful switch is /XO (exclude older files). Prevents an older file from overwriting a newer one.
Also, if you don't use the in-built logging function and just pipe the output into a file yourself with >, switch off the progress (NP). If you don't, you'll get loads of percentage counts in your log file and it looks a horrific mess!!
Moving an entire 3TB drive to another local drive.
Needing to keep ACLs, time stamps, etc
Would I need /ZB at all? In case my permissions are not sufficient?
Also, do I need /COPYALL and /DCOPY:DAT ?
Is there a way to verify file integrity to make sure everything copied correctly and without corruption?
So... like _rsync_ for Windows? Is _rsync_ available via WSL/WSL2?
Not gonna lie, I tought that I would never use this when I saw your video, until I had to backup my sister's external hdd.
This is a god sent, the only thing that I miss is a general progress bar to the whole process...
It sounds like a small thing, but I'm copying a bunch of stuff right now. It's been going on for hours and I wish I had at least a rough idea of where it's at in the process.
Hi Chris, thanks for this video. I will be helping a relative transfer basically everything on her old laptop to a newer one. Is connecting them with a crossover Ethernet cable and then using robocopy the best way to approach this?
Her old laptop is win 7 and the new one is win 10.
Windows Explorer copy and paste on my PC seems to work faster than RoboCopy utility somehow? For instance, I am getting about 113MB/s with the standard explorer copy, and RoboCopy gets roughly the same, but sometimes lower. Another example, between my 2 NVME SSD's, Explorer copy gets 1.2GB/s, and RoboCopy gets ~800MB/s so about 1/3 slower than just normal copy and paste. Any ideas why this might be?
well i had that file name limit error like 10 days ago when i was backing up my wsl files from the backup folder in windows to a hard drive
and if you are wondering why i used the copy paste as everyone does ? - cause i just testing with it
i had another tar.gz file backup as well
This was excellent. Thank you.
Liked the deeper dive into windows powershell
Which one of the following commands is preferable for copying shared files from an AD serv to another (Win servers)
Robocopy Source destination /DCOPY:T /COPYALL /E /R:0
or
Robocopy Source destination /E /ZB /copy:DATSOU /R:3 /W:3 /V
does it do checksumming like rsync?
I want to copy file from downloads to desktop can I do it through robocopy?
Any ideas to copy deduplicated refs? I have about 30tb to move to another synology, and it is painfully slow... I am open for ideas :-)
Thank you very helpful!
thank you for this lesson on robocopy. I did a test, copying files from my C: drive (all of which have dates and times in the past - relative to 4/17/23) to an external USB drive that has bitlocker on it. INITIALLY, the dates and times on the destination drive matched the source drive, but after a minute or so, bitlocker must have done something to them because the dates and times on the destination drive all changed to today's date and time. Are you aware of this occurring with bitlocker? Is there a way around what bitlocker is doing? thanks
Owman Chris, you can't stop yourself hu? In the end you give (even though you leave a lot out) so much options, I still don't know what to do.
Thanks anyway.
👍🏻Greetings from Netherland✌🏻
I like your tutorials Very much
cool, informative, educational, thank you kind sir
where r those backup maintainance robocopy scripts at?
I am going to try this, and you may have changed my life.
Will this fix the issue of copying files to an sd card?
I'm copying a photo directory from my hdd to a microsd via a usb cable plugged into my phone. I've copied a large amount of files but noticed the amount of files did not match the hdd. I selected all the files on the hdd and pasted onto the correct directory in the microsd, a conflict error pops up. If you choose "don't copy" and "do this for all conflicts" the copy process will just end. It won't copy the missing files.
Loads of topics on microsoft forums but they have all been closed with the only solution being to run a sfc scan. Very frustrating and I'm sure I've lost a lot of photos over the years because of it.
edit: nvm, can't even connect my phone to win10 to assign it a drive letter, sigh. Are we living in the future or the past?
SUPER CONFUSED: I am trying to do:
robocopy "D:\BACKUP FOR HDD REPAIR\DASHCAM\" "F:\DASHCAM\" /e /r:0 /w:0 /COPY:DAT /DCOPY:DAT /mt:12
but it keeps saying there is no destination specified????
Why do the miliseconds on the timer go only up to 30?
The GUI copy will not preserve security settings most of the time, usually just replacing the source ACL's with the target folders ACL instead. Very bad if you need those ACL's preserved intact.
I used robocopy extensively to migrate a file server with over 2 TB of roaming profiles years ago. Did one pass to get the bulk data mirrored to the new system (ran overnight, took hours), command saved for later use. Later we just ran it again and again to get the modified files (took minutes each time). The ability to save logs each run helped us sort out a number of access issues reliably in the end too (use /NDL and /NFL to only log errors and the summary).
You can also use it (in rare cases) to salvage data from unreliable systems or drives as it can resume or at least skip files already copied. Had to do this ONE TIME to get data off a system with failing raid drives (lesson here is to actually check your raid and hardware logs regularly and not wait to the alarms to sound, yellow and red drive lights don't always get noticed). We managed to save all but two files if I recall correctly but it was touch and go for sure.
Great video!
If system crashes during robocopy is in progress, some time source folder gets corrupted. I guess it's due to robocopy using mft directly.
I feel like for normal day to day file copying tasks this would be kinda overkill. In the example you gave of that 5GB file you saved roughly 10-20 seconds. It would take that long just to type everything out for the command thusly not saving anytime at all. Now I can clearly see the huge professional level applications for such a thing for sure. Just not sure if this is worth it for the average user. Thank you for teaching me something about windows I didn't know though! Really enjoyed the video nonetheless
Yes he said it's for large files, and mostly for big servers and corporation not the average user
Let's say you copied 500GB of data to another drive and it was going to take 1 hour to complete. You copy/paste and walk away and go grab some lunch. You come back only to find that it did not finish copying because it quit for one of the many reasons Chris spoke of. So many different things can stop the Windows UI copying. With Robocopy and using the right switches that is never going to happen. It will always complete the copy process.
@@surfingsub5854 ok yeah I can see that being really useful. Peace of mind is always a good thing.
I work in migration projects and use robocopy a lot for copying more than 500g of data.
Ex: for copying 2gb folder which contains 2m files in it.
U don't know the pain to wait for copying 2gb of data without robocopy..
Robocopy saves the day!!
It has nothing to do with "the average user". if the aver user has 5gb of normal data, then duh, not worth it. if LOTS of data or like kiran said "2gb folder which contains 2m files", then duh, it is worth it.
How i can trasfer file from windows to android device ?
Why not use Teracopy or other software like that ?
A nice use case is moving game save files to dropbox for those games that Steam does not sync the saves for you. I've got a task set up for that.
Sir, i have been facing an issue, I have an SSD which has around 70 gigs free space, quite recently I was trying to install God of War on it which is just 60 gigs, the problem is somehow while installing or copying from my other drive my entire memory gets full and after restart I think it flushs some archive file and that time I again retain the space.
A similar thing happened during the installation of Cyberpunk 2077 (at that time I had 80gigs free & CP2077 was around 58), for that I had to transfer files in segments, I transferred 1 archive at a time from my HDD which tend to occupy double the memory while transferring and after restart it gets fixed.
So can robocopy help in this or is there any other way.
Thanks 👍
There is a RSYNC that runs in the cmd window WITHOUT needing the Linux subsystem installed.
*"...can use all 16 cores.."*
?? Okay, but where's the BOTTLENECK usually?
I would assume the bottleneck is usually the read or write drive. And if you're reading from, or writing to an HDD wouldn't trying to access multiple files at the same time actually HURT performance because the read/write head is jumping around?
And if you're reading from or writing to an SSD which is RAM (as in Random Access Memory) then what's the point of trying to access files simultaneously?
I need some assistance if you can help me. So I'm trying to copy USB to USB using the robocopy method but I'm having a problem. I don't know if this is correct or not. but if you can tell me how to do it correctly? robocop E:\Karaoke CD (02) D:\Karaoke CD Part (02) CD Drv 01 E:\Karaoke CD (02) USB 02 D:\Karaoke CD Part (02)
Does it support for windows 10?
thanks alot for your decent effort,
What program do you use for screen casting and especially how do you insert yourself in the screencast with cutting everything behind you?
OBS for recording with a Green Screen w/ Alpha Layer
@@ChrisTitusTech Ok, thanks
Nice video.. Thank you I learn new everyday from you..💕💕
True but if you're not rsync'ing or btrfs send'ing and snap'ing from your NAS what are you even doing
Is there a GUI overlay for RoboCopy? That might be more comfortable for many Windows users... Also, how does TeraCopy, and other third-party software to replace Windows default coping service stack up to Window default service, or against RoboCopy?
ChoEazyCopy exists its a gui
Does Robocopy GUI still exist? I searched for it, and Microsoft claims that it exists, but the links were dead-ends.
is the a GUI version of this?
Why should multi threading be the factor that speeds the transfer up? Any modern single CPU thread can handle file transfer io streams.
Thanks, appreciate the info.
This is amazing I just had to sit and listen to
This video enough I wanted to take over 20,000 Files off my external hardrive and put it on my pc very fast if u sit down and follow these steps u will be fine use your note pad to write the codes first so u can copy in post in command prompt without feeling discouraged lol I didn’t need to put my network Ip and it worked
Great video! Does robocopy work with Linux server and Windows client? Is this essentially scp/rsync but for windows?
If your Linux server has samba, then yes. Synology is basically a linux server and that's what he used in this example.
Yes, rsync has some methods that robocopy does not and if you're copying from one network to another and can use rsync that would be better
Great tut! Don't need anything fancier.
Ah yesm the wizard again. Thanks, your videos are Goodstuff!
If its multithreaded does it cause the destination files to be become fragmented?
From what I've read investigating this, it pre-allocates space for each file.
Does windows home have robo copy?
So it does not verify the copied files?
This video came in the right time
I use fastcopy and it has a GUI and it syncs files.
Fastcopy rocks!
I also use Fastcopy. Most people have no idea what it's capable of. Can even shut down the pc when done, do wipe deletions, etc etc the list is long.
I use Teracopy because I can verify the files with different Hash functions. I usually pick SHA-256 or SHA-512
@@apreviousseagle836 Can almost do the same with Fastcopy except Terracopy has a couple more algorithms
Can you simultaneously copy files from say an SD card to both an SSD for fast video editing file storage inside the workstation & a NAS for long-term file storage with Robocopy?
I'm not Chris, but as far as I know, no. You can have multiple robocopy scripts running at the same time, or you can script them to copy one after the other. Say, copy to your SSD, then from there to your NAS.
The 255 path limit is a bad default setting of windows explorer. You can enhance that using a registry key. But yes i also forget it always…
Very good. Thank you
Hold on a second. Why exactly are we not supposed to use this when we have mapped a network drive? It looks like you are using iSCSi? I use SMB via my TrueNAS - I have been using robocopy pretty regularly and saturating the connection 110-140Mb/s. What is the danger exactly?
Excellent subject…… one week short lol. That explained all the duplicate files and why nothing could be found. Even when I located it that did not transfer to the very next file in same folder. Still going through all 7 ssd’s and finding whole directories 3 or 4x deep under user/name/etc. dupchecker helpful as long as you tell it where to look. I wish it would scan everything everywhere and show which are complete and working and which can be trashed. So far though I was able to get back 6Gb of space. All I wanted was keep some important program files and empty each drive start with reformatted and clean pc. That didn’t go as expected. Lol.
Robocopy cut/moving in same directory is very slow. It basically copies the files first and then deletes it. Very annoying.
Saw the title and instantly knew you were talking about robocopy. Use it all the time mainly for the MT option and especially with mapped drives at my enterprise. You mentioned UNC would be faster (we use SMB shares). How? UNC via file explorer is still single threaded mostly and via command line I just get UNC paths not supported.
Give best solution for transfer file between windows, linux,mac(Cross platform)
Thank you!
I mean I am giving you a like before I even finish for Star Wars Galaxies alone!
I have to wait 4-12 hours for copying files from my android phones to windows pc because they slow down a lot. There are so many files in the folder, that it hangs in most file explorers on android. The windows explorer even crashes many times while I copy many of the files but the files are at least somewhat viewable before copying in windows explorer before copying unlike on android.