Note: If you don't like the AI translated voice, you can change the audio track back to English with the "Audio Track" option in the gear settings on the video player. • Also, if you happen to notice any common terms that are repeatedly mis-translated in a particular language, let me know and I'll try to have it correct next time.
Somehow for the first time, the German title and audio track isn’t preselected and I like it but also don’t really know why. I like the real English voice more anyway so it’s not a problem for me. Probably it’s a tweak from uYou+ (the tweaked YT app for iOS I’m using to watch the video).
A more realistic way to simulate bit rot is to open the file in HxD and fill selection with random values (right click > fill selection > random bytes > OK, you can even do multiple passes if you want). Your method is fine, but you are reducing the number of bytes instead of corrupting them.
As someone who literally built a storage server to combat bitrot but did NOT know something like this existed, you have my eternal gratitude for helping to provide yet another data integrety safeguard to my toolkit. Thank you, a thousand times, thank you.
I used to use this program back in the early 2000's, when I used to download stuff from newsgroups. I always thought that "PAR" stood for "parity" files, rather than "Parchives".
I was about to make a comment very much like yours here, so I'd agree 100%. I had lots of usenet servers regularly corrupt parts, so the use of extra parity files was truly essential. I'm also glad we're past that and we have things like BitTorrent (which verifies the data is correct), and filesystems (like Btrfs) that can at least detect corrupted files and give RAID style options (using parity) to recover.
@@H4KnSL4K I don't think it was so much that the news servers corrupted the files, as it was that entire files went missing due to news server retention rules. Of course if you used a huge news server like GigaNews, retention wasn't so much of a problem. But local ISP news servers had tiny storage and low retention rates.
Excellent video. I also learned that the Reed-Solomon Error Corrrction Codes were first introduced in 1960 (Irving S. Reed and Gustave Solomon). Amazing
Before using any drive, make sure to test it at least once (I do multiple times) with a program like h2testw which fills a drive with data and then verifies the integrity to make sure the drive is functional to begin with. I've bought a new SSD before and it already had blocks it couldn't read properly, so good thing I tested it. Obviously a drive can go bad at any moment though.
One of an IT worker's pet peeves is a file that you spent hours on and made lots of progress on getting corrupted. Thanks for giving us a course of action at our disposal in case it happens!
@@shsphotodesign you may store the recovery files in a different location but when you want to recover a file use must move then all back to the same folder.
Funny how a lot of my docx and pdf files got corrupted a week ago while I was trying to copy files from my broken Windows to a pendrive from a Linux live USB. This will be helpful. Thanks Joe 👍
I cannot explain the things I have learned from this youtube channel. It has been by far the most useful channel/page/anything for my overall user experience. If I had not found this channel, I wouldn't know about the program Everything that I use every day, about Power Automate in Windows, and so many more things. And now this! Thank you ThioJoe! Keep it up!
I remember using this more than 10 years ago. I downloaded a big file and it got corrupted during transfer. The guy I got it from just sent me a .par file and he told me to use it to repair the file I downloaded. I was so amazed that the little par file he sent me indeed was able to repair my file. I verified this by confirming the CRC and md5sum is correct. That was my 1st encounter with par files and I was glad to learn about it. Nowadays, I use btrfs file system to resist bit rot and corrupt files. I also do regular backups. Haven't used par files in years but it's nice to know it exists. 🙂
It is important to note that you cannot re-use recovery files. Say you need 50 recovery blocks but you only have 44, you cannot recover it. That is why you can adjust the amount of recovery files and blocks. You could have 4 recovery files each with 50 recovery blocks and send 1 of each to your friends or family, and then if you need more just ask them for the recovery file to get the amount of recovery blocks you need. You can also have over 100% redundancy. It all depends on how important the file you want to recover is to you.
What do you mean "cannot reuse"? So if i recovered my files, i need to recreacre theese backup stuff again? Maybe you have a link to some article with details how it works? Thanks!
@@exercitus8535 shouldn't they be the same? Or they meant that if you have 4 recovery files and you need 8, you can't copy the 4 recovery files and get it recovered (in other words, every recovery file is different)?
@@exercitus8535 so you can copy the recovery files themselves and get 2 sets of them? Like a complete set and then copy that into another complete set?
A quick way to think how error correction algorithms behave is to think about a sudoku game, you can start by having a complete sudoku filled with numbers, and if you delete one number or change it, there are many ways to recover that missing number. You can even have many numbers missing and you will recover the full state, but past a limit you won't be able to do so. I've studied the hamming error correction algorithm at college and it really was similar, the way to recover a bit was to do lots of boolean operations on the data, crossrelated. The most useful thing of these softwares is that the redundancy files are only a fraction of the size of the original document, but they really can fix any error on any of the bits of the source file. The limitation is if you have too much errors, that's when you exceed the "hamming distance" and the program can't recover, but at least it can tell you there are errors. By increasing the size of the redundancy files, you are increasing the number of possible simultaneous correctable errors.
Winrar is very convient for ease of use but this provides interesting use cases such as creating a par file for an entire 1TB+ HDD and using a much smaller portable drive as redundancy, meaning although not as good as a backup you could cover far more files with that limited space or mix and match, have your photo backup and then give a low percentage redundancy for everything else.
This is essentially how raid works. A disk is assigned for storing parity files (Par) across all drives in % or a dedicated disk for all parity. Depends on the raid level you chose. You are merely emulating this concept on a file basis for backups.
Great video, ThioJoe! I have been using PAR2 files for transfers for years, but never even considered that they could be used to protect against bit rot! Genius!
I've never used this, but not mentioned is that this is really for files that don't undergo change. So yes, it works great for a set of media files, but its not going to work on all your Word documents or spreadsheets that you are updating. This can't distinguish between rot/corruption vs. a legitimate update. You'd have to regen the Par files whenever you change anything in your set. Raid storage will do this all on the fly, but if the corruption is done at a high level (in front of a disk write) then the corruption will be taken as a good change from RAID's point of view. Seems to me it makes most sense if you make a backup of a bunch of data, then use Par to make that backup resistant to rot. I'm sure some backup solutions do this automatically.
To add... creating PAR files are useful when storing lots of files in zip/7z files. Those types of files can lose their a huge chunk of their contents from a tiny corruption since one bit sequence can be used in 100s of files. Using a tool like keeps those small bit flips or corruptions safe and protects the entire archive. The Par files should never be compressed and don't compress well anyway.
Just today I opened an old-ish project off an SSD. Errors all over it. Never in my life will I use SSDs again. Going back to spinning disk. Thanks for the heads up about Multipar! That would've been a life-saver in a lot of situations where the audio in old projects, or the project file itself is corrupted.
lmao no you just got unlucky ssds are pretty good, and also no moving parts, which means no damage from shaking or whatever HDDs are PAINFULLY slow, I'd rather have an SSD that's actually worse in other ways just for the extra speed
@@Xnoob545 SSDs lose data during to bit rot at a much faster pace than spinning disks. It's due to electron traps leaking charge and getting into a new state. Save a coupe of TB of data for a year or two then go back to it later on. There will be countless errors upon reading it, whereas reading off one of the spinning disk backups, very few errors are present (files backed up beforehand). A RAID 5 with spinning disks can handle enough throughout for most tasks (video/audio editing), but SSDs are not the best for professional use without at least some form of backup solution.
Good that you mentioned WinRAR for its Recovery Module! It has saved my archives several times already; it should be a requirement for actual archives to retain integrity of the files. I've been curious if one can do that in normal, non-RAR'd files; finally, this answers that question. Thank you!
Any of the recovery data can be used to repair any part of a corrupted file, doesn't matter what's located where. This is actually something called Reed-Solomon Error Correction; and as far as I'm concerned it's absolutely magic. - ThioJoe 01-18-2023
I remember using the recovery function once in WinRAR for a large file that had been compressed into multiple archives and one of the archives was missing. I couldn't understand how a recovery file that was much smaller in size than the missing archive was able to reconstruct it. I used the same term as Joe did, I said it's like magic!
Terrific video. I was just using Beyond Compare the other day to bit-wise scan hard drives with backups I made +/- 10 years ago. So far no bit rot detected. Pretty cool! But this is another great technique for protection.
- For a moment there, I thought throwback-thursday was back. But it's nice to see that PAR has evolved in the past 20 years and is more usable than ever. 👍 - Parchive files are like recovery-records from compression software, but for regular files. … 11:57 Oh, never-mind, Joe mentioned that. - 12:33 Forsooth, not only does the 7-zip format itself not support recovery records (which is catastrophic for solid archives), the program doesn't even support them for formats that DO support them like RAR and ZIP (I think ZIP has it, it's been over 25 years since I used WinZip 🤔). I think Igor just doesn't know how to handle them. In fact, people have been using PAR2 files as a work-around for this for years. 🤦 - 13:30 It is handy, especially when dealing with flash-media (flash-drive, memory-cards, etc.) which are unreliable and Windows won't even necessarily give you a warning (I've transferred files that SEEMED to transfer just fine but when I tested them, they had corruption, that's insidious af, especially if you don't realize it until after the corruption makes it into your backups!!! 😬) - Now if we can just get a proper, efficient, user-friendly, and free (*cough*rtpatch*cough*😒) patching program… (remember PPF files, Playstation Patch Files by Icarus of Paradox?)
I wish there was a way to add this on the filesystem level easily, so the system would just do a sweep of every file on intervals checking for corruption and auto-repairing without needing a RAID array or something else fancy. (Depending on how frequently you did sweeps / how often the system was kept on you could probably get away with a tiny amount of redundancy too since youd only be repairing a tiny bit at any given time due to bitrot)
There is! There’s dedicated file systems but IIRC they aren’t just drop in replacements or compatible with a FS that a modern OS would use for a system drive.
@@ckingpro so, it cant do it. Obviously you can just copy the data but that doesnt help. At that point just buy two drives, your already halving your storage might as well prep for drive failure too
@@felixjohnson3874 but I just showed you it is a thing without multiple drives. And distros and BSD based OSes also run regular scrub to check for corruption and repair it
@@ckingpro sure its without multiple drives, you only lose half of your storage space, yknow like what would happen if you just bought several drives and put them in the least effecient RAID configuration. The one difference of course is that you cant recover from a drive failing.
Those PAR2 files were sooooooo usefull in that era I used Usenet (newsgroups). It still exist and without those PAR2 files, Usenet is almost unusable (even the best paid Usenet providers have currupted files).
This is what i immediately thought of aswell. Quickpar, WinRAR and newsleecher or grabit. Later versions of newsleecher and grabit did the repair and unpacking automatically even. I'm happy that i don't need usenet anymore, but back then it was great for certain things.
Thank you for the highly educational and informative video. Well-done. 👏 What if you want to create a Par2 for multiple files at once, but you do not want to group them? Is it possible to do this, or would someone have to do, for example, 100 files individually?
I used quickpar a while ago one thing I liked is if you needed 1 extra block instead of figuring out which blocks you didn't have, someone with a complete file could create you a new block starting at the last block number that's why the file names have those numbers
PAR2 uses Reed-Solomon ECC. It does not matter which recovery blocks you have. It matters only how many you need. You can use any recovery blocks to repair the same number of corrupt blocks. That's why they are dividing the blocks in separate files. If only 10 blocks are corrupt, you only need 10 blocks of ECC data. it does not matter from which par2 file.
@@johnsmith8981 Hmm i think I'm gonna check this Acronis software out 🤔 btw my brother cracked the macrium software with pro edition so it's safe for me atleast. But you're right, shouldn't download pirated software.
@@jj1322 your brother is just downloading a cracked copy from someone else online. You're not just trusting your brother you're trusting whoever it is they are getting it from probably a public torrent.
I will this instant(tomorrow it's night rn) put all my family images and videos on my backup hdd in a winrar (may be multipar I'll see) Thanks bro i have lost some videos before to corruption so really helpful 👍
if you have a Synology NAS this is included if you use the active backup for business for full system recovery and files too along with time machine backup for mac just restore the file you want
Thanks so much for this video. I believe this program has a lot of potential, I know it would have been a premium software back in 1990-1995. Unfortunately there is no website that talks about how wonderful their software is and what to use it on. Also unfortunately, you seem to have the only video on this software that has spoken english (as opposed to another language or text on screen with music playing). I would love to watch a video that talks about tons of very specific files this should be used for for everyone and for specific uses for advanced computer users. I'm just not sure what I SHOULD use it for and can't find any other videos or people talking about how they used it to give me ideas. I could go through my computer files and check if there are certain window files that typically get corrupted and don't get updated. I could go through my personal files and try to decide which i should use this on. However, I would rather someone else figure it out for me and give me suggestions that I would have never known. I'm not saying that YOU need to do this, but if anyone wants to give me suggestions as replies or make your own video, I would love to know about it. Also, knowing what setting to put it on for each use would also be important.
Joe, at 5:01 you say "Recovery File Complete", but the actual note says, "1 Recovery File Incomplete". Although the result was what you expected, was concerned this might have repercussions late on down the road.
I'd heard of bit rot and data corruption plenty of times but never really took it seriously because "It'll never happen to me" A few months ago I went to play Pokemon Moon again on my homebrewed 3DS and discovered that every single game had rotted away. Thankfully I still had my backup from the homebrew process, and I now keep everything on a NAS running a ZFS mirror. I still need a backup but it's a step in the right direction
I must say that losing a drive is a frustrating thing to happen, especially if the drive has your OS on it. I lost part of my system drive, which is essentially Optane memory, connected with an SSD via RAID. Now, Optane doesn't actually contain any parts of system, it's essentially used for caching purposes. Seemingly unimportant chip, but once it died, system won't boot anymore, and it's impossible to recover. Apparently losing the cache messed Plug&Play drivers up and system won't recognize the drive anymore. The only thing you can do is mount the system SSD into another machine as a non-system disk, backup any data that you don't want to lose, replace or simply just remove the Optane memory, wipe the SSD clean after the work is done, and mount the SSD back in. Then if you removed Optane, you need to reconfigure the drives array to not use RAID (if you do replace Optane with a new one, skip), and then insert install media, and proceed to install fresh copy of Windows onto the drive, then put your data back. However there's something neat about it, and that is that removin Optane frees up NVMe slot, and you can actually insert a new SSD into it. Then you can install Windows into that, and then you don't have to backup your data from the old drive. Although I would still do it, just so you can get rid of the old system partitions and reformat the drive so you can use maximum of it.
This video will only download in Japanese, I've tried to change audio to English but it still downloads in Japanese. Can you repost another video of this just in English, with-out other choices. So it will download in English. This is such a great video and I'd really like to get it in English to reference back. It is a lot of information, truly have to be able to go back remember. Thank you for another amazing video.
I'm trying to understand the use case for this. It saves on space, but at the cost of file confusion, i.e. you can't look at those par files and know what's in them. With storage being as cheap as it is, what's the real advantage of this over simply making a double duplicate backup, particularly if compression on those backups are used?
So, if I understand this correctly. You can create the recovery files onto a CD/DVD. If that particular file/set of files gets corrupted on the actual storage, you can pull that CD/DVD to recover the missing/corrupted data? Seems a nice option to have.
Doing fast backup into tar.lz4 or wim is just better. Its like twice the data (duplicates), less the space, at multiple locations, different partition, different drive, external drive, NAS, online storage, different PC... TAR are good old "Tape ARchive" that stores lots of information (like user permissions, hard links, security, etc.), proven multiple times, and are sequential, puts everything into a single file without compression, while LZ4 gives 500MB/sec per core or twice more if PC are last gen. So with new CPU and 16 threads you get like 16GB/sec compression speed, limited by network, disk, RAM, or whatever else, while decompression are limited by the RAM speed. You can replace LZ4 with something else (ZIP, 7Z, RAR, LZO, GZIP, whatever), depending on your needs, like what kind of data and choose between compression:speed ratio. I prefer LZ4 because it gives best ratio across various files, best compression for a given speed/time, it reads fast, extracts even faster, multi threaded, pretty quick and efficient. Google are using Brotli, and its little better compression, but takes way longer. Since its a big sequential backup, old HDDs have problem reading that (just for review the files and extracting a specific one), but for SSD, to read sequential big data, just to see what files are there, like millions of them is not a problem anymore, it just goes trough few TBs of data in second. WIM are new "Windows IMage", made by Microsoft, same as TAR, but it has generic ZIP compression and has some similarities with "image" file (like VHD) so old HDDs does not have sequential problem when reading big file containing millions of files. People are not using WIM because its not yet proven like TAR, but i think its generally good.
Hello Theo ,please make it larger what you are showing on the screen.can’t be read and don’t switch off VERYVERY quickly. Thank you for sharing greatly appreciated, lovely xxx
I've been using that winrar trick for a while now, because sure you can have backups, but what if a file randomly becomes corrupted and gets copied over to your backups without ever noticing? Now your cat pics or whatever are just gone even with proper backups.
If you PAR a file like a spreadsheet and then change the file, do you have to manually update the PAR or does it do that automatically? If so, how does it know if you changed the file or if it became corrupted?
I had forgotten about this program, used to need to use it a lot back in the day.. I just now downloaded it for mainland use, don't really need it for sea use anymore.
Tickle me a "normal" windows user who is just getting into larger and larger backups... What is the Pro of this vs just having dedicated drives for backups? Decreased time for replacing corrupted files? Less Drives Required to keep archive backups? I am aware of the bit flip from the ..uh ..space particles messing with the electrons. But this is very rare , and from the data I have seen on the probability calculations....simply having your archive drives ...10 feet from one another ought to be more then enough to compensate for this...SPACE! Its BIG! Internet, please enlighten me, thanks!
Just one thing I noticed: you say "one recovery file complete..." yet the message in the green banner (at 5:01) says "1 Recovery File Incomplete". Does this matter, or is it not an issue? By the way, this is wizardry! My mind is truly blown!
This thought just came to me: what if you used this on a Excel or Word document and then later used this to recover the file if it gets corrupted. Wouldn't you lose the changes you made since you created the .par2 files? Perhaps only use this on files that should never be changed and keep proper backups on your documents.
Hey, @ThioJoe . I ran the installer of the latest version of MultiPar (from your Github download link) through Virustotal and 5 out of 69 anti-virus software detected a trojan. Not sure if this is a false positive as each of these detections come from software I've never heard of. Kaspersky and all other popular anti-virus software say that it's fine. Do you think it's safe to use?
Doing a real service here, it's not like Thio hasn't already got time stamps in the description 🤔🤨 ▼ Time Stamps: ▼ 0:00 - Intro & Basics 0:48 - Demo: Corrupting and Repairing a File 2:48 - Demo: Multiple Corruptions in a File 4:29 - Demo: Corrupting the Recovery File 5:33 - Demo: More Details on How it Works 6:21 - Demo: Replacing Completely Missing Files 7:51 - Demo: Explaining MultiPar's Options 11:38 - A Similar Feature in WinRAR 12:43 - Alternative Programs
Thanks for the video, this helps a lot One question: does this still able to recover files that are modified after creating part2 for them? Let's say i create a Photoshop project file that i modify often, save and quit! Or should i create part2 file for it once i finish working on it?
Hello, thank you for sharing this program; I've been looking for something like this for quite some time. But I have a question: what should I set the media size to? I'll primarily use Multipar for backup archives. Thanks
Note: If you don't like the AI translated voice, you can change the audio track back to English with the "Audio Track" option in the gear settings on the video player.
• Also, if you happen to notice any common terms that are repeatedly mis-translated in a particular language, let me know and I'll try to have it correct next time.
I like how the Italian Ai translated voice doesn't recognize english terms like "File" and reads it like it's "lines".
lemme try Turkish one
it's speed is slow or fast
Somehow for the first time, the German title and audio track isn’t preselected and I like it but also don’t really know why. I like the real English voice more anyway so it’s not a problem for me. Probably it’s a tweak from uYou+ (the tweaked YT app for iOS I’m using to watch the video).
Can you make it not start by default, it's anoying to have to change it back to English every time?
As a data hoarder, i wish i had known about this software sooner!
You should look into a NAS with ZFS
Sadness
Heh so, so many Linux ISOs ;)
I have lost data to bit rot before. I cannot overstate just how useful this tip and tool will be in my arsenal. Thank you... Thank you so much.
A more realistic way to simulate bit rot is to open the file in HxD and fill selection with random values (right click > fill selection > random bytes > OK, you can even do multiple passes if you want). Your method is fine, but you are reducing the number of bytes instead of corrupting them.
As someone who literally built a storage server to combat bitrot but did NOT know something like this existed, you have my eternal gratitude for helping to provide yet another data integrety safeguard to my toolkit. Thank you, a thousand times, thank you.
I used to use this program back in the early 2000's, when I used to download stuff from newsgroups. I always thought that "PAR" stood for "parity" files, rather than "Parchives".
Yes I believe parity bits have something to do with error correction. I could have looked that up before my comment...
Surely, that is what/how the name came to be.
Pretty sure it's Parity Archive, because it clearly works like a RAID5/6 would.
I was about to make a comment very much like yours here, so I'd agree 100%. I had lots of usenet servers regularly corrupt parts, so the use of extra parity files was truly essential. I'm also glad we're past that and we have things like BitTorrent (which verifies the data is correct), and filesystems (like Btrfs) that can at least detect corrupted files and give RAID style options (using parity) to recover.
@@H4KnSL4K I don't think it was so much that the news servers corrupted the files, as it was that entire files went missing due to news server retention rules. Of course if you used a huge news server like GigaNews, retention wasn't so much of a problem. But local ISP news servers had tiny storage and low retention rates.
Excellent video. I also learned that the Reed-Solomon Error Corrrction Codes were first introduced in 1960 (Irving S. Reed and Gustave Solomon). Amazing
Well this sure helps. Few years back I had to factory reset my entire system for corruption. It's awful really.
Before using any drive, make sure to test it at least once (I do multiple times) with a program like h2testw which fills a drive with data and then verifies the integrity to make sure the drive is functional to begin with. I've bought a new SSD before and it already had blocks it couldn't read properly, so good thing I tested it. Obviously a drive can go bad at any moment though.
@@kunka592 noted. Will do that in the future
For Windows, it is the most tangible! I can put a new copy on, not factory, than waiting up to 10 hours for maybe the software to make changes!
One of an IT worker's pet peeves is a file that you spent hours on and made lots of progress on getting corrupted. Thanks for giving us a course of action at our disposal in case it happens!
This is beyond science
@@ThioJoe nice crown
Bit rot
@@ThioJoe Do the recovery files need to stay in the same folder as the original file, or can those recovery files be stored in another location?
@@shsphotodesign you may store the recovery files in a different location but when you want to recover a file use must move then all back to the same folder.
Funny how a lot of my docx and pdf files got corrupted a week ago while I was trying to copy files from my broken Windows to a pendrive from a Linux live USB. This will be helpful. Thanks Joe 👍
google/utub spying on you 24/7
Whattttt that is mind boggling, the fact that it works in the first place is crazy
I cannot explain the things I have learned from this youtube channel. It has been by far the most useful channel/page/anything for my overall user experience. If I had not found this channel, I wouldn't know about the program Everything that I use every day, about Power Automate in Windows, and so many more things. And now this! Thank you ThioJoe! Keep it up!
I remember using this more than 10 years ago. I downloaded a big file and it got corrupted during transfer. The guy I got it from just sent me a .par file and he told me to use it to repair the file I downloaded. I was so amazed that the little par file he sent me indeed was able to repair my file. I verified this by confirming the CRC and md5sum is correct. That was my 1st encounter with par files and I was glad to learn about it.
Nowadays, I use btrfs file system to resist bit rot and corrupt files. I also do regular backups. Haven't used par files in years but it's nice to know it exists. 🙂
It is important to note that you cannot re-use recovery files. Say you need 50 recovery blocks but you only have 44, you cannot recover it. That is why you can adjust the amount of recovery files and blocks. You could have 4 recovery files each with 50 recovery blocks and send 1 of each to your friends or family, and then if you need more just ask them for the recovery file to get the amount of recovery blocks you need. You can also have over 100% redundancy. It all depends on how important the file you want to recover is to you.
What do you mean "cannot reuse"? So if i recovered my files, i need to recreacre theese backup stuff again? Maybe you have a link to some article with details how it works? Thanks!
@@vlkf I think he means that you can't just copy-paste recovery file and have double recovery blocks
@@exercitus8535 shouldn't they be the same? Or they meant that if you have 4 recovery files and you need 8, you can't copy the 4 recovery files and get it recovered (in other words, every recovery file is different)?
@@AccSwtch50 yep, exactly like that. Even though it looks like magic, it isn't:)
@@exercitus8535 so you can copy the recovery files themselves and get 2 sets of them? Like a complete set and then copy that into another complete set?
A quick way to think how error correction algorithms behave is to think about a sudoku game, you can start by having a complete sudoku filled with numbers, and if you delete one number or change it, there are many ways to recover that missing number. You can even have many numbers missing and you will recover the full state, but past a limit you won't be able to do so.
I've studied the hamming error correction algorithm at college and it really was similar, the way to recover a bit was to do lots of boolean operations on the data, crossrelated.
The most useful thing of these softwares is that the redundancy files are only a fraction of the size of the original document, but they really can fix any error on any of the bits of the source file. The limitation is if you have too much errors, that's when you exceed the "hamming distance" and the program can't recover, but at least it can tell you there are errors. By increasing the size of the redundancy files, you are increasing the number of possible simultaneous correctable errors.
Winrar is very convient for ease of use but this provides interesting use cases such as creating a par file for an entire 1TB+ HDD and using a much smaller portable drive as redundancy, meaning although not as good as a backup you could cover far more files with that limited space or mix and match, have your photo backup and then give a low percentage redundancy for everything else.
This is essentially how raid works. A disk is assigned for storing parity files (Par) across all drives in % or a dedicated disk for all parity. Depends on the raid level you chose. You are merely emulating this concept on a file basis for backups.
Great video, ThioJoe! I have been using PAR2 files for transfers for years, but never even considered that they could be used to protect against bit rot! Genius!
Having bought a WinRAR license makes one a man of culture.
or a true librarian!
I've never used this, but not mentioned is that this is really for files that don't undergo change. So yes, it works great for a set of media files, but its not going to work on all your Word documents or spreadsheets that you are updating. This can't distinguish between rot/corruption vs. a legitimate update. You'd have to regen the Par files whenever you change anything in your set. Raid storage will do this all on the fly, but if the corruption is done at a high level (in front of a disk write) then the corruption will be taken as a good change from RAID's point of view. Seems to me it makes most sense if you make a backup of a bunch of data, then use Par to make that backup resistant to rot. I'm sure some backup solutions do this automatically.
I spent quite a time understanding Hamming Codes, but this is another level.
Thanks for showing us another great tool nobody knew about!
I have a 10 year old 1tb external hard drive which gets plugged in at best once a year. Wish I knew about this sooner, thanks so much!
I use Multipar all the time. As a collector file integrity is very important. Thank you for making more people aware! 🙂
Petabytes of Linux ISOs right 😉
@@vasiovasio Hehe, yeah 😀
Good idea to share this. This is an awesome tool that I have been using for years.
To add... creating PAR files are useful when storing lots of files in zip/7z files. Those types of files can lose their a huge chunk of their contents from a tiny corruption since one bit sequence can be used in 100s of files. Using a tool like keeps those small bit flips or corruptions safe and protects the entire archive. The Par files should never be compressed and don't compress well anyway.
Just today I opened an old-ish project off an SSD. Errors all over it. Never in my life will I use SSDs again. Going back to spinning disk. Thanks for the heads up about Multipar! That would've been a life-saver in a lot of situations where the audio in old projects, or the project file itself is corrupted.
use chkdsk
SSD is only good as boot drive. Spin to win
lmao no you just got unlucky
ssds are pretty good, and also no moving parts, which means no damage from shaking or whatever
HDDs are PAINFULLY slow, I'd rather have an SSD that's actually worse in other ways just for the extra speed
@@Xnoob545
SSDs lose data during to bit rot at a much faster pace than spinning disks. It's due to electron traps leaking charge and getting into a new state. Save a coupe of TB of data for a year or two then go back to it later on. There will be countless errors upon reading it, whereas reading off one of the spinning disk backups, very few errors are present (files backed up beforehand).
A RAID 5 with spinning disks can handle enough throughout for most tasks (video/audio editing), but SSDs are not the best for professional use without at least some form of backup solution.
Good that you mentioned WinRAR for its Recovery Module!
It has saved my archives several times already; it should be a requirement for actual archives to retain integrity of the files.
I've been curious if one can do that in normal, non-RAR'd files; finally, this answers that question. Thank you!
Any of the recovery data can be used to repair any part of a corrupted file, doesn't matter what's located where. This is actually something called Reed-Solomon Error Correction; and as far as I'm concerned it's absolutely magic. - ThioJoe 01-18-2023
Yes we all saw the video. Also it was recorded on the 13th.
I remember using the recovery function once in WinRAR for a large file that had been compressed into multiple archives and one of the archives was missing. I couldn't understand how a recovery file that was much smaller in size than the missing archive was able to reconstruct it. I used the same term as Joe did, I said it's like magic!
Terrific video. I was just using Beyond Compare the other day to bit-wise scan hard drives with backups I made +/- 10 years ago. So far no bit rot detected. Pretty cool! But this is another great technique for protection.
nice thing to do on a lightroom catalog. I back up my catalog once a week or more, but the inbetweens may be lost, so it is nice to have this.
I love par files and away though that not enough people knew about them. I am glad to see a bigger channel is boosting the awareness of them.
I needed this soo much! Thx Theo!
ThioJoe usually drops some gems 💎 on this channel. This was the damn HOPE DIAMOND 💎 Awesome share!!! 👌
- For a moment there, I thought throwback-thursday was back. But it's nice to see that PAR has evolved in the past 20 years and is more usable than ever. 👍 - Parchive files are like recovery-records from compression software, but for regular files. … 11:57 Oh, never-mind, Joe mentioned that.
- 12:33 Forsooth, not only does the 7-zip format itself not support recovery records (which is catastrophic for solid archives), the program doesn't even support them for formats that DO support them like RAR and ZIP (I think ZIP has it, it's been over 25 years since I used WinZip 🤔). I think Igor just doesn't know how to handle them. In fact, people have been using PAR2 files as a work-around for this for years. 🤦
- 13:30 It is handy, especially when dealing with flash-media (flash-drive, memory-cards, etc.) which are unreliable and Windows won't even necessarily give you a warning (I've transferred files that SEEMED to transfer just fine but when I tested them, they had corruption, that's insidious af, especially if you don't realize it until after the corruption makes it into your backups!!! 😬) - Now if we can just get a proper, efficient, user-friendly, and free (*cough*rtpatch*cough*😒) patching program… (remember PPF files, Playstation Patch Files by Icarus of Paradox?)
That recovery method is used in Parity drive in Unraid OS.
In another dimension, ThioJoe has An Awesome Tool to Prevent Corruption of Your Most Important Politicians
Yes, I remember these from my days of downloading stuff on usenet. Awesome!
I wish there was a way to add this on the filesystem level easily, so the system would just do a sweep of every file on intervals checking for corruption and auto-repairing without needing a RAID array or something else fancy. (Depending on how frequently you did sweeps / how often the system was kept on you could probably get away with a tiny amount of redundancy too since youd only be repairing a tiny bit at any given time due to bitrot)
There is! There’s dedicated file systems but IIRC they aren’t just drop in replacements or compatible with a FS that a modern OS would use for a system drive.
ZFS can do that with copies=2 without needing a RAID array or a mirrored drive. But, the disadvantage of that is it requires twice the storage.
@@ckingpro so, it cant do it. Obviously you can just copy the data but that doesnt help. At that point just buy two drives, your already halving your storage might as well prep for drive failure too
@@felixjohnson3874 but I just showed you it is a thing without multiple drives. And distros and BSD based OSes also run regular scrub to check for corruption and repair it
@@ckingpro sure its without multiple drives, you only lose half of your storage space, yknow like what would happen if you just bought several drives and put them in the least effecient RAID configuration. The one difference of course is that you cant recover from a drive failing.
This is really cool!
Great vídeo, thanks!!!
Those PAR2 files were sooooooo usefull in that era I used Usenet (newsgroups). It still exist and without those PAR2 files, Usenet is almost unusable (even the best paid Usenet providers have currupted files).
This is what i immediately thought of aswell. Quickpar, WinRAR and newsleecher or grabit. Later versions of newsleecher and grabit did the repair and unpacking automatically even.
I'm happy that i don't need usenet anymore, but back then it was great for certain things.
PAR files ! that takes me back to ye olde days of newsnet.
These days for archiving my important files I use RAR with recovery records enabled.
We are very dependent on your knowledge. Thank you and please don’t give up on us.
Today I Learned
Thank you for the highly educational and informative video. Well-done. 👏
What if you want to create a Par2 for multiple files at once, but you do not want to group them? Is it possible to do this, or would someone have to do, for example, 100 files individually?
I used quickpar a while ago one thing I liked is if you needed 1 extra block instead of figuring out which blocks you didn't have, someone with a complete file could create you a new block starting at the last block number that's why the file names have those numbers
PAR2 uses Reed-Solomon ECC. It does not matter which recovery blocks you have. It matters only how many you need. You can use any recovery blocks to repair the same number of corrupt blocks. That's why they are dividing the blocks in separate files. If only 10 blocks are corrupt, you only need 10 blocks of ECC data. it does not matter from which par2 file.
Most hardware addresses 'flipped bits' by performing error correction on the fly. Also, PAR files are good for releasing patches.
Thank you for uoluading such an informative video, sir.
Macrium Reflect scheduled backup is the ultimate saviour of your windows system.
It's paid software but I prefer Acronis.
@@johnsmith8981 I got the paid version for free from various sussy sources 😏
@@jj1322 I would never pirate backup software LOL. Acronis has deep integration with the bootloader too.
@@johnsmith8981 Hmm i think I'm gonna check this Acronis software out 🤔 btw my brother cracked the macrium software with pro edition so it's safe for me atleast. But you're right, shouldn't download pirated software.
@@jj1322 your brother is just downloading a cracked copy from someone else online. You're not just trusting your brother you're trusting whoever it is they are getting it from probably a public torrent.
I will this instant(tomorrow it's night rn) put all my family images and videos on my backup hdd in a winrar (may be multipar I'll see)
Thanks bro i have lost some videos before to corruption so really helpful 👍
Why not have this kind of parity built into the filesystem?
Would be a necessity really.
very helpful, i needed this. i do archive and didn't know there was anything we can do about the random bitflip problem
Cool. Thanks for sharing.
if you have a Synology NAS this is included if you use the active backup for business for full system recovery and files too along with time machine backup for mac just restore the file you want
Thanks so much for this video.
I believe this program has a lot of potential, I know it would have been a premium software back in 1990-1995.
Unfortunately there is no website that talks about how wonderful their software is and what to use it on.
Also unfortunately, you seem to have the only video on this software that has spoken english (as opposed to another language or text on screen with music playing).
I would love to watch a video that talks about tons of very specific files this should be used for for everyone and for specific uses for advanced computer users.
I'm just not sure what I SHOULD use it for and can't find any other videos or people talking about how they used it to give me ideas.
I could go through my computer files and check if there are certain window files that typically get corrupted and don't get updated.
I could go through my personal files and try to decide which i should use this on.
However, I would rather someone else figure it out for me and give me suggestions that I would have never known.
I'm not saying that YOU need to do this, but if anyone wants to give me suggestions as replies or make your own video, I would love to know about it.
Also, knowing what setting to put it on for each use would also be important.
Update: I just had to look up "parchive" and "Reed Solomon codes" to get a more content to watch.
An Awesome Tool to Prevent Corruption Of Your Most Important Files listening reply easy to listening and replying love your video
I am happy to be able to see your content in Spanish
Joe, at 5:01 you say "Recovery File Complete", but the actual note says, "1 Recovery File Incomplete". Although the result was what you expected, was concerned this might have repercussions late on down the road.
I remember having to use PAR recovery files back in the Usenet / Newsgroup days. Data wasn't always guaranteed to come thru... ;-)
great video. Thanks
I'd heard of bit rot and data corruption plenty of times but never really took it seriously because "It'll never happen to me"
A few months ago I went to play Pokemon Moon again on my homebrewed 3DS and discovered that every single game had rotted away. Thankfully I still had my backup from the homebrew process, and I now keep everything on a NAS running a ZFS mirror. I still need a backup but it's a step in the right direction
Watching this videos made me get a very random idea: Could I delete certain parts of the mp4 file to remove single frames? And if yes, how?
Awesome tool - thanks for sharing.
Great video subject. PAR might mean parity archive.
I must say that losing a drive is a frustrating thing to happen, especially if the drive has your OS on it. I lost part of my system drive, which is essentially Optane memory, connected with an SSD via RAID. Now, Optane doesn't actually contain any parts of system, it's essentially used for caching purposes. Seemingly unimportant chip, but once it died, system won't boot anymore, and it's impossible to recover. Apparently losing the cache messed Plug&Play drivers up and system won't recognize the drive anymore. The only thing you can do is mount the system SSD into another machine as a non-system disk, backup any data that you don't want to lose, replace or simply just remove the Optane memory, wipe the SSD clean after the work is done, and mount the SSD back in. Then if you removed Optane, you need to reconfigure the drives array to not use RAID (if you do replace Optane with a new one, skip), and then insert install media, and proceed to install fresh copy of Windows onto the drive, then put your data back. However there's something neat about it, and that is that removin Optane frees up NVMe slot, and you can actually insert a new SSD into it. Then you can install Windows into that, and then you don't have to backup your data from the old drive. Although I would still do it, just so you can get rid of the old system partitions and reformat the drive so you can use maximum of it.
5:23 reminds me of a 3blue1brown video explaining Hamming Codes which are a simpler type of error correcting code
This video will only download in Japanese, I've tried to change audio to English but it still downloads in Japanese. Can you repost another video of this just in English, with-out other choices. So it will download in English. This is such a great video and I'd really like to get it in English to reference back. It is a lot of information, truly have to be able to go back remember. Thank you for another amazing video.
This really hit the spot as I am a backup freak.
I'm trying to understand the use case for this. It saves on space, but at the cost of file confusion, i.e. you can't look at those par files and know what's in them. With storage being as cheap as it is, what's the real advantage of this over simply making a double duplicate backup, particularly if compression on those backups are used?
Have the same question
This video is ultra helpful.
This software doesn't prevent corruption of files, rather repairs damaged ones.
So, if I understand this correctly. You can create the recovery files onto a CD/DVD. If that particular file/set of files gets corrupted on the actual storage, you can pull that CD/DVD to recover the missing/corrupted data? Seems a nice option to have.
1:58 this isn't a joke btw, Cosmic Rays can actually alter your data in some very rare cases
I used to use that some years back. Just remembered.
Doing fast backup into tar.lz4 or wim is just better. Its like twice the data (duplicates), less the space, at multiple locations, different partition, different drive, external drive, NAS, online storage, different PC...
TAR are good old "Tape ARchive" that stores lots of information (like user permissions, hard links, security, etc.), proven multiple times, and are sequential, puts everything into a single file without compression, while LZ4 gives 500MB/sec per core or twice more if PC are last gen. So with new CPU and 16 threads you get like 16GB/sec compression speed, limited by network, disk, RAM, or whatever else, while decompression are limited by the RAM speed. You can replace LZ4 with something else (ZIP, 7Z, RAR, LZO, GZIP, whatever), depending on your needs, like what kind of data and choose between compression:speed ratio.
I prefer LZ4 because it gives best ratio across various files, best compression for a given speed/time, it reads fast, extracts even faster, multi threaded, pretty quick and efficient. Google are using Brotli, and its little better compression, but takes way longer.
Since its a big sequential backup, old HDDs have problem reading that (just for review the files and extracting a specific one), but for SSD, to read sequential big data, just to see what files are there, like millions of them is not a problem anymore, it just goes trough few TBs of data in second.
WIM are new "Windows IMage", made by Microsoft, same as TAR, but it has generic ZIP compression and has some similarities with "image" file (like VHD) so old HDDs does not have sequential problem when reading big file containing millions of files. People are not using WIM because its not yet proven like TAR, but i think its generally good.
thanks man
Thanks for the video
Hello Theo ,please make it larger what you are showing on the screen.can’t be read and don’t switch off VERYVERY quickly. Thank you for sharing greatly appreciated, lovely xxx
Thanks for this awesome tip. Data corruption is terrible fo' shoo'!
I've been using that winrar trick for a while now, because sure you can have backups, but what if a file randomly becomes corrupted and gets copied over to your backups without ever noticing? Now your cat pics or whatever are just gone even with proper backups.
My head has blown 🤯
If you PAR a file like a spreadsheet and then change the file, do you have to manually update the PAR or does it do that automatically? If so, how does it know if you changed the file or if it became corrupted?
Par is considered ECC for the source file(s). If you change the source file in a valid way, you have to recreate the PAR2 files.
Would you use this alongside a RAID array to protect against data loss and corruption at the same time?
I had forgotten about this program, used to need to use it a lot back in the day.. I just now downloaded it for mainland use, don't really need it for sea use anymore.
Tickle me a "normal" windows user who is just getting into larger and larger backups...
What is the Pro of this vs just having dedicated drives for backups?
Decreased time for replacing corrupted files?
Less Drives Required to keep archive backups?
I am aware of the bit flip from the ..uh ..space particles messing with the electrons. But this is very rare , and from the data I have seen on the probability calculations....simply having your archive drives ...10 feet from one another ought to be more then enough to compensate for this...SPACE! Its BIG!
Internet, please enlighten me, thanks!
AUDIO TRACKS 👍👍👍👍👍👍👍👍👍👍
Thats pretty cool
Thanks! Question: Sometimes I see empty folders that I am sure they should not be empty. Does Windows deletes files if not used for a long time?
i dont have repair option, only says verify, how do i get the repair option?
Just one thing I noticed: you say "one recovery file complete..." yet the message in the green banner (at 5:01) says "1 Recovery File Incomplete". Does this matter, or is it not an issue?
By the way, this is wizardry! My mind is truly blown!
This is great!
This thought just came to me: what if you used this on a Excel or Word document and then later used this to recover the file if it gets corrupted. Wouldn't you lose the changes you made since you created the .par2 files? Perhaps only use this on files that should never be changed and keep proper backups on your documents.
Lol I remember using Par files for UseNet to piece together Gamecube and Wii ....files. Good times, very cool stuff.
Hey, @ThioJoe . I ran the installer of the latest version of MultiPar (from your Github download link) through Virustotal and 5 out of 69 anti-virus software detected a trojan. Not sure if this is a false positive as each of these detections come from software I've never heard of. Kaspersky and all other popular anti-virus software say that it's fine. Do you think it's safe to use?
I didn’t know that this was possible I thought I’d have to use a file system with built in support for redundancy.
0:48 Overall
7:54 How to Settings
5:33 Name of Algorithm/Method
11:38 Winrar tip
Doing a real service here, it's not like Thio hasn't already got time stamps in the description 🤔🤨
▼ Time Stamps: ▼
0:00 - Intro & Basics
0:48 - Demo: Corrupting and Repairing a File
2:48 - Demo: Multiple Corruptions in a File
4:29 - Demo: Corrupting the Recovery File
5:33 - Demo: More Details on How it Works
6:21 - Demo: Replacing Completely Missing Files
7:51 - Demo: Explaining MultiPar's Options
11:38 - A Similar Feature in WinRAR
12:43 - Alternative Programs
awesome
Thanks for the video, this helps a lot
One question: does this still able to recover files that are modified after creating part2 for them? Let's say i create a Photoshop project file that i modify often, save and quit!
Or should i create part2 file for it once i finish working on it?
Hello, thank you for sharing this program; I've been looking for something like this for quite some time. But I have a question: what should I set the media size to? I'll primarily use Multipar for backup archives. Thanks
That setting doesn’t matter unless you are storing the files to a permanent optical disc or something
@@ThioJoe Oh, I see. Thank you for clarifying. I've already started using it.
Neat.