Tested's Media Management Workflow!
Vložit
- čas přidán 26. 06. 2017
- In our latest behind the scenes video, Joey goes in-depth with his media management workflow for shooting and editing our Tested videos. Here's how Joey handles the gigabytes of data from memory cards to DAS systems to long-term archiving on a Synology DiskStation server.
SYNOLOGY DS1817+: www.synology.com/en-us/produc...
LACIE 2TB HDD: www.bhphotovideo.com/c/produc...
PEGASUS RAID: www.bhphotovideo.com/c/produc...
SANDISK 95MB/S CARDS: www.bhphotovideo.com/c/produc...
Shot and edited by Joey Fameli
Join this channel to support Tested and get access to perks:
/ @tested
Tested Ts, stickers, mugs and more: tested-store.com
Subscribe for more videos (and click the bell for notifications): czcams.com/users/subscription_c...
Twitter: / testedcom
Facebook: / testedcom
Instagram: / testedcom
Discord: / discord
Amazon Storefront: www.amazon.com/shop/adamsavage...
Savage Industries T-shirts: cottonbureau.com/stores/savag...
Tested is:
Adam Savage / donttrythis
Norman Chan / nchan
Joey Fameli www.joeyfameli.com
Gunther Kirsch guntherkirsch.com
Ryan Kiser / ryan.kiser
Jen Schachter www.jenschachter.com
Kishore Hari / sciencequiche
Sean Charlesworth / cworthdynamics
Jeremy Williams / jerware
Kayte Sabicer / kaytesabicer
Bill Doran / chinbeard
Ariel Waldman / arielwaldman
Darrell Maloney / brokennerd
Kristen Lomasney / krystynlo
Intro bumper by Abe Dieckman
Thanks for watching! - Věda a technologie
The work behind the scenes like this is almost never talked about by people. Thanks for this insight! The level of detail really helps give me (and I’m sure others) a path to follow as video files for hobby cameras continue to grow beyond the typical direct drive storage methods. 🙂
DUDE, Joey. That condense and transcode in Premiere has saved my LIFE! I work at a church and edit down each service, now I only have to keep the snippets I use as we keep the raw file separately. You just freed up TB after TB on my drives.
I'm also pretty excited about the same segment of the video!
Recently started shooting 2 camera angles, for every episode..sometimes, even 3 camera angles.
On a 45 min episode, 2 min of B-Cam footage might make in into the final edit. However, I still have to back up all 45 min of B-cam footage......or so I thought!
What else am I doing the hard way???? Ha
This is exactly what this industry needs! More video workflow videos pls! It's so difficult to find out what others have done and what works for different levels of production.
I'd love to see your file structuring and naming conventions in depth! Also your project management process. It's rare to see such a great insight into how teams manage these workflows. I've watched this video and your following video almost frame by frame!
As someone who has a very small CZcams channel, this sort of information is really good. Thanks.
Terrific video, really enjoy seeing the behind the scenes and workflow footage!
Really great stuff, I've been looking into some kind of RAID setup and am always curious how others manage their data workflows, thanks for sharing this!
Nice job, thanks for sharing. (I'd love to see more file structure setup)
I love my Synology server so much. It's amazing having access to my files via mobile or off-site computers. The video station is an amazing feature because you don't have to go through the pain of setting up an app because it's built in.
I love it when Joey does videos! As someone originally trained as an editor I love the camera videos since it's not my area of expertise and he always has some great tips! Great tips in this video too though, clever thing with putting the SD cards upside down in the case haha, so simple.
Great run through Joey. Thank you!
Joey is the only person on Tested.com that actually do proper testing.
Don't forget the 3 - 2 - 1 rule for media/file storage! 3 copies of files: 2 local and 1 off-site. Especially when you're dealing with camera/show footage that can't be replaced.
Yeah, that's the kind of shit people say who aren't actually working with large volume media and only get this shit from reading. It's a nice concept, but impractical in reality. There's a good reason why this guy didn't say much about it.
Enjoyed this video! Always happy to see content like this, thanks.
Thanks so much for this video. Been looking for more opinions on video storage at all levels as I move from more hobbyist to a career in video.
You didn't mention any off-site storage. If you don't have any, I'd invest in such posthaste. If your office catches fire and all your backup files are stored in the building, you're screwed.
You might want to look into getting a tape drive and tape storage for that: Tape is still the cheapest medium per Terabyte and very, very long-lived. Since you'd hope to never need your tape archive and only rarely make a new backup, storage speed really isn't much of a priority I'd say.
Edit: What you're looking for would be the IBM LTO Ultrium 7 Data Cartridge, for example.
I agree especially with a SHR volume formatted with only 1 disk fault tolerance. I formatted my SHR volume with 2 disks fault tolerance that way when replacing a disk I am not at risk of losing data if another disk crashes during the operation. I always expect a 5% chance of having a brand new disk fail in the first 5 months of use (important when upgrading all the disks).
He definitely mentioned offsite storage, I love the offsite storage patrol lol
If you think tape archiving is 90s, I pity you. Tape has been continually developed (and is still being developed today), with read/write speeds well in excess of what a normal HDD can do when given continuous data. Also, it's much cheaper per GB and far more long-lived as archival storage.
There's a reason that pretty much every data center and/or supercomputer has a tape deck somewhere on premises.
Cloud is an option, I guess, if you don't mind trusting someone else with your data and have a sufficiently fast upload to make it work.
Justin T whereas your solution is a waste of money...
Actually tape is still considered the most reliable for long-term storage. No moving parts, etc. And waay cheaper too. I manage post production & the server infastructure for a post production company and we will be switching from bare drives to tape for our deep storage, because the cost is getting out of control and the reliability will also be better.
I love your redundant copy system. You never have to worry about having your original. This the principal I used programming and in other business practices.
Joey, RAID 6 (or equivalent) is your friend. I manage two SANs, and a bunch of different brand NAS systems (including four Synology's). Unless you really don't mind restoring all your NAS data from backup, please make sure you have RAID 6 or RAID-Z2.
Especially with drives over 3TB, you WILL have a drive fail during repair. It's not a question of if, it's a question of when. All drives spin up to repair parity, and in that sudden rush of performance, you are more likely to lose a drive.
I never recommend RAID 5 or RAID-Z1 with any RAID set over 4 drives. I've had too many moments where I've had to restore systems from tape (LTO), because the person that built it thought RAID 5, on a 5-10 bay NAS, was enough.
Rather waste 1 more drive. I prefer buy the best enterprise HDD. Do not buy any seagate drives !!! It is cheap but easy to crash !!! Buy WD Gold Drive !!! The best HDD !!!
Awesome video, thanks Joey & Tested!
Your workflow is very similar to mine. Dual card writes, cards in separate protective containers, upside down card vs. upside right cards to give status on card to use and not to use. Although, I never reuse a card while in the field as a precaution. And I have one more step in that I keep with me a 2Tb remote card copy unit which then gives me my original media, plus 1 site backup. The protection and backup workflow is so critical. And then, the same begins locally on the media storage. WELL DONE!
One drive fault tolerance on a 70 TB pool is... terrifying to say the least. I really hope that data is being backed up to an offsite cold storage.
Yeah... It seems someone is lacking some IT-background! RAID is not a backup!
Yep, we live by 3-2-1.... data in 3 places, on 2 different media, and one off site. backing up to an Amazon drive is good or on their glacier storage... it is all how much your willing to loose.
You have the tools to do it right.... use them.
rkan2 yes Raid is just a net.. not much.
I use two drive backup on my 8 drive array, I've never had a drive fail, but I have had drives complain before while rebuilding. I'll take the loss of another drives storage to get a little more comfort.
Synology OS is horrible once you've used any proper NAS/SAN based storage. It's like storage for dummies, RAID6 or 10 as a minimum on critical data.
I didn't realize how much I needed to watch this video until afterwards. Damn that cleared up some really simple workflow mistakes I was making. I had NEVER used the media manager to trim files....soooooooooo useful.
Excellent breakdown. Incredibly important.
I did not know about the project trim function. Thank you!
Thanks for sharing this. It's very interesting!
Awesome Vid Joey! Thanks for the info.
This was very helpful, thanks!
Awesome, thank you for showing how to step up our game
Liked this video before watching it for the mere fact that you made a video on this topic. Thank you(!)
This is very satisfying thanks tested for really getting your audience!
Oh boy 1 disk parity in a 8 drive dev on btrfs. Tested living life on the edge.
Rather waste 1 more drive. I prefer buy the best enterprise HDD. Do not buy any seagate drives !!! It is cheap but easy to crash !!! Buy WD Gold Drive !!! The best HDD !!!
Seagate are reliable and great HDDs - if he has the IronWolf Pros, he has 5 year warranty and free recovery services. The also worked closely with Synology to integrate better monitoring into the drives so you know if something imminent is about to happen.
When was the last time two drives failed at the same time? Never.
I'd be interested in seeing Tested's video editing workflow and a detailed description and reasoning for organizing files.
Thanks for making this!
DS1817+ supports hot swap, so you can swap the failed drive while the system is running - this is recommended to do(the OS on the NAS is also using the raid for storage, in a different partition). People like to say RAID 6 bla bla, but you should always keep a full backup of everything anyway(ransomware and so on - offline copy), so RAID 5 is a very good balance between availability and cost for space.
Great video and explanation! I’m thinking about getting a Synology NAS as well.
Very awesome stuff Joey! Keep sharing content like this!
I currently own a Synology DS 214 play (2-bay) and just started using a FreeNAS on an old desktop I had laying around. More for educational purposes than out of a real need, though I do intend to start producing video content at some point (amateur video editor, photographer, videographer here).
Here's the thing though. Early last year, I had a full NAS failure. Basically, I bought a kit from Amazon that I thought was coming with two WD Reds, but actually came with two Seagate drives... you know where this is going. After coming back from a family vacation trip I found my NAS beeping... S.M.A.R.T. had detected a failure in one drive, and as I was thinking on what to do about it, the other drive also failed. They were mirrored - RAID 1. You can't recover from a situation like this with the NAS alone.
I started reading around and researching stuff, apparently (theoretical, but everything I've seen so far points to this) an overzealous Diskstation update was flagging a whole ton of Seagate drives even when they didn't actually have any problem. I'm not sure on the reasoning behind this, but Seagate drives have been notoriously problematic in some reports, like the BackBlaze ones. Mine were 3Tb NAS grade ones.
My setup was encrypted, so after a week of research I ended up having to purchase recovery software to retrieve my data from one of the drives. I couldn't recover data with free software, and I wasn't able to deal with Linux tools like Gparted, ddrescue, testdisk and whatnot.
Just in case anyone needs this (hopefully never), it was Runtime's NAS Data Recovery software that at the time went for around 100 bucks.
So I got new drives, recovered and transfered all the data, started from scratch. Everything is working fine now, no issues so far. And the new FreeNAS system is using those failed drives without a hitch. I ran diagnostics on them multiple times, I used them to transfer data, and I have been using them as external storage up to now, full year and a half after the NAS flagged it's SMART system.
Anyways, thought of sharing some of my experientation on the subject. o/
Such an awesome video!
I've been using an Odroid xu4 with open media vault, and for my needs it works pretty great. I plan to get one of those Orico 4 or 5 bay encloses and upgrade my storage at some point.
This is some top notch organization! Thanks for showing us :)
loved this video!! thanks!
Great video Joey!
Don't sell yourself short Joey. As an IT guy I geeked out over this video as much as some people geek out over, say, Adam's Chewbacca costume. I've worked with the Synology system before, but still cool to see it implemented and your full practice.
I always import all the footage into individual project libraries on final cut pro x. Then I take that library and copy it onto a second hard drive. While editing the project, I copy the library across every hour or so.
Thank you Joey for doing this video.
This is such a crucial part of the creative/video process and there aren't nearly enough good videos available on this topic.
The best part is that you have essential detailed the system that I have been mentally formulating for the past couple weeks.
Question...
Would it be possible for you to post an example of the Excel doc. you use for your archives?
You briefly mentioned how you manage it but I would be very interested to see some more detail.
Thanks again and congratulations on a great video :)
Joey getting his own video, nice.
It is very interesting to see how things work behind the scenes and not just watching the refined version of a video that goes up on CZcams and/or Tested.com.
Do you also utilize off-site backup services?
Thanks for that, great content
Great video. Thanks.
Awesome stuff! Could we also get a tour of the new Tested offices?
Holy crap a video about Digital Asset Management!!!! I work Media Manage for a studio creating about 20TB a week.
great video Joey!
prefect video to help me manage my youtube video files for editing..i got the same SD card case 😁
would love to have more behind the scenes videos. maybe a day in the life of camera crew and editor.
Nice video, thanks! For your next storage setup, please look into a double parity system and hot spares. Every major storage device can do this. More details below.
I do enterprise grade storage for a living; single parity setups (like yours) cause us data loss all the time. Anything that tends to make one drive fail tends to make several drives fail (old age, heat, powering off drives). Rebuild times are very long on large drives, so it's very common to loose one drive, and then loose a second before the first can be replaced and be functional again.
For your next storage purchase; Rules of thumb are, always use at least double parity (so you can loose two drives), always have a spare drive installed (so that replacement happens automatically and at once), build more raid groups with fewer drives (so if you do loose more then one or two maybe they will be in different raid groups and not cause data loss).
Also make sure your storage system is checking your drives for bit rot. Remember that if you loose one drive in a single parity system, and your have any bad sectors on ANY remaining drives, those sectors are gone forever (every block on every drive has to be read to rebuild the failed drive, during a rebuild isn't the time to find out you've got back sectors). Double parity prevents this too. But bit-rot checks help a lot also.
I have 2 x Synology devices at my office. I worship those things.
Great video!
Oh hey, is that Budapest as your wallpaper? Nice
Nice review, I would strongly recommend going for 2 disk parity when using more than 5 disk in a single bay, try RAID 6 or SHR-2, otherwise you risk loosing your data during RAID rebuilding.
Keep it up!
1 Disk redundancy is not good enough. You would not believe how common it is for a drive to fail when resilverling after you replace a dead drive.
RAID 6 or RAIDz2 is the minium IMO.
Yeah, another drive failing while "repairing" the raid set is not uncommon, because of the strain all those old disks get put under while moving data over to the new disk you just inserted.
Andrew Joy definitely. if one drive fails and the rest of the disks are from the same batch, there is a good chance at least one more showing the strain during reslivering. true they are just being read from but you are moving the head and spinning metal.
some people I talk with try to factor this out by aquiring the disks from different vendors. get two drives from Amazon, two from newegg, two more from Fry's/microcenter. basically try to pull from different production runs.
of course this is moot if you try to use a horrible drive like a "deathstar" which will do you no favors no matter what run it is was from.
running zfs on ubuntu 16.04 with a i3 xeon 3.1 which runs Plex and soon moving to emby since that framework looks like a faster feature release than Plex (hello hardware transcoding).
Yes. If you're lucky, the first drive to fail will just be a fluke. But more likely it will fail because it's old, and when that happens all the other drives are old too, which makes it the worst time to put them all through 20 hours of non-stop access.
ZFS is the new king, not sure if the synology can use it, but it's been proven by huge companies like Netflix and it can deliver higher speeds with the same amount of redundancy or more due to the way it handles corrupted data.
RAID is not a replacement for backups. Hopefully they have a real backup system. Just relying on RAID is a very bad idea. It dosen't matter how many redundant drives you have. There are many other things that can compromise your data than a broken hdd.
Use raid for high availability and store one copy of your backups off-site.
one of the cool things about synology is their sync'ing feature. I upload the files to my editing bay and the sync soft sends the video files to the synology. I'm about to implement a new one. What's supercool is the old synology will sync with the new one (volumes of my choice) at an offsite location. Also is an awesome torrent server!
Cool video! More BHT material and how to's.
Good story. Good insights.
Make videos about Technology.... Buy technology to make videos... Make videos about technology that allows for making videos...of technology.
I want that storage in my life.
Thumbs up, keep these behind the scenes videos coming. :)
You should look into contacting an MSP or someone that actually knows "IT" and for them to advice you in how to properly store data. Your current setup is very high risk since the failure rate of restoring a RAID array is very high with that big disks.
Three key things you might benefit from looking into: ZFS, Coldstorage, cloud backup.
The 1817+ supports expansion units. So when your 8 drives are full, connect two more expansion units to get a NAS with a total of 18 drives.
I'm a long time Synology fan. I'm using an older DS1813+ in SHA-2 mode (I think you're brave / crazy using SHA-1!) with off-site third-party cloud backup. I find the throughput on this model more than enough to edit 1080P or better...
Good Lord, my brain hurts more now than after a typical Tested show...
I really like those HDD cases at 12:03. Who makes them and where did you purchase them?
Synology commercial...love it
Quick tip: you don’t have to turn off the server for a drive swap. The ds1817+ has hot swappable drive bays!
After your NAS fills up, you copy everything to external drives and catalog on .xls. What is your workflow when your DAS fills up? Very helpful info - thank you!
I am setting up my own archive!
Just remember that RAID is not a backup, your setup is mostly fine, but you must ensure that as others have said, that you have off-site backup, alongside cold storage backups if possible, otherwise if something was to happen such as fire, crypto-malware, hardware failure or theft, that you have some way of recovering from it.
I would also recommend a UPS to plug the NAS into, and also a second drive array to mirror the first, but one that is kept isolated from the network except during a weekly backup.
Proper data protection is expensive, but it's peanuts compared to data recovery, for example, a single drive recovery operation costs anywhere from 300 to 2000 dollars, with no guarantee of getting your data back, buying a second hard drive though, is 50 to 300 dollars, and that's just one drive.
As for a RAID array? Multiply that 300 to 2000 dollars by the number of drives, and then double or even triple it for the cost of reassembling that data, and you're still not guaranteed your data.
One last thing, create a plan for dealing with data loss, and practice it, take a dummy folder full of test files and delete it, and try to recover that folder yourself, so that if you're in that situation for real, that you know how to deal with it.
I prefer FedEx over UPS.#To Many Acronyms
pretty cool!
what service do you use for cloud storage
you had spoke of a remote office.. so how do you transfer video files from one nas to the other.. how fast is your ISP service ?
also, do you have 10Gbe network at the office ? or just hookup the 4 - 1Gbe ports for bonding network speeds ?
I have the DS1815 but shuddered when you said you use SHR-1. Would really recommend removing data and reconfiguring to SHR-2 as the most likely time for another drive to fail is during a failed disk rebuild. I run mine with 6TB drives in SHR-2 and a hot spare ready to go on any disk fail. I also have external backups of data too
I'm totally happy with all of the "sponsored content" on youtube now... it's like the whole reason I swapped from standard media. Because I want to be advertised to! wOOOOOO!
Sorry if I missed it but, do you edit off of your local drive and only use the NAS for file storage when the project is done? Also, do you just delete the stuff you didn't use in the project or do you save that footage in case you can use it for future projects?
You should increase the fault tolerance of the Synology hybrid raid to 2 disks. It's not that expensive and especially with archival footage you should have a 2nd failover disk. I say especially with archival footage, as this data isn't accessed a lot normally. That means that the disks that are in there aren't checked a lot for errors. Therefore, during repair errors can suddenly pop up without it ever having noticed them before. That's why you want another failover disk, just to be sure. And compared to the price of 8 disks, a 9th disk costs only 12% more, with a failure probably of at least 50% less (statistic made up, but you get the point ;)).
SHR1 should be fine for him. What he needs to invest in is an offsite backup solution. (Also the nas only has 8 bays). Raid is great for fault tolerance but you should invest in a backup before extra disks because all of your disks can fail at the same time (flooding, natural disasters, etc).
Im curious if you have toyed with LTO disks for long term archival purposes
good Stuff. you can go further on the Synology but one think missing (not in workflow but generally) is an UPS Management...
Joey you're genuinely a heck of a guy on camera. Do it more. Come on Still Untitled please!
Is this trim (Export) Function also availible in sony vegas?
Thank you
Thank you! How do you make the Excel sheets of the drive content?
Hiski Hämäläinen I'd like to know too. Got wayyyy too many hdds on my bookshelf in cold storage. Have to go through a bunch to find that one file I'm looking for
If you have to replace a failing or failed drive and a replacement is no longer manufactured or available, what do you do then? Can you replace the drive with just anything, or does it have to match the others in the RAID?
Cool appliance. Raid has been around for years! For those of you on a budget, you can build your own RAID storage appliance for a fraction of the cost.
Geeks And Guinness and you don't even need to buy a raid card either. zfs is magical in both creation and also exporting (which is a nightmare in itself if you use hardware raid around 2006 or so).
I rebuilt my zfs system once but forgot to export the zpools. thinking the worst just happened and I would have to spends hours rebuilding from offline backup zfs came back and reconized the drives I already had in there were from an old zpool and I had enough disks to recreate it. it was a good feel
The reason you would use this rather than your own raid server is the SHR1&2 which is basically an auto scaling raid 5/6 so when you add drives you don't have to rebuild your drive pool. Not to mention all the added functionality of the OS (Surveillance station, containers, etc)
If your time costs nothing . . .
these videos are great.
HackiH I
Your long term archive disks are a single HD that just sits around for years? No redundancy? BTW, the synology can backup to AWS, or another (offi-site) synology.
So far, im following,
- Master Export
- Trimmed project file export
But what about additional raw footage that didnt make the cut but might be needed in the future?
How do those get stored? Or are those deleted from the get go?
my kinda of jam. Sexiest episode on tested!
It'd be great to see something about how you go about compressing video for YT - I can never seem to quite get the quality I want, but you guys seem to have it down - its a shame it was glanced over in this :(
How does the synology handle plex transcoding?
Heya - just to let you know, the one drive redundancy you're running with is both good and bad. Good, because yep, if one drive fails you're safe. Bad - because you're safe only temporarily. I've recently had a drive failure in our NAS at home (also a Synology), and following the instructions of "pull affected drive, insert new, rebuild" didn't quite work out.
What happened to me is that the logical volume sitting on the disk group became un-synchronized with the new disk group - and it was not mountable. I could see it was there, but couldn't access it.
Synology support was fantastic, by the way - they helped me out and recovered *some* of the content that was missing, basically by working some voodoo that made the damaged volume mountable. Long story short, I still lost some content, but not nearly all of it.
So my advice here - get yourself a second, identical capacity NAS unit (and I also recommend Synology for this) and set one up as a 'cloud station server,' the other as a 'cloud station client', and set both your DS1817+ and the new box with 10GbE NICs over a 10Gb switch. One box becomes your backup, the other stays your primary, and the two will keep their volumes synchronized.
Otherwise, cool stuff and thanks for detailing your management strategy. Much appreciated to see someone else's view on how they put this stuff together :).
I have a Synology NAS with 4 HDDs and never figured it out. From memory only iTunes worked. After lots of frustration I just disconnected it and put it down to a learning experience, or should that be non learning experience?
How long do you keep the archived files before you wipe the hard drives? Or do you keep everything and throw nothing away?
Typically you'll keep everything and throw nothing away. Any backup you have of your data should be held onto for as long as possible.
love these videos @tested! Keep it up!