Data Wrangling For TV Commercials
Vložit
- čas přidán 22. 07. 2024
- This video will run you through the basics of on-set Data Wrangling for TVCs including:
-Hard Drive Labelling (0:11)
-Formatting and Testing Hard Drives (1:14)
-Tweaking macOS Finder Settings (7:28)
-File Structure and Naming Conventions (9:27)
-Optimum Shotput Pro Settings (10:48)
-Using Shotput Pro to Wrangle Data (12:12)
-How to Double Check Yourself to Avoid Mistakes (14:12)
-Other Miscellaneous Data Management Principles
For a more detailed overview please check out my blog:
growthis.blog/2017/08/05/data... - Krátké a kreslené filmy
Just took my first DIT/media manage gig and this helped so much. Cheers!
I'm really glad to hear that this helped you out. I trust your first job went well following the methods above and that your data skills continue to grow.
Feel free to check out my new data management blog: filmdrives.com/blogs/blog
Finally a tutorial that goes in depth, mostly on how to properly setup and download media. The only thing I would mention is to show what happens when you get an error and what to do. Especially when Shotput crashes writing the checksum. Nothing like having a 30-45min download and it’s down to 0% and shotput crashes. Everything is great until it’s not. But how to deal with it when it happens often gets overlooked.
Sooooo what happens and what’s the remedy
Really good vid! So glad he walked us through the process of offloading through Shot Put Pro alongside all his tips on managing your folder structure and overall desktop view of the process. 10/10!
I just want to say this was freaking awesome to watch. As someone who has scrambled around wrangling tasks (I'm a video editor by trade by follow shoots occasionally to be a sort of DIT/wrangler), the idea of creating "systems" such as your left to right workflow has been eye opening. I'm definitely using that the next time I wrangle as I'm almost always brain dead halfway through shoot days :P
This is a hidden GEM for someone like me starting out as a truck loader / data wrangler!! This was so well explained, I learnt more from you than I did from Shot Put's videos... Cheers mate! Your workflow was basically what I had in mind but it was awesome to have you run through the program as well :)
Great video! Thank you for being thorough and straight-to-the-point!
Thank you so much for this very instructive video. I needed this type of practical information to get going in this field. Thank you for sharing information on your favorite tools ( Black Magic, ShotPutPro, and Onyx). You are very organized, and I can see that being so organized is a key to success in Data Wrangling / DIT.
Glad I could help Chris. I've been doing a few more data related blog posts here: filmdrives.com/blogs/blog
I have one post about what should be in a Data Management Kit both hardware and software wise which I think you'll find quite useful.
Thank yo uso much! This video is really helping me start working as a logger!
Great Video! Thank you so much!
You are very generous.
exactly the video i’ve been searching for - thank you so much!
Thanks Richard. I know what you mean, there aren't a lot of other tutorials focusing specifically on good data management practices.
Please check out our new blog for more data related posts: filmdrives.com/blogs/blog
Super Useful! Thanks for this mate!
This video needs more views! Great content.
Thanks James. I appreciate the kind words.
There will be more great content to come but it will be posted on my new data management blog, feel free to take a peek: filmdrives.com/blogs/blog
Awesome video! Thank you for sharing your techniques.
No worries at all Jonathan. More techniques and discussions to come at filmdrives.com/blogs/blog
Great tutorial! Thanks for sharing.
Thanks so much, Shot Put's tutorials are garbage. I appreciate you taking the time to actually show us how to properly use the software. I hope you have a fantastic weekend.
No problem Clint, I appreciate your appreciation! I'm looking at doing similar tutorials for Silverstack, Hedge and DaVinci Resolve in the future. Keep an eye out for them over the coming months at filmdrives.com/blogs/blog
That was a great tutorial! Thank you!!!
Perfect video mate, thank you
Great tutorial, thank you for this!
So thorough thanks
great tutorial, thank you
Great Tutorial, thank you. For the problem with the folder digits: There is actually a option to choose the number of digits just below the the option for the starting number. So you could choose the prefix “A” and then 3 digits and you’re good to go.
Great Tip Johann! Thank you!
great video ! very informative
Excellent! Cheers mate
Thanks for this video!
great video
do you get a video on how you organize your sound folder?
Love the labels you use. How do you print out the labels? Do you do them at home or something?
What does 'Rushes' means?
About to take first DIT job and this eased my nerves thank you!
Copying one card to the master+backup at the same time is a risk right? I should always do it one after another.
What questions do you ask about before starting a job (regarding pc brand, card readers, cameras) if youre material is rented?
Thanks!
great!
helal olsun sana kardeşim.
great video!!! thanks for your generosity. One quick question (at 10:52) in the offload pref/offload sequence do you always use "optimized drive use"?what are the other two option for?? Thanks!!!
Hey Jacopo. No problem, happy to help! This is a screen grab from the Shotput Pro Manual and should answer your question: cdn.shopify.com/s/files/1/2691/1130/files/ShotputPro_OffloadSequence.png?2730
'Optimize Drive Use' works great as it intelligently determines what the current read/write speed is and what the max read/write speed of each bus on your system and balances out the tasks it is performing to ensure efficiency.
Great tutorial! Was wondering - if there are less bytes when you look at the info - What should i do?
Hey Charlie. Thanks for the feedback, glad you found it useful. If the bytes are less in your two destinations compared to your source when you Get Info, I would do the offload again. Very, very rarely you may get a false positive from your offload software, the Get Info Check is designed to keep the software honest. When doing an offload again, I would delete it from my two destinations, empty trash and then copy the source media (camera card) again via the offload software and proceed as normal. Hopefully the second time around the Get Info Check passes.
Great video!
1 question; if I got an error (because cfast card un-mounted during validating) can I somehow run the validation again and replace the the report?
Hey Tyler. Shotput Pro used to have a Checksum Utility that did exactly that, you could also use it to manually run checksums on other files. I used it once or twice but didn't find it very useful. I just had a quick look and couldn't find it in the current version. I'll take a more detailed look tomorrow. If I get a failed validation, regardless if it's from an accidental un-mount, I will redo the entire offload. It's the cleanest way in Shotput to generate a proper report and that is important as you have an accurate paper trail and it covers you encase there is an issue down the line. The only instance when I won't redo an offload for a failed validation is when I know it was accidental, I was there when it happened and that the offload time is 2-3 Hours+ and would significantly cost production were I to start it again. If I go this route I make sure I'm very detailed with my manual checks and these days I have alternate methods to run a checksum and generate paperwork so all bases are covered. I hope that answers your question!
I've been asked to format clients drives for PC and Mac. I use ShotPut Pro and have always formatted drives as Mac OS Extended (Journaled). Is there any file size limits with ExFat? We are shooting some 6K footage and file sizes can be up to 500GB. Is there any limitations with transfer speeds when drive is formatted ExFat opposed to Mac OS Ex (J)? Any insight is appreciated. Thanks, Ryan
Hi Ryan. Thanks for reaching out. There are no file size limits with ExFat, it's like FAT32 on steroids and is quite a good file system. If you are required to provide drives that are both Mac and PC compatible then ExFat would be my first choice. There are no transfer speed differences between Mac OS Ex (J) and ExFat so there is nothing to worry about on that front. Thanks!
Allocation size unit for buffer should not exceed 1024kb in order to MacOs to mount Exfat disks
Hello! Its possibile to see on SHOTPUTPRO the average speed of my offload? And also the single file offloading?
Hi Edoardo. Thanks for reaching out. I don't believe you can actively monitor the offload speed within Shotput Pro. I use iStat Menus which can display system wide data transfer speed, you can use that in absence of a display within Shotput.
Please!!! Silverstack tutorial! 🙏🏽
Nice!
Great video! Have you used Hedge before? I am curious as to what you think about it? Please make more videos!
Thanks Nshan, more videos to come. I recently had a play with Hedge and it definitely shows promise, I like the simplistic design and the sync to your phone features but at the moment it's a bit too simple for what I use it for, I couldn't find a way to setup multiple offload presets so that I can quickly switch between an 'A Camera Offload' and a 'B Camera Offload' which was a dealbreaker for my normal workflow. Paul the Hedge CEO touched base with me the other day and said that they are working on new multicam features so I'll be sure to check back in the future. For the moment a combination of Shotput Pro and Silverstack meets my requirements.
I use Hedge over shotput. One thing to know about Hedge is that it writes the checksum as it’s offloading, which is why it’s faster. But, if if there’s an error, you can still get a checksum as long as the download doesn’t get an error. I know that’s not explaining it right. But if you follow the steps in this video, you can catch any issues no matter what software you’re using.
Probably not a great idea, but is it possible to have two card readers plugged in at once and then set up a batch process to allow you to copy, say A Cam's card to both drives followed by B Cam's cards to both drives sequentially....?
Hey Matt.
No, it's definitely a great idea and it's a strategy that I employ from time to time. I use it when I need to be away from the rig for a longer period of time or when I know a card will only take 15 Mins to offload and have another that takes 15 Mins to offload (A Card/B Card on a 2 Camera Job). If we are going to lunch (45 Mins), I can queue the two cards, go have lunch, come back and check them at the same time. Shotput didn't used to be great at this queuing but I suspect it'd perform better these days, I use Silverstack now for all my work and the queuing there is flawless.
My Film Industry / DIT / Transcoding posts are moving over to my new blog which is specialising in data management solutions. Feel free to check it out: filmdrives.com/blogs/blog
Yes, that is indeed possible. Shotput used to handle it weirdly and it'd try to copy both at the same time when you queued them, this would cause the offload to be extra slow. I use Silverstack now and queuing cards up as you described is easy and effective when you need to be away from the Data Rig for a while. You would think Shotput would have this nailed by now, I'll take a look tomorrow and suss it out.
By the way, this is a great CFast Card Reader designed to work in the setup that you described: www.bhphotovideo.com/c/product/1386132-REG/xcellon_cr_cfasd_4_slot_usb_3_1_pro.html
Have you ever worked on a documentary? I wonder how different it would be. I'm currently try to come up an organization method for the documentary i'm working on. Do you rename the clip to include the scene, shot and take number or the date? Thanks
Docs are the same system. Not sure if I’m understanding about if you rename the clip, etc. The only thing in downloading cards with a checksum that you’d ever rename is some cameras that don’t give individual card names and are just untitled then you’d call them whatever camera they’re being called on set.
You never change the file names from a card unless you’re also doing the post work too. That’s a great way to have post hate you and possibly her fired.
Tutorial on transcode portion?
We can certainly run a tutorial on transcoding with DaVinci Resolve. All of our DIT/Data Management Tutorials and Blog Posts are being hosted here: filmdrives.com/blogs/blog What specifically would you like to learn more about in regards to transcoding?
What if you have scene bins?
Hi Nevada Cube. If you have scene bins they are obviously setup to the specific file structure that you are creating. You can setup unlimited presets within Shotput Pro and can use the naming options to mould the folders to fit within your naming conventions.
Hi Nevada Cube. If you have scene bins they are obviously setup to the specific file structure that you are creating. You can setup unlimited presets within Shotput Pro and can use the naming options to mould the folders to fit within your naming conventions.
Ever consider USB hubs? Might be a good option, as the spinning drives don't really saturate the USB3 bandwidth - you could easily have 2 drives running through the hub with no sacrifice in speed
I have considered, tested and used USB Hubs in the past and opt to stay away from them unless absolutely necessary. You are right, HDDs won't use the entire data bandwidth of USB3 but they are very likely to use the full power bandwidth. Depending on your flavour of USB, in this case USB3.0, you only have 5V to play with which will be the right amount to power 1x HDD. If you start putting 2 or more HDDs off the one USB3.0 Port via a hub there won't be enough bus power to fuel the drives. The drives will still run with low power but not as they are intended, it tends to throttle the speed as low at 10-20MB/s. Of course, if your drive takes external power this is a non-issue. The other point worth noting is that sometimes USB Hubs require external power, if they do it's likely they fuel their ports with the Hubs Power not Bus Power, in which case you won't get throttling due to limited power. Does that make sense? Thanks for raising the point!
@@filmdrives6724 Ohh I was thinking of a good powered USB hub, as I assumed you had power to charge the lappy. But yes, I guess your method can make it possible to unplug from power every now and then :)
Hi mate! Just a silly thought: instead of clicking on the drive to erase/format (in Disk Utility), have you tried clicking on the volume? So instead of erasing "Lacie Rugged", try erasing "Lacie Setup". Might solve the error that you always seem to get.
Hi Diego. Thanks for that thought, not silly at all and I appreciate the thorough explanation. I prefer to format the entire drive as habit just encase there are multiple partitions. Sometimes Lacie Pre-Format their drives as MBR rather than GUID which has been problematic, a clean drive re-format guarantees a new setup exactly the way you want it. But I'll test you method on my job the next two days and see how it goes. Thanks!
Do two cards at the same time. One for A Cam and one for B Cam.
love u
what does "rushes" mean ? i am not native english. sry
I understand that it's uncommon for people to use a Windows machine, but as a DIT isn't it our job to ensure the files can be accessed? If the drive does end up the the hands of a Windows user we would need to export a whole new drive for them. Just because it's more common doesn't mean it's the best solution.
Am I missing something about why to use a Mac exclusive format?
No offense to you PC guys out there 😂
Well, as far as I can understand, the problem you presented is usually solved before shooting with production. Before shoot you need to ask them what machine will be used in post, and if it's a Windows one, you won't be formatting in Mac exclusive formats. If they don't know, well, you need to find out. Ask the editor, he'll like you for that.
Everyone I know uses expat so it’s cross compatible
defaults write com.apple.finder _FXShowPosixPathInTitle -bool true; killall Finder
Please don’t tell people that lacies are the industry standard as lacies are way too slow for most productions these days. I’d say G-tech G-drives have been industry standard for a while now plus they don’t slow right down the second they get hot like lacies.