24. Can Apple Vision Pro Change Filmmaking Forever?
Vložit
- čas přidán 5. 06. 2024
- One month ago, Apple released the Vision Pro augmented reality headset.
Today, we're using it to pull focus.
In this video, we discuss a camera assistant's requirements and challenges on set, and how Vision Pro can solve those problems.
From testing to using it in production, we'll show you our workflow, budget, and even 1st AC Amritha Mary's thoughts on why this could be the future of pulling focus on set.
Do you think the Vision Pro could serve a role in future productions? Let us know your thoughts in the comments, we're always deriving inspiration from your feedback. Don't forget to subscribe and follow our journey in building the future of workflow.
0:00 Intro
0:19 First Test Focus Pulling with Apple Vision Pro
0:41 Who is Strada?
0:58 First Camera Assistant Amritha Mary
1:54 Challenges of a Camera Assistant
3:24 How Vision Pro Addresses Challenges of an AC
3:58 What is Pixel Pitch?
4:55 Wireless Connection with Vision Pro
5:25 Vision Pro Focus Pulling Monitor
5:39 Using the iPhone as a camera in tandem with Vision Pro
5:59 Using APowerMIrror for Monitor Viewing
6:48 Our Monitor Budget
7:40 Monitor Mobiliity
8:28 Camera Metadata with the Blackmagic Camera App
8:58 Tentacle Sync E System for Timecode Sync
9:53 Wireless Lens Control
9:56 Atlas Mercury Anamorphic Primes & Beastgrip DOF Adapter Mark 3
10:27 Tilta Nucleus Kit
11:21 High Mobility
11:53 Edelkrone Slider Plus
14:15 The Potential of Augmented Reality
15:38 The App Store Advantage
Director: Michael Cioni
DP: Shannon Stutenroth
Producer: Pete Cioni
Line Producer: Margot Bradshaw
Production Designer: Amalia Romploulias
Editor: Andrea Otto
Motion Graphics: Andre Castro
Colorist: Nick Lareau
Gaffer: Jacob Abrams
Key Grip: Nick Lundstrom
Swing: Ryan A. Hueter
Sound: Fletcher Alliston
Leadman: Roman Salcedo
Art Assistant: Jordan Stringfield
Camera Operator: JP Dellanno
1st AC: Amritha Mary
2nd AC: Martin Velasquez
BTS Video: Clyde Bessey
BTS 1st AC: Tyler Beckerman
BTS Photographer: Lorelei Rubik
Special thanks to
Dan at Atlas Lens Co
Beastgrip - Krátké a kreslené filmy
Using a monitor in MR vs a monitor IRL isn't going to change things if you stop and think about it.
Better to ask yourself what does MR exclusively unlock? Spatial information. For example, the MR user could be given the superpower to see a dynamically tracked lens frustum, focus plane and DOF range all virtually projecting out of the cinema camera as it moves. This would make pulling focus, even within the most challenging scenarios much more intuitive.
I like the way you think
As a boom op this would also be brilliant. Not falling over monitors on a tight set, or having to glance at an iphone to check framing would be a godsend.
I never thought of that! But the notion of other crew members having access to their own wireless monitor is a brilliant use case. The cost of a monitor + wireless send is a lot more than a Vision Pro. And I love the idea of audio being able to see the set (and booms, etc) without having to ask village. Very cool idea!
Terradek's aren't cheap even streaming to an iphone. Plus I've noticed people getting picky about who gets ON the monitor channel. We've been having people sitting outside the studio in a van with Terradeks hoping for a peek.@@Michael_Cioni
It’s pretty impossible to wear this device long term, it’s too heavy and all of the weight rests on your face.
I’ve spent additional 500$ on top of the 4K to find a comfortable way of wearing it, and unfortunately it does not exist.
Tried all the light shields that could fit, top straps, multi straps, forehead resters, modding other headsets and results are always the same, being uncomfortable and in pain. The AVP grinds your face, unfortunately.
The AVP is fragile, so you def would have to own 2-3 three headsets for redundancy.
I also work in sound on set and post and I’ve struggled to incorporate into any of my workflows.
It sounds great on paper, HQ monitor strapped to your head.
I would like to see the 1st ACs face after a real full set week. They would look horrible, possible bruised and their eyes would be blood shot. It causes major eye strain as well.
It’s nowhere near a device that could be used in the camera department.
“You’re too close to an object” messages would constantly pop up moving on set, I get those just sitting around with AVP.
The pass through is ok in good lightning but we often work at night and pitch black darkness and the AVP is not capable of being used in those conditions either. First it would severely limit your mobility, secondly it would constantly disconnect saying tracking has failed.
While there is no doubt at some point in the future this will be possible with a different iteration of AVP we are not there yet.
It needs the ability to hardwire Ethernet, usb-c etc for real professional use.
I have the developers kit and the USB connection is very limited at this point.
I couldn’t make it through this video fully due to the presenter shout-talking so maybe they raised some of these points.
Like what if the AC is wearing glasses, issues arise with distortions etc.
Another thing that made me laugh was that they had to shoot this on an iPhone to get it to work.
The Immersed Visor headset coming soon might be more appropriate, high res pass through and robust tracking is all you need really. Though I believe that’s not standalone.
The poor performance in low light would be a big problem. How you solve that without having bigger sensors/lenses is an interesting thought.
@@longwelshnight vision lense
This is insane. I love your guys' willingness to break some new ground and explore what's possible with today's tech. You're literally setting the precedent!
That's the right attitude. I expect a lot of "why would you do this?" comments, but you're point about a willingness to explore is what leads to new ideas and opportunities. Thanks for watching!
@@Michael_Cioni can you do this on the quest, since that will bring in more users.
The first thing I tested day one was connecting Vision Pro to my Komodo X via Red Control. Excellent post.
Has anyone mentioned the latency on the Vision Pro? I assume it’s not as fast as a Teradek wireless system
My first thought too, latency 🫨
Asking the right questions
Yeah there's just absolutely no way this is fast enough for pulling focus on pro shoots where the AC has to be accurate the first time and the camera is moving.
The apple vision pro is about 4 miliseconds of delay, while the screen casting app has delay, using a macbook will make it more like 5 miliseconds.
The weight is also a factor not to mention on hotter sets, it will get pretty sweaty. But there is potential.
Atlas mercurys being pulled by a vision pro is MADNESS.
I love it
Mike, I met you in 2010 when you were doing Light Iron Digital. You gave talks when RED was having in-house events in Hollywood. I love that you're still going hard on innovating filmmaking tech. I salute you, chief.
thank you! Stay curious:)
Really cool idea and as a focus puller myself, the first thing I thought when the vision pro was announced was focus pulling.
But in practice, this seems like it gives you exactly what a nice monitor on a stand gives you except with more failure points. There also doesn’t seem to be a zero latency solution for image transition on any camera system. Then you have to worry about battery life. The reported battery life on the system would barely get you through a scene much less a 10 - 12 hour day.
Cool idea. Nice to see it tested. But not viable in a professional setting.
All good points. Definitely too early for this to be widely adopted in everyday productions. But if we want to see tech like this become more mature and meet all our needs at scale, we need to try it, test it, and provide feedback to manufacturers so they can improve the ecosystem. I expect a lot to change in this space over the next 10 years.
@@Michael_Cioni oh 100%. It needs to be tried and tested. I’m happy it was!
The quality of this video is absolutely bonkers! Instant sub, keep up the fantastic work
My first impression of this video was WOW! Brilliant production, thank you for posting.
damn this video was incredible. great work to everyone involved!
Really cool idea, I work as a first I wouldn't mind this but there seems to be some limitations. Personally, latency has to be extremely minimal. So if industry video transmission companies, like Terradek, get involved then I would be game. Also, I move around a lot. Carrying a big Arri or Venice is already really heavy. Still, I can see this being the future, just have to work out the kinks
As a focus puller of 15 years feature film level experience I can tell you this woman isn’t what most on the industry would consider a focus puller.
In future please find feature level people to test this kind of stuff not film student level.
The fact she’s pulling of a laptop screen not looking at the action but essentially away from set is a massive red flag. No professional focus pullers would be caught dead using this system.
I'm a film fan and alway love seeing behind the scenes videos but I'd never consider being part of a production. Still, your videos are so well produced and I really like seeing and hearing about all the different tools and apps you use. Also, I really appreciate that you don't use multiple takes when you're speaking to camera, I always find that technique really annoying! Can't wait for the next video 👍
Amazing work 🎉 it’s so fun seeing the process and knowing that even professionals want to use these technologies or even acknowledge them as potential tools on set. Will be following this journey long into the future. Can’t wait for what’s next!
Awesome and criminally underrated stuff here guys!
"Criminally underrated" - that's funny. I'll take it, tho:) Thanks for the comment!
Im sitting here trying to figure out why Strada doesn't have millions of subscribers.
fr man
Because they are not charismatic I think??
Because this guy is annoying AF and didn't ask any of the right questions as to wether this is really a viable solution or not.
Great effort but you didn’t realize a major issue with the Vision Pro …latency. Until this gets resolved I highly doubt the Vision Pro will be used to pull focus remotely on pro shoots.
We were surprised, but didn't experience noticeable latency that was more than any normal cinema configuration. Felt very much the same in terms of latency (probably somewhere around 300ms give or take in my estimation).
@@Michael_CioniYou seeing latency vs a professional focus puller in a shallow DOF situation are two different things. Plenty of people who use “low latency” wireless image transistion devices don’t think they see any latency. But it can make or break a shot extremely easily. There is a reason why Teradek is the gold standard in the industry
Latency is the big thing for me in this situation. Using a wifi based system is the worst possible option when trying to fight latency. That combined with the inherant latency and inaccuracy from the nucleus is a recipe for disaster on tight shots.
Indeed. I agree with you. I have several Teradek Bolts. To clarify what I'm referring to is the innate latency between a lens, a sensor, an image processing pipeline, a video output signal and the wireless transmission. Even tho Teradek's are marketed as zero latency, there is latency before they are given a signal. In other words, it takes a little time to get light from a lens to an any monitor, and 200-300ms is fairly common. Only film cameras with optical viewfinders (and video taps) are able to avoid this. I agree with you that poor latency can absolutely blow a shot. But whether they know it or not, the professional community has become accustomed to acceptable latency due to digital camera image processing pipelines. I'm intimately familiar with professional camera systems and monitoring latency and from my experience, this very early iteration of AVP wireless monitoring is on par with the latency of traditional cinema systems. Get your hands on an AVP and try it for yourself and LMK if you agree. Thanks for the comment!
Excellent video!!!! I just got mine and I see the potential of this to replace so many things it is unreal.
This would be awesome for me to use as a boom op as well. Getting to see the exact letterbox and how much I can push it!
Holy shazang!!!! I dabble in video making but this is next level mind blowing! I found your channel about a day ago, and I'm hooked! Apple and off the shelf products that you can buy means anyone can make their own movie (albeit not totally big budget film, but hey, less than big budget can win Oscars) and puts the little film companies in the mix! You are breaking the mold and I'm so here for it!
Amazing insight(s). Thanks and kudos to your team for trying new technologies. Subscribed!
This is absolutely amazing, thank you for making this project and sharing it! It’s what I’ve been searching for. I’ve been looking for a solution very similar to this for live video into a Vision Pro. My next hurdle to solve is inputting multiple live video feeds to the headset from different types cameras (at least two.)
I’ll try this on my show. A decent amount of location work with tons of CG elements
Amazing. Now only to see it used on cinema cameras. And if this setup allows to pull focus more precisely than regular video village
No way. Latency
I can't begin to explain how well put together this video is.
Wow! Thanks! Check out Episode 23 and other episodes to learn more! Thanks for the comment;)
Pretty cool idea. Now I want to try it out.
great video. excellent work
Nice idea, but surely at this stage the resolution and lag is going to be way off for accurate focus
This was an amazing video, informative, entertaining, and creative. I think this was the first video from this channel I’ve seen and now I’m subscribed.
Apple should have this video on their page! Great video! Professionally made!
That looks pretty fascinating. However, there are still some problems that need to be resolved. As a camera assistant myself, I always have to be able to rely on the equipment, and that doesn't seem to be the case here yet. Relying on third-party apps can really cause problems on set. In addition, streaming via WIFI and HDMI creates a latency problem, and as a 1.AC every millisecond counts. But nonetheless, I find this kind of experiment really enjoyable and hope that it might become usable at some point.
This is wild. Love it!
I really love your videos. Storytelling, visuals, sympathy, everything is mint. Well done. Seriously.
very good
If black magic comes with the apple vision pro app to seamlessly control the video rig, next gen vision pro will sell like hot cakes(if the price is affordable) 😅.
My head blown out 😲. Interesting ❤
As a 1AC, this is definitely interesting. Where the tech is right now it's not realistic at all for professional productions with the added latency, reliance on iOS and lots of extra points of failure. That being said, when VR/AR headsets get smaller and this tech improves, it might be a viable option for some productions
I agree. This is the "top of the 1st inning" so innovation will be rapidly expanding over the next few years in this field. However, of all the limitations and issues we had to problem-solve, latency wasn't one of them. We found that the latency was around 200-400ms from lens to village, which is pretty similar to an Alexa > Teradek > Monitor.
Wow… just wow… this is awesome…
This is great! I really hope to see some integration with teradek tho
Amazing content! This is the future 🔥
thanks ! aweisom !!!
Another awesome experiment pushing the limit what's possible!
Would love to see how you'd set this up if you flipped the experiment - focus pull with the Vision Pro on a pro set with (mostly) unlimited budget. I could also see a reality soon where the lidar from the camera tracks a mark you set and shows a realtime distance line in AR
Not a bad idea. Whose budget?? ;-P
@@Michael_Cioni Panavision 👀
two things I'd like to know/add: high resolution display: yes the vision pro has one, but at what resolution does the stream come in and what is the latency of it? And also how do you get the streams from other cameras into the VP or is it just for iPhones?
Great question! The Vision Pro is capable of display much GREATER than 4K, and the tool we used APower Mirror says it was capable of 4K, but we weren't sure if we were actually getting that. So my guess is there will be major innovation opportunities to ensure 4K transmissions to VP (much like watching a movie off AppleTV+ or Netflix on VP). So the technology works wirelessly and only needs to be integrated in iPhone mirroring tools (and subsequently other wireless transmission tools). thanks for the comment!
This is promising . I definitely would love to see this tried with a proper cinema camera.
Me too! We're going to NAB in 2 weeks and I'll be speaking with wireless transmission companies to see what they are thinking about in terms of adding an OS solution to the portfolio so Vision Pro can be a recipient of their signals.
@@Michael_Cioni that would be game changing if they did. I’m on the fence about getting one soon .
simply amazing
It’s a great use case for using Vision Pro, imagine when developers start making proper tools for film makers on Vision Pro. The only thing I would say is this is definitely mixed reality and not true AR. I think you couldn’t ever get this good with true AR, no matter the form factor. High quality displays in what is basically a mixed reality headset is the only way moving forward and with slimmer design and a broader field of view, it’s exciting to see what’s next.
In terms of the actual future, I think the Vision Pro is the camera; computational photography simulates cameras and lenses; depth mapping recorded for VFX and of course, the VisionPro viewer of the movie can take over eye-tracking focus from the camera operator. Other spatial camera devices will make positioning and moving the camera possible, so you aren't tied to first person point of view
Criminally underrated channel. So much money and hard work behind this channel and it really shines through
imagine projecting a colored plane of focus and field of view on the camera itself, using lens metadata and a tracker on the camera, the vision pro could project a plane in its 3d scan of the environment so the focus puller doesn't even need to see a camera feed at all. If everyone else on set could see the cameras field of vision in a cone, it'll also be really easy for grips, gaffers and soundies to keep booms and lights out of the frame, art dep could focus in real close on placing props on effective spots, completely removing the guess and check that typically happens with the crew and whoever is viewing a monitor.
Wow. Killer idea.
This is basically what Unreal Engine does. Probably not too difficult for a virtual tracking studio now as it is
What adapter did you use to mount the cine glass on the iphone? Or was i missing something?
Cool video.
Great question! Check out Episode #23 and we go into details about the glass and custom adaptors!
I mean, very interesting concept but i think that there should be a lot more to the apple vision pro than just creating a virtual monitor somewhere on set. If you want more mobility you can rig a 7" directly on the hand unit of the focus kit and walk around with that and I think that would be cheaper (and safer) than having someone with weird goggles on set all day. Also an awful lot of focus puller (maybe more seasoned ones, but everybody should work this way) prefer to use the monitor as a focus checking tool and pull focus by measuring distances, using marks and just seeing distances by eye, it's a bit more difficult and you're basically pulling without seeing the image but in this way you can be always aware of where the camera is in relation with the actors. If someone miss their mark if you are just looking off from the monitor you are not aware of that until you see the picture out of focus and in that case there is nothing you can do, there is an out of focus part in the shot.
I think that if we want rally to implement the Vision Pro on set we should use it to create something that it would be otherwise impossible, maybe combining the lidar scanning, mapping the lenses and the camera so that you can see the frostrum of the lens, the current focus position in AR, the minimum focus distance, maybe you could be able to add marks. Also you could have camera infos floating around without being relegated to a rectangle somewhere.
I don't want to criticise your work and your video, I know that something like I've described doesn't exist yet and the vision pro just came out so everything it's new, but your video made me think a little bit about that
You guys do know the sync e doesnt need that cable to work w bm camera right… it pulls it directly over bluetooth in the same fashion of how the sync e’s talk to each other. no cable needed.
It'll be a great idea to try.
Dude...I've been working on this!!!
Very good video.
This was so cool! How did you work around the Apple Vision Pro's limited battery life??
Batteries on both Vision Pro and iPhone 15 are something you have to factor in. So we bought mobile USBC battery packs that could live on or near devices that needed frequent charging.
Super interesting!
great video
Very interesting video.
You can use monitor control in Sony cameras probably. That is interesting
that was damn interesting. subscribed
Cool idea like to try the job
Nice! Next to put 3d eyes on the camera 😛 together with the sonar distance
Well done, mate!
Thank you!! Check out Episode 23 and see how we did the anamorphic shoot!
Awesome and criminally underrated
4:30 vision pro can at most display 100nits brightnes. Also you view it from so close that it is not enough resolution to not see individual pixels. And you are better off looking at a phone or a real monitor.
Fantastic proof of concept. Early adopters & experimentalists are awesome 👍
This is a call out to like RED, Arri, Sony and Blackmagic. This NNNEEEDDDSS to happen. I love the idea 💡 📸🎥
agreed!
Great idea, but for the price of the Apple Vision Pro, you could buy a 6K Cinema Camera. I wonder which one a small film studio would choose to buy?
Im a 1st ac and this tickles my brain now to find the cash for an vision pro
Comparing the price of a full Preston FIZ kit? This is nothing
@@Eddie32791 oh I do know my brother! I have a WCU-4 and Nucleus out in the rental circuit .I just admired but could not justify paying for the vision pro but this swings it in my favour
How about the delay from apple pro vision to núcleos?
We didn't detect any noticeable latency between the Nucleus and the Vision Pro. All cameras to wireless transmissions have some latency, but we've done this with professional tools thousands of times and honestly it felt like the latency to Vision Pro was similar to Alexa or RED or Sony. It didn't impact out shooting. The weight and Timecode were bigger problems. Thanks for the question!
Let's hope so! Currently, I even cannot use my iPhone 15 pro (USB-C) as a camera monitor... -.-
I’d presume 6:55 you’d hdmi from iPhone usb-c tho to share normally?
8:46 ah
awesomeness
Not really AR... Apple promotes it as "IMMERSIVE Reality," because the user isn't taken out of the environment. Nitpick, I know... but I was deeply impressed with how you've taken a very basic visionOS release and incorporated it into your workflow. As an actor, it'd also be helpful for us to see the directors vision on a shot, so we'd know exactly how to act for camera (for those directors that use a video village). Overall, very very impressive!!!
Finally i see something useful. Cudos
Low Latency? Not sure about that over that Wifi App... What was the delay like?
Every cinema camera from lens to sensor to image processor to wireless transmitter has some latency. We found this felt to be just about the same.
If you put on a VP they are about 12ms latency (imperceptible). The APower Mirror app presented a result that was probably around 300ms, but that includes the delay of the camera. So it was totally useable for focus pulling with room to improve.
I really thought this would be pulling in 3D. Having actual 3D spacing would help focus so insanely much. Why else would one really need a Vision pro.
This is genius
ive always felt like I was born too soon.. this is only more proof
I may be in the minority here, but I have no idea what “focus pulling” meant and was lost most of the video. Just thought I’d give some feedback to improve your videos in the future.
That's crazy that you don't even mention that 1st AC happens to pull focus only on low budget projects, and if so he/she should have a reliable 2nd AC who will take care of at least power management and basic body handling. (normal camera crew is: cam op ( if dp don't operate himself) focuspuller 1st AC 2nd AC, sometimes trainee and VT engineer if its bigger proj)
Couldn't you Do the same with A VR Headset or an FPV Headset?
maybe focuspull but you normally need to see the markings on your remote and also, as she handled the camera while wearing the VP, this wouldn't be possible in VR, only AR let's you interact with your surroundings
Just make sure to have 15 batteries to last the whole shoot.
Finally a video about the AVP worth watching. Thanks for seeing the possibilities, and enabling capabilities instead of just being another whiner complaining about what the AVP isn't.
Hey! Thanks! Glad this was fun to watch and informative! I always look at things from the perspective of glass-half-full. And with brand new technology, it's all about getting excited about what's possible. Thanks for the comment:)
The latency on this must be terrible. I personally put a 12.9inch iPad on a chest rig as a back up monitor to a real focus pulling monitor. I send a latency free signal to the focus monitor and then go out from there to an Accsoon cineview SE. it gives me a more mobile monitor and gives a signal to other departments who want a feed for their smart devices. I wear my nucles on my right leg on a leg gun holster.
It surprised us that there was hardly any noticeable latency added by the APower Mirror (app) wireless transmission from the iPhone (running Blackmagic Camera App). Had we had that issue, we would have brought it up here along with other issues we pointed out (timecode $ weight).
Today this setup did work, but it’s not viable this early. That’s why feedback from professionals like yourself to relevant manufacturers is how improvements will be deployed. Thanks for the comment.
Plot twist: in the future the camera will be the headset.
Wow.. yea I mean the vision pro has high resolution displays but that doesn't mean it can reproduce a high resolution display in 3d space. A floating display in AVP is worse than a $99 budget Walmart monitor.
THIS IS A APPLE VISION PRO AD INSTEAD OF A ACTUAL REVIEW
Amazing. But that flex on the rod freaked me out. 😬
😮
I had a short career as a cinnamon photographer, went too niche 🤷🏻♂
Why would you need it
And move quickly?
I can move quicker without this on my head which is destracting my eyesight and range of view plus not to mention my perception and instinct of how someone else near me is going to move
And relatively cheap?
If you can't do it on a high end monitor then get another job
Are they paying you?
Thanks for the comment. To be clear, no one is paying us:) I've been working on high-end cinema for many years and what helped me develop as both a filmmaker and entrepreneur is to remain curious about the intersection of art and technology. Your points about mobility are valid, which is why it's important to remember AVP is extremely new and AR integrations in the production space are almost totally nonexistent. But curiosity, testing, and user feedback will help fuel innovation in this space that will provide solutions and opportunities that may seem foggy today. This has always been the case with new technology in the production space, but eventually, fads become trends and trends become standards. Appreciate the feedback!
For a bit there I thought this was an April Fool's video!
What about the lag
We found that there wasn't any noticeable lag from camera to headset. There's always a little latency between any lens, sensor, video signal and wireless transmitter, but we felt this was about the same as a normal cinema camera and didn't impact our focus pulling. There are other stability issues with timecode, the app and the weight of the device, but we were surprised latency wasn't an issue!
Very cool, has a lot of potentials! Can I use Quest 3 for filmmaking as well?
Probably tons of delay as they distribute to multiple monitors
We thought that would be an issue, but the truth is, I didn't mention latency in this analysis because on set there simply weren't any latency issues! Most cameras from lens to sensor to video processor to wireless transmitter has 200-400ms latency. We found that the iPhone to village was about the same. In other words, it felt to be about the same latency on any normal system. I didn't scientifically measure it, but from thousands of hours on set, it felt about the same as I would expect from a cinema solution with a Teradek transmitter to an SDI monitor. Thanks for the comment!
At this point this solution way expensive 😊👍🏻
Oooh now we cooking