Although I have no hands-on of the AVP, but I can see it's a parallax effect post-proseed on by the AVP spatial video player, between the player window border and the content video. Many others, including myself, were expecting a parallax between the close-by object and the background of the video itself. This did not happen since in core the technology is stereo vision 3D video.
I'm not sure that I like the dreamy blurry border with its parallax artifact . . . or at least I would like to have the option to be able to turn it off. But then Apple would have to admit that "spatial video" is just normal 3D video.
You are correct Kris. I believe Hugh is partially wrong that it's not 3D video. I'm a long time video encoding researcher and have looked any many factors over the years. With spatial video it has to do with how the AVP player and Meta Quest spatial player hosts the video. Apple takes the 2160 4K-3D video and encodes them to 1440p SBS video for playback. Then crops 10% into the video that is covered up by the soft border. Then an extra 5% of the video is taken to the diffused edge of the player border. This gives viewers the parallax option for shifting in any direction and noticing the video revealing itself. Do you remember when Apple did this same display option for the lock screen, all they did is crop into the image with the viewer to make you feel like to were controlling the view of that image. When I look at spatial video in other players on a Quest 3, I notice the edges of the video, then play them back in the Meta spatial player I can see the 15% of the video that has been cropped in. I think sometime in the near future someone will figure out the metadata that tells the media player to playback the video in those setting.
That’s a good point, and true about the Lock Screen. The GitHub project SpatialMediaKit may be a good starting point for forensics even though itself doesn’t export metadata@@brianparker3194
It is 3D video! They are using software and the cutout window to move the video around (rotating it, zooming in and out etc) in space giving an illusion of looking through a window. Most 3D video out there is static attached to your face. If you move your head the video moves with it. However, you see the full video so distortions would be obvious if they were to zoom in or rotate the video potentially making you dizzy. You can achieve the same effect with a 3D video in any game engine or 3D Video player that allows you to move the window, zoom or crop. They are just doing it automatically.
True, but this just points out the magic is often in software. The average user of this technology is looking at you as a tech nerd when you say "But it is just 3D video!!!". They aren't going to care about the details, they weigh the user experience, which is always a composite of hardware and software... There is a lot of software magic in the AVP, taking that impressive hardware spec and doing things with it that are original from a user experience stand point. Ultimately the user experience is what matters, not the technical details developers or enthusiast might pull out to try and diminish the novelty or claim some equality that really isn't there... Techies always under value software contribution to the experience.
What do you think...the Quest update 62 can let you play those spatial video's. Apart from the resolution, would they look like in the Vision Pro? Or just like ordinary 3D? I haven't found any spatial movies yet. Do you have a source of examples, maybe your own test results? What software do I need to play it in my Quest3?
So the claim is that the spatial video is just 3d video inside a shadowbox, and not actually the slightly volumetric video we were shown in advertisements. It's hard to tell from your example if the actual video has more than just stereoscopic depth. One test would be whether or not you can tilt your head 90 degrees while watching and still see depth. The shadowbox effect looks pretty cool, but I see the complaint. True spatial video is something that exists on Leia's Lumepad. The limitation is the screen viewing angles and the quality of AI-driven depthmaps.
I'm not well versed in video tech but at the end of the day it's just MV-HEVC, which was finalized in 2014? So this is good old stereoscopic, nothing new. Just "retina"™ 3d video.
@@SToad Yes, that's right. It's so *YAWN* old news that, get this, literally no other headset natively supported it! And there was no way for consumers to actually shoot videos in that format without committing to spending a bunch of money on dedicated camera hardware and paid software, and committing to the time spent in the workflow needed to create MV-HEVC videos. And what did the entire VR industry do with this "good old nothing new" format since 2014? Absolute fucking jack shit. Apple came along and in one fell swoop made it easily accessible to anyone with an iPhone, and suddenly you care.
This youtuber is wrong. The spatial videos are just regular 3d videos placed behind a hazy window. Moving the video left and right is NOT parralax and not at all the insane ammount of parallax present is the middle of the false advertisement videos that showed you could rotate your perpective to see completely behind people. I would just be thinking these small additions are nice ideas and cool if Apple didn't once again invent a new term and claim they invented something that already existed, this time since the 19th century. It's sad that most tech youtubers don't know enough about what they're reviewing. But this video showed the 4k video feed from the headset, I saw no one showing that, and that's really cool.
I just got back from my Vision pro demo. While the display is super sharp and the spatial video looked awesome I can tell you there is no parallax. Just visual trickery. The two cameras can not record the info behind the subject or outside the cameras fov. The only way to achieve that would be to deploy ai predictions or to remove the subjects from the image to record the environment in 3d. The tracking was good though. Can’t say it would be a decent purchase even if I had the money to blow though. I kinda got bored during the short demo and my face was already hurting from the weight.
Thank you for jumping in and provide feedback. It is not "seeing" behind the object as stereo rig can not capture that. We need volcap rig. What I am saying is there is Parallax with maybe software level or player level + AVP internal magic to create motion estimate parallax. The 3D video is not stuck to my face. You will need to sideload a actual 3D video (what we used to on Quest 3) and put it side by side with the Spatial Video to really see the visual different. I have tha ah-ha moment when I compare 3D Video to Spatial Video side by side inside the AVP. This require you, of course, to own the AVP, xcode your way into the device and load up regular 3D video instead of Spatial Video. But seeing it side by side really make me think it is way more than just 3D video playback.
Hi Hugh, is it possible for you to share a spatial video captured with the Vision Pro in 3D here on CZcams because I'm interested to see the difference in depth and quality compared to the iPhone 15 Pro Max's spatial video (I assume the Vision Pro has a wider stereo base, but I'm not sure about the video quality). Also I was wondering if the Vision Pro has the same format for capturing spatial videos limited to 1080p/30FPS like the iPhone or does it support better formats like 60FPS?
Love your videos Hugh, but in this case am I missing something or is it just depth map from stereo giving that Facebook like parallax? If it is... It's underwhelming and not 'real' parallax (missing information /pixels) being created. Reminds me of the early days of "rubberstamping" 2d to 3d conversions as done on films like clash of the titans. Can't tell much from your video above, but would be good to know if this is the case.
It is not REAL parallax - as the camera is still Canon R5C with the Dual Fisheye lens - so NO volcap. But it is real 3D. And Vision Pro make it look better on top of the 3D data and create depth parallax off axis movement based on your IMU data of the Vision Pro. So the 3D look a lot more comfortable instead of 3D stuck on people face and not react to head movement if that make sense
It can't 'create new pixels and visual information' from what is not already recorded stereoscopically (by the interaxial of the Canon 3d lens for instance) so what it's doing is, creating a depth map from the disparity (depth from stereo) and that depth map 'stretches' edge pixels to give the 'illusion' of parallax - driven by the IMU. This is my understanding (as I've not delved into the vision pro, nor will I buy one). At best this is pseudo-parallax not true "spatial video"
Great content as always! One correction: PSVR2 already has foveated rendering (with eye tracking) and it’s a cheap device. Still having 7 VR headsets at home I can’t wait to try the Vision Pro once it’s introduced in Europe. :)
Thank you! Almost every glowingly positive review of the AVP I see does this - states that this "new feature" is something NEW to VR when it's been around for years. There is almost nothing new that I can see in the Vision Pro. That isn't to say that the Vision Pro doesn't do it a bit better (as in the case of the Spatial Videos perhaps).
People are confused by the differences between "spatial video" and various 3D formats taken by different devices. It would be very helpful if you could prepare some sample video files for us to comparison-view in our Quest 3. We need similar shots taken by the AVP, the iPhone 15 Pro, standalone 3D cameras like the Qoocam EGO, the Lume Pad 2, and the Quest 3 method.
I've been taking a lot of VR180 pictures with my Lenovo Mirage VR camera. I would love to transform them into the new Apple Spatial Pictures. Any idea how?
Hugh helped with my old school uploads of SBS and OU which play back fine on CZcams. We need full spatial streaming soon! youtube.com/@spatialtests?feature=shared
So is the spatial video format actually storing extra stuff, or does it just put the viewer in a different mode that has some tricks added at the edge of the view box?
From the metadata I can inject into my own 3D video, yes lots more info. But not something you shoot but something the AVP calcuated the data you provide and present that inside AVP. Let me go more indepth in the next tutorial. I did lots of testing but my brain is all jumbo now.
Would you be able to test what the max resolution Vision Pro is able to display? I have several 10K VR180 photos from a GH5 rig which look good on Vision Pro, but was wondering if it would be worth upgrading to a dual Fujifilm GFX 100s rig or similar cameras for 16K or 24K VR180 photos. Also is 8K the limit for video playback? I tried playing the 12K FM DUO sample video you linked in one of your previous videos, but I couldn't get it to play on the Vision Pro.
I’m thrilled to see you’ve worked with ESPN+, I’m hoping you’re working with them on UFC because I’d love to watch MMA in VR180. I think the relatively small octagon would be easy to capture and the experience would sell the technology.
UFC on Meta is pretty dope. Have you seen it? I was not involved with the UFC, but my firends shot it in the ring on last coupled big fights. It is on Horizon Worlds in Meta btw.
Amazing video Hugh, we are slowly getting there! I am not sure btw if VR180 can also be spatial on the Vision Pro btw. And what would be a small camera you recommend to capture spatial video (in a frame or 180degree)? I feel the Canon is too big and will scare the people I plan on interviewing
From my experience, no or not yet. VR180 will lose all parallax illustion including Apple own production on Apple TV. But we don't know what kinda of crazy thing Apple will pull when launch in Europe and other country.
It would be really great to have a VR180 with some parallax, the ultimate immersive experience@@hughhou Personally what do you think is more appealing at the moment, VR180 or spatial video?
Hugh, There is parallax in the content. You are just being fooled by the 3d window that is holding the content. I've been recording a lot of spatial videos, and have been really disappointed in the lack of any paralax. Open them up in the "immersive mode" and you will see what I mean.
Před 6 měsíci
I feel the same, there is no parallax yet. I know that Apple is encoding the lidar data in the videos, but not yet showing the parallax. I'm confident it'll come at a later stage
Not just edge blurring - but also head tracking and hardware level parallax. And yes. The standard called MV-HEVC. Not owned by Apple BTW. They are just the 1st to adopte the format and making somthing amazing and less motion sickness.
AVP 'spatial video' is standard stereoscopic VR180 (or other wide field of view) viewed through a narrower rectangular window. The ability to 'look around' is essentially shifting your gaze (moving the center of the VR180 image) to the side. It does provide some limited illusion of the ability to look around objects, but it is still essentially just stereoscopic video. You could produce a similar illusion by putting a standard 3D video behind a smaller window and shifting the projection to correspond with change in viewing angle. This form of spatial video takes immersive 3D VR180 and throws away both the immersive experience and a lot of the image. The advantage is that you can have multiple images open, it's a trade off between immersion and multi-tasking.
Wait you said "VR180" - so Spatial Video now work with 3D 180 video format now? Last time I check AVP does not support any immersive format. The metadata is there but the player does not do anything. If VR180 work natively - this can be huge. Spatial or NOT, this will be amazing!
@@hughhou I think his example of VR180 as a recording source is simply a bit extreme, as it's far more image than what's required for this sort of "perspective shift". It's more of a 3D video with a smaller "cropped window" into the image than the size of the 3D image, creating an illusion that you're looking at a scene with shifting perspective. But really, you're just moving around the 3D video relative to your 6dof movement. It's still just a fancier way to view 3D. Apple showed videos of people experience the layers of depth within the 3D image shifting as you move in space, which is not what actually happens, so it seems. And this is why people are saying it's a lie. the trees in the background don't shift to the left behind the people in the forground as you shift your view to the right. That's what they implied it does. From what I'm seeing here, it does not. this clip shows the implied perspective shift, though it was just the video shifting itself, which happened to be along with the onlooker's perspective: czcams.com/users/clipUgkxMUk7rEH4OcKSCXxVkCiw2iRnABhsc5IJ?si=VtBnbHIksZLMxwJM
That's really cool that moving your head while viewing the video gives a parallax effect. It's these small touches that put Apple above the competition. There's some very clever algorithms at play. I can only imagine it's using the depth information (or calculated depth information in the case of converted 3D video) to build some kind of layered, textured model in the background. Very clever.
I'm not sure if there is parallax. It looks more like a crop and pan which gives the illusion of seeing more. I could be wrong though since I haven't tried it myself.
I think you're right. It's a parallax effect only with the fuzzy frame around the video. But when I experience it, the illusion is very effective. The experience of viewing spatial video on Vision Pro is noticeably better than viewing the same spatial video converted to SBS using the Spatialify app and viewed on the Quest 3. I haven't found good ways to view my own 3D content (such as from the QooCam EGO) on the Vision Pro. There are a couple of apps on the App Store, but they don't work yet. Their developers are hard at work and release updates often, but they aren't there yet.
Yes! It def look like that way. The end result matter. Everyone see it said it’s better than 3D and it’s better - more comfortable. With eye tracking engaged. It’s more than 3D but def not Volume.
You rock, but this video is misleading. There’s no parallax with Spatial Video except for the frame. It’s 3D video in a cool wrapper. What app are you using to view 3D video on AVP?
Thank you! Just the native app. It can be true the video itself does not have parallax, which I can not validate. But it looks different than 3D - I think the head movement tracking, edges effects, parallax illusion all together make a a more enjoyable viewing experience than just 3D stick to my face. I think that is the point. It is still 3D but it is MORE THAN 3D, not JUST 3D. Have you get the chance to watch the video youself?
@@hughhouThanks for the reply! I wasn’t aware that any native app in Vision Pro had the ability to view stereo video from a camera other than Vision Pro or iPhone 15 Pro. Have you made these files available for download? I’d love to try them on Vision Pro.
@@hughhou I tend to agree with you. The high quality of Spatial Video straight off camera without the complicated and laborious workflow is the big deal in my mind. I prefer the Spatial Video off the iPhone 15 Pro. The depth does not seem so exaggerated as with the Vision Pro and it is in a 16:9 aspect ratio. While not true parallax, the overall experience is better than 3D content I have seen. It will be interesting how Apple innovates here with some AI-driven (potentially Gen AI) augmentation to provide some degree of parallax and some synthetic volume. I think this would be a great area of innovation for Apple that would be highly differentiating. I hope Apple does this because they did mislead everyone with their video that showed full volumetric parallax.
Yes indeed! Quest Pro is not hardware level implementation. PSVR2 only have game engine render not live action solution. This leave almost on one has this feature. We hope PSVR2 change this, but so far not even CZcams app work correctly on PSVR2 - PSVR1 work great...
Thanks, I learned so much from this episode! Important question for you: I've been shooting on the Canon VR Lens and preparing 180º 3D content for viewing on the Apple Vision Pro. To play these video files in a 180º FOV on the Meta Quest, I simply press the button on the bottom right to change viewing mode, but how will I do this on the vision pro? Is it possible? I noticed that you played some of your videos shot on the Canon VR Lens during this video, but they were viewed within a regular window as spatial video. I'd like to view my content in its full FOV so that I can look all around.
Thanks so much for your quick response, though this news makes me sad to hear. I assumed the ability to play 180º VR video would be available within the Vision Pro. I am very excited about your solution, however I fear that its something that won't be easy for non-technical people like us to do. For example, the videos I'm filming will need to be viewed in 180º 3D VR by clients using Vision Pro, but I won't be able to install software or do anything on behalf of my clients using their own Vision Pro. Any thoughts? Also I wanted to thank you in advance for working so hard to solve this problem and many other. You're by far the most knowledgeable in the space. It'd be beneficial for all of us if Apple developers hired you as a consultant!@@hughhou
@@noel-issa I'm in the exact same boat. I was wrong to assume spatial video was basically vr180 video. windowed spatial video doesn't sound that great.
The only real difference between this and normal 3D is just the player. It does some tricks to do that parallax and blurring in the edges. You should be able to say, build a video player that does the same thing for normal SBS videos, etc.
Yes or no. The player use active eye tracking, head tracking and do live calcuation to position the inner content to create an illusion of a parallax. Not headset has this level of accuracy on eye tracking (or at all as Quest 3 has no eye tracking). So software can only do so much - you need tracking data.
You should pin a comment saying that the files are in fact just 3D video in a MV-HEVC container. The parallax effect is 100% just done by moving a viewport around the cropped video.
The Quest Pro also has dynamic foveated rendering (using eye tracking) and several games take advantage of that, most notably Red Matter 2. I believe it does it mostly on the software side, but still. Also, like others have said, that's not really parallax that you're seeing, IMO. You'll notice that the edges of the Spatial Videos are not only blurred but also closer in than the edges of the straight up 3D videos. It's like you've got a blurry picture frame that's blocking the outer edges of the 3D image/video. When you move your head, you can see some of what's being blocked by the frame, which enhances the 3D effect and gives the impression of parallax. That said, it's still a cool effect and very smart of Apple.
Thank you! Yes I mean a cool effect and presentation of 3D - so more than 3D instead of just 3D. They solve a lots of viewing challenge of traditional 3D in VR. I did not said other can not achieve it in software and some sensor data - but they just did what others has not, do desire credit on Apple part.
Oh trust me it is coming on this channel. I am shooting an entire feature film in Italy now and it will be out soon as Spatial Video example that professionally done :)
If the z-depth isn't capture by the camera then we don't have any spatial video. The video cameras need a Lidar scanner to create z-depth + video. I don't know if the new 360° cameras have this feature. Also the gaussian splatting technology is the future for 3d imaging.
I don't think you need z-depth for spatial video. Even AVP, the Lidar is not engaged. Calculating the depth with a very low-resolution depth map can be easily achieved with stereo side-by-side video / data. I think that is what Vision Pro does on the fly. I don't know exactly, but it looks different. It is not 3D.
There is more to the way spatial video is captured. To keep it non-technical, it's compressed into a 2D video with depth information attached, allowing it to be experienced either in normal 3D or a bit more volumetric with some parallax. Also, the camera quality is much better than that of the Quest 3. That said, I personally found only the passthrough to be a big step up. If Meta wanted to, they could release a Quest 3 Pro at double or triple the price with only better cameras, and optionally eye-tracking, to close the gap.
I think he's manually converting SBS 3D video to RGBD video which contains a single 2D video next to a depth map video. It's unclear whether the AVP or IPhone 15 can actually shoot in that format.
this would make perfect sense since Apple had 'portrait mode' on iphones (TDoF sensor) for quite a long time now. I have been using these for creating quick holograms in Lookingglass displays@@hughhou . When watching those (without glasses!) It feels like you can 'look around' the object. This sounds similar. Depth map attached to video and computational resolve. Can't see a reason why this couldn't work on my Quest2 if Apple was to release the code to developers. PS, looks like Polycam might have a foot in the door with 3D models (NeRF, Gaussian splatting)
Wow, thanks for sharing this! You should also test this on Quest 3 after updating to v62. Try the same raw Canon file, plus also the converted Apple Spatial Video. That will be really interesting to see if Apple is doing some “special sauce” with their files.
Is v62 out yet? I have been waiting impatiently lol. I don't have special access like other big Quest influencer so I need to wait for beta firmware available to everyone to test :(
@@just4funvr146 Meta is working on Spatial Video support in the Quest3 and it may appear in v62. The question is whether or not they will support the parallax feature of Spatial Video or will simply convert it to 3D video? Time will tell. Until Hugh's video, I was not even aware that parallax was a feature of Spatial Video. That one feature makes it a much better video format than basic 3D video.
Hugh Hou, And this is why I come here! You do the research, you understand the tech, and are willing to help us understand. Thank you! So about that Spatial Video... Does it work by cropping what you see at first in the window, and as you move closer or around the window, you see other parts of the image? That's how I originally thought it might be working when Apple first showed the tech off. Is that right? If not, magic is a fun answer 😁 I'm supposed to get my Apple Vision Pro in mid to late February. I can't wait! And I also can't wait to see how you tinker and show us how to get the most out of it! Thank you Hugh! I really appreciate it! vgv
You showed your own videos you converted (RGBD?) , but bone that were natively created on the AVP. So the question you didn't answer is if the AVP and / or iPhone 15 can natively shoot in this spatial video format. They both have depth sensors after all. Combine the SBS image with additional low res depth data and some computer vision magic could in theory, yield pretty good results.
I did not converted in RGBD but that is very similar tech like in looking glasses. Yes AVP and iPhone 15 native spatial video is even better in Spatial Video. But I can not have them in 3D to do side by side comparison. They are already spatial. Canon VR180 I can. As it start as 3D but then it can be converted in Spatial Video by simply provide the necessary metadata to AVP to do the calculation on the fly within the headset. I later find out using VR180 3D camera is better than just 3D camera b/c it is like having a reframe Insta360 camera but reframed in 3D as so much information I can play around.
@@hughhouOne other question. Traditional stereo video goes out of alignment when you tilt your head causing parallax issues? Is this still an issue in when playing back spatial video in the AVP. If not, that would indicate the AVP is building a depth map or some cloud point data on the fly during playback.
can't believe you with all your experience is falling for this. I obviously can't see 3D on a 2D video, but I don't see any parallax going on in the image itself, I can't see more to the side of geometry. It was only later that I realized you're speaking about the very cheap trick or only showing a part of the whole picture in frame and then allowing you to see more by moving your head... it's still just regular stereoscopic 3D. you need true holograms to capture light from all angles. Apple magic dust is really clouding your judgment, huh?
Well, 3D video is the foundation of Spatial Video. But I don't think it is fair to call it just 3D. It feel parallax in the render inside AVP, it has offset movement inside AVP which make it comfortable to watch and less motion sick (big problem on 3D in VR right now). Yes, this can be achieve in software level but I have never seen anyone doing it. Also, the 3D edges look very different. Don't know how to explain this, but you need to see it yourself. My point is Spatial Video is MORE than Just 3D - I did not say it is volumetric capture. AVP is doing some heavy calculation rendering to your eye to make spatial video look good and look different than 3D.
@@hughhouI hope you're right and it's actually some distortions to accommodate slight change in perspective. But certainly doesn't look like it here...
Could someone develop something for the Quest 3 that runs in the background so that it can be a simpler form of foveated rendering that locks to windows like when watching a video or using console remote play, thus increasing the resolution for what the user is most likely focused on at the moment then as the box goes away it spreads the resolution back out to the whole screen?? 🤔 in theory, from what he said, couldn't that be done on a Quest 3 since it doesn't have eye tracking capability?
Bro... I sat here for months saying spatial video better have parallax, better have some level of 6doF if they are calling it that. Then everyone says nah its just 3d. NOW I hear otherwise. That's tight!
Don‘t be fooled: it is just good old stereoscopic 3d video presented in a rather fresh way. There is zero parallax in the video content it’s just between the blurry frame and the video content plan behind it. That’s all. This could be implemented on any headset with any stereo 3d content there is. It‘s effective but not magical or even profound.
My guess is that Meta was going to release just a regular 3D movie player for spatial video, but then discovered this parallax effect on the APV and now are working hard on implementing it and that's why the update is taking longer to release. Fingers crossed
i think certain ameteur science fiction sector artists started recording their avp spatial video for multimedia consumption mainly targeting male audience if you know what i mean :))
Espatial video is traditional 3D, there is no discussion, you can make the 3D with more depth or make the images appear more outward, but it is still the traditional 3D and don't be fooled by changing the words
True spatial video cannot be recorded with just two lenses. What you see on the AVP is just 3D video inside a box that adds the illusion that perspective changes when you move your head. But when you look closely, you see it's not true.
@hughhou Is Spatial Video captured from the stereo cameras of the Vision Pro better than the Spatial Video captured from the iPhone 15's single view lenses and then converted to stereo? I would imagine that recording in stereo would allow you to see around near field objects to reveal what is actually behind them. Side note: All those 3D 180 cameras you were looking at during CES are now obsolete. Unless they have depth sensors, they can't really make good Spatial Video and once people see it, they won't want basic 3D video any more.
Hugh responded to gabrielegelfofx about this and I agree - a stereo image can be converted as it is played back (on the fly) since the depth can be calculated with the different views. Not sure about the spacial video file format, but if necessary it could be pre-calculated beforehand if the software for it is available.
@@gaba023 Stereo video without depth is just 3D video. It is not Spatial Video that allows you to move your head left and right and see what is behind objects in the foreground. That is not only a great feature, it also adds greatly to the realism and makes you feel like you are looking at reality rather than pre-recorded video.
@@BlueHound I agree. But depth can be calculated using two offset images or stereo pair. So if the software uses this, it can have depth as you describe.
@@gaba023 AI can also generate depth. Stereo could make it better. However, the best quality would come from lidar. Stereo adds the background behind foreground objects within the lateral distance of your eyes.
Are they using some AI with the spatial video conversion then? Maybe that can be done with the Quest 3 too, unless there's a hardware limitation? Linking it to a PC would remove that limitation.
Is it v62 on Quest 3 is out? We should test it and find out any hardware limitation on Q3 vs AVP on Spatial Video. Meta might adopt this format of Spatial Video as well.
Probably more like a software patent infringement problem for Meta. Cool software features define the user experience and are worthy of being protected as inventions.
You might be thinking of Apple's Immersive Video format. 180 degree, 360 8K playback they're capturing for some new immersive series just becoming available on AppleTV+. Sure it's a rebranding, but these are early days and Apple jumps in late before pushing the tech.
As far as I can tell it's a stereoscopic video shown on a plane behind the canvas window frame. So I guess it is occluding part of your recording to add a foreground element that gives some 6dof interaction. The perspective in the video itself is baked in and won't change with perspective. So "spatial video" is S3D content shown in a see through window player.
It is what is looks like but it is an entire different story when you see it inside AVP. The 3D in the frame also look different. If that make sense. I am not saying it actually capture more info than before, but how they move the frame is very different. Maybe some kind of low resolution depth map driving presentation - it looks natural for sure!
Some things I noticed (just now) when I 1) Converted an SBS video taken with QooCam EGO to spatial video using the Spatialify app, and 2) Viewed on Vision Pro in a window that I could walk around and view from different angles: While the video was playing the 3D effect looked natural as I moved away or closer to the window. It looked good from most angles. When I paused the video, the still frame still looked 3d, but as I moved around the 3d looked distorted as you would expect, as it only should look good from a perspective similar to how the camera saw it. I don't see anything jumping into place when I unpause, so maybe Apple isn't doing anything other than the window parallax thing. But what they're doing is very effective in videos, and doesn't seem to work as well in still-frame 3d images.
XR4 - has @hugh Hou said it right? he showed the "Focal Edition" - but that is not meaning foveated rendering.. the Focal Edition does more. its essentially a passthrough with autofocus the foveated rendering also the "cheap" 3990 XR-4 can do.
@@hughhou thanxs for clarification.. so in your comparisson: you should have copared the cheaper version with eh Vision pro not the 9900EUro Version.. the normal XR4- VErsion without the auto-focus-thingy - i think - will be here in Europe even cheaper than the Vision PRo, which will cost more than 4000 Euros i guess...
@@hughhou v62 is out (in Japan at least). However there is no Spatial Video folder in the files app like they said should exist- and no option to upload videos from Gallery in the Meta Quest App in my case. This could be because I am using an iPhone 12 to capture my 3D content, and convert it with spatialify however. Perhaps they limit the app to show this only if you have an iPhone 15 Pro, I don’t know.
Sounds like BS to me. You can't capture data from different angles unless you have a camera at different angles. If the headset is in fixed position it can't do that. Only google light fields can do that and they record from multiple positions. Plus I struggled to understand anything he said.
Sorry for my accent. Just to clarify, I did not say this is VolCap (light fields). The camera did not capture extra info as I stated in the video. It used the exact same stereo rig for fair comparison. It’s what the Vision Pro playback makes all the difference. I did not say it’s NOT 3D, but I acknowledge Apple effort to make 3D more enjoyable to watch with head tracking and motion parallax - so more than just 3D.
I want to really point out the fact that AVP spatial video look very different than other 3D video on any devices. Meta Quest might copy it in v62. Hard as no eye tracking but Apple set a new bar for 3D media. I do repeat said this is not 6Dof or volumetric
No it is not. I just want to be fair as some of the hate on my other videos or on Reddit kinda dismiss the fact that Spatial Video is more than just 3D. Both tehnically and actual feeling after watching it.
Who are you and what have you done to our Hugh? He would never leap from one gimmick to another without giving the cold truths. While you are calling out a "gimmicky 3D look" you are describing a "gimmicky spatial look". If there is now a process by which that stereoscopy can be refined and displayed more beautifully than before - great - but I think there should be clarity on the differences of captured versus computed depth data with no content creator hyperbole on either side. And our Hugh knows exactly why there is "the dreamlike look" - it's to hide the unfixable broken facts of life with 3D. This is not all gimmick though. There "can" be awesome here in some cases. There are 3 different levels of captured data that could be at play in a "Spatial Video" - 2 of them need computational tricks to be spatial (aka a 3D object in space) beyond "just 3D". A single camera lens and a depth sensor (lidar) is 2D+Depth and for many people that's already called "fake 3D" not full stereoscopy. The lidar gives you a measured indication of a point in space but the single camera means one eyeball's view must be faked. 2D+depth picture viewers like on Facebook let you look around an image in 3D because the depth data makes the captured scene essentially 3D object that can be pivoted. To use the modern jargon - Facebook has a spatial image viewer. Two camera lens and no depth sensors give true visual stereoscopy for the backs of our eyeballs but the location of points in space are left to computational guesswork. In traditional stereoscopy that computation is in the human brain. If you want a point cloud or 3D object to pivot around, a machine has to guess what points from each eye are at what depth. I have yet to see the Hugh spatial workflow but if a depth map is involved to turn the scene into a 3D object that you have some freedom of movement around - that's the point where it becomes a fake. Only with 2+ cameras and 1+ depth sensor do you get both the measured points in space for a 3D object and direct to eyeball image stereoscopy. The beauty of the Apple Vision Pro (and Quest 3) as a capture devices are the multiple cameras (at what IPD?) AND depth scanner. If and when those 3 streams of data are used to capture a scene as a 3D object with some degrees of viewer freedom - that's (almost) new and pretty cool.
Haha agree agree! Spatial Video is based on 3D. The whole presentation with AVP and how it "software" reproject 3D just has a very different vision experience than traditional 3D. Let me circle back on your points but right now I am focusing on actually VR180 on AVP :) Thank you for the indepth comment and yes, you guys keep me in check that is why I love my community.
Před 6 měsíci+1
VR 180 SBS 3D video is real life 3D just as i see in real world. No difference for me.
No, sorry, it's a simple relief video, in a frame that's in relief vision, so double relief, even if the second one isn't good, because you'd have to look at it from the front to see it properly. But it's a joke, especially with a little depth effect in the frame, to shift the relief video towards the background, with blur on the sides, because of course you have to fill the gap. The only parallax you'll see is when you close one eye and then the other, but it's better to do it from the front. Nice tweak, but it's all about communication and make-up, ingenious but totally feasible on a Quest 3. My first YT comment, after 25 years of 3D, but you have to set the record straight, it's pretty, it's a nice effect, but nothing revolutionary. Nothing to do with foveated rendering. Keep up the good work!
lol Calling a parallax box the video sits in as something special is insane and the problem with how Apple markets things. It's literally 3D video that moves within a box to give it a little more style. The video itself does not have any parallax. It's literally a 3D video.
Před 6 měsíci+1
I hate foveated rendering!! I need edge to edge sharp for my life :) Quest 2, doing this, some games doing this. A hate that!
You can stop it. See my 4K 30fps recording? Edge to Edge sharp! I think you will need third party app to lock it in without Xcode tho as I only know how to do it in developer level so far.
So Apple still lied, because even though it's not just 3D, it's still nothing like their ads and promotional material (along the lines of the lenticular front panel advertising).
@@hughhou The amount of parallax in the ads is absurd compared to the actual amount, as if it's a full 170+ degree shot. And in the ad they even showed that to be captured by the iPhone. It's truly nothing alike.
@@Philafxs I noticed that too with the girls and trees in the background. To me it looked like the original capture must have been panning, then when they created the virtual camera for the add, panned in sync with the capture. There is no way you can move 30 degrees to one side and see everything behind the subject. (of course the add is not showing the full true reality since it is not an in-headset view - the add camera is third person behind the guy with the headset on) To me it exaggerates what the device is capable of. But I don't think Hugh is meaning to say it is that good. Just that there is some parallax effect when you move your head that you don't see with ordinary 3D.
@@gaba023 The video on the launch event shows the same, with a girl on the beach filming musicians with her iPhone. It's fine to say spatial video isn't just 3D, but not that Apple didn't lie. Much like how much clearer the user's eyes are on the front panel than is the case in real life, which everyone mentioned. I've seen plenty of people think and speculate that the iPhone was able to shoot 3D180 given that clip, while us VR shooters already knew at least that to be bogus given the lenses. Not much has changed since.
@@Philafxs They had a choice - show only what the device literally can do, which would be boring and unimpressive on a 2D tv or CZcams screen, or show on those screens what it FEELS like when you watch spacial video. I think they had to show what it feels like to sell it. You and I know and understand 3D video, so we see the untruth in the add, but will average Joe buy it then demand a refund because it didn't match the add exactly? We'll see.
Před 6 měsíci+1
Every 3D movie, or any 3D content HAS PARALLAX. Thats how 3D works. To say that other 3D doesnt have parallax is nonsense that just shows you dont know what you are talking about.
I did not say other 3D movie does not have parallax. Just to clarify, The camera did not capture extra info as I stated in the video. It used the exact same stereo rig for fair comparison. It’s what the Vision Pro playback makes all the difference. I did not say it’s NOT 3D, but I acknowledge Apple effort to make 3D more enjoyable to watch with head tracking and motion parallax - so more than just 3D. The point it’s looking different. It knowledge viewers head movement and present info follow head and eye tracking. The end result it more comfortable 3D - not Just 3D. Something better.
So disappointing this spacial video. It doesn’t feel like Apple, it feels like a scam. I almost bought a pro 15 for that, I would have been furious when realising that by myself.
Wait till 16 it build in every phone in 16! 15 is a after tho to put Spatial Video in it. Later you will see the spatial video is a lot better directly on the AVP.
@@hughhou Tomorrow my Vision Pro is arriving. I doubt their "spacial" cheap trick will impress me but maybe. Thanks for your reply. I like your channel.
Yes it's an nice effect I have to admit. But Apple was faking dynamic parallax in its ads, it's extremely misleading so I still consider it a scam that deserve a class action. I thought an AI was completing the background already. We have to wait a year or two maybe.
Absolute clickbait title. It absolutely is 3-D, a two-image stereoscopic 3-D photo. Gimme an effin break. Apple knows exactly what they’re doing, and so do you.
This contains ZERO INFORMATION just a bunch of blah blah blah. WHAT IS THE DIFFERENCE? You never say! What is the DIFFERENCE? Spatial Video is just stereoscopic Video is all i got from this video... Settle down and give some info!
Meh I wouldn’t call it 3d more I call it crop and zoom with auto scrolling like they do with iPhones wallpaper soo yah it’s nothing new and yes it is a lie even if it small because a lie is well, just a lie..
What make it 3d is if the objects is scan in to cgi 3d space and you see that object in environment that you can move around rather a scrolling image then that’s 3d but if it’s just flat image cropped and it just move around give illusion of 3d then it’s not 3d just a illusion of it.. that’s the truth.. but hung you even agreed that what it is I mean that what it is just flat image crop that scroll around. Meaning not 3d
@@hughhou oh by title I thought you was saying it is because that’s what Apple was claiming back in last year keynote that’s why people is saying that which confuse lot of people lol
LOL. I never said I know what his actual memories are Kevin. I said that the Apple Vision Pro is not recording like "your actual memories." How do I know that Kevin? Because I am a Clinical Psychologist who specializes in memory and has for over 2 decades. I know how memory works because of SCIENCE Kevin. And if you had taken even a high school psychology course you would know that our memories - yours, mine, his - are NOT visual "movies", they are scripts. Nor is our actual vision anything like what you're seeing on screen - it's actually very spotty and our brain fills in a lot of holes. Does this answer your question? @@kevinbailey8827
Thank you for your professional explanation. I agree that term sound a little hype. I am and lots of ppl here are visual people (creators) - so vision is key for remind ourselves in the past - hint Memory. Vision Pro or even Quest 3 help that. Just step in the direction but not total replacement. Just trying to strike a fair argument. Thanks for jumping in 🙏🏼 We need all info we can get.
Sorry if I came off a little harsh. I have been both incredibly disappointed and incredibly fascinated by the launch of the Apple Vision Pro. It is a great device and I wholeheartedly welcome Apple into the mix. In my opinion, they did the equivalent of releasing the Concept Car to the public (for those who don't know what I mean, at car shows, manufacturers will often have a vehicle showing off the "tech of the future." It isn't released to the public, and only some of the features ever do make it). I don't think we will ever see anything like the AVP in future generations, but some of this will carry on. My disssapointment and facination has been on how it has been covered. I see so many influencers who know about VR and have reviewed other VR products seem like Virtual Reality is a completely new thing to them. I can't count how many have said "WOW. Now I can watch a movie from the comfort of my own living room and be immersed in a theater!" I even saw one yesterday boasting that with the Vision Pro you can now have piano lessons with your real piano and virtual augments. Neither of these is at all new. Hell, I was watching movies in an immersive theater with the Oculus Go. And have an app to do the piano lessons on my Quest 3. It's so ridiculous that I have honestly questioned how much the reviewers who I have always trusted to give fair and unbiased reviews are being paid by Apple. At the same time, I am facinated by the prospect that they are not being paid by Apple, but are genuinely oblivious that they have lost objectivity and joined in a cult of sorts. It's to a point where I can't trust any review that doesn't point out drawbacks/negatives of the AVP. Not negative reviews, but some sign that they are not just either part of the "If you say one word criticizing, it's just because you're poor" cult or a paid promotor. @@hughhou
Your video also struck a very unrelated nerve for me. In my work with treating trauma, I encounter false memories daily. We all have them. There is a phenomenon known as the Mandella Effect, where small groups of people will all believe that they remember something that never happened. What is amazing is that these people can believe that false memories exist, but THEIR memory couldn't possibly be false. They clearly "SEE" this thing in their memory, like a video recorded in their brain. For example, many people believe that the actor Sinbad made a movie called Shazam!! in the 90s. Never happened. It's likely that these people DID see something with Sinbad around that time and have a memory for that. But it's not a video recording. It's a script. And when we read the script we visualize it. That script is easy to change, resulting in the video produced by that script changing. It's a lot harder to understand that when you believe that your memory for events are actual videos recorded somewhere in your brain. @@hughhou
When you are communicating, take the damn thing off your face. My best and only real 3D camcorders are the JVC ones. Real parallax adjustment manually or automatically. You can watch the parallax change when you point the camera at different subjects. Nothing on the market that is affordable condo any of that. and it has a 3D viewfinder that shows 3D, left side, right side left and right sides, plus it has zoom lenses and stabilization. The camera lenses were spaced properly, unlike all the options available today. Since the Apple Video Pro came out the prices on these cameras have doubled and tripled. So a ten year old camera is better than anything on the market today and it was CHEAPER. Your take on spatial video is an illusion from software - a gimmick. The 3D from close lenses is gold for close up work.
Thank you for the feedback and love to hear form an expert. I also has the Panasonic 3D rig back in my boradcast day shooting sports in 3D. I know exactly what you mean. That is 1080P 3D tho not even 4K. Softwre is NOT a gimmick tho. Also it is software and hardware inside the AVP make the magic happen. Any 3D camera can be Spatial Video Camera - that is my point and you just prove that. But no display can show Spatial Video in its true potential - only AVP. If you watch it on Vision Pro - it is more than 3D. Yes software illusion, but is it illusion tho if it is different and feel better by many non-professional consumer feedback. They all said - this is the 3D I can watch... its comfortable (I show this to all my firends and clients btw, who are not 3D expert)
@@videos3d313 The smaller the distance the better the 3D. People making the cameras now have no clue except fo Canon which teased a close lensed version of their 3D lens for the R5, but they haven't released it. Closer lenses means less cross eyed fatigue and better parallax.
@@hughhou I prefer hardware than actually does the job rather than software to conceal a poorly functioning system. I'm not an expert, but I am above average intelligence and still think. Have the CALF and the EGO and I wish those folks understood 3D. Putting three lenses that one could switch between would be great. You could get three levels of distance from one camera that way.
@@rupertchappelle5303 lol, i made 3d videos since 2008. There is no single perfect base. All depend on subject distance. 6.5cm is what human eye distance is, so best for 3-20m scenes. For closer subject, smaller distance needed, 2-3-4cm like my Sony TD30. For far away mountains i often use 30-50-100-300 (!) cm distance!
Although I have no hands-on of the AVP, but I can see it's a parallax effect post-proseed on by the AVP spatial video player, between the player window border and the content video. Many others, including myself, were expecting a parallax between the close-by object and the background of the video itself. This did not happen since in core the technology is stereo vision 3D video.
I'm not sure that I like the dreamy blurry border with its parallax artifact . . . or at least I would like to have the option to be able to turn it off. But then Apple would have to admit that "spatial video" is just normal 3D video.
You are correct Kris. I believe Hugh is partially wrong that it's not 3D video. I'm a long time video encoding researcher and have looked any many factors over the years. With spatial video it has to do with how the AVP player and Meta Quest spatial player hosts the video. Apple takes the 2160 4K-3D video and encodes them to 1440p SBS video for playback. Then crops 10% into the video that is covered up by the soft border. Then an extra 5% of the video is taken to the diffused edge of the player border. This gives viewers the parallax option for shifting in any direction and noticing the video revealing itself. Do you remember when Apple did this same display option for the lock screen, all they did is crop into the image with the viewer to make you feel like to were controlling the view of that image. When I look at spatial video in other players on a Quest 3, I notice the edges of the video, then play them back in the Meta spatial player I can see the 15% of the video that has been cropped in. I think sometime in the near future someone will figure out the metadata that tells the media player to playback the video in those setting.
That’s a good point, and true about the Lock Screen. The GitHub project SpatialMediaKit may be a good starting point for forensics even though itself doesn’t export metadata@@brianparker3194
It just 3D video shown cropped so they can make fake parallax by moving the view port. The files are standard MV-HEVC.
It is 3D video! They are using software and the cutout window to move the video around (rotating it, zooming in and out etc) in space giving an illusion of looking through a window. Most 3D video out there is static attached to your face. If you move your head the video moves with it. However, you see the full video so distortions would be obvious if they were to zoom in or rotate the video potentially making you dizzy. You can achieve the same effect with a 3D video in any game engine or 3D Video player that allows you to move the window, zoom or crop. They are just doing it automatically.
True, but this just points out the magic is often in software. The average user of this technology is looking at you as a tech nerd when you say "But it is just 3D video!!!". They aren't going to care about the details, they weigh the user experience, which is always a composite of hardware and software... There is a lot of software magic in the AVP, taking that impressive hardware spec and doing things with it that are original from a user experience stand point. Ultimately the user experience is what matters, not the technical details developers or enthusiast might pull out to try and diminish the novelty or claim some equality that really isn't there... Techies always under value software contribution to the experience.
@rmzzz76 super well said! We need to put on the hat of techie and normal consumer to exam the tech in different perspectives.
@@hughhou normal consumer can't tell salespeech from truth. A techie can.
What do you think...the Quest update 62 can let you play those spatial video's. Apart from the resolution, would they look like in the Vision Pro? Or just like ordinary 3D? I haven't found any spatial movies yet. Do you have a source of examples, maybe your own test results? What software do I need to play it in my Quest3?
@@SjaakSchulteisfrom their promotional materials it looks like it might be at least trying to mimic the way AVP does it.
So the claim is that the spatial video is just 3d video inside a shadowbox, and not actually the slightly volumetric video we were shown in advertisements. It's hard to tell from your example if the actual video has more than just stereoscopic depth. One test would be whether or not you can tilt your head 90 degrees while watching and still see depth. The shadowbox effect looks pretty cool, but I see the complaint. True spatial video is something that exists on Leia's Lumepad. The limitation is the screen viewing angles and the quality of AI-driven depthmaps.
Overhyped 3d, lol they should have called it 3d+ or 3d pro to avoid confusion 😂
Yes that will be better as more than 3D. So I don’t get hate by my community just trying to make a point of 3D+ - well Apple need their brand unique
I'm not well versed in video tech but at the end of the day it's just MV-HEVC, which was finalized in 2014? So this is good old stereoscopic, nothing new. Just "retina"™ 3d video.
@@SToad Yes, that's right. It's so *YAWN* old news that, get this, literally no other headset natively supported it! And there was no way for consumers to actually shoot videos in that format without committing to spending a bunch of money on dedicated camera hardware and paid software, and committing to the time spent in the workflow needed to create MV-HEVC videos. And what did the entire VR industry do with this "good old nothing new" format since 2014? Absolute fucking jack shit. Apple came along and in one fell swoop made it easily accessible to anyone with an iPhone, and suddenly you care.
This youtuber is wrong. The spatial videos are just regular 3d videos placed behind a hazy window. Moving the video left and right is NOT parralax and not at all the insane ammount of parallax present is the middle of the false advertisement videos that showed you could rotate your perpective to see completely behind people.
I would just be thinking these small additions are nice ideas and cool if Apple didn't once again invent a new term and claim they invented something that already existed, this time since the 19th century. It's sad that most tech youtubers don't know enough about what they're reviewing.
But this video showed the 4k video feed from the headset, I saw no one showing that, and that's really cool.
I just got back from my Vision pro demo. While the display is super sharp and the spatial video looked awesome I can tell you there is no parallax. Just visual trickery. The two cameras can not record the info behind the subject or outside the cameras fov. The only way to achieve that would be to deploy ai predictions or to remove the subjects from the image to record the environment in 3d. The tracking was good though. Can’t say it would be a decent purchase even if I had the money to blow though. I kinda got bored during the short demo and my face was already hurting from the weight.
Thank you for jumping in and provide feedback. It is not "seeing" behind the object as stereo rig can not capture that. We need volcap rig. What I am saying is there is Parallax with maybe software level or player level + AVP internal magic to create motion estimate parallax. The 3D video is not stuck to my face. You will need to sideload a actual 3D video (what we used to on Quest 3) and put it side by side with the Spatial Video to really see the visual different. I have tha ah-ha moment when I compare 3D Video to Spatial Video side by side inside the AVP. This require you, of course, to own the AVP, xcode your way into the device and load up regular 3D video instead of Spatial Video. But seeing it side by side really make me think it is way more than just 3D video playback.
Hi Hugh, is it possible for you to share a spatial video captured with the Vision Pro in 3D here on CZcams because I'm interested to see the difference in depth and quality compared to the iPhone 15 Pro Max's spatial video (I assume the Vision Pro has a wider stereo base, but I'm not sure about the video quality).
Also I was wondering if the Vision Pro has the same format for capturing spatial videos limited to 1080p/30FPS like the iPhone or does it support better formats like 60FPS?
Vision Pro capture 2200 x 2200 - I will provide good test samples in my next video. Still organizing everything.
@@hughhou That would be great. Wow, 2200 x 2200 seems to be a lot! Is it also higher framerate or still 30FPS?
Thank you ! This is the question I wanted to ask too. Looking forward to the update video
still a 3d video file, with a "spatial viewer" which is a 3d video player with viewpoint transformation feature.
Love your videos Hugh, but in this case am I missing something or is it just depth map from stereo giving that Facebook like parallax?
If it is... It's underwhelming and not 'real' parallax (missing information /pixels) being created.
Reminds me of the early days of "rubberstamping" 2d to 3d conversions as done on films like clash of the titans.
Can't tell much from your video above, but would be good to know if this is the case.
It is not REAL parallax - as the camera is still Canon R5C with the Dual Fisheye lens - so NO volcap. But it is real 3D. And Vision Pro make it look better on top of the 3D data and create depth parallax off axis movement based on your IMU data of the Vision Pro. So the 3D look a lot more comfortable instead of 3D stuck on people face and not react to head movement if that make sense
It can't 'create new pixels and visual information' from what is not already recorded stereoscopically (by the interaxial of the Canon 3d lens for instance) so what it's doing is, creating a depth map from the disparity (depth from stereo) and that depth map 'stretches' edge pixels to give the 'illusion' of parallax - driven by the IMU. This is my understanding (as I've not delved into the vision pro, nor will I buy one).
At best this is pseudo-parallax not true "spatial video"
Great content as always! One correction: PSVR2 already has foveated rendering (with eye tracking) and it’s a cheap device. Still having 7 VR headsets at home I can’t wait to try the Vision Pro once it’s introduced in Europe. :)
Good point! I always forgot about PSVR2 lol! Is it hardware level foveated rendering - or software level support only?
Quest Pro supports foveated rendering but I think it's software. @hughhou covered this in a video about Red Matter.
@@hughhouThe PSVR is easy to forget since it's just for gaming and you really don't hear about it that often
Thank you! Almost every glowingly positive review of the AVP I see does this - states that this "new feature" is something NEW to VR when it's been around for years. There is almost nothing new that I can see in the Vision Pro. That isn't to say that the Vision Pro doesn't do it a bit better (as in the case of the Spatial Videos perhaps).
People are confused by the differences between "spatial video" and various 3D formats taken by different devices. It would be very helpful if you could prepare some sample video files for us to comparison-view in our Quest 3. We need similar shots taken by the AVP, the iPhone 15 Pro, standalone 3D cameras like the Qoocam EGO, the Lume Pad 2, and the Quest 3 method.
Good point! I will def do that!
I've been taking a lot of VR180 pictures with my Lenovo Mirage VR camera. I would love to transform them into the new Apple Spatial Pictures. Any idea how?
You will need to make them into Spatial Photo first, let me figure the step by step out and make a tutorial
Can we upload spatial videos made using the Vision Pro to CZcams? How can we watch them?
Yes that is my next video you read my mind! Coming coming. This is week 1 lol. So many tutorials are in the making.
Hugh helped with my old school uploads of SBS and OU which play back fine on CZcams. We need full spatial streaming soon!
youtube.com/@spatialtests?feature=shared
Nice! HLS streaming is coming so yes highly possible
So is the spatial video format actually storing extra stuff, or does it just put the viewer in a different mode that has some tricks added at the edge of the view box?
From the metadata I can inject into my own 3D video, yes lots more info. But not something you shoot but something the AVP calcuated the data you provide and present that inside AVP. Let me go more indepth in the next tutorial. I did lots of testing but my brain is all jumbo now.
It's the video player that's doing some extra tricks, not the file / recording itself.
Love your videos, and I always knew you would bring the very best content about AVP.
Glad you think so!
Thanks for this video Hugh! So many Apple haters, even ones who don’t even have the device, trying to downplay the importance of this device.
Thank you and indeed!
Great video as always! Basic question but how do you play traditional SBS (not VR180)? Does it require some metadata to show in stereo vs SBS 2D?
Yes! I am working on a tutorial will be release tomorrow to explain exactly that!
Would you be able to test what the max resolution Vision Pro is able to display? I have several 10K VR180 photos from a GH5 rig which look good on Vision Pro, but was wondering if it would be worth upgrading to a dual Fujifilm GFX 100s rig or similar cameras for 16K or 24K VR180 photos. Also is 8K the limit for video playback? I tried playing the 12K FM DUO sample video you linked in one of your previous videos, but I couldn't get it to play on the Vision Pro.
Yes that’s coming. New video show 8K up to 450Mbps in 60fps! But 8K x 8K crash Vision Pro. See my new video just dropped
I’m thrilled to see you’ve worked with ESPN+, I’m hoping you’re working with them on UFC because I’d love to watch MMA in VR180. I think the relatively small octagon would be easy to capture and the experience would sell the technology.
UFC on Meta is pretty dope. Have you seen it? I was not involved with the UFC, but my firends shot it in the ring on last coupled big fights. It is on Horizon Worlds in Meta btw.
Another amazing video!
Glad you enjoyed it!
It’s spatial video capture but 2000 x 2000 (square). My family videos are wide screen. Deal breaker?
Spatial Video on AVP is 2200 x 2200 - not a deal breaker. You can use your iPhone instead.
Amazing video Hugh, we are slowly getting there! I am not sure btw if VR180 can also be spatial on the Vision Pro btw. And what would be a small camera you recommend to capture spatial video (in a frame or 180degree)? I feel the Canon is too big and will scare the people I plan on interviewing
From my experience, no or not yet. VR180 will lose all parallax illustion including Apple own production on Apple TV. But we don't know what kinda of crazy thing Apple will pull when launch in Europe and other country.
It would be really great to have a VR180 with some parallax, the ultimate immersive experience@@hughhou Personally what do you think is more appealing at the moment, VR180 or spatial video?
Now we just need the spatial and panaromic bits to be merged so that we can move around freely around the moments we captured.
That is Volume tho and it is technically hard or impossible with current build.
Releasing such a feature, it means they are already working on the whole thing. I know Apple’s DNA, nothing is impossible 😅
Can you tell me why resolution is low when I stream the Xbox app? Everything is crisp until I get into the gameplay and the it degrades. 😢
Thats prob Microsoft problem?
Hugh, There is parallax in the content. You are just being fooled by the 3d window that is holding the content.
I've been recording a lot of spatial videos, and have been really disappointed in the lack of any paralax.
Open them up in the "immersive mode" and you will see what I mean.
I feel the same, there is no parallax yet. I know that Apple is encoding the lidar data in the videos, but not yet showing the parallax. I'm confident it'll come at a later stage
Thank you for making this video, the other one in reference is SO ill informed it's ridiculous.
Right now all I want is to be able to play my insta360 x3 on my AVP.
Haha you read my mind. That is coming next tomorrow: czcams.com/video/fvUtaRTqWLw/video.html - join the live at 8:30am!
So to be clear, "spatial video" is some additional metadata that apple recognized, and adds some blurring around the 3d video/photo?
Not just edge blurring - but also head tracking and hardware level parallax. And yes. The standard called MV-HEVC. Not owned by Apple BTW. They are just the 1st to adopte the format and making somthing amazing and less motion sickness.
AVP 'spatial video' is standard stereoscopic VR180 (or other wide field of view) viewed through a narrower rectangular window. The ability to 'look around' is essentially shifting your gaze (moving the center of the VR180 image) to the side. It does provide some limited illusion of the ability to look around objects, but it is still essentially just stereoscopic video. You could produce a similar illusion by putting a standard 3D video behind a smaller window and shifting the projection to correspond with change in viewing angle. This form of spatial video takes immersive 3D VR180 and throws away both the immersive experience and a lot of the image. The advantage is that you can have multiple images open, it's a trade off between immersion and multi-tasking.
Wait you said "VR180" - so Spatial Video now work with 3D 180 video format now? Last time I check AVP does not support any immersive format. The metadata is there but the player does not do anything. If VR180 work natively - this can be huge. Spatial or NOT, this will be amazing!
@@hughhou I think his example of VR180 as a recording source is simply a bit extreme, as it's far more image than what's required for this sort of "perspective shift". It's more of a 3D video with a smaller "cropped window" into the image than the size of the 3D image, creating an illusion that you're looking at a scene with shifting perspective. But really, you're just moving around the 3D video relative to your 6dof movement. It's still just a fancier way to view 3D. Apple showed videos of people experience the layers of depth within the 3D image shifting as you move in space, which is not what actually happens, so it seems. And this is why people are saying it's a lie. the trees in the background don't shift to the left behind the people in the forground as you shift your view to the right. That's what they implied it does. From what I'm seeing here, it does not. this clip shows the implied perspective shift, though it was just the video shifting itself, which happened to be along with the onlooker's perspective: czcams.com/users/clipUgkxMUk7rEH4OcKSCXxVkCiw2iRnABhsc5IJ?si=VtBnbHIksZLMxwJM
That's really cool that moving your head while viewing the video gives a parallax effect. It's these small touches that put Apple above the competition. There's some very clever algorithms at play. I can only imagine it's using the depth information (or calculated depth information in the case of converted 3D video) to build some kind of layered, textured model in the background. Very clever.
indeed!!
I'm not sure if there is parallax. It looks more like a crop and pan which gives the illusion of seeing more. I could be wrong though since I haven't tried it myself.
I think you're right. It's a parallax effect only with the fuzzy frame around the video.
But when I experience it, the illusion is very effective. The experience of viewing spatial video on Vision Pro is noticeably better than viewing the same spatial video converted to SBS using the Spatialify app and viewed on the Quest 3.
I haven't found good ways to view my own 3D content (such as from the QooCam EGO) on the Vision Pro. There are a couple of apps on the App Store, but they don't work yet. Their developers are hard at work and release updates often, but they aren't there yet.
Yes! It def look like that way. The end result matter. Everyone see it said it’s better than 3D and it’s better - more comfortable. With eye tracking engaged. It’s more than 3D but def not Volume.
Gen 2 of AVP is going to be 🔥
It is called Apple Vision AIR? It is leaked today.
@@hughhou 👀
You rock, but this video is misleading. There’s no parallax with Spatial Video except for the frame. It’s 3D video in a cool wrapper. What app are you using to view 3D video on AVP?
Thank you! Just the native app. It can be true the video itself does not have parallax, which I can not validate. But it looks different than 3D - I think the head movement tracking, edges effects, parallax illusion all together make a a more enjoyable viewing experience than just 3D stick to my face. I think that is the point. It is still 3D but it is MORE THAN 3D, not JUST 3D. Have you get the chance to watch the video youself?
@@hughhouThanks for the reply! I wasn’t aware that any native app in Vision Pro had the ability to view stereo video from a camera other than Vision Pro or iPhone 15 Pro. Have you made these files available for download? I’d love to try them on Vision Pro.
@@prolostI'm also looking for the app available for AVP that can play back SBS, OU, 180 & 360 videos
@@hughhou I tend to agree with you. The high quality of Spatial Video straight off camera without the complicated and laborious workflow is the big deal in my mind. I prefer the Spatial Video off the iPhone 15 Pro. The depth does not seem so exaggerated as with the Vision Pro and it is in a 16:9 aspect ratio. While not true parallax, the overall experience is better than 3D content I have seen.
It will be interesting how Apple innovates here with some AI-driven (potentially Gen AI) augmentation to provide some degree of parallax and some synthetic volume. I think this would be a great area of innovation for Apple that would be highly differentiating. I hope Apple does this because they did mislead everyone with their video that showed full volumetric parallax.
PSVR2 and quest pro also have eye trackers
Yes indeed! Quest Pro is not hardware level implementation. PSVR2 only have game engine render not live action solution. This leave almost on one has this feature. We hope PSVR2 change this, but so far not even CZcams app work correctly on PSVR2 - PSVR1 work great...
i dont think it really look like any parallax is happening it is just moving subtly
Eye-tracking trick and the whole presentation inside Vision Pro is tricking my brain lol.
You think Strip Clubs will let me in with one of those things on?
Lol. Why not?
Thanks, I learned so much from this episode!
Important question for you: I've been shooting on the Canon VR Lens and preparing 180º 3D content for viewing on the Apple Vision Pro. To play these video files in a 180º FOV on the Meta Quest, I simply press the button on the bottom right to change viewing mode, but how will I do this on the vision pro? Is it possible?
I noticed that you played some of your videos shot on the Canon VR Lens during this video, but they were viewed within a regular window as spatial video. I'd like to view my content in its full FOV so that I can look all around.
No possible yet. You need to write your own player with Swift. I am working hard on it tho and should have solution very soon
Thanks so much for your quick response, though this news makes me sad to hear.
I assumed the ability to play 180º VR video would be available within the Vision Pro. I am very excited about your solution, however I fear that its something that won't be easy for non-technical people like us to do. For example, the videos I'm filming will need to be viewed in 180º 3D VR by clients using Vision Pro, but I won't be able to install software or do anything on behalf of my clients using their own Vision Pro. Any thoughts?
Also I wanted to thank you in advance for working so hard to solve this problem and many other. You're by far the most knowledgeable in the space. It'd be beneficial for all of us if Apple developers hired you as a consultant!@@hughhou
@@noel-issa I'm in the exact same boat. I was wrong to assume spatial video was basically vr180 video. windowed spatial video doesn't sound that great.
The only real difference between this and normal 3D is just the player. It does some tricks to do that parallax and blurring in the edges. You should be able to say, build a video player that does the same thing for normal SBS videos, etc.
Yes or no. The player use active eye tracking, head tracking and do live calcuation to position the inner content to create an illusion of a parallax. Not headset has this level of accuracy on eye tracking (or at all as Quest 3 has no eye tracking). So software can only do so much - you need tracking data.
You should pin a comment saying that the files are in fact just 3D video in a MV-HEVC container. The parallax effect is 100% just done by moving a viewport around the cropped video.
The Quest Pro also has dynamic foveated rendering (using eye tracking) and several games take advantage of that, most notably Red Matter 2. I believe it does it mostly on the software side, but still. Also, like others have said, that's not really parallax that you're seeing, IMO. You'll notice that the edges of the Spatial Videos are not only blurred but also closer in than the edges of the straight up 3D videos. It's like you've got a blurry picture frame that's blocking the outer edges of the 3D image/video. When you move your head, you can see some of what's being blocked by the frame, which enhances the 3D effect and gives the impression of parallax.
That said, it's still a cool effect and very smart of Apple.
Thank you! Yes I mean a cool effect and presentation of 3D - so more than 3D instead of just 3D. They solve a lots of viewing challenge of traditional 3D in VR. I did not said other can not achieve it in software and some sensor data - but they just did what others has not, do desire credit on Apple part.
When on a smaller budget, would the limited resolution footage of the Kandao Qoocam Ego still look ok on the headset?
I will answer that with real footage in my next video
Looking forward to that! Great to see your in depth videos and experiments. Thanks for all the inspiring videos! @@hughhou
4:10 "this feature alone worth $3000", why? Playstation VR 2 also has foveated rendering and the cost is $550
True!
We need more spatial video playback example videos, the demo birthday video in the stores is decent but need more creative content
Oh trust me it is coming on this channel. I am shooting an entire feature film in Italy now and it will be out soon as Spatial Video example that professionally done :)
@@hughhou wow, I can only hope you also upload to Google drive for a higher quality version! Great videos btw
If the z-depth isn't capture by the camera then we don't have any spatial video. The video cameras need a Lidar scanner to create z-depth + video. I don't know if the new 360° cameras have this feature. Also the gaussian splatting technology is the future for 3d imaging.
I don't think you need z-depth for spatial video. Even AVP, the Lidar is not engaged. Calculating the depth with a very low-resolution depth map can be easily achieved with stereo side-by-side video / data. I think that is what Vision Pro does on the fly. I don't know exactly, but it looks different. It is not 3D.
Is there a difference between vision pro spatial video and quest 3 Passthrough recording? If so what's the difference?
There is more to the way spatial video is captured. To keep it non-technical, it's compressed into a 2D video with depth information attached, allowing it to be experienced either in normal 3D or a bit more volumetric with some parallax.
Also, the camera quality is much better than that of the Quest 3. That said, I personally found only the passthrough to be a big step up. If Meta wanted to, they could release a Quest 3 Pro at double or triple the price with only better cameras, and optionally eye-tracking, to close the gap.
I think he's manually converting SBS 3D video to RGBD video which contains a single 2D video next to a depth map video. It's unclear whether the AVP or IPhone 15 can actually shoot in that format.
The camera is the major difference.
this would make perfect sense since Apple had 'portrait mode' on iphones (TDoF sensor) for quite a long time now. I have been using these for creating quick holograms in Lookingglass displays@@hughhou . When watching those (without glasses!) It feels like you can 'look around' the object. This sounds similar. Depth map attached to video and computational resolve. Can't see a reason why this couldn't work on my Quest2 if Apple was to release the code to developers. PS, looks like Polycam might have a foot in the door with 3D models (NeRF, Gaussian splatting)
Wow, thanks for sharing this! You should also test this on Quest 3 after updating to v62. Try the same raw Canon file, plus also the converted Apple Spatial Video. That will be really interesting to see if Apple is doing some “special sauce” with their files.
Is v62 out yet? I have been waiting impatiently lol. I don't have special access like other big Quest influencer so I need to wait for beta firmware available to everyone to test :(
@@hughhou I don’t think so… I’m impatiently waiting too!
Can one use Skybox VR video player to view spatial videos on the AVP?
No but I have other solution coming out tomorrow
Could actual Spatial Video be implemented on the Meta Quest 3, do you think? The lack of parallax is quite noticeable in 3D video.
v62 will answer this question very very soon.
With iPhone you can capture 4K 30fps Videos with the App spatialcamera and Upload and view on Quest 3
@@just4funvr146 Meta is working on Spatial Video support in the Quest3 and it may appear in v62. The question is whether or not they will support the parallax feature of Spatial Video or will simply convert it to 3D video? Time will tell. Until Hugh's video, I was not even aware that parallax was a feature of Spatial Video. That one feature makes it a much better video format than basic 3D video.
VR 180 4k is much more immersive compared with 3d or 3d parallax? Why dont we make more VR 180 4k?
We just drop 8K VR 180 on Vision Pro now totally possible- check my new video
I think there's a reason why they used the word "spatial" and not 3D. As long as there's depth, then it is Spatial
Good point
Hugh, do you have any converted Kandao EGO footage for download and review in AVP?
I am working on it.
My Playstation 5 VR2 has eye tracking and has foviated rendering and HDR and headset rumble but it doesn't cost $3500
I always forgot PSVR2! Very good point and thank you for pointing that out
It's not portable and doesn't support 3D ...
Hugh Hou,
And this is why I come here! You do the research, you understand the tech, and are willing to help us understand. Thank you!
So about that Spatial Video... Does it work by cropping what you see at first in the window, and as you move closer or around the window, you see other parts of the image? That's how I originally thought it might be working when Apple first showed the tech off. Is that right? If not, magic is a fun answer 😁
I'm supposed to get my Apple Vision Pro in mid to late February. I can't wait! And I also can't wait to see how you tinker and show us how to get the most out of it!
Thank you Hugh! I really appreciate it!
vgv
Yes kinda! But more complicated than that. Head and eye tracking to in place for motion parallax - its not true volume but step forward in the game.
Thanks Hugh! I'm so looking forward to getting my Vision Pro. In the next couple of weeks, fingers crossed :)
vgv
You showed your own videos you converted (RGBD?) , but bone that were natively created on the AVP. So the question you didn't answer is if the AVP and / or iPhone 15 can natively shoot in this spatial video format. They both have depth sensors after all. Combine the SBS image with additional low res depth data and some computer vision magic could in theory, yield pretty good results.
I did not converted in RGBD but that is very similar tech like in looking glasses. Yes AVP and iPhone 15 native spatial video is even better in Spatial Video. But I can not have them in 3D to do side by side comparison. They are already spatial. Canon VR180 I can. As it start as 3D but then it can be converted in Spatial Video by simply provide the necessary metadata to AVP to do the calculation on the fly within the headset. I later find out using VR180 3D camera is better than just 3D camera b/c it is like having a reframe Insta360 camera but reframed in 3D as so much information I can play around.
Interesting. Thanks for the detailed response. Can't wait for the follow up video. Good work!
@@hughhouOne other question. Traditional stereo video goes out of alignment when you tilt your head causing parallax issues? Is this still an issue in when playing back spatial video in the AVP. If not, that would indicate the AVP is building a depth map or some cloud point data on the fly during playback.
can't believe you with all your experience is falling for this. I obviously can't see 3D on a 2D video, but I don't see any parallax going on in the image itself, I can't see more to the side of geometry. It was only later that I realized you're speaking about the very cheap trick or only showing a part of the whole picture in frame and then allowing you to see more by moving your head... it's still just regular stereoscopic 3D. you need true holograms to capture light from all angles.
Apple magic dust is really clouding your judgment, huh?
Well, 3D video is the foundation of Spatial Video. But I don't think it is fair to call it just 3D. It feel parallax in the render inside AVP, it has offset movement inside AVP which make it comfortable to watch and less motion sick (big problem on 3D in VR right now). Yes, this can be achieve in software level but I have never seen anyone doing it. Also, the 3D edges look very different. Don't know how to explain this, but you need to see it yourself. My point is Spatial Video is MORE than Just 3D - I did not say it is volumetric capture. AVP is doing some heavy calculation rendering to your eye to make spatial video look good and look different than 3D.
@@hughhouI hope you're right and it's actually some distortions to accommodate slight change in perspective. But certainly doesn't look like it here...
@@VRnamek He's sharing his experience, you are looking at tech specs which you have no access to. 🤣
Could someone develop something for the Quest 3 that runs in the background so that it can be a simpler form of foveated rendering that locks to windows like when watching a video or using console remote play, thus increasing the resolution for what the user is most likely focused on at the moment then as the box goes away it spreads the resolution back out to the whole screen?? 🤔 in theory, from what he said, couldn't that be done on a Quest 3 since it doesn't have eye tracking capability?
It’s coming officially on v62 update slowly rolling out to everyone! On Quest 3
@hughhou wow really!! Awesomeness 👌 👏🏾 👍🏽
Bro... I sat here for months saying spatial video better have parallax, better have some level of 6doF if they are calling it that. Then everyone says nah its just 3d. NOW I hear otherwise. That's tight!
Don‘t be fooled: it is just good old stereoscopic 3d video presented in a rather fresh way. There is zero parallax in the video content it’s just between the blurry frame and the video content plan behind it. That’s all.
This could be implemented on any headset with any stereo 3d content there is. It‘s effective but not magical or even profound.
My guess is that Meta was going to release just a regular 3D movie player for spatial video, but then discovered this parallax effect on the APV and now are working hard on implementing it and that's why the update is taking longer to release. Fingers crossed
Yes! Is v62 out? Look like the demo has the same effects as AVP! I am so excited to try that!!
It's out but I haven't tried the demo video yet. I did get the update a few days ago but haven't had much time. I'll try it today.
@@hughhou
LIKE YOUR CES 3D VIDEO HUGH!😊
Thanks! 😃
I am a developer and have Xcode. So How do I add the code to achieve spatial video as you described?
Wow good information
Thanks
i think certain ameteur science fiction sector artists started recording their avp spatial video for multimedia consumption mainly targeting male audience if you know what i mean :))
Lol science fiction ...
Awesome
Thank you!
Espatial video is traditional 3D, there is no discussion, you can make the 3D with more depth or make the images appear more outward, but it is still the traditional 3D and don't be fooled by changing the words
True spatial video cannot be recorded with just two lenses. What you see on the AVP is just 3D video inside a box that adds the illusion that perspective changes when you move your head. But when you look closely, you see it's not true.
So can we play our 180 Canon VR videos on the Vision Pro? I have one and I really want to get the full 180 to play on it.
I am working on that tutorial but it is not good enough yet for my standard.
@@hughhouWe will wait patiently for that video so you can perfect it; always quality videos from you Hugh!
We trust you!! @@hughhou
The Apple Vision Pro is so much better then the varjo it’s more comfortable weighs a lot less and is standalone also is way cheaper!
Have you tried the new Varjo - I am very curious on that as well. I have the Varjo Air
Sorry but I wish Spatial video was better than 3D stereoscopic video but it’s not. But you can keep selling it that way, just like Apple.
It is a little better and as capture technogy catch up with wider FOV, it will be. Right now it is Week 2 of AVP and Spatial Video :(
3D vid
3D vid, fuzzy frame
@hughhou Is Spatial Video captured from the stereo cameras of the Vision Pro better than the Spatial Video captured from the iPhone 15's single view lenses and then converted to stereo? I would imagine that recording in stereo would allow you to see around near field objects to reveal what is actually behind them. Side note: All those 3D 180 cameras you were looking at during CES are now obsolete. Unless they have depth sensors, they can't really make good Spatial Video and once people see it, they won't want basic 3D video any more.
Yes that is my next video but short answer yes.
Hugh responded to gabrielegelfofx about this and I agree - a stereo image can be converted as it is played back (on the fly) since the depth can be calculated with the different views. Not sure about the spacial video file format, but if necessary it could be pre-calculated beforehand if the software for it is available.
@@gaba023 Stereo video without depth is just 3D video. It is not Spatial Video that allows you to move your head left and right and see what is behind objects in the foreground. That is not only a great feature, it also adds greatly to the realism and makes you feel like you are looking at reality rather than pre-recorded video.
@@BlueHound I agree. But depth can be calculated using two offset images or stereo pair. So if the software uses this, it can have depth as you describe.
@@gaba023 AI can also generate depth. Stereo could make it better. However, the best quality would come from lidar. Stereo adds the background behind foreground objects within the lateral distance of your eyes.
Perfectly stated!!!!
Thank you!!
Are they using some AI with the spatial video conversion then? Maybe that can be done with the Quest 3 too, unless there's a hardware limitation? Linking it to a PC would remove that limitation.
Is it v62 on Quest 3 is out? We should test it and find out any hardware limitation on Q3 vs AVP on Spatial Video. Meta might adopt this format of Spatial Video as well.
@hughhou Are converting to RGBD video? Single 2D video next to a Depthmap video encoded in the same video?
Probably more like a software patent infringement problem for Meta. Cool software features define the user experience and are worthy of being protected as inventions.
@@rmzzz76 I doubt something like that could be protected. There's going to be ways to make something similar.
@@hughhou V62 is out but mine hasn't updated yet. I could force update it, but I think I'll wait. Let everyone else iron out the bugs.
I'm guessing Spacial video doesn't work with either 360 or 180 degree videos, just 3D side by side ones. Still, cool technology all the same.
Too early to tell. I can and I was able to make it works but lots of problem in playback frame dropping so not ready for review yet.
You might be thinking of Apple's Immersive Video format. 180 degree, 360 8K playback they're capturing for some new immersive series just becoming available on AppleTV+. Sure it's a rebranding, but these are early days and Apple jumps in late before pushing the tech.
As far as I can tell it's a stereoscopic video shown on a plane behind the canvas window frame. So I guess it is occluding part of your recording to add a foreground element that gives some 6dof interaction. The perspective in the video itself is baked in and won't change with perspective. So "spatial video" is S3D content shown in a see through window player.
It is what is looks like but it is an entire different story when you see it inside AVP. The 3D in the frame also look different. If that make sense. I am not saying it actually capture more info than before, but how they move the frame is very different. Maybe some kind of low resolution depth map driving presentation - it looks natural for sure!
Looking forward to see it on the AVP for myself!
Some things I noticed (just now) when I
1) Converted an SBS video taken with QooCam EGO to spatial video using the Spatialify app, and
2) Viewed on Vision Pro in a window that I could walk around and view from different angles:
While the video was playing the 3D effect looked natural as I moved away or closer to the window. It looked good from most angles. When I paused the video, the still frame still looked 3d, but as I moved around the 3d looked distorted as you would expect, as it only should look good from a perspective similar to how the camera saw it.
I don't see anything jumping into place when I unpause, so maybe Apple isn't doing anything other than the window parallax thing. But what they're doing is very effective in videos, and doesn't seem to work as well in still-frame 3d images.
XR4 - has @hugh Hou said it right? he showed the "Focal Edition" - but that is not meaning foveated rendering.. the Focal Edition does more. its essentially a passthrough with autofocus
the foveated rendering also the "cheap" 3990 XR-4 can do.
Yes. Most Varjo headset has foveated render not just the focal edition. Also PSVR2 and arguably Quest Pro.
@@hughhou thanxs for clarification..
so in your comparisson: you should have copared the cheaper version with eh Vision pro not the 9900EUro Version..
the normal XR4- VErsion without the auto-focus-thingy - i think - will be here in Europe even cheaper than the Vision PRo, which will cost more than 4000 Euros i guess...
does Meta spatial video solution works as good as the experience on avp?
I don't know yet. Is v62 out now?
@@hughhouyes! But still waiting meta quest app update🥲
@@hughhou v62 is out (in Japan at least).
However there is no Spatial Video folder in the files app like they said should exist- and no option to upload videos from Gallery in the Meta Quest App in my case.
This could be because I am using an iPhone 12 to capture my 3D content, and convert it with spatialify however.
Perhaps they limit the app to show this only if you have an iPhone 15 Pro, I don’t know.
If you have v62 on the public test channel, it's not the full release and lacks many features. I have v62 PTC and can't watch spatial videos either.
@@AlanTruly I see! Thank you!!!
360 Vroomers is one youtuber who says it's just regular 3D movies.
Mic? Did you said that? I did not know. I hear from The Verge and other eariler reviewers like CNET.
Yes, Mic is his name and he made a whole video about Apple not telling the truth about spatial videos.
Sounds like BS to me. You can't capture data from different angles unless you have a camera at different angles. If the headset is in fixed position it can't do that. Only google light fields can do that and they record from multiple positions. Plus I struggled to understand anything he said.
Sorry for my accent. Just to clarify, I did not say this is VolCap (light fields). The camera did not capture extra info as I stated in the video. It used the exact same stereo rig for fair comparison. It’s what the Vision Pro playback makes all the difference. I did not say it’s NOT 3D, but I acknowledge Apple effort to make 3D more enjoyable to watch with head tracking and motion parallax - so more than just 3D.
I want to really point out the fact that AVP spatial video look very different than other 3D video on any devices. Meta Quest might copy it in v62. Hard as no eye tracking but Apple set a new bar for 3D media. I do repeat said this is not 6Dof or volumetric
i honestly hope this video wasn't sponsored by... you know :)
No it is not. I just want to be fair as some of the hate on my other videos or on Reddit kinda dismiss the fact that Spatial Video is more than just 3D. Both tehnically and actual feeling after watching it.
@@hughhou will check, but you still cannot see behind objects in center of the screen, right? so no parallax here.
Who are you and what have you done to our Hugh? He would never leap from one gimmick to another without giving the cold truths.
While you are calling out a "gimmicky 3D look" you are describing a "gimmicky spatial look".
If there is now a process by which that stereoscopy can be refined and displayed more beautifully than before - great - but I think there should be clarity on the differences of captured versus computed depth data with no content creator hyperbole on either side.
And our Hugh knows exactly why there is "the dreamlike look" - it's to hide the unfixable broken facts of life with 3D.
This is not all gimmick though. There "can" be awesome here in some cases.
There are 3 different levels of captured data that could be at play in a "Spatial Video" - 2 of them need computational tricks to be spatial (aka a 3D object in space) beyond "just 3D".
A single camera lens and a depth sensor (lidar) is 2D+Depth and for many people that's already called "fake 3D" not full stereoscopy. The lidar gives you a measured indication of a point in space but the single camera means one eyeball's view must be faked.
2D+depth picture viewers like on Facebook let you look around an image in 3D because the depth data makes the captured scene essentially 3D object that can be pivoted. To use the modern jargon - Facebook has a spatial image viewer.
Two camera lens and no depth sensors give true visual stereoscopy for the backs of our eyeballs but the location of points in space are left to computational guesswork. In traditional stereoscopy that computation is in the human brain. If you want a point cloud or 3D object to pivot around, a machine has to guess what points from each eye are at what depth.
I have yet to see the Hugh spatial workflow but if a depth map is involved to turn the scene into a 3D object that you have some freedom of movement around - that's the point where it becomes a fake.
Only with 2+ cameras and 1+ depth sensor do you get both the measured points in space for a 3D object and direct to eyeball image stereoscopy.
The beauty of the Apple Vision Pro (and Quest 3) as a capture devices are the multiple cameras (at what IPD?) AND depth scanner.
If and when those 3 streams of data are used to capture a scene as a 3D object with some degrees of viewer freedom - that's (almost) new and pretty cool.
Haha agree agree! Spatial Video is based on 3D. The whole presentation with AVP and how it "software" reproject 3D just has a very different vision experience than traditional 3D. Let me circle back on your points but right now I am focusing on actually VR180 on AVP :) Thank you for the indepth comment and yes, you guys keep me in check that is why I love my community.
VR 180 SBS 3D video is real life 3D just as i see in real world. No difference for me.
That is coming NEXT!!! 1 step at a time - working extra hard to push out all the tutorial and review lol. You are ahead of me my friend.
No, sorry, it's a simple relief video, in a frame that's in relief vision, so double relief, even if the second one isn't good, because you'd have to look at it from the front to see it properly. But it's a joke, especially with a little depth effect in the frame, to shift the relief video towards the background, with blur on the sides, because of course you have to fill the gap. The only parallax you'll see is when you close one eye and then the other, but it's better to do it from the front. Nice tweak, but it's all about communication and make-up, ingenious but totally feasible on a Quest 3. My first YT comment, after 25 years of 3D, but you have to set the record straight, it's pretty, it's a nice effect, but nothing revolutionary. Nothing to do with foveated rendering. Keep up the good work!
Thank you and very well said. Thank you for the insight. Def nothing to do with FOV render. I throw it there as a different topic for 180 / 360.
lol Calling a parallax box the video sits in as something special is insane and the problem with how Apple markets things. It's literally 3D video that moves within a box to give it a little more style. The video itself does not have any parallax. It's literally a 3D video.
I hate foveated rendering!! I need edge to edge sharp for my life :) Quest 2, doing this, some games doing this. A hate that!
You can stop it. See my 4K 30fps recording? Edge to Edge sharp! I think you will need third party app to lock it in without Xcode tho as I only know how to do it in developer level so far.
Faux 6dof is still not 6dof.
I never said it is 6DoF btw lol. It is just 3D with an upgrade. So more than JUST 3D. 6DoF need VolCap or game engine :)
@@hughhou I know, but barely an upgrade.
I just come here to see if this is news still LOL.
How you pay 5k when the top one is 4200…
Apple Care cost 500, presecription insert plus taxes
@@hughhou ahh ok gotcha! lol I was about to say they release another one? 😆
So Apple still lied, because even though it's not just 3D, it's still nothing like their ads and promotional material (along the lines of the lenticular front panel advertising).
For my own personal experience - it is very close to their Ads experience if the capture is using AVP onboard camera for spatial video.
@@hughhou The amount of parallax in the ads is absurd compared to the actual amount, as if it's a full 170+ degree shot. And in the ad they even showed that to be captured by the iPhone. It's truly nothing alike.
@@Philafxs I noticed that too with the girls and trees in the background. To me it looked like the original capture must have been panning, then when they created the virtual camera for the add, panned in sync with the capture. There is no way you can move 30 degrees to one side and see everything behind the subject. (of course the add is not showing the full true reality since it is not an in-headset view - the add camera is third person behind the guy with the headset on) To me it exaggerates what the device is capable of. But I don't think Hugh is meaning to say it is that good. Just that there is some parallax effect when you move your head that you don't see with ordinary 3D.
@@gaba023 The video on the launch event shows the same, with a girl on the beach filming musicians with her iPhone.
It's fine to say spatial video isn't just 3D, but not that Apple didn't lie. Much like how much clearer the user's eyes are on the front panel than is the case in real life, which everyone mentioned.
I've seen plenty of people think and speculate that the iPhone was able to shoot 3D180 given that clip, while us VR shooters already knew at least that to be bogus given the lenses. Not much has changed since.
@@Philafxs They had a choice - show only what the device literally can do, which would be boring and unimpressive on a 2D tv or CZcams screen, or show on those screens what it FEELS like when you watch spacial video. I think they had to show what it feels like to sell it. You and I know and understand 3D video, so we see the untruth in the add, but will average Joe buy it then demand a refund because it didn't match the add exactly? We'll see.
Every 3D movie, or any 3D content HAS PARALLAX. Thats how 3D works. To say that other 3D doesnt have parallax is nonsense that just shows you dont know what you are talking about.
I did not say other 3D movie does not have parallax. Just to clarify, The camera did not capture extra info as I stated in the video. It used the exact same stereo rig for fair comparison. It’s what the Vision Pro playback makes all the difference. I did not say it’s NOT 3D, but I acknowledge Apple effort to make 3D more enjoyable to watch with head tracking and motion parallax - so more than just 3D. The point it’s looking different. It knowledge viewers head movement and present info follow head and eye tracking. The end result it more comfortable 3D - not Just 3D. Something better.
So disappointing this spacial video. It doesn’t feel like Apple, it feels like a scam. I almost bought a pro 15 for that, I would have been furious when realising that by myself.
Wait till 16 it build in every phone in 16! 15 is a after tho to put Spatial Video in it. Later you will see the spatial video is a lot better directly on the AVP.
@@hughhou Tomorrow my Vision Pro is arriving. I doubt their "spacial" cheap trick will impress me but maybe. Thanks for your reply. I like your channel.
Yes it's an nice effect I have to admit. But Apple was faking dynamic parallax in its ads, it's extremely misleading so I still consider it a scam that deserve a class action. I thought an AI was completing the background already. We have to wait a year or two maybe.
👍
Absolute clickbait title. It absolutely is 3-D, a two-image stereoscopic 3-D photo. Gimme an effin break. Apple knows exactly what they’re doing, and so do you.
김치 귀여워
Yes!
This contains ZERO INFORMATION just a bunch of blah blah blah. WHAT IS THE DIFFERENCE? You never say! What is the DIFFERENCE? Spatial Video is just stereoscopic Video is all i got from this video... Settle down and give some info!
dislike for 30 fps
So spatial video is worse quality parrallax video... I'll pass
How so?
Meh I wouldn’t call it 3d more I call it crop and zoom with auto scrolling like they do with iPhones wallpaper soo yah it’s nothing new and yes it is a lie even if it small because a lie is well, just a lie..
What make it 3d is if the objects is scan in to cgi 3d space and you see that object in environment that you can move around rather a scrolling image then that’s 3d but if it’s just flat image cropped and it just move around give illusion of 3d then it’s not 3d just a illusion of it.. that’s the truth.. but hung you even agreed that what it is I mean that what it is just flat image crop that scroll around. Meaning not 3d
Wait hold on, I said it is NOT 3D. Spatial is not 3D - it is a different thing. What you describe is exactly 3D. So you are correct.
@@hughhou oh by title I thought you was saying it is because that’s what Apple was claiming back in last year keynote that’s why people is saying that which confuse lot of people lol
Can’t understand this MFKA
It is absolutely nothing like your actual memories. You have absolutely no clue what you're talking about.
How do you know his actual memories? You only know your own.
LOL. I never said I know what his actual memories are Kevin. I said that the Apple Vision Pro is not recording like "your actual memories." How do I know that Kevin? Because I am a Clinical Psychologist who specializes in memory and has for over 2 decades. I know how memory works because of SCIENCE Kevin. And if you had taken even a high school psychology course you would know that our memories - yours, mine, his - are NOT visual "movies", they are scripts. Nor is our actual vision anything like what you're seeing on screen - it's actually very spotty and our brain fills in a lot of holes. Does this answer your question?
@@kevinbailey8827
Thank you for your professional explanation. I agree that term sound a little hype. I am and lots of ppl here are visual people (creators) - so vision is key for remind ourselves in the past - hint Memory. Vision Pro or even Quest 3 help that. Just step in the direction but not total replacement. Just trying to strike a fair argument. Thanks for jumping in 🙏🏼 We need all info we can get.
Sorry if I came off a little harsh. I have been both incredibly disappointed and incredibly fascinated by the launch of the Apple Vision Pro. It is a great device and I wholeheartedly welcome Apple into the mix. In my opinion, they did the equivalent of releasing the Concept Car to the public (for those who don't know what I mean, at car shows, manufacturers will often have a vehicle showing off the "tech of the future." It isn't released to the public, and only some of the features ever do make it). I don't think we will ever see anything like the AVP in future generations, but some of this will carry on.
My disssapointment and facination has been on how it has been covered. I see so many influencers who know about VR and have reviewed other VR products seem like Virtual Reality is a completely new thing to them. I can't count how many have said "WOW. Now I can watch a movie from the comfort of my own living room and be immersed in a theater!" I even saw one yesterday boasting that with the Vision Pro you can now have piano lessons with your real piano and virtual augments. Neither of these is at all new. Hell, I was watching movies in an immersive theater with the Oculus Go. And have an app to do the piano lessons on my Quest 3. It's so ridiculous that I have honestly questioned how much the reviewers who I have always trusted to give fair and unbiased reviews are being paid by Apple. At the same time, I am facinated by the prospect that they are not being paid by Apple, but are genuinely oblivious that they have lost objectivity and joined in a cult of sorts.
It's to a point where I can't trust any review that doesn't point out drawbacks/negatives of the AVP. Not negative reviews, but some sign that they are not just either part of the "If you say one word criticizing, it's just because you're poor" cult or a paid promotor.
@@hughhou
Your video also struck a very unrelated nerve for me. In my work with treating trauma, I encounter false memories daily. We all have them. There is a phenomenon known as the Mandella Effect, where small groups of people will all believe that they remember something that never happened. What is amazing is that these people can believe that false memories exist, but THEIR memory couldn't possibly be false. They clearly "SEE" this thing in their memory, like a video recorded in their brain. For example, many people believe that the actor Sinbad made a movie called Shazam!! in the 90s. Never happened. It's likely that these people DID see something with Sinbad around that time and have a memory for that. But it's not a video recording. It's a script. And when we read the script we visualize it. That script is easy to change, resulting in the video produced by that script changing. It's a lot harder to understand that when you believe that your memory for events are actual videos recorded somewhere in your brain.
@@hughhou
Saying Apple didn’t lie is a lie itself.
When you are communicating, take the damn thing off your face.
My best and only real 3D camcorders are the JVC ones. Real parallax adjustment manually or automatically. You can watch the parallax change when you point the camera at different subjects.
Nothing on the market that is affordable condo any of that. and it has a 3D viewfinder that shows 3D, left side, right side left and right sides, plus it has zoom lenses and stabilization. The camera lenses were spaced properly, unlike all the options available today. Since the Apple Video Pro came out the prices on these cameras have doubled and tripled. So a ten year old camera is better than anything on the market today and it was CHEAPER.
Your take on spatial video is an illusion from software - a gimmick. The 3D from close lenses is gold for close up work.
jvc has 6.5cm lens distance? It's still smaller, isn't?
Thank you for the feedback and love to hear form an expert. I also has the Panasonic 3D rig back in my boradcast day shooting sports in 3D. I know exactly what you mean. That is 1080P 3D tho not even 4K. Softwre is NOT a gimmick tho. Also it is software and hardware inside the AVP make the magic happen. Any 3D camera can be Spatial Video Camera - that is my point and you just prove that. But no display can show Spatial Video in its true potential - only AVP. If you watch it on Vision Pro - it is more than 3D. Yes software illusion, but is it illusion tho if it is different and feel better by many non-professional consumer feedback. They all said - this is the 3D I can watch... its comfortable (I show this to all my firends and clients btw, who are not 3D expert)
@@videos3d313 The smaller the distance the better the 3D. People making the cameras now have no clue except fo
Canon which teased a close lensed version of their 3D lens for the R5, but they haven't released it.
Closer lenses means less cross eyed fatigue and better parallax.
@@hughhou I prefer hardware than actually does the job rather than software to conceal a poorly functioning system. I'm not an expert, but I am above average intelligence and still think. Have the CALF and the EGO and I wish those folks understood 3D. Putting three lenses that one could switch between would be great. You could get three levels of distance from one camera that way.
@@rupertchappelle5303 lol, i made 3d videos since 2008. There is no single perfect base. All depend on subject distance. 6.5cm is what human eye distance is, so best for 3-20m scenes. For closer subject, smaller distance needed, 2-3-4cm like my Sony TD30. For far away mountains i often use 30-50-100-300 (!) cm distance!
I like your content but brooooo, learn to speak englishhhhh, and I'm not a native
Fair I am learning!!