Light Fields - Videos From The Future! 📸
Vložit
- čas přidán 11. 01. 2021
- ❤️ Check out Lambda here and sign up for their GPU Cloud: lambdalabs.com/papers
📝 The paper "Immersive Light Field Video with a Layered Mesh Representation" is available here:
augmentedperception.github.io...
❤️ Watch these videos in early access on our Patreon page or join us here on CZcams:
- / twominutepapers
- / @twominutepapers
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Alex Serban, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Eric Haddad, Eric Lau, Eric Martel, Gordon Child, Haris Husic, Jace O'Brien, Javier Bustamante, Joshua Goller, Lorin Atzberger, Lukas Biewald, Matthew Allen Fisher, Michael Albrecht, Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Ramsey Elbasheer, Robin Graham, Steef, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh.
If you wish to support the series, click here: / twominutepapers
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/
#lightfields - Věda a technologie
Moving the camera while seeing paused motion blur is such a cool effect.
Yeah the shot with the flame got me a vibe of star trek "holo situation" or something like that. Or even an advanced futuristic police crime scene analysis feel
Like glitchy?
Yeah
Almost like the bullets in the matrix
also
moon?
@@CosmiaNebula Yes, my sun! Hi, noon!
This is absolutely incredible. Imagine a movie where you can actually look around the scene! What a time to be alive!
Upskirts finally take on a whole new definition
It would make for an incredibly immersive crime movie where the viewer could look around a scene for evidence at the same time as the actors.
Eg looking behind a picture frame or ornament.
>What a time to be alive!
You are right. 10-20 years from now most of us won't have indoor plumbing any longer.
It has its limits. As soon as you leave the volume that was originally occupied by the camera sphere, there are gaps in the data that become increasingly obvious.
not just look arround, but walk inside it!
This is one of the few papers that really made me think "this is the future". Feels like wizardry to rotate and pan around a video, rather than a still image. Not a super practical thing, but it's dang cool!
Two immediate applications I can think of:
1. making better video instructions for the assembly of complex 3D parts in manufacturing.
2. VR porn. Because, why not?
Not a super practical thing? Ask any Movie director
@@kamathln I mean, not for normal videos, sure this has its own utility but you normally want to center on whatever the thematic of the video is and not something off the scene
@@javierflores09 I think regular movie directors (i.e. not ones who want to focus specifically on providing lightfield movies) will actually want this as well:
It means they can much more seamlessly adjust the camera position in post, allowing for insane levels of fine control. So they'll end up with a static view (or perhaps a binocular view ala current 3D movies) but it's extremely finely adjusted to their every whim.
It's also gonna improve the workflow of what I *think* is called Z-compositing? Wherein all your data is in RGBD and so you can composite some CG elements into the scene at exactly chosen depths and because your footage has depth info too, clipping and occlusion will automatically work correctly instead of requiring expensive, error-prone semi-manual frame-by-frame masking.
Essentially this gives even more information, including to stuff that's completely occluded by other stuff. Like, you could digitally add in a reflective surface and it would potentially reveal what's behind a thing *in the digital reflection*
etc.
So yeah I'm sure film makers will love this.
@@javierflores09 Try centering on hyper-active children/pets :D
"Hold on to your papers" that joke always makes me smile 😂
Imagine People walking around with that Monster taking Selfies 🤭
*karen has entered the chat*
What do you mean “it’s not for sale”????
I DEMAND TO SPEAK TO YOUR MANAGER!!!!
@@jearlblah5169 😐 why
I could see this being used for movie production; imagine being able to film once and then tweak the camera movement in post production.
It would quintuple the data storage requirements tho
That's what Lytro were aiming for with their Immerge camera. I don't think anybody was interested though, cos they folded and got gobbled up by Google.
imagine shooting with a rig that has like 20 movie cameras attached to it... maybe its just easier to know what you are doing in the first place
It is possible czcams.com/video/9qd276AJg-o/video.html
Look at intel volumetric video studio
@@NoogahOogah which almost all technology advancement has done lol
at each video, we get closer and closer to magic.
Any technology advanced enough is indestinguishable from magic
Any magic arcane enough is indestinguishable from science
@@overloader7900 I really like how you said that:) I'll be quoting you
@@user-iv1qq3vb9j Is one of the three Arthur Clarke's laws
@@martiddy And its variation, thanks for telling
Download the VR version from the website!
I realised this was the first time ever that I could experience *true* 3D video (not just stereo). It is something else! I kept wanting to pet the damn dog! :D
I just tried their demo in VR and it was very impressive. Things felt so real, like I was there (I know, of course it's VR, but still!). Now I am sad, that I cant watch every video like this. Also the one sitting at a lake or at the ocean were very relaxing. This has definitely the possibility to be used for virtual tourism. AMAZING!
This was exactly the sort of thing I was looking for after the last video with the video selfies. This looks like an incredibly promising way to make VR video more immersive than ever before.
VR porn ist gonna be a saviour for early funding of this technology I'd wager
1:27 That animation was so cool!
Wow, add tobii tracking device and this will revolutionise the media!
What is a tobii tracking device?
@@jambalaya201 Tracks exactly which part of the screen your eyes are looking at
What about just using a VR headset? This is basically designed for VR.
@@rtyzxc It will forever be less convenient than looking at a screen. With tobii eye tracking you can reach a middle ground between immersion and convenience.
Just use camera eye tracking.... It's almost like we know a guy with a bit of expertise in camera based detection...
Imagine watching back your memories played back in that.
it feels so weird for humanity to be so perfectly preserved for the first time ever
My jaw dropped when I saw the dog behind the fence! It was almost perfect, such an amazing implementation.
I first saw this tech used in vr with google lightfields (on steam), if this becomes affordable I would be really happy
We can download a 8GB demo !
augmentedperception.github.io/deepviewvideo/
That was already a few years ago. Although they were still pictures, it still looked amazing as if being there.
watching these videos only for "What a time to be alive!" words
I really can't wait for this to catch on. True VR videos? YES PLEASE!
Wow that was amazing.
There's so much development in this field, please, cover them more
*waiting room for 5 papers down the line*
This is so magic… I was think about this yesterday and don't know what keyword I should search in Google, then you creat this video today! WoW!!
I really hope this leads to better predictive image in-painting soon. Having something of this quality on a phone with minimal hardware changes would be absolutely incredible
This domain always has been fascinating to me. I once reimplemented on GPU the method of the paper "Soft 3D Reconstruction for View Synthesis", and it made me learn so much about light and depth.
Fecking amazing! Tutorial videos (and all videos/movies) of the future will have so much more!!
This feels way more enjoyable to me than 360 VR. I like the idea of a 3D video space where there's parallax if I happen to shift in my seat or tilt my head. And more importantly, the viewer has an actual sense of scale and depth- They can lean in to get a closer look at something in a Maker video, for example.
Imagine now how Walking Simulator games, would be! WHAT A TIME TO BE ALIVE!
For anyone wanting to view Light Field photos (not video) in VR, Google released a free app for SteamVR in 2018 that's pretty awesome:
store.steampowered.com/app/771310/Welcome_to_Light_Fields/
The Google lights fields demo on my vive left me wanting more amazing light field content. I'm happy to see the amount of progress being made in this fascinating field.
This technique with 360 degree viewing looks like it would allow for really immersive VR movies
This could work to accelerate VR-as-a-service platforms, where the remote game engine renders an entire light field for the approximate location of the VR headset, which is delivered once every second or two with extremely high compression. The client's PC decompresses the light field and maps the actual head movements and rotations onto the now-old data, simulating a highly responsive remote rendering service of a semi-static environment.
For VR, it's not just about rotating your head, it's 6 degrees of freedom. The image is still correct when you move your head and this is why lightfield images feel so real in VR, it's like the realism of a photo/video combined with the freedom of real-time 3d.
Having both played Cyberpunk and just seen these videos on my VR Headset, it is almost frightening how close this is to a supposed Brain dance Experience from the game. The indefinitely repeating glitches on the side of the view field make it even more realistic. Awesome how close it even is from being actually there in the scene. Just want to up the resolution a notch. Now you have to interpolate space from only two cameras and their movements in space, get them small and fit them into eye implants. Voilà, Kiroshi eye implants from the game.
I watched this entire video with my mouth gaped in awe. I cannot wait until 2 maybe 3 papers down the line where the artefacts are less noticeable and the content may be optimised in a way so that it could work over a limited connection such as mobile data. Hopefully another 5/10 years down the line we will be able to achieve the same camera angles all available on your smart phone. who knows! truly what a time to be alive.
Your channel is amazing man really make it enjoyable to learn and it’s interesting and needful concepts!!!
You are very kind. Thank you so much! 🙏
This is actually the future. Can't wait to watch these kinds of videos on my VR headset.
I love your quiet almost-ASMR voice! It's great!
Light fields are amazing - that's the field in which I'm doing my PhD. There is so much promise in this technology, and we've come a long way since the brilliant Lytro camera.
360 degree cameras can accomplish some of this, which is ok, but we definitely can improve.
the thing with 360 degree cameras is that it still only captures from one position, that being where the camera is. with this technology, you can kind of move around as the camera and experience real depth effects, unlike a 360 video where you basically move around a gigantic frame, but depth may not be accurate
The moment you mentioned reflective surfaces I caught myself pretty much only looking at the mirror in the shop. :D
Absolutely stunning work.
What's weird is that the camera of the future looks a lot like the human eye. What a time to be alive!
This needs to get promoted big time. It needs a web platform providing content from spectacular places around the world. It needs a dedicated app for the wide spread use with mobile devices via browser and cardboard experiences.
Go to rallies, fairs, TV studios, landmarks and great vistas to create a bulk of content.
How odd that the relatively young concept of wide angle stereoscopy already seems to have found its match.
I'm curious to see what's the future will hold for this technology.
I think lots of people on here are confused about the difference between 360 video, photogrammetry, and light fields. It's not just about stitching multiple videos together and slapping it on a 3D model. A true light field is the 5D value of every point of light within a volume. They are capturing a few 2D slices (where the sensor collects the photons) and the algorithm is filling in all the rest for the (hemi)spherical volume. Then it's compressing that down into something that can be streamed, which is probably a 1,000:1 ratio. It's like compressing an IMAX feature film into the size of a JPG file!
I've been having this idea for years:
Take a huge concert in the daylight.
The lead musician asks everyone to take a short video at the same time, for example of a jump he does or something cool.
Then these 30,000 people upload their video on a given platform (RIP 4G antennas).
=> You get a messy, inconsistant, but free and unimaginably huge dataset you can perform 3D reconstruction on!
After a lot of cleaning and reconstructing, you could walk through the crowd, turn around the stage, etc.
I know some stadiums are equipped with 4D Replay or other technologies that allow for similar experiences, but only for a very limited portion of the venue.
I can see this being useful for video/movie editors to perfect that camera angle / pan after the fact!
Can't wait for the Dall-E paper from OpenAi (and its video)
While we wait for this channel to cover it, I found Yannic Kilcher's video on it to be pretty good : czcams.com/video/j4xgkjWlfL4/video.html
@@MrJaggy123 watched that already :3
Damn you got lucky then
AI is going to teach us something about light we didn't already know existed in real life
100%, it's only a matter of time before AI "scientists" become a reality.
@@Baleur
And then AI takes over and kills/enslaves us all
@@moahammad1mohammad What makes you think an AI would have any reason or motivation to kill or enslave us?
@@sgbench *Programs robot to walk*
*Person stands in front of robot*
*Robot doesn't see person*
*Person codes robot to see person as obstacle*
*Robot "overcomes obstacle"*
lolol
@@markmywords5342 In your example, a person programmed the robot to be evil. The robot didn't become evil on its own.
Virtual theatre plays! Live streamed in VR! Potentially a great way to keep theatres open during a pandemic, if only this were commercially available
There is a Demo for this in VR, which makes it even more impressive, as the content can be rendered in stereoscopic mode, because all information necessary for that is included in the dataset.
A good paper this week. Thank you.
Imagine they implement this technology in VR movie experiences where you become a character inside a movie like a horror one for example, and than you experience the entire spectacle first hand.
Omg the reflections updating blew my mind the most
Reminds me a bit of the Lytro Lightfield Camera.
Tiny box capturing all incoming light and lets you change the focal plane as well as your point of view by a few degrees.
Imagine what you could do with an array of Lytros and VR+Eyetracking (incl. pupil dilation to calculate the eyes aperture to adjust depth of field) for replay. Total immersion
So interesting! Thank you Károly!
I just realized that it makes a lot of difference in watching VR. It is the small head movements that do not show in the image that cause nausea and break the illusion of reality in VR. So yeah, much more relevant than a silly 3d effect.
Remember the head pivots around the neck joint, not the eyes, so there is always a parallax effect that needs to be handled even when the viewer is sitting still in the vr scene while just looking around.
Papers squeezed in preparation!
Now THIS is something I am genuinely very impressed with - I was already impressed with the Tom Grennan video on the PSVR, but that was a lot of work. This does it completely automatically. I'm definitely going to keep an eye on those papers.
This is amazing!
Can’t wait to watch VR movies with this technique in the future.
yeah I checked this out in VR a few months ago - absolutely the future!
This is looking incredibly promising! I still remember seeing a demo at a conference of what LYTRO was doing with lightfields (2016-ish), I was blown away at the time. Too bad they went 📉 but I'm glad to see lightfield technology still going strong
This totally fucking rocks! Just awesome to see where this will go for entertainment and teaching/instruction.
I do think we'll see bitrate for these formats decrease in the near future. All current codecs including H265 are made to compress a series of nearly similar 2D images through time, not to compress multiple parallel streams. Just like how there's similar information from one frame to the next, there will also be very similar information from one stream to the next in this case.
Great piece of feedback. Thank you so much for posting this!
I can already imagine how much more immersive this will make VR videos. No more of that jarring/nauseating feeling from having the world shift with head movement.
Still love hearing the word "doctor" at the beginning of his new videos!
To those who like this idea - take a look into Lytro camera. They managed to produce their first light field camera model available for purchase and then they went under. Not too many people interested in the technology. With this special camera you can capture more information and store it in a file that needs to be reconstructed in their software, where you can chose the angle and focus of your picture and then export it in a normal photo. This is mainly why nobody was interested. Other than that - this technology looks like magic compared to what we are used to. If 3D TVs didn't die out - this would have complemented them very well. But nobody wanted to invest more money in 3d. Companies that worked on light field on 3d TVs with no glasses needed also went in oblivion. If that didn't happen - today instead of fighting over hdr- we would have be looking at TVs with popping picture you can look at from all angles.
In VR this light field technology allows for removing the bulky heavy lenses that sit on your nose and thus making VR glasses as thin as regular glasses. Also - this technology allows for your eyes to move around and focus inside the image, making the experience much more realistic and less vomit inducing. However - Nvidia - the only people behind it - also backed out and stopped developing their 3d and VR glasses.
How about that?
Soooo, photogrammetry applied to each frame of a video scene captured from various angles to reconstruct the 3D scene?
in VR this is mindblowing.
I can see this becoming a new method of viewing content... like video over books.
Super!
That's an old stuff that's going with it's roots up to Lytro cameras from 2006. But still the stuff doesn't stop to impress and be "the future" for 15 years straight in a row.
No, it's not. They're using the term "light field" to mean something very different than the light-field cameras like Lytro's.
@@IceMetalPunk Actually it is. The method of taking many images from many different angles at the same time, was described in 1908 and that was called Light Field and it's exactly what was done in this case. Lytro innovation was to make an optical raster film of micro lenses that's supposed to write all viewing angles in one single frame at one single exposure moment. Limiting resolution and angle differences and as a result ability to move camera. Still technology in general the same.
Means you can probably get results like these in a camera embedded on your phone
@@LKRaider Technically yes. But the quality of execution might vary. And it will take a lot of fiddling. And the bigger the "virtual aperture"(distance between farthest points from which you got pictures) the more you'll have the ability to move the camera. And technically if you get pictures from every possible angle, you'll get the ability to freely move the camera inside the scene. If your "aperture" wold be small, you'll be able to move camera only in that vicinity, and may be have ability to change focus and some other properties, like optical aperture postfactum. Of Course also you can capture still scene, making many photos from many different points. Ether way, it's all by its core just interpolation between those frames, based on coordinates of actual cameras position and the position of the virtual camera. The main point of this research, as I understand in the actual way they do this interpolation that is more likely has better quality and may be a bit faster than previous attempts. Here they build some kind of mesh representation of the scene, which has some similiarity with 3d scanning. But 3d scanning by itself very similar to light field concept.
Amazing.
Here is how to do it with a single market-ready phone camera: create a camera app that when you click the button to take an image, it starts filming a video, allowing you to make a verticle circular motion with the camera in a 2 foot diameter, then you click the button the second time to take the photo. Then the camera app constructs the 3D image using the algorhythm and the images from the video. You can fight artifacts and blurring by letting the program draw detail information in a single 10x10 pixel area from a single frame that most closely matches the surrounding pixels.
astonishing!!
There is a studio in France that uses a ligh stage and shoots 3D video. So they do a photogrammetry reconstruction for every single frame.
OMG is this cool.
This is sooo crazy.
I did not expect this while clicking on this Video
this is amazing
Combining image inpainting with the data of two or three extra cams seems like a natural next step, imho.
It's like we have the real world on your computer
This is like Photogrammetry but for video, very impressive
Can't wait to see a paper that defines a new compression method specifically for light fields. H265 is good but I seriously doubt that it's designed with light fields in mind.
Amazing
AWESOME
Our world contains a lot more information than regular person can think of. Imagine capturing all of it :)
This would be amazing on a flat screen with a head tracker. Or even better one day if they can make a light-field display, i.e. a two dimensional array of tiny projectors that project directly into the eyes of any number of viewers.
Damn this is something else!! 💎
I know that Cyberpunk has been slated - but this reminds me very much of the braindances from that game - you can explore previous memories & events in a similar way to find out additional clues around the scene!
That's so cool
I love viewing these in VR
amazing.
Check out "Welcome to Light Fields" on Steam for VR
That's awesome stuff. I look forward to the hardware for it becoming cheap, commonplace, and integrated.
I'm also looking for some more solutions to data compression. Have the AI scientists designed something for general compression of noisy data yet?
Oh god this is incredible
beautiful.
We may need a custom entropy encoding format that look across all view ports for these highly coupled videos. Most of the image across different physical or logical cameras are essentially the same. Only some highlighted area shown in 1:44 are actually different so we can skip other part and only keep a key frame of it.
amazing
This channel is basically a window into the future
nice!
Awesome
:OOOOOOOO oh wa...... this is huge!!