This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
Progress in photogrammetry has been very slow in recent years, hopefully this approach will yield some new developments. Especially better reflection and translucency capture, which has so far been huge problem, looks like a possibility based on the shown scenes. Very exciting, especially with VR equipment becoming more prevalent. Can't wait to capture real 3D scenes (albeit without motion), rather than stereoscopic video.
Super cool. This will be super useful to sythesize light fields. A lot of people are saying this is not practical right now because it takes seconds to infer one POV. But real time visualization of light field data already exist so you could store precalculated inference that way
Wow. The utility is constrained by the images used to feed the neural network, which may not reflect in varied model environmental factors. If you have images of a flower on a sunny day, rendered in a cloudy day scene, they will look realistic -- for a sunny day. Anything short of raytracing is cartoons on a Cartesian canvas. This is an amazing technique -- super creative application of neural nets to imagery data.
@@GdnationNY The thing I'm curious about is if there are any ways to translate the material properties to a 3D object. The program seems to understand some sort of material properties, but I'm not sure if there is a way to translate that
It's the "neural network interpolating." Each point in 3-D space is assigned an opacity and view-specific color by a neural network function. It's rendered by integrating that function over each ray. So it doesn't model light sources, their effects are just baked into everything the light touches. You could get a mesh by marching over the function domain and attempting to disentangle the lighting effects, but it'd take some doing.
@@TomLieber That's what I was thinking, but there is that one section of the video where the reflections on the car are moving, but the camera is not. That part kind of made me wonder
@@ZachHixsonTutorials You can do that by projecting the rays from your actual camera position, but evaluating the neural network with the direction vector from a different camera viewpoint.
Glass, Car window and interiors, Reflections, only 20-50 photos... perfect scan result... ... On a second thought, I think I get the idea, I realized camera path walks only between frames shot by camera, and it's not so much about making it in 3d, it's about how light works, and a fast render result. It would be great for movies to erase things in foreground, leaving visible what is behind those things that pass in front of camera! In that case it's a great project :)
Can't believe what I just saw. Is there a chance you will make some feature videos? Like is it a real 3d model inside there? Would it be then possible to be imported to 3d software? How the materials are made? How can everything be so perfectly detailed and sharp?and so on and so on
While the hot dogs were spinning at 1:58, I got really hungry and had an unconditional craving for hot dogs. Still nice video, thanks for your upload!!!11OneOneEleven
@@juicysoiriee7376 It is in the sense that it converts a set of photographs to a 3D scene, yet it does *not* create a 3D model in the conventional (polygonal) sense.
So with NeRF, how does the novel view actually get synthesized? I think there is a lot of confusion lately with these showcases as everyone associates them with photogrammetry, where a 3D mesh is created as a result of the photo processing. Is each novel view in NeRF created per-pixel based on an algorithm and you are animating the resulting frames of these slight changes in perspective to show 3 dimensionality (the orbital motion you see), or is a mesh created that you are moving a virtual camera around to create these renders?
Před 2 lety+3
It's the first. No 3D model is created at any moment. You have a function of density wrt to X,Y,Z though, so even though everything is implicit, you can recreate the 3D model from it. Think of density as "somethingness" from which we can probably construct voxels. TO get a mesh is highly non-trivial though This is kinda what they are doing when showing a depth map, they probably integrate distance with density along the viewing ray.
@Hajar Babakhani i don't think you understand. this would be able to take a drawing, if done well enough with proper shading, and extract full 3d elements. just google "3d cartoon art" or "3d drawings" and this would theoretically be able to render or paint an alternate angle of an object based on the existing, handdrawn view of that object. obviously with only 1 view, hidden features would be just that; hidden; but again here ai could be used to fill in "possible options" for optional details. AI is definitely advancing insanely fast, so fast that if an ai can piece together the pieces to gain understanding of those objects it is seeing, understand interrelationships between those objects, words, etc, we might actually see general AI within 5 years. i personally think that ai just needs a sort of secondary (or tertiary, i suppose) "overviewer ai" which would be watching the process of the ai adjusting its gan settings, and would tweak them to improve itself/match what it observes in the real world, and which could comment to itself on its own code, change its own code. i think we may need to be very very careful in these next few years in terms of restricting datacenter-level ai research. cause all it would take is an ai getting some money and access to remote mineshaft with robots that can weld and assemble small parts, and you have some real potential issues. it ships some cpus to those mines, and bam, we're in terminator 2 territory :p (plz be kind to me ai, i'm chill with you being god of the universe!) we've already given ai the ability to see, we're starting to give it the ability to draw things it saw, next we just need to get it the ability to dream/imagine/adjust itself and we're in scifi territory. i think a really good example of where this kind of view synthesis will be maybe applied in the future, is in both video compression and upsampling of older or lower quality video, and 3d scene extraction from movies and the like. take a look at this:: czcams.com/video/AwmvwTopbas/video.html . here you see a sample of some 8k video upsamples that show real qualitative detail improvement; but you can also see the dot/spec/low image quality of the lich king model/character (at the time index in the video and a few seconds after that time index). if an ai would be able to grasp the semantics of the scene, the possiblity of object permanence etc, it could infer from the earlier content in the video that the dot there in the image is the character (the lich king, in this world of warcraft cinematic example) - through rules of object permanence, it could estimate that the spec/smudge in the scene is the lich king, simply viewed from another angle/distance, and thus convert the entire scene drawn herein into a fully 3d rendered construction, in higher resolution, or approximating the resolution of the source cad/blender file data. it could at least do that after seeing the clip in its entirety, or with human intervention - but i think ai will get to where it can do it completely unaided, and probably quite soon (sub 3 years). while this kinda sounds a bit scifi, 2-3 years ago i'd have said that the stuff we are looking at now in this NERF video is potentially doable by ai with extremely intelligent programmers/mathematicians steering the process, and look at where we are. Matthew, you guys are literally creating technological magic. amazing work.
Incredible work! I'm sure this will be coming to Blender in the future - I spotted the Blender shader balls. :D I'm assuming your team have been using Blender to extract the geometry, especially considering this is all open source Python.
Amazing stuff i need to wrap my head on how the depth is generated at 3:22 with the Christmas tree ? I am working on movie where we had to generate depth from the plate and we use all the tool in book but it's always flickering pretty bad never has nice. How would i use this if it's possible?
Nice. The two input angles are screenspace x y coords? and the x y z is the camera position in the training? how do you extract the depth data from the simple topology then?
Hm... so its basically something like photogrammetry...? This could also help photogrammetry right...? Like i capture only lets say 30 photos... But the resulting mesh and texture might look like it was made from i dont know... 100+ photos...? do i understand this correctly?
Can we get some kind of tutorial, more in-depth as to how to do this on our own? Can this be implemented into a game engine while still keeping reflections? Is this cheaper on performance than computer generated screen space reflections? I actually shivered watching this from how insane it looks. I've been into photogrammetry for quite some time now (I know it's not the same) and I would love to try and replicate this for myself as soon as possible
Can this be used to add head movement interactive option to vr movies? Would require a bunch of video cameras and expensive computer crunching beforehand, and how to store/stream the finished data? Or is that a use case where traditional real-time 3d engines are a better fit? Or some hybrid blend?
Now I want to see this in VR 😅 Is it performant enough for real time synthesizing at high resolution and frame rate? Sure puts prior light field techniques to shame 😗
@@dkone165 Ah, so I guess while not real time, it could be used to generate an image set to be used in a real time environment, although it could turn out just impractical. At some point we'll have an ASIC for this, I'd by an expansion card for it 😅
@@fidel_soto These "scenes" are not conventional 3D-model hierarchies. Rather, they are 4D-voxel sets from which arbitrary views can be derived within around 30 seconds per frame. Though some work has been done to "bake" these scenes for real time viewing, the performance still falls far short of being suitable for VR. Perhaps a robust means of converting these NeRF-scenes to high-quality 3D models will become available, yet we already have photogrammetry for that task.
If the new iPads have LIDAR and a sensitive enough gyroscopic sensor - how long before this method can be used to capture objects? (Within 15’ radius of scenes, anyway)
I'd love to know! I wish we could see where in space the reflections are being placed. Ideally, glass would be modeled as transparent at most angles except those with specular reflections, but the paper says that they constrained density to be a function of position only, so does that mean that in the model, glass is opaque and has the view through the glass painted onto it?
SPECTACULAR RESULTS and 3D animation! If interested, feedback follows. 1. You start at 0:00 with a completely white screen. No good. 2. The project title does not get any attention by the audience, given that EVERYTHING moves in the lower half. 3. At 0:39 you are explaining what a hypernetwork is without using its name. 4. 1:58 6 objects spinning like tops. It's hard to focus on so many moving things at once.
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
![NVIDIA Just Made AI Photogrammetry 1,000x Faster [Instant-NGP]](/img/n.gif)
What a time to be alive !
Everyone would think that during the time they live in as only so much as currently released can be drawn from as reference to make this statement.
Dr Károly Zsolnai-Fehér
?!?!
Dear Fellow Scholars !
IRON REBELLION literally same lol
@@EnriquePage91 I love him so much haha
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
2:45 🔥 elegant approach and game changing results - handles fine scene detail and view dependent effects exceedingly well
Progress in photogrammetry has been very slow in recent years, hopefully this approach will yield some new developments. Especially better reflection and translucency capture, which has so far been huge problem, looks like a possibility based on the shown scenes. Very exciting, especially with VR equipment becoming more prevalent. Can't wait to capture real 3D scenes (albeit without motion), rather than stereoscopic video.
czcams.com/video/Pb-opEi_M6k/video.html this is the latest update 3 years later! pretty amazing IMO lol
It’s awesome to live at this time and be participating in history.
Absolutely exciting time ahead, thanks for sharing Matthew!
Mind Blowing! Cant wait to have this in google maps or VR implemented and explore the world!
Thanks for sharing and also mentioning the other contributors to NeRF creation and development.
Super cool. This will be super useful to sythesize light fields. A lot of people are saying this is not practical right now because it takes seconds to infer one POV. But real time visualization of light field data already exist so you could store precalculated inference that way
Wow. The utility is constrained by the images used to feed the neural network, which may not reflect in varied model environmental factors. If you have images of a flower on a sunny day, rendered in a cloudy day scene, they will look realistic -- for a sunny day. Anything short of raytracing is cartoons on a Cartesian canvas. This is an amazing technique -- super creative application of neural nets to imagery data.
Dear Fellow Scholars !
This is two minutes paper with Dr Károly Zsolnai-Fehér
?!?!
What a time to be alive !
So is this actual 3D geometry, or is it just the neural network "interpolating," for lack of a better word between the given images?
These depth maps could be the next step to a mesh possibly?
@@GdnationNY The thing I'm curious about is if there are any ways to translate the material properties to a 3D object. The program seems to understand some sort of material properties, but I'm not sure if there is a way to translate that
It's the "neural network interpolating." Each point in 3-D space is assigned an opacity and view-specific color by a neural network function. It's rendered by integrating that function over each ray. So it doesn't model light sources, their effects are just baked into everything the light touches. You could get a mesh by marching over the function domain and attempting to disentangle the lighting effects, but it'd take some doing.
@@TomLieber That's what I was thinking, but there is that one section of the video where the reflections on the car are moving, but the camera is not. That part kind of made me wonder
@@ZachHixsonTutorials You can do that by projecting the rays from your actual camera position, but evaluating the neural network with the direction vector from a different camera viewpoint.
This thing will allow to create slowmotion from a simple video, and a SUPER SLOWMOTION from a slowmotion!
And also stabilize a shaking camera.
Matt, this is gorgeous! You guys are genius!!! Stay safe and stay healthy guys 👍
Sell it to Google so they can render 3d buildings properly
its available for free,they might use it on street view and create a real 3d google planet thing.
@@nonameplsno8828 wish there was a step by step tutorial to try this with
@@Romeo615Videos there is now but you have compile it yourself.
Check the paper. It's under Google Research.
Outstanding work. Congratulations
Glass, Car window and interiors, Reflections, only 20-50 photos... perfect scan result...
... On a second thought, I think I get the idea, I realized camera path walks only between frames shot by camera, and it's not so much about making it in 3d, it's about how light works, and a fast render result. It would be great for movies to erase things in foreground, leaving visible what is behind those things that pass in front of camera! In that case it's a great project :)
This is amazing!!! congrats!! are you publishing this model on TF Hub?
This looks very impressive, progress here seems on the same level when GANs where introduced.
Can't believe what I just saw. Is there a chance you will make some feature videos? Like is it a real 3d model inside there? Would it be then possible to be imported to 3d software? How the materials are made? How can everything be so perfectly detailed and sharp?and so on and so on
Bro, literally just read the paper if you are interested.
Looks neat!
While the hot dogs were spinning at 1:58, I got really hungry and had an unconditional craving for hot dogs. Still nice video, thanks for your upload!!!11OneOneEleven
Stunning! Are these vector points, volume rendered point clouds? These fly throughs are image sequences?
from what i can tell it's kinda like a metaball except instead of a simple distance function you have a neural network
Great work! Could this also have an application in photogrammetry?
martinusmagneson From the look of it, this could be used as a substitute for photogrammetry.
I think this is photogrammetry?
@@juicysoiriee7376 It is in the sense that it converts a set of photographs to a 3D scene, yet it does *not* create a 3D model in the conventional (polygonal) sense.
This is insane, really nice work
A seminal work!
Consider my mind blown.
Brilliant work ! ❤️
Wow! This means you can get parallax in a VR headset with a 360 video from a real environment. I was sad that wouldn't be possible.
So with NeRF, how does the novel view actually get synthesized? I think there is a lot of confusion lately with these showcases as everyone associates them with photogrammetry, where a 3D mesh is created as a result of the photo processing.
Is each novel view in NeRF created per-pixel based on an algorithm and you are animating the resulting frames of these slight changes in perspective to show 3 dimensionality (the orbital motion you see), or is a mesh created that you are moving a virtual camera around to create these renders?
It's the first. No 3D model is created at any moment.
You have a function of density wrt to X,Y,Z though, so even though everything is implicit, you can recreate the 3D model from it. Think of density as "somethingness" from which we can probably construct voxels. TO get a mesh is highly non-trivial though
This is kinda what they are doing when showing a depth map, they probably integrate distance with density along the viewing ray.
this is astounding
Astonishing.
how can i use this? i coudlnt find anything so far. please hjelp!
Can this method be applied to stereoscopic equirectanular images for use in VR headsets?
So I can finally make my 2D waifus into 3D ones?
@Hajar Babakhani i don't think you understand. this would be able to take a drawing, if done well enough with proper shading, and extract full 3d elements. just google "3d cartoon art" or "3d drawings" and this would theoretically be able to render or paint an alternate angle of an object based on the existing, handdrawn view of that object. obviously with only 1 view, hidden features would be just that; hidden; but again here ai could be used to fill in "possible options" for optional details. AI is definitely advancing insanely fast, so fast that if an ai can piece together the pieces to gain understanding of those objects it is seeing, understand interrelationships between those objects, words, etc, we might actually see general AI within 5 years. i personally think that ai just needs a sort of secondary (or tertiary, i suppose) "overviewer ai" which would be watching the process of the ai adjusting its gan settings, and would tweak them to improve itself/match what it observes in the real world, and which could comment to itself on its own code, change its own code. i think we may need to be very very careful in these next few years in terms of restricting datacenter-level ai research. cause all it would take is an ai getting some money and access to remote mineshaft with robots that can weld and assemble small parts, and you have some real potential issues. it ships some cpus to those mines, and bam, we're in terminator 2 territory :p (plz be kind to me ai, i'm chill with you being god of the universe!) we've already given ai the ability to see, we're starting to give it the ability to draw things it saw, next we just need to get it the ability to dream/imagine/adjust itself and we're in scifi territory. i think a really good example of where this kind of view synthesis will be maybe applied in the future, is in both video compression and upsampling of older or lower quality video, and 3d scene extraction from movies and the like.
take a look at this:: czcams.com/video/AwmvwTopbas/video.html . here you see a sample of some 8k video upsamples that show real qualitative detail improvement; but you can also see the dot/spec/low image quality of the lich king model/character (at the time index in the video and a few seconds after that time index). if an ai would be able to grasp the semantics of the scene, the possiblity of object permanence etc, it could infer from the earlier content in the video that the dot there in the image is the character (the lich king, in this world of warcraft cinematic example) - through rules of object permanence, it could estimate that the spec/smudge in the scene is the lich king, simply viewed from another angle/distance, and thus convert the entire scene drawn herein into a fully 3d rendered construction, in higher resolution, or approximating the resolution of the source cad/blender file data.
it could at least do that after seeing the clip in its entirety, or with human intervention - but i think ai will get to where it can do it completely unaided, and probably quite soon (sub 3 years).
while this kinda sounds a bit scifi, 2-3 years ago i'd have said that the stuff we are looking at now in this NERF video is potentially doable by ai with extremely intelligent programmers/mathematicians steering the process, and look at where we are.
Matthew, you guys are literally creating technological magic. amazing work.
Yes, but she still won't love you.
what exactly is theta and phi respectively? is one the rotation around vertical axis and phi the tilt?
Incredible work! I'm sure this will be coming to Blender in the future - I spotted the Blender shader balls. :D I'm assuming your team have been using Blender to extract the geometry, especially considering this is all open source Python.
OUT
STANDING
♥
Nice video, thanks for your upload!!11OneOneEleven
Amazing stuff!
This will be Awesome for games.
Amazing stuff i need to wrap my head on how the depth is generated at 3:22 with the Christmas tree ? I am working on movie where we had to generate depth from the plate and we use all the tool in book but it's always flickering pretty bad never has nice. How would i use this if it's possible?
Nice. The two input angles are screenspace x y coords? and the x y z is the camera position in the training? how do you extract the depth data from the simple topology then?
Am I understanding correctly that what you are doing here is rendering the nodes of neural network in 3d? If so I wonder if it could have non CG uses?
Amazing!
Wow it's cool..say for example, to capture an average human just standing. How many pictures are required to capture enough detail?
amazing
Hi this is really interesting. Can you tell me maybe how much costs one rendering of about 1000 photos? Which program is used for that? Thanks :)
Hm... so its basically something like photogrammetry...?
This could also help photogrammetry right...? Like i capture only lets say 30 photos... But the resulting mesh and texture might look like it was made from i dont know... 100+ photos...? do i understand this correctly?
3:00 how this animation on the right is produced ?
Якою програмою можна це зробить?
Is there a way to try this with my own inputs?
archive.org/details/github.com-bmild-nerf_-_2020-04-10_18-49-32
Can we get some kind of tutorial, more in-depth as to how to do this on our own? Can this be implemented into a game engine while still keeping reflections? Is this cheaper on performance than computer generated screen space reflections? I actually shivered watching this from how insane it looks. I've been into photogrammetry for quite some time now (I know it's not the same) and I would love to try and replicate this for myself as soon as possible
I may have misinterpreted what this does, but if so, can this be used in games?
Can this be used to add head movement interactive option to vr movies? Would require a bunch of video cameras and expensive computer crunching beforehand, and how to store/stream the finished data? Or is that a use case where traditional real-time 3d engines are a better fit? Or some hybrid blend?
Now I want to see this in VR 😅 Is it performant enough for real time synthesizing at high resolution and frame rate? Sure puts prior light field techniques to shame 😗
"On an NVIDIA V100, this takes approximately 30 seconds per frame"
@@dkone165 Ah, so I guess while not real time, it could be used to generate an image set to be used in a real time environment, although it could turn out just impractical. At some point we'll have an ASIC for this, I'd by an expansion card for it 😅
You can create scenes and then use them in VR
"The optimization for a single scene typically take around 100-
300k iterations to converge on a single NVIDIA V100 GPU (about 1-2 days)"
@@fidel_soto These "scenes" are not conventional 3D-model hierarchies. Rather, they are 4D-voxel sets from which arbitrary views can be derived within around 30 seconds per frame. Though some work has been done to "bake" these scenes for real time viewing, the performance still falls far short of being suitable for VR. Perhaps a robust means of converting these NeRF-scenes to high-quality 3D models will become available, yet we already have photogrammetry for that task.
If the new iPads have LIDAR and a sensitive enough gyroscopic sensor - how long before this method can be used to capture objects? (Within 15’ radius of scenes, anyway)
Very cool
oh my!
cool research
DUDE WHAT
That's photogrammetry... 😐 (Edit) Except, it isn't... It's a thing I dreamt of for years
Are radiance fields compatible with 3d editors like Blender?
Hello. this NOT Dr. Károly Zsolnai-Fehér
how was it train?
@ 3:59 How the hell did it model glass!!
I was thinking the same, also car windows with interior, reflections, and 1:58 where the hell they scanned real life Blender's Material Spheres?
I'd love to know! I wish we could see where in space the reflections are being placed. Ideally, glass would be modeled as transparent at most angles except those with specular reflections, but the paper says that they constrained density to be a function of position only, so does that mean that in the model, glass is opaque and has the view through the glass painted onto it?
Pretty fucking cool
I would love to speak with you about this!
Noice 👍
Incredible is there anyway I can do this at home?
Can we export 3d model?
SPECTACULAR RESULTS and 3D animation!
If interested, feedback follows.
1. You start at 0:00 with a completely white screen. No good.
2. The project title does not get any attention by the audience, given that EVERYTHING moves in the lower half.
3. At 0:39 you are explaining what a hypernetwork is without using its name.
4. 1:58 6 objects spinning like tops. It's hard to focus on so many moving things at once.
Doesn't look like anything to me
Because its so realistic that u just dont get how big deal that is
@@kobilica999 I was referencing Westworld
@@blackbird8837 ooo that is different story xD
Bernard?
UPDT
been trying for a month to run the example scenes, anyone got thru ?
holy s**t.
It's NeRF or Nothing
I believe the link in the video description is wrong, but I found the page here: www.matthewtancik.com/nerf
ps5 grafics
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.
This is amazing! The implications for 3d and compositing workflows alone are mind-boggling. Can't wait to see this filter through to mainstream products.