BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Reintegration
Vložit
- čas přidán 9. 09. 2024
- ACM Transactions on Graphics 2017
Project Page: graphics.stanfo...
Real-time, high-quality, 3D scanning of large-scale scenes is key to mixed reality and robotic applications. However, scalability brings challenges of drift in pose estimation, introducing significant errors in the accumulated model. Approaches often require hours of offline processing to globally correct model errors. Recent online methods demonstrate compelling results, but suffer from: (1) needing minutes to perform online correction preventing true real-time use; (2) brittle frame-to-frame (or frame-to-model) pose estimation resulting in many tracking failures; or (3) supporting only unstructured point-based representations, which limit scan quality and applicability. We systematically address these issues with a novel, real-time, end-to-end reconstruction framework. At its core is a robust pose estimation strategy, optimizing per frame for a global set of camera poses by considering the complete history of RGB-D input with an efficient hierarchical approach. We remove the heavy reliance on temporal tracking, and continually localize to the globally optimized frames instead. We contribute a parallelizable optimization framework, which employs correspondences based on sparse features and dense geometric and photometric matching. Our approach estimates globally optimized (i.e., bundle adjusted poses) in real-time, supports robust tracking with recovery from gross tracking failures (i.e., relocalization), and re-estimates the 3D model in real-time to ensure global consistency; all within a single framework. We outperform state-of-the-art online systems with quality on par to offline methods, but with unprecedented speed and scan completeness. Our framework leads to as-simple-as-possible scanning, enabling ease of use and high-quality results.
Wow.... Its easy to see this integrating into AR scene reconstruction.
The performance is absolutely incredible and the results are perfect.
you have a meshy appartment!
Thanks for inviting me your home! ^-^
Damn, this looks good. (VR|AR) + (RGBD scanning) = $$$$$$$ * Lots of fun
This is just f-cking amazing man
Is the output a mesh grid or just a point cloud?
I know this video is 4 years old but I wanted to know if you could set up stationary cameras around a square room, and be able to use your program to record live motion. I'm looking for a security camrea with no blind spots and that if someone steals some boxes i would be able to take inventory of what they took maybe even scan upcs as they are taking them and also to have a 3d image of the person(s) with facial recognition useing Facebooks auto tagging feature..
With pretty loop closure detection,GJ :)
This looks similar to some simultaneous localization and mapping (SLAM) systems. Can it generate large-scale 3D maps of buildings or neighborhoods?
Damn! This is way too awesome!
Any source codes?
AWESOME STUFF! Question: Let's say one has a scanned or any 3d data like that and wants to calibrate this data dimensions to real world data. What software would You recommend to do that ? I suppose simple measuring tool in 3dsmax and scaling would do the job , but maybe there is some more sophisticated tool for doing that ? Thank You in advance.
damn that's amazing, how does it track the user's position/locks the bottom right part (purely imu?)
or knows from rough/major landmarks?
Hello. Are you working on that project or similar?? Where i can find people on that?? Thanks
+Matthias Is it possible to detect hand moment?
This is amazing. Mesh looks clear as industrial laser scanners do. Will it work with "Structure Sensor MK2"?
Will you ever make this open source?
dude... get used to google.. github.com/niessner/BundleFusion
in luis' defense, he asked before it was published. unfortunately that it's advancement is crippled with non-commerical and whacky-doodle license.
Create a ball drone with a 360 camera attached to it. Put it in the center of the room , then program the drone to fly to 1.8 meters and start scanning... Instant 3D interior auto scanning.
So cool!
yeh whats matterport ?
Lots of jobs available to you in the self-driving car field. Amazing work, especially because it's only one camera, not two or more. Neither is it some kind of spinning mirror thing or radar. Well done.
This is a depth camera though. Fantastic work nonetheless.
Awesome.
can this be used for video games?
sure, but you have to rebuild 3d mesh any way, and map it proper, game engines need clean mesh, and textures should be pics based not a vertex texture, but sure... u may bake color maps from that high poly vertex textured model, still a lot of work.
It has been used for AR games and for VR inside out motion tracking.
Neurangelo ☝️🤓
Is this rgb camera or rgb-d?
looks like RGB-D cause the depth map looks neat
I'm very interested in the software development of this😃
Incredible work! Is the depth sensor custom? Or a repurposed kinect or something?
It looks like the Structure Sensor.
mantap
okay cool, now MAKE IT AN APP (or a piece of software which we all can use! :-) )
alway discasting 3D scan in 2018
I foresee a military application of this for urban warfare.
the applications for this ...
Jafar Tigue It can be used to indoor tracking (GPS precision don't let that). Positional tracking on mobile virtual reality... many others.
It can be used by the FBI to find your basement dungeon.
Amazing!
Will you ever make this open source?
+snaileri Yes -- once it's officially published.
I want it :D
6 month and still not published, can you create a newsletter to notifyus when it will be published?
rtab-map is a published alternitive that uses some of the same ideas.
the moment when comments are worth reading! thanks!