NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields

- Code: github.com/Ton...
- Paper: arxiv.org/abs/...
**Abstract**:
We propose a novel geometric and photometric 3D mapping pipeline for accurate and real-time scene reconstruction from monocular images. To achieve this, we leverage recent advances in dense monocular SLAM and real-time hierarchical volumetric neural radiance fields. Our insight is that dense monocular SLAM provides the right information to fit a neural radiance field of the scene in real-time, by providing accurate pose estimates and depth-maps with associated uncertainty. With our proposed uncertainty-based depth loss, we achieve not only good photometric accuracy, but also great geometric accuracy. In fact, our proposed pipeline achieves better geometric and photometric accuracy than competing approaches (up to 179% better PSNR and 86% better L1 depth), while working in real-time and using only monocular images.

[@mustafabhadsorawala652]

This is brilliant! I worked on NeRF based localization for a project and it was hard enough to create a NeRF offline. This is simply sorcery :D
I am going to have to dive into the paper.

[@Newsik]

This is amazing. If we add light simulations and texture variations it can be a really good system for synthetic data creation in the CV field.

[@Instant_Nerf]

Yeah that’s cool like LiDAR on iPhone adds the data .. very cool idea . Can’t wait to have it Implemented in an app

[@oguzhanercan4701]

great work

[@quangvinhtran319]

​ @Antoni Rosinol What you are doing is really great. However, when I tried to export a PLY file from the instant-ngp GUI, the resulting OBJ file could not be opened. I suspect there is an issue with your version of instant-ngp, and I am not sure if you have encountered the same problem.

[@chi-yaohuang3257]

Does your nerf result contribute to the localization of droid-SLAM?

[@antonikaras9299]

It seems pretty cool! Have you tried running it on an embedded platform, like Nvidia Jetson Xavier, or Orin ?

[@nuclear_AI]

Hi Antoni, congrats on the amazing work!! Could you possibly advise on the hardware requirements for running this?
Thanks in advance💪

[@antonirosinol7918]

Ideally use 2x RTX 2080 Ti, one GPU running SLAM, the other running NERF.
You can fit all in one GPU, but you may run out of memory, I'd suggest having at least 11Gb.

[@nuclear_AI]

@@antonirosinol7918Attempting to get it running on a 12GB GEFORCE RTX 3060 tomorrow...
Wish me luck 🤞

[@fabulani]

@@nuclear_AI How did that go? I'm interested in attempting this, but lack the memory

[@nuclear_AI]

@@fabulani embarrassingly I'm still not there mate.
I seem to be the only person attempting an install on a fresh Win11 machine. I'm finding a few compatability issues with dependency versions that I can't find my way pastl.
Technical knowledge is the current limiting factor, but I'm gaining that knowledge one mistake at a time 💪
🤓😎

[@talhaejaz7651]

Hi, have you implemented this on Windows or Ubuntu?

[@EmptyDreamTV]

loding virtual environment