Video

Thank you for your interest in our work. Now finalising the code, it will come in next few days.

For now you get pringles

Yes, it's now available here: github.com/makezur/super_primitive

When in the future everyone has vr glasses with this technique built in. Slowly rendering the entire world.

Hi Kun. Thank you for your interests in our work. I am sorry I was busy writing thesis. My email address should work well right now and not sure if it is some wierd server issues. If you can not contact me via imperial email, you can also drop me a message to z.shuaifeng@foxmail.com if you like. I will release the rendered Replica sequences after the recent thesis DDL and sorry for the delay.

Hi kwea123 (AI葵), thank you for your interests and feedback. I also learn a lot from your tutorial videos of NeRF which are very helpful. I agree that the information in this video is a bit dense and we have tried to keep a good balance between video length and presentation experience. I could possibly make another longer version on project page so that people can better follow the details.

About pixel-wise denoising: The performance of pixel-denoising task is quite surprising at first glance, especially when some fine-structures can be well persevered. In the denoising task, we randomly vary the labels of randomly selected 90% pixels for each training label image. In my opinion, I think there are several factors making this happen: (1)Coherent consistency and smoothness within NeRF and view-invariant property of semantics are the key. (2)The underlying geometry and appearance play a very important role so pixels with similar texture and geometry tend to have same classes. The photometric loss is important here as an auxiliary loss. I personally think denosing task here is a “NeRF-CRF” given that CRF also refines semantics by modeling similarity in geometry and appearance in an explicit way. (3)There are still average 10% pixels unchanged per frame and in addition a 3D position may have corrupted label in one view but may have a correct label in another view. I also tried 95% or even higher noise ratio, and as expected the fine-structures become much harder to recover with less accurate boundaries, etc. The quantitative results does not aim to show which task is easier or harder in any sense but mainly to show that Semantic-NeRF has the ability to recover from those noisy labels. Note that the evaluation are computed using full label frames including chairs and other classes as well.

It is true that a larger scaling factor (x16, x32) has a risk of missing tiny structures. And we indeed observe, for example, prediction of windows frames (red) around blinds (purple) in spx8 is more accurate than that of spx16. Again, the tables does not mean to compare these two tasks but to show the capability of Semantic-NeRF. A better way to think about super-resolution and propagation is how they sample the sparse/partial labels. Super-resolution (e.g, SPx16) sparsely decimate label maps following a regular grid pattern with a space of 16 pixels while label propagation (LP) select a “seed” randomly from each class per frame.

In SP, a class/instance larger than 16 pixels is very likely to be sampled at least once (i.e., having one/more seeds on this class/instance). Therefore I think the main difference is the coverage of seeds: SP spreads the seeds within class while LP learn from more labels from a local proximity. This is also one of reasons why prediction of light (pink) on the ceiling (yellow) in SP has better quality (Fig.7 and 10) than that in LP(Fig. 8), partly because the appearance and geometry of light and ceiling are too similar for LP to interpolate and the spread of seeds in SP helps

Hope this information and my understanding is helpful. If you have any further questions, please feel free to discuss via emails.

Thank you! We will start working on releasing the code as soon as we finalise the RA-L journal publication.

@@dysonroboticslaboratoryati9846 Great Wish you the best of luck (y)