Depth Compositing Integration

[felixgeen]

First of all thank you so much for focusing so much energy on this project! I’m excited for all things NERF.

I have been playing with nerf a lot since it came out last year and to implement it properly into my Workflows my main wish for it currently is to have some way of compositing objects “inside” the nerf. Even if this means splitting the nerf into two render buffers (near and far) so objects can be placed in between these two layers.

it would be very cool to somehow be able to do proper depth composting using nerf.

another option that would enable this would be rendering out DEEPEXR (DEEP) which produces multi-sampled depth renders at the expense of file size renders. If you’ve ever tried depth composting objects with just a basic black&white 32bit depth map then you’ll know that it doesn’t work due to the edges of objects needing multiple samples to comp correctly.

would be very interested to hear your thoughts on how you might be able to bring depth compositing to your nerf implementation.

[JamesPerlman]

I appreciate your kind words! I love doing this and I'm happy you're excited about it.

Firstly, I'm going to assume you're talking about combining triangle-mesh objects with volumetric NeRFs. I figure this is a feature many people are going to have a need for. There might be some clever way to do AI-assisted depth compositing, I haven't looked into that but essentially it would be something that reduces the edge aliasing artifacts you're referring to with standard 32-bit depth maps (luma mattes for example).

Initially, standard depth compositing is going to be the only option. Beyond the essentially 2D approaches like we've already discussed, there are certainly some other options:

1. Build a NeRF object renderer directly into a fork of some preexisting open-source project, like Cycles. The time and effort investment is quite large for this path, and it doesn't sound like too much fun to me. But it would essentially have a NeRF be a first-class object like a VDB.

2. Bake your 3D object as a NeRF. This one actually seems really fun. Essentially in Blender you could render your object from multiple view angles and create a NeRF which could be rendered alongside other NeRFs with ease! This could be pretty easily automated too, no COLMAP necessary since we have ground truth camera poses available immediately. Downside to this technique is there is a limitation to how detailed the object can be, and an automated approach might not be one-size-fits-all. Depending on the shape of the thing, you'd need to be careful with your camera placement. Not to mention all the other current pitfalls of NeRF: forget surfaces that reflect other parts of the scene, and don't even think about getting accurate specular highlights on heavily-curved surfaces.

3. Bake the mesh object as an SDF. This also seems really fun, perhaps the most fun and interesting of these options. SDFs would be way faster to render than a baked NeRF, but might suffer from the same limitations regarding reflective surfaces. This might be overcome by adding some more features to the render, but the beauty of NeRF is in its simplicity. I'd actually really like to explore baking meshes as SDFs for other purposes like using boolean modifiers, so this one is my fav. Combining SDF raymarching with NeRF raymarching doesn't seem too terribly complex either.

So, those are my thoughts! It's worth mentioning I have not yet searched for any research papers regarding this subject. It's very likely that other teams are a lot further along on this, and I'm sure we'll be seeing hybrid renderers in the near future, not just compositing like what's available now.