Question of viewer

[Sun-Jing-Kang]

How to view the trained model? Thanks!

I found the Scaffold-GS-viewer can't be used, the origin GS-viewer view like full black scene.

[YihangChen-ee]

Hi Sun-Jing-Kang,

HAC follows the base structure of Scaffol-GS, but there some minor modifications in the 3D model, which leads to the failure of viewing using the Scaffold-GS-viewer.

You can have a read on how we decode the model and render the testing images using the decoded model, and modify the Scaffold-GS-viewer or our HAC model to fit each other.

We will consider to directly support the Scaffold-GS-viewer in the future.

:)

[arcman7]

Does this HAC project support real time rendering?

[YihangChen-ee]

Yes. HAC essentially renders images under the 3DGS pipeline, thus is able to achieve real-time rendering. Please refer to the paragraph Inference time in our paper.

[arcman7]

Cool. If I am understanding you correctly, these bitstreams can be rendered by any rasterization method that is faithful the to the 3DGS rasterization implementation? Meaning SIBR or something similar to it can be used to render the output bitstreams from HAC?

Edit: I read the code, and I see here:

https://github.com/YihangChen-ee/HAC/blob/c23a4dc85edf9e5b70a7b735df284143cd477994/gaussian_renderer/__init__.py#L172C8-L172C59

and here:

HAC/gaussian_renderer/__init__.py

Line 186 in c23a4dc

xyz, color, opacity, scaling, rot, time_sub = generate_neural_gaussians(viewpoint_camera, pc, visible_mask, is_training=is_training, step=step)

that before gaussian rasterization can occur, you need to call generate_neural_gaussians. In practice does this slow down the fps of the rendering? If generate_neural_gaussians is sufficiently fast then it won't matter much, but it's hard for me to tell right now.

[YihangChen-ee]

Hi, the bitstreams are not directly used for rendering though. Bitstreams are used to decode the 3DGS model (more precisely, Scaffold-GS model in our method), ie, the locations and the attributes. Note this decoding process only need to be run once to obtain the 3DGS model before the rendering process. After the model is decoded and obtained, the rendering process (ie, the rasterization process) is totally the same to the Scaffold-GS, which is real-time.

[arcman7]

Oh, very nice! I did not realize that. I'm excited to try this out :)

[smart4654154]

Yes. HAC essentially renders images under the 3DGS pipeline, thus is able to achieve real-time rendering. Please refer to the paragraph Inference time in our paper.

Is there a real-time viewing tool available now? Similar to SIBR in 3DGS

[YihangChen-ee]

Yes. HAC essentially renders images under the 3DGS pipeline, thus is able to achieve real-time rendering. Please refer to the paragraph Inference time in our paper.

Is there a real-time viewing tool available now? Similar to SIBR in 3DGS

You could use the real-time viewing tool of Scaffold-GS since we build the HAC framework based on Scaffold-GS. There are some minimal differences though, please remember to change the attribute scale to the version before activation in the decoded weights. (You could refer to the variable name decoded_version in our code to find these differences.)