This is the code for Two photos to find them all: PixelGrouping by Tim van Gelder, Leon Hulshof, Thomas Komen, Jerrijn Mandjes and Max de Redelijkheid.
In this method, pixelSplat (code) and Gaussian Grouping (code) are combined to produce an algorithm capable of 3D object segmentation as proposed in Gaussian Grouping, while retaining the improvements upon the 3D scene reconstruction as proposed in pixelSplat.
This version of the codebase matches the original code used for producing the results presented in the paper.
Since the architecture of PixelGrouping is based on the pixelSplat method, the installation and troubleshooting follows the same steps as presented for their method, restated here for convenience:
To get started, create a virtual environment using Python 3.10+:
python3.10 -m venv venv
source venv/bin/activate
# Install these first! Also, make sure you have python3.11-dev installed if using Ubuntu.
pip install wheel torch torchvision torchaudio
pip install -r requirements.txt
pip install submodules/diff-gaussian-rasterizationIf your system does not use CUDA 12.1 by default, see the troubleshooting tips below.
Troubleshooting
The Gaussian splatting CUDA code (diff-gaussian-rasterization) must be compiled using the same version of CUDA that PyTorch was compiled with. As of December 2023, the version of PyTorch you get when doing pip install torch was built using CUDA 12.1. If your system does not use CUDA 12.1 by default, you can try the following:
- Install a version of PyTorch that was built using your CUDA version. For example, to get PyTorch with CUDA 11.8, use the following command (more details here):
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118- Install CUDA Toolkit 12.1 on your system. One approach (try this at your own risk!) is to install a second CUDA Toolkit version using the
runfile (local)option here. When you run the installer, disable the options that install GPU drivers and update the default CUDA symlinks. If you do this, you can point your system to CUDA 12.1 during installation as follows:
LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64 pip install -r requirements.txt
# If everything else was installed but you're missing diff-gaussian-rasterization, do:
LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64 pip install git+https://github.com/dcharatan/diff-gaussian-rasterization-modifiedThe checkpoints for this model will be available soon