Accepted to ICRA 2025!

Deep Height Decoupling for Precise Vision-based 3D Occupancy Prediction

Yuan Wu^1*, Zhiqiang Yan^1* , Zhengxue Wang¹, Xiang Li², Le Hui³, Jian Yang^1†

^*equal contribution    ^†corresponding author   
¹Nanjing University of Science and Technology    ²Nankai University    ³Northwestern Polytechnical University   

[Paper] [Project Page]

Method

DHD comprises a feature extractor, HeightNet, DepthNet, MGHS, SFA, and predictor. The feature extractor first acquires 2D image feature. Then, DepthNet extracts context feature and depth prediction. HeightNet generates the height map to determine the height value at each pixel. Next, MGHS integrates the output of HeightNet and DepthNet, acquiring height-refined feature and depth-based feature. Finally, the dual features are fed into the SFA to obtain the aggregated feature, which serves as input for the predictor.

Get Started

Installation and Data Preparation

Step1、Prepare environment as that in Install.

Step2、Prepare nuScene and generate pkl file by runing：

python tools/create_data_bevdet.py

The final directory structure for 'data' folder is like

└── data
  └── nuscenes
      ├── v1.0-trainval
      ├── maps  
      ├── sweeps  
      ├── samples
      ├── gts
      ├── bevdetv2-nuscenes_infos_train.pkl 
      └── bevdetv2-nuscenes_infos_val.pkl

Train & Test

# train:
tools/dist_train.sh ${config} ${num_gpu}
# train DHD-S:
tools/dist_train.sh projects/configs/DHD/DHD-S.py 4

# test:
tools/dist_test.sh ${config} ${ckpt} ${num_gpu} --eval mAP
# test DHD-S:
tools/dist_test.sh projects/configs/DHD/DHD-S.py model_weight/DHD-S.pth 4 --eval mAP

Model weights

The pretrained weights in 'ckpts' folder can be found here.

All DHD model weights can be found here.

Experiment

Quantitative comparison

Visual comparison

Acknowledgements

Many thanks to the authors of BEVDet and FlashOcc.

Citation

@article{wu2024deep,
  title={Deep Height Decoupling for Precise Vision-based 3D Occupancy Prediction},
  author={Wu, Yuan and Yan, Zhiqiang and Wang, Zhengxue and Li, Xiang and Hui, Le and Yang, Jian},
  journal={arXiv preprint arXiv:2409.07972},
  year={2024}
}