[pcl::PLYWriter::writeASCII] Input point cloud has no data! On Jetson AGX Orin

[vsevolod-misiul]

Hi,
I'm encountering an issue while running BundleSDF on the Jetson AGX Orin on aarch64. I’ve followed the steps in your tutorial to run the example with the milk dataset. Here’s a detailed description of my setup and the problem:

• I modified the Dockerfile to make it compatible with the Jetson platform.
• Built the project using build.sh.
• Downloaded and placed the following files under the paths specified in the tutorial: XMem-s012.pth, outdoor_ds.ckpt, HO3D dataset, Milk dataset (as per the example).
• Updated the HO3D_ROOT path accordingly.

Problem:
When I run the script, the following logs appear, indicating an issue with processing frames:

[bundlesdf.py] Waiting for GUI
[bundlesdf.py] percentile denoise start
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 1668813025164826994
[bundlesdf.py] process frame 1668813025164826994
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
Segmentation fault (core dumped)

I added debug prints to bundlesdf.py and found that the script stops at the following lines in the process_new_frame function:

def process_new_frame(self, frame):
....
if frame._id==0 and np.abs(np.array(frame._pose_in_model)-np.eye(4)).max()<=1e-4:
--->   frame.setNewInitCoordinate()

[wenbowen123]

@vsevolod-misiul did you check your depth range as mentioned in readme?

[vsevolod-misiul]

@wenbowen123 I ran BundleSDF on the milk dataset provided in the repository, so depth range should be correct. During debugging, I found that the issue is related to the depth_ matrix in the Frame object. In the Frame::processDepth method, some transformations are applied to _depth_gpu. However, I noticed that these transformations have no effect. As a result, after all the transformations, such as:

  CUDAImageUtil::erodeDepthMap(depth_tmp_gpu, _depth_gpu, erode_radius, _W,_H, erode_diff, erode_ratio, zfar);
  CUDAImageUtil::gaussFilterDepthMap(_depth_gpu, depth_tmp_gpu, bf_radius, sigma_D, sigma_R, _W, _H, zfar);
  CUDAImageUtil::gaussFilterDepthMap(depth_tmp_gpu, _depth_gpu, bf_radius, sigma_D, sigma_R, _W, _H, zfar);

depth_tmp_gpu ends up filled with zeros. The main difference between your Dockerfile and my setup is that I’m running it with CUDA 11.7.1 on the board. Could this difference be the cause of the problem?

[vsevolod-misiul]

UPD:
I think nothing from the CUDAImageUtil namespace is working because CUDAImageUtil::convertDepthFloatToCameraSpaceFloat4 and CUDAImageUtil::computeNormals also have no effect. After these functions are called, both xyz_map and normals are filled with zeros.

[wenbowen123]

I remember in some older issues, there were indeed reporting of this cuda version mismatch resulting in depth all zeros.

[vsevolod-misiul]

Could you please explain why you use these specific library versions in the Dockerfile? Can they be updated to newer versions? I’m particularly interested in whether I can use newer versions of OpenCV and PyTorch PyAudio and PyVision.

[wenbowen123]

there is no particular reasons. You can upgrade, though you need to make sure the new versions are compatible with each other.