Semi-stable parts of the Gen3D progress as of Sep 18 #193
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds in a template for the full inference algorithm, and implementations of some of the subsidiary methods. I added `notebooks/bayes3d_paper/tester.ipynb` which runs this partial inference algorithm. I have made several changes to the existing code: - I have changed the interfaces to the pixel kernels slightly, and added docstrings more fully specifying the interfaces the rest of the codebase expects these kernels to satisfy. - I have added utility methods to `PixelsPointsAssociation`. - I have added methods for getting the active pixel kernels to the Image Kernel. - I have renamed the `possible_values` of the discrete kernels to `support` (I thought this was a more standard name - but we can revert this if desired).
This PR includes some additional tests to make sure the behavior of the `logpdf` method is what we expect when it receives invalid RGBD values (`[-1., -1., -1., -1.]`). Since the distribution is defined on the pixel level, it doesn't really expect the observed RGBD to be invalid, so when we use it to vmap over vertices, we need to handle the case where a vertex does not hit the image plane manually. I'm also updating the image kernel slightly to make sure that we can handle those corner cases. (note that this is different from the case where a pixel does not have an associated vertex, which can be handle natively in the pixel distribution) # Test Plan ```bash pytest tests/gen3d/test_pixel_rgbd_kernels.py ```
Disclaimer: I've tested this out in the scratch section of the `tester.ipynb` notebook, but I haven't got the time to write the unit tests for the image kernel or the projection stuff yet. The `sample()` method let us sample RGBD image like this now, which might be useful for debugging purpose:  (note that the sampled depth looks really dark on the actual object itself because our far is set to be very large, so uniform(near, far) can results in depth values much larger than the depth of the object)
Co-authored-by: George Matheos <[email protected]>
Co-authored-by: George Matheos <[email protected]>
@nishadgothoskar this seems to remove the NaN bugs. Inference seems like it works alright, but it does get off eventually -- and I think it isn't honing in closely enough to the right pose. So I think it is a good next step to do as you suggested and add several coarse-to-fine steps.
This adds back in the logq scores in the inference algorithm. It seems to work pretty well! Interestingly, on the dataset in the tester.ipynb notebook, having the logq scores changes the qualitative behavior at frame ~23, when the camera exposure changes. Without the logq scores, the tracking works okay through this period, while with the logq scores, the algorithm loses track of the object. This warrants further debugging, since it probably indicates the math is wrong somewehre. I suspect it may have to do with the fact that we are currently not including scores for pixels that are unobserved. I wonder if this makes the algorithm too likely to want to resample poses that have many points off the screen, since the scoring doesn't realize how much extra randomness this induces. (Put another way, if we include this score, we should lower the P scores of poses with many points off screen, since we will have a log(1/(far - near)) term for each pixel that no point is hitting, and there are more of these for poses where many points are off screen.)
Unfortunately this pushes up the test time for the `tests/gen3d/` directory to 3 mins. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
#180) This PR introduces some utility snippets to load the precomputed YCBV results from FoundationPose so that we can make some comparisons. To fetch the precomputed results, use the `b3d_pull` command. e.g. ```bash python -m b3d.bucket_utils.b3d_pull ``` This should fetch and place the precomputed results under `assets/shared_data_bucket/foundation_pose_tracking_results`. Some examples on how to use this loader can be found in the test file in this PR. It can also be used to make sure that the results are fetched correctly: ```bash pytest tests/gen3d/test_metrics.py ``` I'm going to put together another PR to compute the actual metrics in a bit...
… aggregation (#181) As the title suggests, this PR adds the metrics that are commonly used in 6DoF pose tracking into our repo. Note that currently the metrics are computed using numpy/scipy/sklearn, so they are not compatible with other JAX transformations (e.g. vmap or jit). This is because many of the methods do not have a ready-to-use JAX equivalent, and implementing one ourselves might not be worthwhile given that we likely only need to compute those scores at the very end of the inference. I'm also including an example of using these scoring function in the [test file here](https://github.com/probcomp/b3d/blob/74c2a50205ec6a90855d3271b22edf571373df75/tests/gen3d/test_metrics.py#L54-L81). They can also be triggered by ```bash pytest tests/gen3d/test_metrics.py ``` Check out the earlier PR #180 for the instruction of obtaining the precomputed FoundationPose tracking results.
Usage
```bash
python <pose_tracking_result_dir> <output_dir>
```
This script will load the precomputed pose tracking results on YCB-V
dataset and compute the aggregated ADD and ADD-S metrics per object. It
will print out the result and store them in a CSV format in the output
directory (if not provided, the output directory will be set to the
input directory by default)
Example outputs:
```
ADD-S ADD
002_master_chef_can 0.865135 0.497027
003_cracker_box 0.686422 0.667845
004_sugar_box 0.980075 0.962711
005_tomato_soup_can 0.568405 0.381333
006_mustard_bottle 0.965702 0.534649
007_tuna_fish_can 0.379667 0.255333
008_pudding_box 0.647143 0.549286
009_gelatin_box 0.480714 0.365000
010_potted_meat_can 0.655817 0.510817
011_banana 0.984375 0.962031
019_pitcher_base 0.976282 0.958077
021_bleach_cleanser 0.966097 0.935419
024_bowl 0.768561 0.113258
025_mug 0.540915 0.395854
035_power_drill 0.757128 0.719007
036_wood_block 0.958243 0.901757
037_scissors 0.971098 0.940854
040_large_marker 0.555329 0.400526
051_large_clamp 0.729881 0.433571
052_extra_large_clamp 0.952831 0.308614
061_foam_brick 0.265658 0.148158
```
|
@eightysteele do you know why there is a credential issue arising in CI here? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.