Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring gen3d features into the IPE model #209

Open
wants to merge 120 commits into
base: gen3d--stable-20240918
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
30e1443
fixed abother demo
HaoliangWang Aug 26, 2024
5e0a903
trying tests on physion
HaoliangWang Aug 27, 2024
f2d0ea9
trying in conda rather than pixi
HaoliangWang Aug 27, 2024
36b2c9f
ignore all files in .pixi
HaoliangWang Aug 27, 2024
f329e0c
test whether physion mesh has color
HaoliangWang Aug 27, 2024
7953d77
misc
HaoliangWang Aug 28, 2024
ccaeb49
wierd failure
HaoliangWang Aug 28, 2024
a3db16a
working example
HaoliangWang Aug 29, 2024
875ee1f
loading the correct meshes
HaoliangWang Aug 29, 2024
23632d8
xx
HaoliangWang Aug 29, 2024
ef8fdd5
debugging works
HaoliangWang Aug 30, 2024
d756c86
more debugging on vis
HaoliangWang Aug 31, 2024
e8b738e
I think the poses are correct now
HaoliangWang Sep 3, 2024
ac51fca
center of mass -> center of bounding box
HaoliangWang Sep 3, 2024
955f4af
more debugging on vis
HaoliangWang Sep 3, 2024
5e55d50
now using meshes from obj files instead of hdf5
HaoliangWang Sep 4, 2024
4dab43d
initialize trace working now!
HaoliangWang Sep 4, 2024
df2f845
testing tracking
HaoliangWang Sep 4, 2024
33e7251
don't push traces to git
HaoliangWang Sep 4, 2024
a8f101e
delted example trace
HaoliangWang Sep 4, 2024
eabbfaf
tracking somewhat works
HaoliangWang Sep 5, 2024
8cba7cf
playing with different parameters
HaoliangWang Sep 5, 2024
2e8689e
renaming
HaoliangWang Sep 5, 2024
88202bf
some progress on point cloud and number of objects
HaoliangWang Sep 5, 2024
6bcbda6
initialize using poses from point cloud
HaoliangWang Sep 6, 2024
fd10ee7
tracking works for initialization from point cloud but assumes the ob…
HaoliangWang Sep 6, 2024
bb9df30
misc
HaoliangWang Sep 7, 2024
87c6feb
rgb likelihood somewhat working
HaoliangWang Sep 7, 2024
6991982
Merge branch 'main' into haoliang_ipe_b3d
HaoliangWang Sep 8, 2024
f2e87ff
rgb likelihood working for example dominoes trial
HaoliangWang Sep 8, 2024
20fdcb5
beforedoing scale inference
HaoliangWang Sep 10, 2024
1eec454
still wotking on scale inference
HaoliangWang Sep 10, 2024
43d6013
more utils to get the interpenetration between meshes
HaoliangWang Sep 10, 2024
432d143
size inference somewhat working
HaoliangWang Sep 10, 2024
b340205
check interpenetration somewhat works
HaoliangWang Sep 13, 2024
416e8f3
to big to work, otherwise fine
HaoliangWang Sep 14, 2024
bde5f7a
max capacity
HaoliangWang Sep 14, 2024
f618747
obj id and scale inference
HaoliangWang Sep 16, 2024
4ff90aa
trying Arijit's way of improving efficiency
HaoliangWang Sep 19, 2024
d62d3d4
from Nishad
HaoliangWang Sep 19, 2024
b2b5785
obj id inference works
HaoliangWang Sep 20, 2024
1a59c83
before transfering files
HaoliangWang Oct 15, 2024
8722c39
linking joint working
HaoliangWang Oct 16, 2024
7aa3873
working
HaoliangWang Oct 17, 2024
282d3ed
the scores are not aligned
HaoliangWang Oct 18, 2024
465b556
misc
HaoliangWang Oct 18, 2024
8792a9d
more cutoff from discrete distribution before sampling
HaoliangWang Oct 18, 2024
005896d
wip
HaoliangWang Oct 18, 2024
b1b4678
changed gridding schedule
HaoliangWang Oct 19, 2024
b97a8b9
interp test
HaoliangWang Oct 20, 2024
7dd8269
misc
HaoliangWang Oct 23, 2024
0902c4f
y above 0 in json file
HaoliangWang Oct 25, 2024
6ab28f3
trying out combinatorial
HaoliangWang Oct 29, 2024
02c6029
combinatorial inference not working
HaoliangWang Oct 31, 2024
3eec774
made save and load trace easy
HaoliangWang Oct 31, 2024
f950d21
make min example that doesn't work
HaoliangWang Nov 1, 2024
d143048
implemented scaling composite objects
HaoliangWang Nov 6, 2024
be8df17
reorganize files
HaoliangWang Nov 8, 2024
f9d0f9b
model is now working for composite objects
HaoliangWang Nov 26, 2024
594aac2
delete useless files
HaoliangWang Nov 26, 2024
ceb8d88
misc
HaoliangWang Nov 28, 2024
3fb168d
misc
HaoliangWang Nov 28, 2024
267fbae
misc
HaoliangWang Nov 29, 2024
2fc0b97
misc
HaoliangWang Nov 29, 2024
43311eb
misc
HaoliangWang Nov 29, 2024
a2068e6
misc
HaoliangWang Nov 30, 2024
17c4454
misc
HaoliangWang Nov 30, 2024
f6ba0a4
func to detect scale of composite objs from depth info
HaoliangWang Dec 1, 2024
3fc2e21
patch for missing ids
HaoliangWang Dec 1, 2024
b4e5918
preprosess scipt in progress
HaoliangWang Dec 2, 2024
51d56a1
compute new velocity wip
HaoliangWang Dec 2, 2024
cc3acd3
reorganize
HaoliangWang Dec 3, 2024
4b7b770
added floor penetration and initial attempt on adding a dynamics HMM …
HaoliangWang Dec 3, 2024
31fb2e8
latest commit
HaoliangWang Dec 4, 2024
ebefd43
for debug purposes
HaoliangWang Dec 4, 2024
13c9c36
misc
HaoliangWang Dec 4, 2024
11b2ac0
ready to run estimate pose and scale
HaoliangWang Dec 4, 2024
f138f86
wip on tracking, finished b3d substitute for FP
HaoliangWang Dec 5, 2024
580824f
wip for b3d tracking
HaoliangWang Dec 5, 2024
ca9a938
tracking can sucessfully run
HaoliangWang Dec 5, 2024
924dc32
debugged all_areas
HaoliangWang Dec 5, 2024
d812532
added inferring scale for all frames
HaoliangWang Dec 6, 2024
486fda5
debugged and is running
HaoliangWang Dec 6, 2024
eb88ef7
merged two files
HaoliangWang Dec 6, 2024
eaa0eb1
wip
HaoliangWang Dec 7, 2024
3ba6200
wip
HaoliangWang Dec 10, 2024
a62c36b
correct implementation of velocity calculation and match noise parame…
HaoliangWang Dec 11, 2024
5100cd2
make animations
HaoliangWang Dec 12, 2024
a014533
most up-to-date for Josh's neurips presentation on 12/13/2024
HaoliangWang Dec 15, 2024
b575edb
use hmm for ipe model (#207)
HaoliangWang Dec 16, 2024
6265b5c
adapting hmm structure to b3d, haven't run yet
HaoliangWang Dec 17, 2024
58189a4
more refactoring before debugging
HaoliangWang Dec 17, 2024
b3f9f9d
debugging
HaoliangWang Dec 17, 2024
8a9f979
debugging
HaoliangWang Dec 17, 2024
0c30c19
refactor to debug
HaoliangWang Dec 17, 2024
1e520af
make pr
HaoliangWang Dec 17, 2024
8fa8a31
changing jax.lax.map to jax.vmap works, haven't visualized yet. n_pos…
HaoliangWang Dec 18, 2024
596c54c
env fixing
HaoliangWang Dec 18, 2024
30a8217
working!
HaoliangWang Dec 18, 2024
6040d68
minor
HaoliangWang Dec 19, 2024
d55e821
swap in previous pose
HaoliangWang Dec 20, 2024
667b300
minor
HaoliangWang Dec 20, 2024
def6e4c
added background occluders and distractors
HaoliangWang Dec 24, 2024
5fb06c7
running full set of collide
HaoliangWang Jan 2, 2025
2660c44
further debugging
HaoliangWang Jan 2, 2025
72d3276
trying to fix OOM issue but failed
HaoliangWang Jan 3, 2025
bfcf74e
run by script
HaoliangWang Jan 4, 2025
91d9e2d
approximate vmf with gaussian
HaoliangWang Jan 6, 2025
03ee3e2
speeded by replacing vmf with gaussian, and visualize across scripts
HaoliangWang Jan 6, 2025
b19e2ca
logging both sampled velocities and optim velocities, also profiling …
HaoliangWang Jan 8, 2025
474e0f7
inference_step loop faster but c2f still recompiles
HaoliangWang Jan 8, 2025
8e78777
faster now
HaoliangWang Jan 9, 2025
3765561
floor penetration filtering, next todo is gaussian likelihood
HaoliangWang Jan 10, 2025
e7c3fd4
swap in previous pose at best location in array (#211)
HaoliangWang Jan 11, 2025
47855d3
gaussian blur likelihood works, but tpp memory costly, abandon
HaoliangWang Jan 11, 2025
008df3a
increase penetration penalty seems to work
HaoliangWang Jan 11, 2025
ebdd379
faster
HaoliangWang Jan 13, 2025
afc8314
attempt to distinguish sleeping and active objects
HaoliangWang Jan 13, 2025
e5e8f99
revert back
HaoliangWang Jan 13, 2025
c075ad2
continue to revert back
HaoliangWang Jan 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
misc
HaoliangWang committed Aug 28, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 7953d774e29d8636810fd7d60cb659070562b6df
2 changes: 1 addition & 1 deletion demo.py
Original file line number Diff line number Diff line change
@@ -59,7 +59,7 @@ def test_demo():
# Arguments of the generative model.
# These control the inlier / outlier decision boundary for color error and depth error.
color_error, depth_error = (60.0, 0.01)
inlier_score, outlier_prob = (5.0, 0.00001)
inlier_score, outlier_prob = (5.0, 1.0)
color_multiplier, depth_multiplier = (10000.0, 500.0)
model_args = bayes3d.ModelArgs(
color_error,
411 changes: 411 additions & 0 deletions place_disk.ipynb

Large diffs are not rendered by default.

110 changes: 87 additions & 23 deletions read_hdf5.ipynb
Original file line number Diff line number Diff line change
@@ -2,9 +2,27 @@
"cells": [
{
"cell_type": "code",
"execution_count": 31,
"execution_count": 1,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Jax plugin configuration error: Exception when calling jax_plugins.xla_cuda12.initialize()\n",
"Traceback (most recent call last):\n",
" File \"/opt/conda/envs/ipe_genjax/lib/python3.11/site-packages/jax/_src/xla_bridge.py\", line 605, in discover_pjrt_plugins\n",
" plugin_module.initialize()\n",
" File \"/opt/conda/envs/ipe_genjax/lib/python3.11/site-packages/jax_plugins/xla_cuda12/__init__.py\", line 78, in initialize\n",
" options = xla_client.generate_pjrt_gpu_plugin_options()\n",
" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
" File \"/opt/conda/envs/ipe_genjax/lib/python3.11/site-packages/jaxlib/xla_client.py\", line 229, in generate_pjrt_gpu_plugin_options\n",
" raise ValueError(\n",
"ValueError: XLA_PYTHON_CLIENT_ALLOCATOR env var must be \"default\", \"platform\", \"bfc\", or \"cuda_async\", got \"“platform”\"\n",
"An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.\n"
]
}
],
"source": [
"import os\n",
"import h5py\n",
@@ -28,7 +46,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 3,
"metadata": {},
"outputs": [
{
@@ -202,7 +220,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 4,
"metadata": {},
"outputs": [
{
@@ -211,7 +229,7 @@
"(145, 1024, 1024)"
]
},
"execution_count": 11,
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
@@ -729,9 +747,21 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 5,
"metadata": {},
"outputs": [],
"outputs": [
{
"ename": "NameError",
"evalue": "name 'img_arr' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[5], line 4\u001b[0m\n\u001b[1;32m 2\u001b[0m near_plane\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0.1\u001b[39m\n\u001b[1;32m 3\u001b[0m far_plane\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m100\u001b[39m\n\u001b[0;32m----> 4\u001b[0m height, width \u001b[38;5;241m=\u001b[39m \u001b[43mimg_arr\u001b[49m\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m1\u001b[39m], img_arr\u001b[38;5;241m.\u001b[39mshape[\u001b[38;5;241m2\u001b[39m]\n\u001b[1;32m 6\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m h5py\u001b[38;5;241m.\u001b[39mFile(hdf5_file_path, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m 7\u001b[0m camera_pose \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marray(f[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mazimuth\u001b[39m\u001b[38;5;124m'\u001b[39m][\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcam_0\u001b[39m\u001b[38;5;124m'\u001b[39m])\n",
"\u001b[0;31mNameError\u001b[0m: name 'img_arr' is not defined"
]
}
],
"source": [
"vfov=55 \n",
"near_plane=0.1\n",
@@ -776,20 +806,19 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.48848364, 0.87257305, 0. ],\n",
" [-0.87257305, 0.48848364, 0. ],\n",
" [ 0. , 0. , 1. ]])"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
"ename": "NameError",
"evalue": "name 'camera_pose' is not defined",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[6], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mcamera_pose\u001b[49m\n",
"\u001b[0;31mNameError\u001b[0m: name 'camera_pose' is not defined"
]
}
],
"source": [
@@ -818,7 +847,7 @@
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": 7,
"metadata": {},
"outputs": [
{
@@ -839,7 +868,7 @@
" Mesh(vertices=(38,), faces=(72,), vertex_attributes=(38,))]"
]
},
"execution_count": 23,
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
@@ -917,9 +946,41 @@
},
{
"cell_type": "code",
"execution_count": 32,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bowl.obj\n",
"<trimesh.Trimesh(vertices.shape=(192, 3), faces.shape=(380, 3), name=`bowl.obj`)>\n",
"cone.obj\n",
"<trimesh.Trimesh(vertices.shape=(191, 3), faces.shape=(378, 3), name=`cone.obj`)>\n",
"cube.obj\n",
"<trimesh.Trimesh(vertices.shape=(56, 3), faces.shape=(108, 3), name=`cube.obj`)>\n",
"cylinder.obj\n",
"<trimesh.Trimesh(vertices.shape=(96, 3), faces.shape=(188, 3), name=`cylinder.obj`)>\n",
"dumbbell.obj\n",
"<trimesh.Trimesh(vertices.shape=(160, 3), faces.shape=(316, 3), name=`dumbbell.obj`)>\n",
"octahedron.obj\n",
"<trimesh.Trimesh(vertices.shape=(38, 3), faces.shape=(72, 3), name=`octahedron.obj`)>\n",
"pentagon.obj\n",
"<trimesh.Trimesh(vertices.shape=(74, 3), faces.shape=(144, 3), name=`pentagon.obj`)>\n",
"pipe.obj\n",
"<trimesh.Trimesh(vertices.shape=(196, 3), faces.shape=(392, 3), name=`pipe.obj`)>\n",
"platonic.obj\n",
"<trimesh.Trimesh(vertices.shape=(42, 3), faces.shape=(80, 3), name=`platonic.obj`)>\n",
"pyramid.obj\n",
"<trimesh.Trimesh(vertices.shape=(29, 3), faces.shape=(54, 3), name=`pyramid.obj`)>\n",
"sphere.obj\n",
"<trimesh.Trimesh(vertices.shape=(162, 3), faces.shape=(320, 3), name=`sphere.obj`)>\n",
"torus.obj\n",
"<trimesh.Trimesh(vertices.shape=(128, 3), faces.shape=(256, 3), name=`torus.obj`)>\n",
"triangular_prism.obj\n",
"<trimesh.Trimesh(vertices.shape=(38, 3), faces.shape=(72, 3), name=`triangular_prism.obj`)>\n"
]
},
{
"data": {
"text/plain": [
@@ -938,7 +999,7 @@
" <trimesh.Trimesh(vertices.shape=(38, 3), faces.shape=(72, 3), name=`triangular_prism.obj`)>]"
]
},
"execution_count": 32,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
@@ -948,7 +1009,10 @@
"for dirpath, dnames, fnames in os.walk(mesh_path):\n",
" for f in sorted(fnames):\n",
" if f.endswith(\".obj\"):\n",
" trimeshes.append(trimesh.load_mesh(os.path.join(dirpath, f), process=False, validate=False))\n",
" mesh = trimesh.load_mesh(os.path.join(dirpath, f), process=False, validate=False)\n",
" print(f)\n",
" print(mesh)\n",
" trimeshes.append(mesh)\n",
"trimeshes\n"
]
},