Skip to content

Latest commit

 

History

History
237 lines (169 loc) · 18.2 KB

README.md

File metadata and controls

237 lines (169 loc) · 18.2 KB

Awesome Embodied AI Awesome

A curated list of awesome projects, resources, and research related to Embodied AI and Humanoid Robotics.

Contents

Introduction

Embodied Intelligence refers to AI systems that learn and interact through physical embodiment, often in robotics platforms. This list focuses on projects combining artificial intelligence with physical interaction capabilities.

Scene Understanding

Image Processing

Description Paper Code
SAM Segmentation Paper Code
YOLO-World Open-Vocabulary Detection Paper Code

Point Cloud Processing

Description Paper Code
SAM3D Segmentation Paper Code
PointMixer Understanding Paper Code

Multi-Modal Grounding

Description Paper Code
GPT4V MLM(Image+Language->Language) Paper
Claude3-Opus MLM(Image+Language->Language) Paper
GLaMM Pixel Grounding Paper Code
All-Seeing Pixel Grounding Paper Code
LEO 3D Paper Code

Data Collection

From Video

Description Paper Code
Vid2Robot Paper
RT-Trajectory Paper
MimicPlay Paper Code

Hardware

Description Paper Code
UMI Two-Fingers Paper Code
DexCap Five-Fingers Paper Code
HIRO Hand Hand-over-hand Paper

Generative Simulation

Description Paper Code
MimicGen Paper Code
RoboGen Paper Code

Action Output

Generative Imitation Learning

Description Paper Code
Diffusion Policy Paper Code
ACT Paper Code

Affordance Map

Description Paper Code
CLIPort Pick&Place Paper Code
Robo-Affordances Contact&Post-contact trajectories Paper Code
Robo-ABC Paper Code
Where2Explore Few shot learning from semantic similarity Paper
Move as You Say Affordance to motion from diffusion model Paper
AffordanceLLM Grounding affordance with LLM Paper
Environment-aware Affordance Paper
OpenAD Open-Voc Affordance Detection Paper Code
RLAfford End-to-End affordance learning Paper
General Flow Collect affordance from video Paper Code
PreAffordance Pre-grasping planning Paper
ScenFun3d Fine-grained functionality Paper Code

Question & Answer from LLM

Description Paper Code
COPA Paper
ManipLLM Paper
ManipVQA Paper Code

Language Corrections

Description Paper Code
OLAF Paper
YAYRobot Paper Code

Planning from LLM

Description Paper Code
SayCan API Level Paper Code
VILA Prompt Level Paper

Open Source Projects

Simulation Environments

  • Genesis - Genesis is a physics platform designed for general-purpose Robotics/Embodied AI/Physical AI applications. It is simultaneously multiple things: A universal physics engine re-built from the ground up, capable of simulating a wide range of materials and physical phenomena.; A lightweight, ultra-fast, pythonic, and user-friendly robotics simulation platform. [⭐22747]
  • Habitat Sim - A flexible, high-performance 3D simulator for Embodied AI research. Habitat-Sim is typically used with Habitat-Lab, a modular high-level library for end-to-end experiments in embodied AI. [⭐5000]
  • Isaac Sim - NVIDIA Isaac Sim™ is a reference application built on NVIDIA Omniverse that enables developers to simulate and test AI-driven robotics solutions in physically based virtual environments.
  • MuJoCo - Physics engine for robotics, biomechanics, and graphics
  • PyBullet - Physics simulation for robotics and machine learning
  • Gazebo - Robot simulation environment
  • SAPIEN - A SimulAted Part-based Interactive ENvironment
  • Webots - Webots is an open source and multi-platform desktop application used to simulate robots. It provides a complete development environment to model, program and simulate robots.

Perceptions

Teleops & Retargeting

  • AnyTeleop - A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System
  • Dex Retargeting - Various retargeting optimizers to translate human hand motion to robot hand motion.

Feedback Generation

  • ChatTTS - A generative speech model for daily dialogue
  • Real3D-Portrait - One-shot Realistic 3D Talking Portrait Synthesis
  • Hallo2 - Long-Duration and High-Resolution Audio-driven Portrait Image Animation

Research Frameworks

Learning Frameworks

  • MoveIt - Motion planning framework
  • OMPL - Open Motion Planning Library
  • Drake - Planning and control toolkit
  • GR-1 - ByteDance Research: Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
  • Universal Manipulation Interface(UMI)
  • Meta Motivo - A first-of-its-kind behavioral foundation model to control a virtual physics-based humanoid agent for a wide range of whole-body tasks.
  • OpenVLA - OpenVLA sets a new state of the art for generalist robot manipulation policies. It supports controlling multiple robots out of the box and can be quickly adapted to new robot setups via parameter-efficient fine-tuning.

Datasets

Robot Learning Datasets

Human Motion Datasets

Benchmarks

  • CALVIN - CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
  • DexYCB - A Benchmark for Capturing Hand Grasping of Objects

Companies & Robots

Research Labs

Companies

Humanoid Robotics

  • Atlas - Advanced humanoid robot by Boston Dynamics
  • Digit - Bipedal robot by Agility Robotics
  • Tesla Optimus - Tesla's humanoid robot project
  • Figure 01 - General purpose humanoid robot
  • Phoenix - Open-source humanoid robot design
  • Fourier GR-2 - Fourier General Purpose Humanoid Robotics

Research Papers

Researchers

Yuke Zhu Yuzhe Qin Jim Fan

Community

Conferences

  • RSS - Robotics: Science and Systems
  • ICRA - International Conference on Robotics and Automation
  • CoRL - Conference on Robot Learning
  • IROS - International Conference on Intelligent Robots and Systems

Forums & Discussion

Contributing

Please read the contribution guidelines before making a contribution.

License

This project is licensed under the MIT License - see the LICENSE file for details.