Awesome World Models

Table of Contents

• Projects
  • Interactive World Models
  • World Models based on Gaussian Splats & NeRFs
  • World Models for Robotics and Autonomous Driving
  • Auto-regressive Video
• Datasets
• Articles

Projects

Interactive World Models

• World Models: World Models [Paper] [Website] (NeurIPS, 2018.03)

• IRIS: Transformers are Sample-Efficient World Models [Paper] [Code] (ICLR, 2023.02)

• Genie: Generative Interactive Environments [Paper] [Website] (arXiv, 2024.02)

  • Third-party open source implementation: insait-institute/GenieRedux

• DIAMOND: Diffusion for World Modeling: Visual Details Matter in Atari [Paper] [Code] [NeurIPS, 2024.05]

• GameNGen: Diffusion Models Are Real-Time Game Engines [Paper] [Website] (arXiv, 2024.08)

  • Third-party open source implementation: arnaudstiegler/gameNgen-repro

• Oasis: A Universe in a Transformer [Website] [Code] (2024.10)

• Lucid v1: A world model that does go brrr on consumer hardware [Website] [Code] (2024.10)

• GameGen-X: Interactive Open-world Game Video Generation [Paper] [Website] [Code] (arXiv, 2024.11)

• The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control [Paper] [Website] (arXiv, 2024.12)

• Genie 2: A large-scale foundation world model [Website] (2024.12)

• Playable Game Generation [Paper] [Code] (2024.12)

• GameFactory: A generalizable world model that learns from a small-scale dataset of Minecraft game videos [Paper] [Website] (2025.01)

World Models based on Gaussian Splats & NeRFs

• World Labs: Generating Worlds [Website] (2024.12)

• GenEx: Generating an Explorable World [Paper] [Website] (arXiv, 2024.12)

World Models for Robotics and Autonomous Driving

• 3D-VLA: A 3D Vision-Language-Action Generative World Model [Paper] [Website] (arXiv, 2024.03)

• GAIA-1: A Generative World Model for Autonomous Driving [Paper] (arXiv, 2023.09)

• Navigation World Models: Navigation World Models [Paper] [Website] (arXiv, 2024.12)

Auto-regressive Video

• ART•V: Auto-Regressive Text-to-Video Generation with Diffusion Models [Paper] [Website] (2023.11)

Datasets

• Counter-Strike Deathmatch with Large-Scale Behavioural Cloning: [Paper] [Code] (NeurIPS, 2021.04)

• MineRL: Towards AI in Minecraft [Website] [Code] (2020)

• GameFactory-Dataset: 70 hours of Minecraft gameplay video with action annotations [Dataset] [Paper] [Website] (2025.01)

Articles

• Introducing General World Models (Runway) (2024.09)

• World Models Survey: Understanding World or Predicting Future? A Comprehensive Survey of World Models [Paper] (arXiv, 2024.11)

• Awesome Autoregressive Models in Vision