Welcome to the Caltech Aerial RGB-Thermal dataset repository! This repository hosts the first publicly available dataset tailored for aerial robotics operating in diverse natural landscapes across the continental United States. Our dataset comprises synchronized RGB, thermal, GPS, and IMU data, providing a comprehensive resource for researchers and practitioners in the field. For further details, please see our paper.
Key Features:
-
🏞️ Diverse Terrains: Our dataset captures a wide range of terrains, including rivers, lakes, coastlines, deserts, and forests, ensuring robustness to various environmental conditions.
-
🖼️ Semantic Segmentation Annotations: We provide semantic segmentation annotations for 10 classes commonly encountered in natural settings, facilitating the development of perception algorithms resilient to adverse weather and nighttime conditions.
-
📊 Benchmarking: We introduce new benchmarks for thermal and RGB-T semantic segmentation, RGB-T image translation, and visual-inertial odometry, presenting challenging tasks for evaluation and comparison.
-
🌟 Challenging Domain Shifts: We provide splits for the data based on time and geography, enabling studies of geographic domain adaptation. Temporal splits enable new studies into better handling thermal inversion (see image below).
- The dataset can be downloaded here: https://data.caltech.edu/records/cks6g-ps927.
- The labeled subset can be directly downloaded.
- The raw data must be downloaded from the S3 bucket (follow instructions in the link).
- Raw data is located at
onr-processed - Annotated thermal data is located here under
labeled_thermal_singles. Images are stored according to their capture location and trajectory id following the pattern:
labeled_thermal_singles/CAPTURE_PLACE/TRAJECTORY_ID/{masks|thermal8|thermal16}
- Annotated paired RGB-T data is located under
labeled_rgbt_pairs.
| Index | Class | Hex Code | Color |
|---|---|---|---|
| 0 | unknown | #FF2400 |  |
| 1 | background | #000000 |  |
| 2 | bare ground | #F2D8C4 |  |
| 3 | boulders / rocky terrain | #594636 |  |
| 4 | human-made structures | #A6A6A6 |  |
| 5 | road | #52595A |  |
| 6 | shrubs | #9BE600 |  |
| 7 | trees | #008A35 |  |
| 8 | sky | #00D8F5 |  |
| 9 | water | #0D7FFC |  |
| 10 | vehicles | #FFF900 |  |
| 11 | person | #FE00AA |  |
We typically ignore unknown and background classes.
Thermal data splits are located under caltech_aerial_thermal_dataset/splits/thermal_splits and are subdivided into general (random), geographic (state-based and terrain-based), and temporal (sunrise vs. day vs. night) splits.
The split for RGB-T paired imagery is created from the general (random) split and is available under caltech_aerial_thermal_dataset/splits/rgbt_splits. There are less samples in this split due to sensor failures during some flights.
COMING SOON!
The dataset is provided primarily as ROS1 rosbags. As rosbags may not be a convenient filetype for all users, extraction scripts have been provided to extract csv and jpg/png files. Please see the readme at /extract_data/rosbag/README.md for more information.
Where available, the raw onboard logs from Ardupilot have been included in the dataset. As not all users will be able to read these directly, extraction scripts have been included as part of this repository. Please see the readme at /extract_data/ardupilot/README.md for more information.
COMING SOON!
COMING SOON!
All images with semantic segmentation annotations are already rectified. If you want to rectify all images, some code is provided to assist you in this:
To rectify raw imagery, use the MonoRectifier class provided in caltech_aerial_thermal_dataset/utils/rectifier.py. Use the appropriate calibration files provided under calibrations/*.yaml.
To stereo rectify a trajectory, follow the example bash scripts here:
caltech_aerial_thermal_dataset/bash/bulk_stereo_rectify.sh
caltech_aerial_thermal_dataset/bash/stereo_rectify.sh
and check out the command-line arguments listed in stereo_rectify.py
If you find issues with this repo, or have code to contribute, please submit and issue and/or a PR above.
The dataset is for non-commercial use only and is licensed under the terms of the Creative Commons Attribution ShareAlike 4.0 license.
If you found our work useful, please cite
@inproceedings{lee2025caltech,
title={Caltech Aerial RGB-Thermal Dataset in the Wild},
author={Lee, Connor and Anderson, Matthew and Ranganathan, Nikhil and Zuo, Xingxing and Do, Kevin and Gkioxari, Georgia and Chung, Soon-Jo},
booktitle={European Conference on Computer Vision},
pages={236--256},
year={2025},
organization={Springer}
}