Skip to content

Latest commit

 

History

History
134 lines (100 loc) · 6.65 KB

README.md

File metadata and controls

134 lines (100 loc) · 6.65 KB

TSformer-VO: an end-to-end Transformer-based model for monocular visual odometry

IEEE arXiv License: MIT

Official repository of "Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach"

Abstract

Estimating the camera’s pose given images from a single camera is a traditional task in mobile robots and autonomous vehicles. This problem is called monocular visual odometry and often relies on geometric approaches that require considerable engineering effort for a specific scenario. Deep learning methods have been shown to be generalizable after proper training and with a large amount of available data. Transformer-based architectures have dominated the state-of-the-art in natural language processing and computer vision tasks, such as image and video understanding. In this work, we deal with the monocular visual odometry as a video understanding task to estimate the 6 degrees of freedom of a camera’s pose. We contribute by presenting the TSformer-VO model based on spatio-temporal self-attention mechanisms to extract features from clips and estimate the motions in an end-to-end manner. Our approach achieved competitive state-of-the-art performance compared with geometry-based and deep learning-based methods on the KITTI visual odometry dataset, outperforming the DeepVO implementation highly accepted in the visual odometry community.

Contents

  1. Dataset
  2. Pre-trained models
  3. Setup
  4. Usage
  5. Evaluation

1. Dataset

Download the KITTI odometry dataset (grayscale).

In this work, we use the .jpg format. You can convert the dataset to .jpg format with png_to_jpg.py.

Create a simbolic link (Windows) or a softlink (Linux) to the dataset in the dataset folder:

  • On Windows: mklink /D <path_to_your_project>\TSformer-VO\data <path_to_your_downloaded_data>
  • On Linux: ln -s <path_to_your_downloaded_data> <path_to_your_project>/TSformer-VO/data

The data structure should be as follows:

|---TSformer-VO
    |---data
        |---sequences_jpg
            |---00
                |---image_0
                    |---000000.png
                    |---000001.png
                    |---...
                |---image_1
                    |...
                |---image_2
                    |---...
                |---image_3
                    |---...
            |---01
            |---...
		|---poses
			|---00.txt
			|---01.txt
			|---...

2. Pre-trained models

Here you find the checkpoints of our trained-models. The architectures vary according to the number of frames (Nf) in the input clip, which also influences the last MLP head.

Google Drive folder: link to checkpoints in GDrive

Model Nf Checkpoint (.pth) Args (Model Parameters)
TSformer-VO-1 2 checkpoint_model1 args.pkl
TSformer-VO-2 3 checkpoint_model2 args.pkl
TSformer-VO-3 4 checkpoint_model3 args.pkl

3. Setup

  • Create a virtual environment using Anaconda and activate it:
conda create -n tsformer-vo python==3.8.0
conda activate tsformer-vo
  • Install dependencies (with environment activated):
pip install -r requirements.txt

4. Usage

PS: So far we are changing the settings and hyperparameters directly in the variables and dictionaries. As further work, we will use pre-set configurations with the argparse module to make a user-friendly interface.

4.1. Training

In train.py:

  • Manually set configuration in args (python dict);
  • Manually set the model hyperparameters in model_params (python dict);
  • Save and run the code train.py.

4.2. Inference

In predict_poses.py:

  • Manually set the variables to read the checkpoint and sequences.
Variables Info
checkpoint_path String with the path to the trained model you want to use for inference. Ex: checkpoint_path = "checkpoints/Model1"
checkpoint_name String with the name of the desired checkpoint (name of the .pth file). Ex: checkpoint_name = "checkpoint_model2_exp19"
sequences List with strings representing the KITTI sequences. Ex: sequences = ["03", "04", "10"]

4.3. Visualize Trajectories

In plot_results.py:

  • Manually set the variables to the checkpoint and desired sequences, similarly to Inference

5. Evaluation

The evaluation is done with the KITTI odometry evaluation toolbox. Please go to the evaluation repository to see more details about the evaluation metrics and how to run the toolbox.

Citation

Please cite our paper you find this research useful in your work:

@article{Francani2025,
  author={Françani, André O. and Maximo, Marcos R. O. A.},
  journal={IEEE Access}, 
  title={Transformer-Based Model for Monocular Visual Odometry: A Video Understanding Approach}, 
  year={2025},
  volume={13},
  number={},
  pages={13959-13971},
  doi={10.1109/ACCESS.2025.3531667}
}

References

Code adapted from TimeSformer.

Check out our previous work on monocular visual odometry: DPT-VO