Wiki Link
For every dataset sequence, we provide a script for replaying and remapping.
For EuRoC datasets, we provide groundtruth.txt in tum format, all you need to do is download the sequence.
For TUM RGB-D datasets, just download the sequennce and groundtruth, and rename the groundtruth file to groundtruth.txt.
For KITTI datasets, we provide the groundtruth, all you need to do is download the raw data and use kitti2bag to transform it to ROS 1 bags.
For EuRoC and TUM RGB-D datastes, we provide a download script: SLAM_Hive/slam_hive_datasets/download_all.py.
Welcome to the SLAM Hive Benchmarking Suite
Wiki Help · The ShanghaiTech Mapping Robot Datasets · Code · Mobile Autonomous Robotic Systems Lab @ ShanghaiTech University
SLAM Hive is a docker based sofware package for the purose of evaluating the performance of SLAM (Simultaneous Localization and Mapping) algorithms by running them in docker containers with robotic datasets. The resulting paths are then compared with the ground truth paths of the datasets. The SLAM Hive software is available on GitHub: https://github.com/SLAM-Hive. Here on https://slam-hive.net a version of the suite configured as "view only" is running, showing the results of several hundreds of mapping runs and enabling users to analyze the results in detail.
SLAM Hive is developed and maintained by the Mobile Autonomous Robotic Systems Lab of ShanghaiTech University. To get help or contribute please use the GitHub of the slam_hive_web repository.
More details about SLAM Hive can be found in our papers (please cite):
2024 Journal preprint (arXiv PDF):
@misc{liu2024benchmarkingslamalgorithmscloud,
title={Benchmarking SLAM Algorithms in the Cloud: The SLAM Hive Benchmarking Suite},
author={Xinzhe Liu and Yuanyuan Yang and Bowen Xu and Delin Feng and S{\"o}ren Schwertfeger},
year={2024},
eprint={2406.17586},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2406.17586},
}
ICRA 2023 paper (arXiv PDF):
@inproceedings{yang2023slam,
title={The SLAM Hive Benchmarking Suite},
author={Yang, Yuanyuan and Xu, Bowen and Li, Yinjie and Schwertfeger, S{\"o}ren},
booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
pages={11257--11263},
year={2023},
organization={IEEE}
}
Below please find the list of supported SLAM algorithms and datasets:
Supported Algorithms List
| Algorithm | Image (REPOSITORY:TAG) | Scripts Path |
|---|---|---|
| EVO | slam-hive-evaluation:evo | SLAM_Hive/slam_hive_algos/evo |
| slam-hive-evaluation:evo_latex | SLAM_Hive/slam_hive_algos/evo_latex | |
| ORB-SLAM2 | slam-hive-algorithm:orb-slam2-ros-mono | SLAM_Hive/slam_hive_algos/orb-slam2-ros-mono |
| slam-hive-algorithm:orb-slam2-ros-stereo | SLAM_Hive/slam_hive_algos/orb-slam2-ros-stereo | |
| slam-hive-algorithm:orb-slam2-ros-rgbd | SLAM_Hive/slam_hive_algos/orb-slam2-ros-rgbd | |
| VINS-Mono | slam-hive-algorithm:vins-mono | SLAM_Hive/slam_hive_algos/vins-mono |
| VINS-Fusion | slam-hive-algorithm:vins-fusion-mono-imu | SLAM_Hive/slam_hive_algos/vins-fusion-mono-imu |
| slam-hive-algorithm:vins-fusion-stereo | SLAM_Hive/slam_hive_algos/vins-fusion-stereo | |
| slam-hive-algorithm:vins-fusion-stereo-imu | SLAM_Hive/slam_hive_algos/vins-fusion-stereo-imu | |
| ORB-SLAM3 | slam-hive-algorithm:orb-slam3-ros-mono | SLAM_Hive/slam_hive_algos/orb-slam3-ros-mono |
| slam-hive-algorithm:orb-slam3-ros-mono-inertial | SLAM_Hive/slam_hive_algos/orb-slam3-ros-mono-inertial | |
| slam-hive-algorithm:orb-slam3-ros-stereo | SLAM_Hive/slam_hive_algos/orb-slam3-ros-stereo | |
| slam-hive-algorithm:orb-slam3-ros-stereo-inertial | SLAM_Hive/slam_hive_algos/orb-slam3-ros-stereo-inertial | |
| slam-hive-algorithm:orb-slam3-ros-rgbd | SLAM_Hive/slam_hive_algos/orb-slam3-ros-rgbd | |
| LIO-SAM | slam-hive-algorithm:lio-sam | SLAM_Hive/slam_hive_algos/lio-sam |
| LOAM | slam-hive-algorithm:loam | SLAM_Hive/slam_hive_algos/loam |
| A-LOAM | slam-hive-algorithm:a-loam | SLAM_Hive/slam_hive_algos/a-loam |
| LEGO-LOAM | slam-hive-algorithm:lego-loam | SLAM_Hive/slam_hive_algos/lego-loam |
| slam-hive-algorithm:lego-loam-imu | SLAM_Hive/slam_hive_algos/lego-loam-imu | |
| NDT-LOAM | slam-hive-algorithm:ndt-loam | SLAM_Hive/slam_hive_algos/ndt-loam |
| DSO TODO | slam-hive-algorithm:dso | SLAM_Hive/slam_hive_algos/dso |
| LIVOX-LOAM TODO | slam-hive-algorithm:livox-loam | SLAM_Hive/slam_hive_algos/livox-loam |
Supported Datasets List
| Dataset | Sequence | Download | Groundtruth | Script |
|---|---|---|---|---|
| EuRoC MAV | MH_01_easy | MH_01_easy.bag | We provide groundtruth already converted to tum format. They are in SLAM_Hive/slam_hive_datasets/<>/groundturth.txt | In SLAM_Hive/slam_hive_datasets/<>/rosbag_play.py |
| MH_02_easy | MH_02_easy.bag | |||
| MH_03_medium | MH_03_medium.bag | |||
| MH_04_difficult | MH_04_difficult.bag | |||
| MH_05_difficult | MH_05_difficult.bag | |||
| V1_01_easy | V1_01_easy.bag | |||
| V1_02_medium | V1_02_medium.bag | |||
| V1_03_difficult | V1_03_difficult.bag | |||
| V2_01_easy | V2_01_easy.bag | |||
| V2_02_medium | V2_02_medium.bag | |||
| V2_03_difficult | V2_03_difficult.bag | |||
| TUM RGB-D | rgbd_dataset_freiburg1_xyz | rgbd_dataset_freiburg1_xyz.bag | groundtruth.txt | In SLAM_Hive/slam_hive_datasets/<>/rosbag_play.py |
| rgbd_dataset_freiburg1_rpy | rgbd_dataset_freiburg1_rpy.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_xyz | rgbd_dataset_freiburg2_xyz.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_rpy | rgbd_dataset_freiburg2_rpy.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg1_360 | rgbd_dataset_freiburg1_360.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg1_floor | rgbd_dataset_freiburg1_floor.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg1_desk | rgbd_dataset_freiburg1_desk.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg1_desk2 | rgbd_dataset_freiburg1_desk2.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg1_room | rgbd_dataset_freiburg1_room.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_360_hemisphere | rgbd_dataset_freiburg2_360_hemisphere.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_360_kidnap | rgbd_dataset_freiburg2_360_kidnap.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_desk | rgbd_dataset_freiburg2_desk.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_large_no_loop | rgbd_dataset_freiburg2_large_no_loop.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg2_large_with_loop | rgbd_dataset_freiburg2_large_with_loop.bag | groundtruth.txt | ||
| rgbd_dataset_freiburg3_long_office_household | rgbd_dataset_freiburg3_long_office_household.bag | groundtruth.txt | ||
| KITTI | kitti_2011_10_03_drive_0027_synced | kitti_2011_10_03_drive_0027_synced.bag | We provide groundtruth already converted to tum format. They are in SLAM_Hive/slam_hive_datasets/<>/groundturth.txt | In SLAM_Hive/slam_hive_datasets/<>/rosbag_play.py |
| kitti_2011_10_03_drive_0042_synced | kitti_2011_10_03_drive_0042_synced.bag | |||
| kitti_2011_10_03_drive_0034_synced | kitti_2011_10_03_drive_0034_synced.bag | |||
| kitti_2011_09_30_drive_0016_synced | kitti_2011_09_30_drive_0016_synced.bag | |||
| kitti_2011_09_30_drive_0018_synced | kitti_2011_09_30_drive_0018_synced.bag | |||
| kitti_2011_09_30_drive_0020_synced | kitti_2011_09_30_drive_0020_synced.bag | |||
| kitti_2011_09_30_drive_0027_synced | kitti_2011_09_30_drive_0027_synced.bag | |||
| kitti_2011_09_30_drive_0028_synced | kitti_2011_09_30_drive_0028_synced.bag | |||
| kitti_2011_09_30_drive_0033_synced | kitti_2011_09_30_drive_0033_synced.bag | |||
| kitti_2011_09_30_drive_0034_synced | kitti_2011_09_30_drive_0034_synced.bag |