SLAM Hive Wiki

Explanation:

This custom analysis task is to explore how the resolution and framerate of Image data affect the accuracy of Vision SLAM algorithms, and how to balance algorithm accuracy, resource consumption, and image data quality (resolution and framerate).

4 V-SLAM algorithms with different modes (totally 10 combination). 5 Image frame rate: 20Hz, 10Hz, 5Hz, 2Hz, 1Hz 6 Image Resolution: 1, 0.8, 0.6, 0.5, 0.4, 0.2 5 sequences of EuRoC Dataset Totally:1500 configurations

This task can generate static 2D and 3D scatter. And in the web page, you can see the static scatter, create the scatter using different metrics and parameters, download the raw data of these charts, and also see the dynamic scatter online.

configuration_choose:
  comb_configuration_id: # all the comb_configuration ID. Totally 50 groups.
  - 147
  - 148
  - 149
  - 150
  - 151
  - 141
  - 142
  - 143
  - 144
  - 145
  - 133
  - 134
  - 135
  - 136
  - 137
  - 128
  - 129
  - 130
  - 131
  - 132
  - 123
  - 124
  - 125
  - 126
  - 127
  - 118
  - 119
  - 120
  - 121
  - 122
  - 152
  - 153
  - 154
  - 155
  - 156
  - 163
  - 164
  - 165
  - 166
  - 167
  - 169
  - 170
  - 171
  - 172
  - 173
  - 173
  - 174
  - 175
  - 176
  - 177
  - 178
  combination_rule: # config id - 0; comb config id - 1; limitation rules - 2;
    first_one: # (1) [-] (0 [Intersection] 2) (Actually Using all configs in 1)
    - 1
    first_rule:
    - U
    second_one:
    - 0
    - 2
    second_rule:
    - I
  configuration_id: []
  limitation_rules:
    algorithm_id: null
    dataset_id: null
    evaluation_value:
      ate_max_maximum: null
      ate_max_minimum: null
      ate_max_nolimitation: 1
      ate_mean_maximum: null
      ate_mean_minimum: null
      ate_mean_nolimitation: 1
      ate_median_maximum: null
      ate_median_minimum: null
      ate_median_nolimitation: 1
      ate_min_maximum: null
      ate_min_minimum: null
      ate_min_nolimitation: 1
      ate_rmse_maximum: null
      ate_rmse_minimum: null
      ate_rmse_nolimitation: 1
      ate_sse_maximum: null
      ate_sse_minimum: null
      ate_sse_nolimitation: 1
      ate_std_maximum: null
      ate_std_minimum: null
      ate_std_nolimitation: 1
      cpu_max_maximum: null
      cpu_max_minimum: null
      cpu_max_nolimitation: 1
      cpu_mean_maximum: null
      cpu_mean_minimum: null
      cpu_mean_nolimitation: 1
      ram_max_maximum: null
      ram_max_minimum: null
      ram_max_nolimitation: 1
      rpe_max_maximum: null
      rpe_max_minimum: null
      rpe_max_nolimitation: 1
      rpe_mean_maximum: null
      rpe_mean_minimum: null
      rpe_mean_nolimitation: 1
      rpe_median_maximum: null
      rpe_median_minimum: null
      rpe_median_nolimitation: 1
      rpe_min_maximum: null
      rpe_min_minimum: null
      rpe_min_nolimitation: 1
      rpe_rmse_maximum: null
      rpe_rmse_minimum: null
      rpe_rmse_nolimitation: 1
      rpe_sse_maximum: null
      rpe_sse_minimum: null
      rpe_sse_nolimitation: 1
      rpe_std_maximum: null
      rpe_std_minimum: null
      rpe_std_nolimitation: 1
    parameters_value:
evaluation_form:
  1_trajectory_comparison:
    choose: 0
  2_accuracy_metrics_comparison:
    choose: 0
  3_accuracy_metrics_comparison:
    algorithm_id:
    - 12
    calculate_method: 1
    choose: 0
    dataset_id:
    - 2
    metric: ate_rmse
  4_usage_metrics_comparison:
    choose: 0
  5_scatter_diagram:
    choose: 0
    x-axis: ate_mean
    y-axis: cpu_mean
  6_scatter_diagram: # choose this
    choose: 1
    extend_choose: 1 # choose Extend-Evo metrics
    extend_multiple: # the multiple by which the different inervals increase the origin value
    - 1
    - 2
    - 5
    - 10
    extend_threshold: # set the inervals - [1, 0.75) [0.75, 0.5) [0.5, 0.25) [0.25, 0]
    - 0.75
    - 0.5
    - 0.25
    x-axis: cpu_mean # x axis 
    y-axis: ate_mean # y axis
  7_3d_scatter_diagram: # choose this
    choose: 1         
    extend_choose: 1    # same as 6_scatter_diagram
    extend_multiple:
    - 1
    - 2
    - 5
    - 10
    extend_threshold:
    - 0.75
    - 0.5
    - 0.25
    x-axis: general+image_width
    y-axis: general+image_frequency
    z-axis: ate_mean
  8_repeatability_test:
    choose: 0
    metric: null
  algorithm_dataset_type: 3
group_description: 10 modes of 4 vision algorithm on 5 sequences; 5 image rate and
  6 image resolution - totally 1500 configurations
group_name: Paper_Vision_SLAM_Image_Exploration_Experiment

Explanation:

This analysis task is to calculate the metric of some configurations

10 modes 5 dataset sequences totally: 50

This YAML file's metric is “ATE RMSE”, you can also change to select other metrics.

configuration_choose:
  comb_configuration_id: []
  combination_rule:
    first_one:
    - 0
    first_rule:
    - I
    second_one:
    - 2
    - 1
    second_rule:
    - U
  configuration_id:
  - 274
  - 275
  - 276
  - 277
  - 278
  - 279
  - 282
  - 285
  - 287
  - 288
  - 706
  - 1735
  - 1585
  - 722
  - 754
  - 762
  - 1885
  - 2040
  - 2190
  - 2340
  - 1765
  - 1615
  - 778
  - 794
  - 826
  - 834
  - 1915
  - 2070
  - 2220
  - 2370
  - 1795
  - 1645
  - 850
  - 866
  - 882
  - 890
  - 1945
  - 2100
  - 2250
  - 2400
  - 1825
  - 1675
  - 906
  - 978
  - 954
  - 962
  - 1975
  - 2130
  - 2280
  - 2430
  limitation_rules:
    algorithm_id: null
    dataset_id: null
    evaluation_value:
      ate_max_maximum: null
      ate_max_minimum: null
      ate_max_nolimitation: 1
      ate_mean_maximum: null
      ate_mean_minimum: null
      ate_mean_nolimitation: 1
      ate_median_maximum: null
      ate_median_minimum: null
      ate_median_nolimitation: 1
      ate_min_maximum: null
      ate_min_minimum: null
      ate_min_nolimitation: 1
      ate_rmse_maximum: null
      ate_rmse_minimum: null
      ate_rmse_nolimitation: 1
      ate_sse_maximum: null
      ate_sse_minimum: null
      ate_sse_nolimitation: 1
      ate_std_maximum: null
      ate_std_minimum: null
      ate_std_nolimitation: 1
      cpu_max_maximum: null
      cpu_max_minimum: null
      cpu_max_nolimitation: 1
      cpu_mean_maximum: null
      cpu_mean_minimum: null
      cpu_mean_nolimitation: 1
      ram_max_maximum: null
      ram_max_minimum: null
      ram_max_nolimitation: 1
      rpe_max_maximum: null
      rpe_max_minimum: null
      rpe_max_nolimitation: 1
      rpe_mean_maximum: null
      rpe_mean_minimum: null
      rpe_mean_nolimitation: 1
      rpe_median_maximum: null
      rpe_median_minimum: null
      rpe_median_nolimitation: 1
      rpe_min_maximum: null
      rpe_min_minimum: null
      rpe_min_nolimitation: 1
      rpe_rmse_maximum: null
      rpe_rmse_minimum: null
      rpe_rmse_nolimitation: 1
      rpe_sse_maximum: null
      rpe_sse_minimum: null
      rpe_sse_nolimitation: 1
      rpe_std_maximum: null
      rpe_std_minimum: null
      rpe_std_nolimitation: 1
    parameters_value: []
evaluation_form:
  1_trajectory_comparison:
    choose: 0
  2_accuracy_metrics_comparison:
    choose: 0
  3_accuracy_metrics_comparison: # Because we only provide one configuration for one group(algorithm+dataset), so this task actually just select the value of the metric. In general case, we provide 2 ways to calculate the metric (average and best).
    algorithm_id:
    - 2
    - 3
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
    - 11
    - 12
    calculate_method: 1
    choose: 1
    dataset_id:
    - 2
    - 3
    - 5
    - 6
    - 7
    metric: ate_rmse # change here to change metric
  4_usage_metrics_comparison:
    choose: 0
  5_scatter_diagram:
    choose: 0
    x-axis: ate_mean
    y-axis: cpu_mean
  6_scatter_diagram:
    choose: 0
    x-axis: null
    y-axis: null
  7_3d_scatter_diagram:
    choose: 0
    x-axis: general+image_width
    y-axis: general+imu_frequency
  8_repeatability_test:
    choose: 0
    metric: ate_mean
  algorithm_dataset_type: 3
group_description: test3
group_name: Paper_3_accuracy_metrics_comparison_ATE_RMSE_Experiment

Explanation:

This analysis task is to compare some Lidar based SLAM algorithms in trajectory, Evo metrics and CPU and RAM usage.

6 modes 1 dataset sequences totally: 6

note: 1) Don't select too many configurations to draw them in one diagram. 2) If your selected configurations contain failed trajectories, system will not generate diagram and return an error.

configuration_choose:
  comb_configuration_id:
  - 5
  combination_rule:
    first_one:
    - 0
    first_rule:
    - I
    second_one:
    - 2
    - 1
    second_rule:
    - I
  configuration_id: [994, 995, 996, 997, 998, 999, 1000]

  limitation_rules:
    algorithm_id:
    - 12
    - 11
    - 10
    dataset_id:
    - 15
    evaluation_value:
      ate_max_maximum: null
      ate_max_minimum: null
      ate_max_nolimitation: 1
      ate_mean_maximum: null
      ate_mean_minimum: null
      ate_mean_nolimitation: 1
      ate_median_maximum: null
      ate_median_minimum: null
      ate_median_nolimitation: 1
      ate_min_maximum: null
      ate_min_minimum: null
      ate_min_nolimitation: 1
      ate_rmse_maximum: null
      ate_rmse_minimum: null
      ate_rmse_nolimitation: 1
      ate_sse_maximum: null
      ate_sse_minimum: null
      ate_sse_nolimitation: 1
      ate_std_maximum: null
      ate_std_minimum: null
      ate_std_nolimitation: 1
      cpu_max_maximum: null
      cpu_max_minimum: null
      cpu_max_nolimitation: 1
      cpu_mean_maximum: null
      cpu_mean_minimum: null
      cpu_mean_nolimitation: 1
      ram_max_maximum: null
      ram_max_minimum: null
      ram_max_nolimitation: 1
      rpe_max_maximum: null
      rpe_max_minimum: null
      rpe_max_nolimitation: 1
      rpe_mean_maximum: null
      rpe_mean_minimum: null
      rpe_mean_nolimitation: 1
      rpe_median_maximum: null
      rpe_median_minimum: null
      rpe_median_nolimitation: 1
      rpe_min_maximum: null
      rpe_min_minimum: null
      rpe_min_nolimitation: 1
      rpe_rmse_maximum: null
      rpe_rmse_minimum: null
      rpe_rmse_nolimitation: 1
      rpe_sse_maximum: null
      rpe_sse_minimum: null
      rpe_sse_nolimitation: 1
      rpe_std_maximum: null
      rpe_std_minimum: null
      rpe_std_nolimitation: 1
    parameters_value:
    - nFeatures < 4000
evaluation_form:
  1_trajectory_comparison:
    choose: 1
  2_accuracy_metrics_comparison:
    choose: 1
  3_accuracy_metrics_comparison:
    algorithm_id:
    - 2
    - 5
    calculate_method: 1
    choose: 0
    dataset_id:
    - 2
    - 13
    metric: ate_rmse
  4_usage_metrics_comparison:
    choose: 1
  6_scatter_diagram:
    choose: 0
    x-axis: cpu_mean
    y-axis: ate_mean
  7_3d_scatter_diagram:
    choose: 0
    x-axis: null
    y-axis: null
    z-axis: null
  8_repeatability_test:
    choose: 0
    metric: null
  algorithm_dataset_type: 1
group_description: Lidar SLAM
group_name: Paper_1_2_4_Lidar_SLAM_Trajectory_Evo_Usage_Experiment

Explanation:

This analysis task is to compare the stability of a configuration. We run 20 times of one configuration.

1 configuration with 20 times totally: 20

configuration_choose:
  comb_configuration_id: []
  combination_rule:
    first_one:
    - 0
    first_rule:
    - U
    second_one:
    - 1
    - 2
    second_rule:
    - I
  configuration_id:
  - 531
  limitation_rules:
    algorithm_id: null
    dataset_id: null
    evaluation_value:
      ate_max_maximum: null
      ate_max_minimum: null
      ate_max_nolimitation: 1
      ate_mean_maximum: null
      ate_mean_minimum: null
      ate_mean_nolimitation: 1
      ate_median_maximum: null
      ate_median_minimum: null
      ate_median_nolimitation: 1
      ate_min_maximum: null
      ate_min_minimum: null
      ate_min_nolimitation: 1
      ate_rmse_maximum: null
      ate_rmse_minimum: null
      ate_rmse_nolimitation: 1
      ate_sse_maximum: null
      ate_sse_minimum: null
      ate_sse_nolimitation: 1
      ate_std_maximum: null
      ate_std_minimum: null
      ate_std_nolimitation: 1
      cpu_max_maximum: null
      cpu_max_minimum: null
      cpu_max_nolimitation: 1
      cpu_mean_maximum: null
      cpu_mean_minimum: null
      cpu_mean_nolimitation: 1
      ram_max_maximum: null
      ram_max_minimum: null
      ram_max_nolimitation: 1
      rpe_max_maximum: null
      rpe_max_minimum: null
      rpe_max_nolimitation: 1
      rpe_mean_maximum: null
      rpe_mean_minimum: null
      rpe_mean_nolimitation: 1
      rpe_median_maximum: null
      rpe_median_minimum: null
      rpe_median_nolimitation: 1
      rpe_min_maximum: null
      rpe_min_minimum: null
      rpe_min_nolimitation: 1
      rpe_rmse_maximum: null
      rpe_rmse_minimum: null
      rpe_rmse_nolimitation: 1
      rpe_sse_maximum: null
      rpe_sse_minimum: null
      rpe_sse_nolimitation: 1
      rpe_std_maximum: null
      rpe_std_minimum: null
      rpe_std_nolimitation: 1
    parameters_value:
    - nFeatures < 4000
evaluation_form:
  1_trajectory_comparison:
    choose: 0
  2_accuracy_metrics_comparison:
    choose: 0
  3_accuracy_metrics_comparison:
    algorithm_id:
    - 12
    calculate_method: 1
    choose: 0
    dataset_id:
    - 2
    metric: ate_rmse
  4_usage_metrics_comparison:
    choose: 0
  5_scatter_diagram:
    choose: 0
    x-axis: ate_mean
    y-axis: cpu_mean
  6_scatter_diagram:
    choose: 0
    x-axis: general+image_frequency
    y-axis: ate_mean
  7_3d_scatter_diagram:
    choose: 0
    x-axis: general+image_width
    y-axis: general+imu_frequency
    z-axis: ate_mean
  8_repeatability_test:
    choose: 1
    metric: ate_rmse
  algorithm_dataset_type: 0
group_description: random test; 531 (bad dataset)
group_name: Paper_8_repeatability_Experiment

Explanation: This experiment explores how the quality of Image data (resolution and framerate) affects Visual-based SLAM algorithms.

We run 4 visual-based SLAM algorithms on different sequences of the EuRoC dataset and create 1500 configurations totall. The detail parameter space is shown in \ref{table: exploration3}. And in this experiment, the Extend metric parameters are: internal range: $[1, 0.75), [0.75, 0.5), [0.5, 0.25), [0.25, 0]$; multiple value: $[1, 2, 2.5, 3]$.

In the practical application of SLAM algorithm, it is often necessary to meet some specific scenarios, such as: limited computing resources, accuracy requirements, etc. 
Therefore, in order to analyze such a large number of tasks succinctly, we utilize our system search engine to performs a simple pre-process of the data - Filtering through these 1500 configurations using the conditions: $[0.02 \le ATE.Mean \le 0.1; 0.6 \le CPU.Mean \le 2.0; 600 \le Memory.Max \le 1000]$ results in 278 matching configurations, which are then analyzed with Extend metrics.

Category: orb-slam2-mono:

1. 147
2. 148
3. 149
4. 150
5. 151

orb-slam3-mono:

1. 133
2. 134
3. 135
4. 136
5. 137

orb-slam3-inertial:

1. 128
2. 129
3. 130
4. 131
5. 132

vins-mono:

1. 152
2. 153
3. 154
4. 155
5. 156

vins-fusion-mono-imu:

1. 163
2. 164
3. 165
4. 166
5. 167

orb-slam2-stereo:

1. 141
2. 142
3. 143
4. 144
5. 145

orb-slam3-stereo:

1. 123
2. 124
3. 125
4. 126
5. 127

vins-fusion-stereo:

1. 169
2. 170
3. 171
4. 172
5. 173

orb-slam3-stereo-inertial:

1. 118
2. 119
3. 120
4. 121
5. 122

vins-fusion-stereo-imu:

1. 174
2. 175
3. 176
4. 177
5. 178

configuration_choose:
  comb_configuration_id:
  - 147
  - 148
  - 149
  - 150
  - 151
  - 141
  - 142
  - 143
  - 144
  - 145
  - 133
  - 134
  - 135
  - 136
  - 137
  - 128
  - 129
  - 130
  - 131
  - 132
  - 123
  - 124
  - 125
  - 126
  - 127
  - 118
  - 119
  - 120
  - 121
  - 122
  - 152
  - 153
  - 154
  - 155
  - 156
  - 163
  - 164
  - 165
  - 166
  - 167
  - 169
  - 170
  - 171
  - 172
  - 173
  - 174
  - 175
  - 176
  - 177
  - 178
  combination_rule:
    first_one:
    - 1
    - 2
    first_rule:
    - I
    second_one:
    - 0
    second_rule:
    - I
  configuration_id: []
  limitation_rules:
    algorithm_id:
    - 2
    - 3
    - 5
    - 6
    - 7
    - 8
    - 9
    - 10
    - 11
    - 12
    dataset_id:
    - 2
    - 3
    - 5
    - 6
    - 7
    evaluation_value:
      ate_max_maximum: null
      ate_max_minimum: null
      ate_max_nolimitation: 1
      ate_mean_maximum: null
      ate_mean_minimum: null
      ate_mean_nolimitation: 1
      ate_median_maximum: null
      ate_median_minimum: null
      ate_median_nolimitation: 1
      ate_min_maximum: null
      ate_min_minimum: null
      ate_min_nolimitation: 1
      ate_rmse_maximum: 0.1
      ate_rmse_minimum: 0.02
      ate_rmse_nolimitation: 0
      ate_sse_maximum: null
      ate_sse_minimum: null
      ate_sse_nolimitation: 1
      ate_std_maximum: null
      ate_std_minimum: null
      ate_std_nolimitation: 1
      cpu_max_maximum: null
      cpu_max_minimum: null
      cpu_max_nolimitation: 1
      cpu_mean_maximum: 2
      cpu_mean_minimum: 0.6
      cpu_mean_nolimitation: 0
      ram_max_maximum: 1000
      ram_max_minimum: 600
      ram_max_nolimitation: 0
      rpe_max_maximum: null
      rpe_max_minimum: null
      rpe_max_nolimitation: 1
      rpe_mean_maximum: null
      rpe_mean_minimum: null
      rpe_mean_nolimitation: 1
      rpe_median_maximum: null
      rpe_median_minimum: null
      rpe_median_nolimitation: 1
      rpe_min_maximum: null
      rpe_min_minimum: null
      rpe_min_nolimitation: 1
      rpe_rmse_maximum: null
      rpe_rmse_minimum: null
      rpe_rmse_nolimitation: 1
      rpe_sse_maximum: null
      rpe_sse_minimum: null
      rpe_sse_nolimitation: 1
      rpe_std_maximum: null
      rpe_std_minimum: null
      rpe_std_nolimitation: 1
    parameters_value: []
evaluation_form:
  1_trajectory_comparison:
    choose: 0
  2_accuracy_metrics_comparison:
    choose: 0
  3_accuracy_metrics_comparison:
    algorithm_id:
    - 12
    calculate_method: 1
    choose: 0
    dataset_id:
    - 2
    metric: ate_rmse
  4_usage_metrics_comparison:
    choose: 0
  5_scatter_diagram:
    choose: 0
    x-axis: ate_mean
    y-axis: cpu_mean
  6_scatter_diagram:
    choose: 1
    extend_choose: 1
    extend_multiple:
    - 1
    - 2
    - 2.5
    - 3
    extend_threshold:
    - 0.75
    - 0.5
    - 0.25
    x-axis: cpu_mean
    y-axis: ate_mean
  7_3d_scatter_diagram:
    choose: 1
    extend_choose: 1
    extend_multiple:
    - 1
    - 2
    - 2.5
    - 3
    extend_threshold:
    - 0.75
    - 0.5
    - 0.25
    x-axis: general+image_width
    y-axis: general+image_frequency
    z-axis: ate_mean
  8_repeatability_test:
    choose: 0
    metric: null
  algorithm_dataset_type: 3
group_description: 10 modes of 4 vision algorithm on 5 sequences; 5 image rate and
  6 image resolution - totally 1500 configurations
group_name: Paper_Vision_SLAM_Image_Exploration_Experiment_limitation