This page shows some custom analysis examples that we have created.
=== Paper_Vision_SLAM_Image_Exploration_Experiment ===
Explanation:
This custom analysis task is to explore how the resolution and framerate of Image data affect the accuracy of Vision SLAM algorithms, and how to balance algorithm accuracy, resource consumption, and image data quality (resolution and framerate).
4 V-SLAM algorithms with different modes (totally 10 combination).
5 Image frame rate: 20Hz, 10Hz, 5Hz, 2Hz, 1Hz
6 Image Resolution: 1, 0.8, 0.6, 0.5, 0.4, 0.2
5 sequences of EuRoC Dataset
Totally:1500 configurations
This task can generate static 2D and 3D scatter. And in the web page, you can see the static scatter, create the scatter using different metrics and parameters, download the raw data of these charts, and also see the dynamic scatter online.
configuration_choose:
comb_configuration_id: # all the comb_configuration ID. Totally 50 groups.
- 147
- 148
- 149
- 150
- 151
- 141
- 142
- 143
- 144
- 145
- 133
- 134
- 135
- 136
- 137
- 128
- 129
- 130
- 131
- 132
- 123
- 124
- 125
- 126
- 127
- 118
- 119
- 120
- 121
- 122
- 152
- 153
- 154
- 155
- 156
- 163
- 164
- 165
- 166
- 167
- 169
- 170
- 171
- 172
- 173
- 173
- 174
- 175
- 176
- 177
- 178
combination_rule: # config id - 0; comb config id - 1; limitation rules - 2;
first_one: # (1) [-] (0 [Intersection] 2) (Actually Using all configs in 1)
- 1
first_rule:
- U
second_one:
- 0
- 2
second_rule:
- I
configuration_id: []
limitation_rules:
algorithm_id: null
dataset_id: null
evaluation_value:
ate_max_maximum: null
ate_max_minimum: null
ate_max_nolimitation: 1
ate_mean_maximum: null
ate_mean_minimum: null
ate_mean_nolimitation: 1
ate_median_maximum: null
ate_median_minimum: null
ate_median_nolimitation: 1
ate_min_maximum: null
ate_min_minimum: null
ate_min_nolimitation: 1
ate_rmse_maximum: null
ate_rmse_minimum: null
ate_rmse_nolimitation: 1
ate_sse_maximum: null
ate_sse_minimum: null
ate_sse_nolimitation: 1
ate_std_maximum: null
ate_std_minimum: null
ate_std_nolimitation: 1
cpu_max_maximum: null
cpu_max_minimum: null
cpu_max_nolimitation: 1
cpu_mean_maximum: null
cpu_mean_minimum: null
cpu_mean_nolimitation: 1
ram_max_maximum: null
ram_max_minimum: null
ram_max_nolimitation: 1
rpe_max_maximum: null
rpe_max_minimum: null
rpe_max_nolimitation: 1
rpe_mean_maximum: null
rpe_mean_minimum: null
rpe_mean_nolimitation: 1
rpe_median_maximum: null
rpe_median_minimum: null
rpe_median_nolimitation: 1
rpe_min_maximum: null
rpe_min_minimum: null
rpe_min_nolimitation: 1
rpe_rmse_maximum: null
rpe_rmse_minimum: null
rpe_rmse_nolimitation: 1
rpe_sse_maximum: null
rpe_sse_minimum: null
rpe_sse_nolimitation: 1
rpe_std_maximum: null
rpe_std_minimum: null
rpe_std_nolimitation: 1
parameters_value:
evaluation_form:
1_trajectory_comparison:
choose: 0
2_accuracy_metrics_comparison:
choose: 0
3_accuracy_metrics_comparison:
algorithm_id:
- 12
calculate_method: 1
choose: 0
dataset_id:
- 2
metric: ate_rmse
4_usage_metrics_comparison:
choose: 0
5_scatter_diagram:
choose: 0
x-axis: ate_mean
y-axis: cpu_mean
6_scatter_diagram: # choose this
choose: 1
extend_choose: 1 # choose Extend-Evo metrics
extend_multiple: # the multiple by which the different inervals increase the origin value
- 1
- 2
- 5
- 10
extend_threshold: # set the inervals - [1, 0.75) [0.75, 0.5) [0.5, 0.25) [0.25, 0]
- 0.75
- 0.5
- 0.25
x-axis: cpu_mean # x axis
y-axis: ate_mean # y axis
7_3d_scatter_diagram: # choose this
choose: 1
extend_choose: 1 # same as 6_scatter_diagram
extend_multiple:
- 1
- 2
- 5
- 10
extend_threshold:
- 0.75
- 0.5
- 0.25
x-axis: general+image_width
y-axis: general+image_frequency
z-axis: ate_mean
8_repeatability_test:
choose: 0
metric: null
algorithm_dataset_type: 3
group_description: 10 modes of 4 vision algorithm on 5 sequences; 5 image rate and
6 image resolution - totally 1500 configurations
group_name: Paper_Vision_SLAM_Image_Exploration_Experiment
------
=== Paper_3_accuracy_metrics_comparison_ATE_RMSE_Experiment ===
Explanation:
This analysis task is to calculate the metric of some configurations
10 modes
5 dataset sequences
totally: 50
This YAML file's metric is "ATE RMSE", you can also change to select other metrics.
configuration_choose:
comb_configuration_id: []
combination_rule:
first_one:
- 0
first_rule:
- I
second_one:
- 2
- 1
second_rule:
- U
configuration_id:
- 274
- 275
- 276
- 277
- 278
- 279
- 282
- 285
- 287
- 288
- 706
- 1735
- 1585
- 722
- 754
- 762
- 1885
- 2040
- 2190
- 2340
- 1765
- 1615
- 778
- 794
- 826
- 834
- 1915
- 2070
- 2220
- 2370
- 1795
- 1645
- 850
- 866
- 882
- 890
- 1945
- 2100
- 2250
- 2400
- 1825
- 1675
- 906
- 978
- 954
- 962
- 1975
- 2130
- 2280
- 2430
limitation_rules:
algorithm_id: null
dataset_id: null
evaluation_value:
ate_max_maximum: null
ate_max_minimum: null
ate_max_nolimitation: 1
ate_mean_maximum: null
ate_mean_minimum: null
ate_mean_nolimitation: 1
ate_median_maximum: null
ate_median_minimum: null
ate_median_nolimitation: 1
ate_min_maximum: null
ate_min_minimum: null
ate_min_nolimitation: 1
ate_rmse_maximum: null
ate_rmse_minimum: null
ate_rmse_nolimitation: 1
ate_sse_maximum: null
ate_sse_minimum: null
ate_sse_nolimitation: 1
ate_std_maximum: null
ate_std_minimum: null
ate_std_nolimitation: 1
cpu_max_maximum: null
cpu_max_minimum: null
cpu_max_nolimitation: 1
cpu_mean_maximum: null
cpu_mean_minimum: null
cpu_mean_nolimitation: 1
ram_max_maximum: null
ram_max_minimum: null
ram_max_nolimitation: 1
rpe_max_maximum: null
rpe_max_minimum: null
rpe_max_nolimitation: 1
rpe_mean_maximum: null
rpe_mean_minimum: null
rpe_mean_nolimitation: 1
rpe_median_maximum: null
rpe_median_minimum: null
rpe_median_nolimitation: 1
rpe_min_maximum: null
rpe_min_minimum: null
rpe_min_nolimitation: 1
rpe_rmse_maximum: null
rpe_rmse_minimum: null
rpe_rmse_nolimitation: 1
rpe_sse_maximum: null
rpe_sse_minimum: null
rpe_sse_nolimitation: 1
rpe_std_maximum: null
rpe_std_minimum: null
rpe_std_nolimitation: 1
parameters_value: []
evaluation_form:
1_trajectory_comparison:
choose: 0
2_accuracy_metrics_comparison:
choose: 0
3_accuracy_metrics_comparison: # Because we only provide one configuration for one group(algorithm+dataset), so this task actually just select the value of the metric. In general case, we provide 2 ways to calculate the metric (average and best).
algorithm_id:
- 2
- 3
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
calculate_method: 1
choose: 1
dataset_id:
- 2
- 3
- 5
- 6
- 7
metric: ate_rmse # change here to change metric
4_usage_metrics_comparison:
choose: 0
5_scatter_diagram:
choose: 0
x-axis: ate_mean
y-axis: cpu_mean
6_scatter_diagram:
choose: 0
x-axis: null
y-axis: null
7_3d_scatter_diagram:
choose: 0
x-axis: general+image_width
y-axis: general+imu_frequency
8_repeatability_test:
choose: 0
metric: ate_mean
algorithm_dataset_type: 3
group_description: test3
group_name: Paper_3_accuracy_metrics_comparison_ATE_RMSE_Experiment
------
=== Paper_1_2_4_Lidar_SLAM_Trajectory_Evo_Usage_Experiment ===
Explanation:
This analysis task is to compare some Lidar based SLAM algorithms in trajectory, Evo metrics and CPU and RAM usage.
6 modes
1 dataset sequences
totally: 6
note: 1) Don't select too many configurations to draw them in one diagram. 2) If your selected configurations contain failed trajectories, system will not generate diagram and return an error.
configuration_choose:
comb_configuration_id:
- 5
combination_rule:
first_one:
- 0
first_rule:
- I
second_one:
- 2
- 1
second_rule:
- I
configuration_id: [994, 995, 996, 997, 998, 999, 1000]
limitation_rules:
algorithm_id:
- 12
- 11
- 10
dataset_id:
- 15
evaluation_value:
ate_max_maximum: null
ate_max_minimum: null
ate_max_nolimitation: 1
ate_mean_maximum: null
ate_mean_minimum: null
ate_mean_nolimitation: 1
ate_median_maximum: null
ate_median_minimum: null
ate_median_nolimitation: 1
ate_min_maximum: null
ate_min_minimum: null
ate_min_nolimitation: 1
ate_rmse_maximum: null
ate_rmse_minimum: null
ate_rmse_nolimitation: 1
ate_sse_maximum: null
ate_sse_minimum: null
ate_sse_nolimitation: 1
ate_std_maximum: null
ate_std_minimum: null
ate_std_nolimitation: 1
cpu_max_maximum: null
cpu_max_minimum: null
cpu_max_nolimitation: 1
cpu_mean_maximum: null
cpu_mean_minimum: null
cpu_mean_nolimitation: 1
ram_max_maximum: null
ram_max_minimum: null
ram_max_nolimitation: 1
rpe_max_maximum: null
rpe_max_minimum: null
rpe_max_nolimitation: 1
rpe_mean_maximum: null
rpe_mean_minimum: null
rpe_mean_nolimitation: 1
rpe_median_maximum: null
rpe_median_minimum: null
rpe_median_nolimitation: 1
rpe_min_maximum: null
rpe_min_minimum: null
rpe_min_nolimitation: 1
rpe_rmse_maximum: null
rpe_rmse_minimum: null
rpe_rmse_nolimitation: 1
rpe_sse_maximum: null
rpe_sse_minimum: null
rpe_sse_nolimitation: 1
rpe_std_maximum: null
rpe_std_minimum: null
rpe_std_nolimitation: 1
parameters_value:
- nFeatures < 4000
evaluation_form:
1_trajectory_comparison:
choose: 1
2_accuracy_metrics_comparison:
choose: 1
3_accuracy_metrics_comparison:
algorithm_id:
- 2
- 5
calculate_method: 1
choose: 0
dataset_id:
- 2
- 13
metric: ate_rmse
4_usage_metrics_comparison:
choose: 1
6_scatter_diagram:
choose: 0
x-axis: cpu_mean
y-axis: ate_mean
7_3d_scatter_diagram:
choose: 0
x-axis: null
y-axis: null
z-axis: null
8_repeatability_test:
choose: 0
metric: null
algorithm_dataset_type: 1
group_description: Lidar SLAM
group_name: Paper_1_2_4_Lidar_SLAM_Trajectory_Evo_Usage_Experiment
-------
=== Paper_8_repeatability_Experiment ===
Explanation:
This analysis task is to compare the stability of a configuration. We run 20 times of one configuration.
1 configuration with 20 times
totally: 20
configuration_choose:
comb_configuration_id: []
combination_rule:
first_one:
- 0
first_rule:
- U
second_one:
- 1
- 2
second_rule:
- I
configuration_id:
- 531
limitation_rules:
algorithm_id: null
dataset_id: null
evaluation_value:
ate_max_maximum: null
ate_max_minimum: null
ate_max_nolimitation: 1
ate_mean_maximum: null
ate_mean_minimum: null
ate_mean_nolimitation: 1
ate_median_maximum: null
ate_median_minimum: null
ate_median_nolimitation: 1
ate_min_maximum: null
ate_min_minimum: null
ate_min_nolimitation: 1
ate_rmse_maximum: null
ate_rmse_minimum: null
ate_rmse_nolimitation: 1
ate_sse_maximum: null
ate_sse_minimum: null
ate_sse_nolimitation: 1
ate_std_maximum: null
ate_std_minimum: null
ate_std_nolimitation: 1
cpu_max_maximum: null
cpu_max_minimum: null
cpu_max_nolimitation: 1
cpu_mean_maximum: null
cpu_mean_minimum: null
cpu_mean_nolimitation: 1
ram_max_maximum: null
ram_max_minimum: null
ram_max_nolimitation: 1
rpe_max_maximum: null
rpe_max_minimum: null
rpe_max_nolimitation: 1
rpe_mean_maximum: null
rpe_mean_minimum: null
rpe_mean_nolimitation: 1
rpe_median_maximum: null
rpe_median_minimum: null
rpe_median_nolimitation: 1
rpe_min_maximum: null
rpe_min_minimum: null
rpe_min_nolimitation: 1
rpe_rmse_maximum: null
rpe_rmse_minimum: null
rpe_rmse_nolimitation: 1
rpe_sse_maximum: null
rpe_sse_minimum: null
rpe_sse_nolimitation: 1
rpe_std_maximum: null
rpe_std_minimum: null
rpe_std_nolimitation: 1
parameters_value:
- nFeatures < 4000
evaluation_form:
1_trajectory_comparison:
choose: 0
2_accuracy_metrics_comparison:
choose: 0
3_accuracy_metrics_comparison:
algorithm_id:
- 12
calculate_method: 1
choose: 0
dataset_id:
- 2
metric: ate_rmse
4_usage_metrics_comparison:
choose: 0
5_scatter_diagram:
choose: 0
x-axis: ate_mean
y-axis: cpu_mean
6_scatter_diagram:
choose: 0
x-axis: general+image_frequency
y-axis: ate_mean
7_3d_scatter_diagram:
choose: 0
x-axis: general+image_width
y-axis: general+imu_frequency
z-axis: ate_mean
8_repeatability_test:
choose: 1
metric: ate_rmse
algorithm_dataset_type: 0
group_description: random test; 531 (bad dataset)
group_name: Paper_8_repeatability_Experiment
------
=== Paper_Vision_SLAM_Image_Exploration_Experiment_limitation ===
Explanation:
This experiment explores how the quality of Image data (resolution and framerate) affects Visual-based SLAM algorithms.
We run 4 visual-based SLAM algorithms on different sequences of the EuRoC dataset and create 1500 configurations totall. The detail parameter space is shown in \ref{table: exploration3}. And in this experiment, the Extend metric parameters are: internal range: $[1, 0.75), [0.75, 0.5), [0.5, 0.25), [0.25, 0]$; multiple value: $[1, 2, 2.5, 3]$.
In the practical application of SLAM algorithm, it is often necessary to meet some specific scenarios, such as: limited computing resources, accuracy requirements, etc.
Therefore, in order to analyze such a large number of tasks succinctly, we utilize our system search engine to performs a simple pre-process of the data - Filtering through these 1500 configurations using the conditions: $[0.02 \le ATE.Mean \le 0.1; 0.6 \le CPU.Mean \le 2.0; 600 \le Memory.Max \le 1000]$ results in 278 matching configurations, which are then analyzed with Extend metrics.
Category:
orb-slam2-mono:
- 147
- 148
- 149
- 150
- 151
orb-slam3-mono:
- 133
- 134
- 135
- 136
- 137
orb-slam3-inertial:
- 128
- 129
- 130
- 131
- 132
vins-mono:
- 152
- 153
- 154
- 155
- 156
vins-fusion-mono-imu:
- 163
- 164
- 165
- 166
- 167
orb-slam2-stereo:
- 141
- 142
- 143
- 144
- 145
orb-slam3-stereo:
- 123
- 124
- 125
- 126
- 127
vins-fusion-stereo:
- 169
- 170
- 171
- 172
- 173
orb-slam3-stereo-inertial:
- 118
- 119
- 120
- 121
- 122
vins-fusion-stereo-imu:
- 174
- 175
- 176
- 177
- 178
configuration_choose:
comb_configuration_id:
- 147
- 148
- 149
- 150
- 151
- 141
- 142
- 143
- 144
- 145
- 133
- 134
- 135
- 136
- 137
- 128
- 129
- 130
- 131
- 132
- 123
- 124
- 125
- 126
- 127
- 118
- 119
- 120
- 121
- 122
- 152
- 153
- 154
- 155
- 156
- 163
- 164
- 165
- 166
- 167
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
combination_rule:
first_one:
- 1
- 2
first_rule:
- I
second_one:
- 0
second_rule:
- I
configuration_id: []
limitation_rules:
algorithm_id:
- 2
- 3
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
dataset_id:
- 2
- 3
- 5
- 6
- 7
evaluation_value:
ate_max_maximum: null
ate_max_minimum: null
ate_max_nolimitation: 1
ate_mean_maximum: null
ate_mean_minimum: null
ate_mean_nolimitation: 1
ate_median_maximum: null
ate_median_minimum: null
ate_median_nolimitation: 1
ate_min_maximum: null
ate_min_minimum: null
ate_min_nolimitation: 1
ate_rmse_maximum: 0.1
ate_rmse_minimum: 0.02
ate_rmse_nolimitation: 0
ate_sse_maximum: null
ate_sse_minimum: null
ate_sse_nolimitation: 1
ate_std_maximum: null
ate_std_minimum: null
ate_std_nolimitation: 1
cpu_max_maximum: null
cpu_max_minimum: null
cpu_max_nolimitation: 1
cpu_mean_maximum: 2
cpu_mean_minimum: 0.6
cpu_mean_nolimitation: 0
ram_max_maximum: 1000
ram_max_minimum: 600
ram_max_nolimitation: 0
rpe_max_maximum: null
rpe_max_minimum: null
rpe_max_nolimitation: 1
rpe_mean_maximum: null
rpe_mean_minimum: null
rpe_mean_nolimitation: 1
rpe_median_maximum: null
rpe_median_minimum: null
rpe_median_nolimitation: 1
rpe_min_maximum: null
rpe_min_minimum: null
rpe_min_nolimitation: 1
rpe_rmse_maximum: null
rpe_rmse_minimum: null
rpe_rmse_nolimitation: 1
rpe_sse_maximum: null
rpe_sse_minimum: null
rpe_sse_nolimitation: 1
rpe_std_maximum: null
rpe_std_minimum: null
rpe_std_nolimitation: 1
parameters_value: []
evaluation_form:
1_trajectory_comparison:
choose: 0
2_accuracy_metrics_comparison:
choose: 0
3_accuracy_metrics_comparison:
algorithm_id:
- 12
calculate_method: 1
choose: 0
dataset_id:
- 2
metric: ate_rmse
4_usage_metrics_comparison:
choose: 0
5_scatter_diagram:
choose: 0
x-axis: ate_mean
y-axis: cpu_mean
6_scatter_diagram:
choose: 1
extend_choose: 1
extend_multiple:
- 1
- 2
- 2.5
- 3
extend_threshold:
- 0.75
- 0.5
- 0.25
x-axis: cpu_mean
y-axis: ate_mean
7_3d_scatter_diagram:
choose: 1
extend_choose: 1
extend_multiple:
- 1
- 2
- 2.5
- 3
extend_threshold:
- 0.75
- 0.5
- 0.25
x-axis: general+image_width
y-axis: general+image_frequency
z-axis: ate_mean
8_repeatability_test:
choose: 0
metric: null
algorithm_dataset_type: 3
group_description: 10 modes of 4 vision algorithm on 5 sequences; 5 image rate and
6 image resolution - totally 1500 configurations
group_name: Paper_Vision_SLAM_Image_Exploration_Experiment_limitation