Differences

This shows you the differences between two versions of the page.

--- customanalysis [2024/06/03 10:54] – created liuxzh12023
+++ customanalysis [2025/03/30 14:01] (current) – liuxzh12023
@@ Line 1: / Line 1: @@
-Some Text about Custom Analysis!
+===== Custom Analysis =====
+SLAM-Hive system provides meta analysis function named "Custom Analysis".
+You can use the historical results in database to do more comprehensive analysis.
+In our website https://slam-hive.net, we provide some examples of different types of custom analysis for you.
+------
+==== View-only Version ====
+In the system, we first provide some pre-created analysis results, and you can browse them.
+Note: You can also create the custom analysis task in our "view-only" version, but the result will be not shown at the "Group List".
+Instead, when the task finished, the system will return you the task ID, and you should remember it.
+And when you want to see your result, you can input the link: https://slam-hive.net/analysis/show/YOUR_ID.
+And if you build you own SLAM Hive website and you want to show some your analysis results on it, you should change their YAML file's "is_visualized" from 0 to 1 locally. Then they can be shown in group list.
+=== 1. Create Custom Analysis Task ===
+Here we provide you with a basic YAML file to show you how to create a task, and in [[customanalysis:example | Example]] we provide more detailed examples to show you how to create and use different types of analysis task.
+<file yaml>
+group_name: Simple_example  # custom analysis task name.
+group_description: xxxxxx # some simple description of the task.
+evaluation_form:
+  # you should choose 0-3 based on your configurations.
+  algorithm_dataset_type: 0 # 0: same_algorithm_same_dataset.          can choose: 1 2 3 4 6 7 8
+                          # 1: different_algorithm_same dataset        can choose: 1 2 3 4 6 7
+                          # 2: same_algorithm_different_dataset        can choose: 3 6 7
+                          # 3: different_algorithm_different_dataset.  can choose: 3 6 7
+_trajectory_comparison: # compare the trajectories.
+    choose: 0 # choose this analysis (0) or not (1)
+_accuracy_metrics_comparison: # compare the ATE and RPE metrics of trajectories
+    choose: 0
+_accuracy_metrics_comparison: # generate a histogram of different (algorithm + dataset) group.
+    choose: 0
+    calculate_method: 1 # 0: average (because one algo + one dataset may include different configs)
+                      # 1: choose the best one
+                      # then for each (algorithm, dataset), get one configuration, then calculate the average value
+    metric: ate_rmse # eg: ATE-RMSE
+    algorithm_id: [12] # choose the algorithms by ID
+    dataset_id: [2] # choose the dataset sequences by ID
+_usage_metrics_comparison: # compare the CPU and Memory usage.
+    choose: 0
+_scatter_diagram: # generate a scatter to show the metrics of different configurations' task.
+    choose: 1
+    x-axis: general+image_frequency # X axis's metric
+    y-axis: ate_mean # Y axis's metric
+    # You can choose: 1) Evo metrics; 2) Resource Usage metrics; 3) Configuration parameters (provide the parameter template's name; and you should choose one that all configurations use this template).
+_3d_scatter_diagram: # 3D version of the scatter
+    choose: 1
+    x-axis: general+image_width
+    y-axis: general+imu_frequency
+    z-axis: ate_mean
+  # For the scatter and 3D scatter, when you create a analysis, you can also use this task to create more scatters with other axis's metrics; And you can also see the online dynamic scatter on the webpage.
+_repeatability_test: # Test one configuration's stability.
+    choose: 0 # If you use this analysis, you should only choose one configuration.
+    metric: ate_mean
+#
+configuration_choose: # implicit limitation：chosen configs has running task and evaluation
+  # three ways to choose confg
+  configuration_id: [] # provide configurations id
+  comb_configuration_id: [72] # provide combination configurations id (Equal to choose all configurations ID contained by these comb configs).
+  limitation_rules:   # by a rule;
+    algorithm_id: [12, 13]
+    dataset_id: [2,3]
+    parameters_value: ["nFeatures < 4000"]
+    evaluation_value:
+      ate_rmse_nolimitation: 1              # 0 or 1
+      ate_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      ate_rmse_maximun:
+      ate_mean_nolimitation:    1           # 0 or 1
+      ate_mean_minimum:
+      ate_mean_maximun:
+      ate_median_nolimitation:   1           # 0 or 1
+      ate_median_minimum:
+      ate_median_maximun:
+      ate_std_nolimitation:   1           # 0 or 1
+      ate_std_minimum:
+      ate_std_maximun:
+      ate_min_nolimitation:    1          # 0 or 1
+      ate_min_minimum:
+      ate_min_maximun:
+      ate_max_nolimitation:  1            # 0 or 1
+      ate_max_minimum:
+      ate_max_maximun:
+      ate_sse_nolimitation:    1          # 0 or 1
+      ate_sse_minimum:
+      ate_sse_maximun:
+      rpe_rmse_nolimitation:     1         # 0 or 1
+      rpe_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      rpe_rmse_maximun:
+      rpe_mean_nolimitation:    1          # 0 or 1
+      rpe_mean_minimum:
+      rpe_mean_maximun:
+      rpe_median_nolimitation:    1          # 0 or 1
+      rpe_median_minimum:
+      rpe_median_maximun:
+      rpe_std_nolimitation:    1          # 0 or 1
+      rpe_std_minimum:
+      rpe_std_maximun:
+      rpe_min_nolimitation:    1          # 0 or 1
+      rpe_min_minimum:
+      rpe_min_maximun:
+      rpe_max_nolimitation:   1           # 0 or 1
+      rpe_max_minimum:
+      rpe_max_maximun:
+      rpe_sse_nolimitation:     1         # 0 or 1
+      rpe_sse_minimum:
+      rpe_sse_maximun:
+      cpu_max_nolimitation:    1          # 0 or 1
+      cpu_max_minimum:
+      cpu_max_maximun:
+      cpu_mean_nolimitation:    1          # 0 or 1
+      cpu_mean_minimum:
+      cpu_mean_maximun:
+      ram_max_nolimitation:     1         # 0 or 1
+      ram_max_minimum:
+      ram_max_maximun:
+  combination_rule:   # how to combine three ways：U - union set; I - intersection set; complement set;
+    first_one: [1]
+    first_rule: ["U"]
+    second_one: [0,2]
+    second_rule: ["I"] # (1) - (0 I 2)
+</file>
+=== 2. Show the Custom Analysis Results ===
+The "SHOW" webpage shows all the results created by this analysis task.
+You can also download the diagrams (.png .pdf .pgf) and the raw data (.csv).
+== Online Dynamic Diagram Results ==
+For the "6_scatter_diagram" and "7_3d_scatter_diagram", we also provide online dynamic scatters for users. It's more convenient for users to see the detail information of each configurations(points) in the scatter.
+You can also create more scatters reusing this task's rules and choose other metrics or parameters.