Differences

This shows you the differences between two versions of the page.

--- customanalysis [2024/07/23 10:20] – liuxzh12023
+++ customanalysis [2025/03/30 14:01] (current) – liuxzh12023
@@ Line 4: / Line 4: @@
 You can use the historical results in database to do more comprehensive analysis.
+In our website https://slam-hive.net, we provide some examples of different types of custom analysis for you.
@@ Line 15: / Line 17: @@
 Instead, when the task finished, the system will return you the task ID, and you should remember it.
 And when you want to see your result, you can input the link: https://slam-hive.net/analysis/show/YOUR_ID.
+And if you build you own SLAM Hive website and you want to show some your analysis results on it, you should change their YAML file's "is_visualized" from 0 to 1 locally. Then they can be shown in group list.
+=== 1. Create Custom Analysis Task ===
-=== 1. Create  ===
+Here we provide you with a basic YAML file to show you how to create a task, and in [[customanalysis:example | Example]] we provide more detailed examples to show you how to create and use different types of analysis task.
-The "**Mapping Task List**" shows all the created mapping run tasks.
+<file yaml>
+group_name: Simple_example  # custom analysis task name.
-You can click "**Show Detail**" button to see the details of the result. Details: [[mappingtask:detail | mapping task result detail wiki]].
+group_description: xxxxxx # some simple description of the task.
-You can also see the mapping tasks' state (Running or Finished or Failed).
+evaluation_form:
+  # you should choose 0-3 based on your configurations.
+  algorithm_dataset_type: 0 # 0: same_algorithm_same_dataset.          can choose: 1 2 3 4 6 7 8
+                          # 1: different_algorithm_same dataset        can choose: 1 2 3 4 6 7
+                          # 2: same_algorithm_different_dataset        can choose: 3 6 7
+                          # 3: different_algorithm_different_dataset.  can choose: 3 6 7
+_trajectory_comparison: # compare the trajectories.
+    choose: 0 # choose this analysis (0) or not (1)
+_accuracy_metrics_comparison: # compare the ATE and RPE metrics of trajectories
+    choose: 0
+_accuracy_metrics_comparison: # generate a histogram of different (algorithm + dataset) group.
+    choose: 0
+    calculate_method: 1 # 0: average (because one algo + one dataset may include different configs)
+                      # 1: choose the best one
+                      # then for each (algorithm, dataset), get one configuration, then calculate the average value
+    metric: ate_rmse # eg: ATE-RMSE
+    algorithm_id: [12] # choose the algorithms by ID
+    dataset_id: [2] # choose the dataset sequences by ID
+_usage_metrics_comparison: # compare the CPU and Memory usage.
+    choose: 0
-=== 2. Search Engine for Mapping Task ===
+_scatter_diagram: # generate a scatter to show the metrics of different configurations' task.
+    choose: 1
-You can use our "Search Engine" (shown as a search box) to search your required mapping tasks conveniently.
+    x-axis: general+image_frequency # X axis's metric
+    y-axis: ate_mean # Y axis's metric
-The system provides multiple search conditions:
+    # You can choose: 1) Evo metrics; 2) Resource Usage metrics; 3) Configuration parameters (provide the parameter template's name; and you should choose one that all configurations use this template).
-  - Algorithms Selection (If no limitation for algorithm, select "All").
-  - Datasets Selection (If no limitation for algorithm, select "All").
+_3d_scatter_diagram: # 3D version of the scatter
-  - Parameter Selection: input a set of constraints on the parameters (The rules are similar to the "SQL").
+    choose: 1
+    x-axis: general+image_width
-For example:
+    y-axis: general+imu_frequency
-<file>
+    z-axis: ate_mean
-nFeatures >=2000
+  # For the scatter and 3D scatter, when you create a analysis, you can also use this task to create more scatters with other axis's metrics; And you can also see the online dynamic scatter on the webpage.
-nLevels = 8
-...
+_repeatability_test: # Test one configuration's stability.
+    choose: 0 # If you use this analysis, you should only choose one configuration.
+    metric: ate_mean
+#
+configuration_choose: # implicit limitation：chosen configs has running task and evaluation
+  # three ways to choose confg
+  configuration_id: [] # provide configurations id
+  comb_configuration_id: [72] # provide combination configurations id (Equal to choose all configurations ID contained by these comb configs).
+  limitation_rules:   # by a rule;
+    algorithm_id: [12, 13]
+    dataset_id: [2,3]
+    parameters_value: ["nFeatures < 4000"]
+    evaluation_value:
+      ate_rmse_nolimitation: 1              # 0 or 1
+      ate_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      ate_rmse_maximun:
+      ate_mean_nolimitation:    1           # 0 or 1
+      ate_mean_minimum:
+      ate_mean_maximun:
+      ate_median_nolimitation:   1           # 0 or 1
+      ate_median_minimum:
+      ate_median_maximun:
+      ate_std_nolimitation:   1           # 0 or 1
+      ate_std_minimum:
+      ate_std_maximun:
+      ate_min_nolimitation:    1          # 0 or 1
+      ate_min_minimum:
+      ate_min_maximun:
+      ate_max_nolimitation:  1            # 0 or 1
+      ate_max_minimum:
+      ate_max_maximun:
+      ate_sse_nolimitation:    1          # 0 or 1
+      ate_sse_minimum:
+      ate_sse_maximun:
+      rpe_rmse_nolimitation:     1         # 0 or 1
+      rpe_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      rpe_rmse_maximun:
+      rpe_mean_nolimitation:    1          # 0 or 1
+      rpe_mean_minimum:
+      rpe_mean_maximun:
+      rpe_median_nolimitation:    1          # 0 or 1
+      rpe_median_minimum:
+      rpe_median_maximun:
+      rpe_std_nolimitation:    1          # 0 or 1
+      rpe_std_minimum:
+      rpe_std_maximun:
+      rpe_min_nolimitation:    1          # 0 or 1
+      rpe_min_minimum:
+      rpe_min_maximun:
+      rpe_max_nolimitation:   1           # 0 or 1
+      rpe_max_minimum:
+      rpe_max_maximun:
+      rpe_sse_nolimitation:     1         # 0 or 1
+      rpe_sse_minimum:
+      rpe_sse_maximun:
+      cpu_max_nolimitation:    1          # 0 or 1
+      cpu_max_minimum:
+      cpu_max_maximun:
+      cpu_mean_nolimitation:    1          # 0 or 1
+      cpu_mean_minimum:
+      cpu_mean_maximun:
+      ram_max_nolimitation:     1         # 0 or 1
+      ram_max_minimum:
+      ram_max_maximun:
+  combination_rule:   # how to combine three ways：U - union set; I - intersection set; complement set;
+    first_one: [1]
+    first_rule: ["U"]
+    second_one: [0,2]
+    second_rule: ["I"] # (1) - (0 I 2)
 </file>
-------
-==== Full Version ====
-In this page, you can create the evaluation in various methods.
-Now we provide "evo" methods to evaluation the trajectory.
-=== 1 Create One Evaluation ===
-You can click the "**Create**" (in Mapping Task List) to create a evaluation task.
-See [[ evaluation: | Evaluation wiki]] for more details.
+=== 2. Show the Custom Analysis Results ===
-=== 2 Create a Batch of Evaluation Tasks ===
+The "SHOW" webpage shows all the results created by this analysis task.
-We provide "Batch Create Evaluation Task" for you to create evaluation tasks in batches. You can input the IDs of mapping tasks and then click the button, then the evaluation tasks will run.
+You can also download the diagrams (.png .pdf .pgf) and the raw data (.csv).
+== Online Dynamic Diagram Results ==
-=== 3 Create Compared Evaluation Task  ===
+For the "6_scatter_diagram" and "7_3d_scatter_diagram", we also provide online dynamic scatters for users. It's more convenient for users to see the detail information of each configurations(points) in the scatter.
-You can also compared these trajectories in same charts.
+You can also create more scatters reusing this task's rules and choose other metrics or parameters.
-We provide "Create Multiple Evaluation Task" for you to create compared evaluation task. You can input the IDs of mapping tasks and then click the button, then the evaluation tasks will run.