Differences

This shows you the differences between two versions of the page.

--- customanalysis [2024/07/03 05:43] – liuxzh12023
+++ customanalysis [2025/03/30 14:01] (current) – liuxzh12023
@@ Line 3: / Line 3: @@
 SLAM-Hive system provides meta analysis function named "Custom Analysis".
-You can use the historical results in database to do comprehensive analysis.
+You can use the historical results in database to do more comprehensive analysis.
+In our website https://slam-hive.net, we provide some examples of different types of custom analysis for you.
 ------
@@ Line 11: / Line 14: @@
 In the system, we first provide some pre-created analysis results, and you can browse them.
-Note: You can also create the custom analysis task in our "view-only" version, but the result will be not shown at the "Group List". Instead, when the task finishes, the system will return you the task ID, and you can
+Note: You can also create the custom analysis task in our "view-only" version, but the result will be not shown at the "Group List".
+Instead, when the task finished, the system will return you the task ID, and you should remember it.
+And when you want to see your result, you can input the link: https://slam-hive.net/analysis/show/YOUR_ID.
+And if you build you own SLAM Hive website and you want to show some your analysis results on it, you should change their YAML file's "is_visualized" from 0 to 1 locally. Then they can be shown in group list.
-=== 1. Mapping Task List ===
-The "**Mapping Task List**" shows all the created mapping run tasks.
+=== 1. Create Custom Analysis Task ===
-You can click "**Show Detail**" button to see the details of the result. Details: [[mappingtask:detail | mapping task result detail wiki]].
+Here we provide you with a basic YAML file to show you how to create a task, and in [[customanalysis:example | Example]] we provide more detailed examples to show you how to create and use different types of analysis task.
-You can also see the mapping tasks' state (Running or Finished or Failed).
+<file yaml>
+group_name: Simple_example  # custom analysis task name.
-=== 2. Search Engine for Mapping Task ===
+group_description: xxxxxx # some simple description of the task.
-You can use our "Search Engine" (shown as a search box) to search your required mapping tasks conveniently.
+evaluation_form:
+  # you should choose 0-3 based on your configurations.
+  algorithm_dataset_type: 0 # 0: same_algorithm_same_dataset.          can choose: 1 2 3 4 6 7 8
+                          # 1: different_algorithm_same dataset        can choose: 1 2 3 4 6 7
+                          # 2: same_algorithm_different_dataset        can choose: 3 6 7
+                          # 3: different_algorithm_different_dataset.  can choose: 3 6 7
+_trajectory_comparison: # compare the trajectories.
+    choose: 0 # choose this analysis (0) or not (1)
+_accuracy_metrics_comparison: # compare the ATE and RPE metrics of trajectories
+    choose: 0
+_accuracy_metrics_comparison: # generate a histogram of different (algorithm + dataset) group.
+    choose: 0
+    calculate_method: 1 # 0: average (because one algo + one dataset may include different configs)
+                      # 1: choose the best one
+                      # then for each (algorithm, dataset), get one configuration, then calculate the average value
+    metric: ate_rmse # eg: ATE-RMSE
+    algorithm_id: [12] # choose the algorithms by ID
+    dataset_id: [2] # choose the dataset sequences by ID
+_usage_metrics_comparison: # compare the CPU and Memory usage.
+    choose: 0
-The system provides multiple search conditions:
+_scatter_diagram: # generate a scatter to show the metrics of different configurations' task.
-  - Algorithms Selection (If no limitation for algorithm, select "All").
+    choose: 1
-  - Datasets Selection (If no limitation for algorithm, select "All").
+    x-axis: general+image_frequency # X axis's metric
-  - Parameter Selection: input a set of constraints on the parameters (The rules are similar to the "SQL").
+    y-axis: ate_mean # Y axis's metric
+    # You can choose: 1) Evo metrics; 2) Resource Usage metrics; 3) Configuration parameters (provide the parameter template's name; and you should choose one that all configurations use this template).
-For example:
-<file>
+_3d_scatter_diagram: # 3D version of the scatter
-nFeatures >=2000
+    choose: 1
-nLevels = 8
+    x-axis: general+image_width
-...
+    y-axis: general+imu_frequency
+    z-axis: ate_mean
+  # For the scatter and 3D scatter, when you create a analysis, you can also use this task to create more scatters with other axis's metrics; And you can also see the online dynamic scatter on the webpage.
+_repeatability_test: # Test one configuration's stability.
+    choose: 0 # If you use this analysis, you should only choose one configuration.
+    metric: ate_mean
+#
+configuration_choose: # implicit limitation：chosen configs has running task and evaluation
+  # three ways to choose confg
+  configuration_id: [] # provide configurations id
+  comb_configuration_id: [72] # provide combination configurations id (Equal to choose all configurations ID contained by these comb configs).
+  limitation_rules:   # by a rule;
+    algorithm_id: [12, 13]
+    dataset_id: [2,3]
+    parameters_value: ["nFeatures < 4000"]
+    evaluation_value:
+      ate_rmse_nolimitation: 1              # 0 or 1
+      ate_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      ate_rmse_maximun:
+      ate_mean_nolimitation:    1           # 0 or 1
+      ate_mean_minimum:
+      ate_mean_maximun:
+      ate_median_nolimitation:   1           # 0 or 1
+      ate_median_minimum:
+      ate_median_maximun:
+      ate_std_nolimitation:   1           # 0 or 1
+      ate_std_minimum:
+      ate_std_maximun:
+      ate_min_nolimitation:    1          # 0 or 1
+      ate_min_minimum:
+      ate_min_maximun:
+      ate_max_nolimitation:  1            # 0 or 1
+      ate_max_minimum:
+      ate_max_maximun:
+      ate_sse_nolimitation:    1          # 0 or 1
+      ate_sse_minimum:
+      ate_sse_maximun:
+      rpe_rmse_nolimitation:     1         # 0 or 1
+      rpe_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
+      rpe_rmse_maximun:
+      rpe_mean_nolimitation:    1          # 0 or 1
+      rpe_mean_minimum:
+      rpe_mean_maximun:
+      rpe_median_nolimitation:    1          # 0 or 1
+      rpe_median_minimum:
+      rpe_median_maximun:
+      rpe_std_nolimitation:    1          # 0 or 1
+      rpe_std_minimum:
+      rpe_std_maximun:
+      rpe_min_nolimitation:    1          # 0 or 1
+      rpe_min_minimum:
+      rpe_min_maximun:
+      rpe_max_nolimitation:   1           # 0 or 1
+      rpe_max_minimum:
+      rpe_max_maximun:
+      rpe_sse_nolimitation:     1         # 0 or 1
+      rpe_sse_minimum:
+      rpe_sse_maximun:
+      cpu_max_nolimitation:    1          # 0 or 1
+      cpu_max_minimum:
+      cpu_max_maximun:
+      cpu_mean_nolimitation:    1          # 0 or 1
+      cpu_mean_minimum:
+      cpu_mean_maximun:
+      ram_max_nolimitation:     1         # 0 or 1
+      ram_max_minimum:
+      ram_max_maximun:
+  combination_rule:   # how to combine three ways：U - union set; I - intersection set; complement set;
+    first_one: [1]
+    first_rule: ["U"]
+    second_one: [0,2]
+    second_rule: ["I"] # (1) - (0 I 2)
 </file>
-------
-==== Full Version ====
-In this page, you can create the evaluation in various methods.
-Now we provide "evo" methods to evaluation the trajectory.
-=== 1 Create One Evaluation ===
-You can click the "**Create**" (in Mapping Task List) to create a evaluation task.
-See [[ evaluation: | Evaluation wiki]] for more details.
+=== 2. Show the Custom Analysis Results ===
-=== 2 Create a Batch of Evaluation Tasks ===
+The "SHOW" webpage shows all the results created by this analysis task.
-We provide "Batch Create Evaluation Task" for you to create evaluation tasks in batches. You can input the IDs of mapping tasks and then click the button, then the evaluation tasks will run.
+You can also download the diagrams (.png .pdf .pgf) and the raw data (.csv).
+== Online Dynamic Diagram Results ==
-=== 3 Create Compared Evaluation Task  ===
+For the "6_scatter_diagram" and "7_3d_scatter_diagram", we also provide online dynamic scatters for users. It's more convenient for users to see the detail information of each configurations(points) in the scatter.
-You can also compared these trajectories in same charts.
+You can also create more scatters reusing this task's rules and choose other metrics or parameters.
-We provide "Create Multiple Evaluation Task" for you to create compared evaluation task. You can input the IDs of mapping tasks and then click the button, then the evaluation tasks will run.