SLAM Hive Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Custom Analysis =====

SLAM-Hive system provides meta analysis function named "Custom Analysis".

You can use the historical results in database to do more comprehensive analysis.

In our website https://slam-hive.net, we provide some examples of different types of custom analysis for you.


------

==== View-only Version ====

In the system, we first provide some pre-created analysis results, and you can browse them.

Note: You can also create the custom analysis task in our "view-only" version, but the result will be not shown at the "Group List". 
Instead, when the task finished, the system will return you the task ID, and you should remember it.
And when you want to see your result, you can input the link: https://slam-hive.net/analysis/show/YOUR_ID.
And if you build you own SLAM Hive website and you want to show some your analysis results on it, you should change their YAML file's "is_visualized" from 0 to 1 locally. Then they can be shown in group list.


=== 1. Create Custom Analysis Task ===

Here we provide you with a basic YAML file to show you how to create a task, and in [[customanalysis:example | Example]] we provide more detailed examples to show you how to create and use different types of analysis task.

<file yaml>
group_name: Simple_example  # custom analysis task name.

group_description: xxxxxx # some simple description of the task.

evaluation_form:
  # you should choose 0-3 based on your configurations.
  algorithm_dataset_type: 0 # 0: same_algorithm_same_dataset.          can choose: 1 2 3 4 6 7 8
                          # 1: different_algorithm_same dataset        can choose: 1 2 3 4 6 7
                          # 2: same_algorithm_different_dataset        can choose: 3 6 7
                          # 3: different_algorithm_different_dataset.  can choose: 3 6 7
  1_trajectory_comparison: # compare the trajectories.
    choose: 0 # choose this analysis (0) or not (1)
  
  2_accuracy_metrics_comparison: # compare the ATE and RPE metrics of trajectories
    choose: 0  
  
  3_accuracy_metrics_comparison: # generate a histogram of different (algorithm + dataset) group.
    choose: 0
    calculate_method: 1 # 0: average (because one algo + one dataset may include different configs)
                      # 1: choose the best one   
                      # then for each (algorithm, dataset), get one configuration, then calculate the average value
    metric: ate_rmse # eg: ATE-RMSE
    algorithm_id: [12] # choose the algorithms by ID
    dataset_id: [2] # choose the dataset sequences by ID
  
  4_usage_metrics_comparison: # compare the CPU and Memory usage.
    choose: 0  

  6_scatter_diagram: # generate a scatter to show the metrics of different configurations' task.
    choose: 1
    x-axis: general+image_frequency # X axis's metric
    y-axis: ate_mean # Y axis's metric
    # You can choose: 1) Evo metrics; 2) Resource Usage metrics; 3) Configuration parameters (provide the parameter template's name; and you should choose one that all configurations use this template).
    
  7_3d_scatter_diagram: # 3D version of the scatter
    choose: 1
    x-axis: general+image_width
    y-axis: general+imu_frequency
    z-axis: ate_mean
  # For the scatter and 3D scatter, when you create a analysis, you can also use this task to create more scatters with other axis's metrics; And you can also see the online dynamic scatter on the webpage.  
    
  8_repeatability_test: # Test one configuration's stability.
    choose: 0 # If you use this analysis, you should only choose one configuration.
    metric: ate_mean
    
# 
configuration_choose: # implicit limitation：chosen configs has running task and evaluation
  # three ways to choose confg
  configuration_id: [] # provide configurations id
  comb_configuration_id: [72] # provide combination configurations id (Equal to choose all configurations ID contained by these comb configs).
  limitation_rules:   # by a rule;
    algorithm_id: [12, 13]
    dataset_id: [2,3]
    parameters_value: ["nFeatures < 4000"]
    evaluation_value:
      ate_rmse_nolimitation: 1              # 0 or 1
      ate_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
      ate_rmse_maximun:
      ate_mean_nolimitation:    1           # 0 or 1
      ate_mean_minimum:                  
      ate_mean_maximun:
      ate_median_nolimitation:   1           # 0 or 1
      ate_median_minimum:                  
      ate_median_maximun:
      ate_std_nolimitation:   1           # 0 or 1
      ate_std_minimum:                  
      ate_std_maximun:
      ate_min_nolimitation:    1          # 0 or 1
      ate_min_minimum:                  
      ate_min_maximun:
      ate_max_nolimitation:  1            # 0 or 1
      ate_max_minimum:                  
      ate_max_maximun:
      ate_sse_nolimitation:    1          # 0 or 1
      ate_sse_minimum:                  
      ate_sse_maximun:
      rpe_rmse_nolimitation:     1         # 0 or 1
      rpe_rmse_minimum:                   # if just minimum and no maximum, can just fill the minimum
      rpe_rmse_maximun:
      rpe_mean_nolimitation:    1          # 0 or 1
      rpe_mean_minimum:                  
      rpe_mean_maximun:
      rpe_median_nolimitation:    1          # 0 or 1
      rpe_median_minimum:                  
      rpe_median_maximun:
      rpe_std_nolimitation:    1          # 0 or 1
      rpe_std_minimum:                  
      rpe_std_maximun:
      rpe_min_nolimitation:    1          # 0 or 1
      rpe_min_minimum:                  
      rpe_min_maximun:
      rpe_max_nolimitation:   1           # 0 or 1
      rpe_max_minimum:                  
      rpe_max_maximun:
      rpe_sse_nolimitation:     1         # 0 or 1
      rpe_sse_minimum:                  
      rpe_sse_maximun:
      cpu_max_nolimitation:    1          # 0 or 1
      cpu_max_minimum:                  
      cpu_max_maximun:
      cpu_mean_nolimitation:    1          # 0 or 1
      cpu_mean_minimum:                  
      cpu_mean_maximun:
      ram_max_nolimitation:     1         # 0 or 1
      ram_max_minimum:                  
      ram_max_maximun:
  combination_rule:   # how to combine three ways：U - union set; I - intersection set; complement set;
    first_one: [1]
    first_rule: ["U"]
    second_one: [0,2]
    second_rule: ["I"] # (1) - (0 I 2)
</file>




=== 2. Show the Custom Analysis Results ===

The "SHOW" webpage shows all the results created by this analysis task.
You can also download the diagrams (.png .pdf .pgf) and the raw data (.csv).

== Online Dynamic Diagram Results ==

For the "6_scatter_diagram" and "7_3d_scatter_diagram", we also provide online dynamic scatters for users. It's more convenient for users to see the detail information of each configurations(points) in the scatter.
You can also create more scatters reusing this task's rules and choose other metrics or parameters.