Evaluation, Reproducibility, Benchmarks Meeting 14

Minutes of meeting 14

Date: 24th November 2021

Present: Nicola, Jorge, Lena

Initial draft put up for discussion by Lena:
- 1. Raw metric analyses
  - Analyzing per class
  - Making sample dependencies transparent
- 2. Aggregation
  - Respecting hierarchical data structure
  - Handling missing values
  - Dealing with unequal class importance and frequency
  - Considering dependencies between classes
- 3. Stratification
  - Handling biases
  - Leveraging meta data
- 4. Comparing and ranking (optional)
  - Respecting relations between metrics
  - Performing statistical tests
  - Performing uncertainty-aware rankings
  - Considering non-determinism of ML
  - Considering clinical relevance
- 5. Reporting
  - Choosing reporting precision
  - Reporting on the quality of the reference
  - Extrapolating results
Jorge: need to distinguish between requirements and recommendations
- high risk -> requirement; otherwise recommendation
- Can we make a self-assessment form such that people can assess risks and thus decide on requirements vs recommendation?
Open: How to consider clinical relevance

Copyright (c) MONAI Consortium