Evaluation, Reproducibility, Benchmarks Meeting 15

Minutes of meeting 15

Date: 23rd February 2022

Present: Nicola, Carole, Keyvan, Lena, Annika

---

TOP 1: MONAI metric implementation

*Delphi recommendation framework: List of metrics including reference implementation (desired: MONAI implementation for all metrics ready by paper submission)

• Provide list of metrics to MONAI developers
  • We need to identify how the metrics can be implemented and the optimal implementation
  • → Take this to the expert groups: Carole will formulate the required questions. We should ask them:
    • Ambiguities in the implementation
    • Reference implementation
    • Reference publication
    • If a metric can be implemented in multiple ways: Which ones? Which one is preferable?
    • Metric prioritization
  • Idea: Write follow-up paper with implementation details, lead by Carole

---

TOP 2: Semantic segmentation metric mapping

• Question: How should we deal with lots of text?
• For the toolkit it is preferable to provide all discussion points
• For paper: Refer to paragraph/additional figure/table
• Main paper: Only include the meta road
  • One figure with the complete process: Generate FP → Identify problem category → Problem category specific meta roads (keep it simple, e.g. “Select overlap metric (if any) and only keep very straightforward sub-processes (e.g. volume)) → Branches come together again for aggregation