Evaluation, Reproducibility, Benchmarks Meeting 3

Minutes of meeting 3

Date: 22nd July 2020

Present: Lena Maier-Hein (Lead), David Zimmerer, Jens Petersen, Michela Antonelli, Nicola Rieke, Ada

---

TOP 1: Task force reports

Easy data access task force (Michela)

• Amazon committed to host data with multi-region access
• Poll to challenge organizers in progress
• Brainstorming on questions to MICCAI challenge organizers
  • Michela presents current draft
  • WG complemented poll
• Decision for task force to finalize the poll and then contact challenge organizers

Metrics task force (Carole):

• Task force met several times and added documents to Google Drive
• Focus on data formats so far
• Decision to start with segmentation metrics
• Decision to provide description o metrics to MONAI development team such that they can take care of actual implementation/integration
• Decision to link metric task force with BIAS initiative

---

TOP 2: Update from data working group (Carole)

• Meeting of data working group and benchmarking working group representatives:
  • Decision to separate description of data and description of experiments
  • Decision to have agile process for defining data structure/formats
  • Decision to have one MIC (MSD) and one CAI (ROBUST-MIS) example

---

TOP 3: Structure of working group

• Current task forces are fine:
  • Quick data access (lead: Michela) --> Nicola will join
  • Metrics (lead: Carole)
  • Benchmarking (lead: Annika)
• Decision to pause meeting in August (task forces meet separately)

---

TOP 4: DELPHI process on metrics with BIAS initiative

• Members: Carole, Jens, Jorge, Annika, Lena, Michela, Nicola