Evaluation, Reproducibility, Benchmarks Meeting 4

Minutes of meeting 4

Date: 23rd September 2020

Present: Lena Maier-Hein (Lead), Jens Petersen, Michela Antonelli, Nicola Rieke, Annika Reinke, Carole Sudre

---

TOP 1: WG slide(s) for MONAI Bootcamp

• Bootcamp: ~200 participants; slides will be on YouTube and other channels
• 1st slide:
  • Members + roles (lead, taskforce leads) with pictures
  • Mission statement
• 2nd slide:
  • Goals
  • Mailing list: Annika will create a dkfz mailing list - [email protected]
  • Link to survey with feedback "What do you want from MONAI":
    • Feedback from participants
      • Problems and pitfalls on existing metrics
      • Problems wrt benchmarking
      • Problems wrt challenge organization and participation
    • What do you want from MONAI?
      • Metrics
        • BPR for metrics
        • Reference implementations of metrics
        • Metrics for which tasks (MC question/ranking)
          • Segmentation
          • Classification
          • ...
      • Benchmarking and reproducibility
        • Easy access to existing data sets
        • Infrastructure for benchmarking on existing data sets (including results of SOA methods)
        • Model zoo
      • Support for challenge organization
      • Other (free text)

---

TOP 2: Task force reports

Easy data access task force (Michela)

• Finalized the poll for challenge organizers: https://docs.google.com/forms/d/e/1FAIpQLSffBrP6dRF5R8yzmUj_y67BbsF4RiEX3UGPfYZcxyMYBS8JUQ/viewform :
  • Section 1: General questions
  • Section 2: Technical questions regarding data licence and data access (optional)
  • TODOs:
    • Add comment box for yes/no questions
    • Add privacy policy
    • Add year and website (if accessible)
    • Differentiate between training/test data
    • Add possibility to insert multiple challenge names
  • Think of moving the poll to surveymonkey
  • Next steps:
    • Revise poll
    • Prepare email draft
    • Send to organizers

Metrics task force (Carole):

• Focused on segmentation metrics first
• Put issues on github
• Working on NA handling
• Next step: Regression metrics

Benchmarking task force (Annika):

• Next step: Work on stable release of visualization tool and present it

---

TOP 3: DELPHI process on metrics with BIAS initiative

• Further members
• Which tasks? => Wait for feedback of MONAI Bootcamp
  • Segmentation
  • Classification
  • Detection
  • ...
• Workshop
  • Goals:
    • List of metrics for each task
    • List of advantages and disadvantages
    • List of implementations
    • Strategy for DELPHI plan
    • Similar/complementary metrics, when to use which
• Date:
  • Find slots in November/December (Send poll)