Improving Performance Analysis

[chapman39]

This is an issue tracking features, tasks, discussions relating to improving Serac's benchmarking suite. The following list are tasks/ features (more or less in order in which they should be done):

• Run benchmarks in CI (currently only compiling, not running)
• Name resulting .cali files something more useful than 240904-185751_3556490_bmOIfLlF37Y2.cali
• Adapt current benchmarks to be longer (10+ minutes)
• Start tracking benchmarks in a shared location on LC by creating cron jobs (dane, lassen) that will run weekly as service user, using combinations of
  • gcc, clang
  • toss4, blueos, toss4_cray
  • 1, 4, 16 tasks
  • ?
• Document which solvers/ preconditioners perform best Create documentation and Performance test suite #1193
• Utilize Hatchet or Thicket to compare benchmarks between a PR and develop, to see whether your changes improve/ worsen performance
  • Create easy script to call for this
• Run benchmarks with a "short" (<5 seconds to run) and "long" option (>30 minutes to run), so we can regularly test the benchmarks work, but also get useful timings from them on a weekly basis
• Automate some sort of "archive" system to regularly snapshot the current collection of shared caliper files into a tarball (the advent of accidentally modifying/ removing the cali files). Could be a weekly/ monthly cron job.
• Caliper files should include Serac commit metadata, so we can track roughly when changes in performance has occurred Track serac version and compiler when benchmarking #1240