Implement support for an emergency killswitch.

[patnebe]

Context

• Implement an emergency killswitch mechanism to stop the agent if a file is on disk and prevent it from starting
• This can be used as a quick rollback/backout mechanism if any issues are encountered post-deployment

Changes

• Add a new CLI argument specifying the path to the killswitch file
• Split main into code paths -- continuous_profling_mode (assumed if no duration is specified) vs adhoc_profiling_mode
• Move the non-continuous profiling code path into a new method called collect_profiles_adhoc
• Add a new Runner object which will be used to run the profiler when in continuous profiling mode. Within the runner
  • Prevent the profiler from starting if the kill switch file is detected
  • Start the profiler in a separate thread
  • Check for the kill switch file every 30 seconds, and stop/start the profiler depending on the profiler_run_state.

Test Plan

• CI
• Manual test

[patnebe]

Some thoughts on this.

Wrapping the Profiler in an Arc<Mutex<>> + Profiler::run() being a blocking call means that the mutex won't be unlocked until Profiler::run() returns or the thread running the profiler exits. One side effect is that only one instance of the profiler can be run at any given time. This sounds like desirable behaviour, but maybe there are reasons why we wouldn't want this.

[patnebe]

I see one major downside of creating the profiler + the profiler_stop_signal* channel endpoints outside the Runner struct and passing them in.

In theory, the profiler can be stopped if a signal is sent to it from outside this class resulting in the runner thinking that the profiler is running when it's not. Will think more about how best to handle this.

[patnebe]

Maybe keep track of the join handle so we can cleanup?

[javierhonduco]

Thanks for implementing this feature. I'll review this PR more in detail once it's ready. Something that we should consider is that perhaps the killswitch file should be in a hardcoded path rather than being configurable. This could help as a well-known location that doesn't change could speed up enabling the killswitch in a fool-proof way.

To handle the possibility of running in on-off mode with a killswitch (which is something that we might or might not want to allow), perhaps a flag could be added (--unsafe / --i-know-what-i-am-doing) that allows running the profiler even in the presence of a killswitch. What do you think?