ICE-proof handling of spot terminations

[cristim]

Github issue

Issue type

• Feature Idea

Summary

Currently AutoSpotting may not be able to properly handle sudden spot terminations that escalate into an ICE for the initial on-demand instance type.

The idea is to intercept these ICE events with the recently introduced event-based model, find the group they were happening for and launch spot in that group or even fallback to on-demand instances of different instance types to compensate for the sudden capacity loss in that group.