Make schedulers composable

[htejun]

(This is more of a high-level / vague design discussion than something which is immediately implementable)

We are seeing three macro layers of scheduling emerging among the implemented schedulers:

1. Timeline management: Primarily concerned with scheduling tasks on a group of tightly coupled processors. scx_lavd, scx_bpfland, scx_rustland and scx_rlfifo fall in this category.
2. Load balancing: Given multiple scheduling domains, each running its own timeline management, balance the load across them. scx_rusty.
3. Soft partitioning: Soft partition a machine to serve several distinct workload sets on the machine. scx_layered and scx_mitosis.

Note that the boundary lines aren't black and white. Timeline schedulers are adding topology awareness and scx_rusty borrowed timeline code from lavd. However, for the most part, outside of the main layer of focus, the existing schedulers are doing something minimal to get by - e.g. scx_layered has a really rudimentary timeline implementation and is currently in the process of adding some topology awareness.

These layers stack on top of each other quite well. Load balancing doesn't really have to care what timeline management is doing inside each load balancing domain, and soft partitioning doesn't have to care what load balancing and timeline management are doing within a soft partition.

Instead of trying to build each scheduler to do everything, it'd be great if we can come up with a framework so that different layers can be composed together so that, e.g, scx_layered can use rusty to load balance in each soft partition and rusty in turn can use lavd or bpfland to manage the timeline on each load balancing domain.

• The framework shouldn't be too imposing and allow as much flexibility and latitude in each layer.
• Making BPF code composable is challenging. BPF folks say that freplace can be used to implement quasi callback mechanism and we likely won't have to resort to #include and large switch blocks.