add kep 4986 sakkara cluster topology

[atantawi]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add kep for Sakkara plugin, a hierarchical cluster topology group scheduler

Which issue(s) this PR fixes:

Fixes #4986
kubernetes/enhancements#4986

Special notes for your reviewer:

NONE

Does this PR introduce a user-facing change?

NONE

[k8s-ci-robot]

Hi @atantawi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-sigs-scheduler-plugins canceled.

Name	Link
🔨 Latest commit	02fd878
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-scheduler-plugins/deploys/6794159a1819a100082e534f

[googs1025]

/ok-to-test

[atantawi]

/retest-required

[atantawi]

/retest

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atantawi
Once this PR has been reviewed and has the lgtm label, please assign denkensk for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[13567436138]

when to implement this plugin？

[atantawi]

when to implement this plugin？

I am planning to raise a PR and put the code in there shortly.

[13567436138]

@atantawi awesome

[googs1025]

/cc

I will try to understand this plugin.

[13567436138]

How short
I can not wait

[atantawi]

/retest

[k8s-ci-robot]

@atantawi: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-scheduler-plugins-verify	02fd878	link	true	/test pull-scheduler-plugins-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[googs1025]

initial thought for me is: can similar problems be solved by using existing affinity or anti-affinity or Topology Spread Constraints? But this seems to be "multi-level", and there are many things to consider. 🤔

[googs1025]

I am not sure whether clusters need such a complex definition (at least I do not have such a complex definition internally... this might be my problem). In my work, there is only a one-dimensional level, which is described as "node pool". And I optimize scheduling based on this dimension. 🤔

[googs1025]

one question: will the number of configmaps for group constraints increase as more group settings are added? Can we merge configs into the same one? More configs sometime mean more difficult maintenance

[atantawi]

There is only one configMap (alternatively, a CR or an extension of the PodGroup) per group. Group constraints is a key in the map (or a spec).

[atantawi]

How short I can not wait

#877

[Huang-Wei]

Not a blocker for the KEP, but I highly recommend to start with CRD since day 1.

[atantawi]

Agreed. Should we introduce a new CR or extend the PodGroup?

[Huang-Wei]

How do we present the topological info if the cluster has a couple of thousands nodes?

[atantawi]

Good question! We need a tool to generate/update the tree definition. Or, we should consider an alternative way to code the tree. One possibility is to have a node (a leaf in the cluster topology tree) code the path to the root as labels. Problem is the plugin will need to create the tree every time a group needs to be scheduled. (Unless it's cached and updated whenever node labels change.)

[Huang-Wei]

It's an anti-pattern to update a pod's labels to store extra info. Update a pod is expensive, no need to say updating a gang of pods.

[atantawi]

The most important data is the rank. Having it as a pod label simplifies the logic of the container getting it initially as an environment variable. Alternatively, rank values for all pods in the group may be written in the CR for the group. The challenge is for the containers to get them? I guess if we're using a configmap for the group, the container could mount it.

[Huang-Wei]

After reading the Design section, I still cannot figure out the exact responsibility of each phase. Like, which phase does the "pre-scheduling" to update the pod's label, what if the "pre-scheduling" result is in conflict with the real-time scheduling decision, how preemption works, etc.

It's quite essential to illustrate: by given a group of pods carrying the group constraints, in an e2e manner how they're processed in each phase of a scheduling cycle.

[atantawi]

OK, I'll work on providing a clarifying diagram and maybe a table with description.

[alculquicondor]

cc @dom4ha @sanposhiho

[dom4ha]

/cc @johnbelamaric
There are a few aspects that we should discuss regarding the overall approach to topology aware scheduling

1. Resource management in scheduler becomes more complex, mainly because of rapid DRA development. Some of the resource types that are being introduced already enforce topology constraints implicitly (see KEP-4815 DRA Partitionable devices support for multi-host kubernetes/enhancements#5069). This mean that we cannot assume that there is only one topology.
2. DRA becomes a new method to define and manage resource (especially dynamically allocated), but also, it introduces a new way of expressing resource requirements and evolve toward covering all type of resources that pods may need to run.
3. We (and a few other parties) are currently exploring a possibility of pre-allocating (reserving) resources. There is a proposal for making the first such attempt to leverage DRA framework for that. Note that reservation not necessarily need to consider final Pods at the time of allocation, but may need to consider topology (directly or indirectly)
4. More fine grained allocations (not proposed yet anywhere) could become dynamically scheduled/allocated (somehow) and will not leave too much space for scheduler to decide on the final pods placement.
5. Scheduler might play the crucial role in dynamic scheduling/allocation of such resources/reservations, but this is a very new and vague idea not discussed/brainstormed on the community forum yet (except the last SIG meeting). Note however that ResourceClaims themselves are in fact a sort of reservation method already.
6. Finally, any TAS implementation makes sense when we talk about group/gang scheduling. This concept is not well defined yet at Kubernetes level, so before we provide solutions for it, we need to introduce such well thought-out concept, which may be a challenge in itself. I know that gang-scheduling can be modeled in the current scheduler framework, but not sure if this is sufficient for long term maintainabliity.

I believe that TAS scheduling is important and it's great that you are sharing your approach and offering contribution. I believe that we should think how the proposal interacts with the aspects that I mentioned (frankly, I know questions but not answers yet). I'm especially curious whether the reservation ideas is something that we want to pursue (as community).

To scope the discussion for now, my major concern is whether the proposal that is based on topology scheduling enclosed into a single plugin is capable to take into consideration external constraints (including implicit topology constraints). IIUC the solver knows the topology, knows the group of pods, but does it know about external constraints put by other plugins?