Scaling down approach in AKS

[danpowell88]

We're running our ARC in AKS and as such we aren't able to control the default scheduler (which seems to be ) to take advantage of bin packing.

Ideally we'd be able to tell it to use the MostAllocated strategy and then run something like descheduler to kill off idle pods and have them start up on higher allocated nodes so the other ones can be scaled down.

Curious how others are doing this or any alternative approaches that could work, currently we have our min runners set to 0 to work around this but thats really not ideal from a job startup point of view.

The only thing that comes to mind for me is something like this.

1. Hook into runner job start scripts to set a pod annotation telling us the pod is "running"
2. Deploy a K8s job that periodically runs and cordons a node, checks if all of the pods are not "running" and if so drains the node the deletes it