Faster reconciliation of dependency chains

[ajhall]

I have a moderately long chain of Kustomization dependencies (let's say crds -> system -> cilium -> istio -> workloads), and I make most of my changes to the workloads Kustomization. Whenever I push a new commit, I have to wait for the whole dependency chain to reconcile before my change to workloads gets applied. This means that every push takes a few minutes to reconcile.

I see a few possible ways to handle this.

Existing options:

• Break the dependency chain: Even though crds -> system -> cilium -> istio are hard dependencies for workloads, they change less often and are usually up to date already. In the cases where workloads needs something newly introduced by one of those dependencies, I could just let it fail and retry until the other Kustomizations are done.
• Lower the retryInterval: Make the Kustomizations later in the chain retry their reconciliation more frequently to minimize the downtime between Kustomizations.

My ideal options would be more like this:

• A manual method: Add an additional parameter to flux reconcile kustomization that figures out the dependency chain and reconciles the whole thing in sequence. For example, flux reconcile kustomization workloads --with-dependencies would reconcile crds, then system, then cilium, then istio, then workloads.
• An automatic method: When a Kustomization finishes reconciliation, make the kustomize-controller look for downstream dependencies and mark them for immediate reconciliation as well. That way we wouldn't have to wait for the retryInterval on each Kustomization, it would just be triggered for reconciliation as soon as its dependencies are done. This seems like the nicest solution to me.

I could probably contribute code toward these ideas if they seem viable.