Prevent accidental rollbacks on flux bootstrap

[ahothan]

In a large organization it is difficult to coordinate the version of flux cli used by all actors (developers, testers, automation tools) and the main risk is to see clusters rolling back to older flux versions because the flux cli was not upgraded to the right version.

A typical use case is when a cluster gets deployed with a recent version of flux, then gets uninstalled (flux uninstall) and bootstrapped again but with a stale version of flux cli.

In the current flux version, a flux rollback will happen automatically (ie flux cli will rollback the flux controller versions) without any notice and this may actually cause lots of problems.
For example if the git repo for the cluster has flux Kustomization v1 and a bootstrap is executed with flux cli 0.41.2 on the same path/tag/branch, the deployment will fail in a very messy way.

To prevent this from happening, it would be most useful to add the following checks to flux cli bootstrap:

• if the cluster already runs flux controllers, fail if the corresponding flux version is newer than the flux cli version
• if the cluster does not have flux running, fail if the bootstrap path/branch in the source (git repo, oci...) has a newer flux version

To allow occasional downgrades, we could add an option to skip the above checks (eg flux bootstrap --force)

[benoittgt]

I am wondering if you have answers with #4235 and the 2 merged pull requests.

[stefanprodan]

Yes this has been implemented in Flux 2.3

[ahothan]

this additional check is a great feature, thanks @stefanprodan for adding it.
in this specific use case it does not help because we only use flux bootstrap to deploy flux - including for the first time on a fresh cluster, so we do not have that app.kubernetes.io/managed-by: Helm label.
I just feel like downgrading a flux version with bootstrap should always be prompted and optionally forced.
For now to solve this issue we added flux version check in a wrapper script to flux bootstrap, which works