[SURE-9228] Fleet pod fleet-cleanup does not receive tolerations from values.yaml

[kkaempf]

SURE-9228

Issue description:

When adding a specific toleration to values.tolerations, the fleet-cleanup-clusterregitrations pod does not use them. See here for the code and the image attached for an example.

Business impact:

The pod cannot run in a cluster where all nodes are tainted.

Repro steps:

• Create a new cluster using k3s/rke2
• Taint the node
• Install rancher

Files, logs, traces

-- see JIRA --

Actual behavior:

Pod does not run in a cluster with all nodes tainted and chart installation fails.

Expected behavior:

Pod uses the tolerations from values.tolerations and can run in a tainted node.

[weyfonk]

Additional QA

Problem

Installing Fleet charts in a cluster where all nodes are tainted would fail, because Fleet jobs did not support node tolerations coming from chart values. This would result in Fleet failing to install.
Affected jobs were the cleanup jobs for cluster registrations and completed jobs.

Solution

The Fleet chart now propagates node tolerations from chart values to those jobs, as it does for other deployed pods.

Testing

Engineering Testing

Manual Testing

This has been tested by tainting one node in a k3d cluster, and checking that:

• deployment was successful
• jobs were not scheduled on the tainted node, unless the corresponding toleration was set in chart values

Automated Testing

N/A

QA Testing Considerations

It would make sense to test this in a cluster where all nodes are tainted:

• if no tolerations are specified in chart values, running the above mentioned jobs should still fail
• setting tolerations matching at least one node's taints should enable successful deployment, with both jobs being created and run successfully.

Regressions Considerations

N/A

[mmartin24]

When checked with Rancher v2.11-1d345932e5f78b3da07ebe13691c8224c62b6240-head / fleet:106.0.0+up0.12.0-alpha.4 the fix could not be observed because the toleration value is not being passed to the fleet-cleanup job via helm operation pod. This needs to be addressed by another team.

We tried to pass the tolerations when installing Rancher via extraTolerations flag and observed that Fleet had those values correctly passed; however, the helm-operation pod did not get scheduled in any of the nodes and therefore no fleet clean-up job/pod was created.

Then we tried manually adding the toleration to the helm operation pod and saw that the toleration value was still not being passed to the fleet- clean-job

[manno]

Waiting for fixes on helm-operations pods and #3313 (comment).

[weyfonk]

Additional QA

Problem

Installing Fleet charts in a cluster where all nodes are tainted would fail, because Fleet jobs did not support node tolerations coming from chart values. This would result in Fleet failing to install. Affected jobs were the cleanup jobs for cluster registrations and completed jobs.

Solution

The Fleet chart now propagates node tolerations from chart values to those jobs, as it does for other deployed pods.

Testing

Engineering Testing

Manual Testing

This has been tested by tainting one node in a k3d cluster, and checking that:

* deployment was successful

* jobs were not scheduled on the tainted node, unless the corresponding toleration was set in chart values

Automated Testing

N/A

QA Testing Considerations

It would make sense to test this in a cluster where all nodes are tainted:

* if no tolerations are specified in chart values, running the above mentioned jobs should still fail

* setting tolerations matching at least one node's taints should enable successful deployment, with both jobs being created and run successfully.

Regressions Considerations

N/A

This can also be tested as follows, without needing to install any chart:

1. Add tolerations to a separate values file, eg. values.yaml
2. Ensure that helm repo list lists https://rancher.github.io/fleet-helm-charts/; if not, add it.
3. Refresh Helm repositories with helm repo update
4. Check the Fleet Helm repo for its latest available version, e.g. helm search repo fleet-repo --devel
5. Use that latest version to template a Fleet chart installation with helm template fleet fleet-repo/fleet --version=$latest_version (where $latest_version is the version found in the previous step), checking that no extra tolerations are applied to jobs fleet-cleanup-clusterregistrations and job_cleanup_gitrepojobs.yaml
6. Repeat the previous step with flag -f values.yaml and check that tolerations set in that file are applied to both jobs