Weird behaviour when migrating Flux from "master" to "main" branch

[oliverfernandez]

I am trying to switch the branch Flux reconciles from, moving from master to main

The steps I followed are

1. Create the new main branch in my repo, and set it as default branch
2. In the new main branch, I updated my gotk-sync.yaml to set ref: main for GitRepository
3. I landed the 2. in master as well

Once I did 3. , Flux started to reconcile from main, and I see resources being created matching my configuration from main branch properly.

However, my source-controller is still checking master for reconciliation. I see these logs in my source-controller POD

{"level":"info","ts":"2024-11-13T17:56:37.952Z","msg":"no changes since last reconcilation: observed revision 'main@sha1:b1f0710726842240e48c9cf173492ecb065c8d78'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"78755e0c-6a23-454d-a8b5-025446aada32"}

And

{"level":"info","ts":"2024-11-13T17:10:56.946Z","msg":"no changes since last reconcilation: observed revision 'master@sha1:50a0553338ea27211812ffc0d6003d283799786c'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"bc911ce9-1fdc-4baf-9ba2-0c59b5ce88e1"}

Furthermore, I see my Kubernetes deployments still trying to create replicasets from a reference from master, together with main

I probably messed with Flux state with this branch migration, but I'm unable to figure out how to fix it.

[stefanprodan]

From the logs timestamp looks like the last revision comes from main. To verify that the branch has been set correcty in cluster do flux export source git flux-system, if there is not main you can edit it directly in-cluster and then run flux reconcile source git flux-system. In any case, please double check that main is set in Git in both branches, before editing the in-cluster object.

[oliverfernandez]

Yes, main is set as ref in both branches

When I run flux export source git flux-system I get the following

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    reconcile.fluxcd.io/requestedAt: "2024-11-13T16:34:42.937524Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  timeout: 1m0s
  url: ssh://git@*****

Anyway, I run flux reconcile source git flux-system and got

► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:0c9f21cfc1b6e0d07cf017e57e4b53e09dd5d253

But the problem persists...

Today I still see logs from my source-controller trying to reconcile from master branch

{"level":"info","ts":"2024-11-14T06:47:46.599Z","msg":"no changes since last reconcilation: observed revision 'master@sha1:50a0553338ea27211812ffc0d6003d283799786c'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"91822972-5ee2-43e4-b868-98d8120fe485"}
{"level":"info","ts":"2024-11-14T09:30:17.680Z","msg":"no changes since last reconcilation: observed revision 'master@sha1:50a0553338ea27211812ffc0d6003d283799786c'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"ae6dae56-9d76-4fdb-b8df-4c18b6808c6b"}
{"level":"info","ts":"2024-11-14T10:39:52.010Z","msg":"no changes since last reconcilation: observed revision 'master@sha1:50a0553338ea27211812ffc0d6003d283799786c'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"f426c0de-ebe7-4db4-8525-e781d7ccd412"}

It puzzles me that the timestamp of those logs are very irregular, opposed to the logs from the main branch, which are every 1 minute (matching my configuration)

{"level":"info","ts":"2024-11-14T10:38:48.880Z","msg":"no changes since last reconcilation: observed revision 'main@sha1:0c9f21cfc1b6e0d07cf017e57e4b53e09dd5d253'","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"5a5e3222-cda6-4284-9447-39db20ab3748"}
{"level":"info","ts":"2024-11-14T10:38:57.832Z","msg":"artifact up-to-date with remote revision: '19.1.3'","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-redis-main","namespace":"default"},"namespace":"default","name":"default-redis-main","reconcileID":"89458662-5a8b-4d09-811e-3e6e90f6939c"}
{"level":"info","ts":"2024-11-14T10:40:00.225Z","msg":"artifact up-to-date with remote revision: '19.1.3'","controller":"helmchart","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"HelmChart","HelmChart":{"name":"default-redis-main","namespace":"default"},"namespace":"default","name":"default-redis-main","reconcileID":"ac2418d0-75fa-43f0-8642-9ad7132ee720"}

Is there something else I can try?

[stefanprodan]

Hmm this is really strange, maybe the controller-runtime cache got corrupted, delete the source-controller pod to force a cache reset.

[oliverfernandez]

Deleting the source-controller POD didn't fix the issue.

What finally fixed the issue today is to uninstall Flux from the Kubernetes cluster, and bootstrap it again.