How to manage internal state on a self-hosted instance

[kingdonb]

A discussion similar to the question:

Solving a chicken-and-egg problem when using Flux pointed to a in-cluster slef-hosted Gitlab that doesn't exist yet?

fluxcd/source-controller#1504 @nogweii - you may be interested in this thread.

We are talking about a GitLab instance that runs on a Kubernetes cluster, and the source of truth for the cluster is in there, as a Git repository. This sounds backwards, but it is probably not that uncommon of a configuration, and we certainly have users trying this now. Why would you host the source of truth inside the cluster? It avoids an external dependency at runtime, I could imagine this scenario on for example a submarine where there are coders who need to work in the airgap, it's not such a strange use case.

It's just not so easy to support, since there is a chicken-and-egg problem. How do you capture enough state from the deployment so that you can re-bootstrap the cluster, restore the source of truth from the external backup, and then pivot to the cluster's internal source of truth? You need some continuous backup process, to be sure that you're not memory-holing changes when the cluster goes down and needs to be restored, but those technologies aren't so exotic, we shouldn't be surprised if users say they do that. We should have some kind of answer for this question.

(So I'm thinking of building a workflow for capturing in-cluster Flux state and echoing it out to OCI, as a backup... but first I want to poll the community and ask how we're solving this problem, if there are already good processes nailed down that target this idea?)

I have a different parallel structure (ClickOps dashboard workflow) and I'd like to GitOps-ify it. Using Cozystack, there is a dashboard where admins can click on options in a catalog, paste in Helm values, run upgrades, and it's all managed using Flux HelmReleases on the cluster, inside of various tenants. This would not be so bad if there was a way to capture all that state, and echo it back out to a versioned store.

For a variety of reasons, I do not want to use Git. I intend to use OCIRepository, you can use OCI or Git, but regardless of how the current desired state is stored and versioned, I hope you can see this is the same problem!

Even if it's a different facet of the general problem: how can we keep the desired state's source of truth in the cluster without compromising on the repeatability and recoverability of that state after a total cluster loss? Can we get some feedback from the community about how this is handled in your orgs. Are there any other obvious use cases for this - the one that comes to mind is a mobile store-front that continues to operate and can receive an update, delivered via sneakernet, even when it is in disconnected mode.

Is this a problem that other people are grappling with? What are the obvious solutions we have overlooked, if there are any?

[nogweii]

This is the workflow I was hoping / imagining existed:

1. I would run flux bootstrap and point it to my git host (Gitlab in my original statement) and so flux would be installed into the cluster and...immediately start failing to fetch the repository, since it's not available. But that wouldn't be fatal, it'd just be noisy in the events & logs.
2. All of the source controllers would support a manual upload API of some sort (a git bundle, a compressed OCI tarball, etc). Getting the backup artifact into the cluster would, I assume, be left as an exercise to the reader. (But, kubectl cp exists. Or maybe flux is extended to streamline that?)
3. I then successfully upload the backup artifact to the source controller, and it immediately turns around and updates it status to offer the artifact as the "latest version". (I guess you'd need to also indicate the version of the artifact? git bundles and oci images probably have that inline, but arbitrary tarballs wouldn't... hm.)
4. The Kustomization, Helm, etc controllers all run and do their thing and provision all of the workloads in the cluster, including my git host, as they can all fetch from the source controller.
5. I may need to update the flux configuration to use a new secret if I had to regenerate API or SSH keys or something. Then, it will finally stop erroring and can successfully download from whatever source.

So basically, I was expecting to be able to do the source controller's job for it, at least at the very beginning. This would allow for someone who is developing their flux configuration on their laptop first before even deploying, or doing an offline restore.

This raises a couple of questions, which I think have known answers:

1. How does a source controller know that version M (manual) is newer or older than version R (retrieved)? Well, force pushes are a thing in git, and deleting or overwriting an object in an OCI store is as well. I don't think that is a large concern.
2. What is the flux team's story for high-sec offline bootstrapping? e.g. I'm a paranoid NSA IT guy tasked with rebuilding our Kubernetes cluster, but disconnected from the world. I have a massive NAS of container images, so that's not a concern. Is the expectation there would also be some service running on the NAS to provide an API to download flux configs?

[nogweii]

This is assuming that running flux bootstrap will generate in-cluster GitRepository & Kustomization objects that are functionally identical to whatever is stored in the backup so that when flux manages flux it doesn't need to change anything. Oh, and that the version of flux in gotk-components.yaml is the same (or fully backwards compatible, I guess?).

[kingdonb]

I was thinking, just point it at a source that is online, which describes everything that needs to come online (and pivots to repoint itself at the GitLab once enough is online that it can be used as the source again)

[nogweii]

That could work, but you'd need to figure out the mechanism for doing the transition from source A to B.

And that still requires a dependency external to the cluster to bootstrap it, which was my original hope to avoid.

[NikoKS]

@nogweii, have you find any solution that works for you?

[nogweii]

I have not found a solution yet, but I've also distracted myself with a cluster rebuild (switching to Talos) and a full network rebuild. 😅 Lots of churn, so my flux deployment has been on hold for the past couple of months.

Hopefully I'll soon have everything up and running, but that's not the case right now.

[studioph]

I'm glad I'm not the only person who wants to keep things self-hosted and limit the number and importance of external services.

I would imagine that it should be possible to edit/update the running Flux config resource(s) to point to a different git repository, assuming the content is the same.

My current plan to try is having an instance of Gitea running in Docker on my laptop to point Flux to initially, mirror/migrate the repo to the "prod" Gitea in the cluster once it's stood up by Flux, then update the Flux sync resource to point to the in-cluster instance.

This is based on my poking around the helm charts and reading things like fluxcd-community/helm-charts#88 which seem to indicate that the GitRepository resource created by the flux2-sync chart is equivalent to what's done during the bootstrap command.

I could be wrong about all of that, will report back whether it worked.

[NikoKS]

@studioph, please do update your findings 😁

[studioph]

So I was able to successfully "migrate" Flux from a "bootstrap" git repo to a "prod" repo with the following steps:

1. Have an existing git repo to pass to flux bootstrap. In my case it was a Docker instance of Gitea

• I learned the hard way that supposedly you can't use HTTP basic auth with plain HTTP (not HTTPS) when bootstrapping. Unsure if this is a Flux limitation or a go-git limitation, since there are plenty of examples of using basic auth with plain HTTP on the interwebs.
• Since setting up TLS was way more work than I wanted to do for this experiment I switched to SSH.

2. Run flux bootstrap with your temporary git repo

• Make sure you added the SSH key you want to use to the temp git repo

3. Add (or maybe it already exists) whatever self-hosted git solution you want (again for me it was Gitea)

• Wait for Flux to standup the self-hosted git
• I left the gitea SSH service as the default ClusterIP since it only needs to be accessible to Flux for this test anyways. This also means you don't need to also standup ingress, loadbalancers, etc to get flux working with it.

Here's where it can get a bit tricky with the ordering:
4. Update gotk-sync.yaml flux created in the bootstrap git repo with the the URL of the self-hosted git repo

• Note that you haven't created the new repo yet in the self-hosted git install
• At this point Flux will sync and start erroring that it can't connect to the self-hosted repo

5. Mirror/pull the bootstrap git repo to the self-hosted git install. For me this was doing New Migration in Gitea

• I have them in this order that way the self-hosted repo state is already pointing to itself
• At this point Flux should error when trying to connect to the self-hosted repo due to old values in the secret

6. Edit the flux-system secret:

• If you're using the same SSH key for the self-hosted repo, all you have to update is the known_hosts key. You can get the appropriate entry for your self-hosted git install by running ssh-keyscan <GIT_SERVER> | base64 (ignore the lines it prints that start with #)
• If using a different SSH key, you'll also have to update the identity and identity.pub keys with your private the public key respectively

At this point Flux should now be syncing with your self-hosted git repo.

I will say having gone through this exercise and poked around some other people's Github repos that are using Flux for their homelabs that this way is probably more roundabout than is needed. I remember seeing a comment on an issue or discussion in this repo that bootstrap is just a convenience concept and not actually necessary if you think about it.

I've come across some people that "bootstrap" by installing flux via helm/manifests where the flux configs are also in that same repo.

It would make sense then that you could install flux and your self-hosted git service initially with something like helmfile, then just push/import to the self-hosted repo, and from there flux will take over. While it does mean having 2 "copies" of some configuration that installs the service to the cluster, it can't result in that much extra work? I suppose it all comes down to how complicated/extensive the helm values overrides for the git service are, and how many dependent components (like storage) also need to be stood up at the same time.

[bzp99]

@studioph thanks for sharing, this is exactly what I was looking for. The missing link for me was using my workstation for a temporary Git repository. Perhaps I would not even go so far as to spin up a Dockerized Gitea and just create a simple bare repository that Flux can reach from the cluster.

The only inconvenience seems to be having to migrate repositories during the process but I imagine having a bootstrap-only and an ‘everything’ repo where one is a subset of the other is a viable solution. But it also means keeping the two repos in sync.

I was unfortunately too impatient to wait for your solution to be posted here and just went with the route of using flux install and then flux create commands to set up things ‘directly’ (but still exporting everything flux create sent to the cluster using --export for later inclusion in my new GitOps repo). For me this was a number of things including some storage management (NFS subdir provisioner), ingress-nginx, Keycloak, and then finally Gitea. Finally, I created a new Git repo in the new Gitea instance, ran flux bootstrap with that, and pushed all the previously exported YAMLs for the ‘manually’ deployed services.

[studioph]

Yea what you did is similar to the conclusion I reached where you install the bare minimum components needed to run your self-hosted git in some other fashion, then install flux and let it take over. I'm going to try doing that with helmfile next when I get some more free time.