helm chart for flux2

[nab-gha]

Some customers exclusively use helm to deploy applications to Kubernetes. They will want be able to install gotk using helm.

At the customer I'm working with I manually created a helm chart that containing source-controller, helm-controller and the Kraan controller that uses it, see https://github.com/fidelity/kraan/tree/master/chart. I added facility to provide http proxy in the chart. This approach involves some manual steps when upgrading the source-controller and helm-controller in the Kraan application.

Maybe we could add functionality to the toolkit installer to generate helm charts so that a public helm chart for the gotk components can be maintained?

[hiddeco]

At this moment we do not plan to maintain a Helm chart in the foreseeable future.

Helm charts tend to be maintenance (and support) heavy, and in combination with the limitations around rolling out Custom Resource Definitions it is not a perfect fit for any of the GitOps Toolkit components at present. Introducing a way to generate the Hem chart would not reduce this, and will still result in people wanting to add features to the chart generator, as proven by your:

I added facility to provide http proxy in the chart. This approach involves some manual steps when upgrading the source-controller and helm-controller in the Kraan application.

I strongly believe it should be possible for the users that really need it to collaborate on a "unofficial" Helm chart until we may have an official chart one day. That is the magic of OSS. This means the maintainers can focus on things that serve all users, as there are countless things that still need to be done.

[nab-gha]

@hiddeco I agree that this is lower priority than adding functionality to gotk.

The problem with an "unofficial" approach is that it quickly becomes out of date as gotk evolves.

Would the maintainers of gotk be open to a community contribution to the installer to generate helm charts?

[hiddeco]

That would circle back to

Introducing a way to generate the Hem chart would not reduce this, and will still result in people wanting to add features to the chart generator, [...]

for which I am not really in favor. But this has not been debated so the view of the others may be different.

[scottrigby]

A big part of the benefit of Helm's post-renderer feature is allowing users to add their use cases without every imaginable use case having to be added to a chart. I'll check in with the team about this, as this approach significantly reduces chart maintenance.

[squaremo]

My experience from Flux's Helm chart is that the text templates pretty quickly become difficult to maintain, and the values specification a giant cruftball. The only remedy for this I can come up with is to make your programs have (next to) nothing configurable, and stand firm on adding non-functional embellishments.

I don't think adding kustomize to the mix makes the situation better -- it expands the contract of the chart to its entire output, since any change could break a patch somewhere.

[scottrigby]

Agree, charts need to be well maintained to stay simple. Luckily we have someone on our team with lots of experience maintaining Helm chart source and repos 😄

[scottrigby]

related: link to discussion on this from last Flux meeting (Nov 5th), see last bullet item in notes.

[scottrigby]

Another fun note. We currently also have other ways of installing Flux than the CLI, for example the Terraform provider. And in dev mode, you may also install Flux with just kubectl (then create git repo history on cluster, then place file you created with kubectl in your git repo), the same steps that would be required with a Flux 2 Helm chart.

[hiddeco]

The Terraform provider re-uses part of the Flux CLI, as it makes use of the manifestgen packages to generate the manifests. It requires thus much less maintenance as it builds upon already existing bits and pieces: Terraform module <- manifestgen <- Kustomize bases.

The same applies to the kubectl part, as we offer a simple generated install.yaml with every release (see attachments on https://github.com/fluxcd/flux2/releases), that is again, generated from the Kustomize bases.

[dmccaffery]

A few comments:

1. There are some serious flaws in the current kustomizations:

• ClusterRole / ClusterRoleBinding is used regardless of whether or not watch-all-namespaces is true
• ClusterRoleBinding is associated with the default service account (bad ju-ju)
• The name of cluster-wide resources is hardcoded and not configurable (without many handcrafted patches) -- which means that multiple installations into different namespaces will step on each other and could break each other if the versions do not remain synchronised

2. There are also some serious flaws with the design of the terraform provider -- mostly as a result of 1 above

• the gavinbunney/kubectl provider that is recommended is naive and doesn't ensure ownership of resources contrary to the design intent of terraform -- multiple applies from different plans can overwrite the same resources (even in the same namespace) -- or resources not created with terraform in the first place
• this conflates terraform design with kubernetes design in horrible ways that are inconsistent:

3. Agree that any implementation of a helm chart SHOULD NOT exceed the capabilities of the kustomize generation and baseline requirements of the system, except to implement best practices around labels, annotations, and naming conventions of resources as the kustomization should be doing anyway -- any extension of the default capabilities should be achieved via parent charts owned / maintained by the customer and/or post-renderer.
4. Helm, at its core, is just go templates -- it doesn't have to be any more complicated than the go templates that are already defined in the src: https://github.com/fluxcd/flux2/blob/main/pkg/manifestgen/install/templates.go#L27 -- I would argue that kustomize is considerably more difficult to write and maintain -- the patches construct is fine for extremely simple scenarios, but falls over with even the slightest hint of complexity (such as a feature flag) -- lets hope the watch-all-namespaces argument stays at index 0 forever, for instance:

   flux2/pkg/manifestgen/install/templates.go

   Line 65 in bd41406

   path: /spec/template/spec/containers/0/args/0

As of now, the installation and maintenance mechanisms provided for flux2 do not meet the needs of any large-scale deployment. We manage clusters with thousands of pods and over a hundred engineers. These are not multi-tenant clusters, either. They are very large product-specific clusters.

[cwrau]

You really took the words out of my mouth with number 4, that's why I really don't get the hype around kustomize.

Sure, if you just need to change the names of the resources and have common labels then it's perfect, but anything even slightly more complex is way out of scope

And as you pointed out with the reference, even when using kustomize, people still use templates, so why not use Helm immediately? That way, it'd be just one standard tool instead of a standard tool and non-standard workarounds around that tool

[dmccaffery]

To move forward, I'm happy to contribute to a community-driven helm chart -- the question is where this chart should live. Would a new organisation called "fluxcd-community" be appropriate (similar to prometheus-community)?

[dmccaffery]

Hmm. If what you say is true and the versions of the controllers, cli, and crds must all exactly align across all controllers in order to function; then this is a distributed monolith.

I'll rethink the strategy, then. My assumption was that each controller operated independently and that any api between them would limit breaking changes. In other words, it should be safe to upgrade one controller independently of the others as long as it's not versioned as a breaking change.

I don't understand why these are broken down into poly repos if they are tightly coupled in this way. Regardless, I will rethink the strategy. I was purposefully avoiding tight coupling under a false assumption that this was the design intent.

[stefanprodan]

it should be safe to upgrade one controller independently of the others as long as it's not versioned as a breaking change

In theory this is true, but as all reconciliation APIs are depending on the source ones this could break if it's not validated by e2e.

[hiddeco]

Hmm. If what you say is true and the versions of the controllers, cli, and crds must all exactly align across all controllers in order to function; then this is a distributed monolith.
I don't understand why these are broken down into poly repos if they are tightly coupled in this way.

They are all depending on unstable APIs that are still under development, this means things may be added to a controller another controller depends on, and from that moment having a version equal or higher to when a feature was introduced becomes a requirement to support all features.

This does not make it a monolith, it simply means the services are all still under active development. Once they reach 1.0.0, this problem goes away.

[makkes]

We are currently using flux install --export and doing some transformations on the results. Maybe that could serve as an automation start?

.PHONY: flux-update
.ONESHELL:
flux-update: $(FLUX_BIN)
	tmpdir=$(shell mktemp -d)
	curdir=$(shell pwd)
	cd $${tmpdir}
	$${curdir}/$(FLUX_BIN) install -n flux-system --export > flux.yaml
	$${curdir}/$(KUBERNETES_SPLIT_YAML_BIN) --outdir=./flux --tpl_flat "{{.X.Outdir}}/flux-{{.X.ShortKind}}_{{.Metadata.Name}}.yaml" flux.yaml
	rm flux/flux-namespace_flux-system.yaml
	find flux -name '*.yaml' -not -name 'flux-crd_*.yaml' -print0 | xargs -0 sed -i 's/flux-system/{{ .Release.Namespace }}/'
	chmod 0664 ./flux/flux-*.yaml
	rm -rf $${curdir}/chart/templates/flux/* $${curdir}/chart/crds/flux/*
	mv ./flux/flux-crd* $${curdir}/chart/crds/flux/
	mv ./flux/flux-*.yaml $${curdir}/chart/templates/flux/
	cd $${curdir}
	rm -rf $${tmpdir}

[pierluigilenoci]

In case there is a need, I am also available to participate in the maintenance of the chart.

[mewzherder]

@hiddeco sorry to come in late, but since @alisondy @dholbach @staceypotter and I are working to build out Contributor Experience for Flux, this naming will create a lot of confusion. Can we support the suggestions for fluxcd-community instead? If I understand the repo correctly, these are community contributions, and not about building out a positive contributor experience (which is what we commonly call ContribuX). Note that this is coming from a discussion that we're having amongst us w/ @scottrigby and others. Thanks for the consideration. ;)

[pierluigilenoci]

Is there any news about this?
In case of need, I am also available to maintain the chart.

[haarchri]

@stefanprodan can you create the contrib repo ? afterwards i am also available to maintain the chart with the others together - feedback is welcome

[pierluigilenoci]

@hiddeco thank you

[dmccaffery]

I've been meaning to merge mine but have been pulled into other things. Now that the dust has settled around flux v2, I'm happy to bring in what I have as a starting point.

[pierluigilenoci]

thanks @dmccaffery

[haarchri]

thanks - my idea is to merge first an very basic helm-chart like the following https://github.com/haarchri/helm-charts - and then we can start to work for generator and more flexible configuration - what you guys think ? if we would go i can create the PR in community repo

[pierluigilenoci]

I'm ok with that.

[haarchri]

we released a very basic pre-alpha 0.0.1 version available here: https://github.com/fluxcd-community/helm-charts - for all feel to open issues & create PRs

[pierluigilenoci]

@paulcarlton-ww @haarchri @stefanprodan @hiddeco at this point I believe that this issue can be closed.

[elliottpope]

Can this discussion be labeled with the new area/install label since it took me a while to find this and I was specifically looking at why the Helm chart is not officially supported to inform whether or not I use Terraform or Helm to install Flux v2