Support custom tools for file reconcilation

[rverma-dev]

This proposal is to make flux expand its tooling for custom configuration files. There are multiple projects which are providing a custom configuration file e.g.

1. Istioctl : CRD IstioOperator
2. Eksctl: CRD EKSCluster

So far we can provide a hint to flux which tool to use for applying whether it's kubectl/kustomize/helm/terraform.... by creating its own controller, but it has its own challenges and further the integration of new controller to a runtime binary is not well document.

Can we introduce a way to extend the flux controller for this specific use case, there are multiple options

1. Explore kustomize plugins (seems like WIP Optionally enable external plugins kustomize-controller#163)
2. An annotation-based approach combined with overriding base image (add additional binaries directly) and use annotations like (flux-apply: istioctl, flux-apply-args: [])...
3. Have a way to generate custom system-controller(s) with a binary and args (Preferred Approach)

Please share if there are any existing ways of doing this...

[kingdonb]

Flux does not run external binaries; if you want this behavior in your CI workflow then it will have to run in a CI environment. If your resources can be applied to the cluster in a declarative way (if istioctl has an --export option?) then you can import them to the cluster via GitOps and Flux. But if there are binaries that need to run during the deployment, Flux will not be able to do that. It is for security concerns, this is part of the design of Flux to avoid RCE and other issues that can occur when you shell out to run another binary and pass it parameters.

You can either do it in CI (the CI workflow can run the tasks) or you can use something like Terraform Controller and load those binaries into your runner pods – only the AWS cli is included for now as I understand it. You will be able to accomplish this if you build your own runner pods, (but I believe the security is locked down by default so that you should not be able to load binaries into the container image at runtime with an initContainer for example)

[rverma-dev]

Isn't this behaviour is skewed, we implemented terraform controller mostly because some maintainers needed it but what if one needed a CDK/cloudformation/pulumi controller etc... Shouldn't we find a way for the community to adopt and extend flux with the right set of APIs/extensions.

[kingdonb]

Perhaps I was too hasty to reject this proposal, I only meant to confer the policy and position of current Flux controllers from today. We welcome those types of integrations to come from the community, and we support them with our documentation and best-practices advice which I'll be happy to share more about around how to build controllers and Kubernetes operators.

Adding a parallel effort alongside of kustomize-controller to support one of these other targets CDK/CF/Pulumi sounds like a great idea! I think that is the intention of the design, that there could be more than just one Kustomize controller (like Helm controller, we already have more than one "applier" type controller in Flux), but certain possibilities where the workloads are not necessarily on Kubernetes (like tf-controller) have a use case that skews farther from Kubernetes. It is unlikely that the current Flux team will spend much time focusing there, we are focused on Kubernetes and cloud-native tools mostly under the umbrella of the CNCF.

Some of the tools that you mentioned are for deploying onto Kubernetes as well as other targets, and some might not be used in Kubernetes at all. This is a broadening of scope (but importantly, with the tf-controller as prime example, this does not mean they cannot join the GitOps Toolkit extended family of controllers.)

Some of the other tools that you mentioned also have integration points that exist which work through the Kubernetes API – like EKSctl which as I understand it can be controlled via Kubernetes resources that you place in Git, through the Cluster API project. These are similar projects to Flux, they are controllers or operators based on kubebuilder. Shelling out to binaries is generally considered as a negative pattern in Flux at large, because the translation from typed data to string and back again when performing shell exec is such a fraught and dangerous pattern. The temptation is to pass some information as an argument, and then you have to make sure that it is properly escaped, while sometimes this is unavoidable, it is better to use a Go library to preserve the type safety and prevent that whole class of issues. This is mainly why you don't find custom tools or guidance around how to bundle in custom tools with Flux.

A few of the projects that you mentioned like CDK could integrate with Flux already through a CI pipeline for certain use cases. We saw guidance published for a while suggesting (I published some of this) that you use a CI job that runs a manifest generator in CI and commits to Git, where cdk8s can run for example. Now we have OCI support in Flux which accomplishes the same thing in a way, but in a more secure and performant model, generally keeping Git upstream of CI (and container registries/OCI repositories downstream) which is a big change for Flux users who have invested in automation.

These are big new ideas and boy do we ever need help to get the message out around those. The solution to your issue may already be in there, maybe just a thin veneer left between the problem you're experiencing and how Flux could better support it. People are thinking of ways to bundle all sorts of information inside of OCI artifacts that may integrate with other downstream projects like the ones you mentioned in a way that doesn't require Flux to change anything about its design.

I've avoided peppering links throughout this text because I don't want to come off the wrong way, but if you do have questions about any part of these ideas that you'd like to know more than other parts, please consider me an open book, I share whatever information I can to get Flux community members better integrated and help make more parts of the community better developed and more inclusive. We've explored all these ideas before and are open to exploring them again with whomever is most interested. I try to be responsive on Slack, and we have many community members who are equally invested and interested to help evaluate and re-align proposals from interested volunteers that pop up in the community.

The Flux team is focused on GA, making Flux as secure and stable as possible, so that enterprises and infrastructure builders can make the best use of GitOps Toolkit as well as Flux – broadly so greater adoption can take hold and Flux can be the trusted name in Kubernetes CD, both for end-users and for cloud vendors as well as infrastructure providers that may fall somewhere between those.

I have more thoughts, but I wanted to push "send" and get my main points across before completing the Great English Novel.

[kingdonb]

(Now here comes the hard sell)

Creating a new controller flatly requires one or more dedicated maintainers to own it in order for a successful controller project to take shape. The biggest problem adding new integration points in Flux is that if nobody from the Flux team actively uses that integration and tests it every day, when the target is a popular tool in active development, those integration points are generally not stable enough for Flux maintainers to build atop of in a set-and-forget way that never requires our attention to maintain or fix for the foreseeable future. Every new release from the partner project would require us to test that integration again, and breaking changes from upstream which we will have to force our users to cope with are an even further possibility; this comes up again for every new integration added and the risks compound even further the more exciting and actively developed projects get some integration that hooks directly into Flux.

So, we would need companies or individuals finding enough value in these custom or external tools, perhaps from folks who are also Flux adopters, to step up and volunteer to participate in the development of such a controller. The maintainer of an open source project is a necessarily volunteer position, and we can expect that only someone who is truly invested in a positive outcome would be able to steward an effort such as this, or otherwise take ownership in order for such a project to come to fruition. (This is the point where I believe one should put the "call to action" 👈 pick one you believe strongly in, pitch it to the community, and/or spend some time coding it up. Make an investment!) One of the most important and original considerations across Flux but also all of Open Source is that whomever does the work gets to decide how. This is the importance of volunteers. Absolutely nothing happens without them.

The Flux project has also been focused on providing clear guidance around how to build controllers. Part of GA focus is developing this best-practice advice on how to build a controller, documenting and disseminating that advice, but also ensuring that our own set of CRDs and controllers follow the standards we set out as rigorously and as closely as we can, to ensure we practice what we preach and set a good example, from an expectation that people will sometimes read Flux code, then mimic and perform a similar dance in their own separate context, some CDK controller or similar endeavor. We have begun to see the fruits of this general effort through tf-controller that you also mentioned – it should generally try to conform to all the guidance that we push in fluxcd/pkg and elsewhere.

So in principle I agree that we should support as many such integration points as possible, but I would stop far short of saying those tools need to be integrated directly in the Kustomize Controller which has an already defined mission and scope, and is now edging towards a stable release. It cannot have scope creep involving an ever-expanding suite of target integrations, while still providing a cohesive, stable, focused experience that is consistent for all users and well-documented from end to end.

We want to see more projects like tf-controller, absolutely. Further – we are also under pressure from the CNCF to broaden the pool of maintainers and contributors, inside or outside of the companies that already participate in Flux development. This is not in any way the same position as arguing that those integrations do not belong in Flux! We are definitely here for all of that fun stuff, but we need to do it in a way that instills confidence in the current crop of controllers and maintains predictability for people that have already bet on the stability of Flux v2. This, rather than setting up surprises for our users and integrators expecting a generally stable experience.

So, look to tf-controller for precedent and guidance about the way to implement a new family member that joins the GitOps Toolkit extended family outside of Flux. If you are interested in contributing a controller for one of the projects that you mentioned, that is something we'd very much like to see happen. Maybe one day we see some of this code gets donated to the Flux project, or even starts a new community with its own separate governance?

[rverma-dev]

@kingdonb thanks for a details reply, I do understand the challenges and the alternative strategies. There are few questions I am really puzzled about how to proceed though. But before I shoot let me give the context of our setup.

Our current setup is divided in three phases

1. Landing zone + VPC + TGW + Config + cross account role delegation etc... setup, which is mostly a one time boilerplate code and requires seldom changes. This is currently being done using AWS CDK(cloud formation) + Codepipeline (https://docs.aws.amazon.com/cdk/v2/guide/cdk_pipeline.html)

2. Setup of a master orchestrator clusters ( This is the cluster where we deploy our central services SIEM/LOG Management etc...)
   The master eks cluster is currently bootstrapped through CDK only and its usually doesn't changed a lot. Further we have setup flux manually there and it syncs to master repo. Its same repo where we store eksctl configs for all the team's clusters. Each team's clusters(multiple/multi-tenant) are manually managed by eksctl by a central team ( You mentioned EKSctl which can be controlled via Kubernetes resources that you place in Git, through the Cluster API project, I believe you are talking about CAPA, which is vanilla eks and schematics are quiet different than eksctl config. We in fact looked at projects like CAPA and Crossplane for these kind of setups but eksctl is more api rich, concise and for us is perceived as best tool for this job.

3. Team clusters [ Now this is the hardest part], Further along with each team's cluster teams bootstrap their infrastructures, mostly through AWS CDK (Cloudformation) only. Again Crossplane (too verbose and little less updates than cdk) may look as viable option but our team really loved the conciseness from the CDK L2/L3 stacks. We can migrate to cdktf but the cdktf synth generates too much boiler plot which we want to avoid storing in our git.

I am mostly looking for strategies where we can automate the
a. Second phase i.e bootstrapping/modifying eksctl configs using gitops. Not eksctl has rich set of apis too in addition to binaries. b. Third phase: where we can enable GITOPS for the cdks.

I am really looking for any advises/links on how to handle the tasks in a pure gitops philosophy.

[kingdonb]

Flux itself does not drive imperative commands but Flux-adjacent tf-controller can definitely drive those imperative commands. It is at odds with pure GitOps philosophy to manage artifacts imperatively, as Rule Number 1 at https://opengitops.dev is Declarative Artifacts

You mentioned EKSctl which can be controlled via Kubernetes resources that you place in Git, through the Cluster API project, I believe you are talking about CAPA, which is vanilla eks and schematics are quiet different than eksctl config

That's correct, I was talking about CAPA, and I was not aware of this functional limitation between the projects as I don't use either eksctl or CAPA on a daily basis, but thanks for this clarity as I was missing an important part of understanding your requirements here.

https://github.com/mumoshu/terraform-provider-eksctl Have you considered to try tf-controller with the eksctl provider?

There are other questions here which I'm not prepared to address right now from my own experience, but if you're open to meet after checking out the tf-controller solution, we can try to put you together with an architect and try to get all of your questions answered here. I do work for a company that you may know, and this is a CNCF project but we (humans behind said project) do help businesses and sign contracts sometimes.

I have limited experience with the cloudformation and CDK stuff, I'm not sure if they output an intermediary layer, or if CloudFormation is that layer, how it integrates with tf-controller, or whether you need another tool to work directly with cloudformation if tf-controller can't do it. I have a feeling that all of your questions are in scope for the project.

We can migrate to cdktf but the cdktf synth generates too much boiler plot which we want to avoid storing in our git.

This is what I'm talking about. You're way over my head here right now, I've got some googling to do to follow up and better my own understanding of CDK and ancillary projects now. But it sounds like this could be used with the new OCIRepository resource to avoid putting a bunch of noise you don't want into any GitRepository. This is exactly what we developed the feature in Flux 0.32 for. The tf-controller is rapidly integrating this and my understanding is, it's basically usable with the OCIRepository support now, already, the only thing that's missing are some great examples showing everyone how they're meant to use all these features in concert. So I've got my work cut out for me now 🥸

Thanks for these great questions 👍