Error creating rolebindings

[craigmunro]

We've been successfully using Flux in a multi-tenant cluster for a while now but have come across an issue we can't resolve.

We have the following in a Git repository:

---
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: true
metadata:
  name: ignite
  namespace: example
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: example
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: example
subjects:
- apiGroup: ""
  kind: ServiceAccount
  name: ignite 
roleRef:
  kind: Role 
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

And a Kustomization in the namespace like:

spec:
  force: true
  interval: 1m0s
  path: example
  prune: true
  serviceAccountName: flux
  sourceRef:
    kind: GitRepository
    name: gke-iac
    namespace: example
  suspend: false
  targetNamespace: example
  validation: server

We then get the following:

kubectl get kustomizations.kustomize.toolkit.fluxcd.io gke-iac
NAME      AGE   READY   STATUS
gke-iac   44d   False   RoleBinding/example/read-pods namespace not specified, error: roles.rbac.authorization.k8s.io "pod-reader" not found...

Creating the same resources directly using the same service account works:

kubectl create -f role.yaml --as system:serviceaccount:example:flux
serviceaccount/ignite created
role.rbac.authorization.k8s.io/pod-reader created
rolebinding.rbac.authorization.k8s.io/read-pods created

If the Role is added to Git first and then the RoleBinding is added everything works too.

The kustomize controller has the following additional arguments:

        - --no-cross-namespace-refs=true
        - --default-service-account=flux

We're using Flux v0.35.0 and running on GKE v1.22.12-gke.1200

Are we missing something or is there an issue here?

[stefanprodan]

I've tested this on my cluster and works ok:

$ flux tree ks apps-sync 
Kustomization/flux-system/apps-sync
├── Namespace/apps
├── ServiceAccount/apps/ignite
├── Role/apps/pod-reader
├── RoleBinding/apps/read-pods

My Flux Kustomization is v1beta2 without force nor validation, can you post here the full yaml.

[craigmunro]

Thanks for the prompt response. Removing force and validation didn't make a difference.

Here's the full kustomization yaml:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  creationTimestamp: "2022-08-29T07:04:34Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 12
  name: gke-iac
  namespace: example
  resourceVersion: "209219971"
  uid: fa2d9580-dce2-4208-8eec-ba2fa791526e
spec:
  force: false
  interval: 1m0s
  path: example
  prune: true
  serviceAccountName: flux
  sourceRef:
    kind: GitRepository
    name: gke-iac
    namespace: example
  suspend: false
  targetNamespace: example
status:
  conditions:
  - lastTransitionTime: "2022-10-12T15:46:12Z"
    message: |
      RoleBinding/prv2-e2-prv-example/read-pods namespace not specified, error: roles.rbac.authorization.k8s.io "pod-reader" not found
    reason: ReconciliationFailed
    status: "False"
    type: Ready
  inventory:
    entries:
    - id: prv2-e2-prv-example_ignite__ServiceAccount
      v: v1
    - id: prv2-e2-prv-example_pod-reader_rbac.authorization.k8s.io_Role
      v: v1
    - id: prv2-e2-prv-example_read-pods_rbac.authorization.k8s.io_RoleBinding
      v: v1
  lastAppliedRevision: feature/test-role/8f27e5ca1d9d7cfcafa81e8b6cb74bcb6c499f75
  lastAttemptedRevision: feature/test-role/8f27e5ca1d9d7cfcafa81e8b6cb74bcb6c499f75
  observedGeneration: 1

[stefanprodan]

No idea what verb Kubernetes uses for dry-run, same issue here fluxcd/kustomize-controller#748

[demikl]

There's a bind verb on roles resources that you can provide to one of your flux serviceaccount roles so that the dry-run stage of the kustomization-controller will succeed.

Adding this verb as in impact on privilege escalations, so be aware of this potential issue by reading: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping

[craigmunro]

Thanks, that fixed the issue :-)

[MaxRink]

@stefanprodan that probably should be addressed as one of the weird serverside apply quirks, as mostly disabling escalation prevention doesn't look like a good approach to me

[Unveraenderbar]

Have to second @MaxRink:
It is true that the error of applying linked roles+bindings together in one kustomization disappears if one has the bind verb on roles -- but this is IMHO not a proper solution for the drift detection problem of the kustomize controller.
Because it is very well possible to apply a role binding and the role it refers to with 1 kubectl apply, even w/o holding the bind privilege.
It is only the dry-run that fails, as the API server cannot assess whether the role binding would succeed w/o having the role deployed.
To me, it seems a flat dry run is just insufficient for detecting the real drift.

Having to manage roles and bindings referring them w/o binding privilege for roles is IMHO a quite common scenario in multi-tenancy Flux deployments, and right now the only practicable workaround is deploying the role in a separate kustomization than the referring binding.
But this results in a lot of unnecessary boilerplate.

@stefanprodan any news here?
If improving the dry-run process is out of scope, maybe an annotation for exempting certain resources from dry running would be an option?