Helm Chart - gha-runner-scale-set-controller RBAC issues

[brett-au]

My controller pod was having issues with being able to create roles and rolebindings in the arc-runners namespace. It also had issues reading the kubernetes secret used by the runners in arc-runner

Examples:

2023-12-15T21:34:58Z	ERROR	Reconciler error	{"controller": "autoscalinglistener", "controllerGroup": "actions.github.com", "controllerKind": "AutoscalingListener", "AutoscalingListener": {"name":"arc-runner-set-754b578d-listener","namespace":"arc-systems"}, "namespace": "arc-systems", "name": "arc-runner-set-754b578d-listener", "reconcileID": "6a5fd4c1-7905-485d-8a91-1cb830afd914", "error": "rolebindings.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:arc-systems:arc-gha-rs-controller\" cannot create resource \"rolebindings\" in API group \"rbac.authorization.k8s.io\" in the namespace \"arc-runners\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

2023-12-15T21:25:35Z	ERROR	AutoscalingListener	Unable to create listener role	{"autoscalinglistener": {"name":"arc-runner-set-754b578d-listener","namespace":"arc-systems"}, "namespace": "arc-runners", "name": "arc-runner-set-754b578d-listener", "rules": [{"verbs":["patch"],"apiGroups":["actions.github.com"],"resources":["ephemeralrunnersets"],"resourceNames":["arc-runner-set-nktnd"]},{"verbs":["patch"],"apiGroups":["actions.github.com"],"resources":["ephemeralrunners","ephemeralrunners/status"]}], "error": "roles.rbac.authorization.k8s.io is forbidden: User \"system:serviceaccount:arc-systems:arc-gha-rs-controller\" cannot create resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"arc-runners\""}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingListenerReconciler).createRoleForListener
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalinglistener_controller.go:616
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingListenerReconciler).Reconcile
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalinglistener_controller.go:186

2023-12-15T20:06:28Z	ERROR	AutoscalingRunnerSet	Failed to initialize Actions service client for creating a new runner scale set	{"autoscalingrunnerset": {"name":"arc-runner-set","namespace":"arc-runners"}, "error": "failed to find GitHub config secret: secrets \"gha-runner-scale-set-secret\" is forbidden: User \"system:serviceaccount:arc-systems:arc-gha-rs-controller\" cannot get resource \"secrets\" in API group \"\" in the namespace \"arc-runners\""}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:417
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).Reconcile
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:191
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

Version running:

- name: gha-runner-scale-set-controller
  version: 0.7.0

No custom settings inside values.yaml

I got it working with the following changes to RBAC:

Added the following verbs

verbs:
  - create
  - delete

to the following resources

roles
roleBindings
serviceaccounts
pods
ephemeralrunners/status

I also had to add

- apiGroups:
- ''"  
resources:
- secrets
verbs:
- create
- delete
- get
- list
- patch 
- update

I did not test how restrictive I could make the RBAC rules, and I thought to create a PR for these changes but wanted to reach out and see if others had the same issue as it may be a configuration error on my side. The more I thought about it though the controller is supposed to stand up listeners, and it can't create runners/listeners with out the ability to create. It also can't use the k8s secret for the runners unless it has the ability to view secrets.

Let me know if others have the same problem, if so I'll make a PR to reflect the changes I had to make. Thank you

[billimek]

Observed a similar issue here and we also needed to manually deploy a 'fix' to get the chart to work,

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: role-creator
rules:
  - apiGroups:
      - rbac.authorization.k8s.io
    resources:
      - roles
    verbs:
      - create
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-creator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: role-creator
subjects:
  - kind: ServiceAccount
    name: actions-runner-scale-set-controller-gha-rs-controller
    namespace: actions-scale-set

[brett-au]

FYI I got around this issue using Kustomize, this just allows me to override the cluster role RBAC until the bug is resolved but keep the changes in our code base.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: arc-systems

helmCharts:
- name: gha-runner-scale-set-controller
  repo:oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
  version: 0.7.0
  valuesFile: values.yaml
  releaseName: arc
  namespace: arc-systems

patches:
- path: clusterrole.yaml
  target:
    kind: ClusterRole

[brett-au]

#3212