GKE Autopilot - Any working examples?

[adamstrawson]

Am I fighting a losing battle, or has anyone managed to get actions-runner-controller working within GKE Autopilot?

Seems the strictness that comes with autopilot makes this incredibly difficult to run, and I've spent most of the day overcoming one issue to then hit another.

If anyone has any working runnerDeployment resources that they'll be able to share that works in GKE Autopilot I'll forever be grateful.

Here's currently where I'm at:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: gradle-runner
spec:
  template:
    spec:
      serviceAccountName: actions-runner
      image: XXX
      resources:
        requests:
          cpu: '30.0'
          memory: 35Gi
      dockerdWithinRunnerContainer: true
      containerMode: kubernetes
      workVolumeClaimTemplate:
        storageClassName: standard-rwo
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
      organization: XXX
      workDir: /home/runner/work
      labels:
      - gradle-runner

The two latest errors i'm trying to battle are:

2023-05-23T17:16:45Z	ERROR	runner	Retrying as failed to create ServiceAccount general-purpose-runner-v5txr-gv8ng resource	{"runner": "actions-runner-system/general-purpose-runner-v5txr-gv8ng", "error": "serviceaccounts is forbidden: User \"system:serviceaccount:actions-runner-system:actions-runner-controller\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace \"actions-runner-system\""}

And the runners I do get to run:

time="2023-05-23T17:13:09.034645601Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.4 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.

We're currently running these without issue on standard GKE clusters, but trying to migrate some workloads over to Autopilot, s we know they're problems specifically to running there.

[kevholmes]

@adamstrawson I have yet to see anyone doing this successfully but I would love to be proven wrong. I spent a week on this last summer and ran into a number of blocking issues as you've found. I haven't seen that ServiceAccount error but the real show-stopper for us was that we use docker cli predominantly and want to migrate to using their native Actions - and that requires a) privileged containers (root for cli or Actions or rootless if using the ARC rootless dind img with cli but Actions won't work...) which isn't supported in Autopilot still iirc or b) a runtime like nestybox but that's not supported in Autopilot.

[ronjarrell]

Man, I wish I had read this two weeks ago...

[rquiz]

Ran into the above limitation today due to Autopilot preventing privileged containers. Dropping a note & the error here for anyone searching.

ERROR: error while calling deploymentClient.Create for "builder-cbb2c87c-a765-47dc-8102-********": admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
  Violations details: ***"[denied by autogke-disallow-privilege]":["container buildkitd is privileged; not allowed in Autopilot"]***

[UDtorrey]

Just ran into this issue of privileged containers not allow in Autopilot

Any reason Github cant/doesnt get allowlisted for GKE autopilot privileged containers?

https://cloud.google.com/kubernetes-engine/docs/resources/autopilot-partners#allowlisted-partner-workloads

[FearlessHyena]

It would be great to see GitHub actively pursuing this and getting themselves allowlisted

[alexberry]

plus one for this, or find a workaround that does not require the privelege.

[Hi-Fi]

Isn't it easiest to just use Kubernetes mode for the spawning of the additional containers? Would be also safer than having privileges.