Strange Docker Authentication Behaviour

[jonwest]

Oh hey! I'm diving into self-hosted GitHub Actions, and I'm using ARC. For the most part, all is well. I'm able to trigger jobs without issue, and running actions from the shell using those runners is fine. Where things get hairy is that I'm trying to use multiple containers within a job (as distinct steps, not services). If I run a step like:

- name: Do the thing
  uses: "docker://gcr.io/some-private-image"
  with:
    entrypoint: "bash"
    args: "-c echo 'Hello World!'"

As an example, I can pull both public and private images without issue.

If instead I want to reference that as an external action:

- name: Do the thing externally
  uses: ./.github/actions/do-the-thing.yaml

And it is like:

- name: Do the thing
  runs:
    using: "docker"
    image: "gcr.io/some-private-image"
    args:
      - "-c"
      - "echo 'hello world!'"

I get a 403, permission denied as if it's not authenticated. If I use a distinct step before the action that pulls the image, everything is good and the external action works without a hitch.

The ARC runner is configured to have two containers in the pod—a runner which is running an Ubuntu 20 based image, and docker which is running an extended version of docker:dind—both of which have docker-credential-gcr installed and configured.

I've verified that in the runner container, both the runner user, as well as root are able to pull the images without issue when I exec into the pod. For the docker container, it is running as root, and that user can also pull those images without issue—all of these tests have been done with both interactive and non-interactive shells to eliminate that as a variable, still no change.

While it's possible I could just have the distinct pull step, I'd rather not, since that's brittle—the caller shouldn't need to know the image tag in the action, and args within the uses: docker:// step syntax being a string rather than an array makes writing anything beyond a one-liner a bit dicey, given it must be quoted.

Any ideas? This seems so strange!

[jonwest]

Maybe to help move things along—what I'm primarily interested in is figuring out where this authentication step is taking place. If I am referencing an action as a step, the Docker authentication doesn't appear to be handled by the runner user of the runner container, the root user of the runner container, or the root user of the docker sidecar, so where, or how is the authentication of the docker pull before the docker run of the action container taking place?