GitHub Actions Runner VM token compromised - no damage done

[joeyparrish]

Overview

In the interest of transparency, I'm disclosing that a malicious PR compromised our GitHub Actions Runner VMs. A token from the VM's service account was exfiltrated. This was noticed quickly (within about 3 hours), and workflows and VMs were were shut down.

Right before the shut down, the attacker tried to access the VM's Docker daemon, and was interrupted in the attempt.

The attack was carried out by a security researcher, and no damage was done.

VM token

The exfiltrated token expired after one hour and was not used for anything as far as Google Security can tell.

While I was investigating the attack and working on remediation, GitHub announced the availability of Linux arm64 runners hosted and managed by GitHub. This removes the need for our own self-hosted runner VMs. Our VMs were deleted today. This is super simple and we are super lucky on the timing of this feature from GitHub.

PR As A Point Of Ingress

The other tricky aspect of this attack was that the attacker was able to execute the PR workflow in spite of the "require approval for first-time contributors" rule enabled on the repo. It turns out that the pull_request_target trigger bypasses this rule. This, combined with the PR-author-controlled build scripts, is just begging for attacks "research".

So why use pull_request_target? It's because we were using the vars context for repo-specific settings. This context is not available in a pull_request trigger, or rather, it pulls variables from the PR-author's repo, not the target repo.

Before we started using vars for repo-specific settings, we used to use "environments" to set flags on the repo. These can be read publicly no matter the context, and that system was in use by Shaka Packager before GitHub introduced vars.

So to resolve that, we go back to pull_request triggers and the use of "environments" to set flags for things like self-hosted runners (not presently needed) or debug flags (still sometimes needed).

Remediation Work

The following changes were made across repositories to clean things up:

• Stop using self-hosted arm64 Linux runners now that ubuntu-24.04-arm is available:
  • ci: Stop using self-hosted Linux/arm64 runners shaka-packager#1474
  • ci: Stop using self-hosted Linux/arm64 runners #65
  • ci: Stop using self-hosted Linux/arm64 runners shaka-streamer#232
• Delete Google Cloud VMs for self-hosted runners
• Delete and recreate any tokens still needed in the affected Google Cloud project
• Standardize on "environments" and stop using pull_request_target, to enforce "require approval" rules for workflows:
  • ci: Update comments for the settings workflow shaka-packager#1476
  • ci: Switch pull_request_target to pull_request #66
  • ci: Switch pull_request_target to pull_request shaka-streamer#233

[github-actions]

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.