Keeping your GitHub Actions and workflows secure Part 4: New vulnerability patterns and mitigation strategies

This post is the fourth in a series of posts about GitHub Actions security. Part 1, Part 2, Part 3

We recently implemented CodeQL support for GitHub Actions workflows. We ran these queries at scale to test them against diverse open source projects. Having triaged and reported numerous alerts, we have identified some new common patterns that often lead to vulnerabilities in GitHub workflows:

Misuse of pull_request_target trigger

The pull_request_target event trigger, while offering powerful automation capabilities in GitHub Actions, harbors a dark side filled with potential security pitfalls. This event trigger, designed to execute workflows within the context of Pull Request’s base branch, presents special characteristics that severely increases the impact in case of any vulnerability. A workflow activated by pull_request_target and triggered from a fork operates with significant privileges in contrast to the pull_request event:

• It is able to read repository and organization secrets.
• It is allowed to have write permissions.
• It circumvents the usual safeguards of pull request approvals, allowing workflows to run unimpeded even if approval mechanisms are configured for standard pull_request events.
• It normally runs in the context of the default branch, which, as we will see, may allow malicious actors to poison the Action’s cache and move laterally to other, more privileged workflows even when removing all permissions to the vulnerable workflow.

When working with pull_request_triggered workflows, we have to be very careful and pay special attention to the following scenarios:

• Code Execution from Untrusted Sources: The ability to execute code from forked pull requests is a double-edged sword. A malicious actor can submit a seemingly innocuous pull request that, when triggered by pull_request_target, unleashes havoc. This malicious code, running in the context of the target repository’s environment, could exfiltrate secrets or even tamper with repository contents and releases. The danger lies in the inadvertent checkout and execution of code from untrusted sources. Common mistakes include using the actions/checkout action with a reference to the pull request’s head branch.
• Time Of Check Time Of Use (TOCTOU) Attacks. Even with approval requirements in place (eg: require the Pull Request to have a specific label which attackers are not able to set on their own), attackers can leverage the element of time to bypass security measures. A malicious actor can submit a seemingly harmless pull request, patiently wait for approval, and then swiftly update the pull request with malicious code before the workflow execution. The workflow, relying on a mutable reference like a branch name, falls prey to this “time of check to time of use” (TOCTOU) attack, unwittingly executing the newly injected malicious code. Confusion surrounding context variables, specifically head.ref and head.sha, can lead to vulnerabilities. head.ref, pointing to the branch, is susceptible to manipulation by attackers. In contrast, head.sha, referencing the specific commit, provides a reliable and immutable pointer to the reviewed and approved code. Using the incorrect variable can create an opening for attackers to inject malicious code after approval.
• Lurking Threats in Non-Default Branches. Vulnerabilities often linger in non-default branches, even after being addressed in the default branch. An attacker can target these vulnerable versions of the workflows residing in non-default branches, exploiting them by submitting pull requests specifically to those branches.
• Cache Poisoning. As a mitigation, developers may strip out all read and write permissions from these workflows when in need to run untrusted code. However, the seemingly innocuous permissions: {} configuration can unexpectedly pave the way for cache poisoning attacks. This attack vector involves injecting malicious content into the cache, which can subsequently affect other workflows relying on the poisoned cache entries. Even if a workflow doesn’t have write access to the repository, it can still poison the cache, leading to potential code execution or data manipulation in other workflows that utilize the compromised cache.

If we really need to use this trigger event, there are a few ways to harden the workflows to prevent any abuses:

• Repository Checks. Implementing stringent repository checks is crucial for thwarting attacks originating from forked pull requests. One effective method is to configure workflows to execute only for pull requests originating from the base repository, effectively blocking any attempts from external forks. This can be achieved by using conditions such as github.event.pull_request.head.repo.owner.login == "myorg" to restrict workflow execution.
• Actor Checks. Verifying the permissions of the pull request author is another layer of defense. By restricting workflow execution to trusted actors, such as members of the organization or approved collaborators, the risk of malicious code injection from unauthorized sources can be significantly reduced. Never use a hardcoded list of user names since the users may lose the permissions with time or the user names left abandoned for an attacker to claim.
• Workflow Splitting. The above mitigations may not be useful if we need to run a workflow for forks or arbitrary users. In those cases splitting workflows into unprivileged and privileged components is a powerful security strategy. Unprivileged workflows, triggered by pull_request events, handle the initial processing of pull requests without access to sensitive secrets or write permissions. Privileged workflows, activated by workflow_run events, are invoked only after the unprivileged workflow has completed its checks. This separation ensures that potentially malicious code from forked pull requests never executes within a privileged context. The unprivileged workflow will generally need to communicate and pass information to the privileged one. This is a crucial security boundary, and as we will see in the next section, any data coming from the unprivileged workflow should be considered untrusted and potentially dangerous.

Security boundaries and workflow_run event

The workflow_run event trigger in GitHub Actions is designed to automate tasks based on the execution or completion of another workflow. It may grant write permissions and access to secrets even if the triggering workflow doesn’t have such privileges. While this is beneficial for tasks like labeling pull requests based on test results, it poses significant security risks if not used carefully.

The workflow_run trigger poses a risk because it can often be initiated by an attacker. Some maintainers were surprised by this, believing that their triggering workflows, which were run on events such as release, were safe. This assumption was based on the idea that since an attacker couldn’t trigger a new release, they shouldn’t be able to initiate the triggering workflow or the subsequent workflow_run workflow.

The reality is that an attacker can submit a pull request that modifies the triggering workflow and even replace the triggering events. Since pull_request workflows run in the context of the pull request’s HEAD branch, the modified workflow will run and, upon completion, will be able to trigger an existing workflow_run workflow. The danger arises from the fact that even if the triggering pull_request workflow is not privileged, the triggered workflow_run workflow will have access to secrets and write-scoped tokens, even if the initial workflow did not have those privileges. This enables privilege escalation attacks, allowing attackers to execute malicious code with elevated permissions within the CI/CD pipeline.

Another significant pitfall with the workflow_run event trigger is artifact poisoning. Artifacts are files generated during a workflow run that can be shared with other workflows. Attackers can poison these artifacts by uploading malicious content through a pull request. When a workflow_run workflow downloads and uses these poisoned artifacts, it can lead to arbitrary code execution or other malicious activities within the privileged workflow. The issue is that many workflow_run workflows do not verify the contents of downloaded artifacts before using them, making them vulnerable to various attacks.

Securing workflow_run workflows requires a multi-faceted approach. By understanding the inherent risks and implementing the recommended mitigations, developers can leverage the automation benefits of workflow_run while minimizing the potential for security compromises.

Effective Mitigations:

• Limit Workflow Scope with Branch Filters: Specify the branches where workflow_run workflows can be triggered using the branches filter. This helps restrict the scope of potential attacks by preventing them from being triggered on branches from forks.
• Verify Event Origin: Incorporate a check like github.event_name != 'pull_request' to prevent workflow_run workflows from being triggered by pull requests from forks. This adds an extra layer of protection by ensuring that the triggering workflow originates from a trusted source.
• Treat Artifacts as Untrusted: Treat all downloaded artifacts as potentially malicious and implement rigorous validation checks before using them. Always unzip artifacts to a temporary directory like /tmp to prevent potential file overwrites, i.e. the pollution of the workspace.
• Avoid Defining Environment Variables: Minimize the use of environment variables, especially when handling untrusted data. Environment variables can be vulnerable to injection attacks, potentially allowing attackers to modify their values and execute malicious code.
• Handle Output Variables with Caution: Exercise caution when defining output variables from artifact’s content, as they can also be vulnerable to manipulation by attackers. Always validate the contents of output variables (for example that it is a number, not a string) before using them in subsequent steps or other workflows.

Non-Effective Mitigations:

• Repository Checks: We found several workflows (eg: AWS Karpenter Provider, or Cloudflare Workers SDK) relying solely on repository checks, such as verifying the repository owner (github.repository_owner == 'myorg'), which is not effective in mitigating workflow_run risks since the workflow always run in the context of the default branch which belongs to the organization.

IssueOops: Security Pitfalls with issue_comment Trigger

The issue_comment event trigger in GitHub Actions is a powerful tool for automating workflows based on comments on issues and pull requests. When applied in the context of IssueOps, it can streamline tasks like running commands in response to specific comments. However, this convenience comes with significant security risks that must be carefully considered.

• TOCTOU Vulnerabilities: Similar to the pull_request_target event trigger, workflows using issue_comment can be vulnerable to TOCTOU attacks. If the workflow checks out and executes code from a pull request based on an issue comment, an attacker could exploit the time window between the comment and the workflow execution. An attacker might initially submit a harmless pull request, waiting for an administrator to review and approve the workflow by adding a comment. Once the approval is given, the attacker could quickly update the pull request with malicious code, which would then be executed by the workflow. \
• Bypassing Pull Request Approval Mechanisms: The issue_comment event trigger is not subject to the pull request approval mechanisms intended to prevent abuse. Even if the workflows triggered by pull request require approvals, an attacker can trigger an issue_comment workflow by simply adding a comment to the pull request, potentially executing malicious code without any review. \

Mitigating the Risks:

• Shifting to Label gates: Instead of relying on issue comments to trigger critical workflows, consider adopting Label gates approaches. Label gates use labels to trigger specific actions, allowing for more granular control and better security. Since the Labeled activity type for a pull_request trigger contains details about the latest commit SHA of the Pull Request, there is no need for workflow to resolve a Pull Request number into the latest commit SHA, as it is the casse with issue_comment, and therefore there is no window for an attacker to modify the Pull Request. Remember to use the commit SHA rather than the HEAD reference to prevent TOCTOU vulnerabilities. \

Ineffective or Incomplete Mitigations:

• Actor Checks: Relying solely on actor checks (verifying the identity or permissions of the commenter) is ineffective. The actor triggering the workflow might not be the same one trying to exploit it, further rendering actor checks unreliable. This is the case for TOCTOU vulnerabilities where an attacker can submit a legitimate Pull Request and wait for an admin to trigger an IssueOp and then swiftly mutate the Pull Request by adding a new commit with malicious code on it. \
• Date Checks: Comparing timestamps to verify that the last commit occurred before the triggering comment is also unreliable. Currently, GitHub has no reliable way to figure out the date when a commit was pushed to a repository. An attacker can forge commit dates, rendering these checks useless in preventing malicious code execution. \
• Repository Checks: Verifying the origin repository is not a useful mitigation for issue_comment event triggers. The issue_comment event always executes within the context of the target repository’s default branch, making repository checks redundant. \

Conclusion

We discovered a variety of new patterns of potential supply chain attacks through insecure GitHub Actions workflows. The new CodeQL packs allow you to scan your repository for those patterns. Enable CodeQL for GitHub Actions in your open source project to protect the ecosystem we all depend on!