As security researchers, the GitHub Security Lab team constantly embarks on an emotional journey with each new vulnerability challenge. The excitement of starting new research, the disappointment that comes with hitting a plateau, the energy it takes to stay focused and on target… and hopefully, the sheer joy of achieving a tangible result after weeks or months of working on a problem that seemed unsolvable.

Regardless of how proud you are of the results, do you ever get a nagging feeling that maybe you didn’t make enough of an impact? While single bug fixes are worthwhile in improving code, it’s not sufficient enough to improve the state of security of the open source software (OSS) ecosystem as a whole. This holds true especially when you consider that software is always growing and changing—and as vulnerabilities are fixed, new ones are introduced.

At GitHub, we host millions of OSS projects which puts us in a unique position to take a different approach with OSS security. We have the power and responsibility to make an impact beyond single bug fixes. This is why a big part of the GitHub Security Lab mission is to find ways to scale our vulnerability hunting efforts and empower others to do the same.

Our goal is to turn single vulnerabilities into hundreds, if not thousands, of bug fixes at a time. Enabled by the GitHub engineering teams, we aim to establish workflows that are open to the community that tackle vulnerabilities at scale on the GitHub platform.

Ultimately, we want to establish feedback loops with the developer and security communities, and act as security facilitators, all while working with the OSS community to secure the software we all depend upon.

GitHub and CERT

Dial up modem sounds are a thing of the past, but the protocols we used while listening to a modem’s cheerful sounds are still present and active all over the internet.

A modem's cheerful sound

Nostalgia aside, things got interesting when GitHub Security Lab received an email from Vijay Sarvepalli from the CERT team informing us that IOActive’s Ilja Van Sprundel found a remote vulnerability in pppd (point-to-point protocol daemon). When we investigated the unpatched pppd Extensible Authentication Protocol (EAP) code, we noted the following construct:

/* Not so likely to happen. */
if (vallen >= len + sizeof (rhostname)) {
  ppp_dbglog(("EAP: trimming really long peer name down"));
  MEMCPY(rhostname, inp + vallen, sizeof (rhostname) - 1);
  rhostname[sizeof (rhostname) - 1] = '\0';

Given that previously vallen was checked to be less than len, the vulnerability occurs because it can never be the case that vallen >= len + sizeof(rhostname). Therefore, rhostname never gets trimmed against oversized input, and the rhostname array may overflow.

At the Security Lab, we love to learn about new vulnerabilities so we can model them in CodeQL. CodeQL is a special querying language that allows you to query code as data to find instances of vulnerabilities. However, the real reason the CERT contacted us was to help patch thousands of GitHub projects that were using this exact same vulnerable code pattern. More information about this could be found on the CERT blog post and CVE. This includes various projects, such as complete Android forks to robot-soccer projects.

Robot soccer

We’re always extremely cautious when it comes to sending mass project notifications at GitHub. While we prefer not to disturb maintainers, this vulnerability was public, potentially remotely exploitable, and a significant amount of these projects were most likely unaware of its existence. We ultimately decided to create a mass pull request aimed at fixing this vulnerability in as many projects as possible. This pull request automatically proposes minor changes to project code for all affected projects, and can be applied by maintainers using a few clicks.

After confirmation from GitHub leadership about moving forward, our very own @pwntester embarked on the implementation. First, he collected potentially vulnerable OSS projects using GitHub code search, followed by trimming duplicates. After this initial process, he used a customized version of a script that community member Jonathan Leitschuh developed for a similar mass security pull request effort. We’ll cover more about this pull request later in this post.

With a bit of API juggling, script tweaking, and some patience, we started the process of creating pull requests against a large number of vulnerable GitHub projects. To date 1,964 different projects have received this security pull request and in the first 72 hours, 45 have actually merged the patch.

Some of the projects that merged the pull request include:

Monitor the progress of this mass pull request

GitHub and the community

As you might know, the Security Lab runs a bug bounty program. It’s different from other bounty programs in that we don’t award unique vulnerability findings, but rather novel CodeQL queries that model vulnerability patterns that the CodeQL team can include in our standard query pack. Once a CodeQL query makes it into our standard query pack, it’s run on all open source projects for free, thus amplifying its impact.

The other bounty we run encourages the community to write CodeQL queries that help find and report more than three vulnerabilities in OSS projects. Ultimately, the goal is to scale and move away from the individual fix model of traditional security patching.

These bounties have made a great impact to date, and one of our favorite stories from a community member Jonathan Leitschuh who submitted a CodeQL query for detecting insecure Maven Java artifact downloads that were susceptible to person-in-the-middle tampering. He received a bounty for that query, but then he decided to use the query to actually fix this problem at scale. He wrote a bot that opened over 1,700 pull requests for any affected GitHub projects.

import java
import semmle.code.xml.MavenPom

private class DeclaredRepository extends PomElement {
  DeclaredRepository() {
    this.getName() = "repository" or
    this.getName() = "snapshotRepository" or
    this.getName() = "pluginRepository"
  }

  string getUrl() { result = getAChild("url").(PomElement).getValue() }

  predicate isInsecureRepositoryUsage () {
    getUrl().regexpMatch("(?i)^(http|ftp)://(?!localhost[:/]).*")
  }
}

from DeclaredRepository repository
where repository.isInsecureRepositoryUsage()
select repository,
  "Downloading or uploading artifacts over insecure protocol (e.g. http or ftp) to/from repository " +
  repository.getUrl()

Jonathan’s success with fixing this issue across hundreds of projects partially inspired the workflow we adopted to coordinate our own mass pull request effort with CERT. It’s a great example of the kind of feedback loop between us and the community that we want to further develop and enable. We love to see the community going the extra mile, which is why we ended up doubling Jonathan’s bounty payment as a token of appreciation for his continued collaboration with our team and the wider community.

Going forward

As a security research team, it’s in our DNA to keep trying no matter how insurmountable the OSS security challenge may seem. Sometimes we’ll succeed and often we’ll fail, but we always walk away from the situation with more insight and knowledge. We hope to iteratively improve our success rates and apply the concept of embracing failure as we know it’s a fundamental ingredient of growth to software security itself.

Working closely with the wider GitHub organization, we hope to produce workflows that enable us and our community to make GitHub the home for open source security. If coding is social, then security should be social too.

As always, we’d love to hear about your unique security challenges, from code-review to coordinated disclosure, and everything in between. If there’s anything we can help you with, reach out to the @GHSecurityLab team. You can also get involved and help us secure open source at scale! Visit our bounty program, we look forward to your submissions.