Abstract
In my current workplace we use GitHub to manage our source code. We have our own GitHub org and use GitHub Enterprise. GitHub has many useful security feature which we leverage to mitigate threats to our source code. I’ll highlight in this article some weaknesses with those features and some improvements I believe GitHub could make which would benefit the community as a whole.
Introduction
We think a lot about security at my current workplace. One particular area where we are highly concerned about security is in our consumer applications. We want to ensure, as much as reasonably possible, that we don’t ship malware to our customers.
There are many vectors which an attacker could use to inject malware into our applications:
- Our source code
- 3rd party source code
- Our build system
- Our signing certificates
We take the view that, for all the above vectors, we should be immune to a single developer machine compromise. We make the assumption that a developer machine, or even a developer, will eventually be compromised – it’s a case of when not if. We thus need to ensure that none of the above attack vectors can be exploited if a single machine is compromised.
In this blog I’m going to focus on the challenges ensuring this for the first item – our source code.
How We Protect Our Source Code
In order to protect our source code we want to enforce the requirement that at least two humans must have approved code before it is shipped to customers. In general this means at least one coder and one code reviewer. We also want to ensure that we know who those humans are, i.e. that they are humans we employ.
To do this we leverage protected branches. We mandate that we only ship code to customers from “release branches” – either master branch or a branch which matches the pattern release/*. I won’t go into how we enforce that only code from these branches can be shipped to customers, but our processes are such that I’m confident that this is well enforced
On release branches we enable the following branch protections:
- Require pull request reviews before merging
- At least 1 approving review
- Dismiss stale pull request approvals when new commits are pushed
- Requires signed commits
- Include administrators
Requiring signed commits proves identity of the developer as best as possible. We use hardware tokens (Yubikeys) to hold the signing key and generate those keys using a secure process. As above, I won’t go into details on this point but my confidence is high that the process is very secure.
In what follows I’ll discuss some of the flaws with the above process, how we mitigate and how I’d like to see GitHub change their permissions and settings to help remove the flaws and the need for mitigations.
Issues with Permissions Granularity
We generally avoid giving our developers GitHub Admin permissions on the repos they work on. We also massively restrict who has GitHub Org Admin – a.k.a. “God Mode”. The problem with Admin accounts is that they can change all settings on a repo. In particular, this means they can remove the branch protections described in the above section.
The obvious question would be: “Why does anyone need GitHub Admin in the first place?”. The most common reason for it is to manage repo permissions – “Contributors” in GitHub terminology. We like to keep access to our repos reasonably tight and unfortunately this means that a common administrative task is to change permissions. We handle this by having only a handful of admins who manage this using a separate GitHub account on a separate (“clean”) laptop. This is frankly a PITA!
If we were to liberally hand out admin permissions, then the only mitigation to the threat of those users changing branch permissions (to bypass the two humans rule) would be to have some alerting in place to ensure that protections haven’t been removed or code wasn’t merged without review. The former can be done with a SIEM and the latter using some custom external service (using GitHub APIs). We have implemented both of these, which is good practice from a defence in depth viewpoint. However, I’d much prefer it if we didn’t have to rely on these fallbacks. I also think it would be beneficial for the community if they didn’t have to build such solutions; I’d bet most companies don’t do this due to lack of awareness or lack of resources.
Note: I was somewhat excited to see a new “Maintainer” permission level recently added, but unfortunately it doesn’t provide much in the way of interesting access IMO. It certainly doesn’t solve this problem
Proposed Solution
What I would like to see is a change in the permissions model which allows a developer to have permissions to change the Contributors to a repo without having more serious permissions like the ability to change branch protections. Specifically, I would like GitHub to modify permissions to optionally allow a repo contributor to be able to grant access to another user. A user with this new permission should only be able to grant permissions levels to others which are less than or equal to their own levels, e.g. it would be silly if someone with write access could grant admin access to someone else.
This simple change would solve all of the above problems. This doesn’t mean that developers won’t need admin access to repos from time to time. However, from my experience I’d say that 95% of admin tasks are access related.
Issues with Signatures
Signed commits are a great way of providing authenticity to code – especially when used in conjunction with hardware tokens which support “touch to sign”. However, one flaw with GitHub is that any changes made via the UI are signed with a generic GitHub key. This undermines the proof of identity provided by hardware tokens.
If my GitHub account or machine is compromised then a malicious 3rd party can use the GitHub UI to change code and sign it with the GitHub key. This can be partially mitigated by enforcing 2FA on GitHub accounts and using a hardware token as the second factor. However, if a developer’s machine is compromised then the 2FA is somewhat redundent.
We mitigate this risk ourselves by verifying signatures using a separate external system. We check that all commits are signed by only our keys and disallow any commits signed by the GitHub key. Building such a system is non-trivial and, as above, I’d bet most companies don’t do this due to lack of awareness or lack of resources.
Proposed Solution
What I’d like to see is a repo level option (possibly even org level) which disables any features which allow commits to be signed by the GitHub key. This setting should only be modifiable by a repo (or org) admin.
Is this excessive paranoia – maybe? However, since the solution to this would be relatively simple then it’s perfectly reasonable to argue that this is worth doing
Final Comments
In all of the above, I’ve made the implicit solution that we trust GitHub. If GitHub itself is compromised then all bets are off. However, we’re prepared to put our trust into GitHub as they (and Microsoft) are highly motivated to ensure a high security bar for their product