Preventing AWS credential leaks in Git with pre-commit

Ever since I started using distributed version control systems one of my fears has been accidentally publishing security credentials that could be maliciously used against me. Add to this services like AWS where you can run up a large bill very quickly, and an employer that tries to open source everything possible, and eventually you know you’re going to slip up and expose something you shouldn’t. In the past I’ve cobbled together git commit hooks to provide a basic safety net but I’m looking for a more designed solution and pre-commit is heavily in the running.

First let’s define some background. pre-commit (by Yelp) is “a framework for managing and maintaining multi-language pre-commit hooks”. Git hooks are scripts that git executes at certain points, such as before a commit is allowed to complete. This is the best time to catch in the unwanted content as it’s not been written to anything apart from your local filesystem. For our example we’ll create a new virtualenv and git repo, setup a basic pre-commit config with aws credential protection and try to commit one of our keys to git.

# create the virtualenv we'll use for testing
$ virtualenv pre-commit-testing

Using real prefix '/usr'
New python executable in pre-commit-testing/bin/python2
Also creating executable in pre-commit-testing/bin/python
Installing setuptools, pip, wheel...done.


$ cd pre-commit-testing

# add the virtualenvs settings to our working environment
$ source bin/activate

(pre-commit-testing)[dwilson@home pre-commit-testing]$


# install pre-commit
$ pip install pre-commit

# confirm everything is working
$ pre-commit --help

usage: pre-commit [-h] [-V] {install,uninstall,clean,autoupdate,run,help}

Now everything’s installed we’ll configure the detect-aws-credentials plugin using the .pre-commit-config.yaml file. This plugin will notify us, and prevent our git commit from succeeding, if we include one of our AWS secret keys in our changes.

cat > .pre-commit-config.yaml <<EOF
-   repo: https://github.com/pre-commit/pre-commit-hooks.git
    sha: edb5d5219e4e56c79c13da537dfa59c8c6d08127
    hooks:
    -   id: detect-aws-credentials
EOF

It’s worth a little delving in to how this actually works. The detect-aws- credentials plugin reads all ‘aws_secret_access_key’ values from ~/.aws/credentials and checks each of them against the contents of every file about to be checked in. On a little tangent this functions differently to how my own custom commit hook worked. I did a regex string match for anything matching ‘AKIA’, as all AWS access keys began with that value.

$ git init
Initialized empty Git repository in ~/pre-commit-testing/.git/


# now we have a repo we'll update pre-commit and install the git hook.
$ pre-commit autoupdate

Updating https://github.com/pre-commit/pre-commit-hooks.git...already up to
date.


$ pre-commit install

pre-commit installed at ~/pre-commit-testing/.git/hooks/pre-commit

We’ll now add a known credential to test against. Remember to remove this once you’ve finished testing pre-commit.

cat >> ~/.aws/credentials <<EOF

[commit-test]
aws_access_key_id = AKIA1111111111111111
aws_secret_access_key = 1111111111111111111111111111111111111111
EOF

Scaffolding complete we can progress to the actual heart of our test, making a commit fail. We’ll create a file in our repo, do one successful commit lacking a secret key, followed by a commit containing the aws_secret_access_key from above that will fail.

$ echo hello > dangerous_file
$ git add dangerous_file
$ git commit -v -m 'No secrets.' dangerous_file

Detect AWS Credentials................................Passed
[master (root-commit) 620fa5a] No secrets.
 1 file changed, 1 insertion(+)
 create mode 100644 dangerous_file

$ echo 1111111111111111111111111111111111111111 >> dangerous_file
$ git commit -v -m 'Shush, secrets.' dangerous_file

Detect AWS Credentials................................Failed

AWS secret key found: dangerous_file

# and now you can remove the testing creds from ~/.aws/credentials

Success! Well, a failure, which in this case is what we want. pre-commit has stopped us from committing an AWS secret key to our git repo.

So, is pre-commit for you? Using it requires some changes to your work flow. An extra config file and remembering to run certain commands on creation of every git repo does add a little mental baggage. However if you’re pushing a lot of code to public repositories then this extra work could quickly prove worth the effort. As they say, ‘an ounce of prevention is better than a pound of cure’.