Contaminate AWS instances on ssh login
One of the principles of running large numbers of instances is that
consistency is key. Config deviations cause oddities that’ll drain your
time with investigations and nothing causes entropy on your hosts like an
admin investigating an issue. In this post we’ll configure our instances
to mark themselves as contaminated
when someone logs in. We can then
use other tooling to query, collate and probably reap, machines corrupted by the
keystrokes of humans.
While the example here is step-by-step and interactive, you’d normally
bake this in to your AMI or deploy it very early in your config stage,
possibly using cloud-init
. For our test we’ll spin up an instance and
grant it an ec2 instance profile so it can alter its own tags.
In terms of moving parts we’ll install the awscli
package, add a short
script that’ll tag the instance when run and configure PAM to invoke the
script when an ssh session opens to the machine.
# install required dependency
sudo apt-get install awscli
cat > /usr/local/bin/add-dirty-tag <<EOF
#!/bin/bash
[ "$PAM_TYPE" = "open_session" ] || exit 0
INSTANCE_ID=$(ec2metadata --instance-id)
REGION=$(ec2metadata --availability-zone)
REGION=${REGION%?} # remove the last letter
aws ec2 --region $REGION create-tags --resources $INSTANCE_ID --tags Key=contaminated,Value=true
EOF
sudo chmod a+rx /usr/local/bin/add-dirty-tag
Now we have a script to add the ‘contaminated’ tag to our instance we’ll
configure PAM to run it when a new ssh session starts. On a Ubuntu system the config
should be placed in /etc/pam.d/sshd
.
# tag the machine as contaminated if anyone sshs in.
session optional pam_exec.so /usr/local/bin/add-dirty-tag
It’s worth opening another ssh session and logging in to confirm this works. That will leave you with an established connection in case you misconfigure PAM in some way. Once you’ve successfully logged in and caused the new tag to be added to the instance you can run a cli filter from outside the instance to show all hosts that have been interactively connected to:
aws --region eu-west-1 ec2 describe-instances \
--filters "Name=tag:contaminated,Values=true" \
--query 'Reservations[].Instances[].{id: InstanceId}'
[
{
"id": "i-x134x34x"
}
]
If you decide to adopt an approach like this you can expand the values
stored in the tag using the values PAM exposes, such as $PAM_USER
or
$PAM_RUSER
and a time stamp. There’s also nothing stopping you from
adding something more structured. A concise JSON dict maybe.
Just be careful that you don’t overwrite the details on each
successive login.