When it comes to Amazon Web Services support Terraform has coverage that’s second to none. It includes most of Amazons current services, rapidly adds newly released ones, and even helps granularise existing resources by adding terraform specific extensions for things like individual rules with aws_security_group_rule. This awesome coverage makes it even more jarring when you encounter one of the rare edge cases, such as VPC default security groups. It’s worth taking a step back and thinking about how Terraform normally works. Read on →

Terraform code reuse leads to modules. Modules lead to variables and outputs. Variables and outputs lead to massive amount of boilerplate documentation. terraform-docs lets you shortcut some of these steps and jump straight to consistent, easy to use, automatically generated documentation instead. Terraform-docs, a self-contained binary implemented in Go, and released by Segment, provides an efficient way to add documentation to your terraform code without requiring large changes to your workflow or massive amounts of additional boilerplate. Read on →

When I first started my Prometheus experiments with docker-compose one of the most awkward parts of the process, especially to document, were the manual steps required to click around the Grafana dashboard in order to add the Prometheus datasource. Thanks to the wonderful people behind Grafana there has been a push in the newest major version, 5 at time of writing, to make Grafana easier to automate. And it really does pay off. Read on →

I’m a big fan of baking testing into build and delivery pipelines so when a new tool pops up in that space I like to take a look at what features it brings to the table and how much effort it’s going to take to roll out. The Aqua Security microscanner, from a company you’ve probably seen at least one excellent tech talk from in the last year, is a quite a new release that surfaces vulnerable operating systems packages in your container builds. Read on →

Despite the number of Amazon Web Services that have the word simple in their titles, keeping on top of a large cloud deployment isn’t an easy ask. There are a lot of important, complex, aspects to consider so it’s advisable to pay attention to the best practices, reference architectures, and benchmarks published by AWS and their partners. In this post we’ll take a look at the CIS security benchmark and a tool that will save you a lot of manual verifying. Read on →

After finding a bug in my custom written, bulk code comment / uncomment, vim function I decided to invest a little time to find a mature replacement that would remove my maintenance burden. In addition to removing my custom code I wanted a packaged solution, to make it easier to include across all of my vim installs. After a little googling I found the ideal solution, the vim-commentary plugin. It ticks all my check boxes: mature enough all the obvious bugs should have been found receives attention when it needs it has a narrow, well defined, focus as a user it works the way I’d have approached it And while it’s not a selection criteria, Tim Pope writing it is a big plus I use the Vundle package manager for vim so installing commentary was quick and painless. Read on →

After adding AlertManager to my Prometheus test stack in a previous post I spent some time triggering different failiure cases and generating test messages. While it’s slightly satisfying seeing rows change from green to red I soon wanted to actually send real alerts, with all their values somewhere I could easily view. My criteria were: must be easy to integrate with AlertManager must not require external network access must be easy to use from docker-compose should have as few moving parts as possible A few short web searches later I stumbled back onto a small server I’ve used for this in the past - MailHog. Read on →

What’s the use of monitoring if you can’t raise alerts? It’s half a solution at best and now I have basic monitoring working, as discussed in Prometheus experiments with docker-compose, it felt like it was time to add AlertManager, Prometheus often used partner in crime, so I can investigate raising, handling and resolving alerts. Unfortunately this turned out to be a lot harder than ‘just’ adding a basic exporter. Before we delve into the issues and how I worked around them in my implementation let’s see the result of all the work, adding a redis alert and forcing it to trigger. Read on →

How much of your system does your internal monitoring need to consider down before something is user visible? While there will always be the perfect chain of three or four things that can cripple a chunk of you customer visible infrastructure there are often a lot of low importance checks that will flare up and consume time and attention. But what’s the ratio? As a small thought experiment on one project I’ve recently started to leave a new, very simple four panel, Grafana dashboard open on a Raspberry PI driven monitor that shows the percentage of the internal monitoring checks that are currently in a successful state next to the number of user visible issues and incidents. Read on →

As 2018 rolls along the time has come to rebuild parts of my homelab again. This time I’m looking at my monitoring and metrics setup, which is based on sensu and graphite, and planning some experiments and evaluations using Prometheus. In this post I’ll show how I’m setting up my tests and provide the Prometheus experiments with docker-compose source code in case it makes your own experiments a little easier to run. Read on →