Upgrading docker-compose Prometheus - July 2020

sysadmin

Summer is in the air and it seems like time to replace my entire home lab monitoring system once more. Sensu has been plodding along nicely but I’m in this for the learnin’ so I’m looking for something more interesting that a major version bump and move to Golang. I’m thinking of giving Prometheus a spin to see how it’s changed over the last few years and as a first step I decided it was time to upgrade my local test bed Docker Compose Prometheus and add some bells and whistles.

It’s been over two years since I last worked on and tagged this semi- flexible way to run a local prometheus with its supporting infrastructure so the first steps were to bump and test the image versions. All the usual components, alertmanager, redis and postgres exporters and data sources, and the always dashing grafana are still included and working with newer, patched images from dockerhub. In addition to the upgrades I’ve finally added a push gateway; I have a few batch exporters I want to trial.

Other than the upgrades and some minor tweaks like adding password protection to the redis service the most interesting part of the refresh has been adding Weave Scope, a visualisation and monitoring tool for Docker and Kubernetes. As I’ve added more services and exporters the number of different arrangements and configurations has exploded and I decided it was time to find a more visual way to describe exactly what was running. Weave Scopes ability to automatically generate a map of the services and show some connectivity details between the containers made it the perfect tool for an initial experiment. And it’s quite pretty too.

A box for each container with lines to the other containers it contacts.

I’ve not fully decided to go all in on Prometheus yet as I’ve hit limitations with file_sd_configs not being able to contain all the desired configuration and it’s made me a little weary. With this set of docker-compose files I can at the least run some experiments and see if the benefits are going to be worth the trade offs, and when it comes to experiments that’s a pretty nice place to start.