Upgrading docker-compose Prometheus - July 2020
Summer is in the air and it seems like time to replace my entire home lab monitoring system once more. Sensu has been plodding along nicely but I’m in this for the learnin’ so I’m looking for something more interesting that a major version bump and move to Golang. I’m thinking of giving Prometheus a spin to see how it’s changed over the last few years and as a first step I decided it was time to upgrade my local test bed Docker Compose Prometheus and add some bells and whistles.
It’s been over two years since I last worked on and tagged this semi- flexible way to run a local prometheus with its supporting infrastructure so the first steps were to bump and test the image versions. All the usual components, alertmanager, redis and postgres exporters and data sources, and the always dashing grafana are still included and working with newer, patched images from dockerhub. In addition to the upgrades I’ve finally added a push gateway; I have a few batch exporters I want to trial.
Other than the upgrades and some minor tweaks like adding password
protection to the
redis service the most interesting part of the
refresh has been adding
a visualisation and monitoring tool for Docker and Kubernetes. As I’ve
added more services and exporters the number of different arrangements
and configurations has exploded and I decided it was time to find a more
visual way to describe exactly what was running. Weave Scopes ability to
automatically generate a map of the services and show some connectivity
details between the containers made it the perfect tool for an initial
experiment. And it’s quite pretty too.
I’ve not fully decided to go all in on Prometheus yet as I’ve hit
file_sd_configs not being able to contain all the
desired configuration and it’s made me a little weary. With this set of
docker-compose files I can at the least run some experiments and see
if the benefits are going to be worth the trade offs, and when it comes
to experiments that’s a pretty nice place to start.