Last year at one of the many Belgium tech events Kris mentioned a conference called LOAD (2010) to me. I was a little late in booking the hotel and in the end I couldn’t make it over - and judging by the quality of this years event that was a big mistake. While it’s nice to spend time in the devops world and talk about communication, processes and how to merge development and operational tool-chains sometimes it’s nice to focus on solid, production grade sysadmining; and LOAD was the perfect conference for it. Read on →

Our source code has always been air gapped from the Internet. The forensic examination confirmed that software development servers and workstations were not affected by the incident – from HBGary Anyone else find it hard to accept that none of the developers, testers, documentation writers or build people ever accessed source code from their Internet connected laptops / workstations? Especially considering the state of their other security measures. Don’t get me wrong, in some cases it’s a sensible solution ( off-line key signing for example) but for entire teams working on a shared code base?

Sometimes it’s the little niggles that annoy people the most. As my team progress in to puppet they have an annoying habit of asking very good questions; which can sometimes be a struggle to answer. Todays best question was - “How do I tell if this file is under puppets control?” While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. Read on →

<tl;dr>Log nrpe-runner state changes when puppet runs to see what broke or was fixed.</tl;dr> While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I’ve come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net. Read on →

At work we try, and sometimes even succeed, in using Test Driven Deployment so as one of my background projects I’ve been wrapping certain tools in to cucumber friendly forms. Over the last couple of days I’ve been grabbing ten minutes here and there to incorporate Puppet 2.6 in to the pile. Feature: Puppetwrappers Puppet Provider Examples Scenario: Confirming package installation When a machine has been puppeted Then the bash package should be installed Scenario: Confirm doodoodoo package is absent When a machine has been puppeted Then the doodoodoo package should not be installed Scenario: Confirm cron service is running When a machine has been puppeted Then the cron service should be running Scenario: Confirm tomcat6 service is not running When a machine has been puppeted Then the tomcat6 service should not be running Scenario: Confirm dwilson is in libvirtd group When a machine has been puppeted Then dwilson should be a member of libvirtd Scenario: Confirm dwilson has a uid of 1000 When a machine has been puppeted Then dwilson should have a uid of 1000 Scenario: Confirm dwilson has a given shell When a machine has been puppeted Then dwilson should have the /bin/bash shell I really like using the puppet providers for this because of the abstraction benefits they provide. Read on →

Tonights (the March 2011) London OpenSolaris User Group (LOSUG) was a little different to usual and while the topics have always been quite diverse we’ve never had as seditious a talk as one covering the Solaris fork, OpenIndiana, Illumos and the OpenSolaris community. Alasdair Lumsden did an excellent job of explaining the new projects, why they exist and what they’re aiming for. As someone who took a few steps back when Oracle purchased Solaris it was an interesting catch up. Read on →

It all started with one of those annoying little items on the todo list - find all the unpuppeted ssh authorized_keys files on a machine and alert on them. On first impressions it was going to be quite manual (always a bad sign), involve digging in to legacy installs and would be something we’d need to re-verify occasionally. It couldn’t be that bad though could it? After all how many places can an unmanaged-by- puppet sshkey live? Read on →

One of puppets more under-appreciated features is its ability to abstract and smooth the edges of certain operating system tasks and behaviours. Even something as trivial as installing a package can actually become a portability nightmare once you consider the number of different systems in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands needed to use them. You end up either hard coding commands, and sacrificing portability, or writing your own detection, lookup and invocation logic. Read on →

It might be a sign that I spend too much time online but the quicker a system gives me feedback the more useful I find it. While I love knowing my Nagios safety net has me covered when making changes sometimes waiting for that cgi to refresh can take too long, especially if I’m taking a iterative / test driven approach to the changes I’m making. For those use cases I wrote nrpe-runner. Read on →

While adopting a configuration management tool like Chef and Puppet will have a large, nearly immediate, effect on your work flow even after using the tools for a while you’ll still get a little smile at all the little niceties you continuously discover. One recent small win we had recently was bringing some apache configs files under Puppet command. When we started we had the following block of config: RewriteCond %{REMOTE_ADDR} !10.23.143.33 RewriteCond %{REMOTE_ADDR} !10.23.143.2 RewriteCond %{REMOTE_ADDR} !10.23.143.3 It’s not hard to read and roughly understand what it does, but you have no real context; magic numbers keep things terse but are rarely the most helpful when in the land of a strange system. Read on →