Dean Wilson@UnixDaemon: In search of (a) life --

Introduction To DSAC

Tue, 05 Jul 2011 17:45:00 GMT

A while ago @ripienaar and I had a chat in a pub about monitoring, event systems and lots of related subjects. As we all know he's way more productive than is fair and so while he's been doing a BUNDLE of work with on subjects like monitoring frameworks and event correlation I've been doing some thinking (and no actual coding) about event auditing, continuous compliance and security event management.

Now I've finished the $TIMESINK_PROJECT I'm soon going to actually need some of this stuff so I've started putting together a prototype framework that I'm calling DSAC - Dump Send and Correlate. The code is in a very early stage at the moment but is dealing with a small number of agents on a test network of a couple of hundred nodes. I'm going to start documenting the sections as it becomes ready for more public consumption but I thought I'd show my architectural plans for version 0.1.

The architecture is quite simple at the moment. Every node runs the "consumer and dispatch" stack which generates events, currently all events are made from cron invoked agents. A separate process, also cron invoked (for now) then runs through the spool and invokes all the dispatchers that have registered an interest in the output of that agent. Simple dispatcher examples are an AMQ pusher or a MySQL loader.

At the other end of the process, and quite symmetrically, we have the consumer stack. This reads from the nice big fuzzy cloud of transient data loss and spools files for later processing. We then have another process pick the files up and run them through a number of processors.

I've got working prototypes of a simple bulk archiver and some debugging aids but I can also envision some more useful real time dashboards. The last stage at the moment are the simple reports. I'm currently focusing on the easier reports that will help me show changes to an auditor, package updates, service status changes and user logins but this step will hopefully expand to encompass a lot of our rote compliance needs.

Once I've tidied up the code (and picked up some more ruby!) I'll start putting the bits I work on in my spare time on github.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Simple Puppet module grepper (prototype)

Mon, 20 Jun 2011 23:36:00 GMT

Search for puppet resources values using puppet, not just plain text

One of the ideas that has been sitting on my todo list is having a command that lets me grep a puppet manifest for certain properties, values or even just resources in a smarter way than just running a raw grep over files. While a simple grep works in some cases it is annoyingly fragile when you're trying to ignore literal strings in resource types that you're not interested in or narrow your search down to resources that have a property that can also appear in other types.



  # Show all file resources with a mode of 644
  $ pm-grep -t file -p mode -v 644 files.pp

  # Show all host resources with an alias of any value
  $ pm-grep -t host -p host_aliases hosts.pp

  # Check a number of pp files at once
  $ find /etc/puppet/modules/ -name "*.pp" | xargs -n 1 pm-grep -t file -p mode

pm-grep (puppet manifest grep) isn't anywhere near finished but it does work on simple manifests. It yet doesn't handle corner cases, global parameter defaults and a number of other more advanced techniques but it does fulfil some of my needs and has given me some more to mull over for version 2.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Smarter Service Status in Puppet

Thu, 16 Jun 2011 16:22:00 GMT

While most people know you can use puppet to ensure a service is running the mechanism it uses to determine if a service is actually running is often unexplored.

By default (at least up to Puppet 2.6) puppet assumes that a service doesn't supply a working status option and so will look up the services name in the process table to check if it's running. If your service does support the status argument you can set 'hasstatus => true' and the platforms service provider will be used to interrogate the services current status.

While most services only report a simple status of running or not running puppet, when you've specified 'hasstatus => true' puppet will consult a second property, if it's present, - status - which is where things get a little more interesting and extendable.


  # puppet manifest
  service { "httpd":
      ensure    => "running",
      hasstatus => true,
      status    => "/usr/local/bin/puppet-status-http-check",
  }


  # puppet-status-http-check - example check

  #!/usr/bin/perl
  use strict;
  use warnings;

  my @checks = (
    "/usr/lib/nagios/plugins/check_procs -C httpd",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u /about",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u / -s udlab",
  );

  for my $check ( @checks ) {
    $check .= " 2>&1 > /dev/null"; # suppress output
    system( $check ) == 0 or exit 1;
  }

  # when running under debug you'll see a line like:
  debug: Service[httpd](provider=redhat): Executing '/usr/local/bin/puppet-status-http-check'

By specifying our own command in the status property we can do more complex, and domain specific, status checks. For example we don't so much care that apache is running as that it's serving our chosen vhosts correctly. You can use any command as the right hand side of status and puppet will treat a return code of 0 as confirmation that the service is running and anything else as a failure; which will trigger an attempt to restart the service in our example.

One possibility is to tie this in to nrpe-runner with a carefully chosen command name pattern to reap all the benefits of your already defined nagios checks.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

VMware vSphere 4.1 HA and DRS deepdive - Short Review

Sun, 22 May 2011 08:49:00 GMT

It's been years since I've read a book on VMWare. Between the maturity and ease of use of their GUI tools and my own continual move towards Free virtualisation I've not had the professional need or the spare time to invest but when a book comes as highly recommended as the VMware vSphere 4.1 HA and DRS Technical deepdive does you have to make some room on your (virtual) bookshelf.

Despite its small page count this book covers its subject material in a simple, direct and technically clear way. There is very little fluff and while you could find some of the details buried in VMWare KB articles or white papers its presence here in such a well combined and cohesive form more than justifies the books frankly tiny price tag (at least in the kindle store).

I came away from this book with enough of an understanding of the technologies covered to see where they'd fit, the issues we'd need to monitor for and some of the edge cases that would bite us in deployment. And that's a good return for the small investment of time reading this book takes.

The only downside of the book is that it could really do with another editorial pass or two. While this doesn't alter the quality of the technical content it does make the reading experience a little jarring.

If you want to get in to vSphere HA / DRS then this is a recommended read. Score - 7/10

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Wrapping MCollective with Nagios

Sat, 14 May 2011 16:55:00 GMT

I've been doing a little tinkering with pre/post release checklists and compliance reporting using cucumber and some Nagios wrapping (among other things) in my test lab and recently needed to do some higher level entire environment checks before moving on to the next step. While it's possible to wrap something like nmaps ping check and then Nagios each target it does feel like stepping back a few years in the tool chain.

Luckily I'm running MCollective, so all this synchronous discovery and polling is in my past. After a little bit of delving in to the existing package and service clients I've come up with a prototype environment wide MCollective backed service check and an MCollective backed package check.

I'm not sure if I'd be willing to replace existing low level checks (for things like cron and ssh processes) with this just yet but it does show how easy it is to wrap MCollective with third party code in order reap its benefits from further down the tool chain. With a little scaffolding hopefully it'll be useful in validating individual policies in security policies and guidelines. But more about that later.

Phase two is probably to pull the scripts together (and just use another parameter to select the resource to check) and to be green or red based on percentage. As an example, requiring 40% of the web servers to be returning 200 before starting the next batch of host upgrades.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

ep.io and VMWare at London Devops - May 2011

Sat, 14 May 2011 11:56:00 GMT

I never thought I'd use a cliche like "David vs Goliath" but considering the two speakers at London Devops it does seem a little apt. Andrew Godwin from ep.io, a Python hosting platform, was the first speaker, and he did an excellent job of explaining their internal platform, how they make their decisions and what makes them special. While it was both an interesting and engaging talk it did leave me a little worried about the size of the operation.

While small companies are great to deal with in the right situations they can also be a risk due to their low survival odds, questionable ability to grow alongside you and inability to throw resources at an awkward but urgent problem. On the other hand they can provide better levels of support, knowledge and assistance if you can find a good one and treat them more as partners than vendors, and I suspect that ep.io is going to be one of the good ones.

Then we had the VMWare talk. Until a couple of years ago, when budgets shrank again and Xen and KVM began to rise, I was a big fan and a happy user of VMWare products both on server and desktop. While I've not kept up with all the product details it's hard not to have heard of CloudFoundry.

The two speakers, one from RabbitMQ and one from SpringSource (both now part of the VMWare org chart) had very different speaking styles, the speaker from RabbitMQ had a keen wit and kept the tone light with lots of amusing comments like "VMWare is about 9000 staff, about 8000 of them write device drivers" and while the man from SpringSource spent the whole time complaining about how slow his laptop was. At one point the audience nearly had a whip-round to cover the cost of a couple of GB of RAM for him. As for the content it left me a little adrift. I came out of the talk without knowing much more than I went in with. Although I always have to smile when I hear people from SpringSource describe their product line, Spring Tomcat, Spring AMQ, Spring ls and Spring Bash (I might have made the last two up) so it wasn't a complete waste.

Obviously there will be comparisons made between the talk platforms being discussed and one of the most interesting aspects of the evening for me was how well ep.io came out of the deal. They've got an architecture every bit as well thought out as that of VMWares, they're already looking at the next set of problems that both platforms are going to experience and they came across as remarkable professional for such a small team.

CloudFoundry on the other hand will probably have a bigger effect on my working life. VMWare is often quite an easy sell due to its track record and feature set and I can see more companies talking parts of CloudFoundry on board than I can see them hosting with ep.io. So it's one to spend a little time investigating. The fact that it's open source will just make the whole process easier.

The talks were very well attended with 70-80 people in the audience and once again we should say thank you to the Guardian for providing the venue and Gareth for organising it.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Linux Open Administration Days 2011

Tue, 19 Apr 2011 15:29:00 GMT

Last year at one of the many Belgium tech events Kris mentioned a conference called LOAD (2010) to me. I was a little late in booking the hotel and in the end I couldn't make it over - and judging by the quality of this years event that was a big mistake.

While it's nice to spend time in the devops world and talk about communication, processes and how to merge development and operational tool-chains sometimes it's nice to focus on solid, production grade sysadmining; and LOAD was the perfect conference for it. Over two days, two tracks of talks and one of tutorials, a selection of top notch speakers covered kerberos, LDAP, packaging (Debs and RPMs), storage systems, single sign on, advanced networking, virtualisation, security, HA and monitoring. Some of the talks presented were perfectly timed (DNSSEC and IPv6 from a working admins perspective), some were very solid updates on technologies we sometimes take for granted (PKI, LDAP, SSO and HA clustering) and some covered more vertical admin niches (inventory systems, Exchange replacements and small business servers).

The conference felt like a large local LUG meeting. The people were friendly, the sessions and speakers encouraged the audiences involvement both in and outside of the talks and even when the event was over everyone seemed happy to stay and chat about what they'd seen or further discuss subjects with the speakers (although I suspect the free food and drink didn't hurt in keeping the conference going after hours!)

The LOAD organisers did a marvellous job of finding so many talented speakers and promoting home grown talent. I'd only seen maybe a dozen of the people speak before and the amount of preparation each and every speaker had obviously invested made being in the audience a pleasure. No one was "quipping" about the fact they'd only just written their slides or started to prepare and a number of the speakers tailored their talks based on the other sessions to help reduce duplication and present their own take on certain subjects - and their talks, and the conference, were enhanced by it.

This post may seem a little gushing but this was the best sysadmin conference I've been to for years. I've come back with information that's going to help me do my job better and it's going to be one of the first conferences I book next year.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

HBGary Open Letter - Air Gap

Tue, 19 Apr 2011 13:33:00 GMT

Our source code has always been air gapped from the Internet. The forensic examination confirmed that software development servers and workstations were not affected by the incident -- from HBGary

Anyone else find it hard to accept that none of the developers, testers, documentation writers or build people ever accessed source code from their Internet connected laptops / workstations? Especially considering the state of their other security measures.

Don't get me wrong, in some cases it's a sensible solution ( off-line key signing for example) but for entire teams working on a shared code base?

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Listing Puppet Managed Files

Tue, 22 Mar 2011 22:54:00 GMT

Sometimes it's the little niggles that annoy people the most. As my team progress in to puppet they have an annoying habit of asking very good questions; which can sometimes be a struggle to answer. Todays best question was - "How do I tell if this file is under puppets control?"

While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. While this gets you most of the way the problem is a little harder than it looks because of an edge case. If puppet is managing an entire directory then the files in that directory are not explicitly listed in the catalog.

So we need to look in two places, the catalog and state.yaml. Remembering the greps (and the line transformations needed) requires more mental space than I'm willing to invest so I've written puppet-ls to do all the work for me.


$ puppet-ls /etc/mcollective
/etc/mcollective/facts.yaml
/etc/mcollective/server.cfg

Run the command, specify the directory to check and any shown files are puppet managed. It's not a ground breaking script but it can help people migrating to puppet as they bring more of their systems under its control.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Nagios Wrapped Puppet Runs

Mon, 21 Mar 2011 22:56:00 GMT

Log nrpe-runner state changes when puppet runs to see what broke or was fixed.

While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I've come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net.

One of the lesser known features introduced in Puppet 0.25.4 (and still valid in 2.6) were the prerun_command and postrun_command hooks. These two config settings allow you to specify a command to run at the beginning (which can stop the puppet run from happening) and at the end of a puppet run. While they were originally devised to make integration with etckepper simpler we can also use them to add some additional monitoring to our runs.

We've already covered my nrpe-runner, which lets you run Nagios checks locally for immediate feed back but now let's expand the idea a little for puppet integration. Our plan is simple, invoke nrpe-runner and gather the output, run puppet, re-run the nrpe-runner and see which checks puppet has fixed or broken.

First of all we deploy nrpe-runner, our nrperunner json differ and the (below) wrapper script we use for when puppet's finished running.


$ cat nrpe-wrapper

#!/bin/bash
/home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/post_puppetrun 
logger -t "puppet-nrpe" `/home/deanw/puppet-wrapper/nrperunner-json-differ /tmp/pre_puppetrun /tmp/post_puppetrun`

We then add the config to puppet.confs main section. While it's possible to insert longer lines for each command and skip the wrapper script puppet is a little fiddly about these settings and a separate script is easier to use.


$ cat /etc/puppet/puppet.conf
[main]
  ... snip ...
    prerun_command  = /home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/pre_puppetrun
    postrun_command = /home/deanw/puppet-wrapper/nrpe-wrapper

Now we've done all the prep (and if needed restarted puppet) let's break something and see if we get both a fix and confirmation:


# stop something we know puppet will fix.
$ /etc/init.d/mcollective stop

$ puppetd -vt
info: Retrieving plugin
 .. snip ...
notice: //mcollective::server/Service[mcollective]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 5.51 seconds

# see if we logged the fix... we did!
$ tail -n 1 /var/log/messages
Mar 21 22:07:21 lb03-dynm puppet-nrpe: mcollective_procs changed from 2 to 0

While our simple wrapper just sends the output directly to syslog hopefully you've got an idea how powerful this integrated immediate feedback can be. While it's always been possible for us to dig back through the logs and spot something breaking after a puppet run, by explicitly wrapping the run we can cut done the investigation time while also providing information for later review and discussion.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Puppet Cucumber Providers

Thu, 17 Mar 2011 19:16:00 GMT

At work we try, and sometimes even succeed, in using Test Driven Deployment so as one of my background projects I've been wrapping certain tools in to cucumber friendly forms. Over the last couple of days I've been grabbing ten minutes here and there to incorporate Puppet 2.6 in to the pile.


Feature: Puppetwrappers
  Puppet Provider Examples

  Scenario: Confirming package installation
    When a machine has been puppeted
    Then the bash package should be installed

  Scenario: Confirm doodoodoo package is absent
    When a machine has been puppeted
    Then the doodoodoo package should not be installed

  Scenario: Confirm cron service is running
    When a machine has been puppeted
    Then the cron service should be running 

  Scenario: Confirm tomcat6 service is not running
    When a machine has been puppeted
    Then the tomcat6 service should not be running

  Scenario: Confirm dwilson is in libvirtd group
    When a machine has been puppeted
    Then dwilson should be a member of libvirtd

  Scenario: Confirm dwilson has a uid of 1000
    When a machine has been puppeted
    Then dwilson should have a uid of 1000

  Scenario: Confirm dwilson has a given shell
    When a machine has been puppeted
    Then dwilson should have the /bin/bash shell

I really like using the puppet providers for this because of the abstraction benefits they provide. I can write steps to test packages, services or aspects of a user and not have to worry if a developer runs it on Fedora or Debian.

This is only a first draft, and the cucumber wording needs changing, but I thought I'd put it online to show how expressive cucumber can be for system tasks and how easy, and concise, it is to reuse the puppet providers. You can grab the puppet step code and the Puppet providers features to drop in to your own test harnesses and have a play with. The implementation is pretty simple, for example the code below is everything you need for the service scenarios:


Then /^the (.+) service should be running$/ do | service |
  service_status = Puppet::Type.type(:service).new(:name => service, :hasstatus => true).provider.status
  service_status.should == :running
end

Then /^the (.+) service should not be running$/ do | service |
  service_status = Puppet::Type.type(:service).new(:name =>service).provider.status
  service_status.should == :stopped
end

It's worth mentioning that all the above will only work in 2.6 and above as the internal details returned by the providers are different to those in 2.5.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

OpenIndiana - LOSUG March 2011

Thu, 17 Mar 2011 00:03:00 GMT

Tonights (the March 2011) London OpenSolaris User Group (LOSUG) was a little different to usual and while the topics have always been quite diverse we've never had as seditious a talk as one covering the Solaris fork, OpenIndiana, Illumos and the OpenSolaris community.

Alasdair Lumsden did an excellent job of explaining the new projects, why they exist and what they're aiming for. As someone who took a few steps back when Oracle purchased Solaris it was an interesting catch up. The short version seems to be that "Illumos is a derivative of OS/Net (aka ON), which basically is a Solaris/OpenSolaris kernel with the bulk of the drivers, core libraries, and basic utilities." (from Wikipedia) and is being quite heavily invested in by companies (such as Joynet and Nexenta) and by individuals that were previous employed by Sun to work on Solaris. OpenIndiana is to become an OpenSolaris distribution and packaged software ecosystem.

To me the project has a similar feel to the early days of CentOS and Scientific Linux and I think my biggest take home is that Illumos and OpenIndiana, when taken together, want to be to Solaris what CentOS and Scientific Linux are to Red Hat Enterprise Server. Unfortunately they have a massive disadvantage as Solaris, unlike upstream Red Hat, isn't entirely open. One of the most immediately visible casualties is the excellent ZFS, which is closed source upstream and will both lag behind and diverge from the official Solaris version. Which I consider to be a great loss.

On a more cheerful note the OpenIndiana project is looking for people with an interest in taking free Solaris forward and is still young enough that there are plenty of interesting aspects to get involved with. Websites, CI environments (I'm guessing they won't use Hudson. Heh) and all the other usual roles a large opensource project needs filling are up for grabs.

The talk itself was quite well attended, with what looked to be 35-40 people in the audience, and well presented. It's also the first time I've been in to Oracles Moorgate offices and they're actually quite nice and modern. The open sided lift and the suspended spiral staircase that only serves three floors were personal highlights.

I wish the project well and hope it enjoys success while being able to retain some of what made Solaris great. I may even take the DVD for a spin...

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Find Unpuppeted SSH Keys

Mon, 14 Mar 2011 23:30:00 GMT

It all started with one of those annoying little items on the todo list - find all the unpuppeted ssh authorized_keys files on a machine and alert on them. On first impressions it was going to be quite manual (always a bad sign), involve digging in to legacy installs and would be something we'd need to re-verify occasionally. It couldn't be that bad though could it? After all how many places can an unmanaged-by- puppet sshkey live?

Essentially the task can be broken in to three main parts. The first, quite easy part, is to grab a list of all the users (hello /etc/passwd) and look for known key file names in their home directories. The second part, which was a little harder, is to build a list of all the authorized_keys files that puppet knows it's managing for this host. Lastly once you have the two collections find the differences. Instead of doing static analysis on the puppetmasters classes and modules we're going to focus on how to do it using the compiled desired state of what the local machine should look like, according to the puppet catalog.

The catalog (which lives at /var/lib/puppet/client_yaml/catalog/$fqdn.yaml in modern puppet) is a yaml-based representation of what puppet knows about how the local system should be configured. It contains details of all the resources to be managed on the local machine and their desired end state; which makes it perfect for our needs. I'm not going to go into the catalog in depth in this post but hopefully this little example will whet your appetite and spark some ideas.

Our example, the audit-sshkey-files nagios check, was actually quite easy to write (after some digging in to puppet and borrowing some code from Puppet Catalog Diff by R.I.Pienaar) and should hopefully show how much you can gain from using the meta-data puppet provides.

While most of the audit-sshkey-files script is boilerplate the most important snippet is below:


  if target.type == "File" and target.title.include? "/authorized_keys"
    @puppet_keys.push target.title
    return target.title
  end

All we're doing is building a list of any resources that are of type file and include the string "/authorized_keys" in their name (resource title in puppet terms). While this may not seem like much it's potentially game changing, any resources or relationships that you've modelled in puppet can be later mined to add context to your other tools. You can (as we have here) audit security related files or find user ids puppet doesn't know about and so might be inconsistent over systems. By using the catalog and the relationships and meta-data it provides you can make much more of your investment in deploying systems with puppet, and hopefully this little example presents an easy way to get started.

Now I've gushed about what the puppet catalog can do for you there are two caveats, firstly about my example. It isn't a complete solution, for example it doesn't look for other allowed "authorized_keys" filenames that are defined in the sshd_config file. But it does the 80% of what I needed in our environment and by managing the sshd_config file in puppet (as you should be) it's easy for me to double check I'm looking for the correct files. Secondly about the Puppet catalog itself. Harnessing its contents doesn't exactly have a shallow learning curve and documentation is a little thin on the ground. The original author of puppet Luke Kanies is working on some alternative ways of accessing this kind of information (such as via his Puppet Interfaces project) and as more people build their puppet deployments you can expect so see more and more harnessing of this additional structure.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Reusing Puppets Package providers

Mon, 14 Mar 2011 00:18:00 GMT

One of puppets more under-appreciated features is its ability to abstract and smooth the edges of certain operating system tasks and behaviours. Even something as trivial as installing a package can actually become a portability nightmare once you consider the number of different systems in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands needed to use them. You end up either hard coding commands, and sacrificing portability, or writing your own detection, lookup and invocation logic.

That sounds like, dull, scut work so how does puppet deal with this? And how can we reuse this work to simplify our own code? In slightly simplified terms, Puppet has a package type, which is backed by a number of providers. Each of these providers actually implement the required functionality for a given package manager and contains all the code we need. So how do we harness this existing work? Quite easily. Luckily for us, puppets providers are written in ruby code and are simple to call in our own scripts:



# show package version
$ irb

irb(main):001:0> require 'puppet'
=> true

irb(main):002:0> Puppet::Type.type(:package).new(:name => "bash").provider.properties
=> { :provider=>:yum, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
=>   :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }

# do the same thing with an explicitly specified provider.
irb(main):003:0> Puppet::Type.type(:package).new(:name => "bash", :provider => "rpm").provider.properties
=> { :provider=>:rpm, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
     :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }

While that snippet will hopefully whet your appetite if you need a more worked example I've put a small Puppet Package Provider wrapper up on github. The script will enable you to do the basic install, update and delete without knowing or caring what the underlying package manager is. Hopefully these little code snippets will help you stop thinking of puppet as "just" a tool and show how parts of its code base can be used as a framework to improve other parts of your tool chain.

As an aside it's also worth mentioning that you can globally Change the Package provider in puppet if you're not happy with its auto-detection.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Introducing NRPE Runner

Wed, 09 Mar 2011 19:05:00 GMT

It might be a sign that I spend too much time online but the quicker a system gives me feedback the more useful I find it. While I love knowing my Nagios safety net has me covered when making changes sometimes waiting for that cgi to refresh can take too long, especially if I'm taking a iterative / test driven approach to the changes I'm making. For those use cases I wrote nrpe-runner.

The way I typically use Nagios is to have the Nagios server run the checks on the remote host via the NRPE plugin. The checks to be run on the host are normally stored in a config file with each entry looking like this:


command[local_mail]=/usr/local/libexec/nagios/check_local_mail

While this allows you to run each check to confirm that it's still OK I wanted the ability to run all the commands in the file at once, which I can now do with nrpe-runner. If every thing's fine then it exits silently, to confirm that it's actually run I can summarise and even filter the checks to run:



# show everything as it's run whatever the return status
/usr/local/sbin/nrpe-runner -a
check_swap => SWAP OK - 100% free (16041 MB out of 16041 MB) |swap=16041MB;12031;9624;0;16041
... snipped ...
freemem => OK: 12% (1732M) free memory.

# show a summary
$ /usr/local/sbin/nrpe-runner -s
Ran 39 checks - OK 39. WARN 0, CRIT 0, UNKNOWN 0

# run any checks with ntp in the name (the part between [])
$ /usr/local/sbin/nrpe-runner -s -n ntp
Ran 3 checks - OK 3. WARN 0, CRIT 0, UNKNOWN 0

# run all process checks (checks the command after the '=')
$ /usr/local/sbin/nrpe-runner -s -c proc
Ran 17 checks - OK 17. WARN 0, CRIT 0, UNKNOWN 0

# show all checks named ntp
$ /usr/local/sbin/nrpe-runner -a -n ntp
ntp_skew_primary => NTP OK: Offset -0.003149271011 secs|offset=-0.003149s;5.000000;9.000000;
ntp_process => PROCS OK: 1 process with command name 'ntpd', args '-u ntp:ntp'
ntp_skew_secondary => NTP OK: Offset -0.002887368202 secs|offset=-0.002887s;5.000000;9.000000;

nrpe-runner also has the option to dump the results as json, which I'll be exploring a little further in my next couple of blog posts. While it's not exactly the same as having the checks run by nagios (the user and environment are often different) I've found that shortening the interval between running puppet or yum and seeing the nagios feedback has helped my work-flow quite a lot when making exploratory system changes - and even more when nothing should have changed but does...

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Puppet CookBook is live

Tue, 04 Jan 2011 22:59:00 GMT

Between Xmas and New Year I had some spare time to invest on a side project I've been looking forward to working on for quite a while. I'm pleased to announce the opening of the Puppet CookBook.

I've introduced Puppet to quite a few companies, sysadmins and development teams over the years and a lot of the same issues, concepts and needs repeatedly crop up. By explaining how puppet works in terms of tasks and desired outcomes rather than in raw feature descriptions I hope to show some of its power and flexibility in easy to use examples in a different way to most of the existing documentation.

The site isn't exactly brimming over with content yet (and it's pretty ugly) but I'm adding a handful of posts each week and hope to cover some more advanced topics over the next couple of months. You can follow the Puppet CookBook Twitter account for update announcements or to send feedback or suggestions for future topics.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Hadoop: The Definitive Guide - Short review

Sun, 12 Dec 2010 09:12:00 GMT

Hadoop is one of those technologies that seems to have forever changed the way parts of the industry work but has had no effect on my actual job. In an attempt to keep myself current for the after techtalk conversations I decided to buy Hadoop: The Definitive Guide, Second Edition by Tom White - and I'm very happy with the choice.

While there are massive amounts of information online about Hadoop and the ecosystem emerging around it I still found HadoopTDG to be a useful book and worth the money (especially on the iPad as it's a bit big for comfortable tube reading). The explanations are clear, there is enough detail without slowing the book to a crawl and some of the more important side projects are covered, showing outsiders like me which subprojects can help build the bridge in to my existing infrastructure.

A good book, covers a lot of ground and provides a good level of detail - 7/10

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Clarifying With Facter

Sat, 11 Dec 2010 21:35:00 GMT

While adopting a configuration management tool like Chef and Puppet will have a large, nearly immediate effect on your work flow even after using the tools for a while you'll still get a little smile at all the little niceties you continuously discover.

One recent small win we had recently was bringing some apache configs files under Puppet command. When we started we had the following block of config:


RewriteCond %{REMOTE_ADDR} !10.23.143.33
RewriteCond %{REMOTE_ADDR} !10.23.143.2
RewriteCond %{REMOTE_ADDR} !10.23.143.3

It's not hard to read and roughly understand what it does, but you have no real context; magic numbers keep things terse but are rarely the most helpful when in the land of a strange system. After putting the configs in to a module and abstracting them a little into a template we have the much nicer:


RewriteCond %{REMOTE_ADDR} !<%= primary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= secondary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= ipaddress_eth0_mgmt %>

As part of the tidy up we also renamed some of the (remarkably large amount of) Ethernet interfaces to describe what they were for, rather than leaving them as eth12:34

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

London Perl Workshop 2010

Wed, 08 Dec 2010 23:23:00 GMT

Over the years I've had the opportunity to attend a lot of different events focused on quite a few different programming languages, but none of them match the sheer enthusiasm and love of the language that you get from London PM. While there is always a contingent of LPMers at Perl conferences held further abroad the London Perl Workshop is my yearly chance to see lots of old friends, what they've been up to and discuss what's coming next in our field.

Other than the 3 1/2 hour tube problems getting to the venue (and having to leave the pub early) I had a great time, the organisation and volunteers were as always exceptional and it was a great idea to try and get some speakers from outside the community - and doubly so when you're lucky enough to get the seriously clued PostgreSQL expert Simon Riggs.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

MCollective Plugin - FileMD5er

Mon, 15 Nov 2010 23:26:00 GMT

I've been watching the Marionette Collective for a while, and even gave it a small trial in a couple of testing environments, but this weekend was the first time I've experimented with it at a slightly larger scale (just over a hundred small VM nodes - you have to love EC2) and I'm still impressed.

I can see how it's going to make parts of my work flow easier, and in an attempt to learn a little more about how the plugin system works under the hood I decided to write a small agent, FileMD5er. The agent itself is very simple and addresses a small annoyance I've scripted around for a while. When you're bringing files under Puppet (or Chef) management you need to dig through the hosts and locate any files with differences compared to the most common adhoc file. With a quick mc-filemd5er /path/to/file I can easily spot any machines that have a slightly different version of the file, and then fold them in to centralised management.

Writing the plugin itself was quite easy. The two problems I encountered were finding the right generation of existing plugin to crib from (some of the official MCollective Plugins are of a newer format than others) and not naming the class and the .rb file the same name. Which caused it to half work.

I'll be putting more of my MCollective Plugins on Github as the become a little more generic and hopefully useful to someone else.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!