Dean Wilson@UnixDaemon: In search of (a) life --

Simple Puppet module grepper (prototype)

Mon, 20 Jun 2011 23:36:00 GMT

Search for puppet resources values using puppet, not just plain text

One of the ideas that has been sitting on my todo list is having a command that lets me grep a puppet manifest for certain properties, values or even just resources in a smarter way than just running a raw grep over files. While a simple grep works in some cases it is annoyingly fragile when you're trying to ignore literal strings in resource types that you're not interested in or narrow your search down to resources that have a property that can also appear in other types.



  # Show all file resources with a mode of 644
  $ pm-grep -t file -p mode -v 644 files.pp

  # Show all host resources with an alias of any value
  $ pm-grep -t host -p host_aliases hosts.pp

  # Check a number of pp files at once
  $ find /etc/puppet/modules/ -name "*.pp" | xargs -n 1 pm-grep -t file -p mode

pm-grep (puppet manifest grep) isn't anywhere near finished but it does work on simple manifests. It yet doesn't handle corner cases, global parameter defaults and a number of other more advanced techniques but it does fulfil some of my needs and has given me some more to mull over for version 2.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Smarter Service Status in Puppet

Thu, 16 Jun 2011 16:22:00 GMT

While most people know you can use puppet to ensure a service is running the mechanism it uses to determine if a service is actually running is often unexplored.

By default (at least up to Puppet 2.6) puppet assumes that a service doesn't supply a working status option and so will look up the services name in the process table to check if it's running. If your service does support the status argument you can set 'hasstatus => true' and the platforms service provider will be used to interrogate the services current status.

While most services only report a simple status of running or not running puppet, when you've specified 'hasstatus => true' puppet will consult a second property, if it's present, - status - which is where things get a little more interesting and extendable.


  # puppet manifest
  service { "httpd":
      ensure    => "running",
      hasstatus => true,
      status    => "/usr/local/bin/puppet-status-http-check",
  }


  # puppet-status-http-check - example check

  #!/usr/bin/perl
  use strict;
  use warnings;

  my @checks = (
    "/usr/lib/nagios/plugins/check_procs -C httpd",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u /about",
    "/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u / -s udlab",
  );

  for my $check ( @checks ) {
    $check .= " 2>&1 > /dev/null"; # suppress output
    system( $check ) == 0 or exit 1;
  }

  # when running under debug you'll see a line like:
  debug: Service[httpd](provider=redhat): Executing '/usr/local/bin/puppet-status-http-check'

By specifying our own command in the status property we can do more complex, and domain specific, status checks. For example we don't so much care that apache is running as that it's serving our chosen vhosts correctly. You can use any command as the right hand side of status and puppet will treat a return code of 0 as confirmation that the service is running and anything else as a failure; which will trigger an attempt to restart the service in our example.

One possibility is to tie this in to nrpe-runner with a carefully chosen command name pattern to reap all the benefits of your already defined nagios checks.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Listing Puppet Managed Files

Tue, 22 Mar 2011 22:54:00 GMT

Sometimes it's the little niggles that annoy people the most. As my team progress in to puppet they have an annoying habit of asking very good questions; which can sometimes be a struggle to answer. Todays best question was - "How do I tell if this file is under puppets control?"

While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. While this gets you most of the way the problem is a little harder than it looks because of an edge case. If puppet is managing an entire directory then the files in that directory are not explicitly listed in the catalog.

So we need to look in two places, the catalog and state.yaml. Remembering the greps (and the line transformations needed) requires more mental space than I'm willing to invest so I've written puppet-ls to do all the work for me.


$ puppet-ls /etc/mcollective
/etc/mcollective/facts.yaml
/etc/mcollective/server.cfg

Run the command, specify the directory to check and any shown files are puppet managed. It's not a ground breaking script but it can help people migrating to puppet as they bring more of their systems under its control.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Nagios Wrapped Puppet Runs

Mon, 21 Mar 2011 22:56:00 GMT

Log nrpe-runner state changes when puppet runs to see what broke or was fixed.

While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I've come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net.

One of the lesser known features introduced in Puppet 0.25.4 (and still valid in 2.6) were the prerun_command and postrun_command hooks. These two config settings allow you to specify a command to run at the beginning (which can stop the puppet run from happening) and at the end of a puppet run. While they were originally devised to make integration with etckepper simpler we can also use them to add some additional monitoring to our runs.

We've already covered my nrpe-runner, which lets you run Nagios checks locally for immediate feed back but now let's expand the idea a little for puppet integration. Our plan is simple, invoke nrpe-runner and gather the output, run puppet, re-run the nrpe-runner and see which checks puppet has fixed or broken.

First of all we deploy nrpe-runner, our nrperunner json differ and the (below) wrapper script we use for when puppet's finished running.


$ cat nrpe-wrapper

#!/bin/bash
/home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/post_puppetrun 
logger -t "puppet-nrpe" `/home/deanw/puppet-wrapper/nrperunner-json-differ /tmp/pre_puppetrun /tmp/post_puppetrun`

We then add the config to puppet.confs main section. While it's possible to insert longer lines for each command and skip the wrapper script puppet is a little fiddly about these settings and a separate script is easier to use.


$ cat /etc/puppet/puppet.conf
[main]
  ... snip ...
    prerun_command  = /home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/pre_puppetrun
    postrun_command = /home/deanw/puppet-wrapper/nrpe-wrapper

Now we've done all the prep (and if needed restarted puppet) let's break something and see if we get both a fix and confirmation:


# stop something we know puppet will fix.
$ /etc/init.d/mcollective stop

$ puppetd -vt
info: Retrieving plugin
 .. snip ...
notice: //mcollective::server/Service[mcollective]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 5.51 seconds

# see if we logged the fix... we did!
$ tail -n 1 /var/log/messages
Mar 21 22:07:21 lb03-dynm puppet-nrpe: mcollective_procs changed from 2 to 0

While our simple wrapper just sends the output directly to syslog hopefully you've got an idea how powerful this integrated immediate feedback can be. While it's always been possible for us to dig back through the logs and spot something breaking after a puppet run, by explicitly wrapping the run we can cut done the investigation time while also providing information for later review and discussion.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Puppet Cucumber Providers

Thu, 17 Mar 2011 19:16:00 GMT

At work we try, and sometimes even succeed, in using Test Driven Deployment so as one of my background projects I've been wrapping certain tools in to cucumber friendly forms. Over the last couple of days I've been grabbing ten minutes here and there to incorporate Puppet 2.6 in to the pile.


Feature: Puppetwrappers
  Puppet Provider Examples

  Scenario: Confirming package installation
    When a machine has been puppeted
    Then the bash package should be installed

  Scenario: Confirm doodoodoo package is absent
    When a machine has been puppeted
    Then the doodoodoo package should not be installed

  Scenario: Confirm cron service is running
    When a machine has been puppeted
    Then the cron service should be running 

  Scenario: Confirm tomcat6 service is not running
    When a machine has been puppeted
    Then the tomcat6 service should not be running

  Scenario: Confirm dwilson is in libvirtd group
    When a machine has been puppeted
    Then dwilson should be a member of libvirtd

  Scenario: Confirm dwilson has a uid of 1000
    When a machine has been puppeted
    Then dwilson should have a uid of 1000

  Scenario: Confirm dwilson has a given shell
    When a machine has been puppeted
    Then dwilson should have the /bin/bash shell

I really like using the puppet providers for this because of the abstraction benefits they provide. I can write steps to test packages, services or aspects of a user and not have to worry if a developer runs it on Fedora or Debian.

This is only a first draft, and the cucumber wording needs changing, but I thought I'd put it online to show how expressive cucumber can be for system tasks and how easy, and concise, it is to reuse the puppet providers. You can grab the puppet step code and the Puppet providers features to drop in to your own test harnesses and have a play with. The implementation is pretty simple, for example the code below is everything you need for the service scenarios:


Then /^the (.+) service should be running$/ do | service |
  service_status = Puppet::Type.type(:service).new(:name => service, :hasstatus => true).provider.status
  service_status.should == :running
end

Then /^the (.+) service should not be running$/ do | service |
  service_status = Puppet::Type.type(:service).new(:name =>service).provider.status
  service_status.should == :stopped
end

It's worth mentioning that all the above will only work in 2.6 and above as the internal details returned by the providers are different to those in 2.5.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Find Unpuppeted SSH Keys

Mon, 14 Mar 2011 23:30:00 GMT

It all started with one of those annoying little items on the todo list - find all the unpuppeted ssh authorized_keys files on a machine and alert on them. On first impressions it was going to be quite manual (always a bad sign), involve digging in to legacy installs and would be something we'd need to re-verify occasionally. It couldn't be that bad though could it? After all how many places can an unmanaged-by- puppet sshkey live?

Essentially the task can be broken in to three main parts. The first, quite easy part, is to grab a list of all the users (hello /etc/passwd) and look for known key file names in their home directories. The second part, which was a little harder, is to build a list of all the authorized_keys files that puppet knows it's managing for this host. Lastly once you have the two collections find the differences. Instead of doing static analysis on the puppetmasters classes and modules we're going to focus on how to do it using the compiled desired state of what the local machine should look like, according to the puppet catalog.

The catalog (which lives at /var/lib/puppet/client_yaml/catalog/$fqdn.yaml in modern puppet) is a yaml-based representation of what puppet knows about how the local system should be configured. It contains details of all the resources to be managed on the local machine and their desired end state; which makes it perfect for our needs. I'm not going to go into the catalog in depth in this post but hopefully this little example will whet your appetite and spark some ideas.

Our example, the audit-sshkey-files nagios check, was actually quite easy to write (after some digging in to puppet and borrowing some code from Puppet Catalog Diff by R.I.Pienaar) and should hopefully show how much you can gain from using the meta-data puppet provides.

While most of the audit-sshkey-files script is boilerplate the most important snippet is below:


  if target.type == "File" and target.title.include? "/authorized_keys"
    @puppet_keys.push target.title
    return target.title
  end

All we're doing is building a list of any resources that are of type file and include the string "/authorized_keys" in their name (resource title in puppet terms). While this may not seem like much it's potentially game changing, any resources or relationships that you've modelled in puppet can be later mined to add context to your other tools. You can (as we have here) audit security related files or find user ids puppet doesn't know about and so might be inconsistent over systems. By using the catalog and the relationships and meta-data it provides you can make much more of your investment in deploying systems with puppet, and hopefully this little example presents an easy way to get started.

Now I've gushed about what the puppet catalog can do for you there are two caveats, firstly about my example. It isn't a complete solution, for example it doesn't look for other allowed "authorized_keys" filenames that are defined in the sshd_config file. But it does the 80% of what I needed in our environment and by managing the sshd_config file in puppet (as you should be) it's easy for me to double check I'm looking for the correct files. Secondly about the Puppet catalog itself. Harnessing its contents doesn't exactly have a shallow learning curve and documentation is a little thin on the ground. The original author of puppet Luke Kanies is working on some alternative ways of accessing this kind of information (such as via his Puppet Interfaces project) and as more people build their puppet deployments you can expect so see more and more harnessing of this additional structure.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Reusing Puppets Package providers

Mon, 14 Mar 2011 00:18:00 GMT

One of puppets more under-appreciated features is its ability to abstract and smooth the edges of certain operating system tasks and behaviours. Even something as trivial as installing a package can actually become a portability nightmare once you consider the number of different systems in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands needed to use them. You end up either hard coding commands, and sacrificing portability, or writing your own detection, lookup and invocation logic.

That sounds like, dull, scut work so how does puppet deal with this? And how can we reuse this work to simplify our own code? In slightly simplified terms, Puppet has a package type, which is backed by a number of providers. Each of these providers actually implement the required functionality for a given package manager and contains all the code we need. So how do we harness this existing work? Quite easily. Luckily for us, puppets providers are written in ruby code and are simple to call in our own scripts:



# show package version
$ irb

irb(main):001:0> require 'puppet'
=> true

irb(main):002:0> Puppet::Type.type(:package).new(:name => "bash").provider.properties
=> { :provider=>:yum, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
=>   :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }

# do the same thing with an explicitly specified provider.
irb(main):003:0> Puppet::Type.type(:package).new(:name => "bash", :provider => "rpm").provider.properties
=> { :provider=>:rpm, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
     :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }

While that snippet will hopefully whet your appetite if you need a more worked example I've put a small Puppet Package Provider wrapper up on github. The script will enable you to do the basic install, update and delete without knowing or caring what the underlying package manager is. Hopefully these little code snippets will help you stop thinking of puppet as "just" a tool and show how parts of its code base can be used as a framework to improve other parts of your tool chain.

As an aside it's also worth mentioning that you can globally Change the Package provider in puppet if you're not happy with its auto-detection.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Puppet CookBook is live

Tue, 04 Jan 2011 22:59:00 GMT

Between Xmas and New Year I had some spare time to invest on a side project I've been looking forward to working on for quite a while. I'm pleased to announce the opening of the Puppet CookBook.

I've introduced Puppet to quite a few companies, sysadmins and development teams over the years and a lot of the same issues, concepts and needs repeatedly crop up. By explaining how puppet works in terms of tasks and desired outcomes rather than in raw feature descriptions I hope to show some of its power and flexibility in easy to use examples in a different way to most of the existing documentation.

The site isn't exactly brimming over with content yet (and it's pretty ugly) but I'm adding a handful of posts each week and hope to cover some more advanced topics over the next couple of months. You can follow the Puppet CookBook Twitter account for update announcements or to send feedback or suggestions for future topics.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Clarifying With Facter

Sat, 11 Dec 2010 21:35:00 GMT

While adopting a configuration management tool like Chef and Puppet will have a large, nearly immediate effect on your work flow even after using the tools for a while you'll still get a little smile at all the little niceties you continuously discover.

One recent small win we had recently was bringing some apache configs files under Puppet command. When we started we had the following block of config:


RewriteCond %{REMOTE_ADDR} !10.23.143.33
RewriteCond %{REMOTE_ADDR} !10.23.143.2
RewriteCond %{REMOTE_ADDR} !10.23.143.3

It's not hard to read and roughly understand what it does, but you have no real context; magic numbers keep things terse but are rarely the most helpful when in the land of a strange system. After putting the configs in to a module and abstracting them a little into a template we have the much nicer:


RewriteCond %{REMOTE_ADDR} !<%= primary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= secondary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= ipaddress_eth0_mgmt %>

As part of the tidy up we also renamed some of the (remarkably large amount of) Ethernet interfaces to describe what they were for, rather than leaving them as eth12:34

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

MCollective Plugin - FileMD5er

Mon, 15 Nov 2010 23:26:00 GMT

I've been watching the Marionette Collective for a while, and even gave it a small trial in a couple of testing environments, but this weekend was the first time I've experimented with it at a slightly larger scale (just over a hundred small VM nodes - you have to love EC2) and I'm still impressed.

I can see how it's going to make parts of my work flow easier, and in an attempt to learn a little more about how the plugin system works under the hood I decided to write a small agent, FileMD5er. The agent itself is very simple and addresses a small annoyance I've scripted around for a while. When you're bringing files under Puppet (or Chef) management you need to dig through the hosts and locate any files with differences compared to the most common adhoc file. With a quick mc-filemd5er /path/to/file I can easily spot any machines that have a slightly different version of the file, and then fold them in to centralised management.

Writing the plugin itself was quite easy. The two problems I encountered were finding the right generation of existing plugin to crib from (some of the official MCollective Plugins are of a newer format than others) and not naming the class and the .rb file the same name. Which caused it to half work.

I'll be putting more of my MCollective Plugins on Github as the become a little more generic and hopefully useful to someone else.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Simplifying File Permissions in Puppet Manifests

Wed, 23 Sep 2009 22:16:00 GMT

I've been a user of Puppet for about three years now and while on a recent dig in to some of my older classes it was a little embarrassing to see lots of file types used like this:


  file { "/srv/whi/maps":
    ensure => present,
    source => "puppet://$servername/whi/maps.conf"
    owner  => whi,
    group  => whi,
    mode   => 644
  }
  file { "/srv/whi/elocs":
    ensure => present,
    source => "puppet://$servername/whi/eloc.conf"
    owner  => whi,
    group  => whi,
    mode   => 644
  }

Luckily as we get more experienced with a tool we can often go back and improve on the first steps. By using an explicit File { settings } inside a class you can assign a sensible set of defaults to all the instances of the same type that lack overriding settings. So we can shorten the previous example to -


File {
  owner => whi,
  group => whi,
  mode  => 644
}

file { "/srv/whi/maps":
  ensure => present,
  source => "puppet://$servername/whi/maps.conf"
}

file { "/srv/whi/elocs":
  ensure => present,
  source => "puppet://$servername/whi/eloc.conf"
}

While this isn't a huge win in raw characters typed (although in longer manifests they start to mount up) it does move all the common settings in to a single location (keeping us clear of DRY violations) and it leaves only the differences between file type definitions.

You can also apply those kind of settings (such as Exec { path => "path:list" } at the server level by including them in a top level file and then overriding them as needed in each module. If you do this then you need to be aware that any declared type that doesn't override it gets the global setting, which can lead to the odd action from afar head scratching.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Stand Alone Puppet

Thu, 17 Sep 2009 21:33:00 GMT

While Puppet can be used to manage large, complex environments it's also a useful tool at the lower end of the spectrum. Using just the puppet executable and a small inline class or two you can write very useful manifests in only a handful of lines.


class build-host {
  package { "build-essential": ensure => installed }
  package { "subversion":      ensure => installed }

  file { "/home/dwilson/repos/":
    ensure => directory,
    owner  => dwilson,
    group  => dwilson,
  }
}

node default {
  include build-host
}

To invoke the class you just run puppet -v build-host.pp. It's also worth pointing out the node name of default. This saves you manually changing the manifest whenever you move to another machine. While it wouldn't be hard to replace the above example with a shell script, by using puppet you can easily access the built-in abstractions (which package manager to use, how should you add users) and remove a lot of scaffolding code. And then when you're done you can promote the class to your managed infrastructure.

I've used this to bootstrap provisioning servers (why should the provisioning host be the only machine that wasn't provisioned?), test small but annoying new classes on scratch servers and I'm currently working on integrating it with a small subversion backup testing project in my spare time at work (so very slowly).

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!