Tue, 22 Mar 2011
Listing Puppet Managed Files
Sometimes it's the little niggles that annoy people the most. As my team
progress in to puppet they have an annoying habit of asking very good
questions; which can sometimes be a struggle to answer. Todays best
question was - "How do I tell if this file is under puppets
control?"
While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. While this gets you most of the way the problem is a little harder than it looks because of an edge case. If puppet is managing an entire directory then the files in that directory are not explicitly listed in the catalog.
So we need to look in two places, the catalog and state.yaml. Remembering the greps (and the line transformations needed) requires more mental space than I'm willing to invest so I've written puppet-ls to do all the work for me.
$ puppet-ls /etc/mcollective
/etc/mcollective/facts.yaml
/etc/mcollective/server.cfg
Run the command, specify the directory to check and any shown files are puppet managed. It's not a ground breaking script but it can help people migrating to puppet as they bring more of their systems under its control.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/22 22:54 | /tools/puppet | Permanent link to this entry | This entry and same date
Mon, 21 Mar 2011
Nagios Wrapped Puppet Runs
<tl;dr>Log nrpe-runner state changes when puppet runs to see what
broke or was fixed.</tl;dr>
While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I've come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net.
One of the lesser known features introduced in Puppet 0.25.4 (and still valid in 2.6) were the prerun_command and postrun_command hooks. These two config settings allow you to specify a command to run at the beginning (which can stop the puppet run from happening) and at the end of a puppet run. While they were originally devised to make integration with etckepper simpler we can also use them to add some additional monitoring to our runs.
We've already covered my nrpe-runner, which lets you run Nagios checks locally for immediate feed back but now let's expand the idea a little for puppet integration. Our plan is simple, invoke nrpe-runner and gather the output, run puppet, re-run the nrpe-runner and see which checks puppet has fixed or broken.
First of all we deploy nrpe-runner, our nrperunner json differ and the (below) wrapper script we use for when puppet's finished running.
$ cat nrpe-wrapper
#!/bin/bash
/home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/post_puppetrun
logger -t "puppet-nrpe" `/home/deanw/puppet-wrapper/nrperunner-json-differ /tmp/pre_puppetrun /tmp/post_puppetrun`
We then add the config to puppet.confs main section. While it's possible to insert longer lines for each command and skip the wrapper script puppet is a little fiddly about these settings and a separate script is easier to use.
$ cat /etc/puppet/puppet.conf
[main]
... snip ...
prerun_command = /home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/pre_puppetrun
postrun_command = /home/deanw/puppet-wrapper/nrpe-wrapper
Now we've done all the prep (and if needed restarted puppet) let's break something and see if we get both a fix and confirmation:
# stop something we know puppet will fix.
$ /etc/init.d/mcollective stop
$ puppetd -vt
info: Retrieving plugin
.. snip ...
notice: //mcollective::server/Service[mcollective]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 5.51 seconds
# see if we logged the fix... we did!
$ tail -n 1 /var/log/messages
Mar 21 22:07:21 lb03-dynm puppet-nrpe: mcollective_procs changed from 2 to 0
While our simple wrapper just sends the output directly to syslog hopefully you've got an idea how powerful this integrated immediate feedback can be. While it's always been possible for us to dig back through the logs and spot something breaking after a puppet run, by explicitly wrapping the run we can cut done the investigation time while also providing information for later review and discussion.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/21 22:56 | /tools/puppet | Permanent link to this entry | This entry and same date
Thu, 17 Mar 2011
Puppet Cucumber Providers
At work we try, and sometimes even succeed, in using Test Driven
Deployment so as one of my background projects I've been wrapping certain tools in to
cucumber friendly forms. Over the last couple of days I've been grabbing
ten minutes here and there to incorporate Puppet 2.6 in to the pile.
Feature: Puppetwrappers
Puppet Provider Examples
Scenario: Confirming package installation
When a machine has been puppeted
Then the bash package should be installed
Scenario: Confirm doodoodoo package is absent
When a machine has been puppeted
Then the doodoodoo package should not be installed
Scenario: Confirm cron service is running
When a machine has been puppeted
Then the cron service should be running
Scenario: Confirm tomcat6 service is not running
When a machine has been puppeted
Then the tomcat6 service should not be running
Scenario: Confirm dwilson is in libvirtd group
When a machine has been puppeted
Then dwilson should be a member of libvirtd
Scenario: Confirm dwilson has a uid of 1000
When a machine has been puppeted
Then dwilson should have a uid of 1000
Scenario: Confirm dwilson has a given shell
When a machine has been puppeted
Then dwilson should have the /bin/bash shell
I really like using the puppet providers for this because of the abstraction benefits they provide. I can write steps to test packages, services or aspects of a user and not have to worry if a developer runs it on Fedora or Debian.
This is only a first draft, and the cucumber wording needs changing, but I thought I'd put it online to show how expressive cucumber can be for system tasks and how easy, and concise, it is to reuse the puppet providers. You can grab the puppet step code and the Puppet providers features to drop in to your own test harnesses and have a play with. The implementation is pretty simple, for example the code below is everything you need for the service scenarios:
Then /^the (.+) service should be running$/ do | service |
service_status = Puppet::Type.type(:service).new(:name => service, :hasstatus => true).provider.status
service_status.should == :running
end
Then /^the (.+) service should not be running$/ do | service |
service_status = Puppet::Type.type(:service).new(:name =>service).provider.status
service_status.should == :stopped
end
It's worth mentioning that all the above will only work in 2.6 and above as the internal details returned by the providers are different to those in 2.5.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/17 19:16 | /tools/puppet | Permanent link to this entry | This entry and same date
OpenIndiana - LOSUG March 2011
Tonights (the March 2011)
London OpenSolaris User Group (LOSUG) was a little different to
usual and while the topics have always been quite diverse we've never
had as seditious a talk as one covering the Solaris fork, OpenIndiana,
Illumos and the OpenSolaris community.
Alasdair Lumsden did an excellent job of explaining the new projects, why they exist and what they're aiming for. As someone who took a few steps back when Oracle purchased Solaris it was an interesting catch up. The short version seems to be that "Illumos is a derivative of OS/Net (aka ON), which basically is a Solaris/OpenSolaris kernel with the bulk of the drivers, core libraries, and basic utilities." (from Wikipedia) and is being quite heavily invested in by companies (such as Joynet and Nexenta) and by individuals that were previous employed by Sun to work on Solaris. OpenIndiana is to become an OpenSolaris distribution and packaged software ecosystem.
To me the project has a similar feel to the early days of CentOS and Scientific Linux and I think my biggest take home is that Illumos and OpenIndiana, when taken together, want to be to Solaris what CentOS and Scientific Linux are to Red Hat Enterprise Server. Unfortunately they have a massive disadvantage as Solaris, unlike upstream Red Hat, isn't entirely open. One of the most immediately visible casualties is the excellent ZFS, which is closed source upstream and will both lag behind and diverge from the official Solaris version. Which I consider to be a great loss.
On a more cheerful note the OpenIndiana project is looking for people with an interest in taking free Solaris forward and is still young enough that there are plenty of interesting aspects to get involved with. Websites, CI environments (I'm guessing they won't use Hudson. Heh) and all the other usual roles a large opensource project needs filling are up for grabs.
The talk itself was quite well attended, with what looked to be 35-40 people in the audience, and well presented. It's also the first time I've been in to Oracles Moorgate offices and they're actually quite nice and modern. The open sided lift and the suspended spiral staircase that only serves three floors were personal highlights.
I wish the project well and hope it enjoys success while being able to retain some of what made Solaris great. I may even take the DVD for a spin...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/17 00:03 | /events | Permanent link to this entry | This entry and same date
Mon, 14 Mar 2011
Find Unpuppeted SSH Keys
It all started with one of those annoying little items on the todo list
- find all the unpuppeted ssh authorized_keys files on a machine and
alert on them. On first impressions it was going to be quite manual
(always a bad sign), involve digging in to legacy installs and would be
something we'd need to re-verify occasionally. It couldn't be that bad
though could it? After all how many places can an unmanaged-by-
puppet sshkey live?
Essentially the task can be broken in to three main parts. The first, quite easy part, is to grab a list of all the users (hello /etc/passwd) and look for known key file names in their home directories. The second part, which was a little harder, is to build a list of all the authorized_keys files that puppet knows it's managing for this host. Lastly once you have the two collections find the differences. Instead of doing static analysis on the puppetmasters classes and modules we're going to focus on how to do it using the compiled desired state of what the local machine should look like, according to the puppet catalog.
The catalog (which lives at /var/lib/puppet/client_yaml/catalog/$fqdn.yaml in modern puppet) is a yaml-based representation of what puppet knows about how the local system should be configured. It contains details of all the resources to be managed on the local machine and their desired end state; which makes it perfect for our needs. I'm not going to go into the catalog in depth in this post but hopefully this little example will whet your appetite and spark some ideas.
Our example, the audit-sshkey-files nagios check, was actually quite easy to write (after some digging in to puppet and borrowing some code from Puppet Catalog Diff by R.I.Pienaar) and should hopefully show how much you can gain from using the meta-data puppet provides.
While most of the audit-sshkey-files script is boilerplate the most important snippet is below:
if target.type == "File" and target.title.include? "/authorized_keys"
@puppet_keys.push target.title
return target.title
end
All we're doing is building a list of any resources that are of type file and include the string "/authorized_keys" in their name (resource title in puppet terms). While this may not seem like much it's potentially game changing, any resources or relationships that you've modelled in puppet can be later mined to add context to your other tools. You can (as we have here) audit security related files or find user ids puppet doesn't know about and so might be inconsistent over systems. By using the catalog and the relationships and meta-data it provides you can make much more of your investment in deploying systems with puppet, and hopefully this little example presents an easy way to get started.
Now I've gushed about what the puppet catalog can do for you there are two caveats, firstly about my example. It isn't a complete solution, for example it doesn't look for other allowed "authorized_keys" filenames that are defined in the sshd_config file. But it does the 80% of what I needed in our environment and by managing the sshd_config file in puppet (as you should be) it's easy for me to double check I'm looking for the correct files. Secondly about the Puppet catalog itself. Harnessing its contents doesn't exactly have a shallow learning curve and documentation is a little thin on the ground. The original author of puppet Luke Kanies is working on some alternative ways of accessing this kind of information (such as via his Puppet Interfaces project) and as more people build their puppet deployments you can expect so see more and more harnessing of this additional structure.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/14 23:30 | /tools/puppet | Permanent link to this entry | This entry and same date
Reusing Puppets Package providers
One of puppets more under-appreciated features is its ability to abstract
and smooth the edges of certain operating system tasks and behaviours.
Even something as trivial as installing a package can actually become a
portability nightmare once you consider the number of different systems
in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands
needed to use them. You end up either hard coding commands, and sacrificing
portability, or writing your own detection, lookup and invocation
logic.
That sounds like, dull, scut work so how does puppet deal with this? And how can we reuse this work to simplify our own code? In slightly simplified terms, Puppet has a package type, which is backed by a number of providers. Each of these providers actually implement the required functionality for a given package manager and contains all the code we need. So how do we harness this existing work? Quite easily. Luckily for us, puppets providers are written in ruby code and are simple to call in our own scripts:
# show package version
$ irb
irb(main):001:0> require 'puppet'
=> true
irb(main):002:0> Puppet::Type.type(:package).new(:name => "bash").provider.properties
=> { :provider=>:yum, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
=> :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }
# do the same thing with an explicitly specified provider.
irb(main):003:0> Puppet::Type.type(:package).new(:name => "bash", :provider => "rpm").provider.properties
=> { :provider=>:rpm, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
:arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }
While that snippet will hopefully whet your appetite if you need a more worked example I've put a small Puppet Package Provider wrapper up on github. The script will enable you to do the basic install, update and delete without knowing or caring what the underlying package manager is. Hopefully these little code snippets will help you stop thinking of puppet as "just" a tool and show how parts of its code base can be used as a framework to improve other parts of your tool chain.
As an aside it's also worth mentioning that you can globally Change the Package provider in puppet if you're not happy with its auto-detection.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/14 00:18 | /tools/puppet | Permanent link to this entry | This entry and same date
Wed, 09 Mar 2011
Introducing NRPE Runner
It might be a sign that I spend too much time online but the quicker a
system gives me feedback the more useful I find it. While I love knowing
my Nagios safety net has me covered when making changes sometimes waiting
for that cgi to refresh can take too long, especially if I'm taking a
iterative / test driven approach to the changes I'm making. For those
use cases I wrote nrpe-runner.
The way I typically use Nagios is to have the Nagios server run the
checks on the remote host via the NRPE plugin. The checks to be run on the host are
normally stored in a config file with each entry looking like this:
command[local_mail]=/usr/local/libexec/nagios/check_local_mail
While this allows you to run each check to confirm that it's still OK I
wanted the ability to run all the commands in the file at once, which I can
now do with nrpe-runner. If every thing's fine then
it exits silently, to confirm that it's actually run I can summarise and
even filter the checks to run:
# show everything as it's run whatever the return status
/usr/local/sbin/nrpe-runner -a
check_swap => SWAP OK - 100% free (16041 MB out of 16041 MB) |swap=16041MB;12031;9624;0;16041
... snipped ...
freemem => OK: 12% (1732M) free memory.
# show a summary
$ /usr/local/sbin/nrpe-runner -s
Ran 39 checks - OK 39. WARN 0, CRIT 0, UNKNOWN 0
# run any checks with ntp in the name (the part between [])
$ /usr/local/sbin/nrpe-runner -s -n ntp
Ran 3 checks - OK 3. WARN 0, CRIT 0, UNKNOWN 0
# run all process checks (checks the command after the '=')
$ /usr/local/sbin/nrpe-runner -s -c proc
Ran 17 checks - OK 17. WARN 0, CRIT 0, UNKNOWN 0
# show all checks named ntp
$ /usr/local/sbin/nrpe-runner -a -n ntp
ntp_skew_primary => NTP OK: Offset -0.003149271011 secs|offset=-0.003149s;5.000000;9.000000;
ntp_process => PROCS OK: 1 process with command name 'ntpd', args '-u ntp:ntp'
ntp_skew_secondary => NTP OK: Offset -0.002887368202 secs|offset=-0.002887s;5.000000;9.000000;
nrpe-runner also has the option to dump the results as json, which I'll be exploring a little further in my next couple of blog posts. While it's not exactly the same as having the checks run by nagios (the user and environment are often different) I've found that shortening the interval between running puppet or yum and seeing the nagios feedback has helped my work-flow quite a lot when making exploratory system changes - and even more when nothing should have changed but does...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/09 19:05 | /tools/commandline | Permanent link to this entry | This entry and same date
Copyright © 2000-2010 Dean Wilson
