Tue, 05 Jul 2011
Introduction To DSAC
A while ago @ripienaar and I had a chat
in a pub about monitoring, event systems and lots of related subjects. As
we all know he's way more productive than is fair and so while he's been
doing a BUNDLE of work with on subjects like monitoring frameworks and event correlation
I've been doing some thinking (and no actual coding) about event
auditing, continuous compliance and security event management.
Now I've finished the $TIMESINK_PROJECT I'm soon going to actually need some of this stuff so I've started putting together a prototype framework that I'm calling DSAC - Dump Send and Correlate. The code is in a very early stage at the moment but is dealing with a small number of agents on a test network of a couple of hundred nodes. I'm going to start documenting the sections as it becomes ready for more public consumption but I thought I'd show my architectural plans for version 0.1.
The architecture is quite simple at the moment. Every node runs the "consumer and dispatch" stack which generates events, currently all events are made from cron invoked agents. A separate process, also cron invoked (for now) then runs through the spool and invokes all the dispatchers that have registered an interest in the output of that agent. Simple dispatcher examples are an AMQ pusher or a MySQL loader.
At the other end of the process, and quite symmetrically, we have the consumer stack. This reads from the nice big fuzzy cloud of transient data loss and spools files for later processing. We then have another process pick the files up and run them through a number of processors.
I've got working prototypes of a simple bulk archiver and some debugging aids but I can also envision some more useful real time dashboards. The last stage at the moment are the simple reports. I'm currently focusing on the easier reports that will help me show changes to an auditor, package updates, service status changes and user logins but this step will hopefully expand to encompass a lot of our rote compliance needs.
Once I've tidied up the code (and picked up some more ruby!) I'll start putting the bits I work on in my spare time on github.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/07/05 17:45 | /tools | Permanent link to this entry | This entry and same date
Mon, 20 Jun 2011
Simple Puppet module grepper (prototype)
<tl;dr> Search for puppet resources values using puppet, not just
plain text</tl;dr>
One of the ideas that has been sitting on my todo list is having a command that lets me grep a puppet manifest for certain properties, values or even just resources in a smarter way than just running a raw grep over files. While a simple grep works in some cases it is annoyingly fragile when you're trying to ignore literal strings in resource types that you're not interested in or narrow your search down to resources that have a property that can also appear in other types.
# Show all file resources with a mode of 644
$ pm-grep -t file -p mode -v 644 files.pp
# Show all host resources with an alias of any value
$ pm-grep -t host -p host_aliases hosts.pp
# Check a number of pp files at once
$ find /etc/puppet/modules/ -name "*.pp" | xargs -n 1 pm-grep -t file -p mode
pm-grep (puppet manifest grep) isn't anywhere near finished but it does work on simple manifests. It yet doesn't handle corner cases, global parameter defaults and a number of other more advanced techniques but it does fulfil some of my needs and has given me some more to mull over for version 2.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/06/20 23:36 | /tools/puppet | Permanent link to this entry | This entry and same date
Thu, 16 Jun 2011
Smarter Service Status in Puppet
While most people know you can use puppet to
ensure a service is running
the mechanism it uses to determine if a service is actually running is often unexplored.
By default (at least up to Puppet 2.6) puppet assumes that a service doesn't supply a working status option and so will look up the services name in the process table to check if it's running. If your service does support the status argument you can set 'hasstatus => true' and the platforms service provider will be used to interrogate the services current status.
While most services only report a simple status of running or not running puppet, when you've specified 'hasstatus => true' puppet will consult a second property, if it's present, - status - which is where things get a little more interesting and extendable.
# puppet manifest
service { "httpd":
ensure => "running",
hasstatus => true,
status => "/usr/local/bin/puppet-status-http-check",
}
# puppet-status-http-check - example check
#!/usr/bin/perl
use strict;
use warnings;
my @checks = (
"/usr/lib/nagios/plugins/check_procs -C httpd",
"/usr/lib/nagios/plugins/check_http -I 127.0.0.1",
"/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u /about",
"/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -u / -s udlab",
);
for my $check ( @checks ) {
$check .= " 2>&1 > /dev/null"; # suppress output
system( $check ) == 0 or exit 1;
}
# when running under debug you'll see a line like:
debug: Service[httpd](provider=redhat): Executing '/usr/local/bin/puppet-status-http-check'
By specifying our own command in the status property we can do more complex, and domain specific, status checks. For example we don't so much care that apache is running as that it's serving our chosen vhosts correctly. You can use any command as the right hand side of status and puppet will treat a return code of 0 as confirmation that the service is running and anything else as a failure; which will trigger an attempt to restart the service in our example.
One possibility is to tie this in to nrpe-runner with a carefully chosen command name pattern to reap all the benefits of your already defined nagios checks.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/06/16 16:22 | /tools/puppet | Permanent link to this entry | This entry and same date
Sat, 14 May 2011
Wrapping MCollective with Nagios
I've been doing a little tinkering with pre/post release checklists and
compliance reporting using cucumber and some Nagios wrapping (among
other things) in my test lab and recently needed to do some higher level
entire environment checks before moving on to the next step. While it's
possible to wrap something like nmaps ping check and then Nagios each
target it does feel like stepping back a few years in the tool
chain.
Luckily I'm running MCollective, so all this synchronous discovery and polling is in my past. After a little bit of delving in to the existing package and service clients I've come up with a prototype environment wide MCollective backed service check and an MCollective backed package check.
I'm not sure if I'd be willing to replace existing low level checks (for things like cron and ssh processes) with this just yet but it does show how easy it is to wrap MCollective with third party code in order reap its benefits from further down the tool chain. With a little scaffolding hopefully it'll be useful in validating individual policies in security policies and guidelines. But more about that later.
Phase two is probably to pull the scripts together (and just use another parameter to select the resource to check) and to be green or red based on percentage. As an example, requiring 40% of the web servers to be returning 200 before starting the next batch of host upgrades.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/05/14 16:55 | /tools | Permanent link to this entry | This entry and same date
Tue, 22 Mar 2011
Listing Puppet Managed Files
Sometimes it's the little niggles that annoy people the most. As my team
progress in to puppet they have an annoying habit of asking very good
questions; which can sometimes be a struggle to answer. Todays best
question was - "How do I tell if this file is under puppets
control?"
While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. While this gets you most of the way the problem is a little harder than it looks because of an edge case. If puppet is managing an entire directory then the files in that directory are not explicitly listed in the catalog.
So we need to look in two places, the catalog and state.yaml. Remembering the greps (and the line transformations needed) requires more mental space than I'm willing to invest so I've written puppet-ls to do all the work for me.
$ puppet-ls /etc/mcollective
/etc/mcollective/facts.yaml
/etc/mcollective/server.cfg
Run the command, specify the directory to check and any shown files are puppet managed. It's not a ground breaking script but it can help people migrating to puppet as they bring more of their systems under its control.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/22 22:54 | /tools/puppet | Permanent link to this entry | This entry and same date
Mon, 21 Mar 2011
Nagios Wrapped Puppet Runs
<tl;dr>Log nrpe-runner state changes when puppet runs to see what
broke or was fixed.</tl;dr>
While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I've come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net.
One of the lesser known features introduced in Puppet 0.25.4 (and still valid in 2.6) were the prerun_command and postrun_command hooks. These two config settings allow you to specify a command to run at the beginning (which can stop the puppet run from happening) and at the end of a puppet run. While they were originally devised to make integration with etckepper simpler we can also use them to add some additional monitoring to our runs.
We've already covered my nrpe-runner, which lets you run Nagios checks locally for immediate feed back but now let's expand the idea a little for puppet integration. Our plan is simple, invoke nrpe-runner and gather the output, run puppet, re-run the nrpe-runner and see which checks puppet has fixed or broken.
First of all we deploy nrpe-runner, our nrperunner json differ and the (below) wrapper script we use for when puppet's finished running.
$ cat nrpe-wrapper
#!/bin/bash
/home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/post_puppetrun
logger -t "puppet-nrpe" `/home/deanw/puppet-wrapper/nrperunner-json-differ /tmp/pre_puppetrun /tmp/post_puppetrun`
We then add the config to puppet.confs main section. While it's possible to insert longer lines for each command and skip the wrapper script puppet is a little fiddly about these settings and a separate script is easier to use.
$ cat /etc/puppet/puppet.conf
[main]
... snip ...
prerun_command = /home/deanw/puppet-wrapper/nrpe-runner -j > /tmp/pre_puppetrun
postrun_command = /home/deanw/puppet-wrapper/nrpe-wrapper
Now we've done all the prep (and if needed restarted puppet) let's break something and see if we get both a fix and confirmation:
# stop something we know puppet will fix.
$ /etc/init.d/mcollective stop
$ puppetd -vt
info: Retrieving plugin
.. snip ...
notice: //mcollective::server/Service[mcollective]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 5.51 seconds
# see if we logged the fix... we did!
$ tail -n 1 /var/log/messages
Mar 21 22:07:21 lb03-dynm puppet-nrpe: mcollective_procs changed from 2 to 0
While our simple wrapper just sends the output directly to syslog hopefully you've got an idea how powerful this integrated immediate feedback can be. While it's always been possible for us to dig back through the logs and spot something breaking after a puppet run, by explicitly wrapping the run we can cut done the investigation time while also providing information for later review and discussion.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/21 22:56 | /tools/puppet | Permanent link to this entry | This entry and same date
Thu, 17 Mar 2011
Puppet Cucumber Providers
At work we try, and sometimes even succeed, in using Test Driven
Deployment so as one of my background projects I've been wrapping certain tools in to
cucumber friendly forms. Over the last couple of days I've been grabbing
ten minutes here and there to incorporate Puppet 2.6 in to the pile.
Feature: Puppetwrappers
Puppet Provider Examples
Scenario: Confirming package installation
When a machine has been puppeted
Then the bash package should be installed
Scenario: Confirm doodoodoo package is absent
When a machine has been puppeted
Then the doodoodoo package should not be installed
Scenario: Confirm cron service is running
When a machine has been puppeted
Then the cron service should be running
Scenario: Confirm tomcat6 service is not running
When a machine has been puppeted
Then the tomcat6 service should not be running
Scenario: Confirm dwilson is in libvirtd group
When a machine has been puppeted
Then dwilson should be a member of libvirtd
Scenario: Confirm dwilson has a uid of 1000
When a machine has been puppeted
Then dwilson should have a uid of 1000
Scenario: Confirm dwilson has a given shell
When a machine has been puppeted
Then dwilson should have the /bin/bash shell
I really like using the puppet providers for this because of the abstraction benefits they provide. I can write steps to test packages, services or aspects of a user and not have to worry if a developer runs it on Fedora or Debian.
This is only a first draft, and the cucumber wording needs changing, but I thought I'd put it online to show how expressive cucumber can be for system tasks and how easy, and concise, it is to reuse the puppet providers. You can grab the puppet step code and the Puppet providers features to drop in to your own test harnesses and have a play with. The implementation is pretty simple, for example the code below is everything you need for the service scenarios:
Then /^the (.+) service should be running$/ do | service |
service_status = Puppet::Type.type(:service).new(:name => service, :hasstatus => true).provider.status
service_status.should == :running
end
Then /^the (.+) service should not be running$/ do | service |
service_status = Puppet::Type.type(:service).new(:name =>service).provider.status
service_status.should == :stopped
end
It's worth mentioning that all the above will only work in 2.6 and above as the internal details returned by the providers are different to those in 2.5.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/17 19:16 | /tools/puppet | Permanent link to this entry | This entry and same date
Mon, 14 Mar 2011
Find Unpuppeted SSH Keys
It all started with one of those annoying little items on the todo list
- find all the unpuppeted ssh authorized_keys files on a machine and
alert on them. On first impressions it was going to be quite manual
(always a bad sign), involve digging in to legacy installs and would be
something we'd need to re-verify occasionally. It couldn't be that bad
though could it? After all how many places can an unmanaged-by-
puppet sshkey live?
Essentially the task can be broken in to three main parts. The first, quite easy part, is to grab a list of all the users (hello /etc/passwd) and look for known key file names in their home directories. The second part, which was a little harder, is to build a list of all the authorized_keys files that puppet knows it's managing for this host. Lastly once you have the two collections find the differences. Instead of doing static analysis on the puppetmasters classes and modules we're going to focus on how to do it using the compiled desired state of what the local machine should look like, according to the puppet catalog.
The catalog (which lives at /var/lib/puppet/client_yaml/catalog/$fqdn.yaml in modern puppet) is a yaml-based representation of what puppet knows about how the local system should be configured. It contains details of all the resources to be managed on the local machine and their desired end state; which makes it perfect for our needs. I'm not going to go into the catalog in depth in this post but hopefully this little example will whet your appetite and spark some ideas.
Our example, the audit-sshkey-files nagios check, was actually quite easy to write (after some digging in to puppet and borrowing some code from Puppet Catalog Diff by R.I.Pienaar) and should hopefully show how much you can gain from using the meta-data puppet provides.
While most of the audit-sshkey-files script is boilerplate the most important snippet is below:
if target.type == "File" and target.title.include? "/authorized_keys"
@puppet_keys.push target.title
return target.title
end
All we're doing is building a list of any resources that are of type file and include the string "/authorized_keys" in their name (resource title in puppet terms). While this may not seem like much it's potentially game changing, any resources or relationships that you've modelled in puppet can be later mined to add context to your other tools. You can (as we have here) audit security related files or find user ids puppet doesn't know about and so might be inconsistent over systems. By using the catalog and the relationships and meta-data it provides you can make much more of your investment in deploying systems with puppet, and hopefully this little example presents an easy way to get started.
Now I've gushed about what the puppet catalog can do for you there are two caveats, firstly about my example. It isn't a complete solution, for example it doesn't look for other allowed "authorized_keys" filenames that are defined in the sshd_config file. But it does the 80% of what I needed in our environment and by managing the sshd_config file in puppet (as you should be) it's easy for me to double check I'm looking for the correct files. Secondly about the Puppet catalog itself. Harnessing its contents doesn't exactly have a shallow learning curve and documentation is a little thin on the ground. The original author of puppet Luke Kanies is working on some alternative ways of accessing this kind of information (such as via his Puppet Interfaces project) and as more people build their puppet deployments you can expect so see more and more harnessing of this additional structure.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/14 23:30 | /tools/puppet | Permanent link to this entry | This entry and same date
Reusing Puppets Package providers
One of puppets more under-appreciated features is its ability to abstract
and smooth the edges of certain operating system tasks and behaviours.
Even something as trivial as installing a package can actually become a
portability nightmare once you consider the number of different systems
in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands
needed to use them. You end up either hard coding commands, and sacrificing
portability, or writing your own detection, lookup and invocation
logic.
That sounds like, dull, scut work so how does puppet deal with this? And how can we reuse this work to simplify our own code? In slightly simplified terms, Puppet has a package type, which is backed by a number of providers. Each of these providers actually implement the required functionality for a given package manager and contains all the code we need. So how do we harness this existing work? Quite easily. Luckily for us, puppets providers are written in ruby code and are simple to call in our own scripts:
# show package version
$ irb
irb(main):001:0> require 'puppet'
=> true
irb(main):002:0> Puppet::Type.type(:package).new(:name => "bash").provider.properties
=> { :provider=>:yum, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
=> :arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }
# do the same thing with an explicitly specified provider.
irb(main):003:0> Puppet::Type.type(:package).new(:name => "bash", :provider => "rpm").provider.properties
=> { :provider=>:rpm, :ensure=>"4.1.7-3.fc14", :release=>"3.fc14",
:arch=>"i686", :epoch=>"0", :name=>"bash", :version=>"4.1.7" }
While that snippet will hopefully whet your appetite if you need a more worked example I've put a small Puppet Package Provider wrapper up on github. The script will enable you to do the basic install, update and delete without knowing or caring what the underlying package manager is. Hopefully these little code snippets will help you stop thinking of puppet as "just" a tool and show how parts of its code base can be used as a framework to improve other parts of your tool chain.
As an aside it's also worth mentioning that you can globally Change the Package provider in puppet if you're not happy with its auto-detection.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/14 00:18 | /tools/puppet | Permanent link to this entry | This entry and same date
Wed, 09 Mar 2011
Introducing NRPE Runner
It might be a sign that I spend too much time online but the quicker a
system gives me feedback the more useful I find it. While I love knowing
my Nagios safety net has me covered when making changes sometimes waiting
for that cgi to refresh can take too long, especially if I'm taking a
iterative / test driven approach to the changes I'm making. For those
use cases I wrote nrpe-runner.
The way I typically use Nagios is to have the Nagios server run the
checks on the remote host via the NRPE plugin. The checks to be run on the host are
normally stored in a config file with each entry looking like this:
command[local_mail]=/usr/local/libexec/nagios/check_local_mail
While this allows you to run each check to confirm that it's still OK I
wanted the ability to run all the commands in the file at once, which I can
now do with nrpe-runner. If every thing's fine then
it exits silently, to confirm that it's actually run I can summarise and
even filter the checks to run:
# show everything as it's run whatever the return status
/usr/local/sbin/nrpe-runner -a
check_swap => SWAP OK - 100% free (16041 MB out of 16041 MB) |swap=16041MB;12031;9624;0;16041
... snipped ...
freemem => OK: 12% (1732M) free memory.
# show a summary
$ /usr/local/sbin/nrpe-runner -s
Ran 39 checks - OK 39. WARN 0, CRIT 0, UNKNOWN 0
# run any checks with ntp in the name (the part between [])
$ /usr/local/sbin/nrpe-runner -s -n ntp
Ran 3 checks - OK 3. WARN 0, CRIT 0, UNKNOWN 0
# run all process checks (checks the command after the '=')
$ /usr/local/sbin/nrpe-runner -s -c proc
Ran 17 checks - OK 17. WARN 0, CRIT 0, UNKNOWN 0
# show all checks named ntp
$ /usr/local/sbin/nrpe-runner -a -n ntp
ntp_skew_primary => NTP OK: Offset -0.003149271011 secs|offset=-0.003149s;5.000000;9.000000;
ntp_process => PROCS OK: 1 process with command name 'ntpd', args '-u ntp:ntp'
ntp_skew_secondary => NTP OK: Offset -0.002887368202 secs|offset=-0.002887s;5.000000;9.000000;
nrpe-runner also has the option to dump the results as json, which I'll be exploring a little further in my next couple of blog posts. While it's not exactly the same as having the checks run by nagios (the user and environment are often different) I've found that shortening the interval between running puppet or yum and seeing the nagios feedback has helped my work-flow quite a lot when making exploratory system changes - and even more when nothing should have changed but does...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/03/09 19:05 | /tools/commandline | Permanent link to this entry | This entry and same date
Tue, 04 Jan 2011
Puppet CookBook is live
Between Xmas and New Year I had some spare time to invest on a side
project I've been looking forward to working on for quite a while. I'm
pleased to announce the opening of the Puppet
CookBook.
I've introduced Puppet to quite a few companies, sysadmins and development teams over the years and a lot of the same issues, concepts and needs repeatedly crop up. By explaining how puppet works in terms of tasks and desired outcomes rather than in raw feature descriptions I hope to show some of its power and flexibility in easy to use examples in a different way to most of the existing documentation.
The site isn't exactly brimming over with content yet (and it's pretty ugly) but I'm adding a handful of posts each week and hope to cover some more advanced topics over the next couple of months. You can follow the Puppet CookBook Twitter account for update announcements or to send feedback or suggestions for future topics.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2011/01/04 22:59 | /tools/puppet | Permanent link to this entry | This entry and same date
Sat, 11 Dec 2010
Clarifying With Facter
While adopting a configuration management tool like Chef and Puppet will
have a large, nearly immediate effect on your work flow even after using the
tools for a while you'll still get a little smile at all the little niceties
you continuously discover.
One recent small win we had recently was bringing some apache configs files under Puppet command. When we started we had the following block of config:
RewriteCond %{REMOTE_ADDR} !10.23.143.33
RewriteCond %{REMOTE_ADDR} !10.23.143.2
RewriteCond %{REMOTE_ADDR} !10.23.143.3
It's not hard to read and roughly understand what it does, but you have no real context; magic numbers keep things terse but are rarely the most helpful when in the land of a strange system. After putting the configs in to a module and abstracting them a little into a template we have the much nicer:
RewriteCond %{REMOTE_ADDR} !<%= primary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= secondary_loadbalancer %>
RewriteCond %{REMOTE_ADDR} !<%= ipaddress_eth0_mgmt %>
As part of the tidy up we also renamed some of the (remarkably large amount of) Ethernet interfaces to describe what they were for, rather than leaving them as eth12:34
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/12/11 21:35 | /tools/puppet | Permanent link to this entry | This entry and same date
Mon, 15 Nov 2010
MCollective Plugin - FileMD5er
I've been watching the Marionette Collective for a
while, and even gave it a small trial in a couple of testing
environments, but this weekend was the first time I've experimented
with it at a slightly larger scale (just over a hundred small VM nodes -
you have to love EC2) and I'm still impressed.
I can see how it's going to make parts of my work flow easier, and in an
attempt to learn a little more about how the plugin system works under the
hood I decided to write a small agent, FileMD5er.
The agent itself is very simple and addresses a small annoyance I've
scripted around for a while. When you're bringing files under Puppet (or
Chef) management you need to dig through the hosts and locate any files
with differences compared to the most common adhoc file. With a quick
mc-filemd5er /path/to/file I can easily spot any machines
that have a slightly different version of the file, and then fold them
in to centralised management.
Writing the plugin itself was quite easy. The two problems I encountered were finding the right generation of existing plugin to crib from (some of the official MCollective Plugins are of a newer format than others) and not naming the class and the .rb file the same name. Which caused it to half work.
I'll be putting more of my MCollective Plugins on Github as the become a little more generic and hopefully useful to someone else.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/11/15 23:26 | /tools/puppet | Permanent link to this entry | This entry and same date
Mon, 23 Aug 2010
Adventures in Cronologger
Cronjobs are one of those necessary evils of any decent sized Unix setup,
they provide often essential pieces of a sites data flows but are often
treated as second class citizens. While I've already mentioned my Cron
commandments I'm always looking for improvements in the
rest of my cron tool set and, with Vladimir Vuksan's cronologger, I may have
found another piece of the puzzle.
The concept is simple, you add a command to the front of your crontabs and it invokes your actual cron command. This wrapper script collects the stdout, stderr and some other details such as exit code and run time. The backend is a couchdb data store and the simple reporting pages are written in PHP, and are easy to work through, crib and base your own reports from. Having all this cron information also helps provide a talking point with development, it's easy to show progress and imbue a sense of actually getting somewhere when the number of cronjobs with errors drops each day, rather than the systems team mentioning that their email boxes are a little emptier since the last release.
While our initial tests seem positive there are a couple of reports and tweaks to the command line data injector that we want for our local usage. The biggest problem with the project may well be that the idea is so obviously correct that we end up re-implementing it in something a little more suitable for our environment. Maybe a Python command line client and Perl Template Toolkit driven reports to replace the PHP. But that's a possibility for later - for now cronologger is a great 80% solver.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/08/23 21:49 | /tools | Permanent link to this entry | This entry and same date
Sun, 06 Jun 2010
Netbeans vs Commandline
The last time we interviewed for Java developers (a couple of jobs
ago) it came as quite a surprise at how few of them could function
without their IDE of choice. A high percentage of the candidates
struggled to compile using javac, had problems navigating the docs and
made a large number of very simple syntax errors that they were obviously
used to their editor dealing with.
At the time the more unix focused team, most of who were very long term vim and emacs users, had a number of discussions about how this should impact our rating of the candidates. One school of thought was that people should use the tools that make them most productive. The other was that people should understand their tool chain. How can you diagnose issues on a production server if you can't even compile a class on the command line? You can tell which side I was on.
I've recently joined a small Java project and after some awkward fiddling around with ant, junit and half a dozen other jars decided to give Netbeans a chance. I was pleasantly surprised at how quickly and easily I got the same project up and running in the IDE. I don't yet have a clue how it's storing the files on disk, constructs the build or test targets and a dozen other little details but at this stage in my basic use of Java it doesn't seem to matter.
It's strange how quickly seductive all the optional extras can be and how easy it is to lose track of what you don't know while adapting to the features they offer. I'm not sure how much of it is better tooling, benefits of a strongly typed static language or just having a dedicated team behind producing a consistent development environment but it felt very easy to take baby steps with. And I'm hoping the tool continues to show me more power as my needs when using it grow.
While I'm at no risk of giving up vim for my day to day work I think I'll be investing some time in to learning one of the big three Java editors (Eclipse, Netbeans or IntelliJ) for while I'm away in the strange world.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/06/06 12:11 | /tools | Permanent link to this entry | This entry and same date
Wed, 07 Apr 2010
Pigz - Shortening backup times with parallel gzip
While searching for a completely different piece of software I stumbled
on to the pigz application, a
parallel implementation of gzip for modern multi-processor, multi-core
machines. As some of our backups have a gzip step to conserve
some space I decided to see if pigz could be useful in speeding them up.
Using remarkably unscientific means (I just wanted to know if it's worth further investigation) I ran a couple of sample compression runs. The machine is a quad core Dell server, the files are three copies of the same 899M SQL dump and the machine is lightly loaded (and mostly in disk IO).
####################################### # Timings for two normal gzip runs dwilson@pigztester:~/pgzip/pigz-2.1.6$ time gzip 1 2 3 real 2m43.429s user 2m39.446s sys 0m3.988s real 2m43.403s user 2m39.582s sys 0m3.808s ####################################### # Timings for three pigz runs dwilson@pigztester:~/pgzip/pigz-2.1.6$ time ./pigz 1 2 3 real 0m46.504s user 2m56.015s sys 0m4.116s real 0m46.976s user 2m55.983s sys 0m4.292s real 0m47.402s user 2m55.695s sys 0m4.256s
Quite an impressive speed up considering all I did was run a slightly different command. The post compression sizes are pretty much the same (258M when compressed by gzip and 257M with pigz) and you can gunzip a pigz'd file, and get back a file with the same md5sum.
# before compression -rw-r--r-- 1 dwilson dwilson 899M 2010-04-06 22:12 1 # post gzip compress -rw-r--r-- 1 dwilson dwilson 258M 2010-04-06 22:12 1.gz # post pigz compress -rw-r--r-- 1 dwilson dwilson 257M 2010-04-06 22:12 1.gzs
I'll need to do some more testing, and compare the systems performance to a normal run while the compression is happening, before I trust it in production but the speed ups look appealing and, as it's Mark Adler code, it looks like it might be an easy win in some of our scripts.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/04/07 08:00 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 30 Sep 2009
Rake - surprisingly enjoyable
I've never really liked make files, I don't think I've ever had to write
enough C to really appreciate (or just tolerate) them, so I was a little
dismissive of Rake - and I was
mostly wrong.
Now we're adding a new member to the systems team I've been doing a lot
of thinking about our tool chain - what knowledge assumptions it makes,
which parts are still more manual than I'd like and where the tool chain has
gaps (this is the most annoying one for me) and rake seemed like a
potential addition to encode some of that process knowledge in to a tool.
I've only added little rakefiles here and there but they do make certain
tasks nicer (plus I like the inline descs).
I've not yet worked out any general rules for when to use a shell script and when to use rake but if nothing else it's helping me spend some time on my ruby skills. The best rake starting points I found were Martin Fowlers rake article and the rake release notes.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/30 21:48 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 23 Sep 2009
Simplifying File Permissions in Puppet Manifests
I've been a user of Puppet for about three years now and
while on a recent dig in to some of my older classes it was a little
embarrassing to see lots of file types used like this:
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
owner => whi,
group => whi,
mode => 644
}
Luckily as we get more experienced with a tool we can often go back and
improve on the first steps. By using an explicit File { settings
} inside a class you can assign a sensible set of defaults to all
the instances of the same type that lack overriding settings. So we can
shorten the previous example to -
File {
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
}
While this isn't a huge win in raw characters typed (although in longer manifests they start to mount up) it does move all the common settings in to a single location (keeping us clear of DRY violations) and it leaves only the differences between file type definitions.
You can also apply those kind of settings (such as Exec { path =>
"path:list" } at the server level by including them in a top level
file and then overriding them as needed in each module. If you do this then
you need to be aware that any declared type that doesn't override it gets
the global setting, which can lead to the odd action from afar head
scratching.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/23 22:16 | /tools/puppet | Permanent link to this entry | This entry and same date
Thu, 17 Sep 2009
Stand Alone Puppet
While Puppet can be used to manage large, complex
environments it's also a useful tool at the lower end of the spectrum.
Using just the puppet executable and a small inline class or
two you can write very useful manifests in only a handful of lines.
class build-host {
package { "build-essential": ensure => installed }
package { "subversion": ensure => installed }
file { "/home/dwilson/repos/":
ensure => directory,
owner => dwilson,
group => dwilson,
}
}
node default {
include build-host
}
To invoke the class you just run puppet -v build-host.pp.
It's also worth pointing out the node name of default. This saves you
manually changing the manifest whenever you move to another machine.
While it wouldn't be hard to replace the above example with a shell
script, by using puppet you can easily access the built-in abstractions
(which package manager to use, how should you add users) and remove a lot
of scaffolding code. And then when you're done you can promote the class to
your managed infrastructure.
I've used this to bootstrap provisioning servers (why should the provisioning host be the only machine that wasn't provisioned?), test small but annoying new classes on scratch servers and I'm currently working on integrating it with a small subversion backup testing project in my spare time at work (so very slowly).
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/17 21:33 | /tools/puppet | Permanent link to this entry | This entry and same date
Wed, 01 Jul 2009
dstat - a window to your system
When it comes to Unix diagnostics I was raised the old fashion way, with
iostat, vmstat and similar tools. However times change and tools evolve.
dstat, while not as
comprehensive as using all the tools one by one, provides a wide range
of system performance details in an easy to use package.
While it's useful enough in its default state there is even more
functionality lurking just below the surface. To see which other modules
are available (but are not enabled by default) run dstat -M
list. To add an extra module to the output use a command like
this one: dstat -a -M topmem -M topcpu
As part of my growing use of the tool I've started to write my own little dstat plugins. I was pleasantly surprised at how easy they were to write and deploy even with my basic python skills. While the memcached plugin was a proof of concept I've not needed much I've found the process count plugin to be very handy.
dstat is becoming one of the overview tools I use when investigating performance issues and it's worthy of a place in your toolbox too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/01 21:32 | /tools/commandline | Permanent link to this entry | This entry and same date
Copyright © 2000-2010 Dean Wilson
