Dean Wilson@UnixDaemon: In search of (a) life --

Adventures in Cronologger

Mon, 23 Aug 2010 21:49:00 GMT

Cronjobs are one of those necessary evils of any decent sized Unix setup, they provide often essential pieces of a sites data flows but are often treated as second class citizens. While I've already mentioned my Cron commandments I'm always looking for improvements in the rest of my cron tool set and, with Vladimir Vuksan's cronologger, I may have found another piece of the puzzle.

The concept is simple, you add a command to the front of your crontabs and it invokes your actual cron command. This wrapper script collects the stdout, stderr and some other details such as exit code and run time. The backend is a couchdb data store and the simple reporting pages are written in PHP, and are easy to work through, crib and base your own reports from. Having all this cron information also helps provide a talking point with development, it's easy to show progress and imbue a sense of actually getting somewhere when the number of cronjobs with errors drops each day, rather than the systems team mentioning that their email boxes are a little emptier since the last release.

While our initial tests seem positive there are a couple of reports and tweaks to the command line data injector that we want for our local usage. The biggest problem with the project may well be that the idea is so obviously correct that we end up re-implementing it in something a little more suitable for our environment. Maybe a Python command line client and Perl Template Toolkit driven reports to replace the PHP. But that's a possibility for later - for now cronologger is a great 80% solver.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

The ThoughtWorks Anthology - Short Review

Sun, 27 Jun 2010 08:37:00 GMT

The ThoughtWorks Anthology is a collection of short articles and essays written by a number of their employees (some of who are now ex-employees) about software development with a heavily agile slant. The topics range from the very high level "Lush Landscape of Languages" and "What is an Iteration manager anyway" to the more technical and technique focused "Refactoring Ant Build Files" and "Object Calisthenics".

While the general quality of the writing is very good, especially my favourite - 'Object Calisthenics', the biggest problem with a book like this is that a lot of the essays authors, and some of their also knowledgeable co-workers, have personal blogs where this quality of information is available on a (near) daily basis, in both greater depth and more a conversational nature.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Netbeans vs Commandline

Sun, 06 Jun 2010 12:11:00 GMT

The last time we interviewed for Java developers (a couple of jobs ago) it came as quite a surprise at how few of them could function without their IDE of choice. A high percentage of the candidates struggled to compile using javac, had problems navigating the docs and made a large number of very simple syntax errors that they were obviously used to their editor dealing with.

At the time the more unix focused team, most of who were very long term vim and emacs users, had a number of discussions about how this should impact our rating of the candidates. One school of thought was that people should use the tools that make them most productive. The other was that people should understand their tool chain. How can you diagnose issues on a production server if you can't even compile a class on the command line? You can tell which side I was on.

I've recently joined a small Java project and after some awkward fiddling around with ant, junit and half a dozen other jars decided to give Netbeans a chance. I was pleasantly surprised at how quickly and easily I got the same project up and running in the IDE. I don't yet have a clue how it's storing the files on disk, constructs the build or test targets and a dozen other little details but at this stage in my basic use of Java it doesn't seem to matter.

It's strange how quickly seductive all the optional extras can be and how easy it is to lose track of what you don't know while adapting to the features they offer. I'm not sure how much of it is better tooling, benefits of a strongly typed static language or just having a dedicated team behind producing a consistent development environment but it felt very easy to take baby steps with. And I'm hoping the tool continues to show me more power as my needs when using it grow.

While I'm at no risk of giving up vim for my day to day work I think I'll be investing some time in to learning one of the big three Java editors (Eclipse, Netbeans or IntelliJ) for while I'm away in the strange world.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Obese Provisioning - Antipattern

Thu, 03 Jun 2010 21:37:00 GMT

One antipattern I'm seeing with increasing frequency is that of obese (or fat, or bloated) system provisioning. It seems as common in people that are just getting used to having an automated provisioning system and are enthusiastic about its power as it is in longer term users who have added layer on layer of cruft to their host builder.

The basic problem is that of adding too much work and intelligence to the actual provisioning stage. Large postrun sections or after_install command blocks should be a warning sign and point to tasks that may well be better off inside a system like Puppet or Chef. It's a seductive problem because it's an easy way to add additional functionality to a host, especially when it allows you to avoid thinking about applying or modifying a general role; even more so if it's one that's already in use on other hosts. Adding a single line in a kickstart or preseed file is quicker, requires no long term thinking and is immediately available.

Unfortunately by going down this path you end up with a lot of one-off host modifications, nearly common additional behaviour and a difficult to refactor build process. A tight coupling between these two stages can make trivial tasks unwieldy and in some cases force work to be made to remove or modify the change for day to day operation after the build has completed.

A good provisioning system should do the bare minimum required to get a machine built. It should be lean, do as little as possible and prepare the host to run its configuration management system. Everything else should be managed from inside that.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

PuppetCamp Europe 2010

Tue, 01 Jun 2010 20:45:00 GMT

To me puppet has always been a major evolutionary step up on the sysadmin tool chain. I consider it important enough to be ranked alongside version control systems and virtualisation as one of those mental leaps that leads to better management and enables more flexible solutions than you could offer before understanding it.

While I'm quite a long term member of the puppet community I'm no where near as active as I should be, but even I couldn't miss the chance to attend PuppetCamp Europe, and I'm glad I didn't! I finally got to meet some of Europes most prolific puppet module releasers in person, discovered that Brice is every bit as nice and as scarily smart in person as he is on-list and that the new PuppetLabs people are a very impressive bunch. Even I've still not had the chance to buy James some of those beers he's racked up over the years on the list.

Puppet may be an open source project but a very high proportion of its development and community support has always come from Puppet Labs, so it's critical to both the product and the users that their staff be as good with the community as they are with the code base, and having met half-a-dozen of them I can honestly say it feels like the project is in safe hands. Jeff gave an excellent talk on using Puppet in environments with strict compliance rules, Markus had a razor sharp grasp of what people were really asking (and gave the answer to what they wanted, not just what they asked) and Luke made the event for many of us, he very patiently gave a lot of advice and information not just about the now but also about the historical whys and theoretical hows.

I had an excellent time (Ghent itself is a lovely place to visit for a couple of days) so I'd like to thank Patrick for organising the event, Luke and Puppet labs for Puppet itself and the participants for making PuppetCamp Europe 2010 such an educational and enjoyable experience.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Hardening Apache - Short Review

Tue, 25 May 2010 21:00:00 GMT

I've had Hardening Apache sitting on my shelves for over five years (Sep 2004 or so Amazon tells me). While I can remember dipping in to it for the Apache chroot chapter it never seemed to progress to the top of the pile, and now I'm cleaning out a lot of my old books I decided to finally give it a chance.

The book is very well written, covers a good range of subjects from building apache from source to adding extra security modules and checking its running state. Those are all good points and if I'd read the book when it came out I'd give it a very decent score, unfortunately I waited to read it.

This is a book that hasn't aged well. The version numbers of apache mentioned, the last update times of the modules (and how many of them have fallen in to the pit of being unmaintained) and the general style of the shell scripts all just come across as very dated and prevent me from recommending this book

Well written but ravaged by time - where's the second edition?

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Pigz - Shortening backup times with parallel gzip

Wed, 07 Apr 2010 08:00:00 GMT

While searching for a completely different piece of software I stumbled on to the pigz application, a parallel implementation of gzip for modern multi-processor, multi-core machines. As some of our backups have a gzip step to conserve some space I decided to see if pigz could be useful in speeding them up.

Using remarkably unscientific means (I just wanted to know if it's worth further investigation) I ran a couple of sample compression runs. The machine is a quad core Dell server, the files are three copies of the same 899M SQL dump and the machine is lightly loaded (and mostly in disk IO).


#######################################
# Timings for two normal gzip runs
dwilson@pigztester:~/pgzip/pigz-2.1.6$ time gzip 1 2 3

real    2m43.429s
user    2m39.446s
sys     0m3.988s

real    2m43.403s
user    2m39.582s
sys     0m3.808s

#######################################
# Timings for three pigz runs

dwilson@pigztester:~/pgzip/pigz-2.1.6$ time ./pigz 1 2 3

real    0m46.504s
user    2m56.015s
sys     0m4.116s

real    0m46.976s
user    2m55.983s
sys     0m4.292s

real    0m47.402s
user    2m55.695s
sys     0m4.256s

Quite an impressive speed up considering all I did was run a slightly different command. The post compression sizes are pretty much the same (258M when compressed by gzip and 257M with pigz) and you can gunzip a pigz'd file, and get back a file with the same md5sum.

# before compression
-rw-r--r-- 1 dwilson dwilson 899M 2010-04-06 22:12 1

# post gzip compress
-rw-r--r-- 1 dwilson dwilson 258M 2010-04-06 22:12 1.gz

# post pigz compress
-rw-r--r-- 1 dwilson dwilson 257M 2010-04-06 22:12 1.gzs

I'll need to do some more testing, and compare the systems performance to a normal run while the compression is happening, before I trust it in production but the speed ups look appealing and, as it's Mark Adler code, it looks like it might be an easy win in some of our scripts.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

HTTP Server Headers via Cucumber

Wed, 31 Mar 2010 21:53:00 GMT

One of my little side projects is moving an old, configured in little steps over a long period of time, website from apache 1.3 to a much more sensible apache 2.2 server. I've been thinking about how to get the most out of the testing I need to do for the move and so today I decided to do some yak shaving and write some simple regression tests, play with Cucumber Nagios, rspec matchers and write a little ruby.

It's not exactly polished but after half an hour (mostly spent wrangling with has_key / have_key) I ended up with the following simplified example for testing HTTP headers:


Feature: http://www.unixdaemon.net/ response headers
 
  Scenario: Server header should be production quality
    When I fetch http://www.unixdaemon.net/
    Then the "Server" header should be "Apache"
 
  Scenario: Response header should contain an Etag
    When I fetch http://www.unixdaemon.net/
    Then the response should contain the "Etag" header
 
  Scenario: The Content-Type header should contain text/html
    When I fetch http://www.unixdaemon.net/
    Then the "Content-Type" header should contain "text/html"
 
  Scenario: The Content-Type header should not contain text/xml
    When I fetch http://www.unixdaemon.net/
    Then the "Content-Type" header should not contain "text/xml"

You can also find the cucumber-nagios steps for testing HTTP headers online. It's only a first step towards the full web server move safety net but it's useful one that'll stay in my toolkit.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

HTML & CSS - The Good Parts - Short Review

Tue, 30 Mar 2010 21:55:00 GMT

I'm guessing that if you're reading this then you've seen my very basic website at some point. I learned some HTML and CSS back when Netscape 4 and HTML 3.2 roamed the earth and while some of my very front end gifted co-workers have bought bits of my knowledge up to date I still don't understand how to properly lay out a CSS only multicolumn page without cheating.

I'm not sure if it's because i had vague expectations on what this book would cover or just if I'm not the target market for HTML & CSS The Good Parts but I've read the thing from cover to cover and nothing really stands out to me. All the right words are spoken, content vs style separation is good etc. but none of it feels new to me, the material is not explained in any new way that really gets the message across where other methods have failed and I very nearly gave up on the book half a dozen times. It's not a bad or horribly written book but it's also not one I could pick three best bits out of.

Make sure you have a skim through before you buy. Score 3/10

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Ada Lovelace Day - 2010

Wed, 24 Mar 2010 23:35:00 GMT

So today is Ada Lovelace day and we're supposed to "celebrate the achievements of women in technology and science." I don't know many women in science but I do know a few in technology and one in particular seems to go from back breaking task to another with politeness and grace I wish I could muster.

So for my 2010 Lovelace day (and because she'll need all the happy thoughts she can get now she's president of the Perl Foundation) I'm naming Karen Pauley. A long standing member of the perl community who's been involved in getting things done for more years than many people realise. Listing all her achievements would take a LOT of screen space (and annoy the hell out of her) but, to name three, her TPF work, YAPC::EU organisation and involvement in more related FOSS communities than you can shake a stick at are no small matter.

Speaking as someone who's seen her speak over half-a-dozen times, it's easy to see that Karen has a gift when it comes to presenting. Whether it's about technology, business or community its rare to hear her speak and not come out feeling both smarter and entertained, a combination we'd all love to be able to perform.

I've been lucky enough to chat with Karen outside of conferences and I've always come away from our email conversations with a smile and often with an idea of two, it's hard not to when you're speaking with someone who's both intelligent and a remarkable communicator. Karen is an exceptional person who we're lucky to have in the perl world, and I'm very fortunate to be able to call a friend.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Giving Cloud Computing An Edge - LOSUG March 2010

Sun, 21 Mar 2010 19:30:00 GMT

The LOSUG seems to be the user group with the least cross over of attendees that I go to. It seems to be a three part mix - Sun engineers going along to meet co-workers and get the external eye on to what's happening in different parts of the project, Unix people with dozens of years of experience who want something technical and interesting that matters on the server and people that don't listen to the speaker and then ask questions that, quite frankly, they should be embarrassed over. It's hard to stress how much I've always enjoyed the talks at LOSUG but some of the questions are just... insane.

Right, now I've got that of my chest - and I'll probably get lynched for it in the future - back to the March presentation by Alasdair Lumsden. I'm not going in to details about it as you can read the Giving Cloud Computing An Edge slides yourself now. It was an interesting talk and provided a nice counterbalance to similar talks I've heard in the past about Xen and UML hosting.

What made this LOSUG different to all the others though is that things are changing. Sun's always been very supportive of LOSUG (and always willing to put their hand in their pockets for food, drink and speakers) and now that Sun is owned by Oracle the group will be less driven by the core organisers. You can find more details (and less of me putting words in peoples mouths) at The Future of LOSUG but I wanted to take this chance to both encourage people to come along and show Oracle that the group's important and to say thank you to Joy Marshall, James MacFarlane and Stuart Smith - who have month in and month out organised an excellent event with speakers you couldn't see anywhere else.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Network Ninja - Short Review

Sun, 21 Mar 2010 18:58:00 GMT

I'd never even heard of this book until Bob used its name in the same sentence as the excellent "Cisco Routers for the Desperate". However while that book is about hands on practical Cisco advice Network Ninja is all about the theory - from IP addressing to routing protocols.

While no one's ever going to confuse 200 easy to read pages with the Stevens books this slender volume is an excellent refresher for the experienced admin who doesn't do too much to the network on a day-to-day basis or for the less experienced admin who wants to know some of the why instead of just the command lines.

An enjoyable and opinionated book that covers a lot of ground in a low page count. Only let down by some bad editing - 7/10

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

LibguestFS GLLUG Talk

Sun, 21 Mar 2010 18:12:00 GMT

Over the years there have been a handful of GLLUG members that have given so many interesting talks that I'll always turn up to watch them - and Richard Jones is definitely in that short list.

The website does an excellent job of explaining: "libguestfs is a library for accessing and modifying virtual machine (VM) disk images. Amongst the things this is good for: making batch configuration changes to guests, viewing and editing files inside guests (virt-cat, virt-edit), getting disk used/free statistics (virt-df), migrating between virtualization systems (virt-p2v), performing partial backups, performing partial guest clones, cloning VMs and changing registry/UUID/hostname info, and much else besides." but it doesn't quite convey how cool it is to spin up access in to a windows machine in a handful of seconds and then dump out the registry key you're looking for - all from a Linux command line.

Oh, and even if you didn't turn up (tsk tsk) you can read all about the libguestfs gllug talk here.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

The Book of Xen - Short Review

Sun, 21 Mar 2010 18:01:00 GMT

Although I've been a big fan of virtualization for many years I've mostly been a VMWare man. UML was good for the time but VMWare workstation and GSX always seemed to be better solutions - and they had the benefits of dealing with Windows. At $WORK we looked at using Xen for our new development environment but it never felt very finished, little things like needing to compile your own dhcp client in order to get PXE booting working always felt very wrong.

But now we're looking to move away from VMWare server for certain parts of our infrastructure everything's back on the table so I went looking for a guide through the lands of Xen in the modern world - and I think I found an excellent one in The Book of Xen.

The book takes you through all the aspects of using Xen that you'd expect, from installing it, configuring the guests (DomU in Xen terminology) to making the most out of the networking options and local storage possibilities. Where it goes that extra mile is in sections like 'Beyond Linux', which guides you through using NetBSD and Solaris with Xen, Profiling and benchmarking under Xen and Lessons from the trenches, in which the authors (who run a Xen hosting service) tell you about their real-world aches and pains.

Apart from the chapter on the commercial Citrix XenServer, which I can understand the inclusion of but isn't useful to me, there was something interesting in every chapter. After working through the book I have a good understanding of what needs attention in a Xen hosting setup and what might be weaknesses. All I need now is a similar book for KVM so I can avoid doing all my own research!.

An excellent guide to Xen that brings a lot of useful material into one place - 7/10

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

London DevOps - March 2010

Thu, 18 Mar 2010 00:13:00 GMT

This month was the first of the London DevOps tech talks. Organised by R I Pienaar and masterfully shepherded on the evening by Chris Read about thirty sysadmins (and some developers, project managers and scrum masters) met for a series of impromptu discussions, beer and pizza

While there was no formal schedule for the evening Chris led the group in a fishbowl, seeding some ideas and then watched the conversations bloom. We went through some tool chain issues, trending, log analysis, how Splunk is the best thing since sliced bread with bacon in it and how Centos does some very interesting things with the data they collect. It was the first fishbowl I'd ever attended and it was actually a lot of fun, especially when people suggested RDF and SPARQL for a common data store.

A short break was taken when the pizza arrived and a number of interesting conversations broke out, how little admin time Apache Solr seems to need (and how odd it is to use rsync and shell scripts to sync out changes), how Redis and CouchDB are making certain problem domains easier to deal with and how the BBC has so many cool people hidden away were among those I ambled in to.

ThoughtWorks kindly donated beer, pizza and most importantly the venue - and for that we should say thank you. Getting a decent venue is always difficult for a new group. Although it's early days the group feels like it's got potential, the conversations were interesting, we don't all agree on where we should be heading and what we need next but the atmosphere was friendly and open. Hopefully these meets will last longer than SAGE-WISE did, with all the developer focused events in London it's nice to get to one that's a little closer to what I do.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

BSD Magazine - A decent read

Thu, 11 Feb 2010 22:11:00 GMT

While looking for an OpenBSD baseball cap on the BSD stalls at FOSDEM I was given a couple of issues of the BSD Magazine to flick through - and it's a lot better than I'd hoped.

As most of the UK Linux magazines have become very desktop focused it's nice to see some actual low-level code - packaging for OpenBSD, writing sound drivers for your NetBSD NSLU2, custom Jabber components and basic GDB were all in the two issues I skimmed. While it's not the dearly departed Sysadmin Magazine, and it could do with an editor or two - much as I could, it is a decent read and I'm considering a subscription.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Grocery Arrival Excitement?

Wed, 13 Jan 2010 20:05:00 GMT

Many years ago, in the first dotcom boom, I worked for a website performance monitoring company. I was one of the early employees (developer number 3 and sysadmin number 2) and I remember being in a meeting with the company CEO who was telling us about a new pitch we were doing for $SUPERMARKET, they were going to try this new idea of shopping online and then delivering it to your door.

The worst part of it was that they didn't just want monitoring, they wanted a full transaction engine (with some basic OCR), a product I can probably get away with confessing that we didn't have at the time of the sales pitch. We all knew the deal, if we didn't get it life was going to be very hard there for the next six months, so we all knuckled under. The road was long, difficult and uphill in the snow in both directions but eventually we got to the day of the pitch. Which we aced in an astounding display of luck - the new app sometimes got itself in to a little bit of a state if their website had a failure - which it did about 20% of the time. They loved the demo and wanted us to give them full coverage while they did maintenance work. If we pulled it off then we'd pretty much get the deal, none of our competition at the time could match the features, it was just the uptime that was a little worrying.

So we went out and bought a dozen small desktops, monitors and networking kit, installed them all in our spare store room, put some tables and chairs in and had a company meeting. The management were completely open about what was happening, they took questions and then asked how far we'd go to help. We covered the whole weekend from Friday night to Monday morning. Nearly the entire company chipped in, from three letter titles to sales to dev to systems to HR. We had eyes on the machines over the whole period, including when the Solaris admin, the only person to let us down, didn't make his time slot. Out of all the transactions the worst was beans, they had a new version of the code on some of the servers and it'd return very odd results for beans and break the transaction runner in horrible ways. I'll never forget the 4am calls asking what we do when they offer you a lawn-mower instead.

I placed my first ever order online with the $SUPERMARKET yesterday and hopefully it should arrive in the next couple of hours. The interface may have changed and so many of its users take the service for granted that it's a little humbling to realise how much the Internet's changed so very many things. I guess this post's about a combination of things, the best job I ever had (the company was sold in the end to one of it's competitors. I left happy in the knowledge that we ate their lunch until they gave up trying to compete and bought us), how dedicated staff can be in the right environment, why you should push the boundaries of your industry and how sometimes even cans of beans can be exciting.

I had to put a single can in the order to complete the circle. Here's to hoping they don't charge me for a lawn-mower.

Update: They didn't deliver on the night, there was a "problem with the payment" so they took the money out, using the same details and delivered it two nights later. I'll class this one as a draw.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Spreadsheets Vs Post-It Notes

Tue, 12 Jan 2010 23:32:00 GMT

I'm a fan of documentation, over the years I've ended up supporting more than one business critical system that has less documentation than you get from a cat /dev/null.

The only downside, and I've been bitten by a couple of things like this over the last week is the case of the spreadsheet vs the post-it note - if you have a lovely, well formatted and information dense spreadsheet that says "A is 1" and when you get to the server room the switch has a post-it, in bad scrawl, that says "B is 2" which do you believe?

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Hadoop Talk - SkillsMatter 2009

Sat, 17 Oct 2009 19:17:00 GMT

After an embarrassing tale of misunderstanding, wrong locations and blind luck I recently ended up at the Introduction to data processing with Hadoop and Pig talk over at SkillsMatter - and it was excellent.

For those that don't know about Hadoop, it's an OpenSource Java framework for data-intensive distributed applications. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google's MapReduce and Google File System (GFS) papers. I was aware of the basics but even in an hour I learned enough to know where to look for more details. Pig on the other hand is (to me) like SQL but for Hadoop, it's a lot easier to use than writing your own Java apps and simpler (and actually possible) for non-developers to read than the reams of classes required for custom jobs.

The speaker was excellent, the presentation was well timed, fluid, concise, paced just the way I like it and other than the question session the evening was very enjoyable. You can find the Hadoop slides online.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

JRuby Cookbook - Short Review

Sat, 17 Oct 2009 19:08:00 GMT

First a disclaimer, I'm not a heavy Ruby or Java guy. Most of my coding for the last couple of years has been perl and shell - because I write little things that I need right now and those two languages excel at that (CPAN is still THE decision clincher).

I recently became involved in a side project that is written in Ruby and Java though and in an excellent timing coincidence a friend returned my previously unread copy of the JRuby Cookbook. The book isn't an introduction to either Java or Ruby (there are already excellent online and dead tree resources for that) but it shows where the two can meet and how to get started at those points. It's not really a book to read back to front but it is a good approach for a cookbook.

If you're curious as to how dynamic languages on static language VMs can complement each other this is a good book to flick through. Score - 6/10 - it's not the book for me right now but it does show a lot of entry points I'll probably come back to later.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!