November 20143
October 20143
September 20142
July 20141
March 20146
February 20141
January 20145
December 20135
November 20132
October 20134
August 20134
July 20132
Full Archives

Sat, 04 Jul 2009

Baby Steps with Python

Thanks to the enthusiasm I've returned from EuroPython with (and the fact I couldn't make it to OpenTech because of washing machine issues) I decided to spend a little bit of time porting Nagios Webchecks to python. As a use case it covers a lot of the functionality I need in my day to day system scripts. The ability to specify command line arguments, read a config file and interpolate a template file for output.

I'm no where near finished (and I need to spend a lot more time with Python in a Nutshell before I'm going to feel comfortable) but there's been nothing too horrible so far. Libraries returning tuples rather than lists still feels a little awkward but I suspect that's just me being new to this. ConfigParser seems to meet my current ini file reading needs (which are pretty simple) and Jinja2 looks like a decent template engine. Biggest personal bugbear so far is that using % when printing a variable feels wrong ( print "Hello %s", name feels a lot more comfortable than print "Hello %s" % name and is a more common format across languages.)

I'll show the code to a co-worker or two on Monday and hopefully pickup some constructive criticism as they destroy my slightly too perlish attempt at python code.

Posted: 2009/07/04 18:30 | /programming | Permanent link to this entry

Mon, 06 Apr 2009

Ruby DNS Testing - First Glance

DNS is one of those 'small config change here, errors a long way over there later' technologies that always leaves me a little worried about the knock on effect of my changes. As a simple, coarse, safeguard at work we use Nagios to check that a canary record in each zone can be resolved from each DNS server. It's far from a perfect solution but it does catch some of the bigger errors and typos.

In order to beef up this safety net (and encourage me to spend time using a language other than Perl) I've been investigating some of the testing options available in ruby, namely RSpec and Cucumber. I have to say the testing libraries themselves are actually nice to use and easy to pick up even for a non-rubyists like me. On the other hand I quickly developed a strong dislike of the ruby resolv library that actually does the DNS queries. The lack of decent tutorials or documentation for anything beyond the very basic uses and the (to me) very awkward API nearly had me running back to the safety of Net::DNS , a mature and widely used perl module. There is a ruby port that I'll have a look at in the future

For the testing itself I started writing RSpec stories against my own DNS and found the API easy to use. Testing existing configs against local policies is simple - for example all our domains should list at least three name servers -

it "should have at least 3 NS records" do
  @resolver = Resolv::DNS.new
  @nameservers = @resolver.getresources( domain, Resolv::DNS::Resource::IN::NS)
  @nameservers.should have_at_least(3).items

After doing some more scut work and testing other record types I moved onwards and upwards to Cucumber. I'm not sure I'd be able to gift someone else with writing the scenarios but they'll be easier to show business people than raw rspec stories. They'll also be very handy in migration meetings (an executable todo list), if it's not on the page it's not getting done.

Feature: Mass DNS Resolution
  In order to present a consistent brand image
  As a System Administrator
  I want to ensure no domains point away from our main IP

  Scenario Outline: Resolve a name to a number
    Given a hostname of <hostname>
    Then I should see the IP address

     | hostname        |
     | example.org     |
     | www.example.org |

Next time I work somewhere with ISO 27xxx compliance requirements I'll see if the controls can be written like this and have automatic verification run from them. Cucumber is a little wordy for my tastes but I can see where that could be a strength when presenting to the right audience (such as compliance auditors).

Next time we have a set of DNS migrations I'll be using at least one of these tools to write before and after test cases to ensure nothing gets missed or slips through the cracks. Once I've been through a couple I'll write up in more detail what we end up with.

Posted: 2009/04/06 15:46 | /programming | Permanent link to this entry

Thu, 01 Jan 2009

Erlang in Practice - PragProg Screencasts

I recently watched the first in the series of the Pragmatic Programmers Erlang in Practice Screencasts (by Kevin Smith - no, not that Kevin Smith). As I've not seen them discussed that much else where I thought I'd jot down my thoughts.

First up a disclaimer/warning - I'm not an Erlang person and despite the title of 'Episode 1' this series of screencasts is not aimed at people with no experience in the language. If you want to learn Erlang then I'd suggest you read Programming Erlang instead. Once you've been through the book then you should consider coming back to this series.

Now, to look at the screencasts from a different angle - production quality and value for money. Despite not knowing enough Erlang to understand all the code presented, I found the quality of the screencast to be perfect for watching on a laptop. The video was clear, the presenters voice didn't make me want to kill him (although this is a highly personal thing) and at five dollars the price was right for an hours worth of content.

So would I buy another one? Yes, but not this series. Until I get a chance to work my way through the Erlang book this series is off limits to me, The Ruby Object Model and Metaprogramming on the other other hand is mighty tempting for under five UK pounds...

Posted: 2009/01/01 20:05 | /programming | Permanent link to this entry

Sat, 08 Nov 2008

Disturbing Diffs - Unsafe open?

-  file_move_safe(move_from_path, move_to_path)
+  move_file(move_from_path, move_to_path)

Is move_file not as safe as file_move_safe? Is it safer? Dare I read the other diffs to find out? Am I better off not knowing?

Posted: 2008/11/08 13:11 | /programming | Permanent link to this entry

Dynamic Languages and joining arrays

I've been spending a fair amount of time recently trying to choose my Language of the year for 2009. I've always been a dynamic language fan (yes, I know this means I should be looking further afield for the next one) and I was surprised at how different even such a common task as joining all the elements of an array together, using a given separator, looks between them.

First let's look at the big three, including perl, my current favourite.

# perl
$ perl -d -e 1;
DB<1> my @names = qw( A B C);
DB<2> print join(" : ", @names), "\n";
A : B : C

# python
>>> names = ['A', 'B', 'C' ]
>>> " : ".join(names)
'A : B : C'

# ruby
irb(main):009:0> names = [ 'A', 'B', 'C' ]
=> ["A", "B", "C"]
irb(main):010:0> names.join(' : ')
=> "A : B : C"

The perl approach is very procedural (ignore the use of the debugger as perl doesn't come with an excellent REPL in the core like the other two) and is the one I'm most familiar with so it's hard for me to be too critical about it. If you like OO then it's not for you.

Next we have Python, which is really growing on me as a language - apart from in this case. Putting the separator first and passing the list in as a parameter just feels very wrong and is the exact opposite of the ruby version, which I much prefer. To me the ruby approach of operating on the array is the most natural version and sits well in my head. As a small 'bonus' I also looked at the PHP equivalent -

$array = array('A', 'B', 'C');
echo implode(" : ", $array);

This is close enough to the perl version that I can't really object to it, other than to (rhetorically) ask why the hell it's called 'implode'?

I guess all I can say in summary is round one to ruby.

Posted: 2008/11/08 12:41 | /programming | Permanent link to this entry

Sat, 06 Sep 2008

My First Day with Python - Initial Thoughts

While I've always been a bit of a perl guy I don't want this post to be "perl has x and python doesn't" in tone. Which is lucky really as Python has exceptions and threading as first class features where as perl has... ahem.

So after spending a chunk of today reading a python book and spending some time writing code here's my initial short list of gripes -

Considering how picky I can be that's a very short list so Python must sit well with me so far. Now, in order, I can't help but read except IOError as 'catch everything apart from IOError'. This one bugs me more than it should but considering how happy native exceptions in the language made me this just felt mean.

Secondly, print adding newlines. While this might seem trivial every other language I use on a daily basis has a print function that doesn't print a newline so this feels weird. At least it's not called say ;)

Now to the one that I'll get no sympathy on - whitespace in blocks. First up let me say I don't mind about the enforced indentation. I indent anyway so it's not a big deal. I guess I'll hit the odd case when it annoys me (probably involving heredocs) but I've got nothing against it. What does irk me is the lack of block delimiters - whitespace just doesn't cut it for me.

I like my { and } delimited blocks, a nasty voice in my head is telling me to add them but just comment them out ( if x == y: # { ) but that seems very wrong. I've always looked at those examples in C programming books that say...

# incorrect
if ( something )
  print("All's well");

# this is wrong because wellness is a separate statement
# and not part of the if

... and thought - "just add the damn braces, you'll be back to add more code later anyway." Now I'm learning a language that seems to want me to slip up like this. I'll either get used to this or move to ruby.

Lastly we have the lack of ++ and --. I know the arguments, I've read them before. I disagree. I've never done anything insane with ++ and where I have used it it's saved me typing. Can we have ++ and remove nested ternary ( ? : ) instead please?

I like Python and I think I'll be investing more time in to learning it.

Posted: 2008/09/06 22:44 | /programming | Permanent link to this entry

Fri, 11 Jul 2008

Randexp - Generating test data with Ruby regexs

While paging through reddit programming recently (seems only fair since they linked to me ;)) I stumbled on to the very nifty Randexp gem, a library that uses regular expression patterns to generate data that would satisfy the pattern. Or in less tech terms - a really good test data generator.

# install randexp

$ irb

require "rubygems"
require "randexp"

# simple fake phone number -
/020(7|8) \d{3} \d{4}/.gen

# build a reusable class.
class Randgen
  def self.version()

# and use it.


I especially like the ability to make your own character classes. I'm not a ruby guy but I can see this being very useful in lots of little data generation scripts and test harnesses.

Posted: 2008/07/11 18:29 | /programming | Permanent link to this entry

Tue, 08 Jan 2008

Bash Puzzles - Brace Expansion

There is nothing like other peoples code to highlight all those little gaps in your knowledge of a programming language. I know what the first one does:

$ mkdir -p {projectone_,projecttwo_,projectthree_}log

$ ls -1

And I was a confident (and a little bit happy) about knowing what the second one does:

$ mkdir -p {project_one,}log

$ ls -1

But I had no clue about this one. And my guess (that it would be treated as a single element and expand to 'projectone_log') was very wrong:

$ mkdir -p {projectone_}log

Posted: 2008/01/08 21:34 | /programming | Permanent link to this entry

Wed, 14 Mar 2007

Playing with Facter

I'm on-call tonight so I invested some time in facter, "A cross-platform Ruby library for retrieving facts from operating systems." While facter is an interesting command line program (its extension mechanism is quite nice) its main claim to fame is that it's used by puppet (which I'm slowly evaluating as a CFEngine replacement) to determine facts about a machine.

While the docs are a little light on the ground the tgz contains a couple of examples and after some playing around I think I've got a basic Linux Bonding fact ready. For your viewing pleasure, the Facter Linux Network Bonding custom fact. It's not amazingly powerful or complex but it does seem to do what I want and it gave me a reason to look around the Ruby Dir class so it's not all bad. I've mostly put it up to show how easy it is for someone with very little ruby knowledge to extend facter.

Note: I also discovered that you can't do a confine :bonding => :true, facter works on literal string values, not on true or false.

Posted: 2007/03/14 23:20 | /programming | Permanent link to this entry

Mon, 12 Mar 2007

daemon_percentages.rb and Ruby Autovivification

Both Jim Weirich and Ben Summers were kind enough to email me about my Daemon Logging Percentages and Playing with Ruby Idioms post. They sent me an explanation on how to do the hash assignment in a way I find much nicer, so with no more delays I present - Option 4:

  tally = Hash.new(0)
  tally[daemon] += 1

It really is that simple - and I still missed it by a mile. I've updated the script to use this and I wanted to say thank you for the pointer, so thank you both.

Posted: 2007/03/12 21:06 | /programming | Permanent link to this entry

Sat, 15 Jul 2006

Don't do This With 'find' - part 1

Hell is other peoples code. -- Not quite Sartre.

I don't mind using other peoples code. I'll even submit patches if I find a problem, but discovering the same mistakes in almost half-a-dozen projects is enough to drive me insane. Here are three common red flags involving the find command and how to get around them:

If you want to be portable, don't use GNU/Linuxisms.

Compare these two commands -

find -name "*.log" # not portable
find . -name "*.log" # works everywhere

The first one works with GNU find so it's fine for (almost) all Linux distributions. It doesn't work on older Solaris, HPUX or AIX machines and saves one dot. Lesson: specify your paths.

Know what happens when you match nothing.

The first example (shown below) works fine as long as it matches. If the find doesn't return anything you get a syntax error.

find . -name "*.tmp" | xargs rm # pray for a .tmp file
find . -name "*.tmp" | xargs -r rm

The second example exits silently if there is nothing for it to do. As noted by Dave Cantrell, the -r option is a GNUism so be careful where you use it. You can also get around this by using find options -exec foo instead of xargs. But that comes with its own baggage. Lesson: Plan for errors.

If you're using GNU/Linuxisms then use them properly.

The GNU utilities add a number of powerful and convenient options. If you've decided that your script only cares about Linux use them and save a lot of hassle.

find . -ls | awk '{ printf("%s %s %d\n", $NF, $5, $7) }' # horrible
find . -printf "%p %u %s\n" # better

The first example is (possibly) more portable but it's horrible. And not something you want to see in an init script. The second one is tied to newer versions of find but is much clearer to read, once you've looked up the escapes using man find. Lesson: Know your options.

Posted: 2006/07/15 11:36 | /programming | Permanent link to this entry

Sun, 05 Jun 2005

Blosxom TagCloud

A tag cloud is a visual depiction of content tags used on a website. Often, more frequently used tags are depicted in a larger font or otherwise emphasized. Selecting a single tag within a tag cloud will generally lead to a collection of items that are associated with that tag. From the Wikipedia Tag cloud entry.

Ever wanted a tag cloud of your Blosxom posts? With just this blosxom-tagcloud.pl script (and three Perl modules from CPAN) you can have one that integrates itself with your Blosxom footer and even allows easy merging of the tag cloud and any static text/HTML you've used in the past.

Overview: blosxom-tagcloud.pl works by scanning a directory tree, the location of which is a config setting, and building a list of all your posts and the categories they are under. Once it's got this list it uses the HTML::TagCloud module to generate some HTML and CSS for their display. It then tries to merge in any static footer you might be using and creates a footer file that can be included directly in your blog. This isn't a Blosxom plugin, this script should be run on a periodic basis to save the machine needing to do a file system walk with every blog hit.

Installation: The installation is easy, download the script, if necessary un-compress it, install the required perl modules (File::Find::Rule, IO::AtomicFile and HTML::TagCloud) and then tweak the configuration settings at the top of the script to suit your Blosxom install. The settings are mostly self-explanatory and I've left examples in the code.

Notes: I'm undecided on the use of tag clouds in a UI so I thought I'd conduct a little experiment and see how useful they are by adding one to my own blog. The perl script below, blosxom-tagcloud.pl, is what I use to generate the footer that contains the tag cloud; to see an example scroll to the bottom of the UnixDaemon Blog. The script will search a given directory tree for Blosxom posts (they must all have the same file extension) and then write out a footer file, which can include another static file, that will be displayed on your blog.

The script itself is pretty short and uses three main modules, all of which are available on CPAN. Two of the modules, File::Find::Rule and HTML::TagCloud do most of the heavy lifting and are essential. You could probably replace the dependency on IO::AtomicFile quite easily but it's not really worth the effort.

Posted: 2005/06/05 10:45 | /programming | Permanent link to this entry

books career cloud events firefox geekstuff linux meta misctech movies nottech perl programming security sites sysadmin tools tools/ansible tools/commandline tools/gui tools/network tools/online tools/puppet unixdaemon

Copyright © 2000-2014 Dean Wilson :: RSS Feed