Dean Wilson@UnixDaemon: ... be good, get good or give up.

Install Package Dependencies After a DPKG Package Install

Tue, 15 Apr 2008 17:44:00 GMT

I've had a couple of people mail asking about my "frigging apt" comment in a previous post (the last paragraph). It's actually as simple as the comment implies. Here's an example -

  
  wget http://ftp.dk.debian.org/debian/pool/contrib/v/vmware-package/vmware-package_0.22_i386.deb
  dpkg -i vmware-package_0.22_i386.deb
  apt-get install -f

  # get prompted about installing lots of packages

I don't have any really well thought out reasons to not like this approach - in the few cases I've tried it I've found it to work; it just feels a little... icky.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Rebuilding Debian Packages - Debian Delvings

Tue, 15 Apr 2008 09:11:00 GMT

Ever wanted a Debian package to be just a -little- bit different? Here's how. While most of the software we're pulling in from Debian is fine for our uses there are a couple of applications that we'd like to be a little different than the stock versions. Rather than go away and package them ourselves (which would require a lot more packing skills and time than I currently have - improving those skills is one of the reasons I'm doing this series) it's possible to download a source version of a Debian package, make a small amendment and then repackage it for personal use. In this example we'll have a look at how to do this with NginX, a very fast and stripped down webserver I'm evaluating for a couple of services.

Firstly let's be nosey and have a look at the Nginx source package. Assuming you have a deb-src line in your /etc/apt/sources.list file getting a copy of the source package is as simple as apt-get source nginx; it's worth noting that this command doesn't need you to be root to run. While we're here let's pull down the dependencies required to rebuild nginx once we've made our modifications - sudo apt-get build-dep nginx - note the build-dep argument - and then pull down the other packages required but are not specific to this package.sudo apt-get install build-essential devscripts fakeroot (sudo or as root).

It's a bit of a tangent (and an annoyance) but the package of Nginx in Debian stable (nginx 0.4.13-2) has a Nginx package removal bug that stops you from removing the package without hacking the init script. I only point this out as when I was trying to learn how to repackage I couldn't remove it and so assumed I'd broken something.

Before we make any amendments to the package let's try and rebuild it. cd in to the versioned source directory (nginx-0.4.13 in our example) and run debuild -us -uc. A few screens of text later and we've got a package - ../nginx_0.4.13-2_i386.deb.

Now we know we can round trip let's be intrusive and make some amendments to the package. We'll add an optional module (http_realip_module) to the nginx package (you should be able to do something like this to add SSL support for example. In this case it's a small, one line, change -

  
 $ cd nginx-0.4.13 # this is the etch version when I wrote this

 $ vi debian/rules # add '--with-http_realip_module' to the ./configure line.

 $ dch --nmu "Rebuild of package and addition of the http_realip_module module"

 $ vi debian/changelog # change the name and email as required.

 # build the new package with the extra module
 $ DEBFULLNAME="Dean Wilson" DEBEMAIL=dwilson@example.com debuild -us -uc

 $ ls -alh ../nginx_0.4.13-2.1_i386.deb

  # if you have a repo drop it in now and regenerate the index for clients to see it.

dch simplifies adding changelog entries to debian/changelog. The two variables before the command tell it who's made the change. In our case we're logging that we added the new module and that this package is a "Non maintainer upload" (NMU). This is important for future users of the package - in essence we're taking responsibility for the repack away from the oringinal maintainer, who knows nothing of the changes you've made, and making ourselves the correct contact point. And if you don't do this lintian will complain when you build the package.

Closing notes - you may need to pin or hold the package to stop it getting overwritten when future (maintainer) upgrades become available - but that's outside the scope of this post.

Bonus Debian Nginx package Evil. Bad idea but useful for some testing I was doing.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Creating Virtual Debian Packages - Debian Delving

Tue, 15 Apr 2008 09:05:00 GMT

Now that we have a local apt repository we can start to fill it with our own custom goodness. One of the first things I'm going to need is virtual packages. At work we'll be using them to pull reusable components together in to a number of full applications (and using the apt mechanisms to force upgrades when a component changes) and to group Nagios plugins (we'll be packaging some of those in a later blog post) in to sensible sections; we're going to have a lot of Nagios plugins.

Unfortunately I can't talk about those in detail on a public website that not enough people read, so we'll need a more contrived example. Although I understand the reasons behind it (and appreciate them on servers) the fact that Debian doesn't have a perl package that INCLUDES THE CORE PERLDOC bites me every now and again so we'll make our own perl package that pulls in a handful of the existing packages.

Tangent - I think the three packages should be perl-base, perl-modules and perl-doc which anal people like me can pick for their systems. The perl package should be a virtual one that includes (and forces the install of) all three of those so you get a more intuitive setup for people that don't need huge amounts of customisation. This could also be done with Ruby to hide some of the bat shit splitting of it's libraries under Debian. But that's all personal opinion; and now I've got virtual packages and an apt repo I can do what I want. Bwhahahaha.

While I was trying to work out how to do this I found the equivs package and it's got one of the best descriptions I've ever read:

 
$ apt-cache show equivs | less
... snip ...
Description: Circumvent Debian package dependencies
 This package provides a tool to create Debian packages that only
 contain dependency information.
 .
 One use for this is to create a meta-package: a package whose sole
 purpose is to declare dependencies and conflicts on other packages so
 that these will be automatically installed, upgraded, or removed.
... snip ...

In additional to being a very clear statement of intent this is obviously the command for us. So how do we use it?

 
# on the build machine, which also hosts our sample repo.
$ apt-get install equivs

$ equivs-control perl-full

# edit the file to suit - example linked to from the next paragraph
vi perl-full

# build the meta package
equivs-build perl-full

# copy it in to our repo - which lives under 
cp perl-full_1.0_all.deb $repo_base/dists/pool/main/

# rebuild the index
pushd $repo_base && apt-ftparchive generate $repo_base/apt-ftparchive.conf

Here's an inline version of the perl-full example control file that I used.

 
Section: misc
Priority: optional
Standards-Version: 3.6.2

Package: perl-full
Version: 1.0
Maintainer: Dean Wilson 
Depends: perl,perl-doc,libdbi-perl
Architecture: all
Description: Perl install that includes perl-doc and libdbi-perl packages.
 I'm fed up of installing perl and not getting perl-doc. This virtual
 package fixes that for my local systems and any other that can reach my
 apt repo
 .
 I don't recommend anyone else using this as it was written as an
 experiment and to voice how much having to install perl-doc bugs me.
 Oh, and this isn't a good example of a description. Look at real Debian
 packages for much better ones.

On the client you can do an apt-get update ; apt-get install -s perl-full and you should get a list of the packages that will be installed, which on a system that's not already got a perl install should now include perl-doc.

While this example is a bit forced (and very opinionated) the technique and technology behind it are useful tools that are worth knowing.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Creating a Personal Apt Repository - Debian Delving

Tue, 15 Apr 2008 09:01:00 GMT

Ever wanted your own apt-repo? If not hit the back button about.... now.

My new employers are going to be very Debian heavy on the systems side of the project I'm on so I'm currently in the process of sharpening my Debian specific skills (I've always tried to avoid Unix solutions that were tied to a single OS or distro but in this case we might as well do it The Debian Way).

One of the first things to come up was the need for a local apt repository - for internal packages, third party ones we wanted to use, some backported from other Debian releases and even some rebuilt ones. In this, the first of who knows how many, Debian focused blog post I'll be describing my first pass at setting up a repo for holding these and how I'm using it. I should probably say that this is all from bits and pieces I've picked up across the web so be careful before you blindly follow my advice.

First of all let's set a victory condition - I want to be able to apt-get install puppet and have it install the version currently from testing (Lenny as I write this) under a stable system (Etch) without any pinning, apt magic etc. Before we start, run apt-get install -s puppet | grep 'Inst puppet' and see what version you get back - I get 'Inst puppet (0.20.1-1 Debian:4.0r3/stable)' so it'll be pretty easy for me to see if this works; when it does the version number will change.

We're going to serve packages over http so you'll need a webserver (apt-get install apache2 in my experiments) and a couple of other debian packages (apt-utils and bzip2). Rather than go in to a line by line description grab my simple make-repo.sh Debian Repository creation script and my sample apt-ftparchive.conf.

Once you've got local copies of both, had a quick look at them (you wouldn't run arbitrary shell scripts you've downloaded from the net would you?) and made any config adjustments run make-repo.sh and watch as it adds directories to your system. You now have a skeleton repo on your machine; but without any packages it's about as much use as a Westham fan at a cup final. So let's add a package.

  
$ cd dists/pool/main/ # this is under your repo base

# grab the puppet version from lenny (at time of writing)
$ wget http://ftp.de.debian.org/debian/pool/main/p/puppet/puppet_0.24.4-3_all.deb

# generate the index files
$ cd /var/www/debian # my repo base
$ apt-ftparchive generate /var/www/debian/apt-ftparchive.conf

On the client side edit /etc/apt/sources.list and add a line that looks like deb http://apt-repo.example.com/debian/ etch main - replacing apt-repo.example.com with your web host. Do an apt-get update and you should get a couple of 'Ign' warnings but all should work. If not, then get debugging. Now for the moment of truth, on the client -

  
$ apt-get install -s puppet | grep 'Inst puppet'
Inst puppet (0.24.4-3 apt-repo.example.com)

You now have a local apt repository with a sensible version of puppet ready for use by all your Etch hosts. It's also a good building block for fulfilling some of our other requirements. But well get to those in other blog posts. Now go and add the process and website checks to Nagios.

Notes - while you can pull the packages down and dpkg -i them one by one on each machine this requires you to copy them to each host and (this is the annoying part) install the dependencies by hand (yes, you can frig this with apt-get -f and let it try to do this for you but that's horrible). I should also say that if I'm doing any of this horribly wrong then feel free to mail me with corrections. I'd love to know how the Debian pros do it. (don't send me emails with the word "Alone").

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Unemployed - Again

Sat, 01 Mar 2008 00:40:00 GMT

I'm far happier than I should be to announce that as of about 39 minutes ago I am once more unemployed. My two month contract finished on time and I'm looking forward to being a strain on society for the entire weekend. I start a new role Monday morning (not sure what I can and can't say about it yet) but I'm looking forward to getting my teeth in to their technical challenges rather than just advising.

Thanks to everyone who sent me links and leads when I posted about leaving my last job, they were much appreciated.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Earthquakes in the UK?

Wed, 27 Feb 2008 01:14:00 GMT

I've just been woken up by the flat moving. Cups rattling, shelves wobbling and my ceiling light clinking against itself. Felt very much like the aftershocks of the earthquake we had a couple of years ago.

Here's hoping no one's hurt.

Update: BBC coverage. So I wasn't the only one to notice then.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

This years blockbusters?

Thu, 17 Jan 2008 20:38:00 GMT

I've just had a skim through the Times onlines 50 movies of 2008 and while I was picking my handful of must sees (Cloverfield, Indy 4, and Hellboy 2) I noticed a couple of interesting comments

While I'm actually very fan boy about the idea of an Avengers movie (mentioned in the "The Incredible Hulk" comments) I felt a cold shiver of dread when I read 'Hayden Christensen, who has also just signed to star in the long-awaited movie version of William Gibson's prescient sci-fi classic Neuromancer,'. Nothing good can come of this; unless we get Dina Meyer back as Jane nee Molly.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

OpenSolaris User Group - Jan 08

Thu, 17 Jan 2008 20:33:00 GMT

I attended the Advances in OpenSolaris Network Administration talk hosted by LOSUG over at London Bridge last night. And no one mentioned MySQL.

I came out of the session with a couple of pages of notes but two things really stuck out - the talk covered the new developments as a sequential feature list rather than showing you something cool or interesting and then explaining how the new technologies made it possible. A couple of slides had command lines on them to illustrate a point but they were little tweaks and changes rather than something that made you sit back and say "I want that." This is in no way uncommon but it is often an opportunity wasted.

This leads to my the second point, a lot of the topics covered were about incremental improvements and addressing concerns from previous versions of Solaris. While these are probably great things if you are stuck with older Solaris installs and looking for a way out the ability to sniff loopback traffic and rename nics didn't really inspire me much, we've had those abilities in the Free and Open worlds for a fair while now.

I should also note that I -really- dislike the term "vanity naming" when it comes to changing the name of a network interface. It just makes the process sound so pointless and frivolous.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Neward, chromatic and External Perspectives

Wed, 16 Jan 2008 11:36:00 GMT

Ted Neward is the latest person to get linked to in the ongoing campaign to prove that parrot isn't dead, sleeping or pining for the fjords (sorry, couldn't resist). While chromatic rebuffs some of Teds points I can't help but think something is missing - a little outside perspective.

chromatic rightfully points out that the project isn't dead (and has actually been pretty visible in the perl sphere since the start of the year) but look at it from more of an outsiders angle - unless you are already in the perl community it's not obviously moving. A release every month is lovely and all but the announcements are apparently not hitting all the right places if people like Ted (who seems to at least keep an eye on a number of different projects and technologies) haven't noticed. And if he's not seen them then the unwashed masses of other developers won't have either. Is this a problem? Depends on what you want from the project. Mindshare is a wonderful thing but introducing people to a technology before it's ready can destroy its chances of success.

PS I wanted to be glib and just say "motion != progress" but that seems unfair considering the amount of time people are investing.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Three Things - 1 of many

Tue, 15 Jan 2008 19:48:00 GMT

This weeks three things are -

MySQL 5.1(.20+) can log errors via syslog (finally)
IBM Blades run quite well despite being very wet. (don't ask)
Amazon Prime is too helpful. (Wooo individual book orders)

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Short Review - Hellboy Animated: Sword of Storms

Tue, 15 Jan 2008 19:36:00 GMT

I liked the original Hellboy, it had great casting, an OK story and Selma Blair being hot. Well, technically on fire, but we'll let that slip. I was expecting great thing from the first of the animated Hellboy films - special effects are great but you can go completely overboard with the monsters in an animated film.

Instead I spent a couple of hours watching Hellboy Sword of Storms plod along at a very slow pace. The story was dull, the animation was fine but nothing exceptional and the only highlight were the occasional witty one liners. Wait for it to hit TV

Score: 3/10 - bring on Hellboy 2

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Proof of Ownership and Third Party Escrow

Tue, 15 Jan 2008 19:28:00 GMT

I own a lot of old comics, piles of DVDs and a somewhat smaller (but still decent size stack) of audio CDs. These take up a lot of physical space, the comics decrease in quality, they all attract dust and are a pain to dig through when I want to find that one song on a compilation CD from 2002. Or was it 2001?

I have a lot of data - iso images and virtual machines are among the biggest disk eaters. A large percentage of it sits on a number of small external hard drives that are cheap, easy to upgrade (buy a bigger one, run an rsync and done) and simple to take offsite.

I'd love to use the latter to solve the former - I'd happily buy two 500GB drives (dirt cheap) rip redundant FLAC and MP3 copies of a number of my CDs (and scan / rip everything else) and then dispose of them to get a shelf back but how do I then prove everything is legit when something crops up and I have to display the fact I paid for them? Amazon orders? Credit card receipts? With the grossly inflated fines being levelled in the US I'd be bankrupted if I couldn't prove I did buy the soundtrack to Finding Nemo (although the fine might be less painful than having it in the public record that I did - it was a present, honest!)

So where are the companies offering indemnity, escrow and proof of legitimate ownership? Why can't I tick a "remember I bought this" button on Amazon and know that they've got my back? (as well as my VMs, online storage and book purchasing history?) Is this impossible to do? Is there no market for it? Where am I going to put that next order from Play when it turns up...

It could even be a way to recover things you've lost, they'd have the proof that you owned it and it'd cost them practically nothing to issue you another copy. As much as I love CSI, Buffy and Stargate I want my floor space back.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Short Review - I Am Legend

Tue, 08 Jan 2008 23:48:00 GMT

I've watched the original Omega Man, enjoyed the Kiwi perspective (named The Quiet Earth) and now I've seen Robert Neville Will Smith style - and it wasn't bad.

The plot is mostly unchanged (although explained through flashbacks), the pacing is decent and the feeling of being alone is well conveyed - the DVD store scene is a great glimpse of a man about to lose it.

The zombie/mutant hordes are a lot more visually impressive than those in the Omega Man (20 years of special effects and it shows) but their near mindless nature does change the tone and pace a bit. Score - 6/10

PS: there was also a version (The Last Man on Earth ) with Vincent Price which I'll be adding to my rental queue.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

O'Reilly Books and Odd Password Requirements

Tue, 08 Jan 2008 23:12:00 GMT

It's not that widely known but O'Reilly offer a user group discount - it's 35% off the cover price and free delivery so it's often cheaper than you can get the books new from anywhere else.

A few days ago I wanted to order a couple of books and because there are no conferences this month (and so no lovely Josette) I signed up online. The process itself was quick, easy and painless but one step stuck out in my mind - "Password cannot contain special characters or spaces". To me this is a weird limitation and one that seems to harken back to the days of old.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Bash Puzzles - Brace Expansion

Tue, 08 Jan 2008 21:34:00 GMT

There is nothing like other peoples code to highlight all those little gaps in your knowledge of a programming language. I know what the first one does:


$ mkdir -p {projectone_,projecttwo_,projectthree_}log

$ ls -1
projectone_log
projectthree_log
projecttwo_log

And I was a confident (and a little bit happy) about knowing what the second one does:


$ mkdir -p {project_one,}log

$ ls -1
log
project_onelog

But I had no clue about this one. And my guess (that it would be treated as a single element and expand to 'projectone_log') was very wrong:


$ mkdir -p {projectone_}log

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Three out of Three - New (contract) Job

Mon, 07 Jan 2008 20:57:00 GMT

Thanks to everyone who sent me leads and links to relevant job adverts but since I posted that I was out of work I've started a two month contract that began this week and runs until the end of Feb.

It's my first contract role (and it's not a typical one by any stretch) and it's taking a little time to get used to considering I've spent most of my working life as a permie. Still, it's got good people and pays the bills so I'm glad I accepted the offer.

It does mean that I won't be hitting LCA this year though, which is a shame because last years was excellent. On the plus side not spending money on the travel and flight means I can consider taking two/three months off and catching up on all those little things I keep putting off until I'm not quite as busy; a time that never arrives.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

The Empty Envelope

Fri, 04 Jan 2008 23:38:00 GMT

This is a weird one (and a bit of a long shot) but someone kindly sent me a letter this week, well I assume they did, as I only got an envelope.

If it was you then please drop me an email. I'm not ignoring, you I just don't know who you are or what you wanted. Not knowing is going to bug me now.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

YSlow - a Grade A Tool

Wed, 02 Jan 2008 22:52:00 GMT

I spent a couple of hours running the YSlow FireFox extension against the main website for one of my little side projects and I couldn't stop fiddling with the sites config until I got the score up. Improving a category until you get an 'A' gives you that same moment of satisfaction as all your tests passing or a file restore working perfectly.

Due to not being amazingly wealthy I cheated with the content delivery network stage and just overrode it with the sites own name.

I'd happily recommend this tool to anyone who wants to learn some good habits. With the exception of the CDN rule it's hard to argue with most of its suggestions. The performance grade is a nice way of giving near instant feedback and encouraging at least basic sensible behaviour.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Linux Journal - Offensive Adverts and the real Problem

Wed, 02 Jan 2008 16:40:00 GMT

Linux Journal is getting some coverage again, last time was an advert, this time it's a headline about Perl that Andy Lester didn't like and caused him to post that "The Linux Journal owes the open source community, especially the Perl community, a big apology." You can read the full complaint yourself over at use.perl 2.0 - sorry - Perl Buzz ;)

I like his post, despite the fact he's got a valid point the delivery irks me more than the underlying issue. It's a great example of person being indignant on behalf of a whole lot of people who don't care, and in many cases have no clue who he is or why he's claiming to speak for them. For some reason I keep wanting to type 'ESR' now... He could have posted in his own (actually very good) use.perl journal and stated that he found it unpleasant/disgusting and people would have nodded or disagreed with him. But no. Instead it has to be a big sweeping thing. I don't mean to pick on Andy but as a member of the Perl, Linux and open source communities I wanted to voice my counter opinion and say - LJ please publish what you want. I have a credit card and the ability to cancel my subscription if I don't like it. Oh, hang on, I did that over two years ago - which leads me to my other point.

Right - ignoring the whole moral outrage thing can we focus on the real issue please? Why does Linux Journal seem to be such a waste of paper these days? From 2000 to 2005 I used to be a devoted reader but over the last couple of years it's just not worth the time or money. I don't know if it's just the explosion of blogs and other decent technical content or just my reading habits changing but I find it hard to justify the cost (and actually finding a copy) just to get the hidden gems that are Reuven Lerner and Mick Bauer. To avoid sounding overly harsh I should probably point out that the articles haven't taken a huge dive in quality so much as so many clued, interesting people are now posting daily in indirect competition. Oh and the new editor was annoying as hell.

Going back to the original Perl headline incident the possible highlight of this for me (which I realise I probably shouldn't find childishly amusing) is that LJ just pulled the original story. No negotiation or endless discussion, they just pulled the whole thing.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Perl 5.10 - My Favourite Three Features

Tue, 01 Jan 2008 11:59:00 GMT

Since the release of Perl 5.10 (back on 2007/12/18) there have been a fair few articles discussing all the shiny new features - including smart matching, a built-in switch and state variables but my favourite three haven't really received much coverage. So I'll add to the pile of blog posts.

First up is a tiny (from the outside anyway) change that may have the biggest impact of all the new features on my day to day perl - the display of the actual name of uninitialized variables.


# older perls
$ perl584 -we 'print $foo, "\n";'
Use of uninitialized value in print at -e line 1.

# perl 5.10
$ perl510 -we 'print $foo, "\n";'
Use of uninitialized value $foo in print at -e line 1.

From the perspective of someone who has to spend the occasional afternoon reading Apache errorlogs I really like this one.

Now we move on to stackable file tests; something I was surprised perl couldn't do when I first noticed it was missing years ago -


# older perls
...
if (-s $file && -r _ && -x _) {
  print "$file isn't zero length and is +rx\n";
}
...

# perl 5.10
...
if (-s -r -x $file) {
  print "$file isn't zero length and is +rx\n";
}
...

Lastly on my little list is named captures - instead of referencing $1 and $2 etc. you can now assign them names at the point of capture and then pull the values out of a hash at a later time -


# requires 5.10 or above. But not 6.

my %date;
my $sample_date = '20071225';

if ( $sample_date =~ /(?\d{4})(?\d{2})(?\d{2})/ ) {
  %date = %+;
}

say "The year is $date{'year'}";

While none of these are massive attention grabbing additions like the powerful smart matching, switch statement or say (one of those is not like the others ;)) they help make the day-to-day stuff a little more pleasant.

Bonus feature -


my $x;
my $default = 'foo';

$x = 0;
$x ||= $default; 

say "\$x is $x";

$x = 0;
$x //= $default;

say "\$x is $x";

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!