Mon, 23 Aug 2010
Adventures in Cronologger
Cronjobs are one of those necessary evils of any decent sized Unix setup,
they provide often essential pieces of a sites data flows but are often
treated as second class citizens. While I've already mentioned my Cron
commandments I'm always looking for improvements in the
rest of my cron tool set and, with Vladimir Vuksan's cronologger, I may have
found another piece of the puzzle.
The concept is simple, you add a command to the front of your crontabs and it invokes your actual cron command. This wrapper script collects the stdout, stderr and some other details such as exit code and run time. The backend is a couchdb data store and the simple reporting pages are written in PHP, and are easy to work through, crib and base your own reports from. Having all this cron information also helps provide a talking point with development, it's easy to show progress and imbue a sense of actually getting somewhere when the number of cronjobs with errors drops each day, rather than the systems team mentioning that their email boxes are a little emptier since the last release.
While our initial tests seem positive there are a couple of reports and tweaks to the command line data injector that we want for our local usage. The biggest problem with the project may well be that the idea is so obviously correct that we end up re-implementing it in something a little more suitable for our environment. Maybe a Python command line client and Perl Template Toolkit driven reports to replace the PHP. But that's a possibility for later - for now cronologger is a great 80% solver.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/08/23 21:49 | /tools | Permanent link to this entry | This entry and same date
Sun, 06 Jun 2010
Netbeans vs Commandline
The last time we interviewed for Java developers (a couple of jobs
ago) it came as quite a surprise at how few of them could function
without their IDE of choice. A high percentage of the candidates
struggled to compile using javac, had problems navigating the docs and
made a large number of very simple syntax errors that they were obviously
used to their editor dealing with.
At the time the more unix focused team, most of who were very long term vim and emacs users, had a number of discussions about how this should impact our rating of the candidates. One school of thought was that people should use the tools that make them most productive. The other was that people should understand their tool chain. How can you diagnose issues on a production server if you can't even compile a class on the command line? You can tell which side I was on.
I've recently joined a small Java project and after some awkward fiddling around with ant, junit and half a dozen other jars decided to give Netbeans a chance. I was pleasantly surprised at how quickly and easily I got the same project up and running in the IDE. I don't yet have a clue how it's storing the files on disk, constructs the build or test targets and a dozen other little details but at this stage in my basic use of Java it doesn't seem to matter.
It's strange how quickly seductive all the optional extras can be and how easy it is to lose track of what you don't know while adapting to the features they offer. I'm not sure how much of it is better tooling, benefits of a strongly typed static language or just having a dedicated team behind producing a consistent development environment but it felt very easy to take baby steps with. And I'm hoping the tool continues to show me more power as my needs when using it grow.
While I'm at no risk of giving up vim for my day to day work I think I'll be investing some time in to learning one of the big three Java editors (Eclipse, Netbeans or IntelliJ) for while I'm away in the strange world.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/06/06 12:11 | /tools | Permanent link to this entry | This entry and same date
Wed, 07 Apr 2010
Pigz - Shortening backup times with parallel gzip
While searching for a completely different piece of software I stumbled
on to the pigz application, a
parallel implementation of gzip for modern multi-processor, multi-core
machines. As some of our backups have a gzip step to conserve
some space I decided to see if pigz could be useful in speeding them up.
Using remarkably unscientific means (I just wanted to know if it's worth further investigation) I ran a couple of sample compression runs. The machine is a quad core Dell server, the files are three copies of the same 899M SQL dump and the machine is lightly loaded (and mostly in disk IO).
####################################### # Timings for two normal gzip runs dwilson@pigztester:~/pgzip/pigz-2.1.6$ time gzip 1 2 3 real 2m43.429s user 2m39.446s sys 0m3.988s real 2m43.403s user 2m39.582s sys 0m3.808s ####################################### # Timings for three pigz runs dwilson@pigztester:~/pgzip/pigz-2.1.6$ time ./pigz 1 2 3 real 0m46.504s user 2m56.015s sys 0m4.116s real 0m46.976s user 2m55.983s sys 0m4.292s real 0m47.402s user 2m55.695s sys 0m4.256s
Quite an impressive speed up considering all I did was run a slightly different command. The post compression sizes are pretty much the same (258M when compressed by gzip and 257M with pigz) and you can gunzip a pigz'd file, and get back a file with the same md5sum.
# before compression -rw-r--r-- 1 dwilson dwilson 899M 2010-04-06 22:12 1 # post gzip compress -rw-r--r-- 1 dwilson dwilson 258M 2010-04-06 22:12 1.gz # post pigz compress -rw-r--r-- 1 dwilson dwilson 257M 2010-04-06 22:12 1.gzs
I'll need to do some more testing, and compare the systems performance to a normal run while the compression is happening, before I trust it in production but the speed ups look appealing and, as it's Mark Adler code, it looks like it might be an easy win in some of our scripts.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2010/04/07 08:00 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 30 Sep 2009
Rake - surprisingly enjoyable
I've never really liked make files, I don't think I've ever had to write
enough C to really appreciate (or just tolerate) them, so I was a little
dismissive of Rake - and I was
mostly wrong.
Now we're adding a new member to the systems team I've been doing a lot
of thinking about our tool chain - what knowledge assumptions it makes,
which parts are still more manual than I'd like and where the tool chain has
gaps (this is the most annoying one for me) and rake seemed like a
potential addition to encode some of that process knowledge in to a tool.
I've only added little rakefiles here and there but they do make certain
tasks nicer (plus I like the inline descs).
I've not yet worked out any general rules for when to use a shell script and when to use rake but if nothing else it's helping me spend some time on my ruby skills. The best rake starting points I found were Martin Fowlers rake article and the rake release notes.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/30 21:48 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 23 Sep 2009
Simplifying File Permissions in Puppet Manifests
I've been a user of Puppet for about three years now and
while on a recent dig in to some of my older classes it was a little
embarrassing to see lots of file types used like this:
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
owner => whi,
group => whi,
mode => 644
}
Luckily as we get more experienced with a tool we can often go back and
improve on the first steps. By using an explicit File { settings
} inside a class you can assign a sensible set of defaults to all
the instances of the same type that lack overriding settings. So we can
shorten the previous example to -
File {
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
}
While this isn't a huge win in raw characters typed (although in longer manifests they start to mount up) it does move all the common settings in to a single location (keeping us clear of DRY violations) and it leaves only the differences between file type definitions.
You can also apply those kind of settings (such as Exec { path =>
"path:list" } at the server level by including them in a top level
file and then overriding them as needed in each module. If you do this then
you need to be aware that any declared type that doesn't override it gets
the global setting, which can lead to the odd action from afar head
scratching.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/23 22:16 | /tools/puppet | Permanent link to this entry | This entry and same date
Thu, 17 Sep 2009
Stand Alone Puppet
While Puppet can be used to manage large, complex
environments it's also a useful tool at the lower end of the spectrum.
Using just the puppet executable and a small inline class or
two you can write very useful manifests in only a handful of lines.
class build-host {
package { "build-essential": ensure => installed }
package { "subversion": ensure => installed }
file { "/home/dwilson/repos/":
ensure => directory,
owner => dwilson,
group => dwilson,
}
}
node default {
include build-host
}
To invoke the class you just run puppet -v build-host.pp.
It's also worth pointing out the node name of default. This saves you
manually changing the manifest whenever you move to another machine.
While it wouldn't be hard to replace the above example with a shell
script, by using puppet you can easily access the built-in abstractions
(which package manager to use, how should you add users) and remove a lot
of scaffolding code. And then when you're done you can promote the class to
your managed infrastructure.
I've used this to bootstrap provisioning servers (why should the provisioning host be the only machine that wasn't provisioned?), test small but annoying new classes on scratch servers and I'm currently working on integrating it with a small subversion backup testing project in my spare time at work (so very slowly).
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/17 21:33 | /tools/puppet | Permanent link to this entry | This entry and same date
Wed, 01 Jul 2009
dstat - a window to your system
When it comes to Unix diagnostics I was raised the old fashion way, with
iostat, vmstat and similar tools. However times change and tools evolve.
dstat, while not as
comprehensive as using all the tools one by one, provides a wide range
of system performance details in an easy to use package.
While it's useful enough in its default state there is even more
functionality lurking just below the surface. To see which other modules
are available (but are not enabled by default) run dstat -M
list. To add an extra module to the output use a command like
this one: dstat -a -M topmem -M topcpu
As part of my growing use of the tool I've started to write my own little dstat plugins. I was pleasantly surprised at how easy they were to write and deploy even with my basic python skills. While the memcached plugin was a proof of concept I've not needed much I've found the process count plugin to be very handy.
dstat is becoming one of the overview tools I use when investigating performance issues and it's worthy of a place in your toolbox too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/01 21:32 | /tools/commandline | Permanent link to this entry | This entry and same date
Mon, 09 Mar 2009
Puppet Scripts - extract-report-issues
I spent a little while digging through the default puppet log types the
other day and after reading through a batch of activity logs I whipped up
extract-report-issues,
a script that can be run on the command line (or daily via cron) and
displays a list of errors and warnings from the specified glob of hosts and
log files. By default it does all hosts for the current day, we've got it
running nightly so we can work through the issues each morning. It's worth
noting that sometimes in the output the same failure occurs more than
once. This is because puppet retries certain operations - such as retrieving
a resource.
There is actually a lot of useful information in the puppet reports. To start with I've added a todo item for a script that notes persistent errors (the same issues over two or three runs) that I'll hopefully get to this month. Maybe.
If you're running puppet in production you owe it to yourself to turn on reporting and set up some processes around it. While puppet makes it easy to perform action at a distance you still need to close the loop somehow.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 20:57 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 04 Mar 2009
First Steps in Github
I finally decided to set up and start using a github account and my early
impressions are that it's quite slick and very userfriendly.
Apart from an annoyance where I couldn't see my pushes for a little while (I think I fell afoul of some caching) setting up an account and adding new repos was simple. Pushing from my actual dev machine just worked and I've now been bitten by the github bug. I feel the need to push code...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/04 22:22 | /tools/online | Permanent link to this entry | This entry and same date
Mon, 02 Mar 2009
Mozilla Ubiquity - Puppet Types
I like Ubiquity. It
puts a lot of the sites I used on a regular basis close to hand without
making me dig through my bookmarks (or del.icio.us account). In a
small burst of productivity, and to avoid real work, I decided to put a
command together for the Puppet Type docs at Reductive Labs.
If you have the Ubiquity plugin installed you should be able to install a copy of the command from the Ubiquity Puppet Types Command page. Writing a simple command like this one was surprisingly easy (especially using command-editor) although I don't (yet) know enough jQuery to add some of the other features I want.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/02 22:55 | /tools/firefox | Permanent link to this entry | This entry and same date
Tue, 03 Feb 2009
Simple, Single Document Bookmarks in vim
I like vim, I think it's a great editor
worth investing time and effort in to learning but I also think it's one
of the most horrible things to watch an inexperienced user typo his way
through while you're urgently waiting for them to finish the damn edit.
My favourite one this week (and it's only Tuesday) is looking for
probably unique phrases that you can later search for to return to a
specific part of a document.
In an attempt to stop my laptop getting any more back of the head shaped
dents in it from when I've failed to restrain myself I thought I should
point out a much simpler way of doing this. Once you're at the part of a
document you want to return to press m<letter>. This
sets a mark. To return to it press '<letter>. That's it.
No more pasting in chunks of a string hoping it only occurs once in the
damn document. If you need to mark a couple of locations then fine just use
different letters to set and return to the places you want. And save me
sending another laptop back in for warranty.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/03 22:05 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 14 Jan 2009
Soon to be With Added Git?
Despite setting up my own
gitweb
install I'm still not using git regularly enough to be comfortable with it
so today I went through the Peepcode Press Git Internals book/PDF. While
the diagrams and details of what happens under the cover are useful it's
the wrong level for me as a basic user. To ease myself in to the move
from subversion for some of my personal projects I found Git Magic to be more
useful.
I know git requires a mental shift and it's a very complex and powerful tool but for my own needs I'll probably never use more than 10% of its capabilities. Unfortunately most of the projects I use and need to submit patches to have switched - so I'll be a happy sheep and go along for the ride. Even if it turns out to be a roller coaster.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/14 18:14 | /tools/commandline | Permanent link to this entry | This entry and same date
Tue, 06 Jan 2009
Diffing Files Over Multiple Servers - rd-differ
Adhoc changes are a very bad thing in many ways, one of the worst is how
often they are not fully implemented across all the servers or even
pulled back to staging. In an attempt to sanity check the config files when
we have to make these little hacks I oddly-proudly present - rd-differ. A tool for
diffing config files over multiple machines.
The idea is simple, you tell it the file or directory you're interested
in, specify a single machine as the baseline and then specify a number of
others as the machines to check against it. A sample invocation looks like this
rd-differ /etc/apache2 10.10.100.111 10.10.100.112
10.10.100.113 and the output is show as a diff.
The files are rsynced down using ssh so your usual keys will work and while the normal output is that of the raw diff it's very easy to wrap the results and add other checks on top of it. The shell's not written to be very defensive (unusual for me) but the code is short enough that it's worth the compromise.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/06 18:26 | /tools/commandline | Permanent link to this entry | This entry and same date
Sat, 08 Nov 2008
Rebooting Via Proc and the magic sysreq key
You know what the best way to start the day is? I'm pretty sure that it
doesn't include a production web server putting its file systems in to
read only mode. When this happens most local commands don't work - init,
shutdown, telnit and reboot all stop being useful and you have to resort
to desperate measures... and here's the desperate
measure of the day.
First, check that your system supports the magic sysreq key -
$ cat /proc/sys/kernel/sysrq
1 # nonzero is good
Now you know you have the power to destroy your system through a single incorrect character, have a look at the Redhat Sysrq command reference (you want the 'sysrq' section). We tried to make it sync the disks and reboot - your requirements may vary.
root@web02:~# echo s > /proc/sysrq-trigger
root@web02:~# echo b > /proc/sysrq-trigger
# machine reboots
As techniques go this one's a little obscure but it's very useful in the right circumstances.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/11/08 12:25 | /tools/commandline | Permanent link to this entry | This entry and same date
Thu, 04 Sep 2008
Ubiquity - More Than Just Shiny Chrome
While Google Chrome has been getting all the press coverage recently Ubiquity, from Mozilla Labs, is where all the interesting action seems to be happening.
Ubiquity ticks all the boxes for me, it's a simple, easy to use idea, that'll save me time. It's easily extensible and already has a huge community of people working, enhancing and just trying new things with it. All the things I've come to expect from Firefox and the Mozilla using community.
I personally think this is an important distinction to make - while Google Chrome is a new browser with some great ideas (and a quickly revised EULA) FireFox is a proven, Free platform that encourages extension and has a track record of doing the right thing.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/09/04 19:58 | /tools/firefox | Permanent link to this entry | This entry and same date
Sat, 23 Aug 2008
Nagios Service and Hosts stats - Graphed in Munin
We've been hitting some load issues on one of our monitoring
machines recently and while it looks like the munin graph generation
is the culprit we also decided to keep an eye on how many services and
hosts Nagios was checking.
One of the downsides of having a very automated server deployment system is how easy it is to suddenly find yourself with an extra dozen hosts you no longer really need. While each check is quite small and quick, add up the frequent runs and multiply it by a reasonable number of servers and you can soon hit problems.
So as a first step towards keeping an eye on those numbers we now have a munin Nagios hosts plugin and a munin Nagios services plugin that show the total number of hosts and services monitored and the states those resources are in.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:20 | /tools/commandline | Permanent link to this entry | This entry and same date
Nagios Checks - Validate HTML and Validate Feed
As part of my ongoing attempt to stop myself from silently making
mistakes (I don't so much mind the ones I notice) I've added another
couple of Nagios
Plugins. This time validate_feed
and validate_html.
As both of these checks call out to an external, third party resource, if you use them be sure to tweak your Nagios polling interval down to a respectful level.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:11 | /tools/commandline | Permanent link to this entry | This entry and same date
Thu, 14 Aug 2008
Filter syslog logs with syslogslicer
While digging through a pile of syslog log files recently I needed
something a little more data format aware than pure grep. So I present the
first version of syslogslicer
- a simple perl script that knows a little bit about the syslog log file
format.
# some example command lines
syslogslicer -p cron -f program,message /var/log/syslog
# print the program and message for all lines with program 'cron'
syslogslicer -p cron -m hourly /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
syslogslicer -p cron -m hourly -s 20080810100000 -e 20080810123000 /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
# between 20080810100000 and 20080810123000
syslogslicer allows you to filter the output by matching text in the program or log message, only print certain output fields and do basic time based filtering. If you've ever wanted to see all the logs raised by postfix with the word 'database' in them between 10 and 11 am then this might be the tool for you.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 12:28 | /tools/commandline | Permanent link to this entry | This entry and same date
Nagios - Check Proxy Check
"This script retrieves a URL via a specified proxy server and
alerts (using the standard Nagios conventions) if the request
fails."
We're running a couple of services through a proxy server for a number of good, and to be honest a couple of not so good but mandated, reasons. The Check Proxy Check Nagios Plugin ensures that if the proxy goes down in a way that stops us pulling pages through it we know.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 09:30 | /tools/commandline | Permanent link to this entry | This entry and same date
Wed, 13 Aug 2008
Nagios Disk Check - Mountpoint or Filesystem?
If you mount filesystems under a specific mount point, and monitor
them with Nagios, then be sure
you understand what happens if the underlying file system goes away.
With:
/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -p /a_mount_point
you'll get the value from the containing file system. In this case
/. If you'd rather know that your chosen mount point has
actually gone away, and that you're no longer checking what you thought
you were, then add the -E option to the command. This will
turn on exact path matching and catch that kind of error.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/13 21:54 | /tools/commandline | Permanent link to this entry | This entry and same date
Copyright © 2000-2010 Dean Wilson
