Mon, 31 Aug 2009
Look for the people that everyone mocks
Pay very close attention to them. Nothing marks a trouble spot in quite
the same way. Either they're being picked on and something needs to be done
or they are the first people you should be helping to pursue new
opportunities. Very far away. Either way you need to know.
Leaving incompetent people in place destroys morale for the good staff and encourages them to look elsewhere. When there's a bad apple or two lowering the teams value, and the quality of their output, pure professional pride only gets you so far. At some point you need to have a culture that rewards effort and punishes wastes of space.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/31 20:58 | /nottech | Permanent link to this entry | This entry and same date
Cloud Application Architectures - Short Review
With all the hype and misdirection around the cloud it's always good to
find a little bit of concrete information. If you're interested in the
general principles of how the cloud (and Amazon Webservices in
particular) could replace some of your existing infrastructure then Cloud Application
Architectures isn't a bad place to start.
The book is a slim tome, it's easy to read in a couple of sittings and covers all the basics. The author felt more than a little biased towards the cloud (IMHO) but what do you expect from someone leading the push? The book is well written, clear in making its points and the worst omission / error are a couple of missing figures, but the text works fine without them.
I'd be remiss if I didn't point out the Berkley Cloud paper (PDF), a great paper for defining terms regarding the cloud, and any of the keynotes by Simon Wardley (especially the one from FrOScon 2009) as other good starting points - both of which are free.
If you're interested in how the cloud could play a part in your environment and want something a little more concrete (and AWS focused) then this book is for you. 7/10.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/31 20:26 | /books | Permanent link to this entry | This entry and same date
Sun, 30 Aug 2009
Project California: a Data Center Virtualization Server - Short Review
When it comes to sysadmin buzzwords Project California: a Data
Center Virtualization Server ticks a lot of the boxes, which is a
little misleading as half the book is about solid hardware level details
that are actually rarely covered.
While this makes the first half more than a little dry it does introduce concepts that many of us take for granted, such as why DDR3 is faster than DDR2. The second half takes you through the Cisco UCS stack and where the benefits are. It's a good starting point but I'd hoped for some more meat, maybe even a case-study or two. The book answered some of my questions but it's not amazingly comprehensive so expect to do a lot more digging after you've finished reading it.
The book is self-published (Via Lulu) so it's probably worth mentioning the quality - it's the same as any other book I've bought recently. No better and no worse, which is actually pretty impressive.
6/10 - dry, fills a niche but covers a lot of general material not specific to UCS.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/30 11:05 | /books | Permanent link to this entry | This entry and same date
FrOSCon 2009
Last weekend I joined the hordes and worked my way from London to Seigburg
for FrOSCon 2009. Along the way I
experienced an airport evacuation due to fire alarm, a delayed flight,
four trains (one in the wrong direction) and numerous kindly old
German ladies that took pity on me and gradually got me in the right
direction. And it was worth every second.
I'd never really considered going to FrOSCon before, the percentage of talks given in German is quite high and I don't speak a word of the language but this year there was an excellent line up of speakers (all presenting in English) in the OpenSQLCamp room. As I'm getting more and more MySQL requests at work I thought it'd be a handy thing to go along and learn from.
In general it reminded me of the earlier FOSDEMs in tone, especially with the selection of project based developer rooms. The only annoyance was that so many sysadmin related talks, with excellent attention grabbing titles, were in German only. I plan on working my way through the slides in the hope that I can get at least a basic idea of their subject matter.
A large percentage of the sessions were recorded and I'm looking forward to catching up on the sessions where I was already booked or engaging in the hallway track. The only annoyance was that so many sysadmin related talks, with excellent titles, were in German only. I plan on working my way through the slides in the hope that I can get at least a basic idea of their subject matter.
Speaker highlights for me were Dag Wieers and Simon Wardley, both are very entertaining, highly practised speakers that grab the audience and don't let go. I especially like watching the audience in swardleys cloud computing talks as they gradually come to see how it's going to impact all of us eventually. If he was paired with someone in the mobile space with the same kind of eloquence we'd have a nice road map of the tech future.
Although the English language talks were a little thin on the ground during the second day the organisers did an excellent job (although the newbies like me could have done a few pointers about things like the BBQ, do we buy tickets? Pay at the counter?) and with a little luck I'll have the money to go back again next year.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/30 11:00 | /events | Permanent link to this entry | This entry and same date
Fri, 28 Aug 2009
Find and replace interview question
We've recently been searching for a junior sysadmin to join the team (and
I'm very happy to say we've now succeeded) so as part of my day to day
tasks I had to come up with a dozen simple questions to weed out the people
that have never used anything but webmin (and there is a surprising number
of them out there). One of the questions seemed to cause a lot of trouble
in the general sense and tripped up the few who even made an attempt -
"How would you change all occurrences of 10.23.34.10 to 10.23.34.101 in a text file?
While most of the candidates failed this one on account of skipping the question we had a couple make attempts (oddly all using the vim replace syntax). While I was hoping for a little bit of sed or perl a solution's a solution. Unfortunately theirs all had edge cases.
First was the obvious trick in the question. All the answers (apart from
one) came back with something like this -
%s/10.23.34.10/10.23.34.101/. First up we have the missing
boundary test. While that regex is fine for 10.23.34.10 it breaks quite
badly on 10.23.34.101 (it replaces it with 10.23.34.1011). Secondly the
dots are unescaped, which means it's not only going to work on IPs, but
number sequences.
I was actually really surprised at how few of the admins could write an awk / perl / shell script and how many paid very little attention to what the regex actually matched. Still, while the question seemed easy enough it had enough awkwardness to enable further discussion about regexs, data checking and the evil of manual changes. And it showed that everybody likes vim :)
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/28 21:54 | /sysadmin | Permanent link to this entry | This entry and same date
Thu, 27 Aug 2009
@reboot - explaining simple cron magic
In a conversation with Stuart the subject of cron
timings came up, and after a brief discussion the ugly head of
@reboot reared. While most people know that you can use the special
'event' syntax to trigger cronjobs at specific times I'd guess a very
small number of them actually know how it works. For example does cron
rerun @reboot jobs when the service is restarted? (hint - no it
doesn't.)
After a quick discussion on how cron knew the machine hadn't really rebooted we had a short list on how it was doing it - tracking uptime, watching run level states, calling the init script only on certain levels... the only problem is that all of those had obvious issues that stopped them being a good choice. So I dug a little deeper.
First I needed a canary cronjob that would show me when @reboot was actually triggered successfully and a cronjob to run it -
$ vi /home/dwilson/log-cron
#!/bin/bash
logger "Cron ran me"
$ chmod a+rx /home/dwilson/log-cron
# and then the crontab
$ sudo vi /etc/cron.d/logme
@reboot dwilson /home/dwilson/log-cron
Once I had this in place I ran through the possible triggers, changing run levels, stopping and starting the script and changing the uptime were the big three - and none of them worked. In the logs were a number of 'Added a cronjob and got - (CRON) INFO (Skipping @reboot jobs -- not system startup) in syslog when I restarted.' lines instead.
After a quick run under strace I gave up under the sheer
weight of output and decided to look at the code. As my test machine was
Debian I added a source line for apt and pulled down the packages
source.
echo "deb-src http://ftp.uk.debian.org/debian etch main contrib non-free " > /etc/apt/sources.list.d/source.list
apt-get update
mkdir cron-src
cd cron-src
apt-get source cron
Now it was time to do some digging and get some line numbers to look at. In the cron directory I ran some greps to get an overview of possible code locations:
cd cron-3.0pl1/
grep -n -i reboot -r .
grep -i -r WHEN_REBOOT *
grep -n -i '@reboot' -r .
// ... snip ... //
cron.c:284:#define REBOOT_FILE "/var/run/crond.reboot"
cron.c:286: if (access(REBOOT_FILE, F_OK) == 0) {
cron.c:293: if ((rbfd = creat(REBOOT_FILE, S_IRUSR&S_IWUSR)) < 0) {
// ... snip ... //
# ls -alh /var/run/crond.reboot
---------- 1 root root 0 2008-11-07 11:07 /var/run/crond.reboot
Looking at the three interesting lines above we see how cron, on Debian at least, knows if it's been a real reboot. It uses the access function to check REBOOT_FILE. Nosing around a little more I also found the creat line and saw that the file had no permissions. The delving was nearly over but there was one thing I didn't get - how did the file get removed?
A quick look at the /var/run Filesystem Hierarchy Standard page cleared this up - 'Files under this directory must be cleared (removed or truncated as appropriate) at the beginning of the boot process.'. Which Debian does in /etc/init.d/bootclean Why is it done on system boot? So that if the system failed it still gets cleaned out.
With a much better idea how this should work, and just to double check, I stopped crond, deleted the /var/run/crond.reboot file by hand and turned crond back on. And my cronjob logged a little line. Not much feedback for all those commands but it was oddly worth it.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 21:52 | /operatingsystems/linux/debian | Permanent link to this entry | This entry and same date
Large uptimes - a wonderful problem to have
When it comes to the list of problems 'our uptimes are too high' isn't
normally in the top five that sysadmins dread.
While having a lengthy uptime used to be a boasting point it can also hide technical issues - such as kernel upgrades you've applied but not enabled (unless you're running something special like ksplice), confidence gaps in high availability systems (when was the last time you did a fail over?) and a general worry that what's running on a host now may not be when it comes back up.
The solution? Embrace the occasional controlled reboot and exercise those HA systems. After all, any machine that can't be rebooted without the customers noticing is a strong candidate for a single point of failure
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:48 | /sysadmin | Permanent link to this entry | This entry and same date
NW Rug - Capistrano Talks
I recently headed up to the August NWRug in
Manchester, firstly because it's been a while since I've seen Will Jessop, the organiser (and more
importantly a mate) and secondly because I was interested in Capistrano.
While we use puppet at work for the more strategic stuff, such as ensuring machines start off with a well-defined configuration, I've been in need of something to perform sets of tasks against defined groups of servers. While I've been using the time honoured pattern of wrapping our inventory commands in ssh loops I've started to have more complex needs. While we're not using Ruby or Rails Cap looks like it should be able to fulfil a lot of my need for immediate action (something that puppet isn't good at).
While the first talk of the evening was about the basics of Cap the second was much more interesting to me. It was an advanced talk given by (I think) the current maintainer. He did an excellent job of explaining some of the less common features and how they related to the Unix philosophy - such as streams and pipes. The speaker obviously knew his stuff and it's given me the push I need to hopefully spend some time in the future looking at the software. If his slides go up they are well worth a read.
The group were a friendly bunch, the venue was nice and the talks educational. If it wasn't for the train cost and the time spent travelling I'd be more than willing to attend more of their meetings.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:38 | /events | Permanent link to this entry | This entry and same date
Moon - Short Review
If you've not seen or read much in the way of Scifi then Moon may be an innovative movie that surprises you with its
plot twists, (a film with a plot? Quick, change screens, Transformers is on
next door) surprises and human/machine interaction. If you're reading this
blog then I'm guessing you'll find it slow, predictable and a bit
meandering.
5/10. Very average.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:09 | /movies | Permanent link to this entry | This entry and same date
Wed, 19 Aug 2009
Testing A Production DNS Re-point
We recently consolidated a number of websites used by one of our brands
back down to a sensible number (sensible being one). Which, while only a
single action point on an email, turned out to be a large amount of DNS
and apache vhost wrangling. In order to give myself a safety net, and
an easy to check todo list, I decided to invest ten minutes in
writing a small test script.
Despite all my best intentions and experimenting with testing DNS with Cucumber and RSpec when the issue came up for real, and on a short deadline, I fell back to old habits and reached for perl. Net::DNS is an excellently documented module and the little utility modules like File::Slurp and URI::Title are perfect for quick tasks like this. The full test_dns_repoints.pl code, which is only 47 lines is something I can see myself using again and again as we repeat this kind of work.
It's amazing how much a little effort can gain you when it comes to testing your infrastructure. Having that safety net, especially when you're a one man team, is a very reassuring feeling and it's one I'm trying to introduce in to more areas of even one off jobs.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/19 07:44 | /sysadmin | Permanent link to this entry | This entry and same date
Copyright © 2000-2010 Dean Wilson
