Thu, 27 Aug 2009
@reboot - explaining simple cron magic
In a conversation with Stuart the subject of cron
timings came up, and after a brief discussion the ugly head of
@reboot reared. While most people know that you can use the special
'event' syntax to trigger cronjobs at specific times I'd guess a very
small number of them actually know how it works. For example does cron
rerun @reboot jobs when the service is restarted? (hint - no it
doesn't.)
After a quick discussion on how cron knew the machine hadn't really rebooted we had a short list on how it was doing it - tracking uptime, watching run level states, calling the init script only on certain levels... the only problem is that all of those had obvious issues that stopped them being a good choice. So I dug a little deeper.
First I needed a canary cronjob that would show me when @reboot was actually triggered successfully and a cronjob to run it -
$ vi /home/dwilson/log-cron
#!/bin/bash
logger "Cron ran me"
$ chmod a+rx /home/dwilson/log-cron
# and then the crontab
$ sudo vi /etc/cron.d/logme
@reboot dwilson /home/dwilson/log-cron
Once I had this in place I ran through the possible triggers, changing run levels, stopping and starting the script and changing the uptime were the big three - and none of them worked. In the logs were a number of 'Added a cronjob and got - (CRON) INFO (Skipping @reboot jobs -- not system startup) in syslog when I restarted.' lines instead.
After a quick run under strace I gave up under the sheer
weight of output and decided to look at the code. As my test machine was
Debian I added a source line for apt and pulled down the packages
source.
echo "deb-src http://ftp.uk.debian.org/debian etch main contrib non-free " > /etc/apt/sources.list.d/source.list
apt-get update
mkdir cron-src
cd cron-src
apt-get source cron
Now it was time to do some digging and get some line numbers to look at. In the cron directory I ran some greps to get an overview of possible code locations:
cd cron-3.0pl1/
grep -n -i reboot -r .
grep -i -r WHEN_REBOOT *
grep -n -i '@reboot' -r .
// ... snip ... //
cron.c:284:#define REBOOT_FILE "/var/run/crond.reboot"
cron.c:286: if (access(REBOOT_FILE, F_OK) == 0) {
cron.c:293: if ((rbfd = creat(REBOOT_FILE, S_IRUSR&S_IWUSR)) < 0) {
// ... snip ... //
# ls -alh /var/run/crond.reboot
---------- 1 root root 0 2008-11-07 11:07 /var/run/crond.reboot
Looking at the three interesting lines above we see how cron, on Debian at least, knows if it's been a real reboot. It uses the access function to check REBOOT_FILE. Nosing around a little more I also found the creat line and saw that the file had no permissions. The delving was nearly over but there was one thing I didn't get - how did the file get removed?
A quick look at the /var/run Filesystem Hierarchy Standard page cleared this up - 'Files under this directory must be cleared (removed or truncated as appropriate) at the beginning of the boot process.'. Which Debian does in /etc/init.d/bootclean Why is it done on system boot? So that if the system failed it still gets cleaned out.
With a much better idea how this should work, and just to double check, I stopped crond, deleted the /var/run/crond.reboot file by hand and turned crond back on. And my cronjob logged a little line. Not much feedback for all those commands but it was oddly worth it.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 21:52 | /operatingsystems/linux/debian | Permanent link to this entry | This entry and same date
Large uptimes - a wonderful problem to have
When it comes to the list of problems 'our uptimes are too high' isn't
normally in the top five that sysadmins dread.
While having a lengthy uptime used to be a boasting point it can also hide technical issues - such as kernel upgrades you've applied but not enabled (unless you're running something special like ksplice), confidence gaps in high availability systems (when was the last time you did a fail over?) and a general worry that what's running on a host now may not be when it comes back up.
The solution? Embrace the occasional controlled reboot and exercise those HA systems. After all, any machine that can't be rebooted without the customers noticing is a strong candidate for a single point of failure
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:48 | /sysadmin | Permanent link to this entry | This entry and same date
NW Rug - Capistrano Talks
I recently headed up to the August NWRug in
Manchester, firstly because it's been a while since I've seen Will Jessop, the organiser (and more
importantly a mate) and secondly because I was interested in Capistrano.
While we use puppet at work for the more strategic stuff, such as ensuring machines start off with a well-defined configuration, I've been in need of something to perform sets of tasks against defined groups of servers. While I've been using the time honoured pattern of wrapping our inventory commands in ssh loops I've started to have more complex needs. While we're not using Ruby or Rails Cap looks like it should be able to fulfil a lot of my need for immediate action (something that puppet isn't good at).
While the first talk of the evening was about the basics of Cap the second was much more interesting to me. It was an advanced talk given by (I think) the current maintainer. He did an excellent job of explaining some of the less common features and how they related to the Unix philosophy - such as streams and pipes. The speaker obviously knew his stuff and it's given me the push I need to hopefully spend some time in the future looking at the software. If his slides go up they are well worth a read.
The group were a friendly bunch, the venue was nice and the talks educational. If it wasn't for the train cost and the time spent travelling I'd be more than willing to attend more of their meetings.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:38 | /events | Permanent link to this entry | This entry and same date
Moon - Short Review
If you've not seen or read much in the way of Scifi then Moon may be an innovative movie that surprises you with its
plot twists, (a film with a plot? Quick, change screens, Transformers is on
next door) surprises and human/machine interaction. If you're reading this
blog then I'm guessing you'll find it slow, predictable and a bit
meandering.
5/10. Very average.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/08/27 18:09 | /movies | Permanent link to this entry | This entry and same date
Copyright © 2000-2010 Dean Wilson
