take that cfengine

We've had a long running problem with cfengine at glasgow - 2.2.3 (the latest DAG) didn't expand out HostRange properly on the non-workernodes (ie where we need it most - disksvr, gridsvr, natbox groups). today I spent far too long battling with both 2.2.8 and the latest svn release (don't go there - its far too fussy about the exact release of aclocal you use) and neither of them worked properly.

I finally got a minature testcase configuration file to work, then got *really* confused when I used our live config as a testcase file sucessfully, but not the normal incantation.

it turned out to be the fact we'd defined

domain = ( beowulf.cluster )
in update.conf

however, setting this broke the way cfengine handles FQDNs on the dual-homed nodes (which are gla.scotgrid.ac.uk and beowulf.cluster). Commented it out leaving cfengine to guess the right thing to do, and it all seems OK.

I have since upgraded uniformly to 2.2.3 across all the SL4 x86_64 machines and tested OK.

While doing this I noticed we hadn't defined the WMS as a mysqld node so we weren't monitoring it in nagios or backing up the database. Oops. Sorted.

SAM Failures across scotgrid: Someone else's problem

All 3 scotgrid sites have just failed the atlas SAM SE tests (atlas_cr, atlas_cp, atlas_del) as have quite alot of the rest of the UKI-* sites.

Once again this isn't a Tier-2 issue but an upstream problem with the tests themselves


ATLAS specific test launched from monb003.cern.ch
Checking if a file can be copied and registered to svr018.gla.scotgrid.ac.uk

   ------------------------- NEW ----------------
srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/
+ lcg-cr -v --vo atlas file:/home/samatlas/.same/SE/testFile.txt -l lfn:SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438 -d srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
Using grid catalog type: lfc
Using grid catalog : lfc0448.gridpp.rl.ac.uk
Using LFN : /grid/atlas/dq2/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
[BDII] sam-bdii.cern.ch:2170: Can't contact LDAP server
lcg_cr: Host is down
+ out_error=1
+ set +x
   -------------------- Other endpoint same host -----------

Nagios, nagios, where are you?

Nagios seems not to be sending us alarms properly on SAM test failures (we had 2 yesterday - one for an internal SAM problem and the other for an SRMv1 timeout issue).

Andrew is investigating.

Alert! Alert!

'twas the night before holidays, when all through the servers not a pager was stirring...

Hmm lulled into a false sense of security by the appearance of [WLCG Nagios] entitled emails alerting me about proxy expiry on the shared nagios system I foolishly thought all was well. However we wern't getting any 'real' alerts from the system to the individual sites.

Turned out to be a configuration issue in the /etc/nagios/uki-scotgrid-*/contacts.cfg

We had

    service_notification_options    n
    host_notification_options       n

meaning no notifications were sent - changed this to

    service_notification_options    w,u,c,r,f
    host_notification_options       d,u,r,f,s

which means we get alerted on pretty much every state change - for more details see the manual

on a more annoying note - I left my macbook PSU back in the UK and there's a limited no of apple resellers here :-(

Holy lactating keyboards Batman, there goes the disk!

There's a pre-amble to this post which isn't really to do with ScotGrid directly, but has some bearing subsequent event. If I tell you the actors were a glass of milk, a three year old and my laptop keyboard then you can doubtless assemble the plot yourself. So yesterday some important keys were completely non-functional on my laptop - one of these keys was a character in my password so I wasn't able to login at all. Despite my best MacBook disassembling and cleaning efforts I couldn't recover the keyboard - there seems to be gunk etched onto the conducting keyboard sheet. Now that I am back in the office I can use a USB keyboard, but I need to get a new MacBook one quickly.

Anyway, upshot is that my online activities were squeezed into an ancient IBM R32 laptop running XP with 256MB of RAM. So, I was not watching (bookmarks were effectively unavaliable) when the CE ran out of disk space and we went belly up for SAM tests yesterday lunchtime.

Problem was easily identifed this morning and, thanks to the magic of LVM, corrected. When the machines were setup /var was not separate from /, so we've now added a separate /var logical volume with 30GB in it.

We looked into why this did not trigger and alarm and it seems that there was a mistake in the nagios configuration which was looking at free inodes, rather than free space. Corrected now.

I also took the opportinutity to re-run YAIM on the CE and fix the broken VOMS mappings. My atlas production role now maps correctly to an prdatlas account. This might cause a failure to clear from globus the stack of jobs running under my "local" gla012 account, but the actual production system doesn't require any outputs from globus so the real payload will run fine.

Sadly, the segfaulting on the CE still happens, however it seems the the problem was spotted earlier and that a fix is on the way: https://savannah.cern.ch/bugs/?35981.

Oh no, not again...

We went though a little rash of SAM test failures last night. This turned out to be an LHCb user who was submitting jobs which filled up the scratch area on the worker nodes and turned them into blackholes.

Obligatory GGUS ticket was raised.

We do alarm against disk space filling up on the worker nodes, but it was still 4 hours before action was taken and the nodes set offline before being cleaned. In that time an awful lot of jobs were destroyed. Make me think we might want to automate the offlining of nodes which run out of disk space, pending investigations.

nagios event handlers

I've gone over to the Dark Side (no, not Python) and have just implemented my first nagios event handler - This *should* automatically try and fix the problem we have with our Dirvish backup scrips - namely that we end up with too many copies of the database dumps held.

So - cue nagios' event handlers - the only issue should be that nagios (and the event handler) runs as user nagios and most sysadmin stuff needs root. If you're willing to trust it to sudo then it should be OK.

Grid-Monitoring Nagios tests

Finally (after a long interval) I reinvestigated getting the LCG Grid Service Monitoring Working Group nagios tests installed at Glasgow.

I had tried once before, but it needed nagios on our UI. This time I added a UI to our nagios host (nice n simple - simply added the hostname into the relevant UI group in cfengine). Works fairly well - I've got it installed and polling the SAM tests via the sam-API and picking up the results. I still need to get certificate proxy renewals working, and merge the records together with our existing definitions for the hosts (we use non-qualified names, the wlcg.cfg used FQDNs)

As the screen shot shows - We've already got a lot of green - and if we can nail the cert problems I'll switch it over to use the normal notification system

nagios monitors are Go!

it's been long overdue on the TODO list but we finally got nagios nrpe installed and configured on the worker nodes. We're now checking for locally logged in users (should only be sysadmin staff locally), high loads, processes, zombies and most importantly disk free.

Few pointers that may help others. 1) cfengine splays for 30 mins. This means if you enable a check before the plugsins are pushed out to the node it fills your mailbox. 2) if you normally use

define service{
hostgroup_name workernodes

then you'll find your testing runs on ALL workernodes. use host_name node001 (or equiv) for testing new services.
3) cfengine saves you pushing the same config out manually - and it also has the nice side effect of restarting nrpe (a necessary process) automatically when it realises nrpe.cfg has changed

Nagios Acknowledements

We've been sucessfully using Nagios as one component of our site monitoring. The email and Jabber notification is great, but when someone acks a problem, we have to look at the webpage to see who did it and what comments they added.

One minor tweak to commands.cfg soon fixed that - thanks to the macros $SERVICEACKAUTHOR$, $HOSTACKAUTHOR$, $SERVICEACKCOMMENTS$ and $HOSTACKCOMMENTS$ - described in more detail in the documentation

I'll post the snippets onto the HepSysMan wiki soon...

Jabber Dabba Do!

OK, So in between taking the kids to an Intellectual day out (The new Barnstormer's ace BTW) I have registered a new gmail account for uki.scotgrid.glasgow and played with Net::Jabber on svr031. Took quite a few Perl dependancies to get it working, as you need the IO SSL working for when google switches to TLS.

The supplied nagios "notify_via_jabber" doesn't work out of the box, but I have a simple test script by Thus0 I lifted from the web that works fine - Now all I need to do is rewrite the notify script with the correct incantations from the test one.

Plan is then to have Jabber notifications for certain classes of nagios alert.

UPDATE: Success! added a new test on the disk servers as I knew there were some non-dpm boxes amongst them. Screenshot of gaimPidgin popup below

Spent some of the week doing sysadmin work on the new cluster. Discovered that not all raid cards are configured equal thanks to a set of scripts using Net-SNMP. Although Areca are kind enough to supply the MIB Files for the Raid cards, they're badly written and contain underscores in the definitions. I already discovered a workaround last week but as I've just installed Nagios on the master node (together with the magic mod-ssl incantation to make parts of the site only accessible to named DN's) I discovered that the check_snmp plugin doesn't work with it.

One quick hack of the source file to add in

asprintf (&command_line, "%s -Pu -t %d -r %d -m %s -v %s %s %s:%s %s",

and it works as well as can be expected. Raised a sf.net ticket to try and get something better propogated upstream.

Now working on a Makefile and a pile of shell scripts to automagically generate a set of nagios configs to check each node before Paul integrates it all with MonAMI. Crib notes and more info available on the GridPP Nagios Page