'twas the night before holidays, when all through the servers not a pager was stirring...
Hmm lulled into a false sense of security by the appearance of [WLCG Nagios] entitled emails alerting me about proxy expiry on the shared nagios system I foolishly thought all was well. However we wern't getting any 'real' alerts from the system to the individual sites.
Turned out to be a configuration issue in the /etc/nagios/uki-scotgrid-*/contacts.cfg
meaning no notifications were sent - changed this to
which means we get alerted on pretty much every state change - for more details see the manual
on a more annoying note - I left my macbook PSU back in the UK and there's a limited no of apple resellers here :-(