mmmm. Shiny!

Yesterday we took delivery of one of the new workernodes that we're purchasing for the 'Phase 2' expansion of Glasgow.

Basically - Supermicro 6015TW-T servers with dual motherboards, each with 2* quad core 2.5GHz intel CPUs. Oh, and 2G RAM per core and a 500G HDD per motherboard.

Integration into the YPF installer was suprisingly painless - I generated a hundred or so new SSH keys and configs for the new boxes (still to do the cfengine ones - they're slightly fiddlier). updated the database of MAC addresses, wrote out the dnsmasq config and restarted the dnsmasq daemon. did a 'setboot' and lo, up n running.

Annoyingly I had to make some minor BIOS changes to these - we always want the nodes to power up in the event of a 'power failure' (ie, we shot them with the APC masterswitch) and there's no point them asking for a PXE boot off the second NIC (its not connected)

oh and the last bunch of workers had IDE disks not SATA (change kickstart to /dev/sda not /dev/hda for target).

So - status is the 2 machines are up n configured, now to get them into Torque (which is still playing sillybuggers wrt the gLite version - they package a pre-release 2.3.0 and it doesn't have the libtorque.0 for monami. Oh and diagnose -f truncates at 65k characters....

take that cfengine

We've had a long running problem with cfengine at glasgow - 2.2.3 (the latest DAG) didn't expand out HostRange properly on the non-workernodes (ie where we need it most - disksvr, gridsvr, natbox groups). today I spent far too long battling with both 2.2.8 and the latest svn release (don't go there - its far too fussy about the exact release of aclocal you use) and neither of them worked properly.

I finally got a minature testcase configuration file to work, then got *really* confused when I used our live config as a testcase file sucessfully, but not the normal incantation.

it turned out to be the fact we'd defined

domain = ( beowulf.cluster )
in update.conf

however, setting this broke the way cfengine handles FQDNs on the dual-homed nodes (which are gla.scotgrid.ac.uk and beowulf.cluster). Commented it out leaving cfengine to guess the right thing to do, and it all seems OK.

I have since upgraded uniformly to 2.2.3 across all the SL4 x86_64 machines and tested OK.

While doing this I noticed we hadn't defined the WMS as a mysqld node so we weren't monitoring it in nagios or backing up the database. Oops. Sorted.

nanocmos + lcas = FAIL

While working on an unrelated issue on svr021 I noticed an edg-mkgridmap error in the logfile

Aug 29 05:28:14 svr021 edg-mkgridmap[6693]: voms search(https://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk/services/VOMSCompatibility?method=getGridmapUsers): Internal Server Error

Mentioned to mike who promptly went and fixed the issue, only to discover 30 mins later we're failing SAM tests - LCAS voms plugin had once again gone fubar and caused globus-gatekeeper to segfault

Aug 29 12:14:41 svr021 GRAM gatekeeper[662]: Authenticated globus user: [DN REMOVED]
Aug 29 12:14:41 svr021 GRAM gatekeeper[663]: Authenticated globus user: [DN REMOVED] Aug 29 12:14:41 svr021 kernel: globus-gatekeep[662]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4
Aug 29 12:14:41 svr021 kernel: globus-gatekeep[663]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4


the globus gatekeeper log has a bit more info:
TIME: Fri Aug 29 12:14:41 2008
PID: 663 -- Notice: 5: Authenticated globus user: [DN REMOVED]
lcas client name: [DN REMOVED]
LCAS 0:
LCAS 1: Initialization LCAS version 1.3.7
allowing empty credentials
LCAS 2: LCAS authorization request
LCAS 0: lcas_userban.mod-plugin_confirm_authorization(): checking banned users in /opt/glite/etc/lcas/ban_users.db
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
JMA 2008/08/29 12:14:45 GATEKEEPER_JM_ID 2008-08-29.11:14:39.0000014519.0000000000 JM exiting

As before, commenting out the lcas_voms.mod in /opt/glite/etc/lcas/lcas.db allows it to work, at the expense of losing VOMS roles.

We've got it working using the voms_mod at the moment by altering the ACLs on the VOMS server (svr029) for nanocmos. Now to try and debug the lcas plugin failure

CE-sft-lcg-rm-rep fail

Ho Hum - After sorting out the gatekeeper, we still get a sam fail. Wait a minute...

Checking replication to Central SE (lxdpm101.cern.ch)

Replicate the file from the default SE to lxdpm101.cern.ch

+ lcg-rep -v --vo ops -d lxdpm101.cern.ch lfn:sft-lcg-rm-cr-node114.beowulf.cluster.080827052321.475531
Using grid catalog type: lfc
Using grid catalog : prod-lfc-shared-central.cern.ch
Source SE type: SRMv1
Destination SE type: SRMv1
httpg://lxdpm101.cern.ch:8443/srm/managerv1: No space left on device
lcg_rep: No space left on device
+ result=1
+ set +x

No space on device? grr - don't these people have monitoring ;-)

gatekeeper AWOL

Glasgow suffered 3-4 hours CE outage this evening as the globus-gatekeeper on svr021 had gone AWOL. we suffered a few SAM tests before I twigged that the 'connection refused' was coming from our end - 'service globus-gatekeeper restart' nobbled that but not until we'd failed 7 sam tests. Damn.

Glasgow on the move

Because of the current problems at RAL, Glasgow was nominated as a test peripatetic Tier-2, so see how agile ATLAS production was at moving Tier-2 resources in case of T1 downtime (note this test only works if the cloud services, FTS and LFC, are still running - if these are gone then it's almost impossible, today, to use any of the cloud's Tier-2s).

First off Glasgow was sent South East, into the NL cloud. Here we found a problem with the input datasets, because input datasets to T2s (which are subscribed without sources) only look for sources within the cloud (this follows the ATLAS computing model). However, the way around this is to specify the associated T1 (for production) as the source and then DQ2 does the work. The panda developers made the change on Friday, so that NIKHEF was specified as the source for inputs to Glasgow. Likewise, for output back to the NL T1, Glasgow's PRODDISK token was specified explicity as the source.

That done, Glasgow galloped through a couple of hundred jobs for the NL cloud, before they ran out of jobs:

Flushed with this success we've just shoved Glasgow into the FR cloud for a while, as they still have jobs left to run. Within an hour we're running a couple of 100 jobs.

SAM Failures across scotgrid: Someone else's problem

All 3 scotgrid sites have just failed the atlas SAM SE tests (atlas_cr, atlas_cp, atlas_del) as have quite alot of the rest of the UKI-* sites.

Once again this isn't a Tier-2 issue but an upstream problem with the tests themselves


ATLAS specific test launched from monb003.cern.ch
Checking if a file can be copied and registered to svr018.gla.scotgrid.ac.uk

   ------------------------- NEW ----------------
srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/
+ lcg-cr -v --vo atlas file:/home/samatlas/.same/SE/testFile.txt -l lfn:SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438 -d srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
Using grid catalog type: lfc
Using grid catalog : lfc0448.gridpp.rl.ac.uk
Using LFN : /grid/atlas/dq2/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
[BDII] sam-bdii.cern.ch:2170: Can't contact LDAP server
lcg_cr: Host is down
+ out_error=1
+ set +x
   -------------------- Other endpoint same host -----------

Supported VO Tweaks

I have enabled the 'gaussian' VO on the ScotGrid UIs at the request of a local user. This took a little longer than usual as the set-up is slightly unusual - we want to have voms and job submission correctly configured, but do not want to support the VO on our WMS (which we consider our fragile service).

YAIM 4 makes enabling a VO on a specific node type easy - we have an override stanza in services/glite-UI which adds gaussian (through 'VOS="$VOS gaussian"').

The trick bit is the cfengine magic to redefine the WMS and the LB services for gaussian only as YAIM sets all of these to be the same:

ui::
{ /opt/glite/etc/gaussian/glite_wms.conf
ReplaceFirst "https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server" With "https://rb1.cyf-kr.edu.pl:7443/glite_wms_wmproxy_server"
}
{ /opt/glite/etc/gaussian/glite_wmsui.conf
ReplaceAll "svr022.gla.scotgrid.ac.uk" With "lb.grid.cyf-kr.edu.pl"
}

Gaussian are a special VO who only seem to exist to allow access to a commercial software package: "Gaussian VO enables use of commercial chemical package Gaussian on EGEE Grid".

Grids and licensed software still have some serious paradigm issues when you have to setup an entire VO to use a piece of software.

Nagios, nagios, where are you?

Nagios seems not to be sending us alarms properly on SAM test failures (we had 2 yesterday - one for an internal SAM problem and the other for an SRMv1 timeout issue).

Andrew is investigating.

DPM Upgrade

Mike and I upgraded the DPM yesterday to 1.6.10. There are no schema changes in 1.6.7->1.6.10, so the upgrade involves 'downtime' of about 20s. We didn't put this into the CIC portal, but I suppose in retrospect we should have declared an 'at risk' period.

We're still running the i386 version of DPM (on top of x86_64). At some point it would be desirable to upgrade to x86_64; however, as i386 works just fine and this will involve real downtime, there is no urgent pressure to do so.

Greig has noted that dpm-updatespace in 1.6.10 has a bug in it: https://gus.fzk.de/pages/ticket_details.php?ticket=38330.

Alert! Alert!

'twas the night before holidays, when all through the servers not a pager was stirring...

Hmm lulled into a false sense of security by the appearance of [WLCG Nagios] entitled emails alerting me about proxy expiry on the shared nagios system I foolishly thought all was well. However we wern't getting any 'real' alerts from the system to the individual sites.

Turned out to be a configuration issue in the /etc/nagios/uki-scotgrid-*/contacts.cfg

We had

    service_notification_options    n
    host_notification_options       n

meaning no notifications were sent - changed this to

    service_notification_options    w,u,c,r,f
    host_notification_options       d,u,r,f,s

which means we get alerted on pretty much every state change - for more details see the manual

on a more annoying note - I left my macbook PSU back in the UK and there's a limited no of apple resellers here :-(

Space tokens aplenty!

Glasgow have deployed the ATLAS PROD, USER and GROUP disk space tokens, in line with the requirements of this GGUS ticket.

As we're running DPM, the procedure was fairly trivial, and is documented on the ScotGrid wiki.

We've also fixed and re-enabled the DPM information provider script, which Graeme reported broken in this blog posting.

An ldapsearch query shows that we're now advertising the new tokens correctly:


# atlas:ATLASPRODDISK:online, svr018.gla.scotgrid.ac.uk, resource, grid
dn: GlueSALocalID=atlas:ATLASPRODDISK:online,GlueSEUniqueID=svr018.gla.scotgri
d.ac.uk,Mds-Vo-name=resource,o=grid
objectClass: GlueSATop
objectClass: GlueSA
objectClass: GlueSAPolicy
objectClass: GlueSAState
objectClass: GlueSAAccessControlBase
objectClass: GlueKey
objectClass: GlueSchemaVersion
GlueSARoot: atlas:/dpm/gla.scotgrid.ac.uk/home/atlas
GlueSAPath: /dpm/gla.scotgrid.ac.uk/home/atlas
GlueSAType: permanent
GlueSALocalID: atlas:ATLASPRODDISK:online
GlueSAName: Replica online storage for VO atlas
.
.
.


All in all, a productive few minutes' work...I wonder how those with dCache are coping...

Attack of the Clones

So, I finally got round to booting the other three new servers we'd recently purchased (these are nice simple dual core, dual CPU boxes (Dual-Core AMD Opteron(tm) Processor 2216) on Tyan Thunder n3600M (S2932) Motherboards)

I'd always been exceptionally suspicious of the PXE MAC address thet the 1st one offered (60:50:40:30:20:10) rather than the imprinted one on the rj45 socket starting with 00:E0:81 but the fact that three out of four machines all had the same mac address meant a phone call to the vendor was called for.

The one that works has the 2.01 BIOS, the others were already shipped with 2.02 and downgrading didn't cure the amnesia. I don't see a utilty to reburn the mac address ont the machine, so have left that in the capable hands of the vendor for now. Googling seems to suggest that nvidia cards are somewhat prone to wierdisms like this. Bother.

Powercut at Durham

Ok, first blog post here...

On sunday, our machine room's UPS caused a brief power failure, which unfortunately tripped some breakers so we had to call out the electricians before we could start restoring service.

The UPS will take some time to repair, so Durham will be at risk for a while.

On the plus side, the changes involved with the SE rebuild have been proven to survive a reboot!

EDIT:
Just realised that I forgot to say that the site is back online, and has been since sunday evening, it is just currently at mercy of the power company.

ECDF - More VOs

Sam reports that:

"biomed, pheno, vo.scotgrid.ac.uk and vo.gridpp.ac.uk VOs have been enabled on the SL3 CE (ce.glite), the SRM (for storage), and the monitoring/accounting box (mon.glite)."

This seems to have produced a glitch on account records publication on the MON box, but a fix is in the pipeline.

ECDF SAM Tests Fixed

Sam managed to fix SAM last week. It turned out that the two CEs were interfering with each other in the samsgm account area and producing corrupted GASS caches. It's still a bit of a mystery as to why this happened - shared accounts for *sgm has been normal (if not best) practice for ages. However, it's being worked around and the site is up and stable again.

Durham SE Issues

Durham suffered a complete SE failure last week. A RAID card failure took down the old SE, gallows, and then an LVM metadata corruption took out the new disk server on se01.

The list of lost ATLAS files has been reported (https://savannah.cern.ch/bugs/?38037) and we're waiting for the catalog to be cleaned up to restart production here (well, when there are any jobs to run).

We took the opportunity to retire gallows and now se01 is the sole SE at Durham. It should suffice for ATLAS production where we only need a few TB cache anyway.

In the meantime there was a power outage in the Durham machine room over the weekend. David had to get the university to reset some breakers but things seem to be running well now.

Publish and be damned...

It all went pear shaped yesterday as information publishing fell over on the SE. It seems when I quickly "fixed" the DPM information provider script to get the correct hostname I forgot to chomp() the output from "hostname -f". So the hostname variable had a trailing newline which corrupted the information system. The BDII logs started to throw errors like "First line of LDIF entry does not begin with 'dn:' at /opt/glite/libexec/glite-info-generic line 17".

Unfortunately the BDII then considered the whole of the SE information package corrupt (rather than just that provider's output) and our SE promptly dissappeared from the information system with the attendent RM test failures.

This situation then persisted for most of the day until Andrew noticed it "by eye". So we had another failure - nagios didn't send an alarm properly when we started to fail. If that had happened it would have been fixed in a hour, but instead we were failing for 8 hours.

From the dizzy heights of SAM perfection we fell to 98% for the month, 95% for the week. It wasn't quite hubris, but it was ironic that I was blogging about Glasgow's reliability at the very moment we were broken.

At the moment I have removed the info provider for tokens, and I will more carefully put it back on Tuesday.

Problems up north

We have two major problems in ScotGrid right now:

ECDF: Have been failing SAM tests for over a week now. The symptom is that the SAM test is submitted successfully, runs correctly on the worker node, but then job outputs never seem to get back to the WMS, so eventually the job is timed out as a JS failure. As usual we cannot reproduce the problem with dteam or ATLAS jobs (in fact ATLAS condor jobs are running fine) so we are hugely puzzled. Launching a maual SAM test throught the CIC portal doesn't help because the test gets into the same state and hangs for 6 hours - so you cannot submit another one. Sam has asked for more network ports to be opened to have a larger globus port range, but the network people in Edinburgh seem to be really slow in doing this (and it seems it is not the root cause anyway).

Durham: Have suffered a serious pair of problems on their two SE hosts. The RAID filasystem on the headnode (gallows) was lost last week and all the data is gone. Then this week the large se01 disk server suffered an LVM problem and we can no longer mount grid home areas or access data on the SRM. Unfortunately Phil is on holiday, David is now off sick and I will be away on Monday - hopefully we can cobble something together to get the site running on Tuesday.

Thankfully, dear old Glasgow T2 is running like a charm right now (minor info publishing and WMS problems aside). In fact our SAM status for the last month is 100%, head to head with the T1! Fingers crossed we keep it up.

Much improved CE

Our bugbear in the past was always the lcg-CE, which was a service it was easy to overload and cause the site to hit meltdown (we have a lot of examples collected in http://scotgrid.blogspot.com/search/label/CE.).

A few months ago a new daemon, the globus cache marshal, was introduced which promised to substantially reduce the load on the old CE.

Recently we have had a few job spikes from local atlas and pheno users and I'm very happy to say that the CE seems much healtier than in the past. Having more than 1.5k jobs running and queued the load on the CE was modest and the CPU usage was < 20%.

This is a huge improvement over past performance and has removed a major source of site instability.

Who's crying now?

Our poor WMS was killed from the end of last week when a pheno user submitted about 20k jobs into it. Worse, they hadn't used a proxy renewal service, so their VOMS extension on the proxy expired and so the jobs they had submitted suffered shallow failures, promting further resubmission attempts and further load.

We have contacted the user, but they were having a great deal of trouble even cancelling the jobs. We've left things for 2 days now, but the situation is really not improving. I can't see much hope for the machine in its current state - we'll probably have to blow it away and start again next week.

For the moment we've asked our users to revert to using ye olde RB, which we were just about to switch off but hadn't actually decomissioned.

There's a serious question now about what level of WMS service we want to provide. It's a complex service and rather difficult to debug when it goes wrong. Certainly an upgrade to the SL4 version should be done, but do we want to have 2 WMS hosts and possibly even a separate LB service?

disinformation

We just got ticketed for a failing SE sam test. Most odd as Steve Lloyds SAM results were all green. Re-read the ticket and it turns out we were publishing info for svr018.beowulf.cluster rather than the external interface name. Despite this being noted before Graeme hacked it around and raised a Savannah Ticket

And don't do it again...

I re-enabled Heinz today on the cluster, our infamous RSA cracking biomed user.

Tony finally spoke to Cal and this seemed clarify that

• VOs now know they have a more serious responsibility to discipline their users
• Heniz knows he cannot run this work again under biomed
  

He'd been suspended for six months which seems like an appropriate punishment.

So, how was it for you...?

I made some comments on Glasgow's site availability from October last year until now. It was Q1 this year that was very hard on us (strictly last week of Feb until end of March) - we lost 5 days to the CE crash and reinstall of core nodes, then were pestered by minor networking and user level problems. All of these conspired to reduce our availability to 83% that quarter.

However, we seem to be very much on top of things now, with 95% so far for Q2 and 100% in the last week (we were lucky that our servers had certificates from the old CA though).

I suppose this also accounts for the lack of blog entries this month - things are going well.

Full steam ahead, Captain. The weather is fine...

liblcas_lcmaps fix

There is a fix for the segfaulting bug I reported against the globus gatekeeper and gridftp server, https://gus.fzk.de/pages/ticket_details.php?ticket=35694.

If you download the patched version of liblcas_lcmaps_gt4_mapping.so.0.0.0 and install it to /opt/glite/lib then the problem is resolved.

I applied the patch last night and since then we haven't suffered a single segfault.

Maarten said that the official fix should reach production in the next couple of weeks, but I'm happier having it on site now.

CRHell...

So, even after fixing our CE disk space problems we still were failing SAM tests. The error was one I'd never seen before: "7 an authentication operation failed".

The GOC wiki hinted at a few things which can cause this, but as it only seemed to affect SAM tests (Steve Lloyd jobs and my ATLAS production were both running fine) there wasn't a lot that could really be debugged locally - we had even had a successful SAM test from Rafa's SAM Admin interface at 11am.

One of the classic X509 errors though, is CRLs being out of date. When I checked the CRL files on the CE is was clear something was amiss. A few were dated today, but most were approaching 5 days old. When I ran the CRL updater by hand I got the error:

fetch-crl[19144]: 20080507T154251+0100 updating CRL 'CERN Trusted Certification Authority (1d879c6c)'
fetch-crl[19144]: 20080507T154251+0100 File /etc/grid-security/certificates//1d879c6c.r0 valid: no
fetch-crl[19144]: 20080507T154251+0100 Attempt to overwrite /etc/grid-security/certificates//1d879c6c.r0 failed since the original is not a valid CRL file

What's worse is that not only did this CRL fail to update, it caused the updater script to bomb and no CRL after this point even attempted to update!

In the end I had to delete all of the CRLs on the CE and re-run the script to get fresh copies.

What exactly happened I do not know, but the relevant command in fetch-crl is:

openssl crl -hash -in CERT_REVOKE_FILE -noout -inform PEM -text

This should produce, as a first line, the hash for the CRL, which is the same as the file name. however, testing one of the bad files I got instead:

unable to load CRL
12697:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:731:

So clearly they had gone evil in some way.

I think this is poor behaviour from the script, so I have submitted a GGUS ticket (https://gus.fzk.de/pages/ticket_details.php?ticket=36191).

X509 error messages just suck so badly.

It's all gone 'orribly wrong

OK - My bad. I spotted we failed a sam test yeterday (got a mail from the automated alert) - didn't realise it doesn't send multiple ones if you keep failing....

Am sure Graeme will post more but we'd filled up / (as /var wasn't on a separate partition - it is now, and a nice healthy 30G) on the CE. Puzzlingly nagios hadn't bothered to alert that we'd gone warning at 8% free or critical at 4% free and was "OK - /0 free"

case sensitivity in check_disk: -w is for disk space, -W is for inodes. grr. Typo-tastic. I lowercased the offending config and let cfengine ripple it out. While it did so I noticed cfengine restarted ntpd on the 3 nat boxes (that also act as the timeservers for the cluster) - somehow it was copying both a standard then a local /etc/ntp.conf into place each time and restarting as planned on a new config file.

my bad - we use class 'natboxes' for the group and I'd specified any.!(master|nat):: changing it to any.!(master|natboxes) worked fine - no restart since and none of the workernodes are seeing any upstream timeservers on INIT or LOCAL

Holy lactating keyboards Batman, there goes the disk!

There's a pre-amble to this post which isn't really to do with ScotGrid directly, but has some bearing subsequent event. If I tell you the actors were a glass of milk, a three year old and my laptop keyboard then you can doubtless assemble the plot yourself. So yesterday some important keys were completely non-functional on my laptop - one of these keys was a character in my password so I wasn't able to login at all. Despite my best MacBook disassembling and cleaning efforts I couldn't recover the keyboard - there seems to be gunk etched onto the conducting keyboard sheet. Now that I am back in the office I can use a USB keyboard, but I need to get a new MacBook one quickly.

Anyway, upshot is that my online activities were squeezed into an ancient IBM R32 laptop running XP with 256MB of RAM. So, I was not watching (bookmarks were effectively unavaliable) when the CE ran out of disk space and we went belly up for SAM tests yesterday lunchtime.

Problem was easily identifed this morning and, thanks to the magic of LVM, corrected. When the machines were setup /var was not separate from /, so we've now added a separate /var logical volume with 30GB in it.

We looked into why this did not trigger and alarm and it seems that there was a mistake in the nagios configuration which was looking at free inodes, rather than free space. Corrected now.

I also took the opportinutity to re-run YAIM on the CE and fix the broken VOMS mappings. My atlas production role now maps correctly to an prdatlas account. This might cause a failure to clear from globus the stack of jobs running under my "local" gla012 account, but the actual production system doesn't require any outputs from globus so the real payload will run fine.

Sadly, the segfaulting on the CE still happens, however it seems the the problem was spotted earlier and that a fix is on the way: https://savannah.cern.ch/bugs/?35981.

Farewell ScotGrid-Edinburgh!

Not withstanding the problems this week at ECDF, in general the new Edinburgh resource has been working well. Consequently maintaining the old site was just a drag on our people, with no real purpose anymore. Sam broadcast the intention to close the site a couple of weeks ago and we changed its status in the GOC to "closed".

Greig and Sam have ensured that the site's old storage is accessible through ECDF, though we've advised VOs to move their data off this SE as the hardware is ageing and unreliable.

Edinburgh is dead! Long live Edinburgh!

ECDF down for the moment

ECDF have been having real trouble with GPFS in the last week, which gave us some miserable results (23% pass rate on SAM, c.f., UK average of 75%). For the moment the systems team have suspended job submission and the site went into downtime on Friday.

This may or may not be related to the problems we see with the globus job wrapper code on ECDF, where the GPFS daemon consumes up to 300% CPU due to a strange file access pattern in the job home directory. Sam is working on installing an SL4 CE (based on the GT4 code) to see if this improves matters.

Who's watching the watcher?

We had a problem on the DPM headnode with the new VOMS certificate for Zeus not being installed. When I checked I found it was in the repository, but had not been copied to the server. What gives? It turned out that the cfengine version we had on svr018 (2.2.3-1.el4) was not defining the gridsrv class properly (through the HostRange expansion), so then the grid class was not defined and consequently the vomsdir was not being checked.

I downgraded to 2.1.22 and this fixed the problem. But there is a mystery here - why does 2.2.3-1 work fine on the worker nodes?

I see 2.2.6 has been released. Maybe we should go back to rolling out own build of cfengine?

#include <documentation.h>

I quote from the DPM Developer Documentation

LFC/DPM Database schema
TO DO : describe non straight forward tables/fields....

so, with that in mind, I set about pulling out the number of SRM 2.2 requests vs the no of SRMv1 requests at the site. v1 should be constant (what with all the new users coming onboard) and SRM 2.2 being a rapid increase since we enabled it? well it's not easy to grep from the logs so I thought I'd poke the DB. First off in dpm_db.dpm_req r_type a char(1) field normally has g (get?) and p (put?) but we have just over 1500 rows where type is 'B' (broken?). hmm - all from flavia's DN and clienthost of lxdev25

my plots of the dpm usage are far too spikey to make sense of at the moment, but I'll work on presenting the info a bit clearer.
In the meantime I discovered that it's pretty obvious when we set torque to fill the jobslots in host order (made it easier to drain nodes off) and when we send nodes away to vendors.

assimilation

I'd noticed that over the last month the load on our DPM headnode had been higher than before we switched on MonAMI checking of the DPM. Of course I instantly blamed the developer of said product. However, I disabled monami to prove that the load went down and lo... no change. Hmm.

I then started working out how to optimise the MySQL memory usage - we have about a 1.8G ascii file when I do a mysqldump -A and yet the innodb file takes up a whopping 4.4G on disk with tiny constantly rolling transaction logs of 5M.

As paul was here at cern (it's the WLCG Workshop this week) we got together to hammer out some changes to our implementation. When we logged onto svr018 (The DPM Headnode) I noticed that monami was running again. Turns out that cfengine was "helpfully" restarting the process for us. Grr.

So, an evening of infrastructure management changes:
- we had i386 monami rpms installed - we'd hard coded the repo path rather than using the $basearch variable in our local mirror.
- we had to ensure that we had backup=false in cfengine - where we had a config_dir directive (such as /etc/monami.d and /etc/nrpe.d) the applications were often trying to use someconfigfile.cfg and someconfigfile.cfg.cfsaved - ditto cron.d etc etc.
- we were sometimes trying to run 64 bit binaries on 32 bit architectures as we'd copied them straight from cfagent (normally nrpe monitors) - We've now using $(ostype) in cfagent which expands to linux_x86_64 and linux_i686 on our machines. Although cfengine sets a hard class of 32_bit and 64_bit but you can't use that in a variable.
- we now have the 'core' nrpe monitors (disk space, load, some process checks) installed on ALL servers not just the workernodes. Ahem. Thought we'd implemented that before.
- we've upgraded to the latest CVS rpm of monami on some nodes and we've got grooovy mysql monitoring. - oh and the load's gone down too.

Oh no, not again...

We went though a little rash of SAM test failures last night. This turned out to be an LHCb user who was submitting jobs which filled up the scratch area on the worker nodes and turned them into blackholes.

Obligatory GGUS ticket was raised.

We do alarm against disk space filling up on the worker nodes, but it was still 4 hours before action was taken and the nodes set offline before being cleaned. In that time an awful lot of jobs were destroyed. Make me think we might want to automate the offlining of nodes which run out of disk space, pending investigations.

Splunk / nagios / logrotate

Well, I upgraded to nagios3 this evening on the cluster and noticed it had a new enable_splunk_integration option in the cgi.cfg - I'd looked at splunk before and thought 'hmm, nice idea, not sure it'll work with the grid stuff' but decided to give it a whirl

first up - nagios gotchas - We had the dag rpm installed which hasn't been updated to the 3.0 let alone the 3.0.1 release so went for the manual compile option. Discovered that the (gd|libjpeg|libpng)-devel packages weren't installed - quickly fixed by yum.

took the ./configure line from the spec as a guide - however it managed to splat the cgi's into /usr/sbin rather than /usr/lib64/nagios/cgi - thanks :-( soon found em and moved em round. seems to be working OK - not installed the newer wlcg monitors yet - thats the next task.

Splunk - looks flash but is it any good? There's no sign of any educational pricing on their website and the 'free' version has one HUGE weakness - no user authorisation / login. Temp workaround of some iptables rules to reduce risk and had a play. Defined /var/log on our central syslog server as a datasource and watched it go.

well, sort of... it promptly filled /opt/splunk as it makes an indexed copy of anything it finds, - I think for a real install we'd need some new space on a disk. secondly it quicky swallowed more than its 500M/day 'free' allowance - grabbed a 30day trial licence of the enterprise version and lo it now complains that I've had 2 licence violations of over 5G/day indexed. Harumph.

not sure if this would settle down once it goes through the backlog of the archived logfiles - perhaps if I implement only a syslog FIFO for it it'd be happier. Also we have the 'traditional' logrotate style of .1 .2 .3 etc rather than the more dirvish friendly dateext option - we should really swap... if the RHEL logrotate supports it :-/

"rpm -q logrotate --changelog" doesnt mention it although its fixed in fedora

The other issue is that splunk thrashes the box as it indexes, and it's just stopped as its filled the disk again. Ho Hum.

A long time coming: UKI-SCOTGRID-ECDF on APEL

So, yes, it's probably taken a little longer than it might have, but UKI-SCOTGRID-ECDF is now publishing all its accounting data back to early January.
Here

Of course, ops has a disturbingly high share of the Grid usage at the moment, but hopefully we will start to get ATLAS (and maybe even LHCb) jobs filtering in in the near future...

ECDF running for ATLAS

ECDF have now passed ATLAS production validation. The last link in the chain was ensuring that their SRMv1 endpoint was enabled on the UK's DQ2 server at CERN - this allows the ATLAS data management infrastructure to move input data to ECDF.

After that problem was corrected this morning the input data was moved from RAL, a production pilot picked up the job and ran it then the output data was moved back to RAL.

I have asked the ATLAS production people to enable ECDF in the production system and I have turned up the pilot rate to pull in more jobs.

We had a problem with the software area not being group writable (for some reason Alessando's account mapping changed), but this has now been corrected and an install of 14.0.0 has been started.

It's wonderful to now have the prospect of running significant amounts of grid work on the ECDF resource. Well done to everyone!

Only one bite at the cherry...

I have modified the default RetryCount on our UIs to now set zero retries. Automatic retries were actually working quite well for us when we were losing a lot of nodes to MCE errors (in the days before the upgrade to SL4, x86_64) - users' jobs would automatically rerun if they got lost and there was no need for them to worry about failures. However, recently we see users submitting more problematic jobs to the cluster - some which fail to start at all, some which run off into wallclock limits, others which stall half way through. Often we have to gut the batch system with our special spoon and in this case having to do it four times because the RB/WMS keeps resubmitting the job is less then helpful.

For once cfengine's editfiles stanza was useful and a simple:

ui::
{ /opt/glite/etc/glite_wmsui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}
{ /opt/egd/etc/edg_wl_ui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}

got the job done.

Emergency Outage

Due to a vulnerability to the security flag in the ipv4 header, we will be taking uki-scotgrid-glasgow offline today for an upgrade. In order to minimise downtime we shall be rebooting all worker nodes simultaneously rather than draining queues.

We aim to have this work completed by 12:00 midday today.

Please see RFC 3514 for more details. We advise other sites to perform this upgrade asap.

brain dead batch systems

why oh why are some of the batch utilities so brain dead? simple case of 'qstat -r' should show who's running jobs right? wrong as it outputs based on a fixed 8 character width for username. doh. so 'prdatlas' and biomed06 seem to be busy. Well not quite as if I do a qstat -f | egrep " e(group|user) " | sort -u I see that it's prdatlas028 and several biomed06? users. grr...

I may install Job Monarch from sara but in the meantime it'll be some hacky PHP to parse the outputs a bit more cleanly

Also, despite having 493 running jobs at the moment (we're down on our capacity as I'm still doing a rolling upgrade to SL4.6 and a new kernel) there are a very small number of users on the system

svr031:~# qstat -f | grep euser | sort -u | wc -l
14

not good, Especially if they decide to take a break.

p p p pick up a pakiti

We've been using pakiti at Glasgow for some time now for keeping an eye on which nodes are out of date. One minor niggle is that it doesn't keep track of the grub default kernel (ie what should come in on reboot) compared to the running kernel

We already had a v simple shell script that did that:

 pdsh -w node[001-140] chkkernel.sh | dshbak -c
----------------
node[001,005,007,014,016-020,022-023,025,028,031-061,063-085,087-090,092,095-096,098-101,103-104,106-107,109-110,113,115,118-120]
----------------
 Running: 2.6.9-67.0.7.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status OK
----------------
node[062,091,093-094,097,102,105,108,111-112,114,116-117,121-127,129,131,133-134,136-139]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[003,009,011,013,015,021,027,029]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[128,130,132]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[002,004,006,010,012,024,026,030,140]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[086,135]
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status OK
----------------
node008
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error

but I finally got it integrated with some patching - see http://www.scotgrid.ac.uk/wiki/index.php/Pakiti

result - pretty green / red status on the "default kernel' column.

The patches have been emailed to Romain so may well appear upstream eventually

Edinburgh as an LHCb Tier-1?

I've just been accused (jokingly, I hope) of trying to turn Edinburgh into LHCb's 7th Tier-1. The attached plot shows the recent data transfers that I have been running into our dCache. The rates are good (~35MB/s), but not particularly special. However, against a background of zero, it certainly made LHCb jump up and take notice ;) Maybe this will convince them that Tier-2s really can be used for analysis jobs...

I should note that during these transfers one of the dCache pools was about to melt (see below). I've since reduced the max number of movers on each pool to something more reasonable. For the tests, I created a small application that spawned ~50 simultaneously lcg-cp's that were all transferring files from CERN CASTOR to Edinburgh. Who needs FTS when you've got DIRAC and lcg_utils? Now all I need is someone else's proxy and I'll never be caught... ;) But, on a serious note, I suppose this does show that people can create tools to abuse the system and get round the official FTS channels, which could impact the service for other users.

The curse of 79...

Since the dawn of the Glasgow cluster we have been cursed with a low level of globus 79 errors. We did not understand these well, but always believed that they were caused by a confusion in the gatekeeper, where the X509 authentication seemed to suffer a race condition and get muddled between users.

However, since upgrading to an SL4 CE and installing it on a different machine we still get these cropping up (an example).

The GOC Wiki suggests this can be caused by firewall trouble or an incorrect GLOBUS_TCP_PORT_RANGE. Now, this is (and was) correctly defined on both machines to be the standard 20000-25000. However, I have decided to change it to 50000-55000 in case we are tripping some generic nasty filter somewhere else on campus.

Since I did that, last night, we haven't had a 79 error - however this proves nothing so far as we can easily go for a week without one of these happening.

I also contacted the campus networking people to ask if there was any known port blocks in this range.

Data Management and MICE

I had a chat to one of our MICE PhD students a couple of weeks ago and I was explaining how to use EGEE data management (SRMs, LFCs, FTS, lcg utils, etc.). His comment afterwards was "I didn't know I was going to do a PhD in data management...".

The problem is that all these tools are very low level, so any user community has to build a higher level infrastructure on top of this. Obviously the LHC experiments have done this extensively, but it is frustrating that there is no simple generic data management service for smaller VOs who lack the resources of the larger VOs.

I wonder if this accounts for the popularity of SRB in other communities? It may have some limitations, but it clearly offers a higher level data storage, cataloging and metadata service which must be attractive for smaller user communities. Surely there is a potential project to try and tie all of the EGEE components into a sensible data management system?

Durham - SL4 Install Success!

Durham took the plunge earlier this week to upgrade the CE, SE and all nodes to SL4.6... with success! After our preparation was delayed slightly due to a small UPS failure, we set about installing cfengine to handle the fabric management. This took a little longer than expected but our patience has paid off and it eases the pain of setup and config of clusters. Using the normal RedHat Kickstart to get a base install of SL4.6, we then hand the rest of the setup to cfengine to work its magic (install extra packages, setup config files, run YAIM etc).

Firstly installing a Worker Node was relatively straight forward. Then came the CE along with torque, PBS and the site BDII setup. Thanks to Graeme for help checking our site was working and publishing as expected.

We unexpectedly hit a firewall issue as I had renamed the CE from the old "helmsley.dur.scotgrid.ac.uk" to "ce01.dur.scotgrid.ac.uk"... though I had preserved the IP address. Not what I expected but our network guys were able to fix the rules and we were operational again.

Then the SE followed very quickly afterwards, cfengine and YAIM working their magic very successfully. The procedure was as simple as 1) dump of the database, 2) install SL4.6, 3) Let cfengine do its stuff for a base install, 4) restore the database, 5) Run YAIM. Simple!

Just one gotcha was trying to change the NFS mounted home directories to be local to the nodes. This fails with an error trying to copy the globus-cache-export files. Due to time constraints we have re-enabled the NFS home dirs... but I'm sure this will be simple to fix and I'll look at it next week.

Fair shares and queue time will need reviewing but in all a busy and successful few days. We're passing SAM tests and I've seen Phenogrid, Atlas and Biomed running jobs. Still the UI and a disk server to do, but with cfengine in place, this should be relatively straight forward and will require no downtime.

Another ECDF/Grid requirement mismatch.

While ECDF is, in principle, functional and capable of running jobs, this is a bit useless if no-one can see if you're doing it. So, in the face of APEL accounting still not working for the cluster, I had another look.

There were two problems:
Firstly, the sge account parser was looking in the wrong directory for SGE accounting logs - this fails silently with "no new records found", so I didn't notice before. The configured location actually was correct when I set the thing up, but the mount point had been moved since (as the CE is not on the same box as the SGE Master, we export the SGE account directory over NFS to the CE so it can parse them) with no indication that anything was wrong.

Secondly, after I fixed this...
It turns out that the APEL java.lang.OutOfMemoryError strikes again for ECDF.
The ECDF systems team configure the SGE accounting to roll over accounting logs on a monthly basis. Unfortunately, this leads to rather large accounting files:
# ls --size /opt/sge/default/common/acc* --block-size=1M
1543 /opt/sge/default/common/accounting

(yes, gentlemen and ladies, that is a one and a half gig accounting file...and we're only half-way through the month. The archived accounting logs tip the scales at around a quarter to half a gig compressed, but they compress rather efficiently so the "true" size is much larger - up to 10x larger, in fact.)

I suspect the next step is to arrange some way of chopping the accounting file
into bitesized chunks that the APEL log parser is capable of swallowing.
The irony is that we already parse the accounting logs internally using a thing called ARCo - I've not seen any indication that it would be easy to get APEL to understand the resulting database, though.

Dem info system blues

I fixed a problem on the CE information system tonight. YAIM had gone a little screwy and incorrectly written the lcg-info-dynamic-scheduler.conf file, so I had added the lrms_backend_cmd parameter myself as:

lrms_backend_cmd: /opt/lcg/libexec/lrmsinfo-pbs -h svr016.gla.scotgrid.ac.uk

Adding the host seemed sensible as the CE and the batch system don't run on the same node, right? Wrong! the host paramater ends up being passed down to "qstat -f HOST" which is a broken command - we ended up with zeros everywhere for queued and running jobs and, consequently a large stack of biomed jobs we are unlikely ever to run.

I raised the obligatory GGUS ticket: https://gus.fzk.de/pages/ticket_details.php?ticket=33313

To VOMS or not to VOMS? That is the question (for LCMAPS...)

Our advice to local users of the cluster has traditionally been to not use VOMS credentials. This is to ensure that they are mapped in the batch system to their local account, rather than to a pool account from their VOMS attributes (mappings to local accounts are maintained by the grid-mapfile-local file). In the default configuration of LCMAPS VOMS pool account mappings are made before the grid-mapfile, which is now just a fall back.

However, I could not simply reverse the order of the LCMAPS plugins as this would undo all the good which VOMS brings and move everyone back to a single fixed or pool account mapping no matter what their VOMS credentials (this would probably have affected me worse than anyone as I flit between atlas, atlas/Role=production, vo.scotgrid.ac.uk and dteam!).

So, for local users grid-proxy-init seemed to be the way to go, even if I knew this would come back and be a problem later. However, later became earlier as soon as I started to test the gLite-WMS - here it turns out you must use a VOMS proxy. Simple grid proxies just don't work anymore.

Finally, puzzling over the very poor LCMAPS documentation, and staring at the configuration script I managed to solve the problem by:

1. First running a local account plugin against a grid-mapfile which only contains our local user accounts.
2. Then running the VOMS plugins as usual.
3. Finally, running the grid-mapfile plugin, against the usual /etc/grid-security/grid-mapfile.

This was almost too easy to be true - and indeed it turns out not to quite be that simple as you hit a bug in LCMAPS that you cannot use a module twice - so having lcmaps_localaccount.mod twice is not possible. However, it turns out that one can do it if the module is renamed and physically copied. This works, so we now have an lcmaps_localaccount.mod and a lcmaps_localuseraccount.mod - exactly the same bytes, different names! (To be strictly accurate we have two copies of liblcmaps_localaccount.so.0.0.0, to which these links point.)

And, in the end, I was able to keep myself out of the local user grid-mapfile, so I have the full array of VOMS roles for myself, while the local users are cosily tucked up in their local account areas.

Upgrade to gLite 3.1 - Notebook

It was well over a year since we'd done a "from the top" install of the CE, so a few things were different:

• The information system has been re-branded. It's now configured in /opt/glite/etc/gip, although many of the plugins are still running from /opt/lcg.
• The CE information system is upgraded to use the BDII (on 2170).
• The site BDII also now uses a wrapper script to get all information, rather than coding the list of GRISs/BDIIs to query (GIP file:///opt/glite/libexec/glite-info-wrapper).
  
• LCAS and LCMAPS now also run out of /opt/glite.
• Pool account mappings are now done to a random pool account, rather than the "next" free one. In addition the hardlink syntax used for assigning a pool account to a DN has changed slightly (using colons to indicate VOMS attributes after the DN).

Funeral March for the Lost CE

So, here's the post mortem on the CE hard crash on Wednesday last. About 1700 the load on the CE started to ramp up and it quickly rose to almost 100. I could see this happening just as I was about to go home (typical!) so I started to indulge in a frantic bout of process killing to reduce load and bring the CE back under control. However, despite my best efforts, the CE crashed hard at 1800 (gap in the ganglia plot).

When the machine rebooted, the gatekeeper restarted and again the load began to rise. I then went through a frantic couple of hours trying to do everything I could to reduce the load and try an get the CE back on an even keel - this was made very hard by the fact that with load averages quickly rising to 60+ the machine was extremely sluggish.

I shut down R-GMA, turned off the mail server to no avail. I killed off queued jobs in the batch system, even got as far as disabling VOs, and banning users whose jobs I had cancelled. I even got so desparete as to firewall the gatekeeper from all but the ScotGrid RB! But although I coud slow down the load increase by doing this, by 10pm it became clear that something dreadful had happened to the gatekeeper. Every gatekeeper process which was forked stalled, consuming CPU and managing to do absolutely nothing. As there was no response, the RB then contected the CE again, forking off another gatekeeper and the march to death continued. If I reduced the number of users able to contact the CE this slowed down the rate of resource exhaustion, but could not stop it. Clearly something utterly evil had happened to the gatekeeper state.

At this point I became convinced that nothing could be done to save the remaining queued or running jobs and that the site was going down. I started to think instead about moving our March downtime forwards, to do the SL4 upgrades, and to prise the CE and the batch system apart. And of course, that is just what we did at the end of last week.

Acrobat ate my disk servers!

Glasgow is finally out of downtime. GS worked his grid-fu and managed to upgrade lots to SL4 - Admittedly some things (RGMA) weren't a goer. APEL Accounting could be broken for a while as we've now split the CE (new home = svr021) and the Torque server (still on svr016). My 'simple' job was to take care of the DPM servers...

Simple enough, we hacked into the YAIM site-info.def stuff and separated things out into services/ and vo.d/ - easy. Few gotchas as cfengine was once again reluctant to create the symlinks on the target nodes (however creating the symlinks on the master and replicating those works fine) which we thought might be fixed by an upgrade of cfengine from 2.1.22 to 2.2.3. Big mistake. it broke HostRange function of cfengine.

so we have

 dpmdisk = ( HostRange(disk,032-036) HostRange(disk,038-041) ) 

but cfengine complained that

SRDEBUG FuzzyHostParse(disk,032-041) succeeded for disk033.gla.scotgrid.ac.uk
SRDEBUG FuzzyHostMatch: split refhost=disk033.gla.scotgrid.ac.uk into refbase=disk033.gla.scotgrid.ac.uk and cmp=-1
SRDEBUG FuzzyHostMatch(disk,032-041,disk033.gla.scotgrid.ac.uk) failed 

now I'm not sure if this is due to the problem of short hostname vs FQDN - I've hit a similar issue when I want to copy iptables configs off -

$(skel)/nat/etc/iptables.$(host) mode=0600 dest=/etc/iptables define=newiptables type=sum 

needs iptables.host.gla.scotgrid.ac.uk not just iptables.host on the master repo.

Anyway, this all seems trivial compared to the hassle with the latest SLC 4X that got mirrored up to the servers overnight (the disk servers run SLC4 rather than SL4 as the Areca raid card drivers are compiled in) - dpm-queryconf kept failing with

send2nsd: NS002 - send error : No valid credential found

and yet the certificates were there and valid - openssl verify ... returned OK, ddates were valid, NTP installed etc. The dpm log showed

dpm_serv: Could not establish security context: _Csec_recv_token: Connection dropped by remote end ! 

The really frustrating thing was that the server that I installed from home while munching breakfast (all hail laptops and broadband) worked fine, but those I installed (and reinstalled) later in the office were broken. [hmm. is this a sign that I should stay at home in the mornings and have a leisurely breakfast?]

Puzzling was the fact that the broken servers had more rpms installed than the working ones. - I eventually resorted to installing strace on both boxes and diffing the output of 'strace dpm-qryconf'

the failing one had a big chunk of

open("/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xffff9028)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xffff9028)         = -1 ENOENT (No such file or directory)
open("/lib/libstdc++.so.6", O_RDONLY)   = -1 ENOENT (No such file or directory)
stat64("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0xffff9028)      = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/usr/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/sse2", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/usr/lib/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

whereas the working one didn't call this at all.

I was also bemused as to why acroread had been installed on the server and more annoyingly why I couldn't uninstall,

Yep - someone (step up to the podium Jan Iven) had mispackaged the SLC acroread 8.1.2 update...

rpm -qp ./acroread-8.1.2-1.slc4.i386.rpm --provides
warning: ./acroread-8.1.2-1.slc4.i386.rpm: V3 DSA signature: NOKEY, key ID 1d1e034b
2d.x3d  
3difr.x3d  
ADMPlugin.apl  
Accessibility.api  
AcroForm.api  
Annots.api  
DVA.api  
DigSig.api  
EFS.api  
EScript.api  
HLS.api  
MakeAccessible.api  
Multimedia.api  
PDDom.api  
PPKLite.api  
ReadOutLoud.api  
Real.mpp  
SaveAsRTF.api  
SearchFind.api  
SendMail.api  
Spelling.api  
acroread-plugin = 8.1.2-1.slc4
checkers.api  
drvOpenGL.x3d  
drvSOFT.x3d  
ewh.api  
libACE.so  
libACE.so(VERSION)  
libACE.so.2.10  
libACE.so.2.10(VERSION)  
libAGM.so  
libAGM.so(VERSION)  
libAGM.so.4.16  
libAGM.so.4.16(VERSION)  
libAXE8SharedExpat.so  
libAXE8SharedExpat.so  
libAXE8SharedExpat.so(VERSION)  
libAXSLE.so  
libAXSLE.so  
libAXSLE.so(VERSION)  
libAXSLE.so(VERSION)  
libAdobeXMP.so  
libAdobeXMP.so  
libAdobeXMP.so(VERSION)  
libAdobeXMP.so(VERSION)  
libBIB.so  
libBIB.so(VERSION)  
libBIB.so.1.2  
libBIB.so.1.2(VERSION)  
libBIBUtils.so  
libBIBUtils.so(VERSION)  
libBIBUtils.so.1.1  
libBIBUtils.so.1.1(VERSION)  
libCoolType.so  
libCoolType.so(VERSION)  
libCoolType.so.5.03  
libCoolType.so.5.03(VERSION)  
libJP2K.so  
libJP2K.so  
libJP2K.so(VERSION)  
libResAccess.so  
libResAccess.so(VERSION)  
libResAccess.so.0.1  
libWRServices.so  
libWRServices.so(VERSION)  
libWRServices.so.2.1  
libadobelinguistic.so  
libadobelinguistic.so  
libadobelinguistic.so(VERSION)  
libahclient.so  
libahclient.so  
libahclient.so(VERSION)  
libcrypto.so.0.9.7  
libcrypto.so.0.9.7  
libcurl.so.3  
libdatamatrixpmp.pmp  
libextendscript.so  
libextendscript.so  
libgcc_s.so.1  
libgcc_s.so.1(GCC_3.0)  
libgcc_s.so.1(GCC_3.3)  
libgcc_s.so.1(GCC_3.3.1)  
libgcc_s.so.1(GCC_3.4)  
libgcc_s.so.1(GCC_3.4.2)  
libgcc_s.so.1(GCC_4.0.0)  
libgcc_s.so.1(GLIBC_2.0)  
libicudata.so.34  
libicudata.so.34  
libicui18n.so.34  
libicuuc.so.34  
libicuuc.so.34  
libpdf417pmp.pmp  
libqrcodepmp.pmp  
librt3d.so  
libsccore.so  
libsccore.so  
libssl.so.0.9.7  
libssl.so.0.9.7  
libstdc++.so.6  
libstdc++.so.6(CXXABI_1.3)  
libstdc++.so.6(CXXABI_1.3.1)  
libstdc++.so.6(GLIBCXX_3.4)  
libstdc++.so.6(GLIBCXX_3.4.1)  
libstdc++.so.6(GLIBCXX_3.4.2)  
libstdc++.so.6(GLIBCXX_3.4.3)  
libstdc++.so.6(GLIBCXX_3.4.4)  
libstdc++.so.6(GLIBCXX_3.4.5)  
libstdc++.so.6(GLIBCXX_3.4.6)  
libstdc++.so.6(GLIBCXX_3.4.7)  
nppdf.so  
prcr.x3d  
tesselate.x3d  
wwwlink.api  
acroread = 8.1.2-1.slc4

Yep, thats right - RPM had decided that acroread was a dependency. Grr. Workaround - remirror the CERN SLC repo, (no they hadn't updated since), manually remove the offending rpm, and rebuild the metadata with 'createrepo'

Then make sure that the nodes were rebuilt and only ever looked to our local repository rather than the primary cern / dag ones. (thanks to kickstart %post and cfengine)

Finally we got yaim to *almost* run - it was failing on lcmaps and grid-mapfile creation (fixed by unsupporting a certain VO)
Easy fix in comparison. Anyway - DPM up n running and seems OK. Roll on the next SAM tests.... (or real users)

Phew.

Grid Middleware should not be this hard to install!

Durham - DPM v1.6.7, Space Tokens etc

Well this is the first of hopefully many posts from Durham. So firstly a quick update, well Durham seem have been ticking along nicely - with the exception of a few network and power outages over the last few months. SAM tests are passing and ATLAS, Pheno and many other VOs are successfully running jobs. Versions of the lcg software is a little out of date in places – but this is work in progress.

So with a little encouragement and help from Greig I finally took the plunge and upgraded our SE to DPM v1.6.7. After getting the yum repositories correct, it was a case of stopping the daemons, running yum update, making the DPM schema changes (we were upgrading from an old version of DPM), and then restarting the daemons... done... or so we thought!

Everything was working, file copies in and out of our SE, reserving space tokens, etc... the only gotcha was that we were publishing "GlueSAStateAvailableSpace: 0"... which wasn't true. After a little investigation, and with the help of Greig, we noticed that /opt/lcg/var/gip/plugin/lcg-info-dynamic-se was pointing to a beta version of lcg-info-dynamic-dpm. Changed this to remove the beta and bingo... all working correctly.

I have then setup to publish the space tokens as shown here here, and all is done.

A good days work, until a major network outage at the JANET/NorMAN level knocked us out most of the night... typical. We seem to have recovered now though so we should be back on track.

Thusday Night Status Update

Quick summary of where we are right now:

* YAIM configuration updated and rationalised.

* Batch system has been upgraded to Torque 2.1.9/Maui 3.2.6, running on SL4 x86_64.

* Queues have been reduced to 4, open to most VOs, with queue lengths of 30m, 6h, 3d and 7d.

* CE has been moved to svr021, again running SL4 x86_64.

* Information system has been reconfigured to new gLite versions. After a minor wobble on the CE seems to be working just fine.

* DPM headnode has been upgraded to SL4 x86_64.

And job submission works:

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
Submitted:          Thu Feb 21 23:22:30 2008 GMT
*************************************************************

...

================================================================================

   JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/gla012_sx54e7252PGtxJtB4Y2mIg

================================================================================

And there was much rejoycing.

Significant work remains for tomorrow, but very good progress being made.

Glasgow Downtime

As announced on GOCDB We're taking UKI-SCOTGRID-GLASGOW down until 17:00 localtime tomorrow (friday) to bring forwards the maintenance we'd planned for March. This was due to an unexpected CE failurelast night that meant the queues were empty.

DPM and ATLAS Space Tokens

So, here's the definitive guide to enabling space tokens for ATLAS:

1. Assign the token to atlas/Role=production using the dpm-reservespace command. (Example here.)
2. Create the directory where DDM will put the files, which is the normal path for ATLAS, plus the name of the spacetoken in lower case. e.g.


• dpns-mkdir /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


3. Now change group ownership of this directory to atlas/Role=production, chmod it to 775 and finally add two ACLs which will mean the entire tree will be writable by production roles:




• dpns-chgrp atlas/Role=production /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-chmod 775 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk




4. Enable the space token publisher (this only needs to be done once - subsequent tokens are picked up automatically). Instructions here.
   

New gLite 3.0 WMS at Glasgow

The Glaswegians now have a gLite 3.0 workload management server running on an SL3 machine.

Installation was reasonably easy; points of note being that a gLite user must exist, and that the gLite-WMSLB meta-package does not install the gLite-yaim-lb package. 

Jobs submitted on the Glasgow cluster with glite-wms-job-submit will now automatically use the local WMS server. Instructions on how to submit jobs via the WMS are available here and here.

DPM SRMv2 Tweaks

Got back from holiday to find that FDR transfers to Glasgow were failing with a "permission denied" error. Looking through the logs it seemed that the srmMkDir call was failing (unlike SRMv1 you need to make the path before transferring data into the SE).

However, it seems the default umask for DPM srmMkDir is 022, which leaves the directory unwritable. As I had tested the /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk path I left it unwritable by Mario's certificate.

I fixed this with a dpns-chmod, but then I wrote a script to patch up the space token directory tree area using DPM ACLs to ensure the directories are group writable by the production role.

You can get the script here: http://www.physics.gla.ac.uk/~graeme/atlas/scripts/atlas-dpm-token-fix.sh.

/dev/scotgrid - Buffer Overflow

Want more Glasgow news? We've also started a local logbook so we should have a changelog for the system. Not that we didn't keep every small detail documented before anyway. Ahem.

ganglia gmond

During some testing for next months outage, we'd rebuilt several nodes. One thing I noticed was that the NAT boxes had stopped reporting into ganglia. We'd had something similar before with an older version of gmond ignoring the 'mcast_if' parameter (hey, the alternative is to set up the routing tables) - the clunky 'copy over a known newer binary' wasn't going to be sustainable and the sf.net download only had i386.

Howver, kudos to the ganglia developers - one stupidly simple 'rpmbuild' and lo, a pile of x86_64 rpms ready to be copied into the cluster repo directory. Some cfengine voodoo and zip - all diskservers (including the new shiny 48T box) and nat boxes are reporting in. some of the graphs took a wobble but we're all present and correct with 168 machines in the pool.

YUM Updated Mystery Solved

Upgrading the cluster recently has brought nothing but trouble, with seeming conflicts between python-devel and python. Hacking and slashing through this on the UI I have now realised why. The conflict comes not from the x86_64 python RPM, but from the extra i386 python RPM we install to provide a 32bt python (needed by LCG modules like the python lfc plugin).

To avoid having to present the whole i386 repo to YUM, we'd picked out a few choice i386 packages and dropped them into the local cluster repo. However, in the meantime the i386 python had been updated and our version remained old and stale - causing the unsatisfiable dependency.

I have now updated and rebuilt the repo and things seem to upgrade smoothly at last.

However, roll on ye native 64bit middleware. This is too much of a pain at times...

ATLAS Space Token Roles

Following a discussion with ATLAS people I've now clarified that SRMv2 writing will happen using the atlas/Role=production VOMS role.

Therefore sites should restrict access to the ATLASMCDISK and ATLASDATADISK space tokens to this role. To do this, release and then recreate the space reservation:

# dpm-releasespace --token_desc ATLASDATADISK
# dpm-reservespace --gspace 10T --lifetime Inf --group atlas/Role=production --token_desc ATLASDATADISK

Greig's python snippet to list spaces is very helpful.

cluster glue

hmm. Freudian? I originally typed 'cluster clue' as the title.

Regular readers will be aware that we run both ganglia and cfengine. However even our wonderful rebuld system (YPF) doesn't quite close off all the holes in the fabric monitoring. case in point - reimaged a few machines and noticed that ganglia wasn't quite right. It'd copied in the right gmond.conf for that group of machines but hadnt checked that it was listed in the main gmetad.conf as a data_source,

Cue a short Perl script (soon to be available on the scotgrid wiki) to do a sanity check, but it;s this sort of non-joinedupness of all the bits that really annoys me about clusters and distributed systems.

Are there any better tools? (is Quattor the savoiur for this type of problem)

/rant

nagios event handlers

I've gone over to the Dark Side (no, not Python) and have just implemented my first nagios event handler - This *should* automatically try and fix the problem we have with our Dirvish backup scrips - namely that we end up with too many copies of the database dumps held.

So - cue nagios' event handlers - the only issue should be that nagios (and the event handler) runs as user nagios and most sysadmin stuff needs root. If you're willing to trust it to sudo then it should be OK.

DPM Storage Token Information Published

It was a bit fiddly, but with 4 fixes to Michel's script, Glasgow are publishing the space token information for ATLASDATADISK.

I documented the workarounds in the LCG Twiki.

perl-TermReadKey - missing in action

We had a ticket from a Zeus user unable to get a file off our DPM, while her Zeus colleagues could. I spend a long time checking the pool accounts, which were all fine, and checking the Zeus VOMS setup, which was also fine.

Finally, I looked in the logs for the grid-mapfile, where the culprit lay:

"Can't locate Term/ReadKey.pm in @INC..."

On two of the servers, disk034 and disk036, the perl-TermReadKey RPM was missing and it looks like the grid-mapfiles had not been rebuilt for a very long time - from the backups it looks like it was October when they were remade!

OK, nagios check: age of grid-mapfile and lcgdm-mapfile!

Down yum! Down!

As reported previously, We discovered that the nightly yum update was enabled on the servers.

The magic cfengine snippet to disable this is:

groups:
    # we don't want auto yum update stuff
    nonightlyyum = ( `/usr/bin/test -f /var/lock/subsys/yum` )

shellcommands:
    nonightlyyum::
        "/sbin/chkconfig yum  off" umask=022
        "/sbin/service yum stop" umask=022

The same check-for-some-enabled-subsys trick can be used to disable many of the periodic checks run on a standard SL install. (it's what 'service foo status' does for many of them)

SRM2.2 Configuration for FDR/CCRC

I configured Glasgow's DPM for the SRM 2.2 space tokens required by ATLAS for FDR/CCRC:

svr018:~# dpm-reservespace --gspace 10T --lifetime Inf --group atlas --token_desc ATLASDATADISK
ab0f1a60-59d6-4099-82fa-a17711678860

Easy, eh!

I notice that there is no dpm-listspaces command, which means that one has to grub around inside the database to find out what spaces are currently defined.

Two additional notes for other T2s:

1. Transfers from DDM are done using a vanilla atlas proxy for now (belonging to Mario Lassnig), so make sure the token is writable by the atlas group, not, e.g., atlas/Role=production.
2. All that is needed for CCRC is 3TB, however this is based on a 1 week cleaning cycle. If, like Glasgow, you have lots of space, then making the area bigger means the cleaning is not critical. (The space can be updated later with dpm-updatespace.)
   

YAIM stale...

Another problem I found with configuring the new VOs is that the reconfiguration of the information system on the CE failed. Mike had run config_gip, as has been done forever, but it did nothing. So queues and access rights for the VOs were not published.

There are new YAIM functions, like config_gip_ce, as well as a whole different way of structuring site-info.def. I fiddled with various options in the site-info.def file and even in the YAIM function itself, but I couldn't get it to work properly at all.

In some desperation I hacked the ldif file in the end, which is not a long term option; we really need now to look at the whole way that we structure YAIM for the site (as well as plan migrations to SL4 for BDII, CE and DPM...).

OPS goes dark...

Following enabling other VOs yesterday, DPM broke for ops. The error messages were as cryptic as ever:

httpg://svr018.gla.scotgrid.ac.uk:8443/srm/managerv1: Unknown error

And in the SRM logs:

01/29 12:29:02 14830,3 srmv1: SRM02 - soap_serve error : Can't get req uniqueid
01/29 12:05:18 14830,0 srmv1: SRM02 - soap_serve error : CGSI-gSOAP: Could not find mapping for: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=samoper/CN=582979/C
N=Judit Novak

The error seemed to correspond to re-running config_mkgridmap yesterday, however, as Judit (and other ops people) were in the grid-mapfile I was very confused.

Eventually, staring at the lcgdm-mkgridmap.conf I realised that the ops VO was only configured to get voms information from the deprecated lcg-voms.cern.ch server. I reconfigured to get the information from voms.cern.ch and it started to work.

The think I cannot fathom is how is kept working for so long - we have always had lcg-voms.cern.ch as the server for ops.

I updated the ops entry on the GridPP Wiki.

As usual the things which made this much harder then it should have were:

1. It only affected the ops VO (not dteam, atlas or pheno which we can test).

2. The error message was, as usualy, cryptic and unhelpful.

Keeping up with the Jones'

Well we recently had an incident with our NFS server for the cluster (home / software) locking up and needing a cold power cycle. Due to $vendors setup this takes aaaages (in the order of 20 mins) to go through the BIOS selfcheck (hangs at 053C). $vendor would like to poke around system and perhaps perform bios upgrade. Hmm. Oh well, all 10 disk servers are identical so we'll just drain one down and play - it also gives us chance to upgrade (from 1.6.5) to the latest 1.6.7-mumble DPM.


... or so we thought.

disk032:~# rpm -qa | grep DPM
DPM-gridftp-server-1.6.7-1sec
DPM-rfio-server-1.6.7-2sec.slc3
DPM-client-1.6.7-2sec.slc3

"Thats odd - Graeme have you updated these?" nope - Turns out that yum.nightly cron was auto updating on both the disk servers and some of the grid servers... Gaaah. clickity click and we're all ready to play.

In the meantime, dpm-drain migrated most of the data off the server to the other stash of disks but there were still 69 files that failed with 'Internal error' - Am looking through the DB to try and see if I can pull any more info out

Take Me To Your VO...

As noted preciously, we have enabled mice, vo.scotgrid.ac.uk and vo.nanocmos.ac.uk at Glasgow. Mike worked his way through the documentation, which was a out of date because it hadn't incorporated the changes in VO pool account management (which has become much better) which Andrew implemented.

We also plan to setup VOs much more promptly now for local user groups. Now that we have a scotgrid VOMS server this will be much easier.

The wiki has all the updated details.

Resource Broker Blues

Our resource broker was down for the weekend as the network service stalled. Root cause turned out to be a bit of over aggressive cleaning from cfengine. I had wanted to do a better job of cleaning up the /tmp area in the cluster - each worker node had hundreds of condor_g working directories lying around - with nothing in them. cfengine's "tidy" leaves directories alone by default and only cleans files. So I enabled the "rmdirs=sub" option - works beautifully, gets rid of all the cruft in /tmp. So pleased was I that I disengaged by brain and set this option on for /home as well - good idea to clean up those old gass cache areas, isn't it? Well, almost - unfortunately /home has subdirs which are the node pool account home areas and unused pool accounts fall into the clean me up category. All the untouched pool areas then vanished.

This caused a number of people to start getting "unspecified grid manager errors" on globus-job-runs, as well as wiping out the edguser home area on the RB which caused the network server to go into crisis.

It didn't take long to work out what had happened, but fixing it took a while as the resource broker seemed to be quite huffy afterwards.

The only plus side was that I enabled the mice, scotgrid and nanocmos VOs on the RB.

Happy New Year ScotGrid - now with added ECDF...

Well, we didn't get it quite as a Christmas present, but the combined efforts of the scotgrid team have managed to get ECDF green for New Year.

In the week before Christmas Greig and I went through period of intensive investigation as to why normal jobs would run, but SAM jobs would not. Finding that jobs which fork a lot, like SAM jobs, would fail was the first clue. However, it turned out not to be a fork or process limit, butn a limitation on the virtual memory size which was the problem. SGE can set a VSZ limit on jobs, and the ECDF team have set this to 2GB, which is the amount of memory they have per core. Alas for jobs which fork, virtual memory is a huge over estimate of their actual memory usage (my 100 child python fork job registers ~2.4GB of virtual memory, but uses only 60MB of resident memory). That's a 50 fold over estimate of memory usage!

As SAM jobs to fork a lot, they hit this 2GB limit and are killed by the batch system, leading to the failures we were plagued by.

A work around, suggested by the systems team, was to submit ops jobs to the ngs queue, which is a special short running test queue (15 min wall time) which has no VSZ limit on it.

Greig modified the information system to publish the ngs queue and ops jobs started to be submitted to this queue on the last day before the holidays.

Alas, this was not quite enough to get us running. We didn't find out until after new year that we also needed to place a specify a run time limit of 15 minutes on the jobs and submit them to a non-standard project. The last step required me to hack the job manager in a frightful manner as I really couldn't fathom how the perl (yuk!) job manager was supposed to set the project - in fact even though project methods existed they didn't seem to emit anything into the job script.

Finally, with that hack made this morning, ECDF started to pass SAM tests. A long time a coming, that one.

The final question, however, is what to do about this VSZ limit. The various wrappers and accoutrements which grid jobs bring mean that before a line of user code runs there are about 10 processes running, as 600MB of VSZ has been grabbed. This is proving to be a real problem for local LHCb users, because ganga forks a lot and also gets killed off. Expert opinion is that VSZ limits are just wrong.

We have a meeting with the ECDF team, I hope, in a week, and this will be our hot topic.

Big thanks go to Greig for a lot of hard work on this, as well as Steve Traylen, for getting us on the right track, and Kostas Georgiou, for advice about the perils of VSZ in SGE.