assimilation

I'd noticed that over the last month the load on our DPM headnode had been higher than before we switched on MonAMI checking of the DPM. Of course I instantly blamed the developer of said product. However, I disabled monami to prove that the load went down and lo... no change. Hmm.

I then started working out how to optimise the MySQL memory usage - we have about a 1.8G ascii file when I do a mysqldump -A and yet the innodb file takes up a whopping 4.4G on disk with tiny constantly rolling transaction logs of 5M.

As paul was here at cern (it's the WLCG Workshop this week) we got together to hammer out some changes to our implementation. When we logged onto svr018 (The DPM Headnode) I noticed that monami was running again. Turns out that cfengine was "helpfully" restarting the process for us. Grr.

So, an evening of infrastructure management changes:
- we had i386 monami rpms installed - we'd hard coded the repo path rather than using the $basearch variable in our local mirror.
- we had to ensure that we had backup=false in cfengine - where we had a config_dir directive (such as /etc/monami.d and /etc/nrpe.d) the applications were often trying to use someconfigfile.cfg and someconfigfile.cfg.cfsaved - ditto cron.d etc etc.
- we were sometimes trying to run 64 bit binaries on 32 bit architectures as we'd copied them straight from cfagent (normally nrpe monitors) - We've now using $(ostype) in cfagent which expands to linux_x86_64 and linux_i686 on our machines. Although cfengine sets a hard class of 32_bit and 64_bit but you can't use that in a variable.
- we now have the 'core' nrpe monitors (disk space, load, some process checks) installed on ALL servers not just the workernodes. Ahem. Thought we'd implemented that before.
- we've upgraded to the latest CVS rpm of monami on some nodes and we've got grooovy mysql monitoring. - oh and the load's gone down too.

Oh no, not again...

We went though a little rash of SAM test failures last night. This turned out to be an LHCb user who was submitting jobs which filled up the scratch area on the worker nodes and turned them into blackholes.

Obligatory GGUS ticket was raised.

We do alarm against disk space filling up on the worker nodes, but it was still 4 hours before action was taken and the nodes set offline before being cleaned. In that time an awful lot of jobs were destroyed. Make me think we might want to automate the offlining of nodes which run out of disk space, pending investigations.

Splunk / nagios / logrotate

Well, I upgraded to nagios3 this evening on the cluster and noticed it had a new enable_splunk_integration option in the cgi.cfg - I'd looked at splunk before and thought 'hmm, nice idea, not sure it'll work with the grid stuff' but decided to give it a whirl

first up - nagios gotchas - We had the dag rpm installed which hasn't been updated to the 3.0 let alone the 3.0.1 release so went for the manual compile option. Discovered that the (gd|libjpeg|libpng)-devel packages weren't installed - quickly fixed by yum.

took the ./configure line from the spec as a guide - however it managed to splat the cgi's into /usr/sbin rather than /usr/lib64/nagios/cgi - thanks :-( soon found em and moved em round. seems to be working OK - not installed the newer wlcg monitors yet - thats the next task.

Splunk - looks flash but is it any good? There's no sign of any educational pricing on their website and the 'free' version has one HUGE weakness - no user authorisation / login. Temp workaround of some iptables rules to reduce risk and had a play. Defined /var/log on our central syslog server as a datasource and watched it go.

well, sort of... it promptly filled /opt/splunk as it makes an indexed copy of anything it finds, - I think for a real install we'd need some new space on a disk. secondly it quicky swallowed more than its 500M/day 'free' allowance - grabbed a 30day trial licence of the enterprise version and lo it now complains that I've had 2 licence violations of over 5G/day indexed. Harumph.

not sure if this would settle down once it goes through the backlog of the archived logfiles - perhaps if I implement only a syslog FIFO for it it'd be happier. Also we have the 'traditional' logrotate style of .1 .2 .3 etc rather than the more dirvish friendly dateext option - we should really swap... if the RHEL logrotate supports it :-/

"rpm -q logrotate --changelog" doesnt mention it although its fixed in fedora

The other issue is that splunk thrashes the box as it indexes, and it's just stopped as its filled the disk again. Ho Hum.

A long time coming: UKI-SCOTGRID-ECDF on APEL

So, yes, it's probably taken a little longer than it might have, but UKI-SCOTGRID-ECDF is now publishing all its accounting data back to early January.
Here

Of course, ops has a disturbingly high share of the Grid usage at the moment, but hopefully we will start to get ATLAS (and maybe even LHCb) jobs filtering in in the near future...

ECDF running for ATLAS

ECDF have now passed ATLAS production validation. The last link in the chain was ensuring that their SRMv1 endpoint was enabled on the UK's DQ2 server at CERN - this allows the ATLAS data management infrastructure to move input data to ECDF.

After that problem was corrected this morning the input data was moved from RAL, a production pilot picked up the job and ran it then the output data was moved back to RAL.

I have asked the ATLAS production people to enable ECDF in the production system and I have turned up the pilot rate to pull in more jobs.

We had a problem with the software area not being group writable (for some reason Alessando's account mapping changed), but this has now been corrected and an install of 14.0.0 has been started.

It's wonderful to now have the prospect of running significant amounts of grid work on the ECDF resource. Well done to everyone!

Only one bite at the cherry...

I have modified the default RetryCount on our UIs to now set zero retries. Automatic retries were actually working quite well for us when we were losing a lot of nodes to MCE errors (in the days before the upgrade to SL4, x86_64) - users' jobs would automatically rerun if they got lost and there was no need for them to worry about failures. However, recently we see users submitting more problematic jobs to the cluster - some which fail to start at all, some which run off into wallclock limits, others which stall half way through. Often we have to gut the batch system with our special spoon and in this case having to do it four times because the RB/WMS keeps resubmitting the job is less then helpful.

For once cfengine's editfiles stanza was useful and a simple:

ui::
{ /opt/glite/etc/glite_wmsui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}
{ /opt/egd/etc/edg_wl_ui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}

got the job done.

Emergency Outage

Due to a vulnerability to the security flag in the ipv4 header, we will be taking uki-scotgrid-glasgow offline today for an upgrade. In order to minimise downtime we shall be rebooting all worker nodes simultaneously rather than draining queues.

We aim to have this work completed by 12:00 midday today.

Please see RFC 3514 for more details. We advise other sites to perform this upgrade asap.

brain dead batch systems

why oh why are some of the batch utilities so brain dead? simple case of 'qstat -r' should show who's running jobs right? wrong as it outputs based on a fixed 8 character width for username. doh. so 'prdatlas' and biomed06 seem to be busy. Well not quite as if I do a qstat -f | egrep " e(group|user) " | sort -u I see that it's prdatlas028 and several biomed06? users. grr...

I may install Job Monarch from sara but in the meantime it'll be some hacky PHP to parse the outputs a bit more cleanly

Also, despite having 493 running jobs at the moment (we're down on our capacity as I'm still doing a rolling upgrade to SL4.6 and a new kernel) there are a very small number of users on the system

svr031:~# qstat -f | grep euser | sort -u | wc -l
14

not good, Especially if they decide to take a break.

p p p pick up a pakiti

We've been using pakiti at Glasgow for some time now for keeping an eye on which nodes are out of date. One minor niggle is that it doesn't keep track of the grub default kernel (ie what should come in on reboot) compared to the running kernel

We already had a v simple shell script that did that:

 pdsh -w node[001-140] chkkernel.sh | dshbak -c
----------------
node[001,005,007,014,016-020,022-023,025,028,031-061,063-085,087-090,092,095-096,098-101,103-104,106-107,109-110,113,115,118-120]
----------------
 Running: 2.6.9-67.0.7.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status OK
----------------
node[062,091,093-094,097,102,105,108,111-112,114,116-117,121-127,129,131,133-134,136-139]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[003,009,011,013,015,021,027,029]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[128,130,132]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[002,004,006,010,012,024,026,030,140]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[086,135]
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status OK
----------------
node008
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error

but I finally got it integrated with some patching - see http://www.scotgrid.ac.uk/wiki/index.php/Pakiti

result - pretty green / red status on the "default kernel' column.

The patches have been emailed to Romain so may well appear upstream eventually

Edinburgh as an LHCb Tier-1?

I've just been accused (jokingly, I hope) of trying to turn Edinburgh into LHCb's 7th Tier-1. The attached plot shows the recent data transfers that I have been running into our dCache. The rates are good (~35MB/s), but not particularly special. However, against a background of zero, it certainly made LHCb jump up and take notice ;) Maybe this will convince them that Tier-2s really can be used for analysis jobs...

I should note that during these transfers one of the dCache pools was about to melt (see below). I've since reduced the max number of movers on each pool to something more reasonable. For the tests, I created a small application that spawned ~50 simultaneously lcg-cp's that were all transferring files from CERN CASTOR to Edinburgh. Who needs FTS when you've got DIRAC and lcg_utils? Now all I need is someone else's proxy and I'll never be caught... ;) But, on a serious note, I suppose this does show that people can create tools to abuse the system and get round the official FTS channels, which could impact the service for other users.

The curse of 79...

Since the dawn of the Glasgow cluster we have been cursed with a low level of globus 79 errors. We did not understand these well, but always believed that they were caused by a confusion in the gatekeeper, where the X509 authentication seemed to suffer a race condition and get muddled between users.

However, since upgrading to an SL4 CE and installing it on a different machine we still get these cropping up (an example).

The GOC Wiki suggests this can be caused by firewall trouble or an incorrect GLOBUS_TCP_PORT_RANGE. Now, this is (and was) correctly defined on both machines to be the standard 20000-25000. However, I have decided to change it to 50000-55000 in case we are tripping some generic nasty filter somewhere else on campus.

Since I did that, last night, we haven't had a 79 error - however this proves nothing so far as we can easily go for a week without one of these happening.

I also contacted the campus networking people to ask if there was any known port blocks in this range.

Data Management and MICE

I had a chat to one of our MICE PhD students a couple of weeks ago and I was explaining how to use EGEE data management (SRMs, LFCs, FTS, lcg utils, etc.). His comment afterwards was "I didn't know I was going to do a PhD in data management...".

The problem is that all these tools are very low level, so any user community has to build a higher level infrastructure on top of this. Obviously the LHC experiments have done this extensively, but it is frustrating that there is no simple generic data management service for smaller VOs who lack the resources of the larger VOs.

I wonder if this accounts for the popularity of SRB in other communities? It may have some limitations, but it clearly offers a higher level data storage, cataloging and metadata service which must be attractive for smaller user communities. Surely there is a potential project to try and tie all of the EGEE components into a sensible data management system?

Durham - SL4 Install Success!

Durham took the plunge earlier this week to upgrade the CE, SE and all nodes to SL4.6... with success! After our preparation was delayed slightly due to a small UPS failure, we set about installing cfengine to handle the fabric management. This took a little longer than expected but our patience has paid off and it eases the pain of setup and config of clusters. Using the normal RedHat Kickstart to get a base install of SL4.6, we then hand the rest of the setup to cfengine to work its magic (install extra packages, setup config files, run YAIM etc).

Firstly installing a Worker Node was relatively straight forward. Then came the CE along with torque, PBS and the site BDII setup. Thanks to Graeme for help checking our site was working and publishing as expected.

We unexpectedly hit a firewall issue as I had renamed the CE from the old "helmsley.dur.scotgrid.ac.uk" to "ce01.dur.scotgrid.ac.uk"... though I had preserved the IP address. Not what I expected but our network guys were able to fix the rules and we were operational again.

Then the SE followed very quickly afterwards, cfengine and YAIM working their magic very successfully. The procedure was as simple as 1) dump of the database, 2) install SL4.6, 3) Let cfengine do its stuff for a base install, 4) restore the database, 5) Run YAIM. Simple!

Just one gotcha was trying to change the NFS mounted home directories to be local to the nodes. This fails with an error trying to copy the globus-cache-export files. Due to time constraints we have re-enabled the NFS home dirs... but I'm sure this will be simple to fix and I'll look at it next week.

Fair shares and queue time will need reviewing but in all a busy and successful few days. We're passing SAM tests and I've seen Phenogrid, Atlas and Biomed running jobs. Still the UI and a disk server to do, but with cfengine in place, this should be relatively straight forward and will require no downtime.

Another ECDF/Grid requirement mismatch.

While ECDF is, in principle, functional and capable of running jobs, this is a bit useless if no-one can see if you're doing it. So, in the face of APEL accounting still not working for the cluster, I had another look.

There were two problems:
Firstly, the sge account parser was looking in the wrong directory for SGE accounting logs - this fails silently with "no new records found", so I didn't notice before. The configured location actually was correct when I set the thing up, but the mount point had been moved since (as the CE is not on the same box as the SGE Master, we export the SGE account directory over NFS to the CE so it can parse them) with no indication that anything was wrong.

Secondly, after I fixed this...
It turns out that the APEL java.lang.OutOfMemoryError strikes again for ECDF.
The ECDF systems team configure the SGE accounting to roll over accounting logs on a monthly basis. Unfortunately, this leads to rather large accounting files:
# ls --size /opt/sge/default/common/acc* --block-size=1M
1543 /opt/sge/default/common/accounting

(yes, gentlemen and ladies, that is a one and a half gig accounting file...and we're only half-way through the month. The archived accounting logs tip the scales at around a quarter to half a gig compressed, but they compress rather efficiently so the "true" size is much larger - up to 10x larger, in fact.)

I suspect the next step is to arrange some way of chopping the accounting file
into bitesized chunks that the APEL log parser is capable of swallowing.
The irony is that we already parse the accounting logs internally using a thing called ARCo - I've not seen any indication that it would be easy to get APEL to understand the resulting database, though.

Dem info system blues

I fixed a problem on the CE information system tonight. YAIM had gone a little screwy and incorrectly written the lcg-info-dynamic-scheduler.conf file, so I had added the lrms_backend_cmd parameter myself as:

lrms_backend_cmd: /opt/lcg/libexec/lrmsinfo-pbs -h svr016.gla.scotgrid.ac.uk

Adding the host seemed sensible as the CE and the batch system don't run on the same node, right? Wrong! the host paramater ends up being passed down to "qstat -f HOST" which is a broken command - we ended up with zeros everywhere for queued and running jobs and, consequently a large stack of biomed jobs we are unlikely ever to run.

I raised the obligatory GGUS ticket: https://gus.fzk.de/pages/ticket_details.php?ticket=33313

To VOMS or not to VOMS? That is the question (for LCMAPS...)

Our advice to local users of the cluster has traditionally been to not use VOMS credentials. This is to ensure that they are mapped in the batch system to their local account, rather than to a pool account from their VOMS attributes (mappings to local accounts are maintained by the grid-mapfile-local file). In the default configuration of LCMAPS VOMS pool account mappings are made before the grid-mapfile, which is now just a fall back.

However, I could not simply reverse the order of the LCMAPS plugins as this would undo all the good which VOMS brings and move everyone back to a single fixed or pool account mapping no matter what their VOMS credentials (this would probably have affected me worse than anyone as I flit between atlas, atlas/Role=production, vo.scotgrid.ac.uk and dteam!).

So, for local users grid-proxy-init seemed to be the way to go, even if I knew this would come back and be a problem later. However, later became earlier as soon as I started to test the gLite-WMS - here it turns out you must use a VOMS proxy. Simple grid proxies just don't work anymore.

Finally, puzzling over the very poor LCMAPS documentation, and staring at the configuration script I managed to solve the problem by:

1. First running a local account plugin against a grid-mapfile which only contains our local user accounts.
2. Then running the VOMS plugins as usual.
3. Finally, running the grid-mapfile plugin, against the usual /etc/grid-security/grid-mapfile.

This was almost too easy to be true - and indeed it turns out not to quite be that simple as you hit a bug in LCMAPS that you cannot use a module twice - so having lcmaps_localaccount.mod twice is not possible. However, it turns out that one can do it if the module is renamed and physically copied. This works, so we now have an lcmaps_localaccount.mod and a lcmaps_localuseraccount.mod - exactly the same bytes, different names! (To be strictly accurate we have two copies of liblcmaps_localaccount.so.0.0.0, to which these links point.)

And, in the end, I was able to keep myself out of the local user grid-mapfile, so I have the full array of VOMS roles for myself, while the local users are cosily tucked up in their local account areas.

Upgrade to gLite 3.1 - Notebook

It was well over a year since we'd done a "from the top" install of the CE, so a few things were different:

• The information system has been re-branded. It's now configured in /opt/glite/etc/gip, although many of the plugins are still running from /opt/lcg.
• The CE information system is upgraded to use the BDII (on 2170).
• The site BDII also now uses a wrapper script to get all information, rather than coding the list of GRISs/BDIIs to query (GIP file:///opt/glite/libexec/glite-info-wrapper).
  
• LCAS and LCMAPS now also run out of /opt/glite.
• Pool account mappings are now done to a random pool account, rather than the "next" free one. In addition the hardlink syntax used for assigning a pool account to a DN has changed slightly (using colons to indicate VOMS attributes after the DN).

Funeral March for the Lost CE

So, here's the post mortem on the CE hard crash on Wednesday last. About 1700 the load on the CE started to ramp up and it quickly rose to almost 100. I could see this happening just as I was about to go home (typical!) so I started to indulge in a frantic bout of process killing to reduce load and bring the CE back under control. However, despite my best efforts, the CE crashed hard at 1800 (gap in the ganglia plot).

When the machine rebooted, the gatekeeper restarted and again the load began to rise. I then went through a frantic couple of hours trying to do everything I could to reduce the load and try an get the CE back on an even keel - this was made very hard by the fact that with load averages quickly rising to 60+ the machine was extremely sluggish.

I shut down R-GMA, turned off the mail server to no avail. I killed off queued jobs in the batch system, even got as far as disabling VOs, and banning users whose jobs I had cancelled. I even got so desparete as to firewall the gatekeeper from all but the ScotGrid RB! But although I coud slow down the load increase by doing this, by 10pm it became clear that something dreadful had happened to the gatekeeper. Every gatekeeper process which was forked stalled, consuming CPU and managing to do absolutely nothing. As there was no response, the RB then contected the CE again, forking off another gatekeeper and the march to death continued. If I reduced the number of users able to contact the CE this slowed down the rate of resource exhaustion, but could not stop it. Clearly something utterly evil had happened to the gatekeeper state.

At this point I became convinced that nothing could be done to save the remaining queued or running jobs and that the site was going down. I started to think instead about moving our March downtime forwards, to do the SL4 upgrades, and to prise the CE and the batch system apart. And of course, that is just what we did at the end of last week.

Acrobat ate my disk servers!

Glasgow is finally out of downtime. GS worked his grid-fu and managed to upgrade lots to SL4 - Admittedly some things (RGMA) weren't a goer. APEL Accounting could be broken for a while as we've now split the CE (new home = svr021) and the Torque server (still on svr016). My 'simple' job was to take care of the DPM servers...

Simple enough, we hacked into the YAIM site-info.def stuff and separated things out into services/ and vo.d/ - easy. Few gotchas as cfengine was once again reluctant to create the symlinks on the target nodes (however creating the symlinks on the master and replicating those works fine) which we thought might be fixed by an upgrade of cfengine from 2.1.22 to 2.2.3. Big mistake. it broke HostRange function of cfengine.

so we have

 dpmdisk = ( HostRange(disk,032-036) HostRange(disk,038-041) ) 

but cfengine complained that

SRDEBUG FuzzyHostParse(disk,032-041) succeeded for disk033.gla.scotgrid.ac.uk
SRDEBUG FuzzyHostMatch: split refhost=disk033.gla.scotgrid.ac.uk into refbase=disk033.gla.scotgrid.ac.uk and cmp=-1
SRDEBUG FuzzyHostMatch(disk,032-041,disk033.gla.scotgrid.ac.uk) failed 

now I'm not sure if this is due to the problem of short hostname vs FQDN - I've hit a similar issue when I want to copy iptables configs off -

$(skel)/nat/etc/iptables.$(host) mode=0600 dest=/etc/iptables define=newiptables type=sum 

needs iptables.host.gla.scotgrid.ac.uk not just iptables.host on the master repo.

Anyway, this all seems trivial compared to the hassle with the latest SLC 4X that got mirrored up to the servers overnight (the disk servers run SLC4 rather than SL4 as the Areca raid card drivers are compiled in) - dpm-queryconf kept failing with

send2nsd: NS002 - send error : No valid credential found

and yet the certificates were there and valid - openssl verify ... returned OK, ddates were valid, NTP installed etc. The dpm log showed

dpm_serv: Could not establish security context: _Csec_recv_token: Connection dropped by remote end ! 

The really frustrating thing was that the server that I installed from home while munching breakfast (all hail laptops and broadband) worked fine, but those I installed (and reinstalled) later in the office were broken. [hmm. is this a sign that I should stay at home in the mornings and have a leisurely breakfast?]

Puzzling was the fact that the broken servers had more rpms installed than the working ones. - I eventually resorted to installing strace on both boxes and diffing the output of 'strace dpm-qryconf'

the failing one had a big chunk of

open("/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xffff9028)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xffff9028)         = -1 ENOENT (No such file or directory)
open("/lib/libstdc++.so.6", O_RDONLY)   = -1 ENOENT (No such file or directory)
stat64("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0xffff9028)      = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/usr/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/sse2", 0xffff9028)     = -1 ENOENT (No such file or directory)
open("/usr/lib/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

whereas the working one didn't call this at all.

I was also bemused as to why acroread had been installed on the server and more annoyingly why I couldn't uninstall,

Yep - someone (step up to the podium Jan Iven) had mispackaged the SLC acroread 8.1.2 update...

rpm -qp ./acroread-8.1.2-1.slc4.i386.rpm --provides
warning: ./acroread-8.1.2-1.slc4.i386.rpm: V3 DSA signature: NOKEY, key ID 1d1e034b
2d.x3d  
3difr.x3d  
ADMPlugin.apl  
Accessibility.api  
AcroForm.api  
Annots.api  
DVA.api  
DigSig.api  
EFS.api  
EScript.api  
HLS.api  
MakeAccessible.api  
Multimedia.api  
PDDom.api  
PPKLite.api  
ReadOutLoud.api  
Real.mpp  
SaveAsRTF.api  
SearchFind.api  
SendMail.api  
Spelling.api  
acroread-plugin = 8.1.2-1.slc4
checkers.api  
drvOpenGL.x3d  
drvSOFT.x3d  
ewh.api  
libACE.so  
libACE.so(VERSION)  
libACE.so.2.10  
libACE.so.2.10(VERSION)  
libAGM.so  
libAGM.so(VERSION)  
libAGM.so.4.16  
libAGM.so.4.16(VERSION)  
libAXE8SharedExpat.so  
libAXE8SharedExpat.so  
libAXE8SharedExpat.so(VERSION)  
libAXSLE.so  
libAXSLE.so  
libAXSLE.so(VERSION)  
libAXSLE.so(VERSION)  
libAdobeXMP.so  
libAdobeXMP.so  
libAdobeXMP.so(VERSION)  
libAdobeXMP.so(VERSION)  
libBIB.so  
libBIB.so(VERSION)  
libBIB.so.1.2  
libBIB.so.1.2(VERSION)  
libBIBUtils.so  
libBIBUtils.so(VERSION)  
libBIBUtils.so.1.1  
libBIBUtils.so.1.1(VERSION)  
libCoolType.so  
libCoolType.so(VERSION)  
libCoolType.so.5.03  
libCoolType.so.5.03(VERSION)  
libJP2K.so  
libJP2K.so  
libJP2K.so(VERSION)  
libResAccess.so  
libResAccess.so(VERSION)  
libResAccess.so.0.1  
libWRServices.so  
libWRServices.so(VERSION)  
libWRServices.so.2.1  
libadobelinguistic.so  
libadobelinguistic.so  
libadobelinguistic.so(VERSION)  
libahclient.so  
libahclient.so  
libahclient.so(VERSION)  
libcrypto.so.0.9.7  
libcrypto.so.0.9.7  
libcurl.so.3  
libdatamatrixpmp.pmp  
libextendscript.so  
libextendscript.so  
libgcc_s.so.1  
libgcc_s.so.1(GCC_3.0)  
libgcc_s.so.1(GCC_3.3)  
libgcc_s.so.1(GCC_3.3.1)  
libgcc_s.so.1(GCC_3.4)  
libgcc_s.so.1(GCC_3.4.2)  
libgcc_s.so.1(GCC_4.0.0)  
libgcc_s.so.1(GLIBC_2.0)  
libicudata.so.34  
libicudata.so.34  
libicui18n.so.34  
libicuuc.so.34  
libicuuc.so.34  
libpdf417pmp.pmp  
libqrcodepmp.pmp  
librt3d.so  
libsccore.so  
libsccore.so  
libssl.so.0.9.7  
libssl.so.0.9.7  
libstdc++.so.6  
libstdc++.so.6(CXXABI_1.3)  
libstdc++.so.6(CXXABI_1.3.1)  
libstdc++.so.6(GLIBCXX_3.4)  
libstdc++.so.6(GLIBCXX_3.4.1)  
libstdc++.so.6(GLIBCXX_3.4.2)  
libstdc++.so.6(GLIBCXX_3.4.3)  
libstdc++.so.6(GLIBCXX_3.4.4)  
libstdc++.so.6(GLIBCXX_3.4.5)  
libstdc++.so.6(GLIBCXX_3.4.6)  
libstdc++.so.6(GLIBCXX_3.4.7)  
nppdf.so  
prcr.x3d  
tesselate.x3d  
wwwlink.api  
acroread = 8.1.2-1.slc4

Yep, thats right - RPM had decided that acroread was a dependency. Grr. Workaround - remirror the CERN SLC repo, (no they hadn't updated since), manually remove the offending rpm, and rebuild the metadata with 'createrepo'

Then make sure that the nodes were rebuilt and only ever looked to our local repository rather than the primary cern / dag ones. (thanks to kickstart %post and cfengine)

Finally we got yaim to *almost* run - it was failing on lcmaps and grid-mapfile creation (fixed by unsupporting a certain VO)
Easy fix in comparison. Anyway - DPM up n running and seems OK. Roll on the next SAM tests.... (or real users)

Phew.

Grid Middleware should not be this hard to install!

Durham - DPM v1.6.7, Space Tokens etc

Well this is the first of hopefully many posts from Durham. So firstly a quick update, well Durham seem have been ticking along nicely - with the exception of a few network and power outages over the last few months. SAM tests are passing and ATLAS, Pheno and many other VOs are successfully running jobs. Versions of the lcg software is a little out of date in places – but this is work in progress.

So with a little encouragement and help from Greig I finally took the plunge and upgraded our SE to DPM v1.6.7. After getting the yum repositories correct, it was a case of stopping the daemons, running yum update, making the DPM schema changes (we were upgrading from an old version of DPM), and then restarting the daemons... done... or so we thought!

Everything was working, file copies in and out of our SE, reserving space tokens, etc... the only gotcha was that we were publishing "GlueSAStateAvailableSpace: 0"... which wasn't true. After a little investigation, and with the help of Greig, we noticed that /opt/lcg/var/gip/plugin/lcg-info-dynamic-se was pointing to a beta version of lcg-info-dynamic-dpm. Changed this to remove the beta and bingo... all working correctly.

I have then setup to publish the space tokens as shown here here, and all is done.

A good days work, until a major network outage at the JANET/NorMAN level knocked us out most of the night... typical. We seem to have recovered now though so we should be back on track.

Thusday Night Status Update

Quick summary of where we are right now:

* YAIM configuration updated and rationalised.

* Batch system has been upgraded to Torque 2.1.9/Maui 3.2.6, running on SL4 x86_64.

* Queues have been reduced to 4, open to most VOs, with queue lengths of 30m, 6h, 3d and 7d.

* CE has been moved to svr021, again running SL4 x86_64.

* Information system has been reconfigured to new gLite versions. After a minor wobble on the CE seems to be working just fine.

* DPM headnode has been upgraded to SL4 x86_64.

And job submission works:

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
Submitted:          Thu Feb 21 23:22:30 2008 GMT
*************************************************************

...

================================================================================

   JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/gla012_sx54e7252PGtxJtB4Y2mIg

================================================================================

And there was much rejoycing.

Significant work remains for tomorrow, but very good progress being made.

Glasgow Downtime

As announced on GOCDB We're taking UKI-SCOTGRID-GLASGOW down until 17:00 localtime tomorrow (friday) to bring forwards the maintenance we'd planned for March. This was due to an unexpected CE failurelast night that meant the queues were empty.

DPM and ATLAS Space Tokens

So, here's the definitive guide to enabling space tokens for ATLAS:

1. Assign the token to atlas/Role=production using the dpm-reservespace command. (Example here.)
2. Create the directory where DDM will put the files, which is the normal path for ATLAS, plus the name of the spacetoken in lower case. e.g.


• dpns-mkdir /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


3. Now change group ownership of this directory to atlas/Role=production, chmod it to 775 and finally add two ACLs which will mean the entire tree will be writable by production roles:




• dpns-chgrp atlas/Role=production /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-chmod 775 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk




4. Enable the space token publisher (this only needs to be done once - subsequent tokens are picked up automatically). Instructions here.
   

New gLite 3.0 WMS at Glasgow

The Glaswegians now have a gLite 3.0 workload management server running on an SL3 machine.

Installation was reasonably easy; points of note being that a gLite user must exist, and that the gLite-WMSLB meta-package does not install the gLite-yaim-lb package. 

Jobs submitted on the Glasgow cluster with glite-wms-job-submit will now automatically use the local WMS server. Instructions on how to submit jobs via the WMS are available here and here.

DPM SRMv2 Tweaks

Got back from holiday to find that FDR transfers to Glasgow were failing with a "permission denied" error. Looking through the logs it seemed that the srmMkDir call was failing (unlike SRMv1 you need to make the path before transferring data into the SE).

However, it seems the default umask for DPM srmMkDir is 022, which leaves the directory unwritable. As I had tested the /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk path I left it unwritable by Mario's certificate.

I fixed this with a dpns-chmod, but then I wrote a script to patch up the space token directory tree area using DPM ACLs to ensure the directories are group writable by the production role.

You can get the script here: http://www.physics.gla.ac.uk/~graeme/atlas/scripts/atlas-dpm-token-fix.sh.

/dev/scotgrid - Buffer Overflow

Want more Glasgow news? We've also started a local logbook so we should have a changelog for the system. Not that we didn't keep every small detail documented before anyway. Ahem.

ganglia gmond

During some testing for next months outage, we'd rebuilt several nodes. One thing I noticed was that the NAT boxes had stopped reporting into ganglia. We'd had something similar before with an older version of gmond ignoring the 'mcast_if' parameter (hey, the alternative is to set up the routing tables) - the clunky 'copy over a known newer binary' wasn't going to be sustainable and the sf.net download only had i386.

Howver, kudos to the ganglia developers - one stupidly simple 'rpmbuild' and lo, a pile of x86_64 rpms ready to be copied into the cluster repo directory. Some cfengine voodoo and zip - all diskservers (including the new shiny 48T box) and nat boxes are reporting in. some of the graphs took a wobble but we're all present and correct with 168 machines in the pool.

YUM Updated Mystery Solved

Upgrading the cluster recently has brought nothing but trouble, with seeming conflicts between python-devel and python. Hacking and slashing through this on the UI I have now realised why. The conflict comes not from the x86_64 python RPM, but from the extra i386 python RPM we install to provide a 32bt python (needed by LCG modules like the python lfc plugin).

To avoid having to present the whole i386 repo to YUM, we'd picked out a few choice i386 packages and dropped them into the local cluster repo. However, in the meantime the i386 python had been updated and our version remained old and stale - causing the unsatisfiable dependency.

I have now updated and rebuilt the repo and things seem to upgrade smoothly at last.

However, roll on ye native 64bit middleware. This is too much of a pain at times...

ATLAS Space Token Roles

Following a discussion with ATLAS people I've now clarified that SRMv2 writing will happen using the atlas/Role=production VOMS role.

Therefore sites should restrict access to the ATLASMCDISK and ATLASDATADISK space tokens to this role. To do this, release and then recreate the space reservation:

# dpm-releasespace --token_desc ATLASDATADISK
# dpm-reservespace --gspace 10T --lifetime Inf --group atlas/Role=production --token_desc ATLASDATADISK

Greig's python snippet to list spaces is very helpful.

cluster glue

hmm. Freudian? I originally typed 'cluster clue' as the title.

Regular readers will be aware that we run both ganglia and cfengine. However even our wonderful rebuld system (YPF) doesn't quite close off all the holes in the fabric monitoring. case in point - reimaged a few machines and noticed that ganglia wasn't quite right. It'd copied in the right gmond.conf for that group of machines but hadnt checked that it was listed in the main gmetad.conf as a data_source,

Cue a short Perl script (soon to be available on the scotgrid wiki) to do a sanity check, but it;s this sort of non-joinedupness of all the bits that really annoys me about clusters and distributed systems.

Are there any better tools? (is Quattor the savoiur for this type of problem)

/rant

nagios event handlers

I've gone over to the Dark Side (no, not Python) and have just implemented my first nagios event handler - This *should* automatically try and fix the problem we have with our Dirvish backup scrips - namely that we end up with too many copies of the database dumps held.

So - cue nagios' event handlers - the only issue should be that nagios (and the event handler) runs as user nagios and most sysadmin stuff needs root. If you're willing to trust it to sudo then it should be OK.

DPM Storage Token Information Published

It was a bit fiddly, but with 4 fixes to Michel's script, Glasgow are publishing the space token information for ATLASDATADISK.

I documented the workarounds in the LCG Twiki.

perl-TermReadKey - missing in action

We had a ticket from a Zeus user unable to get a file off our DPM, while her Zeus colleagues could. I spend a long time checking the pool accounts, which were all fine, and checking the Zeus VOMS setup, which was also fine.

Finally, I looked in the logs for the grid-mapfile, where the culprit lay:

"Can't locate Term/ReadKey.pm in @INC..."

On two of the servers, disk034 and disk036, the perl-TermReadKey RPM was missing and it looks like the grid-mapfiles had not been rebuilt for a very long time - from the backups it looks like it was October when they were remade!

OK, nagios check: age of grid-mapfile and lcgdm-mapfile!

Down yum! Down!

As reported previously, We discovered that the nightly yum update was enabled on the servers.

The magic cfengine snippet to disable this is:

groups:
    # we don't want auto yum update stuff
    nonightlyyum = ( `/usr/bin/test -f /var/lock/subsys/yum` )

shellcommands:
    nonightlyyum::
        "/sbin/chkconfig yum  off" umask=022
        "/sbin/service yum stop" umask=022

The same check-for-some-enabled-subsys trick can be used to disable many of the periodic checks run on a standard SL install. (it's what 'service foo status' does for many of them)

SRM2.2 Configuration for FDR/CCRC

I configured Glasgow's DPM for the SRM 2.2 space tokens required by ATLAS for FDR/CCRC:

svr018:~# dpm-reservespace --gspace 10T --lifetime Inf --group atlas --token_desc ATLASDATADISK
ab0f1a60-59d6-4099-82fa-a17711678860

Easy, eh!

I notice that there is no dpm-listspaces command, which means that one has to grub around inside the database to find out what spaces are currently defined.

Two additional notes for other T2s:

1. Transfers from DDM are done using a vanilla atlas proxy for now (belonging to Mario Lassnig), so make sure the token is writable by the atlas group, not, e.g., atlas/Role=production.
2. All that is needed for CCRC is 3TB, however this is based on a 1 week cleaning cycle. If, like Glasgow, you have lots of space, then making the area bigger means the cleaning is not critical. (The space can be updated later with dpm-updatespace.)
   

YAIM stale...

Another problem I found with configuring the new VOs is that the reconfiguration of the information system on the CE failed. Mike had run config_gip, as has been done forever, but it did nothing. So queues and access rights for the VOs were not published.

There are new YAIM functions, like config_gip_ce, as well as a whole different way of structuring site-info.def. I fiddled with various options in the site-info.def file and even in the YAIM function itself, but I couldn't get it to work properly at all.

In some desperation I hacked the ldif file in the end, which is not a long term option; we really need now to look at the whole way that we structure YAIM for the site (as well as plan migrations to SL4 for BDII, CE and DPM...).

OPS goes dark...

Following enabling other VOs yesterday, DPM broke for ops. The error messages were as cryptic as ever:

httpg://svr018.gla.scotgrid.ac.uk:8443/srm/managerv1: Unknown error

And in the SRM logs:

01/29 12:29:02 14830,3 srmv1: SRM02 - soap_serve error : Can't get req uniqueid
01/29 12:05:18 14830,0 srmv1: SRM02 - soap_serve error : CGSI-gSOAP: Could not find mapping for: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=samoper/CN=582979/C
N=Judit Novak

The error seemed to correspond to re-running config_mkgridmap yesterday, however, as Judit (and other ops people) were in the grid-mapfile I was very confused.

Eventually, staring at the lcgdm-mkgridmap.conf I realised that the ops VO was only configured to get voms information from the deprecated lcg-voms.cern.ch server. I reconfigured to get the information from voms.cern.ch and it started to work.

The think I cannot fathom is how is kept working for so long - we have always had lcg-voms.cern.ch as the server for ops.

I updated the ops entry on the GridPP Wiki.

As usual the things which made this much harder then it should have were:

1. It only affected the ops VO (not dteam, atlas or pheno which we can test).

2. The error message was, as usualy, cryptic and unhelpful.

Keeping up with the Jones'

Well we recently had an incident with our NFS server for the cluster (home / software) locking up and needing a cold power cycle. Due to $vendors setup this takes aaaages (in the order of 20 mins) to go through the BIOS selfcheck (hangs at 053C). $vendor would like to poke around system and perhaps perform bios upgrade. Hmm. Oh well, all 10 disk servers are identical so we'll just drain one down and play - it also gives us chance to upgrade (from 1.6.5) to the latest 1.6.7-mumble DPM.


... or so we thought.

disk032:~# rpm -qa | grep DPM
DPM-gridftp-server-1.6.7-1sec
DPM-rfio-server-1.6.7-2sec.slc3
DPM-client-1.6.7-2sec.slc3

"Thats odd - Graeme have you updated these?" nope - Turns out that yum.nightly cron was auto updating on both the disk servers and some of the grid servers... Gaaah. clickity click and we're all ready to play.

In the meantime, dpm-drain migrated most of the data off the server to the other stash of disks but there were still 69 files that failed with 'Internal error' - Am looking through the DB to try and see if I can pull any more info out

Take Me To Your VO...

As noted preciously, we have enabled mice, vo.scotgrid.ac.uk and vo.nanocmos.ac.uk at Glasgow. Mike worked his way through the documentation, which was a out of date because it hadn't incorporated the changes in VO pool account management (which has become much better) which Andrew implemented.

We also plan to setup VOs much more promptly now for local user groups. Now that we have a scotgrid VOMS server this will be much easier.

The wiki has all the updated details.

Resource Broker Blues

Our resource broker was down for the weekend as the network service stalled. Root cause turned out to be a bit of over aggressive cleaning from cfengine. I had wanted to do a better job of cleaning up the /tmp area in the cluster - each worker node had hundreds of condor_g working directories lying around - with nothing in them. cfengine's "tidy" leaves directories alone by default and only cleans files. So I enabled the "rmdirs=sub" option - works beautifully, gets rid of all the cruft in /tmp. So pleased was I that I disengaged by brain and set this option on for /home as well - good idea to clean up those old gass cache areas, isn't it? Well, almost - unfortunately /home has subdirs which are the node pool account home areas and unused pool accounts fall into the clean me up category. All the untouched pool areas then vanished.

This caused a number of people to start getting "unspecified grid manager errors" on globus-job-runs, as well as wiping out the edguser home area on the RB which caused the network server to go into crisis.

It didn't take long to work out what had happened, but fixing it took a while as the resource broker seemed to be quite huffy afterwards.

The only plus side was that I enabled the mice, scotgrid and nanocmos VOs on the RB.

Happy New Year ScotGrid - now with added ECDF...

Well, we didn't get it quite as a Christmas present, but the combined efforts of the scotgrid team have managed to get ECDF green for New Year.

In the week before Christmas Greig and I went through period of intensive investigation as to why normal jobs would run, but SAM jobs would not. Finding that jobs which fork a lot, like SAM jobs, would fail was the first clue. However, it turned out not to be a fork or process limit, butn a limitation on the virtual memory size which was the problem. SGE can set a VSZ limit on jobs, and the ECDF team have set this to 2GB, which is the amount of memory they have per core. Alas for jobs which fork, virtual memory is a huge over estimate of their actual memory usage (my 100 child python fork job registers ~2.4GB of virtual memory, but uses only 60MB of resident memory). That's a 50 fold over estimate of memory usage!

As SAM jobs to fork a lot, they hit this 2GB limit and are killed by the batch system, leading to the failures we were plagued by.

A work around, suggested by the systems team, was to submit ops jobs to the ngs queue, which is a special short running test queue (15 min wall time) which has no VSZ limit on it.

Greig modified the information system to publish the ngs queue and ops jobs started to be submitted to this queue on the last day before the holidays.

Alas, this was not quite enough to get us running. We didn't find out until after new year that we also needed to place a specify a run time limit of 15 minutes on the jobs and submit them to a non-standard project. The last step required me to hack the job manager in a frightful manner as I really couldn't fathom how the perl (yuk!) job manager was supposed to set the project - in fact even though project methods existed they didn't seem to emit anything into the job script.

Finally, with that hack made this morning, ECDF started to pass SAM tests. A long time a coming, that one.

The final question, however, is what to do about this VSZ limit. The various wrappers and accoutrements which grid jobs bring mean that before a line of user code runs there are about 10 processes running, as 600MB of VSZ has been grabbed. This is proving to be a real problem for local LHCb users, because ganga forks a lot and also gets killed off. Expert opinion is that VSZ limits are just wrong.

We have a meeting with the ECDF team, I hope, in a week, and this will be our hot topic.

Big thanks go to Greig for a lot of hard work on this, as well as Steve Traylen, for getting us on the right track, and Kostas Georgiou, for advice about the perils of VSZ in SGE.

ECDF Progress

We finally seem to be homing in on the problems at ECDF. Any job which forks off too many processes seems to die in the batch system. Launching a simple fork python job works fine at 20 children, but dies at 50. The same task at Glasgow runs happily with 100 children.

I can see there is no ulimit issue, but something is unhappy. We must track down if it is SGE or some gatekeeper weirdness.

Grid-Monitoring Nagios tests

Finally (after a long interval) I reinvestigated getting the LCG Grid Service Monitoring Working Group nagios tests installed at Glasgow.

I had tried once before, but it needed nagios on our UI. This time I added a UI to our nagios host (nice n simple - simply added the hostname into the relevant UI group in cfengine). Works fairly well - I've got it installed and polling the SAM tests via the sam-API and picking up the results. I still need to get certificate proxy renewals working, and merge the records together with our existing definitions for the hosts (we use non-qualified names, the wlcg.cfg used FQDNs)

As the screen shot shows - We've already got a lot of green - and if we can nail the cert problems I'll switch it over to use the normal notification system

Glasgow joins NGS as partner site

Glasgow was approved as an NGS partner site at the last NGS MB meeting. It seems it took us a long time to get here, but with the addition of gsissh login for NGS members we had a full service which would be useful to NGS members. Our test record for the NGS Inca test suite was good.

David is drafting detailed documentation for NGS members, so watch this space...

One bad apple, sitting in a rack...

Andrew did great work getting all the nodes back on line and dealing with the quirks of cfengine and reconfiguring everything.

Unfortunately we failed 2 SAM tests, with the infamous "Cannot read JobWrapper output, both from Condor and from Maradona". I checked the torque logs and both of these tests ran on node016 - so looked like this was the bad apple.

When I checked, it was clear that yaim had not run, so the PATH was bad (perhaps this was before Andrew fixed cfengine)?

Quick spin with cfengine and "-Drunyaim" and the node was good again.

The existence of links from /etc/profile.d/grid-env.{csh,sh} is an excellent proxy for YAIM having run correctly, so we should implement this as a cfengine test.

140 worker nodes sitting in the rack...

140 workernodes sitting in the rack, and if one small workernode should accidentally kernel panic, there'll be 139 workernodes sitting in the rack. "WooHoo" - For the first time in AGES pbsnodes -l isn't listing any nodes as offline or down. Admittedly I have brought back into service one node that kernel paniced - but $vendor is taking soooo long to get back to us with faults we may as well see how long it takes before it dies again.

So in summary - Glasgow is batting at 560 for nought so far :-)

Java and gite-WN

After sorting out the hassle with cfengine earlier I still couldn't get nodes to build from scratch without intervention. gliite-WN (3.1.1.0) wasn't installing due to failed dependencies on java by bouncycastle. Steve T to the rescue (again) with his notes on JPackage. Sun have updated the jdk to 1.5.0.14 (there are quite a few CVE vulns before -13) but the jpackage is only available to -13. Luckily the CVS log of the spec file (available here) simply bumps the version number between 13 and 14 only. I therefore patched and compiled our own java-sun-1.5.0 package for the cluster.

Name : java-1.5.0-sun Relocations: (not relocatable)
Version : 1.5.0.14 Vendor: (none)
Release : 1jpp Build Date: Fri 07 Dec 2007 16:43:26 GMT
Install Date: (not installed) Build Host: svr031.gla.scotgrid.ac.uk
Group : Development/Interpreters Source RPM: java-1.5.0-sun-1.5.0.14-1jpp.nosrc.rpm
Size : 71332405 License: Sun Binary Code License
Signature : (none)
Packager : Scotgrid Glasgow
URL : http://java.sun.com/j2se/1.5.0
Summary : Java Runtime Environment for java-1.5.0-sun
Description :
This package contains the Java Runtime Environment for java-1.5.0-sun

it seems OK on the worker nodes - the three I rebuilt went straight to glite-WN 3.1.1.0 without any hassle. - just before the announcement that they've bumped the version number to 3.1.1.1. Typical.

cfengine hassle

We've had a strange problem with cfengine recently - it wasn't creating the symlink /usr/java/current -> /usr/java/javer_version_whatever. The log file complained of:

Error while trying to link /usr/java/current -> /usr/java/current

which is most odd as I was trying to link it to $(javaver) which evaluates to jdk1.5.0_12.

After much faffing about it turns out to be a bug in cfengine - it tries to create a circular link on the 1st item in the list and works fine for the 2nd one. as a result our cfengine config is now:

links:
java::
# BUG IN CFENGINE - it will fail the 1st line but do the 2nd OK
/usr/java/current_ver ->! /usr/java/$(javaver) type=relative
/usr/java/current ->! /usr/java/$(javaver) type=relative


which is ugly, hacky and just Wrong

ECDF networking troubles

Just a quick post to say that there was a problem with one of ECDF's network switches on Sunday morning just before 2am that brought down GPFS and also the NFS mounts on the CE. Luckily we're not using GPFS for the Grid storage, but the CE did have problems. The systems team are looking into ways of making things more resilient to failures like this in the future.

Dem Biomed Blues...

Finally I got fed up with the biomed user whose jobs always stall on the cluster. Banned them and sent in a ticket.

I'm not prepared to be tolerant of crap code when we have 100s of queued jobs.

Health and Efficiency (ATLAS style)

Now that the conversion to the new ATLAS MC production system (panda/pallette/pangea?) is underway, I thought it would be interesting to compare the site's view of efficiency in the new system to the old. I had to fix up our local accounting database, which was truncating some of the longer username fields we have now (e.g., prdatlasNNN). After doing that, I could easily distinguish between panda pilots and other production activities.

Since we upgraded to SL4 in September (which was just about the time that Rod started toying with panda) the scores are:

Lexor/Cronus
+-------+----------+-----------+-----------+
| Jobs  | CPU_Hours| Wall_Hours| Eff       |
+-------+----------+-----------+-----------+
| 20047 | 282434.6 | 533904.0  | 0.52899   |
+-------+----------+-----------+-----------+

Panda
+-------+------------+-----------+------------+
| Jobs  | CPU_Hours  | Wall_Hours| Eff        |
+-------+------------+-----------+------------+
| 17746 | 57312.5925 | 59600.919 | 0.96160584 |
+-------+------------+-----------+------------+

This is quite a different view of "efficiency" to the VO's view, because here the actual success or failure of the job is masked - we're only looking at wall time efficiency in the batch system. However, the improvement here is spectacular, so sites should, I think, be very happy with this change.

Note that the panda figures include all the pilots, even the ones which had no jobs to pick up (production stalled a few times because of dCache problems at RAL and other teething troubles). If one masks these jobs out then the efficiency is even better: 98.1%.

ECDF - nearly there...

Thanks to the efforts of Greig and Sam, ECDF now has storage set up. Not a lot of storage (just 40MB), but it proves the headnode is working and the information system is correctly configured.

This means we are now fully passing SAM tests. Hooray!

Of course, passing SAM tests is only the first step though, and there are 3 outstanding issues which I have discovered using an atlas production certificate:

1. I was mapped to a dteam account when I submitted my job (not quite as bad as you think - I am obviously in dteam and this was the default grid-mapfile mapping after LCMAPS had failed).
2. There's no 32bit python - this has been passed to Ewan for dealing with (along with the list of other compat 32 RPMs.
3. There's no outbound http access. This hobbles a lot of things for both ATLAS and LHCb.

It feels like we're in the home straight at last though!

Maui: MAXPROC vs MAXJOBS

One thing which was always desirable in the batch system was to guarantee a number of jobs slots for groups, irrespective of their fairshare usage. We actually want to encourage opportunistic usage of resources, and not punish people by then refusing to run any of their jobs for a week.

However, attempts to set a soft MAXPROC limit always seemed to come to grief. Maui would block jobs beyond the soft limit, even though, as far as I could see, it had been told not to. in frustration I had to set all groups with a soft limit at the cluster size.

Today, I had a chat with the great maui guru, Steve T, who was also somewhat puzzled by maui's behaviour. He pointed out that he'd only ever set MAXJOBS, not MAXPROC. Well, I thought I would give that a whirl and it works!

So, finally we can have a system which protects some slots for VOs and groups, but allows for full opportunistic use of the cluster for everyone.

Thanks Steve!

Maui Madness

Maui has been driving me mad for about 3 weeks now. When we upgraded the cluster I had forgotten that moving to pooled prd and sgm accounts would mean that these groups were independent of the normal VO fairshare. As our engineers started to become more active I was unable to get any ATLAS jobs to start at all - particularly atlasprd jobs. As I tried to add fairshare for the new groups maui started to lose the plot, just dropping groups entirely from its fairshare groups - you can see the effect very clearly from MonAMI's maui plots - groups just evaporate!

Fed up with this, tonight, I stopped maui, removed all its databases and restarted it. This, of course, means it's lost its current fairshare calculations, but at least it now has fairshares for the new groups.

I have also re-jigged the fairshare algorithm to have far less of a decay on it - users who ran 7 day jobs were at a huge advantage because by the time their job had finished its first day of running was weighted by 0.3, so it almost didn't count!

Grids, ScotGrid and GU

I gave a talk to the Distributed IT Group at the University today, entitled Grids, ScotGrid and GU: Computing from here to eternity? It was a general introduction to EGEE grids and contained some specific information on how to get started with the Glasgow cluster.

You can get the talk here.

First Jobs Run at ECDF

Finally, after several months of anguish, SAM jobs are running at ECDF! Note that for the moment they fail replica management tests (there was little point in putting effort into the DPM while the CE was so broken), but at last we're getting output from SAM jobs coming back correctly.

The root cause of this has been networking arrangements which were preventing the worker nodes from making arbitrary outbound connections. Last week we managed to arrange with the systems team to open all ports >1024, outbound, from the workers. Then it was a matter of battering down each of the router blocks one by one (painfully these seemed to take about 2 days each to disappear).

Testing will now continue, but we're very hopeful that things will now come together quickly.

nagios monitors are Go!

it's been long overdue on the TODO list but we finally got nagios nrpe installed and configured on the worker nodes. We're now checking for locally logged in users (should only be sysadmin staff locally), high loads, processes, zombies and most importantly disk free.

Few pointers that may help others. 1) cfengine splays for 30 mins. This means if you enable a check before the plugsins are pushed out to the node it fills your mailbox. 2) if you normally use

define service{
hostgroup_name workernodes

then you'll find your testing runs on ALL workernodes. use host_name node001 (or equiv) for testing new services.
3) cfengine saves you pushing the same config out manually - and it also has the nice side effect of restarting nrpe (a necessary process) automatically when it realises nrpe.cfg has changed

M5: The data cometh...

Data from the ATLAS M5 cosmics run started to flow into the UK yesterday. Looks like Glasgow has managed to get all of the subscribed datasets:

M5.0029118.Default.L1TT-b11100100.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029118.Default.L1TT-b00001000.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MILANO,MWT2_IU,MWT2_UC,RALDISK
M5.0029118.Default.L1TT-b11101000.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029120.Default.L1TT-b00000010.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029118.Default.L1TT-b00000010.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MILANO,MWT2_IU,MWT2_UC,RALDISK
M5.0029118.Default.L1TT-b11101110.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029118.Default.L1TT-b00000011.ESD.v13003010
COMPLETE: AGLT2,BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MILANO,MWT2_IU,MWT2_UC,RALDISK
M5.0029118.Default.L1TT-b11101011.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029120.Default.L1TT-b00000001.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MWT2_IU,MWT2_UC,NAPOLI,RALDISK
M5.0029118.Default.L1TT-b00000110.ESD.v13003010
COMPLETE: BNLDISK,BU_DDM,CNAFDISK,GLASGOW,MILANO,MWT2_IU,MWT2_UC,RALDISK

Biomed Abuse

Alerted by Dave Colling, we found a biomed user who was running about 100 jobs on the cluster all trying to factorise a 768 bit number (and win $50 000 in the process).

Clearly this is abuse of our resources and nothing to do with biomed. They have consumed more than 80 000 normalised CPU hours since September. I'm sure the operational costs of this amount to more than several thousand pounds (should we bill them?).

It was all the more irritating as we had a stack of ATLAS and local users' jobs to run, but the biomed jobs were set to download subjobs from the user's job queue (they were effectively limited pilots) and so they ran right up to the wallclock limit of 36 hours.

I banned the user, deleted their jobs and sent a very angry GGUS ticket.

As a slight aside one notes the efficiency of pilot job systems at hoovering up spare jobs slots and consuming resources on cluster well in excess of the nominal 1% fair share we give to biomed.

NGS VO Supported

I should have now fixed the NGS VO on ScotGrid. This was supposed to happen during the SL4 transition, but got lost in the rush to get the site up and running correctly.

LCMAPS now knows to map certificates from the ngs.ac.uk VO to .ngs pool accounts, and the fallback grid-mapfile is being correctly made.

What remains to be sorted out is if we can support NGS without giving shell access. I am loath to turn on pool mapped gsissh as we'd have to hack the grid-mapfile on the UI (to exclude other VOs) and somehow manage to synch the pool account mappings between the UI and the CE.

At the moment I'm sure it will be broken, because we don't even have shared home areas, but even if introduce these I'm not certain it will be a practical proposition to only offer gsiftp.

Some Queue Work

I introduced a new long queue last night, which is primarily to support long validation jobs run by some of our Durham phenomenology friends. It has a 7 day CPU/Wall limit. As I was messing about with transitions to queues which support more than 1 VO, I added support for ATLAS on the queue, thinking it might be of use to our local ATLAS users.

Unfortunately, as it was advertised in the information system, and was a perfectly valid and matchable queue, we soon got production and user ATLAS jobs coming in on this long queue. So, tonight, I have stopped advertising it for ATLAS, to push things back to the normal 36 hour limited grid queue.

I also discovered that maui does its "group" fairsharing on the submitting user's primary group. This is pretty obvious, of course, but as we've traditionally had one queue per VO, everyone in that VO in a primary group of the same name, somehow I had it muddled in my mind that it was a queue balancing act instead. Turns out this is in fact good, because we can set different fair shares for users sharing the gridpp queue, as long as we keep them in per-project primary groups.

... and we're back

OK - Estates and buildings found enough spare change to last us through for another couple of hours. One workernode casualty (so far) but otherwise things seem to be OK. Will monitor for sanity then remove us from unscheduled downtime.

Glasgow Down - Power Outage

We lost power in the Kelvin Building this morning just before 9am. News is:

Power outage
25/10/07 08:49 BST
We have been informed by Estates and Buildings that there is a power
outage in the Kelvin Building among others. The Kelvin buildings
router provides network services to other buildings and so there will
be no network service in the following buildings:
Kelvin, Davidson, West-medical, Pontecorvo and Anderson colleges,
Joseph-Black, Bower, Estates, Garage, Zoology and shares the medical
school with Boyd-Orr.
Scottish power are working on the problem. Further information as it
becomes available.

We've EGEE broadcast and put in an unscheduled downtime.

Really annoying - we were running along very nicely when this happened (although we'd had to shut down 20 WNs because of the loss of an a/c unit on Tuesday).

Glasgow site missing, presumed AWOL

This morning uki-scotgrid-glasgow and other systems in the same building are offline. We're not sure of cause yet and wil investigate further once someone's on site.

UPDATE - 09:08 - Confirmed as power cut affecting building (see http://www.gla.ac.uk/services/it/helpdesk/#d.en.9346) No ETA yet for restoring services. EGEE Broadcast sent.

News from ATLAS Computing

There is some significant news for sites from ATLAS software week. The decision was taken yesterday to move all ATLAS MC production a pilot job system, with the pilots based on the Panda system developed by US ATLAS. The new system will get a new name, pallete and pallas are the front runners. (I like pallas myself.)

In addition EGEE components, such as LFC, will be standardised on for DDM.

As this is a pilot job system, so the anticipated model is that ATLAS production will keep a steady stream of pilots on ATLAS T2 sites, which pull in real job payloads from the central queue.

This is very like the LHCb MC production model, so as the transition is made to this system sites should start to see much better usage of their resources by ATLAS - just like LHCb are able to scavenge resources from all over the grid.

Canada have recently shifted to Panda production, and greatly increased their ATLAS workrate as a result. There have been some trials of the system in the UK and France, which were also very encouraging.

Of course, there is a very large difference with LHCb, because ATLAS don't just do simulation at T2s, but also do digitisation and reconstruction. These steps require input files and the panda based system for dealing with this is to ensure that the relevant input dataset is staged by the ATLAS distributed data management system (DDM) on your local SE; further the output dataset will be created on your local SE, then DDM will ship it up to the Tier-1 after the jobs have run.

This means that sites will really require to have a working storage system for ATLAS work from now on (in the previous EGEE production model, any of the SEs in your cloud could be used, which masked a lot of site problems but caused us huge data management headaches).

In the end using pilots had two compelling advantages. Firstly, the sanity of the environment can be checked before a job is actually run, which means that panda gets 90% job efficiency (the other EGEE executors struggled to reach 70%). Secondly, and this is the clincher, it means that we can prioritise tasks within ATLAS, which is impossible to do otherwise.

At the moment the push will be to get ATLAS production moved to this new system - probably on a cloud by cloud basis. This should not cause the sites headaches as production is a centralised activity (most sites still have a single atlasprd account anyway). However, the pilots can also run user analysis jobs - and this will require glexec functionality. Alessandra and I stressed to Kors that this must be supported in the glexec non suid mode.

In the UK ATLAS community we now need to get our DQ2 VOBox working properly - at the moment dataset subscriptions in the UK are just much too slow right now.

Postscript: I met Joel at lunchtime, who wanted me to namecheck LHCb's DIRAC system, as panda is based on DIRAC - well, I didn't know that, but I suppose I'll learn a lot more about the internals of these things in the next few months.

EGEE: It's broken but it works...

I had meant to blog this a week ago, but I did my first shift for ATLAS EGEE production. This was really being thrown in at the deep end, as the twiki instructions were, well spartan, to say the least. (And definately misleading in places, actually.)

I was on Data Management, which meant concentrating on problems of data stage-in and stage-out, as well as trying to pick up site which had broken tool sets.

It felt like a bit of a wild ride - there are almost always problems of some kind, and part of the art is clearly sorting out the completely urgent must be delt with now, from the simply urgent, down to the deal with this in a quiet moment.

I found problems at T1s (stage-in, overloaded dCaches, flaky LFCs), T2s (SEs down, quite a few broken lcg-utils, some sites just generically not working but giving very strange errors). I raised a lot number of GGUS tickets, but sometimes it's very difficult to know what the underlying problem is, and it's very time consuming batting the ticket back and forth with the site.

It's a very different experience from being on the site side. Instead of a "deep" view of a few sites you have a "shallow" view of almost all of them. If you want to read my round-up of issues though the week, it's on indico (it's the DDM shifter report).

Disk Servers and Power Outages

Your starter for 10TB: how long does it take to do an fsck on 10TB of disk? Answer: about 2 hours.

Which in theory was fine - in at 10am, out of downtime by 12. However, things didn't go quite according to plan.

First problem was that disk038-41, which had been setup most recently, had the weird disk label problem, where the labels had been created with a "1" appended to them. Of course, this wouldn't matter, except that we'd told cfengine to control fstab, based on the older servers (with no such ones), so the new systems could not find their partitions and sat awaiting root intervention. That put those servers back by about an hour.

Secondly, some of the servers were under the impression that they had not checked their disks for ~37 years (mke2fs at the start of the epoch?), so had to be coaxed into doing so by hand, which was another minor hold up.

Third, I had decided to convert all the file systems to ext3, to avoid protect them in the case of power outages. It turns out that making a journal for a 2TB filesystem actually takes about 5 minutes - so 25 minutes for your set of 5.

And, last but not least, the machines needed new kernels, so had to go through a final reboot cycle before they were ready.

The upshot was that we were 15 minutes late with the batch system (disk037 was already running ext3, fortunately), but an hour and 15 minutes late with the SRM. I almost did an EGEE broadcast, but in the end, who's listening Sunday lunchtime? It would just have been more mailbox noise for Monday, and irrelevant by that time anyway.

As the SRMs fill up, of course, disk checks will take longer, so next time I would probably allow 4 hours for fscking, if it's likely.

A few other details:

* The fstab for the disk servers now names partitions explicitly, so no dependence on disk labels.
* The BDII going down hit Durham and Edinburgh with RM failures. Ouch! We should have seen that one coming.
* All the large (now) ext3 partitions have been set to check every 10 mounts or 180 days. The older servers actually had ~360 days of uptime, so if this is an annual event then doing an fsck once a year should be ok.

Glasgow power outage

Ho Hum, it's that time of year for HV switchgear checking. Rolling programme of work across campus meant that the building housing the glasgow cluster was due for an outage at ungodly-o-clock in the morning. We arranged the outage in advance, booked scheduled downtime. All OK. Then after G had taken some well deserved hols I discovered how dreadful the cic-portal was to send an egee broadcast. I want to tell users of the site it;s going down. Any chance of this in english? RC management? What is 'RC' - doesn't explain it. Then who should I notify? agan no simple descriptions... Grr. Rant..

OK - system went down cleanly easily enough (pdsh -a poweroff or similar) - bringing it back up? hmm. 1st off the LV switchboard needed resetting manually so the UPS has a flat battery. Then one of the PDU's decided to restore all the associated sockets to 'On' without waiting to be told. (so all the servers lept into life before the disks were ready). Then the disk servers decided they needed to fsck (it'd been a year since the last one) - slooooow. Oh, and the disklabels on the system disks were screwed up (/1 and /tmp1 rather than / and /tmp for exmple) - another manual workaround needed.

Finally we were ready to bring the workernodes back - just on the 12:00 deadline. I left graeme still hard at it, but there's a few things we'll need to pull out in a post mortem. I'm sure Graeme will blog some more

svr016 (CE) sick

Our CE appears to have fallen over this morning at 7AM. It's not responding to SSH (well, with a load like that I'm not suprised) - It'll need poking as soon as someone's on site.

UPDATE - 10:00. Unresponsive to even the 'special' keyboard we keep for such events. Needed the big red button pressing. Seems to have come back OK. Monitoring for fallout.

RB Corrupts its Database

Our RB (svr023) seriously died today.

I was alerted to a problem by a pheno user who was having trouble submitting jobs. When I checked the RB I found that the root partition was full, choked by massive /var/log/wtmp files and by an extremely large RB database in /var/lib/mysql.

When had reinstalled the RB 2 weeks ago I had ensured that /var/edgwl was in a large disk area, but to have more than 4GB filled up in less than 2 weeks by the other denizens of /var was completely unexpected.

Emergency procedure was then to move /var/log and /var/lib/mysql over to the /disk partition, creating soft links pointing from the old locations.

This seemed to be going ok, but job submission was still failing. When I checked the error log for mysql I got the message:

071008 16:25:32 [ERROR] /usr/sbin/mysqld: Can't open file: 'short_fields.MYI' (errno: 145)

The short_fields table definitely existed. I even checked it against the last version in /var. Logging in to mysql demonstrated that this table had become corrupted.

I toyed with the idea of trying to save the database, however it would probably have left us with an internally inconsistent RB database, as well as orphaned files in the /var/edgwl bookkeeing area.

Reluctantly I decided that the only sensible recourse was to reinstall svr023, and take the hit of the lost jobs.

This has now been done, and normal service has been resumed.

In the course of the re-install the /var partition has been grown to more than 100GB, which should protect us against large log files for quite a time.

Frankly, all very annoying and I'm quite upset that we lost users' jobs.

Sorry folks. It shouldn't happen again.