The Higgs appears

This year we haven't been great at keeping up with blog posts but there are many reasons behind this. We have installed a new network, additional cores taking us up to 4000 available job slots and have upgraded the server infrastructure throughout  GU Scotgrid's cluster. Also, we have fallen foul of infrastructure issues and have had problems with the old and replacement Air Con systems. Slowly, we are extracting ourselves from these issues and  recently Professor David Britton gave a lecture on the Grids role in the announcement made in July of this year at CERN during this years Turing Festival. A surprise appearance at the event was Professor Higgs himself.

Professors David Britton (left) and Professor Peter Higgs

Also presenting at the event were Professors Tejinder Singh Virdee and John Ellis of Imperial College and Dr Ben Segal from CERN. The event was one of the kickstart activites for the Turing Festival and enabled the public and academics to get a better over view of what has been involved in getting the experiments this far.

My God; it's full of data-transfers!

The Great ATLAS Spacetoken Migration of 2011 kicked off yesterday evening, and with 47TB of data sitting in MCDISK at Glasgow, Brian and We decided to take the opportunity to see how fast we could push it across to DATADISK.
So, since ATLAS Data Management on this case happens over FTS (even though the vast majority of the transfers are internal to a site), we turned up the number of slots for STAR-GLASGOW a bit, from 20 (our default) to 50 (which was fun) up to 80 (although we peaked at around 65 used).
With effectively no limit from FTS, our data rates were... impressive. Although it's an unfair comparison (everyone else was limited by FTS, and we were mostly moving things over the internal network), we managed to hit a peak transfer rate of 1.5GB/s internally (yes, that's 12Gbit/s), and sustain at around 8Gbits/s. That equated to around 2/3s of the total UK data movement over STAR channels, or roughly 2/3s of ATLAS's total traffic in this migration. At that rate, none of our disk servers were stressed, and the network switches were intensely relaxed.

Some exciting graphs follow:

Woops, there go the pool accounts...

We got a ticket on Tuesday because an ATLAS user couldn't get their files back from one disk server which had run out of its 600 (!) ATLAS mapped pool accounts.

I did a bit of a hacky clean up, but actually this is very safe because, unlike a CE, there are no files involved to be mis-inherited by a subsequent user. The only issue would occur at the very moment that a user tried to transfer files.

The clean up removed the oldest mappings, and even the busiest server was down to ~150 mappings and ~450 free slots, so adequate breathing room was gained.

Sam is going to think about this in the storage group concept and write a more general tidier-upper for all VOs.

A fistful of user jobs...

No ATLAS production to do in the UK, but we have a nice full cluster anyway, with more than 1000 user jobs running:

svr016:~# qstat -q

server: svr016.gla.scotgrid.ac.uk

Queue            Memory CPU Time Walltime Node  Run Que Lm  State

---------------- ------ -------- -------- ----  --- --- --  -----

q2d                --   48:00:00 48:00:00   --  134  12 --   E R

atlanaly           --   24:00:00 24:00:00   --  738 789 --   E R

atlprd             --   48:00:00 48:00:00   --    2  48 --   E R

q7d                --   168:00:0 168:00:0   --    0   0 --   E R

route2all          --      --       --      --    0   0 --   E R

q1d                --   24:00:00 24:00:00   --   94  16 --   E R

mpi                --      --    72:00:00   --    0   0 --   E R

atlas              --   24:00:00 24:00:00   --  470 312 --   E R

lhcb               --   48:00:00 48:00:00   --    0   0 --   E R

                                               ----- -----

                                                1438  1177

The atlanaly queue are jobs from the panda backend and the atlas queue takes WMS backend jobs.

Today the particular job mix was kind to the storage, with no overloads being seen, but it's something we constantly have to monitor to pre-empt problems.

Postscript: I had another look and realised that most of the WMS backend jobs were from hammercloud (Sam testing SSDs!). Seems that genuine user WMS jobs were about 20-30, with more than 1000 in the panda backend.

serious multi core

who would have thought it possible ....



well with an large SSD, 24 cores, file stager analysis and it survives! More soon on our testing with some cutting edge equipment.

Take my outputs, damn you...

We recently ran up a very large backlog of production output files waiting to go from Glasgow back to the Tier-1 (reminder, panda doesn't consider a job finished until the outputs are safely stored at the T1). This is clearly seen in the red line on the panglia plot above, which reaches very high values. As we recently cut the timeout for the UK cloud to 2 days for transferring jobs, to improve the responsiveness of the production system, we started to leak out failed jobs (light green line) as panda gave up and decided to rerun.

Fortunately we got a big boost in the number of FTS slots from Glasgow to RAL, increasing from 10 to 25 active transfers (see the bottom FTS monitoring plot). Even so it clearly takes 24 hours for all the backlogs to drain down.

One of the problems here is that the output files are small from simulation (a tiny log file and a 20-50MB HITS file), so the overheads of FTS + SRM are very considerable and the actual bandwidth achieved is quite low. One possibility we are considering in ATLAS is introducing a pre-merge of outputs on the T2, which will allow us to send much bigger files back to the T1 (although a final "super-merge" will probably still be necessary). For this we are waiting for the generic Athena merge transform and then we will need to test integrating this into the mainline production workflow.

Until then we just have to take the operational load of tweaking the FTS settings when necessary.

Deflected Cosmic Rays...

This is the second short "when you're good..." post. During the RAL machine room move, we tested distributing ATLAS cosmics AOD and DPD data from CERN->GLASGOW->UK T2s. After some tweaking of the T2 FTS channels at CERN and tinkering in DDM this has worked a charm. Data distrubution in the UK has gone very well throughout the current combined cosmics data taking runs.

This is the first time that we tried circumventing the T1 for such an organised data distribution and it was a real success for the UK, ATLAS and Glasgow.

When you're good, you're Glasgow...

There hasn't been much time to write in the blog recently, STEP09 madness and all. However, it is wonderful to see that Glasgow was the top ATLAS T2 for analysis during the STEP09 challenges. We analysed more than 1.8B events, mostly through panda, with a 98% success rate.

We also took the largest fraction of data of any UK T2, 40%, and succeeded in getting all the data we were sent (we had little anxiety on the final weekend and want to increase our network heardroom for sure).

Sam and I wrote a full report on our experiences and how we used the opportunity to really probe the limits of the current cluster.

For the future, we really have to worry about how to maintain the i/o rate into the CPUs as the number of cores rises.

Oh my gosh... it's users...

I had been aware of a steady increase in the number of ATLAS user jobs on the cluster in the last few months, which I was delighted to see. I decided to quantify this by querying our accounting database and the users really have arrived.

User jobs since April 1 have consumed 867k hours of wallclock and 686k hours of CPU (80% efficient), c.f. production numbers of 1981k wallclock and 1867k CPU (94% efficient). This means ATLAS users are now consuming 30% of the ATLAS walltime on the cluster.

We've had 235 unique ATLAS users since April and 46 have used more than 1000 hours of wallclock time.

Victim of our own success

I was wondering why Glasgow was not getting more activated ATLAS production jobs and eventually tracked it down to the fact that our cache disk area, ATLASPRODDISK, was almost full with only 170GB free space left. Panda was very sensibly not sending us more jobs until we had somewhere to put the outputs!

I quick whirl with dpm-updatespace and I increased PRODDISK from 2TB to 5TB, which should see us good.

I also discovered that Durham was missing some ATLAS releases, which was why they were missing out on ATLAS jobs today - installations now triggered.

HammerCloud 135---A Load Shared is a Load Halved, to a point.

We performed our splitting of the DPM across two hosts just in time for the most recent HammerCloud test on UK sites:
http://gangarobot.cern.ch/st/test_135/

So, we already have some metrics to compare the old arrangement with the new.
For reference, Graeme blogged about the last big HammerCloud UK test here, where we were getting an event rate of around 10Hz, at the cost of the DPM head node running at an unsustainable load.
Since then, a couple of Hammerclouds have come by, generally coincident with ATLAS production and other stresses on the DPM, and it has just utterly failed to cope.

After our surgery, we did a lot better:




with an event rate of about 14 Hz, a 50% improvement, almost.



and, the load on the DPM head node was very much more acceptable, given the increased power of the hardware:




However, we're still not close to maxing out the pool nodes:



probably because we've hit another, higher, performance bottleneck on the new svr015 "MySQL server" machine:



that orangish stuff is the CPU in I/O Wait state, waiting for seeks within the DB.
We're currently looking at ways of tuning MySQL, or the disk, to improve this performance, since it looks like there's another 30 to 40% of performance there, at least.

Some ideas we've had include splitting the dpm_db and cns_db across different filesystems (since they have very different access patterns for this kind of use), tweaking MySQL settings (although they look generally fine...), or even getting Faster Disks. Roll on solid state drives, we say!

Analysis Challenge: Round 4

We re-run the analysis challenge yesterday with a better MySQL setup so that the higher number of dpns daemons could get db connections. However, the results were much the same as before and the conclusion seems to be that X509 sucks - it's killing the headnode with all of the simultaneous authentications.

We hope to prove later on that this is the real problem and then think about what we can do about it...

Plots from last analyais challenge

Mostly confirmed the results which we saw at the end of last week's test. Load on the DPM headnode is our pressing concern - it's maxing its CPU out even at open rates of a little over 1Hz.

Analysis Challenge: Round 3

Last week's analysis challenge at Glasgow showed extreme load and sluggishness in the DPM (see the attached plots of awfulness). Although we managed a much better event rate we also suffered from incomplete processing and the DPM was a clear bottleneck.

I had a chat with JPB today who spotted the very high memory consumption of the dpm daemon - he thinks there's probably a memory leak and that this might be slowing things down. He also said it might be worth running more dpns daemons as these also do connection athentication.

So, to get ready for tomorrow I have:

1. Allowed core dumps for the DPM and DPNS daemons.
2. Increased the number of threads in the DPNS daemon to 60.
3. Restarted all the daemons.

That last operation freed up about 3GB of memory!

If we still see problems tomorrow then at least we should have some good information for the developers to chew on.

"Oh no! Not again..."

After being all enthusiastic that the gSOAP errors had been nailed, we failed two SE tests in the last 24 hours. Exactly the same issue as before.

As this error message is so vague it looks like lcg-rollout is our only hope.

I note in passing that Glasgow has one of the most reliable SEs in the UK for ATLAS (2.1% job loss, only beaten by Oxford who have 0.8%; UK average in Q3 was 8% loss) so this is particularly galling.

Shouldn't we be making the results as seen by our real customers rather more important than a once an hour stab in the dark from ops?

Glasgow on the move

Because of the current problems at RAL, Glasgow was nominated as a test peripatetic Tier-2, so see how agile ATLAS production was at moving Tier-2 resources in case of T1 downtime (note this test only works if the cloud services, FTS and LFC, are still running - if these are gone then it's almost impossible, today, to use any of the cloud's Tier-2s).

First off Glasgow was sent South East, into the NL cloud. Here we found a problem with the input datasets, because input datasets to T2s (which are subscribed without sources) only look for sources within the cloud (this follows the ATLAS computing model). However, the way around this is to specify the associated T1 (for production) as the source and then DQ2 does the work. The panda developers made the change on Friday, so that NIKHEF was specified as the source for inputs to Glasgow. Likewise, for output back to the NL T1, Glasgow's PRODDISK token was specified explicity as the source.

That done, Glasgow galloped through a couple of hundred jobs for the NL cloud, before they ran out of jobs:

Flushed with this success we've just shoved Glasgow into the FR cloud for a while, as they still have jobs left to run. Within an hour we're running a couple of 100 jobs.

SAM Failures across scotgrid: Someone else's problem

All 3 scotgrid sites have just failed the atlas SAM SE tests (atlas_cr, atlas_cp, atlas_del) as have quite alot of the rest of the UKI-* sites.

Once again this isn't a Tier-2 issue but an upstream problem with the tests themselves


ATLAS specific test launched from monb003.cern.ch
Checking if a file can be copied and registered to svr018.gla.scotgrid.ac.uk

   ------------------------- NEW ----------------
srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/
+ lcg-cr -v --vo atlas file:/home/samatlas/.same/SE/testFile.txt -l lfn:SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438 -d srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
Using grid catalog type: lfc
Using grid catalog : lfc0448.gridpp.rl.ac.uk
Using LFN : /grid/atlas/dq2/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
[BDII] sam-bdii.cern.ch:2170: Can't contact LDAP server
lcg_cr: Host is down
+ out_error=1
+ set +x
   -------------------- Other endpoint same host -----------

ECDF running for ATLAS

ECDF have now passed ATLAS production validation. The last link in the chain was ensuring that their SRMv1 endpoint was enabled on the UK's DQ2 server at CERN - this allows the ATLAS data management infrastructure to move input data to ECDF.

After that problem was corrected this morning the input data was moved from RAL, a production pilot picked up the job and ran it then the output data was moved back to RAL.

I have asked the ATLAS production people to enable ECDF in the production system and I have turned up the pilot rate to pull in more jobs.

We had a problem with the software area not being group writable (for some reason Alessando's account mapping changed), but this has now been corrected and an install of 14.0.0 has been started.

It's wonderful to now have the prospect of running significant amounts of grid work on the ECDF resource. Well done to everyone!

DPM and ATLAS Space Tokens

So, here's the definitive guide to enabling space tokens for ATLAS:

1. Assign the token to atlas/Role=production using the dpm-reservespace command. (Example here.)
2. Create the directory where DDM will put the files, which is the normal path for ATLAS, plus the name of the spacetoken in lower case. e.g.


• dpns-mkdir /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


3. Now change group ownership of this directory to atlas/Role=production, chmod it to 775 and finally add two ACLs which will mean the entire tree will be writable by production roles:




• dpns-chgrp atlas/Role=production /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-chmod 775 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk


• dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk




4. Enable the space token publisher (this only needs to be done once - subsequent tokens are picked up automatically). Instructions here.
   

DPM SRMv2 Tweaks

Got back from holiday to find that FDR transfers to Glasgow were failing with a "permission denied" error. Looking through the logs it seemed that the srmMkDir call was failing (unlike SRMv1 you need to make the path before transferring data into the SE).

However, it seems the default umask for DPM srmMkDir is 022, which leaves the directory unwritable. As I had tested the /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk path I left it unwritable by Mario's certificate.

I fixed this with a dpns-chmod, but then I wrote a script to patch up the space token directory tree area using DPM ACLs to ensure the directories are group writable by the production role.

You can get the script here: http://www.physics.gla.ac.uk/~graeme/atlas/scripts/atlas-dpm-token-fix.sh.