Take my outputs, damn you...

We recently ran up a very large backlog of production output files waiting to go from Glasgow back to the Tier-1 (reminder, panda doesn't consider a job finished until the outputs are safely stored at the T1). This is clearly seen in the red line on the panglia plot above, which reaches very high values. As we recently cut the timeout for the UK cloud to 2 days for transferring jobs, to improve the responsiveness of the production system, we started to leak out failed jobs (light green line) as panda gave up and decided to rerun.

Fortunately we got a big boost in the number of FTS slots from Glasgow to RAL, increasing from 10 to 25 active transfers (see the bottom FTS monitoring plot). Even so it clearly takes 24 hours for all the backlogs to drain down.

One of the problems here is that the output files are small from simulation (a tiny log file and a 20-50MB HITS file), so the overheads of FTS + SRM are very considerable and the actual bandwidth achieved is quite low. One possibility we are considering in ATLAS is introducing a pre-merge of outputs on the T2, which will allow us to send much bigger files back to the T1 (although a final "super-merge" will probably still be necessary). For this we are waiting for the generic Athena merge transform and then we will need to test integrating this into the mainline production workflow.

Until then we just have to take the operational load of tweaking the FTS settings when necessary.

Glasgow on the move

Because of the current problems at RAL, Glasgow was nominated as a test peripatetic Tier-2, so see how agile ATLAS production was at moving Tier-2 resources in case of T1 downtime (note this test only works if the cloud services, FTS and LFC, are still running - if these are gone then it's almost impossible, today, to use any of the cloud's Tier-2s).

First off Glasgow was sent South East, into the NL cloud. Here we found a problem with the input datasets, because input datasets to T2s (which are subscribed without sources) only look for sources within the cloud (this follows the ATLAS computing model). However, the way around this is to specify the associated T1 (for production) as the source and then DQ2 does the work. The panda developers made the change on Friday, so that NIKHEF was specified as the source for inputs to Glasgow. Likewise, for output back to the NL T1, Glasgow's PRODDISK token was specified explicity as the source.

That done, Glasgow galloped through a couple of hundred jobs for the NL cloud, before they ran out of jobs:

Flushed with this success we've just shoved Glasgow into the FR cloud for a while, as they still have jobs left to run. Within an hour we're running a couple of 100 jobs.

News from ATLAS Computing

There is some significant news for sites from ATLAS software week. The decision was taken yesterday to move all ATLAS MC production a pilot job system, with the pilots based on the Panda system developed by US ATLAS. The new system will get a new name, pallete and pallas are the front runners. (I like pallas myself.)

In addition EGEE components, such as LFC, will be standardised on for DDM.

As this is a pilot job system, so the anticipated model is that ATLAS production will keep a steady stream of pilots on ATLAS T2 sites, which pull in real job payloads from the central queue.

This is very like the LHCb MC production model, so as the transition is made to this system sites should start to see much better usage of their resources by ATLAS - just like LHCb are able to scavenge resources from all over the grid.

Canada have recently shifted to Panda production, and greatly increased their ATLAS workrate as a result. There have been some trials of the system in the UK and France, which were also very encouraging.

Of course, there is a very large difference with LHCb, because ATLAS don't just do simulation at T2s, but also do digitisation and reconstruction. These steps require input files and the panda based system for dealing with this is to ensure that the relevant input dataset is staged by the ATLAS distributed data management system (DDM) on your local SE; further the output dataset will be created on your local SE, then DDM will ship it up to the Tier-1 after the jobs have run.

This means that sites will really require to have a working storage system for ATLAS work from now on (in the previous EGEE production model, any of the SEs in your cloud could be used, which masked a lot of site problems but caused us huge data management headaches).

In the end using pilots had two compelling advantages. Firstly, the sanity of the environment can be checked before a job is actually run, which means that panda gets 90% job efficiency (the other EGEE executors struggled to reach 70%). Secondly, and this is the clincher, it means that we can prioritise tasks within ATLAS, which is impossible to do otherwise.

At the moment the push will be to get ATLAS production moved to this new system - probably on a cloud by cloud basis. This should not cause the sites headaches as production is a centralised activity (most sites still have a single atlasprd account anyway). However, the pilots can also run user analysis jobs - and this will require glexec functionality. Alessandra and I stressed to Kors that this must be supported in the glexec non suid mode.

In the UK ATLAS community we now need to get our DQ2 VOBox working properly - at the moment dataset subscriptions in the UK are just much too slow right now.

Postscript: I met Joel at lunchtime, who wanted me to namecheck LHCb's DIRAC system, as panda is based on DIRAC - well, I didn't know that, but I suppose I'll learn a lot more about the internals of these things in the next few months.

ScotGrid and AliEn

After a long hiatus, Dan and I sat down to look at getting the AliEn system submitting properly into ScotGrid. Battling through OO Perl (yuk!) we found the various points where the system seemed to lack configuration information - this was basically a value for the site BDII to query free slots, a CE queue name and the name of the VO (gridpp). When we'd hacked those values into the LCG.pm AliEn file then things started to look a lot rosier.

However, I think that AliEn is going to want a dedicated queue, because of the way it parses the BDII's output, which inclines us towards setting up a VO for panda sooner rather than later.