#include <documentation.h>

I quote from the DPM Developer Documentation

LFC/DPM Database schema
TO DO : describe non straight forward tables/fields....

so, with that in mind, I set about pulling out the number of SRM 2.2 requests vs the no of SRMv1 requests at the site. v1 should be constant (what with all the new users coming onboard) and SRM 2.2 being a rapid increase since we enabled it? well it's not easy to grep from the logs so I thought I'd poke the DB. First off in dpm_db.dpm_req r_type a char(1) field normally has g (get?) and p (put?) but we have just over 1500 rows where type is 'B' (broken?). hmm - all from flavia's DN and clienthost of lxdev25

my plots of the dpm usage are far too spikey to make sense of at the moment, but I'll work on presenting the info a bit clearer.
In the meantime I discovered that it's pretty obvious when we set torque to fill the jobslots in host order (made it easier to drain nodes off) and when we send nodes away to vendors.

Data Management and MICE

I had a chat to one of our MICE PhD students a couple of weeks ago and I was explaining how to use EGEE data management (SRMs, LFCs, FTS, lcg utils, etc.). His comment afterwards was "I didn't know I was going to do a PhD in data management...".

The problem is that all these tools are very low level, so any user community has to build a higher level infrastructure on top of this. Obviously the LHC experiments have done this extensively, but it is frustrating that there is no simple generic data management service for smaller VOs who lack the resources of the larger VOs.

I wonder if this accounts for the popularity of SRB in other communities? It may have some limitations, but it clearly offers a higher level data storage, cataloging and metadata service which must be attractive for smaller user communities. Surely there is a potential project to try and tie all of the EGEE components into a sensible data management system?

News from ATLAS Computing

There is some significant news for sites from ATLAS software week. The decision was taken yesterday to move all ATLAS MC production a pilot job system, with the pilots based on the Panda system developed by US ATLAS. The new system will get a new name, pallete and pallas are the front runners. (I like pallas myself.)

In addition EGEE components, such as LFC, will be standardised on for DDM.

As this is a pilot job system, so the anticipated model is that ATLAS production will keep a steady stream of pilots on ATLAS T2 sites, which pull in real job payloads from the central queue.

This is very like the LHCb MC production model, so as the transition is made to this system sites should start to see much better usage of their resources by ATLAS - just like LHCb are able to scavenge resources from all over the grid.

Canada have recently shifted to Panda production, and greatly increased their ATLAS workrate as a result. There have been some trials of the system in the UK and France, which were also very encouraging.

Of course, there is a very large difference with LHCb, because ATLAS don't just do simulation at T2s, but also do digitisation and reconstruction. These steps require input files and the panda based system for dealing with this is to ensure that the relevant input dataset is staged by the ATLAS distributed data management system (DDM) on your local SE; further the output dataset will be created on your local SE, then DDM will ship it up to the Tier-1 after the jobs have run.

This means that sites will really require to have a working storage system for ATLAS work from now on (in the previous EGEE production model, any of the SEs in your cloud could be used, which masked a lot of site problems but caused us huge data management headaches).

In the end using pilots had two compelling advantages. Firstly, the sanity of the environment can be checked before a job is actually run, which means that panda gets 90% job efficiency (the other EGEE executors struggled to reach 70%). Secondly, and this is the clincher, it means that we can prioritise tasks within ATLAS, which is impossible to do otherwise.

At the moment the push will be to get ATLAS production moved to this new system - probably on a cloud by cloud basis. This should not cause the sites headaches as production is a centralised activity (most sites still have a single atlasprd account anyway). However, the pilots can also run user analysis jobs - and this will require glexec functionality. Alessandra and I stressed to Kors that this must be supported in the glexec non suid mode.

In the UK ATLAS community we now need to get our DQ2 VOBox working properly - at the moment dataset subscriptions in the UK are just much too slow right now.

Postscript: I met Joel at lunchtime, who wanted me to namecheck LHCb's DIRAC system, as panda is based on DIRAC - well, I didn't know that, but I suppose I'll learn a lot more about the internals of these things in the next few months.

DPM Gridftp Resource Consumption

Durham was suffering from excessive resource consumption, from "hung" dpm.gsiftp connections from ATLAS transfers. Because of the way that gridftp v1 servers work huge buffers were being held in memory, leading to resource exhaustion on the machine and a subsequent crash.

Phil and I discussed this, and I noticed that the active network connections were to the RAL FTS server, not to the source SRM, so it looked like it was the control channel which was hung open, not the data channel.

Greig had a look on Glasgow's servers and discovered the same problem, but we were relatively unaffected due to the whopping 8GB of RAM we have in each disk server (and by having 9 disk servers, presumably). Cambridge also reported problems.

The issue is being looked at by the DPM developers, but for the moment Phil's had to write a cron script to kill off the hung ftps to keep gallow's head above water.

The Road To Hell Is Paved With Data Management...

After running rather full for more than a week, we had lots of jobs which seemed to be hanging. On investigation most of these were due to stuck data management commands - mostly lcg-cps. While lcg-cp failed to exit the job was idling as it slowly approached its max wallclock when it would be killed.

As we had a queue of jobs which wanted to run it seemed absurd to leave these slackers until the batch system sent them to their fiery doom. Better to kill them off early and get some well behaved jobs in.

With a small minority of ilc jobs I killed the lcg-cp by hand, and a few of them managed to restart properly. However, when I discovered a biomed user with 110 jobs hung it was going to be far too tedious to try and script checkjob, ssh, grep and kill to attempt to jump start them - so I wielded my trusty qdel.

Unfortunately, qdeling 110 jobs at the one time produced a rather fierce load spike on the CE. The GRIS plugin then timed out and we spasmed into 4444 waiting jobs and the usual 68 years of ERT. Woops! Next time I'll delete them more slowly.

I contacted the users via the CIC portal interface that Alessandra had suggested. For this purpose it seems to work rather well.

Finally, after all of that, we got 3 jobs that were sitting in "W" state. No ideas, nothing useful in the PBS logs and many other things to do, so finally I had to qdel them as well.

It takes a long time to clear out this stuff - a couple of hours at least. Fortunately it doesn't seem to happen too often.

Total Checks Out

The problem with VOMS was that the DN of the gridpp VOMS server was wrong in the setup.

Once I realised that, it all started to work.

Results were that it all works! lcg-cr and lcg-del function perfectly, so it seems the problem is with Hannah's UI.

ScotGrid meets AstroGrid

I was invited to talk about data movement in GridPP/WLCG to the AstroGrid people today. It was really informal and a chance for GridPP and AstroGrid to learn a bit more about one another's problem domains and look at the solutions that each project has adopted.

AstroGrid have a nice virtual filesystem implementation called VOSpace, which allows astronomers to interact with IVOA databases and resources. They now want to extend the concept of these personal virtual spaces into the Astronomy data area. So, a database at the ROE could have a database for an instrument, with tables for different data, and an astronomer would be able to query and store their results in a virtual space within the ROE database (roe/user/graeme/myqueries/hotstars). These results could then be copied out, e.g., into a VOSpace area.

The cunning bit is can you combine query results between databases? i.e., do a join between a database view at ROE and one at Cambridge? Well, this requires a (possibly large) amount of data to be shipped between these two databases.

The question they wanted to ask me was, did I know an efficient way of transferring the data between the databases?

Now, this is quite a different from the LCG problem. So, instead of saying, "buggered if I know" (with my facetious streak, it was tempting ;-), I described the sorts of data flows associated with LCG, the software components we use and the achievements and limitations of our solutions: FTS is great - it ships PB of data around very reliably 24x7, balances VO and site requirements, etc.; FTS is rubbish - it needs Oracle and only talks to SRMs.

Actually, it became quite an interesting and wide ranging chat about grids - different methods of working and how to convince users to use it. They were quite heartened to see that in HEP the VOs really do now overwhelmingly use the grid for their activities.

In the end it seems they actually want to start this quite small - get the virtualised results spaces working first, and then probably tackle the bulk data shipping later. They were definitely interested in looking at an implementation of VOSpace which used an SRM as a backend store (at the moment they haven't really gone beyond the constraints of a single RAID array). And when they do come to look at bulk data movement, then they will look at FTS and RFT as possible methods for doing it.

Nice to make contact with other communities. They get a good view from the top of Blackford Hill...

Final presntation in Melbourne: "Grid Data Management and Storage (An EGEE-Centric View)".

This is organised by VeRSI (The Victorian eResearch Strategic Initiative). They have funding to setup multi-site storage of ~100TB to help scientists in Victoria share data and were very interested in the EGEE DM solutions. I ran though storage, catalogs, SRM 2.2, FTS, etc (presentation here). However, my conclusion was that operations were more important than technology choices - perhaps that is the real lesson from EGEE.

After learning a bit more about their project, I was inclined to recommend dCache for them - they have a muti-site storage problem, with dedicated networking between the data centres and dCache would seem to offer them the most flexible approach. Of course, as with many of the Australian grid projects, SRB seemed to be their de facto soution (needless to say, with shiboleth authentication), however in talking to their SRB expert about the LFC, I finally managed to get a number for the scaling of the SRB MCAT catalog - it can start to have problems when you have over 30,000 files. This seems terribly low - I know the LFC has been tested up to millions on even quite modest hardware (although the LFC is just a file catalog, where as MCAT is also a metadata calalog).

Of course, no software is a panacea, and they all have problems - perhaps the weakness of the EGEE solution is that there are so many bits to it - it would be quite a daunting thing to setup from scratch.

I'll be interested to see what they do decide in the end.

Wow! It's been a busy time here in Melbourne, but just time to blog before getting back on the plane to Scotland.

The first week I was at the ACSW conference in Ballarat. It turned out that AUSGRID was only a 1 day event as part of that, so in fact the primary reason (excuse?) for coming turned out to be the least interesting thing. There were a few good papers as part of that, though. The keynote was by Denis Caromel, from Nice, describing how they have grid "enabled" java, in a package called ProActive. Essentially it replaces all of those awful MPI calls with nice, easy to manipulate java objects. Looks like a splendid way of doing parallel code on the grid. Probably not of huge interest in LCG, where we're mostly embarassingly parallel, but for communities coming from a supercomputing background, or just starting out with grids, it might be a very valuable toolkit to use.

The paper I gave went down well and there was general appeciation of the scale of EGEE and the volume of work we are doing. There was some discussion on SRM vs. SRB, a theme that contunied into my second week.

I met with Lyndsay Hood, the acting program manager of the Australian Partnership for Advanced Computing (APAC), for lunch. We had an interesting chat about their storage issues, where the thing that seems to put them off the EGEE solution is the X509 authentication. Too difficult for normal users, they seem to think. This is something I've heard from people in the UK as well, but no one seems to have a better answer: portals are too limiting, username/password doesn't scale. Shiboleth gets talked about all the time, but it seems to be spoken of as a panecea, rather than a working product that people can use right now.

A general comment on computer science conferences: I've had enough graph theory to last me a lifetime.