SL4 x86_64 UI now Availiable

I reinstalled the site's svr020 UI on Saturday. This involved an incredible amount of pain related to the bizarre inability of SL4 to properly install GRUB on a linux software RAID partition. Although the machine would install absolutely fine, on reboot it would just halt after the GRUB prompt.

In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an

That finally gave me a base SL4 install I could work with.

After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).

Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)

Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).

Job submission with DIRAC

In order to get some "real user" experience of performing physics analysis on the Grid, I have been doing a lot of reading and playing with the LHCb computing software. First of all, there's a lot of it so it takes a while to get to understand what each component does, how they can be linked together, how they are configured and built and how the applications can be run locally or on the Grid to do some real physics.

I was particularly interested in getting some basic jobs running on the Grid, so I quickly started playing with Ganga, the user interface for job configuration and submission. At first I was quite impressed. It was very simple to use Ganga to submit small jobs to the local system, CERN batch or the Grid (via the LHCb DIRAC workload management system). However, a few problems quickly appeared:

1. Jobs were continually failing on the Grid due to poorly configured software installations on the sites. Missing libraries was the main source of problems. It also seems that the latest version of Gauss v30r3 (LHCb MC generation) is a bit broken due to a mis-configured path. These things weren't a problem with Ganga as such, but using it meant that another layer had to potentially be debugged.

2. I found bulk job submission was very difficult in Ganga. Writing the python code to loop over the jobs is easy, but the client just couldn't handle the 100's of jobs going through it. It became very slow and eventually just hung. Even just starting up the client is slow. Maybe running on a non-lxplus machine would be better. There were also inconsistencies between the Ganga job monitoring and that reported by DIRAC.

As an alternative, I decided to bypass Ganga and use the DIRAC API directly. This proved to be quite successful, being much faster for bulk submission. I put together some notes on this, which can be found here:

http://twiki.cern.ch/twiki/bin/view/Main/LHCbEdinburghGroupDIRACAPI

Using DIRAC didn't help with the site mis-configurations (although it is easy to get the job output and check the log files for problems), but I found it a more efficient way of working. I'll try again with Ganga once I understand better the problems that keep on appearing on the Grid.

From my brief foray into running jobs on the Grid, it appears that Ganga/DIRAC do insulate users from malfunctioning middleware, however, there are still real problems when it comes to poorly installed software on the sites. From a deployment point of view, maybe this should be taken as encouragement, as the problem is at the application level and not so much with the middleware. I think we would need to do a more systematic study to find this out (much like Steve's ATLAS jobs).

What is needed is better testing of the sites by through VO-specific SAM tests. This information then has to be fed back into DIRAC (or whatever) so that mis-configured sites can be ignored until their problems are resolved. User will then find running jobs on the Grid a much easier and pleasant experience.

Ganga Quickstart Guide Complete

I have finished the last few sections of the Glasgow Ganga Quickstart Guide.

In particular I have covered bulk job submission using ganga.

Swetha and I used the methods described here to submit a batch of 12 of her jobs onto the cluster this afternoon. She's very pleased to have a scalable method to submit large numbers of jobs onto the cluster - previously she was limited to ~5 jobs on the computing service cluster, which she also had to run interactively. This is much better.

A Morning With Ganga

I (finally) downloaded and installed ganga onto one of the cluster UIs. It's a cinch to install, just run one python script, add an element to your PATH and you're away.

The next step is going through the User Guide. Although this was written for 4.2, there's nothing I found which didn't basically work in 4.3.1.

It's quite easy to setup and run simple jobs:

In [2]:j=Job(application=Executable(exe='/bin/echo',args=['Hello, World']))

In [3]:j.submit()
Ganga.GPIDev.Lib.Job               : INFO     submitting job 0
Ganga.GPIDev.Adapters              : INFO     submitting job 0 to Local backend
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "submitted"
Out[3]: 1

In [4]:outfile=file(j.outputdir+'stdout')
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "running"
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "completed"

In [5]:print outfile.read()
Hello, World

However, that's spawning a job which just runs on the local machine - how easy was it to run on the grid?

Answer: very easy!

In [39]:l1=Job(backend=LCG())
In [42]:l1.application=Executable(exe='/bin/echo',args=['Hello, World'])

In [44]:l1.submit()
Ganga.GPIDev.Lib.Job               : INFO     submitting job 3
Ganga.GPIDev.Adapters              : INFO     submitting job 3 to LCG backend
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "submitted"
Out[44]: 1

In [45]:l1.status
Out[45]: submitted

In [46]:l1.backend 
Out[46]: LCG (
 status = 'Ready' ,
 reason = 'unavailable' ,
 iocache = '' ,
 CE = None ,
 middleware = 'EDG' ,
 actualCE = 'ce.ulakbim.gov.tr:2119/jobmanager-lcgpbs-dteam' ,
 id = 'https://svr023.gla.scotgrid.ac.uk:9000/-wOjj5xKfjcrKIXiCEISNA' ,
 jobtype = 'Normal' ,
 exitcode = None ,
 requirements = LCGRequirements (
    other = [] ,
    nodenumber = 1 ,
    memory = None ,
    software = [] ,
    ipconnectivity = 0 ,
    cputime = None ,
    walltime = None
    )
 )
In [52]:
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "completing"
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "completed"

In [54]:print file(l1.outputdir+'stdout').read()
Hello, World

Wonderful! Didn't have to do any of that nasty edg-job-* stuff. And it ran in Turkey - pretty cool.

I have also now discovered how to define and submit batches of jobs to the grid. This snippet defines a set of 10 jobs:

a=list()
for i in range(10):
    a.append(Executable(exe='/bin/echo', args=[str(i)]))
s=ExeSplitter(apps=a)
j=Job(splitter=s,backend=LCG())
j.submit()

Submitted that and I'm running jobs in China, Italy, Greece, Pakistan, Russia, Austria, France, Spain and Switzerland.

I think this is the first time in a while I've thought "Hey! The grid is actually cool."