Pheno goes Bang!

Crisis on the cluster this morning. After a long night of job submission by a phenogrid user (putting in more than 1000 jobs) the cluster went into a spasm, where the pheno jobs started to hit wait state en masse. Then what I think happened was that as torque saw each pheno job hit wait, failing to start, it immediately picked the next pheno job, tried to start that, failed, tried to start the next, and so on. This resulted in a load storm within torque (loads >100), which was then not even able to answer normal client queries - so maui locked up and the gip plugin started to timeout.

When I realised what was happening (and the pheno jobs were still coming in) I added the user's DN to the LCAS ban_users.db file. I then carried out some debugging tests, restarting maui, clearing out maui stats files, etc. In the end I saw no option but to qdel the user's waiting jobs, to attempt to take the pressure off torque.

Once the jobs were flushed out the system torque quite quickly started to recover. Maui started to respond again and the GIP plugin could get sensible answers.

Why were the jobs going into waiting state? The error the user seemed to be getting back was "Globus error 158: the job manager could not lock the state lock file." This seems to be an error which crops up when the job is being cancelled. There was a strange mix of jobs from this user - some with VOMS extensions, some vanilla proxy. Was this a problem with proxy renewal and the gatekeeper trying to cancel jobs which it no longer had the right to? The problem kicked in at almost exactly the time that the user's original submission proxy expired and the RB would have renewed it from the RAL MyProxy server. The wrong proxy might well also have affected the ability of the jobs to start - hence the wait crisis being sparked.

After I had been satisfied that the cluster was stable again, I took the user out of the banned list. Their jobs are now flowing back into the cluster, interestingly all with the vanilla proxy now.

I will keep a close eye on things and check that things don't go wrong again.

Postscript: VOMS proxy renewal is broken: http://savannah.cern.ch/bugs/?func=detailitem&item_id=15208

Multiple VO Woes

Steve Lloyd and I sat down after lunch today to try and get to the bottom of why his dteam submitted jobs always fail. Strangely this seems to be a RB specific problem. IC always works, Glasgow always fails and RAL seems to come and go.

Using the Glasgow RB we submitted a job to Edinburgh, so that we could trace things through the batch system. The job arrived at Edinburgh, and ran through the batch system. However, it continued to be considered by the RB as

Current Status: Scheduled
Status Reason: Job successfully submitted to Globus

Clearly this was not the case.

We had a good look through the logs on the RB, but there's no particular sign of things going wrong there - although it must be said that the logs are both dense and impenetrable.

When it became clear that there was no easy solution I decided to try and reproduce the problem myself. Now, recall I had joined gridpp a while ago to help our local users and never had any trouble. However, now I can't seem to get a single job running through as a gridpp member - even on the Glasgow cluster. And things are in fact even worse than for Steve, because my gatekeeper process dies almost instantly, so the job never even goes into the batch system:

grep 2007-07-10.14:49:10.0000028268.0000113028 /var/log/messages
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 for /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart on 130.209.239.23
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 mapped to gridpp001 (17601, 10016)
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 has GRAM_SCRIPT_JOB_ID 1184075356:lcgpbs:internal_434559272:9672.1184075355 manager type lcgpbs
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 JM exiting

I'll now try and poke around inside the gatekeeper logs and see if I can come up with any indication why things are going wrong.

And what the hell's this got to do with the RB anyway? It's deeply puzzling and frustrating in equal measure.

New Disk Servers Deployed

We've now completed the deployment of our 5 additional disk servers for ATLAS. This takes our SRM space total to 84TB, with 77TB for ATLAS.

There is about 12TB used so far.

Top Level BDII Updated to Glue 1.3/yaim 3.0.1

I upgraded our top level BDII today, so it should now happily understand the Glue 1.3 schema. This was the deadline for the upgrade set in the EGEE operations meeting - BDIIs have to be upgraded from the top down, because they don't like information provided by lower levels to be in a schema they don't understand.

This was a more significant thing it might first seem, as I also went very carefully through our site-info.def file and used the latest 3.0.1 yaim to configure the node.

It turned out that there were significantly fewer changes than I had thought there would be. The main additions seemed to be exposing new BDII configuration variables, and changing the way that queues are configured, to allow for queues accessible to FQANs. Of course, much of this is not relevant to the BDII/MON box, which doesn't care about users or VOs, but it proves that there's no major errors in the new configuration.

I have noticed that the startup time for the BDII is slower, presumably as it's creating all of its index files; however, the performance should be much better.

Success with MPI

Cracked it! I can now get MPI jobs running on the Glasgow cluster.

First thing to note is that the gatekeeper does not invoke mpirun for the job - this is very good, because it would be almost impossible to get this to work if it did.

The key file is the NODELIST file which the CE will generate and add to the executable's argument list. When given as the argument of the -p4pg option then mpirun will ssh to all of the "slave" nodes and start the binary which is given in the NODELIST file.

By default this breaks for 2 reasons:

1. The gatekeeper only copies the job's sandbox into the working directory of the "master" worker node. So on the "slave" nodes the executable isn't present. (N.B. Even though we have a shared data area for our glaNNN accounts, the working directory is always in /tmp and local to the worker node.)
   
2. The executable listed really needs to be a wrapper script, so it's the wrong thing for mpirun to be starting anyway.

So, wrapper script really has to do the following:

1. Change to a more sensible shared directory (like $CLUSTER_SHARED).
2. Rewrite the NODELIST file so that the name of the correct mpi binary to run is given, instead of the wrapper script itself.
3. Invoke mpirun, giving the new NODELIST file.

Here's an example (with a lot of debugging hooks) which works, running the code:

#! /bin/sh
#
# Argument list is: BINARY -p4pg NODELIST -p4wd PATH
# What's really important for us is the NODELIST file, i.e., $3
cd $CLUSTER_SHARED/mpi
export MYBIN=$1
PGFILE=`pwd`/pgfile.`hostname -s`.$$
echo My Args: $@
echo "----"
echo "Original NODELIST file:
cat $3
echo "----"
cat $3 | perl -ne 'print "$1 $2 /cluster/share/gla012/mpi/$ENV{\"MYBIN\"}\n" if /^([\w\.]+)\s+(\d+)/;' > $PGFILE
echo "----"
echo "New NODELIST file:
cat $PGFILE
echo "----"
/opt/mpich-1.2.7p1/bin/mpirun $MYBIN -p4pg $PGFILE

(Hmm, it's splitting that perl one liner in a really nasty way - no line breaks there.)

There are, however, two problems which I can see.

1. Accounting. Looking at the torque logs it's clear that only the master node's process is being accounted for. The slave node MPI processes are not accounted for. Do we multiply the master node's CPU and Wall by the node number as an interim measure?
2. Orphaned and stray processes. As ssh is used to start the binary on the slave nodes, what happens if the code leaves them behind or they run away?

I wonder if there's a way we can modify mpirun to do things in a torque friendly way? I shall enquire of the MPI gurus.

(For more formal documentation, watch this wiki page....)

MPI Progress

I am making progress with MPI jobs. I can now get MPICH jobs into the batch system via edg-job-submit and they do get a batch system reservation.

It turned out I had to add MPICH as a GlueHostApplicationSoftwareRunTimeEnvironment in the information system. It's also essential to have GlueCEInfoLRMSType as pbs. It doesn't work if you put torque (it must be the only thing on the grid that actually cares!).

The job wrapper then adds some interesting arguments to the executable:

-p4pg NODELIST -p4wd PATH

Where NODELIST looks like this:

node067.beowulf.cluster 0 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node070 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node102 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node139 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh

and PATH is just the working directory for the job. Note the magic number "0" seems to be the place where the job executable runs and "1" are all the nodes where other job slots are reserved for this job.

So clearly the NODELIST file then needs to be taken by mpirun and used to start all the mpi subprocesses. From the EGEE MPI Wiki, the standard method seems to like to use the i2g mpi-start command, so the arguments must be in a form appropriate for it. Open questions remain, though:

1. How to get i2g mpi-start to work. When I give it an MPI binary it seems determined to compile it - however this falls over, even though MPICH 1.2.7 is in the path.
2. How do I ignore mpi-start and run a pre-prepared MPI binary, which will be what a Glasgow use wants to do.
3. How on earth will torque account for all of this properly?

Further reading: EGEE-II-MPI-WG-TEC.doc.

LHCb Stuck Jobs

Coincidentally with drafting the stalled jobs document, we got 23 stalled LHCb jobs last Friday. These jobs had consumed about a minute of CPU then just stopped.

I reported them to lhcb-production@cern.ch and the response from LHCb was very swift and helpful. We did quite a bit of debugging on them - although in the end we had to confess that exactly why these ones had stalled was something of a mystery. At first LHCb thought that NFS might have gone wobbly at our end, so the jobs got stuck reading the VO software. From what I could see this was unlikely, and when NIKHEF, RAL and IN2P3 reported similar problems we were off the hook.

Some useful tools for stuck jobs:

• lsof - see what file handles are open
• strace - what's the job doing right now
• gdb - attach a debugger to the code
  

In fact, a lot of simple diagnostics also help: what's in the job's running directory. What STDOUR/STDERR has been produced to far, etc.

When these jobs are killed it's helpful to poke the stalled process - that way information gets back to the VO. A qdel will see the outputs all lost and the job resubmitted elsewhere, which is far less helpful.

In the end, whatever the bug is, it's down at the 10^-6 level!

Thanks to LHCb for being so responsive.

I also must take my hat off to Paul and his MonAMI torque plugin. His live efficiency plots for the batch system queues made spotting this very easy. In the past this sort of thing would have been noticed on a very hit or miss basis.

Health and Efficiency

As part of investigating the problems of stalled jobs, I have plotted Wall vs. CPU time for ATLAS and LHCb on our cluster.

LHCb jobs are generally quite efficient (as evidenced by their 93% efficiency from the EGEE accounting pages). What's interesting is seeing the cluster of jobs at 11 and 22 hours of CPU time, with a smear in wall clock from prefect efficiency to ~50% (data management stikes again?).

ATLAS jobs have a far more variable profile, with many more short jobs of high efficiency, with a more general, and flatter line out to lower efficiencies. There's a very distinct line of problematic jobs (the spike on the tail).

It seems really that with our new fast CPUs our queue times are really much too long (inherited from the old cluster, if I remember). LHCb and ATLAS both seem happy for queues to be reduced from 96/100 hours to 36/36 hours.

ATLAS Software Week

Last week I was at ATLAS Software Week at CERN.

It was a useful meeting (as ever meeting people and chatting is most important!). Some issues I picked up for ATLAS sites were:

1. Although 13.0.10 has been released there are quite a few things known broken (event generation, for instance). This means we are stuck with having a lot of "old" ATLAS software releases on our sites. At Glasgow we have 86GB of ATLAS software - more then 60% of the total for all VOs.
   
2. Preparations for Computing System Commissioning and the Final Dress Rehearsal are underway. The start date seems to have slipped (was meant to start this week)? Actually, I must find out what the site involvement schedule actually is.
3. The DQ2 data management system was upgraded to 0.3 last week. There were a few teething troubles, but the next release should handle many common problems much better.
   
4. There's pressure not to run too many simulations as part of each job sent to a site - so keep the wallclock down (< 24 hours), but this reduced the file sizes. Small files are a big problem - they are inefficient to transfer and gunge up any tape system. So they should really be merged before any migration to tape. (Problem for CASTOR though, which even puts T0D1 stuff onto tape?)
   
5. Event sizes keep going up. Computing TDR had ESD at 0.5MB, but currently this is 1.6MB (1.8 for MC). Probably a realistic target will be 1.3MB files.
6. Memory footprints are rising too. 2GB necessary for simulation and probably a subset of reconstruction jobs too.
7. To deal with merging and pile-up jobs worker nodes should now be speced with at least 20GB of disk space per core. At the moment, however, jobs will try and limit their ambitions to 10GB. However, this requirement also seems monotonic, so make sure it's accounted for in forthcoming purchases.
   
8. Queues for ATLAS production should be around 24 to 36 hours of cpu and wall time (N.B. this is on modern CPUs). NIKHEF are currently at 24/36 and I'm going to cut Glasgow back to 36 hours.
   
9. If you see stuck ATLAS jobs try and investigate the problem and report to atlas-comp-oper@cern.ch. This will help cut off the nasty tail in the ATLAS efficiency curve.

RB Twiddles

Steve Lloyd's started to monitor our RB. So far things look good! I also discovered that SAM had picked up the RB and was at least checking its host certificate. I have now added the RB to the GOC and requested monitoring - another pair of eyes to pick up problems.

As we are getting rather more officially known, I've reduced the number of advertised VOs on the Glasgow RB to ops, dteam (for testing), atlas, pheno and gridpp. There's a slightly different site-info.def used for the RB to do this (site-info-rb.def) and I changed cfengine to run against this file if the node needs rebuilt.

Even though the RB has only been up for 3 months the MySQL database is now 270MB. Although this is well below the 4GB where Olivier has reported problems, we will have to monitor this to ensure we don't get into stability problems.

At the moment, though, the machine is very lightly loaded.

Tails and Spikes

Tony and I have been trying to draft a policy on killing off jobs which just fail to start properly, so I pulled some stats out of our local accounting MySQL database and plotted a histogram of job efficiencies. This is a very interesting plot - a clear "decay" down from high efficiency into a long tail, then a significant spike of very low efficiency jobs (< 0.02).

Jeremy said that Dario was quite sanguine about killing these sorts of jobs off - things which fail to consume CPU after 6 hours are probably never going to get anywhere.

However, it turns out this is a bit of a can of worms. The RB will resubmit the job (up to 3 times) and the same thing might happen again on a different site. On the other hand, jobs running out of wall clock look the same to the user - and the RB will also resubmit them! If we do kill off jobs, should we email the user? Is this scalable in terms of our time? How much information do we provide to the end user? Will they even care?

It will be an interesting discussion.

Glasgow RB now in BDII

After humming and hawing about our "trial" RB, I decided to bite the bullet and publish it into the site, and thence top level, BDII. This makes things easier for the RTM, among other things. So far the RB has been really trouble free to run and had been lightly loaded - fingers crossed.

I think our "policy" here should be to support VOs in which we have a strong interest, e.g., ATLAS, gridpp, pheno, ScotGrid VOs. This will mean splitting our site-info.def file, though, because there will be a different list of supported VOs on the RB to the CE/SE.

Of course, the LCG-RB's days should be numbered - it seems that the gLite WMS now outperforms it. If this is true then there really is little point in trying to learn about it, instead we should put our efforts into the new system.

BDII Status

After moving back to our own BDII things have been more or less ok - we had a wobble on Sunday with 2 timeouts. Overall the load on svr019 is definitely creeping up again though - is there some systematic involved here?

I met Steve Traylen in R1 at CERN at lunchtime and he said that the indexing patch for the BDII has already been applied at CERN, so it seems that this is something which it's fairly safe to grab. I'll put it on the TODO list for next week.

I seem to be spending a disproportionate amount of my time worrying about BDII failures!

Back to our own BDII

The RAL BDII timed out three times in 24 hours, so I've moved back to the ScotGrid one.

Oh, for a working information system...

lftp woes

Fed up with lftp mirroring being broken.

Discovered a nice simple sanity check though for making sure we don't do a Greig:

grep ^lcd /etc/mirror.conf | awk '{print "ls -ld " $2}' | /bin/sh

Ganga Quickstart Guide Complete

I have finished the last few sections of the Glasgow Ganga Quickstart Guide.

In particular I have covered bulk job submission using ganga.

Swetha and I used the methods described here to submit a batch of 12 of her jobs onto the cluster this afternoon. She's very pleased to have a scalable method to submit large numbers of jobs onto the cluster - previously she was limited to ~5 jobs on the computing service cluster, which she also had to run interactively. This is much better.

Brain transplant leftovers

Bah! yum update has failed to get the headers (404) when tested on a client node and the pxe/dhcp/tftp part of the build process still needs working on. We've also got 2 nodes that require a re-image to bring them under the full control of the build system. Looks like today is going to be spent making sure the YPF is working properly.

Stuff that needs to be done:
* clusterdb to dnsmasq - it should spit out the MAC addressess to file ready to use.
* Check all the apache aliases for build
* check the autokick stuff.

then reimage machines. Hopefully that'll reduce the no of red bits in nagios down to a less boss-scaring amount. Once thats done we can work on removing the false alarms (where we monitor non-existing services that we know are't configured yet)

Switched Back to RAL BDII

We failed another couple of CE-RM tests over the weekend. It's clear this started when the load on the machine crept up from ~0.5 to ~1.0 about 2 weeks ago. I didn't change anything on the box, so I am mystified as to what's caused this change. Perhaps it's a greater load being put on the BDII by expanded use of the RB?

I have switched back to using the RAL BDII for the moment and we haven't failed since then. I may setup an additional top level BDII on svr017, which is the unloaded scotgrid admin node, and see if that has a lower overall load.

I will also upgrade the BDII to the new release, which uses indexes which speeds up queries, and see if that helps.

Queue Tweaks and Maui

As we've been really full recently, I have reduced the maxwallclock available to grid VOs from 148 to 100 hours. The maxcpu time stays the same at 96 hours. I'm growing very frustrated with jobs which just stall at the start - we had 9 atlas jobs which consumed 1s of cpu time in 9 hours, hanging on an lcg-cp.

I also increased the maxcpu and wallclock on the gridpp queue to 168 hours, to make sure that Swetha's bio jobs run through ok - 96 hours was probably too close to the wire. We can cut our local users a bit more slack as their jobs, when they do run, tend to be almost 100% efficient.

I had to reduce the maxprocs on the glee queue to 400 - we can't really afford to get the whole cluster filled with EE jobs as their maxcpu/wallclock is so high at 28 days, and this will completely mess up fairsharing.

We've been suffering this weekend from two very large job surges from pheno and glee. As these groups have a large, but underused, fairshare, they get to start an awful lot of jobs in a short time - and as they run for a very long time then the cluster starts to suffer from very few jobs slots coming free and can't run anything for anyone else. 100hr/560 is 11 minutes, but the job inflow, when local users are involved, is far from uniform.

I would like to move the *sgm jobs from atlas and lhcb into the dteam/ops reserved job slot. I will have to ask Sam how to do this.

A Less Reliable Week

After weeks of perfection, I'm now picky about even a small number of failures.

We had 4 BDII timeouts this week, which is worrying. My inclination is to give R-GMA a hard stare - its close waits have still been misbehaving again and we see load/network spikes which seem to be R-GMA related. However, the BDII was perfect even when R-GMA was occupying more than 1000 close waits, so perhaps it's somewhat unclear. When I looked with top, it actually seemed that most of the CPU was being consumed by slapdadd and slapd. Well, one to keep an eye on.

We also had a copule of JS failures with the error message "Got a job held event, reason: Globus error 79: connecting to the job manager failed.". The GOC wiki suggests this is probably a networking problem, however, that seems unlikely. Is this the gatekeeper deciding that the presented certificate does not match that of the job submitted? There's also a suggestion that the GLOBUS_TCP_PORT_RANGE might be wrong, but we've never changed this from the default 20000-25000 range, so that also seems unlikely.

Again, this probably requires some detailed examination of the gatekeeper logs to see if the connection got through.

Start Them Young...

Spent most of the last 2 days showing S5/6 pupils round the computer cluster and talking about the LHC. It was quite fun, but rather difficult to give a talk in an air conditioned noisy room - but their time was so short there was no chance of taking them to a quieter room to speak to them. I scooped up various posters from around the department (proton model, aerial view of LHC, ATLAS, ScotGrid poster) and had the RTM running on the projector.

Unfortunately RTM was a bit broken, but they could still see Europe pulsing with computer centres and be impressed.

Impressive storage facts (measured in 8GB iPod units):

1. The LHC would fill an 8GB iPod in < 10s
   
2. If you stored a year of LHC data on 8GB iPods the pile would be 12.5km high.
3. If you tried to listen to all of that iPod data as music, it would take you almost 30 000 years.

School kids often affect extreme levels of disengagement, but most of them seemed to enjoy it.

Come and study physics then!

Bad Worker Node, Bad, Bad...

node040 had the wrong gateway, via svr031 instead of nat005. When I gated it through nat005 external networking started to work. When svr031 was reinstalled we turned off packet routing on svr031 - however at the time we switched the rest of the cluster to nat005 (when svr031 lost its brain) node040 was down. It never got the change and the default gateway is set at install time - not controlled by cfengine.

In theory this should never happen again - all the install time network files are now correct and nodes should be re-installed when they are brought back into service. However, a nagios monitor which tests external networking should be implemented.

The reason the problem kicks in when the cluster gets full is that node040 can't run jobs (can't get the input sandbox from the RB), so the RB job wrapper gives up after ~5 minutes. Frees up the job slot and another job gets sucked in to its doom. When the cluster is less full this is much more sporadic, as a resubmitted job is likely to go to a different (functioning) node. When I looked at the logs I saw that node040 killed a remarkable number of jobs in its 8 day reign of terror: 1835 out of 11885 have been sent to their doom (15.4%).

For interest, I wrote a little python log parser, which can print number of jobs, avg. cpu and wall time (in minutes). Even just running it over Thursday's pbs log shows up node040 as a bad place to be:

svr016:/var/spool/pbs/server_priv/accounting# nodestat.py 20070607 | sort -k 2 -n
node043: 4 1069.9 1165.9
node045: 4 1080.3 1178.0
node048: 4 1185.8 1195.9
node013: 5 1171.3 1345.5
node032: 5 959.9 1030.7
[...]
node088: 12 438.0 445.3
node019: 13 433.7 437.4
node023: 13 667.8 719.0
node103: 13 397.1 430.7
node139: 62 70.4 80.3
node040: 646 0.5 4.7

Note that node139 is the node most of the ops tests run on, because they have a reserved job slot - even though it's not strictly tied to a node it rarely runs anywhere else. This is really a pain, because I'm sure we'd have picked up the problem earlier - ops jobs never get resubmitted. Perhaps we should remove that reservation. I'm fairly confident that with 500+ job slots we'll get something coming free in under an hour. (6 days / 500 ~ 20min).

Glasgow CE Flaky

It looks like we're having CE problems at Glasgow. We failed a SAM test at 1am, with the error "Globus error 79: connecting to the job manager failed. Possible reasons: job terminated, invalid job contact, network problems, ...". This is very reminiscent of the errors seen by our engineers who use GRAM submission.

We also seemed to fail to run atlassgm tests for long enough that we've been blacklisted by ATLAS in the FCR - though here the tests are just missing, so I don't know what went wrong.

We passed a test at 1030, so hopefully we'll be back in soon.

It's urgent that I get to the bottom of this.

I have checked the gatekeeper logs, and the jobs are being mapped properly to atlassgm at regular intervals. I have checked the WNs and there's nothing evil here - ssh working, disks not full, nfs mounts ok. The exit status of all the jobs from the batch system is 0. The failing jobs were not consistently given to one WN, which might explain the issue. We even passed tests on node040 yesterday, then failed in the early hours of this morning.

Help!

An Unhappy Night With Steve's Tests

I'm very pleased that Glasgow is the top site for Steve's ATLAS tests, but last night we seemed to have a miserable time, failing ~10/30 tests. These were all ABORTS from the RB, which was the IC RB in each case. And all of the successful jobs also came from the IC RB - so it wasn't that we had completely fallen out. And Glasgow's the only site affected, so I think it must be a site issue. However, there just isn't enough information in the logfiles to be able to tell why the job's aborting.

The first thing I checked was autofs (I added a new map yesterday), but this was ok. /home and /tmp are also fine. I'll have to dig into torque and see what I can find.

It really is annoying hard to pin these things down in the baroque dance which is EDG job submission...

SAM tests changed VOMS Role (without warning!)

We, and a large fraction of the rest of the grid, started to fail replica management tests late on Friday night. At first I thought it must be a catalog problem at cern, so I raised a ticket. However, it turned out that what had actually happened was that the VOMS role used to submit the SAM tests had changed. This caused DPM to map the SAM tester DN into a different group - who then did not have permission to write into the default generated directory for lcg-cr.

This change was made completely unannounced, and I suspect without any real thought as to the implications for sites using DPM 1.6.3 and earlier.

Maarten Litmath helpfully posted a fix-up script on LCG-ROLLOUT, which uses ACLs to grant suitable privileges to lcgadmin and production roles for each supported VO, which I applied to Glasgow and Durham at about midnight last night (I was surely violating cardinal rules of sysadmining, but I couldn't see how it would cause harm - and this time I got away with it). This did fix the problem.

I'm really annoyed that this though. Changes like this should never, ever be made on a Friday! (It seemed the change actually came through at ~10am, but didn't break until midnight, when the next YYYY-MM-DD directory needed to be created.) In addition several people have commented that the fix is to upgrade to DPM 1.6.4 - despite the fact that this is broken in gLite 3.0r25 in two significant ways!

Grrrr. I just hope they don't ask us to explain these SFT failures - they shall have a piece of my mind... (I sound just like my Mum, when she was annoyed - see what the grid's doing to me!).

Transfer Tests

Continuing with transfer tests between RAL-T2 and Glasgow over the weekend to investigate the effect of the Streams setting within glite-transfer-channel-set. Turns out it's not a very clear effect - negligible at best. As Greig has already noted that -T 1 is best for dcache, I propose we leave them at that.

However, totalling up the traffic for the weekend, I've moved 6.3 TB since friday afternoon (41h) which gives us a sustained average bandwith of 343Mb/s

But... Thats only 6.3 out of a requested 7.5. Only 7/15 transfers completed sucessfully with all 500 files being transferred. The others copped out with:

(mostly) FTS Reason: Failed on SRM get: Failed To Get SURL. Error in srm__get: service timeout.
(some) FTS Reason: Failed on SRM get: SRM getRequestStatus timed out on get
(twice) FTS Reason: Failed on SRM put: Failed SRM put on httpg://svr018.gla.scotgrid.ac.uk:8443/srm/managerv1 ; id=... call. Error is File exists

So, considering I was transferring the same 50 seed files, thats quite alot of crapness on behalf of the dCache source. Not sure what caused the two false "file exists" failures. I severely doubt that more than 10% failure rate is acceptable to the experiments.

I'll present the full findings (once I've plotted them) at the GDB Meeting on Tuesday

Worth noting that Pauls MonAMI data was v useful in keeping an eye on the dteam specific data pool usage, together with the health of the DPM service at Glasgow.

Users and Stalled Jobs

I noted that I had qdeled quite a few jobs from the cluster two days ago. Well, the ILC user I contacted through the CIC portal responded, apologised and thanked me for the suggestion of adding a timeout to lcg-cp. However, the biomed user (110 jobs) didn't answer. And in fact they continued to submit jobs onto the cluster and I was forced to clear out another 42 jobs this afternoon. So, if nice doesn't work, try nasty - I have ticketed them and will ban them from the cluster if I don't hear back within 24 hours.

ScotLUG talk

Just a FYI - I shall be speaking at the Scottish Linux Users' Group on Thursday evening giving a brief (3 slides) introduction to "The Grid" from a WLCG viewpoint, the computing demands, andthen moving on to how we use cfengine and caffeine to keep sane (more than 3 slides).

Not that it's a rehash of my HEPiX talk or anything :-)
Details will be on http://www.scotlug.org.uk/wiki/2007-05-31

VOMS is a Good Thing (tm)

Spent some of yesterday playing with VOMS and the static grid-mapfile mappings we put into the system. The point of these was to support users who were not yet in any VO and might be using globus-job-* commands to submit jobs to the cluster. For these users it was obviously important that they mapped to the same account on the batch system as on the UI. (What actually happens is that the UI mirrors its grid-mapfile from the CE.)

However, we now have a different class of user, one who might use gsissh to access the cluster, but use the RB to submit jobs and be in a VO. For this user we need a local mapping in the grid-mapfile, but we need to ensure that job submission is made with a normal VO mapped pool account.

To test this I mapped myself to a local account (gla012) on the UI, using gsissh to login. Then I initalised a VOMS proxy and submitted a job. The job correctly ran as dteam.

This means we can support our multiple classes of users, and also users in multiple VOs, in a relatively straightforward way.

One point for the local account, though, is that we will put the local account into the correct group for the user's expected VO. This means that job submission with a vanilla proxy will work as expected.

(An outstanding problem might well be data write access - at the moment the data users put onto nfs using their local account is fine for writing, but the area cannot be written to. This may be an issue for some people.)

The Road To Hell Is Paved With Data Management...

After running rather full for more than a week, we had lots of jobs which seemed to be hanging. On investigation most of these were due to stuck data management commands - mostly lcg-cps. While lcg-cp failed to exit the job was idling as it slowly approached its max wallclock when it would be killed.

As we had a queue of jobs which wanted to run it seemed absurd to leave these slackers until the batch system sent them to their fiery doom. Better to kill them off early and get some well behaved jobs in.

With a small minority of ilc jobs I killed the lcg-cp by hand, and a few of them managed to restart properly. However, when I discovered a biomed user with 110 jobs hung it was going to be far too tedious to try and script checkjob, ssh, grep and kill to attempt to jump start them - so I wielded my trusty qdel.

Unfortunately, qdeling 110 jobs at the one time produced a rather fierce load spike on the CE. The GRIS plugin then timed out and we spasmed into 4444 waiting jobs and the usual 68 years of ERT. Woops! Next time I'll delete them more slowly.

I contacted the users via the CIC portal interface that Alessandra had suggested. For this purpose it seems to work rather well.

Finally, after all of that, we got 3 jobs that were sitting in "W" state. No ideas, nothing useful in the PBS logs and many other things to do, so finally I had to qdel them as well.

It takes a long time to clear out this stuff - a couple of hours at least. Fortunately it doesn't seem to happen too often.

Transfer Channels

Back to 'proper' work, and I realised that the transfer-channel sizes are still very conservative (with the exception of Imperial). Rather than just increasing to a high value, I thought I'd plot some timings to show the difference of a 0.5TB FTS session.

The 4 steps on the image show: seeding to ral-T2 with lcg-cr from local workstation, then 3 sets of transfers up to uki-scotgrid-glasgow with the files setting on the transfer channel to 5,10 and 15 respectively.

My script *should* have continued with larger steps, but some of us forgot to renew the voms-proxy...

Fiddly old VO configurations

Got an email from a Zeus VO member who couldn't use the SE. Poking around revealed that we hadn't updated the VO membership list to be obtained from VOMS, instead of the old LDAP system.

As YAIM is now in a mess, I had to modify /opt/edg/etc/edg-mkgridmap.conf and /opt/lcg/etc/lcgdm-mkgridmap.conf by hand and then rebuild the mapfiles.

The should fix things, but I so, so wish we had a one-stop-shop for configuring VOs. It's too easy to leave VOs half cocked on the system, and often it's months before anyone notices or complains.

Of DPM, MySQL and MonAMI... (part 2)

Paul put MonAMI back onto the DPM yesterday. We saw a very similar rise in the number of MySQL connections as before, but as we were on the ball with this we were able to look at who was connecting via SHOW PROCESSLIST. Turns out that all the extra connections were from DPM itself. MonAMI was not to blame.

Early this morning the number of connections came back down again, which might indicate that under certain circumstances, DPM starts an extra connection to the database which it then does not let go of for some time (the 24 hour slot for any SRM transaction to complete?). I wonder if this might be the cause of the rare putDone failures we saw.

Thanks to MonAMI we'll be able to watch for this, and correlate any failures with how busy MySQL was.

Paul also did some pretty RRD aggregate plots, which are very much easier to read. Thanks! Note how MonAMI is able to distinguish between atlas and atlas/Role=production, which is incredibly useful.

My name is total (well, maybe...)

I heard from Hannah today that the problems using the LFC at RAL for totalep were down to DNS problams (globus being very picky about hostnames). She's managed to overcome them by hard coding the LFC's hostname into /etc/hosts, or by using a different machine on a subnet with better DNS. Hopefully we'll see some jobs soon.

Spaeking of jobs from minor VOs, I noticed that camont sent quite a few jobs in the last week. In total they've done nearly 500 jobs and consumed about 450 CPU hours (812 kSI2k hours).

Users: The Good, The Bad and The Ugly...

Last Thursday I went to see a few potential ScotGrid users. There are a couple of users in Civil Engineering who'd like to use the cluster. One of them has monte carlo code and this should be no problem (the good!) Another has a real need for MPI code (the bad!), which is challenging, but we obviously hope to build on the good work done here in GridIreland (see their wiki).

Finally, I spoke to a postgrad student in IBLS (Institute of Biomedical and Life Sciences) who has a lot of code to run on protein data (don't really understand the problem, actually). Now, she's got to run code she hasn't written herself, and neither does she have the source code and the damn thing asks questions in an interactive mode before it runs. (The ugly!)

On the computing service cluster she has to run the code in PBS interactive mode, in order to answer the questions. Then she has to hope that the network stays up between her machine and the cluster, because if it goes down (and often it does) it kills her job.

The first thing I did was show her how to use screen, so that at least she can run in a detached terminal. (I found a good tutorial.)

Last night, I started to write a python wrapper for the program which will take a defined set of default options, which can then be overridden on the command line, and pass them to the program. This turned out to be quite troublesome - using the python Popen3 class (pipe open) I just could not get the read() or readline() methods to behave properly with the select() call (even in nonblocking mode). select() would return a filehandle with data only once; and neither did the write() call seem to push the output properly into the program. However, when I switched to using the os.read() and os.write() calls, instead of the class methods, things started to work exactly as I expected. Hopefully this will run the program properly in a non-interactive mode and open up the possibility of running on our cluster.

Of course, there's a different track to this as well - we urgently need to get VOs setup for these users so that they can use edg-job-* commands or ganga to run their jobs. Watch this space...

Jabber Dabba Do!

OK, So in between taking the kids to an Intellectual day out (The new Barnstormer's ace BTW) I have registered a new gmail account for uki.scotgrid.glasgow and played with Net::Jabber on svr031. Took quite a few Perl dependancies to get it working, as you need the IO SSL working for when google switches to TLS.

The supplied nagios "notify_via_jabber" doesn't work out of the box, but I have a simple test script by Thus0 I lifted from the web that works fine - Now all I need to do is rewrite the notify script with the correct incantations from the test one.

Plan is then to have Jabber notifications for certain classes of nagios alert.

UPDATE: Success! added a new test on the disk servers as I knew there were some non-dpm boxes amongst them. Screenshot of gaimPidgin popup below

Of DPM, MySQL and MonAMI...

Paul has installed MonAMI onto out DPM, which has been very useful (and will become more so when we get nagios running again). However, we started to report zero storage over the weekend, which was tracked down to MySQL running out of connections (as DPM doesn't have a monitoring API we have to query the db directly, which is not ideal). When I looked in detail I (eventually) found that MonAMI had eaten all of the MySQL connections by swallowing sockets.

Paul is investigating and seems to have found at least one place where connections could leak (although he's unclear why it was triggered).

However, even stopping MonAMI at 11pm last night didn't entirely resolve the situation. At some point in the early hours MySQL seemed to again run out of connections. This caused some of the DPM threads to go mad and write as fast as they could to the disk. By 6am there was a 2.5GB DPM log file and / was full. Yikes.

This morning I had to stop all of DPM and MySQL, move the giant logfile out of the way, and then do a restart.

Paul will try the fix soon, but this time keep a much closer eye on things.

I believe we should also make sure that /var/log on the servers is a separate large partition in the future. Although we have enough space in / during normal running, clearly an abnormal situation can fill things up pretty quickly - and running out of space on the root file system is not desirable!

iperf on Fibre

We now want to 1) Retest bandwidth to check nothing's broken

So, one quick repeat of the iperf tests that greig and I did earlier and it all looks similar. Still got the asymmetry though. Need to find a chunky machine elsewhere and test.

March of the fibre...

The new ScotGrid cluster was connected directly by fibre from the Nortel switch stack down to the campus WAN router this morning. This cuts out 10m of Cu and a hop through an old baystack router. David reports there was ~2 minutes of network outage, which hopefully no one noticed.

This should open the way to getting a dedicated 1Gb link right to SJ5.

We now want to

1. Retest bandwidth to check nothing's broken
2. Move the gridmon box into the new cluster area
   

Right, come on LHC, where's that data...

A Morning With Ganga

I (finally) downloaded and installed ganga onto one of the cluster UIs. It's a cinch to install, just run one python script, add an element to your PATH and you're away.

The next step is going through the User Guide. Although this was written for 4.2, there's nothing I found which didn't basically work in 4.3.1.

It's quite easy to setup and run simple jobs:

In [2]:j=Job(application=Executable(exe='/bin/echo',args=['Hello, World']))

In [3]:j.submit()
Ganga.GPIDev.Lib.Job               : INFO     submitting job 0
Ganga.GPIDev.Adapters              : INFO     submitting job 0 to Local backend
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "submitted"
Out[3]: 1

In [4]:outfile=file(j.outputdir+'stdout')
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "running"
Ganga.GPIDev.Lib.Job               : INFO     job 0 status changed to "completed"

In [5]:print outfile.read()
Hello, World

However, that's spawning a job which just runs on the local machine - how easy was it to run on the grid?

Answer: very easy!

In [39]:l1=Job(backend=LCG())
In [42]:l1.application=Executable(exe='/bin/echo',args=['Hello, World'])

In [44]:l1.submit()
Ganga.GPIDev.Lib.Job               : INFO     submitting job 3
Ganga.GPIDev.Adapters              : INFO     submitting job 3 to LCG backend
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "submitted"
Out[44]: 1

In [45]:l1.status
Out[45]: submitted

In [46]:l1.backend 
Out[46]: LCG (
 status = 'Ready' ,
 reason = 'unavailable' ,
 iocache = '' ,
 CE = None ,
 middleware = 'EDG' ,
 actualCE = 'ce.ulakbim.gov.tr:2119/jobmanager-lcgpbs-dteam' ,
 id = 'https://svr023.gla.scotgrid.ac.uk:9000/-wOjj5xKfjcrKIXiCEISNA' ,
 jobtype = 'Normal' ,
 exitcode = None ,
 requirements = LCGRequirements (
    other = [] ,
    nodenumber = 1 ,
    memory = None ,
    software = [] ,
    ipconnectivity = 0 ,
    cputime = None ,
    walltime = None
    )
 )
In [52]:
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "completing"
Ganga.GPIDev.Lib.Job               : INFO     job 3 status changed to "completed"

In [54]:print file(l1.outputdir+'stdout').read()
Hello, World

Wonderful! Didn't have to do any of that nasty edg-job-* stuff. And it ran in Turkey - pretty cool.

I have also now discovered how to define and submit batches of jobs to the grid. This snippet defines a set of 10 jobs:

a=list()
for i in range(10):
    a.append(Executable(exe='/bin/echo', args=[str(i)]))
s=ExeSplitter(apps=a)
j=Job(splitter=s,backend=LCG())
j.submit()

Submitted that and I'm running jobs in China, Italy, Greece, Pakistan, Russia, Austria, France, Spain and Switzerland.

I think this is the first time in a while I've thought "Hey! The grid is actually cool."

Urgent Updates Urges

The latest gLite update (r24) is labeled as urgent. Turns out that this is only because the VOMS certificate for lcg-voms.cern.ch is going to expire at the end of the month.

Forunately, we have the lcg-vomscerts RPM directly controlled by cfengine, so it was a simple matter to update this part of the system.

Of course, while we're languishing at r20 the world is marching on, so we'll have to play catch-up sometime. Currently I've sheduled next Thursday as our site upgrade day. I'm glad we have the cluster independent of any other authentication system - it means finding all of those new UIDs for sgm and prd accounts will not be a problem.

(I'm also glad we didn't break our DPM with the current fiasco over gridmap file paths!)

Local Accounting Pages Ready

Billy's been doing a grand job knocking the local accounting pages into shape. This is based on Jamie's original work, but with some of the nastier hacks taken out and a lot of MySQL/PHP performance improvements from Andrew.

We can now see job numbers, CPU times, wall times and efficiencies for each group, plotted on a day/week or month basis.

There's still some work to be done - it would be nice to have a per-user plot, but the core is there and working well.

Oh, and it's checked into subversion finally. No more panics about losing the code.

It's probably in a good enough shape that other sites would find it useful now, actually.

Biomed get busy

The cluster got busy again from yesterday morning, with a whole pile of biomed jobs coming in. It was nice to see the resources being used.

Strangely, users are a bit like buses - you wait ages and then two come at once, because one of our local theorists then submitted several 100 jobs last night too, so we had a job queue for the first time in ages.

Looks like everything ran successfully as well (and Steve's ATLAS jobs still got through) - I always worry that something subtle has broken which will only be revealed when the site gets busy.

I redid the fairshares on the fly though, because now that the theorists have decided to use the pheno VO, we have to reflect their 20% nominal fair-share in Maui (in fact everyone's getting 33% as CompSci and local Bio users are not yet active).

The biomed jobs are still coming in steadily - 2 or 3 a minute. Lovely jobs, actually. Run time is ~4 1/2 hours, so a good turn over rate - and they are 99.99% efficient!

ScotGrid Review Documents Complete

The site responses for the ScotGrid T2 review have now been given to the reviewers. Inspired by Olivier I decided that some plots of CPU delivery per VO and per site would be useful.

This turned out to be surprisingly hard to do - the accounting portal only gives a summary for a time period, not a plot over the time period. So I had to download the last 12 months as individual CSV files and parse them. Of course, each file contains variable numbers of VOs and sites. As this is essentially data in 3 dimensions, i.e., cpuhours(month, vo, site) it's impossible for Excel to deal with it directly.

Time to bring python out of the box to parse the data and print summary CSV files which Excel can do. Took the best part of 3 hours - however, it's now done and any future work like this should be faster.

Plots shown above, just so they get a wider audience.

Glasgow disk benchmarks

I repeated the Edinburgh benchmarks testing at Glasgow. Results can be found here:

http://www.ph.ed.ac.uk/~gcowan1/glasgow-disk-benchmarks-07-05.ps

The behaviour for random reads and writes is very similar to that observed for the Edinburgh disk, although the absolute rates at Glasgow are higher. The sequential behaviour is somewhat different. A single thread gives a substantial perforamce boost for both reads and writes. Filesytem was ext2 with default options. dump2fs reported the large_file feature.

Edinburgh and ATLAS FCR

Edinburgh got bumped by ATLAS for failing JS for quite a while, but as their 5 jobs slots were tied up with Zeus jobs the ATLAS SAM tests couldn't run so they couldn't get back in the ATLAS BDII. Of course this meant that Steve Lloyd's jobs were aborting (no matching resources).

So, Sam has allowed the atlassgm user access to the reserved ops/dteam job slot. This will be fine as long as nothing happens to stick that job in the queue.

However, it's all getting a bit silly - more and more tweaks in the batch system to rush through the test jobs.

Of course, to some extent it's an Edinburgh problem - they are so strapped for CPU that getting anything to run through in a guaranteed time is hard.

I <heart> mod_include

Ah, the joys of Apache. More modules than you can shake a stick at. Trying to debug a really annoying masonic error message of "[error] Re-negotiation handshake failed: Not accepted by client!?" (SSL not working fully) I decided to set up a few "echo var" statememts in the page to see what was defined.

Lo one quick

<h1>Danger, Will Robinson! <!--#if expr="$SSL_CLIENT_S_DN_CN" --><!--#echo var="SSL_CLIENT_S_DN_CN" --> seen approaching!<!--#else -->This is <b>svr031</b><!--#endif --></h1>

and you end up with either

Danger, Will Robinson! This is svr031
or
Danger, Will Robinson! andrew elwell seen approaching!
:-)

Edinburgh disk benchmarks

I ran some Edinburgh disk benchmarks over the past couple of days in order to obtain a comparison of the performance of the IBM disk to the university SAN. It's not an entirely fair comparison as the IBM disk uses fibre channel to talk to pool1 (beefy 16GB RAM server) while the SAN uses FC to talk to pool2 (2GB of RAM). Both systems as configured to use RAID5, but they use a different number of disk in each array. You can find the results here:

http://www.ph.ed.ac.uk/~gcowan1/edinburgh-disk-benchmark-07-05.ps

I used tiobench (threaded IO) to perform the testing. It is quite clear that there is a limit of ~100MB/s on sequential operations with the IBM disk, while for the SAN it appears to be ~50MB/s. I need to check if this is a limit with the FC connection, I had thought we should see better performance. As expected the rate for random IO increases as the block size increases. There does not appear to be any significant difference between the two sets of disks when looking at the random metrics.

I'll post some results from Glasgow soon.

I love my BDII ;-)

Glasgow survived the RAL BDII outage of last night, as our trusty svr019, which is our top level BDII, was not affected.

You can see all the red splurged across Steve's analysis test with only a few sites not affected.

Of course, the BDII is always a single point of failure - and one day our will go wrong, but at the moment, for Glasgow, it's certainly the right choice.

Mark has switched Durham to use bdii.scotgrid.ac.uk and survived the mid-morning wobble.

Until we know for sure that RAL have sorted this I have encouraged Edinburgh to switch as well.

Scalpel, Mr Elwell please...

We started the re-install of svr031 this morning shortly before 11. So far the patient is doing well, coming around after surgery before 1pm.

cfengine is running again. ganglia is installed and Andrew's hacking in the ram disk for the rrds.

So far, so good...

Total Checks Out

The problem with VOMS was that the DN of the gridpp VOMS server was wrong in the setup.

Once I realised that, it all started to work.

Results were that it all works! lcg-cr and lcg-del function perfectly, so it seems the problem is with Hannah's UI.

Graeme Becomes an Oil Man

To help Hannah at Total with LFC issues I have joined the totalep VO.

Unfortunately things then fell over at the first hurdle - voms-proxy-init fails for this VO:

ppepc62:~$ vpi -voms totalep
Enter GRID pass phrase:
Your identity: /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
Creating temporary proxy .................................... Done
Contacting voms.gridpp.ac.uk:15026 [/C=UK/O=eScience/OU=Manchester/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk] "totalep" Failed

Error: Could not establish authenticated connection with the server.
GSS Major Status: Unexpected Gatekeeper or Service Name
GSS Minor Status Error Chain:

an unknown error occurred

None of the contacted servers for totalep were capable of returning a valid AC for the user.

I have put a ticket in (Footprints ISSUE=1207 PROJ=14), but no news yet.

HEPiX Summary

This week I 'ave mostly been attending Spring HEPiX 2007

I've summarised below some of the things I picked up on from the talks - Both slides and the video streams are available in the DESY Indico system.

Day 1 - Site Reports

LAL

WN's running SL4.4 64 bit with the ability to "fix" a VO to SL3 nodes

Evaluating LUSTRE

Lemon/Quattor for monitoring

Sun Thumper good.

LAPP (French T3)

Use GPFS/Woodcrest Blades (HP)

Cacti / Nagios / Ganglia

Developing own accounting tools for parsing Torque / Maui logs

PSI.ch

FreeNX desktops

SL5 Live CD

v12n - Use VMware / Eval Xen.

Use Lustre (on Cray) and GPFS (Linux clusters)

CASPUR

Mix of systems - IBM P5 / 2*Opteron clusters (Infiniband / Qsnet) / NEC SX6 / HP EV7.

GPFS / AFS / NFS

Xen / VMware

BNL

2*8500 robots using HPSS

Linux farm is big (>4700 CPUs, 1.3PB Local disk)

Nagios / Ganglia / Cacti

Temp Monitors distributed in datacentre - alert to RT Tickets.

Use both RT (helpdesk) and AT (assets) - also linked to OSG Footprints

GSI

OTRS Ticketing

Debian

1U boxes with 16 cores/32G RAM / 4 SATA Hot Swap disks for 8K Euro

RAL

2nd 8500 silo, perhaps with tape-passing interface

CASTOR problems

Disk problems fixed

Network upgraded

GridKa

Procurement via benchmark

How to connect 10GE to storage.

PDSF

HPSS

New Building

Blades

ScotGrid

Hey, We're great.

TRIUMF

Using BGP within the ATLAS network!

In-Row cooling

Dirvish / Amanda backups

Funky Videoconference room

INFN-T1

GPFS

RedEye (DIY) Monitoring System

Disk0Tape1 is CASTOR-2, D1T0 - GPFS / StoRM, D1T1 probably CASTOR-2

Nagios / MRTG / ntop

IN2P3

8500 robot

HPSS

CERN

Magnet issues

Firewall changes (no UDP)

Procurement via benchmark

Clampdown on Skype / p2p

Silent Data corruption

SLC3 support. SLC5 tests.

DESY

New Machine room

Nagios

8500 robot

Request Tracker / Zope

SLAC

Sun BlackBox, Thumpers, 8500

Phew. I'll write up the other days ASAP.

Durham Enable Fairshares

Mark, Phil and Nigel have enabled maui fairshares on the Durham cluster.

This is excellent news and should help them get better numbers on, e.g., Steve Lloyd's ATLAS tests.

I'm not quite sure how fair shares and pre-emption interact - I assume that they're somewhat independent.

Grieg re-testing DPM 1.6.3 rfio

Thanks to Greig for retesting our DPM 1.6.3's rfio.

The good news is that the code changes from 1.5.10 drastically reduce the number of rfio_open() failures. These now only crop up at about 70 clients (all within 1 second). The average open time is also much more stable, never rising above 2.5s.

However, we now see a problem with io wait on the disk server, and the total LAN rate struggles to reach 200MB/s, where as 1.5.10 easily reached 400Mb/s.

This remains to be investigated, but the only change on the disk servers is the update of DPM version, so I think it must be a DPM issue.

Biomed and VOMS

We got a ticket from Biomed about their jobs not running properly.

I had a suspicion that this was the case, and when I went to the yaim tool site and found that not only had they changed their VOMS certificate, which I had updated, but they had also changed their VOMS server's DN.

So, I changed their VOMS DN in our site-info.def and re-ran the config_mkgridmap function on the CE, which also configures LCMAPS. After that things seemed fine.

One to watch out for, though, when updated VOMS certificates come through.

Intervention At Edinburgh

I had to intervene at Edinburgh 2 weeks ago (14th, just before I went to London for the T2 review). They had been failing JS since the Friday night. Logging on I could see a stack of ops jobs, but nothing running on several WNs.

I tried starting the oldest ops job using runjob -cx, but that didn't work, giving the error:

04/14/2007 16:54:30;0080;PBS_Server;Req;req_reject;Reject reply code=15057(Cannot execute at specified host because of checkpoint or stagein files), aux=0, type=RunJob, from root@ce.epcc.ed.ac.uk

Not at all clear to me what was going on. I tried running different ops jobs and they all started and ran properly, so in the end I deleted that job from the queue and that seemed to ungunge things.

torque seems to produce rather unhelpful information in these sort of cases, unless I'm just looking in the wrong places.

iperf redux

Last Month Greig and I performed a quick round of iperf benchmarks between Glasgow and Edinburgh (see plot) and got around 520/570 Mb/s. However due to some network tweakage, we've just rerun the plot and got the cracking results below:

Glasgow Worker Nodes Filled Up

Browsing the Glasgow gangila plots on Saturday night I noticed a very weird situation, where the load was going over the number of job slots and an increasing amount of CPU was being consumed by the system.

It took a while to work out what was going on, but I eventually tracked it down to /tmp on certain worker nodes getting full - there was an out of control athena.log file in one ATLAS user's jobs which was reaching >50GB. Once /tmp was full it crippled the worker node and other jobs could not start properly - atlasprd jobs untarring into /tmp stalled and the system CPU went through the roof.

It was a serious problem to recover from this - it required the offending user's jobs to be canceled, and a script to be written which cleared out the /tmp space. After that the stalled jobs also had to be qdeled, because they could not recover.

This did work - the load comes back down under the red line and then fills back up as working jobs come in as can be seen from the ganglia plots.

This clearout was done between 2230 and 2400 in a Saturday night, which royally p***ed me off - but I knew that if I left it until Monday the whole site would be crippled.

I raised a GGUS ticket against the offending user. Naturally there wasn't a response until Monday, however it did prove that it is possible to contact a VO user through GGUS.

Lessons to learn: we clearly need to monitor disk on the worker nodes, both /home and /tmp. The natural route to do this is is through MonAMI, with trends monitored in ganglia and alarms in nagios. Of course, we need to get nagios working again on svr031 - the president's brain will be re-inserted next week! In addition, perhaps we want at least a group quota on /tmp, so that VOs can kill themselves but not other users.

Blogger Catchup

Networking problems and the LT2 reviews have meant that it's been really hard to blog in the last week. Now trying to catchup, so hang on to your hats...

Database housekeeping

I had to perform some housekeeping on the dCache postgreSQL database today. Auto-vacuuming was not enabled by default and postgres was starting to consume lots of CPU cycles. More information over on the storage blog:

http://gridpp-storage.blogspot.com/2007/04/postgresql-housekeeping.html

Edinburgh storage woes

Edinburgh has been suffering lately from a number of failures in the SAM replica management tests. Looking at the dCache webpages reveals the reason:

http://srm.epcc.ed.ac.uk:2288/usageInfo

Rather than displaying the usual pool usage information, the table contained entries like this:

pool1_04 pool1Domain [99] Repository got lost

The "Repository got lost" error can be explained as follows. The dCache periodically runs a background process which attempts to write a small test file onto each pool in an attempt to check that it is still operational. If this fails then the above error message will be generated. According to the dCache developers this will occur if there is a filesystem problem or a disk is not responding quickly enough.

What is strange, however, is that the recent problems have resulted in all of our dCache pools as being marked with the above error. It seems strange that the same filesystem or disk issue would simultaneously affect all of the pools. I have submitted a ticket to dCache support in an attempt to get more information.

The problem can eaily be fixed by restarting the pool process on the affected dCache pool node. In the above case it is pool1.epcc.ed.ac.uk since the pool in question is named pool1_04.

service dcache-pool restart

I have started to add material to this page

http://www.gridpp.ac.uk/wiki/Edinburgh_dCache_troubleshooting

to desribe our dCache setup in more detail. This should give people a better idea of how a working system should be configured. I will add more information when I have time.

Server Spread

Scotgrid-Glasgow continues to grow. Not only do we have the vobox for a user group, but we now have a brace of test Scientific Linux 5 servers kickstarted and ready to go. Changes that we needed to make to our existing setup were minimal, the main one being the need to add noipv6 to the PXE command line, otherwise the installer hunts (and fails) for an IPv6 address by DHCP.

I have also added a "sl5" flavour to our local repository for Misc Stuff RPMs (such as cfengine) in both i386 and x86_64. One niggle is that you have to run createrepo once in each location if you use the $basearch variable in the repo config source.

Next step - install apache onto one of the boxes, tweak config and see if it has all the correct mirroring arrangements and replace the brain dead svr031 versiom.

APEL Configuration Twiddle

Reviewing some of the ScotGrid status pages I noticed we hadn't published accounting data for about a week. Trying to run APEL by hand revealed why - it was set to use the old sBDII on svr016 instead of the new one on svr021. This had been running on for months, even though the site's published GIIS endpoint had been changed to svr021 months ago - and I had finally switched it off about a week ago.

Once this was corrected (in /opt/glite/etc/glite-apel-pbs/parser-config-yaim.xml) things ran through fine.

Defining SPEC Values for the Cluster

I had a long discussion with Mark about getting the SPEC values correct for Durham. There's no really good answer to this apart from go to the SPEC Website and try and find machines with the same processor types and vintage as your own (ideally with the same motherboard). N.B. One should really use the "base" values - these have a conservative set of compiler flags so are more appropriate for pre-compiled EGEE applications - the peak values enable all the bells and whistles on the compiler.

I was also prompted to look at the numbers I had put in for the new Glasgow cluster. Here we have Opteron 280s. There are now 10 measurements for the SI2K of these machines - these are all very close and average to 1533, so that's what I have now put (up slightly from 1450). The FP2K values have a bigger spread (different chipsets?), but in the absence of any guide I again took the average, which was 1770.

I also noticed that CPU2000 has now officially been retired - replaced by CPU2006. This is going to be a problem as CPU2000 will not be available for newer machines, but CPU2006 will not be available for older ones. How do you express that in your JDL?

DPM GRIS goes nuts at Glasgow

I was amazed to find that Glasgow was in a gstat warn condition this evening, because we were reporting 0GB storage available.

When I checked on the SE the plugin was able to run fine, but it didn't seem to be being run properly - so everything was reporting zeros.

I tried restarting the GRIS, but this didn't work (it had been running since last year), so I had to kill off the process and then restart it. Finally, for good measure I restarted rgma-gin, which is one of the plugins providing the dynamic information.

Still, it took a few minutes before the GRIS started to properly provide the correct information.

However, from 6pm until 10pm we apparently had no storage.

Any thoughts on how to monitor the information system? Well, we did go into "warn" status on the SAM sBDII seavail test, so we should ensure that gets trapped by Paul's MonAMI SAM feeds.

Disabling SAME/R-GMA in the Job Wrapper

Alessandra raised a GGUS ticket about the R-GMA client issues in the job wrapper. She got a rapid response, along with a recipe to disable them:

Unfortunately this is a known problem with SAM CE JobWrapper Tests and R-GMA. We are using R-GMA command line utility to publish a small piece of data from worker nodes but unfortunately sometimes R-GMA hangs for quite a lot of time.

We are planning a new release of JobWrapper tests without R-GMA publishing (replaced completely by our internal SAM/GridView transport mechanism). But for the time being the only solution is to disable JobWrapper tests on your site if you observe such a behaviour.

To do this you have to remove all the symlinks that appear in the following two directories on all WNs:

$LCG_LOCATION/etc/jobwrapper-start.d
$LCG_LOCATION/etc/jobwrapper-end.d

I have now put in the necessary cfengine stanza to delete the links and stop this nonsense:

disable:
        worker::
                # Disable the SAM wrapper which uses R-GMA
                /opt/lcg/etc/jobwrapper-start.d/01-same.start
                /opt/lcg/etc/jobwrapper-end.d/01-same.end

Now, how big a difference does it make? Quite a lot for short jobs - the wallclock time for a simple globus-job-run has gone down from 5 minutes to 3 seconds!

The total time for the jobmanager to handle the job has remained quite high - 1m30s c.f. 16s for the pbs jobmanager. However, at least no one is going to be "charged" for the time that the job is with the gatekeeper, unlike the time spent in the batch queue.

Last VO Enabling Twiddle

One last thing, making a VO Tag Directory.

Now it looks like it's all working fine.

GRIS Wobbles in Glasgow CE

Between 12 and 1pm yesterday the site BDII stopped reporting on the queue statuses. We suffered the classic problem of reporting 4444 queued jobs and 0 job slots available.

You can see from the plots from the sBDII that the amount of network traffic clearly dips. In fact the gstat graphs show clear a dip in the number of entries. So it was the CE's GRIS was misbehaving at this point.

I checked the CE for load, job floods, etc. There was nothing abnormal - we got 20 jobs in 1 minute, but we should be able to cope with that ok. There are no logs for the running slap daemon and nothing odd spotted in /var/log/messages.

We also aborted on a few of Steve Lloyd's tests. However, very weirdly the RB logs are showing an attempt to match a queue on host gla.scotgrid.ac.uk. Where did the name of the CE go? Is this related to the CE GRIS getting in a pickle?

So, one of these anomalous blips in the crappy information system again.