Only one bite at the cherry...

I have modified the default RetryCount on our UIs to now set zero retries. Automatic retries were actually working quite well for us when we were losing a lot of nodes to MCE errors (in the days before the upgrade to SL4, x86_64) - users' jobs would automatically rerun if they got lost and there was no need for them to worry about failures. However, recently we see users submitting more problematic jobs to the cluster - some which fail to start at all, some which run off into wallclock limits, others which stall half way through. Often we have to gut the batch system with our special spoon and in this case having to do it four times because the RB/WMS keeps resubmitting the job is less then helpful.

For once cfengine's editfiles stanza was useful and a simple:

ui::
{ /opt/glite/etc/glite_wmsui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}
{ /opt/egd/etc/edg_wl_ui_cmd_var.conf
ReplaceFirst "RetryCount\s+=\s+[1-9];" With "RetryCount = 0;"
}

got the job done.

Resource Broker Blues

Our resource broker was down for the weekend as the network service stalled. Root cause turned out to be a bit of over aggressive cleaning from cfengine. I had wanted to do a better job of cleaning up the /tmp area in the cluster - each worker node had hundreds of condor_g working directories lying around - with nothing in them. cfengine's "tidy" leaves directories alone by default and only cleans files. So I enabled the "rmdirs=sub" option - works beautifully, gets rid of all the cruft in /tmp. So pleased was I that I disengaged by brain and set this option on for /home as well - good idea to clean up those old gass cache areas, isn't it? Well, almost - unfortunately /home has subdirs which are the node pool account home areas and unused pool accounts fall into the clean me up category. All the untouched pool areas then vanished.

This caused a number of people to start getting "unspecified grid manager errors" on globus-job-runs, as well as wiping out the edguser home area on the RB which caused the network server to go into crisis.

It didn't take long to work out what had happened, but fixing it took a while as the resource broker seemed to be quite huffy afterwards.

The only plus side was that I enabled the mice, scotgrid and nanocmos VOs on the RB.

RB Corrupts its Database

Our RB (svr023) seriously died today.

I was alerted to a problem by a pheno user who was having trouble submitting jobs. When I checked the RB I found that the root partition was full, choked by massive /var/log/wtmp files and by an extremely large RB database in /var/lib/mysql.

When had reinstalled the RB 2 weeks ago I had ensured that /var/edgwl was in a large disk area, but to have more than 4GB filled up in less than 2 weeks by the other denizens of /var was completely unexpected.

Emergency procedure was then to move /var/log and /var/lib/mysql over to the /disk partition, creating soft links pointing from the old locations.

This seemed to be going ok, but job submission was still failing. When I checked the error log for mysql I got the message:

071008 16:25:32 [ERROR] /usr/sbin/mysqld: Can't open file: 'short_fields.MYI' (errno: 145)

The short_fields table definitely existed. I even checked it against the last version in /var. Logging in to mysql demonstrated that this table had become corrupted.

I toyed with the idea of trying to save the database, however it would probably have left us with an internally inconsistent RB database, as well as orphaned files in the /var/edgwl bookkeeing area.

Reluctantly I decided that the only sensible recourse was to reinstall svr023, and take the hit of the lost jobs.

This has now been done, and normal service has been resumed.

In the course of the re-install the /var partition has been grown to more than 100GB, which should protect us against large log files for quite a time.

Frankly, all very annoying and I'm quite upset that we lost users' jobs.

Sorry folks. It shouldn't happen again.

RB: "Rather Better"

One thing which went to pot during the upgrade, was the way that the higher UIDed pool accounts cascaded through to the RB. This, unfortunately, meant that any jobs which were running on svr023 were lost (there would have been very few, in fact, which is why we spent more efforts on the UI).

However, in our attempts to get the RB back it became clear that the edg-wl-ftpd service (yet another hacked version of GT2 gridftp) cannot handle UIDs > 16bit. This screwed things up for us, as all our new UIDs are in the range 200000+.

In the end I had to re-hack the perl passwd/group/shadow/users.conf generator, lowering all the UIDs specially for the RB. In fact this was not quite as awful as one might think, as the RB supports only a subset of the VOs we run jobs for on the main site. I also scripted up a "generator" for the RB's site-info.def, that strips down the VOS variable to those we support for job submission. In addition, communication between RB and the UI or the CE is of course mediated by certificate, so having a different pooll account or UID on the RB is not a problem.

There was a supporting tweak to cfengine to take passwd-rb (etc.) as the source passwd file for the RB.

Then the RB was blown away and rebuilt. It seems to have done it rather a lot of good, as now Steve Lloyd's dteam test jobs run properly (see his RB test page).

Multiple VO Woes

Steve Lloyd and I sat down after lunch today to try and get to the bottom of why his dteam submitted jobs always fail. Strangely this seems to be a RB specific problem. IC always works, Glasgow always fails and RAL seems to come and go.

Using the Glasgow RB we submitted a job to Edinburgh, so that we could trace things through the batch system. The job arrived at Edinburgh, and ran through the batch system. However, it continued to be considered by the RB as

Current Status: Scheduled
Status Reason: Job successfully submitted to Globus

Clearly this was not the case.

We had a good look through the logs on the RB, but there's no particular sign of things going wrong there - although it must be said that the logs are both dense and impenetrable.

When it became clear that there was no easy solution I decided to try and reproduce the problem myself. Now, recall I had joined gridpp a while ago to help our local users and never had any trouble. However, now I can't seem to get a single job running through as a gridpp member - even on the Glasgow cluster. And things are in fact even worse than for Steve, because my gatekeeper process dies almost instantly, so the job never even goes into the batch system:

grep 2007-07-10.14:49:10.0000028268.0000113028 /var/log/messages
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 for /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart on 130.209.239.23
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 mapped to gridpp001 (17601, 10016)
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 has GRAM_SCRIPT_JOB_ID 1184075356:lcgpbs:internal_434559272:9672.1184075355 manager type lcgpbs
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 JM exiting

I'll now try and poke around inside the gatekeeper logs and see if I can come up with any indication why things are going wrong.

And what the hell's this got to do with the RB anyway? It's deeply puzzling and frustrating in equal measure.

RB Twiddles

Steve Lloyd's started to monitor our RB. So far things look good! I also discovered that SAM had picked up the RB and was at least checking its host certificate. I have now added the RB to the GOC and requested monitoring - another pair of eyes to pick up problems.

As we are getting rather more officially known, I've reduced the number of advertised VOs on the Glasgow RB to ops, dteam (for testing), atlas, pheno and gridpp. There's a slightly different site-info.def used for the RB to do this (site-info-rb.def) and I changed cfengine to run against this file if the node needs rebuilt.

Even though the RB has only been up for 3 months the MySQL database is now 270MB. Although this is well below the 4GB where Olivier has reported problems, we will have to monitor this to ensure we don't get into stability problems.

At the moment, though, the machine is very lightly loaded.

Glasgow RB now in BDII

After humming and hawing about our "trial" RB, I decided to bite the bullet and publish it into the site, and thence top level, BDII. This makes things easier for the RTM, among other things. So far the RB has been really trouble free to run and had been lightly loaded - fingers crossed.

I think our "policy" here should be to support VOs in which we have a strong interest, e.g., ATLAS, gridpp, pheno, ScotGrid VOs. This will mean splitting our site-info.def file, though, because there will be a different list of supported VOs on the RB to the CE/SE.

Of course, the LCG-RB's days should be numbered - it seems that the gLite WMS now outperforms it. If this is true then there really is little point in trying to learn about it, instead we should put our efforts into the new system.

Resource Broker (Beta) for ScotGrid

Through the wonders of YAIM and cfengine, I was able to setup an lcg-RB on svr023 in two cfengine lines: download metapackage, run configure_node.

And it works! I got output back from my first job:

ppepc62:~/jobs$ edg-job-status https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ


*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        svr016.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-dteam
reached on:         Fri Mar 16 12:24:38 2007
*************************************************************

ppepc62:~/jobs$ edg-job-get-output https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ

Retrieving files from host: svr023.gla.scotgrid.ac.uk ( for https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ )

*********************************************************************************
                        JOB GET OUTPUT OUTCOME

 Output sandbox files for the job:
 - https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ
 have been successfully retrieved and stored in the directory:
 /tmp/jobOutput/graeme_5hT0x7GZluDFMWJ6qT0KLQ

*********************************************************************************

The jobs did, however, run really slowly as R-GMA managed to lock-up twice on me and need restarted. I'm really fed up with this, so I have started an Ops Logbook for the site to at least log these issues in a consistent way.

If anyone has a nagios/cfengine recipe for restarting R-GMA I'd be glad to use it.