And don't do it again...

I re-enabled Heinz today on the cluster, our infamous RSA cracking biomed user.

Tony finally spoke to Cal and this seemed clarify that

• VOs now know they have a more serious responsibility to discipline their users
• Heniz knows he cannot run this work again under biomed
  

He'd been suspended for six months which seems like an appropriate punishment.

Dem Biomed Blues...

Finally I got fed up with the biomed user whose jobs always stall on the cluster. Banned them and sent in a ticket.

I'm not prepared to be tolerant of crap code when we have 100s of queued jobs.

Biomed Abuse

Alerted by Dave Colling, we found a biomed user who was running about 100 jobs on the cluster all trying to factorise a 768 bit number (and win $50 000 in the process).

Clearly this is abuse of our resources and nothing to do with biomed. They have consumed more than 80 000 normalised CPU hours since September. I'm sure the operational costs of this amount to more than several thousand pounds (should we bill them?).

It was all the more irritating as we had a stack of ATLAS and local users' jobs to run, but the biomed jobs were set to download subjobs from the user's job queue (they were effectively limited pilots) and so they ran right up to the wallclock limit of 36 hours.

I banned the user, deleted their jobs and sent a very angry GGUS ticket.

As a slight aside one notes the efficiency of pilot job systems at hoovering up spare jobs slots and consuming resources on cluster well in excess of the nominal 1% fair share we give to biomed.

Biomed Stalled Jobs

Since we came back after the upgrade to SL4 I had noticed a very large number of stalled biomed jobs on the cluster.

These were all jobs which had stalled running python ./get_task.py autodock AVIANFLUDC2_T02IAN3J1170 (or something very like it).

As the cluster hadn't actually been full, and I was very busy, I actually let this situation go for most of the week. However, today I emailed the user (using the CIC portal user look up). I got a very quick response that there was a known problem with an overloaded AMGA server, which was causing these stalls. I was given permission to kill the jobs, which I did.

Although it's a good thing (tm) to get in touch with users, following our stalled jobs guide, it is time consuming and I wish there was some form of automation we could apply.

Bad, bad, biomed....

A very flaky day - we had a biomed user throw jobs into the system which were trashing workernodes by filling up /tmp. This caused lots of nodes to get into a weird state where they seemed to run our of memory (ssh and ntp nagios alarms firing). Jobs couldn't start properly on these nodes, so they became black holes: one of our local users lost 47 of 50 jobs, another lost 124 of 150.

We then started to fail SAM tests, drop out of the ATLAS BDII, Steve's tests then couldn't resource match us, and so on. Bad day.

It took quite a few hours to sort out the mess, and a further few hours to stabalise the site.

Our GGUS ticket names and shames.

It's a very different ball game on the grid - we have 6000+ users who can submit jobs, and it's not hard to kill a worker node. The torque generally handles this badly and all hell breaks loose.

Action plan:

1. Nagios alarms on WN disk space
2. Group quotas on job stratch area

Users and Stalled Jobs

I noted that I had qdeled quite a few jobs from the cluster two days ago. Well, the ILC user I contacted through the CIC portal responded, apologised and thanked me for the suggestion of adding a timeout to lcg-cp. However, the biomed user (110 jobs) didn't answer. And in fact they continued to submit jobs onto the cluster and I was forced to clear out another 42 jobs this afternoon. So, if nice doesn't work, try nasty - I have ticketed them and will ban them from the cluster if I don't hear back within 24 hours.

Biomed get busy

The cluster got busy again from yesterday morning, with a whole pile of biomed jobs coming in. It was nice to see the resources being used.

Strangely, users are a bit like buses - you wait ages and then two come at once, because one of our local theorists then submitted several 100 jobs last night too, so we had a job queue for the first time in ages.

Looks like everything ran successfully as well (and Steve's ATLAS jobs still got through) - I always worry that something subtle has broken which will only be revealed when the site gets busy.

I redid the fairshares on the fly though, because now that the theorists have decided to use the pheno VO, we have to reflect their 20% nominal fair-share in Maui (in fact everyone's getting 33% as CompSci and local Bio users are not yet active).

The biomed jobs are still coming in steadily - 2 or 3 a minute. Lovely jobs, actually. Run time is ~4 1/2 hours, so a good turn over rate - and they are 99.99% efficient!

Biomed and VOMS

We got a ticket from Biomed about their jobs not running properly.

I had a suspicion that this was the case, and when I went to the yaim tool site and found that not only had they changed their VOMS certificate, which I had updated, but they had also changed their VOMS server's DN.

So, I changed their VOMS DN in our site-info.def and re-ran the config_mkgridmap function on the CE, which also configures LCMAPS. After that things seemed fine.

One to watch out for, though, when updated VOMS certificates come through.

Contact with VOs

In my GridPP 18 talk I said we still felt we didn't have enough contact with VOs in general. A specific example of this is biomed. When I added the new VOs on Monday I found there was a problem with the biomed pool accounts, which seemed to have the wrong stub names (boimNNN instead of biomedNNN). As soon as I fixed this we got biomed jobs into the cluster.

So, we were a large site, with a biomed queue enabled, where things were broken from November until March - and we never got a ticket.

Of course, all the VOs are sorting out their operational procedures, and I'm sure that many other VOs would have been no different - but that's the point isn't it? Our contact with non-LHC VOs is poor right now.

I have finally fixed the problem where the BDII was refusing to publish the VO view for biomed on UKI-SCOTGRID-GLASGOW. This had been going on since the cluster was reconfigured. At this time the biomed torque queue was renamed biom (I'm sure I had a good reason for doing this, although I now cannot recall it...), which somehow was causing the dynamic scheduler wrapper to fail to publish for the biomed queue. I had manually hacked LDIF files, rerun YAIM, etc., etc., without success.

This afternoon I just renamed the biomed queue biomed, redefined the queue in site-info.def and reran YAIM's config_gip - finally, the information system started to do the right thing.

How annoying that I wasn't able to really understand the problem.