Stale CREAM and Maui partitioning

Nothing terribly exciting; but we've done a bit of an update on our Maui configuration, and CREAM problem has been cleared a-whey.

Previously, we've had our different era's of compute nodes annotated with the key speed notifier; so that the scheduler understands the fact that some are faster. For the vast majority of grid jobs, this is an utterly irrelevant distinction - so long as the job gets the time it expects (and Maui is scaling requests that go to the slower nodes so they get longer).

However jobs that use more than one process (i.e. MPI jobs) are a different case - if they get scheduled with different classes of nodes, then you get sub-optimal resource useage. So it's useful to keep some distinction between them. Previously we've been using reservations to restrict where jobs can go - but there's an (ill-defined) upper bound to the maximum number of overlapping reservations on a single job slot at once; too many breaks things.

So we had a look at Partitions in Maui, which is really the proper way to handle these. The downside is that you're limited to 3 partitions - there's a compiled in limit of 4, one of which is the special [ALL] partition, and one is DEFAULT. Fortunately, we have 3 era's of kit - and as long as we're happy calling one 'DEFAULT', it all works. And Maui understands never to schedule a job to more than one partition at a time.

So we ended up with a lot of lines like:

    NODECFG[node060] SPEED=0.94 PARTITION=cvold

But in order to make them used by all jobs, we had to adjust the default partitions available to include them all:

    SYSCFG PLIST=DEFAULT:cvold

which gives all users equal access to all the partitions.

Not terribly exciting Maui tweaks, but sometimes that's the way of it.


The CREAM problem manifested itself as a set of jobs that have gone sour. There's a known problem in the current versions of CREAM where if the job is queueing on PBS for over an hour, CREAM (rather, the BLAH parser) thinks it's dead, and kills the job, with reason=999.

What's not made explicit is that you need to have no spaces in that! i.e. you must have

    alldone_interval=7200

because if you put alldone_interval = 7200, then CREAM doesn't understand that. So fixed that, and it was all hunky dory for a while. Then we started getting lots of blocked jobs in Torque again; all from CREAM.

Cue more digging.

Eventually found this in the CREAM logs (after a slight reformatting):

JOB CREAM942835851 STATUS CHANGED: PENDING => ABORTED
[failureReason=BLAH error: no jobId in submission script's output
(stdout:) (stderr:/opt/glite/etc/blah.config: line 81: alldone_interval: command not found-
execute_cmd: 200 seconds timeout expired, killing child process. )

So, two things here. Firstly, alldone_interval with spaces had crept back in, along with the correct version - in our case via cfengine directives, probably down to the Double CREAM plan. More interesting was that having the invalid part of the BLAH config present slows down BLAH (BUpdaterPBS was pegging a CPU at 100%), sufficent that it hits another timeout at 200 seconds to respond at all. And then CREAM kills the job, but doesn't actually tell Torque, sand box is blown away, so it can't finish (nowhere to put output), or, if not started, can't start.

Removing the second (wrong) version of the alldone_interval fixed that - CPU use in the parser dropped to trival levels, and all appears to be happy again. This one's not really CREAMS fault, but it's always good to have an idea of what misconfigured services end up doing, otherwise it's hard to fix those 'not enough coffee' incidents. Hence, this one for Google...

UPDATE: Oops! Spoke too soon. That's the problem defined, but clearly not the solution - as it's happened again. Gonna leave this here as a reminder to self to give it a bit longer before considering something fixed...

Corralling jobs in Maui.

Sometimes, when testing new hardware or software in a limited way, it is important to be able to arrange lightweight, temporary partitions of a cluster for only a given user.

Now, you could repartition the cluster nodes between a "normal" partition and a "testing" partition, but for most pbs/maui clusters (which don't have anything but the 'ALL' partition set), this involves changing configuration for all the nodes, rather than simply the nodes we care about. (And then changing it back when you're finished.)

You might also consider doing this with reservations - indeed, the maui manual suggests that a reservation locked to a user specified with an & prefix will force precisely the behaviour we want - locking the reservation and the user together. This appears not to work under empirical testing.

Instead, the solution we've found to work is (all in maui.cfg):

1. Create a reservation for the user only.
   SRCFG[ssdnodes] PERIOD=INFINITY
SRCFG[ssdnodes] STARTTIME=00:00:00 ENDTIME=24:00:00
SRCFG[ssdnodes] HOSTLIST=node30[0-9]
SRCFG[ssdnodes] USERLIST=ssp001

2. Create a quality of service class with the property that it only runs on that reservation.
   QOSCFG[ssd] QFLAGS=USERESERVED:ssdnodes
3. Make the user a member of that quality of service class only.
   USERCFG[ssp001] QDEF=ssd QLIST=ssd

(In this case, the configuration mutually restricts the user ssp001 and the nodes node300 to node309 to each other.)

This has the benefit that it also generalises to any number of users, as long as you add them to the reservation and the QoS class.

more openmpi tweaking

Whilst testing MPI on our cluster and get it into a usable state I uncovered a rather nasty bug with openmpi-1.3.4. This manifested itself with never being able to run on the node with cores > 4. It was a weird one as openmpi communication over two nodes worked fine with 8 cores on each node but when a job requested cores > 4 on the same node. The job just hung. An strace of the mpiexec process suggested some sort of TIMEOUT/WAIT issue.

On the release note for openmpi-1.4.1 it appears they discovered this bug and provided a fix:
- Fix a shared memory "hang" problem that occurred on x86/x86_64
platforms when used with the GNU >=4.4.x compiler series.

This sounded plausible and in fact an upgrade has fixed the issue.

So now with all 8 cores running on the same node the next issue to arise was one related to Maui. Some time when you requested nodes=8, Maui scheduled the job on 3 cores, a qdel and a resubmission later Maui rescheduled the job onto 5 cores. On one test I even qrun'd the job and it appeared it start on the correct number of nodes but there appeared to be no reason for Maui not getting this correct. So it was time to get out Maui docs.

from the docs;
Maui is by default very liberal in its interpretation of :PPN=. In its standard configuration, Maui interprets this as 'give the job * tasks with AT LEAST tasks per node'. Set the JOBNODEMATCHPOLICY parameter to EXACTNODE to have Maui support PBS's default allocation behavior of nodes with exactly tasks per node.

This seemed to suggest that Maui's default behaviour is to pack a job into as few nodes as possible. So I tried out setting the JOBNODEMATCHPOLICY to EXACTNODE and this seems to have done the trick.

nodes=24 means 24 nodes, not 8, not 6 but 24

This does have a drawback in that it will be 24 separate nodes. This setting relies upon being able to set :ppn (processes per node) to allow nodes=3:ppn=8 giving 24 cores which is really what you want to say. As you probably have a fast machine with loads of memory and cores. Therefore, you could target all the cores rather than 24 nodes. However, it is a start.

Wouldn't it be nice if you could specify :ppn in JDL. The only way round this I can see for now is to manually change the job manager or use the local batch attributes of CREAM to allow a custom cerequirement to be specified. Possible but not nice.

Maui: MAXPROC vs MAXJOBS

One thing which was always desirable in the batch system was to guarantee a number of jobs slots for groups, irrespective of their fairshare usage. We actually want to encourage opportunistic usage of resources, and not punish people by then refusing to run any of their jobs for a week.

However, attempts to set a soft MAXPROC limit always seemed to come to grief. Maui would block jobs beyond the soft limit, even though, as far as I could see, it had been told not to. in frustration I had to set all groups with a soft limit at the cluster size.

Today, I had a chat with the great maui guru, Steve T, who was also somewhat puzzled by maui's behaviour. He pointed out that he'd only ever set MAXJOBS, not MAXPROC. Well, I thought I would give that a whirl and it works!

So, finally we can have a system which protects some slots for VOs and groups, but allows for full opportunistic use of the cluster for everyone.

Thanks Steve!

Maui Madness

Maui has been driving me mad for about 3 weeks now. When we upgraded the cluster I had forgotten that moving to pooled prd and sgm accounts would mean that these groups were independent of the normal VO fairshare. As our engineers started to become more active I was unable to get any ATLAS jobs to start at all - particularly atlasprd jobs. As I tried to add fairshare for the new groups maui started to lose the plot, just dropping groups entirely from its fairshare groups - you can see the effect very clearly from MonAMI's maui plots - groups just evaporate!

Fed up with this, tonight, I stopped maui, removed all its databases and restarted it. This, of course, means it's lost its current fairshare calculations, but at least it now has fairshares for the new groups.

I have also re-jigged the fairshare algorithm to have far less of a decay on it - users who ran 7 day jobs were at a huge advantage because by the time their job had finished its first day of running was weighted by 0.3, so it almost didn't count!

Some Queue Work

I introduced a new long queue last night, which is primarily to support long validation jobs run by some of our Durham phenomenology friends. It has a 7 day CPU/Wall limit. As I was messing about with transitions to queues which support more than 1 VO, I added support for ATLAS on the queue, thinking it might be of use to our local ATLAS users.

Unfortunately, as it was advertised in the information system, and was a perfectly valid and matchable queue, we soon got production and user ATLAS jobs coming in on this long queue. So, tonight, I have stopped advertising it for ATLAS, to push things back to the normal 36 hour limited grid queue.

I also discovered that maui does its "group" fairsharing on the submitting user's primary group. This is pretty obvious, of course, but as we've traditionally had one queue per VO, everyone in that VO in a primary group of the same name, somehow I had it muddled in my mind that it was a queue balancing act instead. Turns out this is in fact good, because we can set different fair shares for users sharing the gridpp queue, as long as we keep them in per-project primary groups.

Queue Tweaks and Maui

As we've been really full recently, I have reduced the maxwallclock available to grid VOs from 148 to 100 hours. The maxcpu time stays the same at 96 hours. I'm growing very frustrated with jobs which just stall at the start - we had 9 atlas jobs which consumed 1s of cpu time in 9 hours, hanging on an lcg-cp.

I also increased the maxcpu and wallclock on the gridpp queue to 168 hours, to make sure that Swetha's bio jobs run through ok - 96 hours was probably too close to the wire. We can cut our local users a bit more slack as their jobs, when they do run, tend to be almost 100% efficient.

I had to reduce the maxprocs on the glee queue to 400 - we can't really afford to get the whole cluster filled with EE jobs as their maxcpu/wallclock is so high at 28 days, and this will completely mess up fairsharing.

We've been suffering this weekend from two very large job surges from pheno and glee. As these groups have a large, but underused, fairshare, they get to start an awful lot of jobs in a short time - and as they run for a very long time then the cluster starts to suffer from very few jobs slots coming free and can't run anything for anyone else. 100hr/560 is 11 minutes, but the job inflow, when local users are involved, is far from uniform.

I would like to move the *sgm jobs from atlas and lhcb into the dteam/ops reserved job slot. I will have to ask Sam how to do this.

Biomed get busy

The cluster got busy again from yesterday morning, with a whole pile of biomed jobs coming in. It was nice to see the resources being used.

Strangely, users are a bit like buses - you wait ages and then two come at once, because one of our local theorists then submitted several 100 jobs last night too, so we had a job queue for the first time in ages.

Looks like everything ran successfully as well (and Steve's ATLAS jobs still got through) - I always worry that something subtle has broken which will only be revealed when the site gets busy.

I redid the fairshares on the fly though, because now that the theorists have decided to use the pheno VO, we have to reflect their 20% nominal fair-share in Maui (in fact everyone's getting 33% as CompSci and local Bio users are not yet active).

The biomed jobs are still coming in steadily - 2 or 3 a minute. Lovely jobs, actually. Run time is ~4 1/2 hours, so a good turn over rate - and they are 99.99% efficient!

Edinburgh and ATLAS FCR

Edinburgh got bumped by ATLAS for failing JS for quite a while, but as their 5 jobs slots were tied up with Zeus jobs the ATLAS SAM tests couldn't run so they couldn't get back in the ATLAS BDII. Of course this meant that Steve Lloyd's jobs were aborting (no matching resources).

So, Sam has allowed the atlassgm user access to the reserved ops/dteam job slot. This will be fine as long as nothing happens to stick that job in the queue.

However, it's all getting a bit silly - more and more tweaks in the batch system to rush through the test jobs.

Of course, to some extent it's an Edinburgh problem - they are so strapped for CPU that getting anything to run through in a guaranteed time is hard.

Durham Enable Fairshares

Mark, Phil and Nigel have enabled maui fairshares on the Durham cluster.

This is excellent news and should help them get better numbers on, e.g., Steve Lloyd's ATLAS tests.

I'm not quite sure how fair shares and pre-emption interact - I assume that they're somewhat independent.

Glasgow Update to gLite 3.0r16

This was the first gLite update with significant component changes:

* DPM was upgraded to 1.6.3, with a schema change and a new SRM v2.2 daemon.
* Torque and Maui were upgraded to v2.1.6 and v3.2.6, respectively, from the previously ancient LCG versions.

The DPM upgrade I tackled first. This was fine and I blogged about it on the storage blog. Just be careful to take a db dump before you try it, just in case things go wrong. There's also a strong warning against running automatic updates on gLite server nodes - this is not supported and some people are reporting DPM database corruption on LCG-ROLLOUT.

The torque/maui upgrade I was a little nervous about as I don't feel I greatly understand these components. However, Steve T had said that minor upgrades are ok (we'd been using the Steve T build for the cluster since the start, so we were already on torque v2), so I took the plunge. First I did a single worker, and restarted pbs_mom, so make sure the 2.1.6 mom didn't have trouble talking to the 2.1.5 server - and it didn't. So then I updated all the WNs, before turning my attention to the server.

Here, I did the usual yum -y update first. Then I restarted pbs_server and maui. pbs_server didn't restart cleanly, claiming something was bound to the port. I had a look, but by the time I did there was nothing - I think it was the server being sluggish to exit. So pbs_server then (re)started fine, and I did an extra maui restart to be on the safe side.

A basic check of the batch system (pbsnodes, qstat, diagnose) looked ok.

I have commented Steve's repository out of by yum.conf - we'll now use the "official" gLite build on Steve's advice.

N.B. I still intend to manage to batch system using cfengine, not YAIM - it's a lot more flexible for us to do this, e.g., the new routing queue being the default one.

Durham Queue reogranisation

Last week Durham implemented new queue structure, we now have three grid queues -

• ops (for ops jobs)
• pheno (for the phenogrid VO)
• grid (for all other grid jobs)

We have also implemented job suspension based upon queue priorities, this allows us to give local queues priority over grid jobs. How to configure preemption has been written up here http://www.gridpp.ac.uk/wiki/Maui_Preemption