Take my outputs, damn you...

We recently ran up a very large backlog of production output files waiting to go from Glasgow back to the Tier-1 (reminder, panda doesn't consider a job finished until the outputs are safely stored at the T1). This is clearly seen in the red line on the panglia plot above, which reaches very high values. As we recently cut the timeout for the UK cloud to 2 days for transferring jobs, to improve the responsiveness of the production system, we started to leak out failed jobs (light green line) as panda gave up and decided to rerun.

Fortunately we got a big boost in the number of FTS slots from Glasgow to RAL, increasing from 10 to 25 active transfers (see the bottom FTS monitoring plot). Even so it clearly takes 24 hours for all the backlogs to drain down.

One of the problems here is that the output files are small from simulation (a tiny log file and a 20-50MB HITS file), so the overheads of FTS + SRM are very considerable and the actual bandwidth achieved is quite low. One possibility we are considering in ATLAS is introducing a pre-merge of outputs on the T2, which will allow us to send much bigger files back to the T1 (although a final "super-merge" will probably still be necessary). For this we are waiting for the generic Athena merge transform and then we will need to test integrating this into the mainline production workflow.

Until then we just have to take the operational load of tweaking the FTS settings when necessary.

CREAM gets an upgrade

The CREAM instance at Glasgow has now been upgraded to the latest SL5 version. This continues the push to migrate those services that can be moved from SL4 to SL5 and should also make it easier to upgrade to the new 1.6 instance when it is released. The only hitch to a relatively painless upgrade was cfengine tweaking LCAS and replacing 64 bit path names with 32 bit paths.

Où est le site bdii

Our upgrade to the SL5 gLite3.2 site bdii has been tormenting me of late as even although the BDII was installed, it was only returning data from a local ldapsearch.

It was listening on port 2170 and the bdii process was running. Then when you tried an ldapsearch from a local machine, it worked. Trying it from a external machine, it could not connect.

First thought was firewall but iptables was not working. Then what about campus firewall. Nope, nothing had changed there. I checked the configs from SL4 to SL5 and they were the same. I turned on logging for slapd and turned up the verbosity. You could then see the DENY's being made by slapd itself.

After much googling I tried slapd in /etc/hosts.allow and this worked! It looks like with the transition to SL5 there is a requirement to add the slapd service to hosts.allow. This looks to have been a bug with openldap in SL4.

With the site bdii upgraded the change over occurred yesterday.

'EventRecords' is full

Our accounting database appears to be full.

org.glite.apel.core.ApelException: java.sql.SQLException: The table 'EventRecords' is full

Hmmm, what to do. Increase or archive?

You can see what is set from: SHOW TABLE STATUS FROM accounting LIKE 'EventRecords';

and if you want to increase you can use:ALTER TABLE accounting MAX_ROWS=1000000000 AVG_ROW_LENGTH=338;

But surely the correct thing would be archive. Handily the archival procedure is documented on the APEL wiki.

It is useful to know that the default size of MyISAM tables in MYSQL4 is 4Gb. Luckily in MYSQL5 and above the table limit is much higher. I wonder if the new SL5 APEL will ship with innodb tables?

Corralling jobs in Maui.

Sometimes, when testing new hardware or software in a limited way, it is important to be able to arrange lightweight, temporary partitions of a cluster for only a given user.

Now, you could repartition the cluster nodes between a "normal" partition and a "testing" partition, but for most pbs/maui clusters (which don't have anything but the 'ALL' partition set), this involves changing configuration for all the nodes, rather than simply the nodes we care about. (And then changing it back when you're finished.)

You might also consider doing this with reservations - indeed, the maui manual suggests that a reservation locked to a user specified with an & prefix will force precisely the behaviour we want - locking the reservation and the user together. This appears not to work under empirical testing.

Instead, the solution we've found to work is (all in maui.cfg):

1. Create a reservation for the user only.
   SRCFG[ssdnodes] PERIOD=INFINITY
SRCFG[ssdnodes] STARTTIME=00:00:00 ENDTIME=24:00:00
SRCFG[ssdnodes] HOSTLIST=node30[0-9]
SRCFG[ssdnodes] USERLIST=ssp001

2. Create a quality of service class with the property that it only runs on that reservation.
   QOSCFG[ssd] QFLAGS=USERESERVED:ssdnodes
3. Make the user a member of that quality of service class only.
   USERCFG[ssp001] QDEF=ssd QLIST=ssd

(In this case, the configuration mutually restricts the user ssp001 and the nodes node300 to node309 to each other.)

This has the benefit that it also generalises to any number of users, as long as you add them to the reservation and the QoS class.

SSDs - the testing begins!

This Monday (finally!) we received (half) of the SSDs we ordered for our storage testing plans.

These are the Intel G2 X-25s which are intended to represent the mid-range of the SSDs available currently (the low end ones are still due to arrive, and our high end card is being tested differently).

Just as a sneak preview, we had a chance to run iozone against one of the X-25s, in the same configuration as I've previously run against our newer disk servers (in RAID6 mode). As you can see from the graphs below, the SSDs behave exactly as we'd expect - the throughput is almost identical on random or ordered reads, whilst the RAID array suffers significantly from having to seek. Indeed, although the 22 drives in the array give it much better read performance when not seeking, the single X25 seems to equal the RAID array's performance when seeking is needed...

Next thing on the list is testing them in Worker nodes against Analysis and Production workloads.

LHCb Production Failures

Over the last week we have been investigating why we have around 50% failure rate with LHCb jobs. All seem to be failing with the same issue which is sometimes not being able to copy their results back to the Tier 0 or subsequent fail-over Tier 1 site. This is not strictly just a Glasgow issue and it has affected Sheffield and Brunel, although the issue appears to have gone away from Brunel.

We have tried pretty much everything, as simple lcg-ls and lcg-cp actually work from the worker nodes so its not a certificate issue. The failures are not particular to a CE. Nothing changed at our site prior to the failure and LHCb say nothing changed at their end. In fact they have sites in the UK such as Manchester working fine.

None of the failures correspond to a particular set of worker nodes which might indicate NAT issues for us as we split our odd and even nodes through separate NAT's. However, it does look like network contention at some point in the process as we see either broken pipes or timeouts in the logs direct from Globus.

2010-03-04 04:04:56 UTC dirac-jobexec.py ERROR: SRM2Storage.__putFile: Failed to put file to storage. file:/tmp/8230840/CREAM603030715/7472318/00005987_00009161_3.dst: globus_xio: System error in writev: Broken pipe
2010-03-04 04:04:56 UTC dirac-jobexec.py ERROR: globus_xio: A system call failed: Broken pipe

The only constant so far is that their appears to be a 50% failure rate from failed uploads which happens consistently from submissions from DIRAC.

Its certainly a puzzler and we are fast running out of ideas!

NATs Maxing Out

During our investigation of our LHCb failures we noticed that our number of conntrack entries on our two NAT hosts were in fact being totally used up i.e. all 43200! By looking at /proc/net/ip_conntrack we noticed that most of the connections were in fact udp DNS lookups by Camont jobs. We also noticed that we had not changed the default timeouts, 32768 for tcp and 3600 for udp. This was probably the reason they were being used up. So we have tweaked the timeouts and increased the maximum.
So our new NAT settings look like this:

/etc/sysctl.conf
original values of 43200, 32768, 3600 respectively.
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 21600
net.ipv4.netfilter.ip_conntrack_max = 65536
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30

Now out NAT's look much healthier. Only problem - it didn't help with LHCb productions jobs not being able to upload their results back to CERN. Back to the drawing board.

local users before pool users

Further to the original post by Graeme 'to voms or not to voms'. The Nikhef documentation has been thoroughly overhauled and I have now been able to switch lcmaps in CREAM and SCAS over to use local unix group mappings before pool accounts, if they exist.

The main changes are changing localaccount to pull in the glasgow centric grid-mapfile.

localaccount = "lcmaps_localaccount.mod"
               " -gridmapfile /usr/local/etc/grid-mapfile-local"
              # " -gridmapfile /etc/grid-security/grid-mapfile"

Some small tweaks are required to move localaccount from the last check to the first check. If this is successful it uses that account, otherwise it moves to check voms and pool accounts.

glexec_get_account:
proxycheck -> localaccount
localaccount -> good | vomslocalgroup
#proxycheck -> vomslocalgroup
vomslocalgroup -> vomspoolaccount | poolaccount
vomspoolaccount -> good | vomslocalaccount
vomslocalaccount -> good | poolaccount
poolaccount -> good #| localaccount

glexec_verify_account:
proxycheck -> localaccount
localaccount -> good | vomslocalgroup
#proxycheck -> vomslocalgroup
vomslocalgroup -> vomspoolaccount | poolaccount
vomspoolaccount -> good | vomslocalaccount
vomslocalaccount -> good | poolaccount
poolaccount -> good #| localaccount

SCAS is works in the same way and all that is required is to change the localaccount setting to pull in our Glasgow local grid-mapfile a'la

localaccount = "lcmaps_localaccount.mod"
                " -gridmapfile /usr/local/etc/grid-mapfile-local"
#               " -gridmapfile /etc/grid-security/grid-mapfile"

Job done. I can now flit between gla or pool accounts depending on my existence in /usr/local/etc/grid-mapfile-local

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
2013.svr008               cream_441636610  ssp001                 0 R q2d            
2014.svr008               cream_963867097  gla057                 0 Q q2d

VMware Web admin vs SL5.4: fight!

Recently, we've acquired some hefty servers for the purposes of running virtual machines (initially for test purposes and cheap dev boxes, but potentially for service hosting depending on how well it goes). We're using VMWare Server, which, although it comes with some command line tools, very much wants you to use the fancy web interface that it runs on non-standard ports.

This was fine, except that it seemed extremely flaky on all our test servers - randomly crashing, sometimes taking out a running VM with it.

It turns out that this is all the fault of our running an up-to-date version of SL. SL5.4 (actually, anything based on RHEL5.4, one assumes) has a version of glibc that VMWare really doesn't get on with well.

Once we copied the 5.3 release of libc.so.6 from a 5.3 server into a suitable place, and pointed VMware's LD_LIBRARY_PATH at it, it seems much more stable.

(The relevant bug report, including fix suggestions is:

http://bugs.centos.org/view.php?id=3884 )

more openmpi tweaking

Whilst testing MPI on our cluster and get it into a usable state I uncovered a rather nasty bug with openmpi-1.3.4. This manifested itself with never being able to run on the node with cores > 4. It was a weird one as openmpi communication over two nodes worked fine with 8 cores on each node but when a job requested cores > 4 on the same node. The job just hung. An strace of the mpiexec process suggested some sort of TIMEOUT/WAIT issue.

On the release note for openmpi-1.4.1 it appears they discovered this bug and provided a fix:
- Fix a shared memory "hang" problem that occurred on x86/x86_64
platforms when used with the GNU >=4.4.x compiler series.

This sounded plausible and in fact an upgrade has fixed the issue.

So now with all 8 cores running on the same node the next issue to arise was one related to Maui. Some time when you requested nodes=8, Maui scheduled the job on 3 cores, a qdel and a resubmission later Maui rescheduled the job onto 5 cores. On one test I even qrun'd the job and it appeared it start on the correct number of nodes but there appeared to be no reason for Maui not getting this correct. So it was time to get out Maui docs.

from the docs;
Maui is by default very liberal in its interpretation of :PPN=. In its standard configuration, Maui interprets this as 'give the job * tasks with AT LEAST tasks per node'. Set the JOBNODEMATCHPOLICY parameter to EXACTNODE to have Maui support PBS's default allocation behavior of nodes with exactly tasks per node.

This seemed to suggest that Maui's default behaviour is to pack a job into as few nodes as possible. So I tried out setting the JOBNODEMATCHPOLICY to EXACTNODE and this seems to have done the trick.

nodes=24 means 24 nodes, not 8, not 6 but 24

This does have a drawback in that it will be 24 separate nodes. This setting relies upon being able to set :ppn (processes per node) to allow nodes=3:ppn=8 giving 24 cores which is really what you want to say. As you probably have a fast machine with loads of memory and cores. Therefore, you could target all the cores rather than 24 nodes. However, it is a start.

Wouldn't it be nice if you could specify :ppn in JDL. The only way round this I can see for now is to manually change the job manager or use the local batch attributes of CREAM to allow a custom cerequirement to be specified. Possible but not nice.

cream sours

Well we now know how much it takes to kill our CREAM instance. Yesterday it stoppped working completely and it appeared to be caught in a tailspin with the Lease and Proxy Renew processes within CREAM. Grepping the logs indicated that most of the Renewals and Lease Manager entries were all related to condor submission from ATLAS.

From speaking to Massimo at INFN it was described how Proxy and Lease renewals are operations which are executed with higher priorities wrt other commands. One hypothesis might be that the CREAM CE was so overloaded doing these commands that it was unable to deal with basic job submission since all the test jobs I submitted never made it out out the REGISTERED state.

It looked bad on Ganglia:


The first course of action was to disable job submission using the command line tool: glite-ce-disable-submission and try to deal with the renewals. This worked for a time but they reoccurred later on that evening.

The timestamps on these ATLAS cream jobs seems to be very old and hinted at stale jobs so the next course of action was to manually purge the database using the tool provided by the CREAM developers: here. The easiest way I could see to do this was to connect to the creamdb, select out the id's and create a script that called the purger for each id. Note: you need jdk 1.6 in order to run the purger!

This ended up removing around 3000 CREAM entries.

Ganglia looked much happier:


So I think you have to be careful when getting submissions from Condor at the moment as it looks to be quite easy to denial of service your CREAM CE.

Roll on CREAM 1.6

- That proxy renewal is not very efficient in the release now in production (already addressed in the coming CREAM CE: see here)
- When there are too many pending commands, new job submissions will be disabled by the limiter: see here

ScotGrid's shrink wrapped UI

In an effort to reduce overhead for new external users who wish to submit to Glasgow I have created a shrink wrapped gLite UI. This comes in the form of a slimmed down Virtual Box SL5 image with pre-installed gLite UI.

The hope is that users from external institutions who wish to run jobs on the EGEE grid and more specifically at ScotGrid will be able to take advantage of this. This is of particular importance for external users of Lumerical's FDTD who are primarily engineers who just want to run the software rather than install an SL5 gLite UI first. The end goal is extending this to help all our users get up and running as quickly as possible.

This will come pre-installed with Glasgow's submission tools such as gqsub and other more specific user scripts. Those wishing a link to download the VM should drop us an email.

Details of the VM image, setting up the UI are available at the wiki.

I am still at a loss how CERN managed to get their VirtualBox image down to 500GB!

pick a torque, any torque

Since our seg-faulting mom issue during our SL5 upgrade using 2.3.6 server & mom I have compiled a variety of Torque versions of late and trialled them out. I have now come to some conclusion and am sticking with the 2.3.* series. 2.3.9 at the moment - well until another bug is found!

2.3 Series

2.3.6 - seg-faulting mom during some unidentifiable race condition
2.3.7 - untested
2.3.8 - Operators/Managers Lists Bug
2.3.9 - Seems stable

2.4 Series - Beta

2.4.2 - OSC MPIEXEC Bug
2.4.3 - OSC MPIEXEC Bug Fixed & Operators/Managers Lists Bug
2.4.4 - OSC MPIEXEC Bug Back in

Leaks caused by frozen ICE

We had a rather quiet time here over winter - a slight hiccough with a disk server, but all rather stable. Other than that, the big freeze didn't result in much.

Except for the ice freezing up, and causing leaky pipes.

That's ICE - the WMS plugin that submits to CREAM. It turns out that it can break the pipes, and leak bits of past jobs. This resulted in an error message like:

Warning - Unable to submit the job to the service: https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server
System load is too high:
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
Threshold for ICE Input JobDir jobs: 1500 => Detected value for ICE Input JobDir jobs /var/glite/ice/jobdir : 1514

from both WMSen. In principle, this is reasonable: it's saying that the WMS is loaded up, so no more jobs for the moment. A decent way of ensuring running jobs are not harmed by new submissions, in the event the system explodes.

Except the system load was the lowest I've seen on them, under 1.0. Dug dug into the underlying Condor instance, which had only a few jobs in it, and the hunt commenced for the 1500 phantom jobs.

As the error message suggests, /var/glite/ice/jobdir/old had 1514 files in it, each one representing a past job. However, most of these were old - over a month old. Given that the WMS is supposed to purge the jobs after a month (if they user doesn't do it earlier), that shouldn't have been the case.

Derek down at RAL confirmed this - it's apparently a known bug; but I can't quite see it on the gLite Known issues page. It looks like most of the UK's WMS's fell to this at the same time. I think that's due to the increased number of CREAM CE's (so the rate of use of ICE is climbing), and the fail over on the clients if one WMS is down - resulting in a nice, even distribution of failure.

In the end, the fix was simple - I moved the files older than a month out of /var/glite/ice/jobdir/old. Deletion aught to be safe, but they're tiny. I'll need to automate that, until such time as the bug is fixed - but also need to watch in case the usage increases further, and 1500 isn't enough to last out a month of use. In that case, I think I'd probably temporily increase the
limit on the WMS (I believe it's in a configuration file), knowing that most of them are stale phantoms.

The only discussion I can find related to that error message resulted in pointing the finger at the glite-wms-ice-safe process. ICE has two processes, and it appears that the ice-safe is the part responsible for cleaning up. However, as far as I can tell, both processes are running on each of our WMS's, so this appears to be a different case from the previous one. It might have been the case that the ice-safe process died, and when it's restarted it's not removing old jobs? I don't know - if I find out I'll update here.

The purpose of this post is to get the error message from the WMS into google, and on the same page as something that talks about the issue; and resolution. In case it freezes up on us again.

(Almost) 100% Success for Glasgow!

We noticed today that (apart from that pesky red mark for CMS over the past 6 months, and some yellow on the WMS tests) we're looking incredibly green and functional on the Glasgow Dashboard at the moment. So, we took a picture before that changed...

(The Glasgow Dashboard is Mike's mashup of all the useful metrics on the web concerning UKI-SCOTGRID-GLASGOW, now over two, alternating, pages. It's actually quite useful, and remarkably festive at this time of year.)

openmpi magic

I have just rebuilt openmpi-1.3.4 for use with CASTEP. This is built to a useful /opt location with Torque, F90 support for gfortran44.

gLite support for OPENMPI is fairly generic and means any openmpi rpm install does not have any useful batch or interconnect support. So anything out of the ordinary requires a custom build.

I will stick the RPM available for download here in the next few days.

The magic for building from a src rpm:

cd /usr/src/redhat/SPECS/
rpmbuild -ba --define '_prefix /opt/openmpi-1.3.4' --define '_mandir %{_prefix}/share/man'
--define 'configure_options --prefix=/opt/openmpi-1.3.4 --with-tm=/usr/
FC=gfortran44 F77=gfortran44 CC=gcc44 CXX=g++44 FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2'
openmpi-1.3.4.spec

Full Instructions are here.

issues with gfortran43/44 and mpich

I am finally getting to the bottom of what has been going wrong with re-compiling MPICH for F90/F95 (required for CASTEP - a demanding Fortran code). I have now narrowed it down to one issue between recompiling MPICH with gfortran43/44 for SL5 usage with CASTEP instead of plain old gfortran.

FC="gfortran44" ; export FC;
F90="gfortran44" ; export F90;
...
--enable-f90modules

The SAM MPICH test runs after recompiling with F90 support using gfortran and in fact it works fine on SL4 and SL5. So that was not the issue.

message size            transfertime            bandwidth
32 bytes                0.000000 sec            inf MB/s
2048 bytes              0.000117 sec            17.476267 MB/s
131072 bytes            0.001445 sec            90.687654 MB/s
8388608 bytes           0.078437 sec            106.946397 

It turns out that MPICH just doesn't work when compiled with gfortran43/44. Leaving me in a bit of an pickle as CASTEP will not compile on SL5 with gfortran, you have to use gfortran43/44!

Time for the backup plan ..... openmpi.

lightening testing of glexec with SCAS

Well since it is looking increasingly lightly that we will be moving to some form of identity switching at our sites to give us more information about who is running their jobs via their pilot frameworks. I thought I would give it whirl.

So in some lightening tests, a phrase I am stealing from lightening talks sometimes given at technical conferences, I am trialling glexec for identity switching coupled with SCAS for centralised allow/deny decisions.

Here is what was tested:

an install of SCAS
and install and test GLEXEC with SCAS on LCG-CE
and install and test GLEXEC with SCAS on CREAM [1]
and install and test GLEXEC on WN (SL4)
and install and test GLEXEC on WN (SL5)

Detailed Instructions and Results can be found here

The short and long of it is that it is very easy to set-up SCAS and use it on whatever service you want. So easy infact that once you SCAS server is up and running you cn direct calls to it from your CE's in a matter of minutes. glexec on the WN is just as easy, all that remains would be for someone to use it.

We currently have not rolled any of this into production but I am confident that it could be done quickly and safely. Since we are into real data taking, safely is the keyword. We want no unnecessary downtimes, which I think is achievable.

Thanks to Oscar at Nikhef for answering questions.

1: there appeared to be a certificate permission issue when calling SCAS from CREAM that prevented job submission. It looks like you need to copy the hostcert/key by hand to another cert owned by the tomcat user.


-rw-r--r-- 1 tomcat tomcat 2187 Dec 4 10:44 tomcathostcert.pem
-r-------- 1 tomcat tomcat 1863 Dec 4 10:44 tomcathostkey.pem


Mysql binary logging revisited

After last time, I'd poked at the LB servers databases, so we were getting effectively lock free backups, on one of the servers.

In the intervening period, after it was seen to be stable, I did the same for the other server.

However, Mike noted that the disk space used for the logs was growing rapidly. (I blame those pesky LHC physicists. Running jobs on our systems - anyone would think there was data to analyse or something ...). Because we're running with LB servers on the same machines as the WMS, this means that the /var partition contains both the database files, and the users sandbox - hence the old binary logs take space away from users stuff. (That's something to think about for the reinstall - might be worth separating them).

Time to automate log triming. Firstly, the manual side: the statement

PURGE BINARY LOGS BEFORE '2009-11-01 00:00:00';

to the server trims out some of the older logs. You can also trim up to a given file.

Better than that, however, is to put

expire_logs_days=8

in the my.cnf. This tells mysql to retire logs older than 8 days at server start up, or when the logs are flushed.

So as long as we ensure that when we take a full backup we flush the logs, then logs are automatically trimmed to just over a weeks worth. Adding that parameter to the mysqldump script, and we're done.

The binary logs have value, independant of the backups - there's a tool to read them, and look at what was happening. Whether 8 days is the best level for us is something that we'll have to monitor - arguments for shorter time periods seem stronger than for longer.

Torque 2.4.2 to the rescue

I previously blogged about our Torque 2.3.6 on SL5 mom's continually seg-faulting. At first we thought it was a bitness issue 32/64 between our SL4 and SL5 mom's running through the same pbs_server. However, a quick test with the SL4 nodes removed proved that this was not the case. A trawl through the source proved unproductive.

Therefore, it was time to go to Plan B. To that end I have built the latest Torque release 2.4.2 and tested this on our pre-prod staging cluster. This worked well with a configuration of 2.3.6 server and 2.4.2 mom's. The next test was a test on a single node in production. This was successful and was running jobs fine when all the other mom's seg-faulted again. The 2.4.2 mom survived this and continued to run. So a full roll-out is under way. We will think about upgrading the server at a later date. The only point to note is that we have to fully drain a node before doing the upgrade which is pain. It does attempt a job conversion but these are unsuccessful as far as we can tell and you end up with dead job holding onto job slots.

So the moral of the story is stay away from 2.3.6 and go to 2.4.2 instead.
It is pretty easy to build but I have hosted our build here for anyone that wants them.

A tale of two job managers

A while back I posted about supporting SL4 and SL5 OS's through the same batch system. Our solution was to use torque submit filters to add additional node properties to the jobs as they passed through the job managers on the CE's. This coupled with specific node properties on all nodes on the cluster worked quite well until I noticed that we were leaking CREAM jobs that should have requested SL5 running on the SL4 nodes.

After some investigation it appeared that when I was testing the filter and running the cream pbs qsub submit by hand I was always setting the number of nodes, even if I only required 1 i.e.

 
as a pool account .... 
/opt/glite/bin/pbs_submit.sh -c /bin/hostname -q q1d -n 1

This meant that there was always a #PBS -l nodes= in the submission script. However, if you call pbs_submit without the -n you get behaviour where no #PBS -l nodes= line appears in the final submission script. This then relies on the default behaviour that if no number of nodes is specified you get 1 node. This meant that my pbs filter did not catch the number of nodes and did not add the node property at all!

As it turns out on deeper investigation into the CREAM pbs_submission script. That when 1 node is required it uses the pbs default behaviour and does not specify a number of nodes. Only when there is more than one does it specify this i.e. MPI. This is a change from the lcg_CE job manager which always specifies a number of nodes be it 1 or more. Something to remember.

To get round this I have added an additional line to the cream pbs submit script to always default to 1 node if not MPI. Not the best but it's a short lived tweak until we get rid of our SL4 support. This should be very soon.

/opt/glite/bin/pbs_submit.sh

[ ! -z "$bls_opt_mpinodes" ] || echo "#PBS -l nodes=1" >> $bls_tmp_file

CE Publishing

The Problem

Publishing an inhomogeneous site 'correctly' is not trivial. This is now required in order to pass the new gstat2 Nagios tests. Things to remember -

* Physical is sockets/CPU's and Logical is Cores.
* Physical * Cores = Logical in order to pass the new central Nagios tests.

If your cluster is inhomogeneous then you need to be able to publish both clusters separately or as one or come up with a fudged number. It is made harder as we have one batch system with multiple CE's submitting to it.

Some Solutions

* Sub-Clusters [ what we have implemented at Glasgow ]
* Publishing decimal for cores

our implementation is discussed here.

Please let me know if anything is wrong with this and I will update.

Segfaulting PbsMoms

We have an issue with segfaulting mom's that seems correlated with the server trying to ping it's moms. The server are version is torque-2.3.6-2cri.x86_64
We are currently supporting two OS's through the same batch system using submit filter and node properties. Therefore, we have two different versions of moms.
Nodes 1->295 have moms torque-2.3.6-2cri.x86_64 and 296->309 have moms torque-2.1.9-4cri.slc4.i386

When the moms segfault we see that the torque-2.1.9 moms stay up and only the torque-2.3.6 moms all die. I ran one of them through GDB and can see the call stack:

(gdb) where
#0 mom_server_find_by_ip (search_ipaddr=177078032) at mom_server.c:450
#1 0x000000000041965e in mom_server_valid_message_source (stream=0) at mom_server.c:2022
#2 0x0000000000419870 in is_request (stream=0, version=1, cmdp=0x7fffff542ae8) at mom_server.c:2125
#3 0x0000000000416997 in do_rpp (stream=0) at mom_main.c:5351
#4 0x0000000000416a52 in rpp_request (fd=) at mom_main.c:5408
#5 0x00002ae8ae9f3bc8 in wait_request (waittime=, SState=0x0) at ../Libnet/net_server.c:469
#6 0x0000000000416c1d in main_loop () at mom_main.c:8046
#7 0x0000000000416ee1 in main (argc=1, argv=0x7fffff5431d8) at mom_main.c:8148
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) bt full
#0 mom_server_find_by_ip (search_ipaddr=177078032) at mom_server.c:450
__v =
pms = (mom_server *) 0x6cbb80
addr =
#1 0x000000000041965e in mom_server_valid_message_source (stream=0) at mom_server.c:2022
addr = (struct sockaddr_in *) 0x187ef434
pms = (mom_server *) 0x0
id = 0x43be08 "mom_server_valid_message_source"
#2 0x0000000000419870 in is_request (stream=0, version=1, cmdp=0x7fffff542ae8) at mom_server.c:2125
command =
ret = 0
pms =
ipaddr =
id = "is_request"


So it looks like time to dive through the source for mom_server_find_by_ip (search_ipaddr=177078032) at mom_server.c:450 or install torque-2.4!

Arc, authorisation and LCMAPS

As a gLite site, it would be ideal if we could have the same user mapping between certificate DN's, and unix user names that is used with our existing CE's.

Which means using the gLite LCMAPS to make decisions about what username each user has.

This is supported in Arc, but it's not in the same fashion.

The best approach appears to be: Have an initial mapping listed in the grid-mapfile (There's utilities to make this easy). This allows a first pass of authorisation. Then, in the gridFTP server, the mapping rules in there are applied next - this is where LCMAPS applies.

Interestingly, Arc makes it very easy to do the thing we found hard with LCMAPS - to have a small set of 'local' users with fixed permanent mappings (independant of VO), and VO based pool accounts for other users.

However, it's in the LCMAPS integration that things get a bit stuck.

It's a silly 32/64 bitness issue. On a 64 bit system, yum pulls out the 64bit Arc - as you might expect. Sadly, there's not a 64 bit version of LCMAPS in the repositories as yet.

So it's a case of hacking what I need out of etics. I'll post a recipe when I have one, but this is a pretty tempory situation - it looks like Oscar pretty much LCAS/LCMAPS ready, but they're not a separate package, so are waiting on the SCAS, CREAM or WMS SL5-64bit packages.

nmon

Seeing sams post about NFS prompted me to mention 'nmon' - its kinda like 'top' on steroids and does particularly useful trend plotting. Originally a hack for AIX but ported to linux once a certain vendor realised people weren't just buying powerpc systems....

Anyway - go grab from http://www.ibm.com/developerworks/aix/library/au-analyze_aix/ - the linux version is now opensource I see.

NFS Load Tweaks: a Brief Guide for the Interested Enthusiast

I was asked about the mystery of NFS server tweaking in a dteam meeting, so I thought I'd compile this brief blog post.
As with all actions, there are two steps: first, gather your information, second, act on this information.

1) Determining your current NFS load statistics.

NFS logs useful information in its /proc entry...

so:

> cat /proc/net/rpc/nfsd

rc 0 28905480 1603148913
fh 133 0 0 0 0
io 3663786355 2268252
th 63 362541 16645.121 3156.556 747.974 280.920 148.129 100.155 61.480
42.249 40.829 90.461
ra 256 1069115586 4089582 3055815 2625032 2228952 2114496 1983622
1765372 1743563 1610465 89609536
net 1634942152 0 1634971040 2214677
rpc 1630024431 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 1573543 1535237104 8743056 1545350887 1532645717 29571823
1179900114 9214599 6691508 538717 366274 0 2801854 39816 505310 4298
2486034 62181794 53164 2414727 0 986878
proc4 2 0 0

This somewhat arcane looking output is full of variously useful
statistics about your nfs daemon.

The "rc" (read cache) field gives the fraction of cache hits, misses
and "nocache" (interactions which bypassed the cache) for read
operations.

The "fh" (file handle) field's most important entry is the first - the
number of stale file handles in the system. If you have flaky NFS, for
example, this will be non-zero.

The io field is simple cumulative io (read, and then written) in bytes.

The "th" (threads) field is the most interesting field for NFS load
optimisation. The first entry is the total number of threads currently
executing. The second is the number of seconds (?) all threads were in use
(which means your NFS was maxed out in active connections). The
remaining 10 entries are a histogram of NFS thread utilisation, in
seconds (it seems to be hard to get NFS to reset this; restarting the
daemon definitely doesn't). Plotting this gives you an idea of how
much time your NFS server spends in various load states.
Ideally, you want the last entry (90-100% use) to be comfortably in
the tail of your distribution...
If you have indications that your server spends a lot of its time with
all threads in use, you should increase the maximum number of threads
- powers of 2 are recommended.

The "ra" (read-ahead cache) field gives similar results, but for the
read-ahead cache. The first number is the size of the cache, the next
10 are a histogram showing how far into the cache entries were found
(so, the first number is the number of times an entry was read from
the first 10% of the cache), and the last is for cache misses.
Obviously, if you're getting a lot of cache misses *and* your cache
hits histogram is heavily right-skewed, it's worth increasing the
cache size. (Conversely, if you have a heavily left-skewed histogram,
and few cache misses, you may be able to manage with a smaller cache.)

The remaining fields are rpc process info fields, which are less
relevant to us for our purposes.

2. Optimising your NFS.

The most important things to ensure are that there are enough
resources for the peak load on your NFS service. NFS will spawn new
threads to handle new active connections, and if its max-threads limit
is too low, you'll get brown-outs under high load.
Starting at least four instances of nfsd per processor (and, on modern
processors, up to 8 per core) is recommended as a sensible
configuration. You can set this on the command line for the nfsd
service by simply using the bare number as an option.

And, of course, if you can bear the risk of data-loss (or silent data
corruption!) on sudden server loss, setting the export option "async"
trivially increases your network throughput by removing the need for
confirmation and syncing of writes between clients and server.
See the NFS config faq at:
http://nfs.sourceforge.net/#section_b
for more details.

You may also wish to do the standard setting of packet sizes with
respect to MTU that you would normally do for a network-based
protocol. The general process (and some more details) are covered at:
http://nfs.sourceforge.net/nfs-howto/ar01s05.html

Arc, and the installation

We've been fiddling with the NorduGrid Arc middleware a bit. Not just out of random curiosity, but more trying to get a handle on the workloads that it suits better than gLite, and vice versa. It does a number of things differently, and by running an Arc CE in parallel with an lcg-CE and CREAM, we can do some solid comparisons. Oh, and the name of the middleware is also much more amenable to puns, so expect a few groaners too.

So, consider this the first in a series. During this process, we expect to end up with a set of notes on how to install and run an Arc setup, for people already familiar with gLite.

Firstly, install. We took a blank SL5 box, added the nordugrid repo's, and then

yum groupinstall "ARC Server"
yum groupinstall "ARC Client"

Well, very nearly. There's one more thing needed, which is to add the EPEL dependancies (libVOMS is the key lib)

yum install yum-conf-epel

The next step is to configure it. That's all done in /etc/arc.conf, and is the subject for later posts.

There is a need for a filesystem shared between the CE and the worker nodes, so we fired up a spare disk server for NFS.

Startup is three systems, already configured in /etc/init.d : gridftp, grid-infosys and grid-manager.

Ta-da! A running Arc CE.

Ok, so there's a fair bit glossed over in the configuration step. Next time, I'll talk about how I configured it to work with our existing queues - and where the expectations for Arc differ from gLite.

worker node on demand

Virtualisation is a hot topic again for grid services and worker node on demand

KVM, XEN, VMWARE - Everyone using different ones.
Virtualisation for cloud - Nimbus, Open Nebula, eucalyptus

the future... ??
1. plain signed virtual images transported from site to site.
2. virtual images including experiment software.
3. connecting to pilot job frameworks, instantiated with virtual images,
4. pilot frameworks replaced by commercial domain schedulers. virtual clusters.

HEPIX is GO!

HEPIX Workshop

Site Reports Session

CERN:
Getting serious about ITIL. Solaris being phased out. Getting serious about 10GigE.
Lustre pilot project. New purchases discussed.

JLAB:
New LQCG Cluster "2009 Quad Infiniband - ARRA Cluster"
Storage - Whitebox 14 AMAXservers Solaris w/ZFS or Lustre
Compute - DellpowerEdge R4102 x4 Ghz QDR Infiiband, 24Gb RAM

Auger Cluster Upgraded
Nehalems - intel x5530 dual cpu, quad core, 24MB RAM, 500GB SATA
(seeing i/o contention on disk when running 14/16 jobs)
OS Switch from Fedora 8 32bit, to CentOS 5.3 64bit

No real Grid Computing
IBM TS3500 tape library installed. StorageTek Powderhorn silos replaced.
80 production VM's VMWare ESX3.5 planned to move to vSphere4.0

GSI:
FAIR - new accelerator discussion. The futuristic talk!
The Cube DataCentre Building: 1000 19" water cooled racks held in 26x26x26 cube building. Lifts to reach the machines. Iron structure for racks to sit on.

CINP2P3 LYON:
T1 4LHC & D0, Babar, SL5 migration in Q2 2010 for both Main Cluster and MPI Cluster. New Purchases and New Server Building.

STORAGE Session

Your File System NexGen openAFS (Jeffery Altman):
YFS now funded by US Gov to create nextgen openAFS. 2 year funding. Deliverables included assessment of current AFS and 2 year upgrade plan to client and server for YFS deliverable. Still open source.

Storm and Lustre:
IOZONE discussion, Hammer-cloud Tests Discussion, Benchmarking summary, Good Results, performance below iozone tests. WMS jobs and Panda jobs different. file::// protocol support performs well but requires the VO to support it. Open questions: Lustre Striping (should yes or no). Performance (Raid config?), Monitoring - still work to be done, Support - Kernel Upgrades can take a while to be made available and Benchmarks - are they realistic? Tuning still to do.

Lustre at GSI:
Users - Alice Analysis for Tier2, GSI Exp, FAIR Simulations. Still on 1.6.7.2 1Pbtye, > 3000 nodes. Foundry RX32 ethernet switch. MDS HA Pair, one standby. 84 OSS, 200 OSTs. MDS 8 core, 3GHz Xeon, 32Bb RAM. Real throughput testing with Alice Analysis Train. 50Gbit/s using 2000 cores. Hardware and Software issues. Complex system and vulnerable to network communications. Using Robin Hood Filesystem Monitor for audit and management. This protects the MDS by directing requests to MYSQL instance. i.e top ten users, file moves etc. Using this rather than e2Scan.

Hadoop on your worker nodes using local hard drives & Fuse:
Hadoop compared against Lustre. Performed well when 8 jobs ran. Replication of files provides redundancy. Cost and maintenance factor very favourable to small sites. Deployed in some sites in the US. Not a really Tier 1 deployable solution. Name node redundancy exists (will lose at most one transaction) - requires additional software.

Virtualization Session

lxcloud at CERN:
Cern has developed a proof of concept for virtualized worker nodes. 'Golden nodes' serving images to the Xen Hypervisors using Open Nebula. Also looked at Platform's VMO. Production lxcloud being built. 10 machines, 24GB, 2TB disk dual Nehalem. Starting with Xen. Production release by March 2010. Memory an issue as the HyperVisor requires some memory i.e. with 16GB RAM you cannot run 8 2GB VM's.

Fermigrid:
Has moved much of its infrastructure to Xen HyperVisor. Looks like a solid infrastructure. Investigating KVM with the possibility of a move in the next few years if it proves to be better. INFN mentioned Xen vs KVM at Hepix Spring 2009 for discussion of differences.

Another new VO at Glasgow

Today I finally got time to create a new VO for our new users in Solid State Physics.

vo.ssp.ac.uk

This is now active across the cluster and users can sign up to the VO from our voms server on svr029 and will be used to host users of CASTEP and other departmental SSP users.

Our local wiki page on running CASTEP at Glasgow. Only the MPI version to get working now.

CASTEP, A Test of True Grid

Along came another users with a requirement for MPI. Can we run it? Well yes you can but remember our interconnects are just plain old Ethernet and nothing fancy like Myrinet or Infiniband. We are not a HPC cluster but an HTC cluster.

So we have been building CASTEP, an f90 code, heavy on the MPI scatter/gather. A test of true grid for any HTC cluster. First off CASTEP requires a minimum of make3.81 and gfortran43. Handy that we moved to SL5 as these are now the standard. Coupled with making sure that the required libs fftw3, blas and lapack are all built with the same compiler, gfortran43. This allowed the single core version to be built and installed onto the grid.

An MPI version is turning out be a bit more work. First off the old, outdated and no longer developed libs MPICH have not been built with .f90 support enabled by default. So we have got hold of the source to do a recompile with .f90 support on for gfortran43. There also appeared to be a bug in the gfortran support. So we had to patch the src rpm to include a patch that we located online. This allowed us to finally build the mpich lib. This has been tested with compilation of an MPI job in c and f90, both of which run successfully.

Unfortunately CASTEP still doesn't run using it so more digging required.

gqsub at EGEE09

Just a short note from the EGEE 09 conference. It's been very gratifying to have had so much interest in gqsub at the conference - I even had emails about it scant hours after the poster was put up (and before the offical poster session!).

In response to the comments recieved, I've put a roadmap of planned features up on the gqsub page, which gives an idea of where it's headed.

In addition, v 1.2.0 is out, which implements auto staging back of output. This means that in cases where there is not a shared filesystem between the UI and the worker node, but there is GridFTP server on the UI, then gqsub will pull out the JDL tricks we used earilier with the Lumerical deployment. This results in the illusion of a shared filesystem - the job is submitted, and the output appears in the right places as if it was done in a shared filesystem.

torque submit filters

After debating whether to add node properties for SL4 and SL5 into the job managers for both cream and the lcg-ce I read Derek's post from RAL about using submit filters. So I thought I would have a go and see if I could tweak the node specification, keep the number of nodes requested intact for MPI and add additional property for the particular CE. Turns out its easy to implement but as usual there is some wierdness. You should be able to write your filter in whatever language you like and just specify the torque.cfg i.e.

Here is a simple example in bash:
/usr/local/sbin# cat torque_submit_filter.sh

#!/bin/sh
while read i
  do
    if [[ $i =~ "^#PBS -l nodes=[0-9]" ]]
    then
      export i="${i}:SL4"
    fi
    echo $i
  done

/var/spool/pbs# cat torque.cfg

SUBMITFILTER /usr/local/sbin/torque_submit_filter.sh

This works with cream but not with the lcg-ce.

So lets try again but this time in perl:

/usr/local/sbin# cat torque_submit_filter.pl

#!/usr/bin/perl -w

use strict;

# Echo all other input
while ()
  {
    # By default just copy the line.
    my $line = $_;

    if ($line =~ m/^#PBS -l nodes=[0-9]/)
      {
        chomp($line);
        $line = $line . ":SL5\n";
      }

    print ($line);
  }

Now this works with both cream and lcg-ce! Obviously you can do whatever takes your fancy to the qsub input and make it more intelligent.

A word of warning. We used the same queues for both CE's which meant that SL4 and SL5 resources were indistinguishable to users unless they used OS specific CE requirements. We ended flooded on the SL4 queues, with lots of free slots on the SL5 queues. So in the end we have created a new set of queues for the SL4 CE. Hopefully this will be explicit enough for users to target the correct CE.

Fun with 5!

So, the SL5 migration was done last week, and as promised at yesterday's dteam meeting I am posting our problems so that other sites can watch out for similar issues (although some of these are deeply related to the way we do things at Glasgow).

First though, the successes:

1. 1800 cores running SL5
2. DPM headnode upgraded to SL5
3. Torque server upgraded to SL5, running torque 2.3.6, maui 3.2.6p21
4. /atlas/uk voms group supported with a separate fairshare
   

Now, the list of problems:

1. We introduced a new python script, gridAccounts.py, to generate pool accounts, retiring the venerable, but incomprehensible, genaccts.pl script we had before (Andy Elwell wrote that and his comment was "OK - I give up with python as I need this NOW..."; my retort was "I HATE PERL SO MUCH. IT'S A SHIT LANGUAGE.", but I had never found the time to rewite it until now). The new script reads standard config files, so it's a lot easier to manage, understand and extend. However, all change is (a bit) dangerous and the new script initially had groups in the wrong order in yaim's user.conf, which caused the groupmapfile to be wrong. This then caused all jobs to fail - the uid/gid of the gridftp session did not match the uid/primary gid of the user and gridftp does not like that at all.

(The reason we have to write users.conf is because we still get yaim to do a lot, although we manage all accounts through cfengine; yaim relies on this file to configure various other aspects of the system, such as grid/group mapfiles.)

2. We were trying to support the /atlas/uk VOMS group as a separate entity. This is simple in theory (!), you're looking for the following entries in voms-grid-mapfile:

"/atlas/uk/Role=NULL/Capability=NULL" .ukatlas
"/atlas/uk" .ukatlas

and this in groupmapfile:

"/atlas/uk/Role=NULL/Capability=NULL" atlasuk
"/atlas/uk" atlasuk

If we were managing these files directly, it would have been no problem. However, convincing YAIM to do this was far from easy. This is not helped by the fact that YAIM is now utterly incomprehensible in many ways (have a look at yaim/utils/users_getvogroup if you don't believe me). Finally we hit on the correct recipe, which is to have these accounts in users.conf, with a new "special" defined:

201601:ukatlas001:201040,201000:atlasuk,atlas:atlas:uk:
201602:ukatlas002:201040,201000:atlasuk,atlas:atlas:uk:
201603:ukatlas003:201040,201000:atlasuk,atlas:atlas:uk:
...

with this line added to groups.conf:

"/VO=atlas/GROUP=/atlas/uk":::uk:

Aside: Sometimes I wonder if YAIM has outgrown its usefulness. From something we could understand and tweak easily it's now a sed|awk|cut|sort|tail black box monster, which uses a computerised format for configuration files. c.f. the configuration we have for our own scripts:

[someuser]
dn = /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=some user
uid = 4832
home = /clusterhome/home/someuser
group = atlas
tier25 = True
vo = gla

And trying to do grid configuration manipulations in a language which doesn't have dictionaries is just ridiculous.

Maybe we'll need to wean ourselves off it eventually?

3. Information publishing on svr018 was broken after the upgrade. There was a cryptic reference to a required 'dpminfo' user which Sam had made in his notes. Adding this user did seem to make things work, though it's not at all clear why. Hopefully Sam will enlighten us later. In passing, note that the resource BDII on the service nodes seems to be 'protected' now, so attempts to reach it from 'outside' fail. This is new behaviour and lost us some time in debugging.

4. Terrible trouble was caused by upgrading the torque server to SL5. Using the SteveT build of torque server (http://skoji.cern.ch/sa1/centos5-torque/) seemed to cause grave problems with moms crashing on the worker nodes. Downgrading the moms to the torque 2.3.0 didn't work as the jobs files (/var/spool/pbs/mon_priv/jobs) seemed to be in incompatible formats and led to crashing moms plus a very confused torque server. Cleaning out all jobs seemed to not work either. The final solution was to rebuild torque 2.3.6 on SL5 - this gave a consistent and compatible server/mon pairing.

A small side effect though, was the the rebuilt maui had a different 'secret' in it, so I have had to hack the info provider on the SL4 CEs to use the --keyfile= argument in the maui client commands. (That's such a stupid 'feature'.)

5. Once we were out of downtime, random transfers to the DPM were failing. Eventually we tracked to the reduction in the number of pool accounts for atlasprd. There was no sync between the passwd fila and to the /etc/grid-security/gridmapdir pool account list, of course, so gridftp was was throwing a "530 Login incorrect. : No local mapping". We realised that

1. /etc/passwd should be handled better on nodes which need to map pool accounts.
2. For the moment never reduce the number of accounts!
3. N.B. on the CEs the gridmapdir is shared, so maintenance probably needs to be delegated
4. If we remove a pool account mapping then you have to remove the link from any DNs to this mapping as well (look for DN filenames with only 1 hard link).
   

OK, that's it. We got there, though not without some anxious moments!

wms myproxy renewal wobbles

During our recent reconfiguration to SL5 we also re-wrote our user account generation script from perl to python. Well Graeme did actually. So now its very easy to understand and extend. A consequence of this was that we created a new directory in /home for each user to keep things neat and tidy. This necessitated the recreation of all home directories across the cluster. A task fraught with danger.

However, we managed it except that I blew away the glite user from the WMS in the process and the .certs and .globus certificates required to run the WMS. After replacing them everything worked fine or so I thought. Recently we received reports that the myproxy renewal was not working and as it transpired the /home/glite/.certs/hostkey.pem and /home/glite/.certs/hostcert.pem must be owned by the glite user and not root for the renewal process to work! One to watch!

SL5 migration and CPU deployment

As of last week have now migrated ScotGrid-Glasgow to SL5. This meant worker nodes, DPM's and batch system all becoming SL5 in one big flurry of activity. We started on Monday morning and came out of downtime on Wednesday evening with SAM tests passing. Since then we have been mopping up the remaining issues that cropped up along the way but more on that later.

So as of 16th September 1800 jobs slots running SL5 out of a total of 1912. The remaining 112 job slots have been held back as SL4 till December to allow those VO's with unpatched software kits or that are simply not ready to move to SL5 to run jobs.

Similar to RAL and other sites we have gone with separate CE's between SL4 and SL5 to allow for those VO's that cannot co-exist on the same CE. These CE's will very shortly be submitting to the same batch system using node requirements :SL4, :SL5 set from the CE as described by SouthGrid from their SL3 to SL4 migration. It does necessitate from job manager tweaking but it works. I may try and switch this to submit filter when I get the chance as job manager tweaking is never very robust.

Canna I no just use qsub?

Ah, the endless refrain.

Anytime a user with cluster experience is introduced to the gLite submission mechanism, some question of that order (although not always with a Scottish accent) is inevitable.

Pulling out my Human-Computer Interaction hat, I first came to the conclusion that, despite the occasional hints to the contrary, users are indeed Human. Hot on the heels of this realisation, a little bit of analysis of the gLite job submission and control tools indicated that, whilst very powerful, they work in a very different fashion to qsub.

It's not clear that qsub is in any sense a better iterface than the native command line tools, but it is clear that it is different.

The general idea was to resolve this difference by providing a different interface to grid job submission that was more familiar to users with existing experience of cluster computing. Wether it's going to be a better approch for a user without that experience is not clear; but it will make it simpler for users to use the Grid as an offload for a local cluster (i.e. use a cluster, when it's full, send the jobs to the Grid).

It turns out that the POSIX defintion of qsub isn't too far away, conceptually, from a Grid system, so all that was needed to act as an interface transalation layer was a relativly straightforward python script.

Rather than relay all the gory details here, let me direct you to the gqsub download page, with the manual.

For users on svr020, it's installed in the default path, so you can just use it. Note that to properly mirror the expected behaviour you probably want to make sure you run from within $CLUSTER_SHARED.

But to answer the original question: "Aye!"

who changed the job wrapper?

It was a long night yesterday as Graeme and I tried to fix our failing ops CE tests. It started on Monday night when SAM mysteriously started failing across all CE's at Glasgow and then Durham. The jobs appeared to run but just stayed at the running state until the WMS presumably killed them and eventually failed ops tests.

After investigation we noticed the 'cannot download .BrokerInfo from' error. A quick look on a node proved that it was owned in /tmp by another user rather than ops. A strace -f -p NNNN on the globus-url-copy command process showed the ops job was getting a permission denied when trying to create/copy the file. A look at past CE-sft-broker tests showed the a very clear difference, in fact there was a missing directory!

-rw-r--r-- 1 sgmops001 opssgm 3085 Aug 31 05:06 /tmp/https_3a_2f_2fwms208.cern.ch_3a9000_2fElSbIsNqd8SN69eCXPN1JA/.BrokerInfo


-rw-r--r-- 1 sgmops001 opssgm 2312 Sep 1 22:34 /tmp/.BrokerInfo

Removing this file allowed the ops test to run but why it was happening was still a mystery. A work around we have deployed is to create an additional directory in cp_1.sh i.e.

# Workaround for gLite WMS jobs, which don't cd into EDG_WL_SCRATCH...
echo In cp_1.sh
echo Making temporary work directory
templ=$TMPDIR/glite_run_XXXXXXXX
temp=$(mktemp -d $templ)
echo Changing work directory to $temp
cd $temp

In the end we had to remove every blocking .BrokerInfo file from /tmp across the cluster and ops jobs started passing again. Further digging showed that the job wrapper has changed somewhere along the line. The old job wrapper had code like this in it.

#if [ ${__job_type} -eq 0 -o ${__job_type} -eq 3 ]; then # normal or interactive
 newdir="${__jobid_to_filename}"
 mkdir ${newdir}
 cd ${newdir}
#elif [ ${__job_type} -eq 1 -o ${__job_type} -eq 2 ]; then # MPI (LSF or PBS)
#fi

This has now been removed and could be causing issues for other sites. Torque and SGE have functionality to ring-fence every job perhaps we would have been safer using it but running jobs from /tmp worked for 3 years. Not any more it would seem.

multiple WMS yaim problems

I was alerted today by our all new shiny jabber chatroom that we were publishing the same WMS via lcg-infosites. A quick check and there it was....

-bash-3.00$ lcg-infosites --vo camont wms
https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server
https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server
https://lcgwms03.gridpp.rl.ac.uk:7443/glite_wms_wmproxy_server
https://wms00.hep.ph.ic.ac.uk:7443/glite_wms_wmproxy_server

We recently moved to the latest WMS 3.1 release last week and I thought it may have been down to that. Upon further inspection I found the following GIP plugin:

svr023:/opt/glite/etc/gip/provider# ./glite-info-provider-service-wmproxy-wrapper

which was publishing the wrong WMS.

GlueServiceEndpoint: https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

In the file the problem was an obvious one:

export WMPROXY_HOST=svr022.gla.scotgrid.ac.uk svr023.gla.scotgrid.ac.uk

This begs the questios, can YAIM deal with more than one WMS and if so how do you specify them. We had always gone for a quoted, space separated list in site-info.def
i.e.

WMS_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"

but perhaps you can't do that any more and you need to override the WMS_HOST in a node specific way. Oh well.

Database backups, and lock time

Running a service creates data. Running a service for a long time creates lots of data.

In this case, the WMS and LB servers - we're sitting with about 18GB on each LB. This is not a problem - they're well indexed against the usual queries (out of the box, no fiddling required), so the old data isn't really noticed.

Until you take a backup.

Then, in order to get a consistant backup, it's locked for however long it takes to dump all that data. Which is about 45 minutes.

That's too long - it means we have some time when it's not available, it's getting noticed. So, how can we take a backup, without locking the database for so long?

There's various options for that, but the best looking (read: simplest) one is to enable binary logging in MySQL. Because the tables used are all InnoDB, which is transactional, this means that the backup can mark a position in the log, and then use that to _not_ backup operations that came after it - which results in a consistant backup. (If your using any MyISM tables, which are not transactional, you can't do this. Hence the use of LVM snapshoting or other exotic techniques).

This it really simple: in the my.cnf for each service, put 'log-bin' (without the quotes) in the [mysqld] section, and restart.

Binary logging is now enabled.

Next, to take a lock free [0] dump, add the --single-transaction flag to mysqldump.

The time taken to actually dump the data to disk won't change, but the database won't be locked for that time.

I did this for one of our LB servers, and then, while the dump was running, submited a job through the WMS. The job was assigned to the LB I was dumping, proving it can be written to, and has now completed, while the dump hasn't yet finished.

I've modified our usual backup script, so that if it detects the presence of /var/lib/mysql/${hostname}-bin.index, which is the index for the binary log, it automatically uses --single-transaction. That way, we still have a single backup script, but it does it the best way possible.

There are a couple of downsides to binary logging: It means the DB has to write more data to disk, so is about 1% slower. As the services are not running at 99% of the cpu, that's ok for us. It also means that each new piece of data is stored twice - once in the DB, and once in the log. Therefore the data storage need grows twice as fast - faster, if there are deletes to the database. I'm looking at an 18GB database - so this won't be a problem. Also, you can purge old logs, so I don't feel that this is a problem any more than the risk of the database expanding over the partition size is.

One thing I'll be looking at is useing the binary logs to take an incremental backup. That'll still not lock the database, but will also be much smaller and faster to take. That's a bit more complicated to arrange, so it'll go into the pile of 'ideas that look nice, but we don't think we need it yet'

As an aside, I think this has to go down as one of the more anticlimatic updates - it was simple, quick and just worked. Unless disk space is very tight, I can't see why one wouldn't enable it.

[0] Technically, it takes a lock, waits for all pending transactions to complete, marks the log position, then releases it. If you have slow operations in flight, it locks it for the duration of that operation.

getting ngs.ac.uk voms to work

I have been looking into an issue with the NGS as they are testing submission to the WMS. A ticket was raised as authentication failed on both our production CE's.

This was recreated with by created an ngs voms proxy.

-bash-3.00$ voms-proxy-init -voms ngs.ac.uk --valid 240:00
Cannot find file or dir: /clusterhome/home/gla057/.glite/vomses
Enter GRID pass phrase:
Your identity: /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=douglas mcnab
Creating temporary proxy ................................................................ Done
Contacting voms.ngs.ac.uk:15010 [/C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/Email=support@grid-support.ac.uk] "ngs.ac.uk" Done

Warning: voms.ngs.ac.uk:15010: The validity of this VOMS AC in your proxy is shortened to 86400 seconds!

Creating proxy ............................................................................ Done
Your proxy is valid until Thu Aug 20 15:34:41 2009


Then with a direct globus-job-run:

-bash-3.00$ globus-job-run svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
GRAM Job submission failed because authentication with the remote server failed (error code 7)
-bash-3.00$ globus-job-run svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
GRAM Job submission failed because data transfer to the server failed (error code 10)


After much investigation, the long and short of it is that even with the correct entries in the groupmapfile and grid-mapfile the issue still occurred. So I checked the VO certificate in /etc/grid-security/vomsdir. This was fine, although there was also the /etc/grid-security/vomsdir/ngs.ac.uk/voms.ngs.ac.uk.lsc which may have been getting used before the VO certificate. So to check I removed the /etc/grid-security/vomsdir/ngs.ac.uk/voms.ngs.ac.uk.lsc

Hey presto, submission worked:

-bash-3.00$ globus-job-run svr026.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
node295.beowulf.cluster
-bash-3.00$ globus-job-run svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
node295.beowulf.cluster


So I think there may be an issue with ngs.ac.uk VO and the lsc file which looked correct.


svr026:/etc/grid-security/vomsdir/ngs.ac.uk# cat voms.ngs.ac.uk.lsc
/C=UK/O=eScience/OU=Manchester/L=MC/CN=voms.ngs.ac.uk/Email=support@grid-support.ac.uk
/C=UK/O=eScienceCA/OU=Authority/CN=CA


This will be an issue in the future on SL5 when VO certificates are deprecated for the lsc file.

the sl5 cluster grows

With a view to a full scale migration of Glasgow's worker nodes from sl4 to sl5 in September we have grown the size of our sl5 test cluster from 8 job slots to 112 job slots.

This is accessible for submission to the following queues:

dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q6h
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q1d
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q2d
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q3d

Currently the CE is only advertising and accepting submissions from Atlas and queues are open to sgm/prd/pil account but I am more than welcome to open them to anyone who wishes to test. Just drop me a line and I will create a test software area for any sgm account to install the application software via SL5 and allow access on the CE & Batch Sys for running the jobs.

So far things have been positive for Atlas with software kits now installing on SL5 and attempting to run kit validation. Currently we are failing KV tests, more precisely it failed in the digitization phase, so we then failed the reconstruction.

Nightly builds continue to be run so slowly but surely I'm sure these issues will be ironed out.

sl5 workers murmurings

Well we have had an sl5 test cluster for a while now but it has ever really seen any action other than the odd random hello world job from testing with cream and the like.
However, as the great sl5 debate raged on we put ourself forward as a test site for atlas along with Oxford, another site with an sl5 cluster for installs of SL4 and future SL5 versions of the atlas software.

However, in order to get my development CE visible to the real world I had to add it into our site bdii. This then attracted ops and dteam jobs as by default they were allowed through the CE. No great shakes and was actually good as it identified problems that had not been seen with simple hello world jobs from within ScotGrid.

The first mishap was a networking issue where the jobs could arrive but couldn't get their job wrapper and payload as most of our workers are NAT'd. Except my development one. A simple fix once we worked out what was wrong.

Two other problems were encountered. Firstly that CE-sft-lcg-rm-free test went into a warn state as the glite-WN package no longer pulls in ldapsearch. This is fixed by installing openldap-clients from sl-base.

Secondly, the many of the jobs that actually did run through the system encountered an error on CE-sft-brokerinfo with something like: error while loading shared libraries: libclassad_ns.so.0: cannot open shared object file: No such file or directory
After some googling, this bug is known about and has been fixed. The fix is adding gridpath_prepend "LD_LIBRARY_PATH" "/opt/classads/lib64/" to /etc/profile.d/grid-env.sh However, at Glasgow we control grid-env.sh though cfengine so I needed to make the appropriate change there too.

After going through this over the last few days I stumbled across Ewan's page as he had encountered the exact same issues. So take heed and do a spot of googling first!

There is also a metapackage available for Sl5 glite3.2 WN's this should hopefully contain all the required dependencies. This is located here. The gotcha with this is that you have to install it with yum localinstall or stick it in a yum repo as rpm -i doesn't work.

I have also just compared what is installed from this against the Atlas SL5 page and there were 4 packages missing: compat-gcc-34-g77, compat-libgcc-296, compat-libstdc++-296, ghostscript-8.15.2

So currently we have ops/dteam jobs running and passing. Atlas software jobs running, completing but not successfully working. More digging is required and I will keep you posted.

Rest in Peace gLite 3.0 ... finally

Today heralds a poignant day for the members of ScotGrid as we finally waved goodbye to the last gLite 3.0 Service (VOMS) and SL3.0 server in our cluster. The sombre mood was only broken by the arrival of the newborn gLite 3.1 VOMS server running on SL4. There was much flag waving and tears of joy as the first voms-proxy-init was issued and the shiny new web interface marvelled at.

Again Jpackage caused a little confusion as tomcat5 pulls jdk6 unless you exclude it or force an install of jdk5. This is preferred for all you firefox users out there. As if tomcat is running under jdk6 you have to remember to turn off TLS1.0 from the preferences menu in order to get the SSL handshaking to work or you get a nice fat error page! Not very useful for an admin screen let me tell you.

This upgrade was tried last year but was hampered by a lack of database migration scripts. This time around and with the help of these instructions it went swimmingly.

So although it was a sad day for gLite 3.0 and SL3 camp and a small victory for gLite 3.1/SL4, the war is not over. With gLite 3.2 and SL5 closing in on all fronts the battle is only just beginning.

p.s. we have an SL5 set-up so if you want to test, please let me know.

Deflected Cosmic Rays...

This is the second short "when you're good..." post. During the RAL machine room move, we tested distributing ATLAS cosmics AOD and DPD data from CERN->GLASGOW->UK T2s. After some tweaking of the T2 FTS channels at CERN and tinkering in DDM this has worked a charm. Data distrubution in the UK has gone very well throughout the current combined cosmics data taking runs.

This is the first time that we tried circumventing the T1 for such an organised data distribution and it was a real success for the UK, ATLAS and Glasgow.

When you're good, you're Glasgow...

There hasn't been much time to write in the blog recently, STEP09 madness and all. However, it is wonderful to see that Glasgow was the top ATLAS T2 for analysis during the STEP09 challenges. We analysed more than 1.8B events, mostly through panda, with a 98% success rate.

We also took the largest fraction of data of any UK T2, 40%, and succeeded in getting all the data we were sent (we had little anxiety on the final weekend and want to increase our network heardroom for sure).

Sam and I wrote a full report on our experiences and how we used the opportunity to really probe the limits of the current cluster.

For the future, we really have to worry about how to maintain the i/o rate into the CPUs as the number of cores rises.

Installing (and fixing) a gLite Tar UI on SL5

First, a little background.

The UI machine is the gLite term for the machine from which you submit jobs (and monitor, receive output etc). This is analogous to the submit machine in Condor, and the head node for a local cluster - except that with the Grid, there is no reason that you can't submit on one UI, monitor from another and collect output on a third. No reason - except perhaps for keeping one's sanity.

Whilst most of the Grid servers are normally dedicated machines, occasionally given over to more than one Grid task, but only doing Grid tasks, the UI is a clear contender for being placed on machine that already have another purpose. In this instance, we have a group of users that have their own cluster, and occasionally off load some computations onto the Grid. It would be ideal if they could submit to either their local cluster or the Grid from the same machine. Cluster head nodes aren't too portable, so the obvious approach is to turn their existing head node into a gLite UI.

Fortunately, the gLite developers forsaw this possibility, and the UI package is available in a single blob that can be installed for an individual user. So that's what I've done - but there's a few caveats, and a couple of bugs to work around.

The tar UI I used was the gLite 3.2.1 production release. This is still early in the 3.2 life cycle, and not all services are available at 3.2, so there might be a few teething issues here, interacting with the older services. At Glasgow we don't have any 3.0 services, which is good, as they're really unsupported.

On to the install: Download the two tarballs, and unpack into a directory (why 2 tarballs, one tarball aught to be enough for anyone). I then promptly fell of the end of the documentation - which assumes that you already know a lot about gLite.

What you have to do it produce a file (the site-info.def) that gives some high level details of what the UI needs to know to work. This file can be created anywhere (I put it in the same directory I unpacked the tarballs into), as you always gives it's path to yaim, the tool that uses it.

The first thing you need to put in is the 4 paths listed on the wiki page. Then you need a few other things:

BDII_HOST=svr019.gla.scotgrid.ac.uk
MON_HOST=svr019.gla.scotgrid.ac.uk
PX_HOST=lcgrbp01.gridpp.rl.ac.uk
WMS_HOST="svr022.gla.scotgrid.ac.uk svr023.gla.scotgrid.ac.uk"
RB_HOST=$WMS_HOST

The BDII host is where the UI gets it's information from - this should be a 'top level' BDII, not a site BDII. None of have the faintest clue why it needs the MON host - that's something I'll dig into later. The PX host is the MyProxy server to use by default. That one should be good for anywhere in the UK. The WMS host is the replacement for the deprecated (but still needed) RB hosts, and points to the WMS to be used for submission (by default).

One thing I found I needed that wasn't documented was a SITE_NAME. I just put the hostname in there - it doesn't appear to be used, but yaim complains if it's not there.

The last thing needed is a list of the VO's to be supported on the UI. When deploying a tar UI this will normally be a very small list - one or two I would expect. Therefore I choose to place them inline. There is a mechanism to put the VO specification in a separate directory, which is used for shared UI machines.

VOS="vo.scotgrid.ac.uk"

VO_VO_SCOTGRID_AC_UK_VOMS_SERVERS="vomss://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk"
VO_VO_SCOTGRID_AC_UK_VOMSES="'vo.scotgrid.ac.uk svr029.gla.scotgrid.ac.uk 15000 /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk vo.scotgrid.ac.uk'"
VO_VO_SCOTGRID_AC_UK_VOMS_CA_DN="'/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA'"

VO specification is in two parts - first we have to list the VO's (space separated list), and then , for each VO, give the VOMS server that defines the membership of the VO, and the certificate DN for the VOMS server. Note that the vo name gets translated to UPPER CASE and all the dots in it become underscores (a fact that's somewhat underdocumented, and results in a complaint about a syntactically invalid site-info.def, and no other message ...)

Once that's all in place, it's time to run yaim to configure things (from the dir I unpacked into):

./glite/yaim/bin/yaim -c -s site-info.def -n UI_TAR

Slight problem with installing certificates: By default these go into /etc/grid-security/certificates, but I'm not running as root. As a local user (for the initial testing), I need to tell yaim where to put them instead. In the site-info.def:

X509_CERT_DIR=${INSTALL_ROOT}/certificates

and make that directory, and re-run the yaim command. Chuntering along for a bit, and then finished with no errors - I did get a couple of warnings, but nothing that looked like a problem in this case.

Last step - testing. First, load up the installed software:

$GLITE_EXTERNAL_ROOT/etc/profile.d/grid-env.sh

and install my certificate on there.

lcg-infosites ... works
voms-proxy-* ... works
glite-wms-job-submit ... Boom!

glite-wms-job-submit: error while loading shared libraries: libboost_filesystem.so.2: wrong ELF class: ELFCLASS32

Hrm. Looks like a 32/64 bit problem. Some pokage later, and it turns out that the shell setup script supplied points only to the $GLITE_EXTERNAL_ROOT/usr/lib directory - and not the lib64, containing the needed libs. A quick hack onto the grid-env.sh, and that's rectified. Now:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl
glite-wms-job-submit: error while loading shared libraries: libicui18n.so.36: cannot open shared object file: No such file or directory

This turns out to be the International Components for Unicode (at least, I think so). The particularly interesting point about this is that the only references I can find to these libraries on SL include one from this very blog and they are all about Adobe Acrobat Reader... because that's the most common software that uses it.

I grabbed the RPM from http://linux1.fnal.gov/linux/scientific/5x/x86_64/SL/, and added it to $GLITE_EXTERNAL_ROOT/usr/lib64 by:

cd $GLITE_EXTERNAL_ROOT
rpm2cpio libicu-3.6-5.11.2.x86_64.rpm | cpio -i

And, finally:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

==========================================================================

Jobs submitted from a pure 64 bit SL5 system. Note that the separator character has changed, from being a line of '*' to a line of '='.

After chuntering away for a while,

[scotgrid@golem ~]$ glite-wms-job-output https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

Error - Operation Failed
Unable to retrieve the output

[scotgrid@golem ~]$ cd /tmp/jobOutput/
[scotgrid@golem jobOutput]$ ls
scotgrid_VW96yirZ4gG6jVXjx9UwBg

Which is a known problem - the UI reports that collecting job output failed, but it does succeed. If you don't need to get the OutputSandbox from the job (e.g. it's all written to an SE), then this isn't a problem.

Now to post some bug reports on the tar UI package... (Update: This is now bug number 52825 for the configuration and bug 52832 for the ICU package)

NGS and EGEE Software Tags

After I reinstalled svr021 (CE) we lost some good work carried out by Andy Elwell to allow NGS software tags to be published along side glite ones through the BDII. I found the original post and re-installed the patch.

Wouldn't it be nice if this was available directly from glite well it is now!

The new script will correctly report information from the ngs-uee-gip-plugin plug-in that the NGS sites use to report applications installed under /usr/ngs.

Jason created the patch, I tested and Laurence Field at CERN has kindly merged the changes into gLite proper and has made an updated RPM available from...

http://etics-repository.cern.ch:8080/repository/download/registered/org.glite/glite-info-generic/2.0.2/noarch/glite-info-generic-2.0.2-5.noarch.rpm

so if you are an NGS affiliate site who runs a glite stack, the rpm above will allow both sets of software tags to be advertised through your BDII. If you try to install ngs-uee-gip-plugin without the new version of glite-info-generic your ngs tags will replace your glite software tags or vice-versa. Now they happily merge rather than replace.

MPI really kicks off at Glasgow

It seems there is a real appetite for MPI codes on ScotGrid at the moment. First off there was Optics running Lumerical's FDTD FDTD and now we have UKQCD running Chroma. Next up is an MPICH install of CASTEP for Solid State Physics. So nearly 2 years after it was first enabled at ScotGrid it is finally seeing it's first tour of duty. Better late than never. We still have some kinks to iron out such as better scheduling of MPI on our cluster but early results from benchmarking are promising even for an ethernet based MPI solution!

Bright and creamy MPI

So, as of the last time MPI was mentioned, it was working. Well, it looks like it wasn't getting much use, because over the year or so, it seems to have fallen into disrepair.

We'd ended up with MPIexec not being installed on the worker nodes, which was blocking the setup of the processor nodes. This even prevented a single process MPI job from running, because that still used MPIexec. In the end, this particular problem was resolved by installing it again (after some careful ramp up to make sure it didn't knock anything else off).

The phrasing of that last sentence is deliberately precise: it turned out that there was another problem lurking in the swamp water that is middleware. In order to test the install of MPIexec, I grabbed a worker node that was out of production for the HEP-SPEC benchmarking. This had, of course, got a new install of the worker node packages, in order to give a consistent platform with other sites.

Experienced Grid hands might just be able to predict what comes next...

After installing MPIexec on that node, and then restricting that node to just our test VO (Maui is awesome for this sort of tweak), we noticed that it wasn't accepting any jobs. Specificially, jobs were arriving, but failing immediatly. Cue finger pointing at MPIexec, and removal of it.

Didn't help.

In the end, Mike resolved this one: An incompatabiliy between the Torque server, and the torque clients with the new Worker node package. Once that was resolved, MPIexec back on, and it was all working fine. Roll out across the cluster, finger crossing and no problems: MPI back in business.

The next step was to actually run MPI jobs - took a couple of attempts with mpi-start, but got there. One problem we have is that the WMS will not send MPI jobs to a site that declares that it is 'torque'. It will only send jobs to sites that declare that declare the LRMS to be 'pbs' or 'lsf'. Given that torque identical to PBS (and more common!), that's a bit silly. This is a known bug, that's been open for 4 years, with a patch available, this is a bit rediculous.

There is a work around, where you can tell the WMS to use a specific LRMS, but you have to also specifiy the target CE - which kind of defeats much of the point of the WMS...

Fortunatly, using the CREAM CE sidesteps most of these issues. Alas, the latest WMS package doesn't work properly with CREAM CE's, so we had to mark our CREAM CE to be 'Special', not 'Production' (effectivly disableing WMS submission to CREAM). Not too big a problem, as we can do direct submission to the CREAM CE for our specific use case, but it's not great in the long term.

Our specific use case is the Lumerical FDTD package, which is installed and working at Glasgow, and has been used by end users. There's some trickyness involed in this, as we're not passing in source code, as mpi-start expects, so I'll write up a bit more how it all fits together at some point.

There might be some Maui fiddling in the imminent future, to assist it to pack MPI jobs on to as few physical machines as possible. The key point is that MPI has been used by end users at Glasgow, which bodes well.