more openmpi tweaking

Whilst testing MPI on our cluster and get it into a usable state I uncovered a rather nasty bug with openmpi-1.3.4. This manifested itself with never being able to run on the node with cores > 4. It was a weird one as openmpi communication over two nodes worked fine with 8 cores on each node but when a job requested cores > 4 on the same node. The job just hung. An strace of the mpiexec process suggested some sort of TIMEOUT/WAIT issue.

On the release note for openmpi-1.4.1 it appears they discovered this bug and provided a fix:
- Fix a shared memory "hang" problem that occurred on x86/x86_64
platforms when used with the GNU >=4.4.x compiler series.

This sounded plausible and in fact an upgrade has fixed the issue.

So now with all 8 cores running on the same node the next issue to arise was one related to Maui. Some time when you requested nodes=8, Maui scheduled the job on 3 cores, a qdel and a resubmission later Maui rescheduled the job onto 5 cores. On one test I even qrun'd the job and it appeared it start on the correct number of nodes but there appeared to be no reason for Maui not getting this correct. So it was time to get out Maui docs.

from the docs;
Maui is by default very liberal in its interpretation of :PPN=. In its standard configuration, Maui interprets this as 'give the job * tasks with AT LEAST tasks per node'. Set the JOBNODEMATCHPOLICY parameter to EXACTNODE to have Maui support PBS's default allocation behavior of nodes with exactly tasks per node.

This seemed to suggest that Maui's default behaviour is to pack a job into as few nodes as possible. So I tried out setting the JOBNODEMATCHPOLICY to EXACTNODE and this seems to have done the trick.

nodes=24 means 24 nodes, not 8, not 6 but 24

This does have a drawback in that it will be 24 separate nodes. This setting relies upon being able to set :ppn (processes per node) to allow nodes=3:ppn=8 giving 24 cores which is really what you want to say. As you probably have a fast machine with loads of memory and cores. Therefore, you could target all the cores rather than 24 nodes. However, it is a start.

Wouldn't it be nice if you could specify :ppn in JDL. The only way round this I can see for now is to manually change the job manager or use the local batch attributes of CREAM to allow a custom cerequirement to be specified. Possible but not nice.

openmpi magic

I have just rebuilt openmpi-1.3.4 for use with CASTEP. This is built to a useful /opt location with Torque, F90 support for gfortran44.

gLite support for OPENMPI is fairly generic and means any openmpi rpm install does not have any useful batch or interconnect support. So anything out of the ordinary requires a custom build.

I will stick the RPM available for download here in the next few days.

The magic for building from a src rpm:

cd /usr/src/redhat/SPECS/
rpmbuild -ba --define '_prefix /opt/openmpi-1.3.4' --define '_mandir %{_prefix}/share/man'
--define 'configure_options --prefix=/opt/openmpi-1.3.4 --with-tm=/usr/
FC=gfortran44 F77=gfortran44 CC=gcc44 CXX=g++44 FFLAGS=-O2 FCFLAGS=-O2 CFLAGS=-O2 CXXFLAGS=-O2'
openmpi-1.3.4.spec

Full Instructions are here.

issues with gfortran43/44 and mpich

I am finally getting to the bottom of what has been going wrong with re-compiling MPICH for F90/F95 (required for CASTEP - a demanding Fortran code). I have now narrowed it down to one issue between recompiling MPICH with gfortran43/44 for SL5 usage with CASTEP instead of plain old gfortran.

FC="gfortran44" ; export FC;
F90="gfortran44" ; export F90;
...
--enable-f90modules

The SAM MPICH test runs after recompiling with F90 support using gfortran and in fact it works fine on SL4 and SL5. So that was not the issue.

message size            transfertime            bandwidth
32 bytes                0.000000 sec            inf MB/s
2048 bytes              0.000117 sec            17.476267 MB/s
131072 bytes            0.001445 sec            90.687654 MB/s
8388608 bytes           0.078437 sec            106.946397 

It turns out that MPICH just doesn't work when compiled with gfortran43/44. Leaving me in a bit of an pickle as CASTEP will not compile on SL5 with gfortran, you have to use gfortran43/44!

Time for the backup plan ..... openmpi.

CASTEP, A Test of True Grid

Along came another users with a requirement for MPI. Can we run it? Well yes you can but remember our interconnects are just plain old Ethernet and nothing fancy like Myrinet or Infiniband. We are not a HPC cluster but an HTC cluster.

So we have been building CASTEP, an f90 code, heavy on the MPI scatter/gather. A test of true grid for any HTC cluster. First off CASTEP requires a minimum of make3.81 and gfortran43. Handy that we moved to SL5 as these are now the standard. Coupled with making sure that the required libs fftw3, blas and lapack are all built with the same compiler, gfortran43. This allowed the single core version to be built and installed onto the grid.

An MPI version is turning out be a bit more work. First off the old, outdated and no longer developed libs MPICH have not been built with .f90 support enabled by default. So we have got hold of the source to do a recompile with .f90 support on for gfortran43. There also appeared to be a bug in the gfortran support. So we had to patch the src rpm to include a patch that we located online. This allowed us to finally build the mpich lib. This has been tested with compilation of an MPI job in c and f90, both of which run successfully.

Unfortunately CASTEP still doesn't run using it so more digging required.

MPI really kicks off at Glasgow

It seems there is a real appetite for MPI codes on ScotGrid at the moment. First off there was Optics running Lumerical's FDTD FDTD and now we have UKQCD running Chroma. Next up is an MPICH install of CASTEP for Solid State Physics. So nearly 2 years after it was first enabled at ScotGrid it is finally seeing it's first tour of duty. Better late than never. We still have some kinks to iron out such as better scheduling of MPI on our cluster but early results from benchmarking are promising even for an ethernet based MPI solution!

Bright and creamy MPI

So, as of the last time MPI was mentioned, it was working. Well, it looks like it wasn't getting much use, because over the year or so, it seems to have fallen into disrepair.

We'd ended up with MPIexec not being installed on the worker nodes, which was blocking the setup of the processor nodes. This even prevented a single process MPI job from running, because that still used MPIexec. In the end, this particular problem was resolved by installing it again (after some careful ramp up to make sure it didn't knock anything else off).

The phrasing of that last sentence is deliberately precise: it turned out that there was another problem lurking in the swamp water that is middleware. In order to test the install of MPIexec, I grabbed a worker node that was out of production for the HEP-SPEC benchmarking. This had, of course, got a new install of the worker node packages, in order to give a consistent platform with other sites.

Experienced Grid hands might just be able to predict what comes next...

After installing MPIexec on that node, and then restricting that node to just our test VO (Maui is awesome for this sort of tweak), we noticed that it wasn't accepting any jobs. Specificially, jobs were arriving, but failing immediatly. Cue finger pointing at MPIexec, and removal of it.

Didn't help.

In the end, Mike resolved this one: An incompatabiliy between the Torque server, and the torque clients with the new Worker node package. Once that was resolved, MPIexec back on, and it was all working fine. Roll out across the cluster, finger crossing and no problems: MPI back in business.

The next step was to actually run MPI jobs - took a couple of attempts with mpi-start, but got there. One problem we have is that the WMS will not send MPI jobs to a site that declares that it is 'torque'. It will only send jobs to sites that declare that declare the LRMS to be 'pbs' or 'lsf'. Given that torque identical to PBS (and more common!), that's a bit silly. This is a known bug, that's been open for 4 years, with a patch available, this is a bit rediculous.

There is a work around, where you can tell the WMS to use a specific LRMS, but you have to also specifiy the target CE - which kind of defeats much of the point of the WMS...

Fortunatly, using the CREAM CE sidesteps most of these issues. Alas, the latest WMS package doesn't work properly with CREAM CE's, so we had to mark our CREAM CE to be 'Special', not 'Production' (effectivly disableing WMS submission to CREAM). Not too big a problem, as we can do direct submission to the CREAM CE for our specific use case, but it's not great in the long term.

Our specific use case is the Lumerical FDTD package, which is installed and working at Glasgow, and has been used by end users. There's some trickyness involed in this, as we're not passing in source code, as mpi-start expects, so I'll write up a bit more how it all fits together at some point.

There might be some Maui fiddling in the imminent future, to assist it to pack MPI jobs on to as few physical machines as possible. The key point is that MPI has been used by end users at Glasgow, which bodes well.

MPI Running Properly at Glasgow

I was very keen to take up Stephen Child's offer to get MPI enabled
properly at Glasgow. (See link for my last attempts at this working,
where I cobbled together something that was far from satisfactory, but
at least proved it was possible in theory to get this to work.)

A problem for MPI at Glasgow is that our pool account home directories
are not shared and that jobs all wake up in /tmp anyway. For local
users we offer the /cluster/share area, which gets around this, but
what to do for generic MPI jobs? We decided it would be a very good
idea to offer some shared area for MPI jobs, and that the right
strategy would be to modify mpi-start to pick the job up by its
bootstraps and drop itself back down into a shared directory area in
/cluster/share/mpi. To do this we decided to generalise the
MPI_SHARED_HOME environment variable. Previously this had been "yes"
or "no", but in the new scheme if it points to a directory then the
script transplants the job to an appropriate subdirectory of this
area.

On the site side I had to make site all the MPI environment variables
were properly defined in the job's environment (which we do with
/etc/profile.d/mpi.sh) and advertise the right MPI attributes in the
information system.

It all went pretty well, until we had an issue with mpiexec not being
able to invoke the other job threads properly. (Mpiexec starts the
other job threads via torque, which means they get accounted for
properly and that we can disable passwordless ssh between the WNs
- which we did at the SL4 upgrade). There was a fear it was due to some
weird torque build problem, but in the end it was a simple issue with
server_name not being properly defined on the workers. A quick but of
cfengine and this was then fixed.

So, Glasgow now supports MPI jobs - excellent. (He will rebuild the
newly featured mpi-start and release it next week.)

Big thanks to Stephen for setting this up for us.

Success with MPI

Cracked it! I can now get MPI jobs running on the Glasgow cluster.

First thing to note is that the gatekeeper does not invoke mpirun for the job - this is very good, because it would be almost impossible to get this to work if it did.

The key file is the NODELIST file which the CE will generate and add to the executable's argument list. When given as the argument of the -p4pg option then mpirun will ssh to all of the "slave" nodes and start the binary which is given in the NODELIST file.

By default this breaks for 2 reasons:

1. The gatekeeper only copies the job's sandbox into the working directory of the "master" worker node. So on the "slave" nodes the executable isn't present. (N.B. Even though we have a shared data area for our glaNNN accounts, the working directory is always in /tmp and local to the worker node.)
   
2. The executable listed really needs to be a wrapper script, so it's the wrong thing for mpirun to be starting anyway.

So, wrapper script really has to do the following:

1. Change to a more sensible shared directory (like $CLUSTER_SHARED).
2. Rewrite the NODELIST file so that the name of the correct mpi binary to run is given, instead of the wrapper script itself.
3. Invoke mpirun, giving the new NODELIST file.

Here's an example (with a lot of debugging hooks) which works, running the code:

#! /bin/sh
#
# Argument list is: BINARY -p4pg NODELIST -p4wd PATH
# What's really important for us is the NODELIST file, i.e., $3
cd $CLUSTER_SHARED/mpi
export MYBIN=$1
PGFILE=`pwd`/pgfile.`hostname -s`.$$
echo My Args: $@
echo "----"
echo "Original NODELIST file:
cat $3
echo "----"
cat $3 | perl -ne 'print "$1 $2 /cluster/share/gla012/mpi/$ENV{\"MYBIN\"}\n" if /^([\w\.]+)\s+(\d+)/;' > $PGFILE
echo "----"
echo "New NODELIST file:
cat $PGFILE
echo "----"
/opt/mpich-1.2.7p1/bin/mpirun $MYBIN -p4pg $PGFILE

(Hmm, it's splitting that perl one liner in a really nasty way - no line breaks there.)

There are, however, two problems which I can see.

1. Accounting. Looking at the torque logs it's clear that only the master node's process is being accounted for. The slave node MPI processes are not accounted for. Do we multiply the master node's CPU and Wall by the node number as an interim measure?
2. Orphaned and stray processes. As ssh is used to start the binary on the slave nodes, what happens if the code leaves them behind or they run away?

I wonder if there's a way we can modify mpirun to do things in a torque friendly way? I shall enquire of the MPI gurus.

(For more formal documentation, watch this wiki page....)

MPI Progress

I am making progress with MPI jobs. I can now get MPICH jobs into the batch system via edg-job-submit and they do get a batch system reservation.

It turned out I had to add MPICH as a GlueHostApplicationSoftwareRunTimeEnvironment in the information system. It's also essential to have GlueCEInfoLRMSType as pbs. It doesn't work if you put torque (it must be the only thing on the grid that actually cares!).

The job wrapper then adds some interesting arguments to the executable:

-p4pg NODELIST -p4wd PATH

Where NODELIST looks like this:

node067.beowulf.cluster 0 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node070 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node102 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node139 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh

and PATH is just the working directory for the job. Note the magic number "0" seems to be the place where the job executable runs and "1" are all the nodes where other job slots are reserved for this job.

So clearly the NODELIST file then needs to be taken by mpirun and used to start all the mpi subprocesses. From the EGEE MPI Wiki, the standard method seems to like to use the i2g mpi-start command, so the arguments must be in a form appropriate for it. Open questions remain, though:

1. How to get i2g mpi-start to work. When I give it an MPI binary it seems determined to compile it - however this falls over, even though MPICH 1.2.7 is in the path.
2. How do I ignore mpi-start and run a pre-prepared MPI binary, which will be what a Glasgow use wants to do.
3. How on earth will torque account for all of this properly?

Further reading: EGEE-II-MPI-WG-TEC.doc.