CASTEP, A Test of True Grid

Along came another users with a requirement for MPI. Can we run it? Well yes you can but remember our interconnects are just plain old Ethernet and nothing fancy like Myrinet or Infiniband. We are not a HPC cluster but an HTC cluster.

So we have been building CASTEP, an f90 code, heavy on the MPI scatter/gather. A test of true grid for any HTC cluster. First off CASTEP requires a minimum of make3.81 and gfortran43. Handy that we moved to SL5 as these are now the standard. Coupled with making sure that the required libs fftw3, blas and lapack are all built with the same compiler, gfortran43. This allowed the single core version to be built and installed onto the grid.

An MPI version is turning out be a bit more work. First off the old, outdated and no longer developed libs MPICH have not been built with .f90 support enabled by default. So we have got hold of the source to do a recompile with .f90 support on for gfortran43. There also appeared to be a bug in the gfortran support. So we had to patch the src rpm to include a patch that we located online. This allowed us to finally build the mpich lib. This has been tested with compilation of an MPI job in c and f90, both of which run successfully.

Unfortunately CASTEP still doesn't run using it so more digging required.

the sl5 cluster grows

With a view to a full scale migration of Glasgow's worker nodes from sl4 to sl5 in September we have grown the size of our sl5 test cluster from 8 job slots to 112 job slots.

This is accessible for submission to the following queues:

dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q6h
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q1d
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q2d
dev010.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q3d

Currently the CE is only advertising and accepting submissions from Atlas and queues are open to sgm/prd/pil account but I am more than welcome to open them to anyone who wishes to test. Just drop me a line and I will create a test software area for any sgm account to install the application software via SL5 and allow access on the CE & Batch Sys for running the jobs.

So far things have been positive for Atlas with software kits now installing on SL5 and attempting to run kit validation. Currently we are failing KV tests, more precisely it failed in the digitization phase, so we then failed the reconstruction.

Nightly builds continue to be run so slowly but surely I'm sure these issues will be ironed out.

sl5 workers murmurings

Well we have had an sl5 test cluster for a while now but it has ever really seen any action other than the odd random hello world job from testing with cream and the like.
However, as the great sl5 debate raged on we put ourself forward as a test site for atlas along with Oxford, another site with an sl5 cluster for installs of SL4 and future SL5 versions of the atlas software.

However, in order to get my development CE visible to the real world I had to add it into our site bdii. This then attracted ops and dteam jobs as by default they were allowed through the CE. No great shakes and was actually good as it identified problems that had not been seen with simple hello world jobs from within ScotGrid.

The first mishap was a networking issue where the jobs could arrive but couldn't get their job wrapper and payload as most of our workers are NAT'd. Except my development one. A simple fix once we worked out what was wrong.

Two other problems were encountered. Firstly that CE-sft-lcg-rm-free test went into a warn state as the glite-WN package no longer pulls in ldapsearch. This is fixed by installing openldap-clients from sl-base.

Secondly, the many of the jobs that actually did run through the system encountered an error on CE-sft-brokerinfo with something like: error while loading shared libraries: libclassad_ns.so.0: cannot open shared object file: No such file or directory
After some googling, this bug is known about and has been fixed. The fix is adding gridpath_prepend "LD_LIBRARY_PATH" "/opt/classads/lib64/" to /etc/profile.d/grid-env.sh However, at Glasgow we control grid-env.sh though cfengine so I needed to make the appropriate change there too.

After going through this over the last few days I stumbled across Ewan's page as he had encountered the exact same issues. So take heed and do a spot of googling first!

There is also a metapackage available for Sl5 glite3.2 WN's this should hopefully contain all the required dependencies. This is located here. The gotcha with this is that you have to install it with yum localinstall or stick it in a yum repo as rpm -i doesn't work.

I have also just compared what is installed from this against the Atlas SL5 page and there were 4 packages missing: compat-gcc-34-g77, compat-libgcc-296, compat-libstdc++-296, ghostscript-8.15.2

So currently we have ops/dteam jobs running and passing. Atlas software jobs running, completing but not successfully working. More digging is required and I will keep you posted.

Installing (and fixing) a gLite Tar UI on SL5

First, a little background.

The UI machine is the gLite term for the machine from which you submit jobs (and monitor, receive output etc). This is analogous to the submit machine in Condor, and the head node for a local cluster - except that with the Grid, there is no reason that you can't submit on one UI, monitor from another and collect output on a third. No reason - except perhaps for keeping one's sanity.

Whilst most of the Grid servers are normally dedicated machines, occasionally given over to more than one Grid task, but only doing Grid tasks, the UI is a clear contender for being placed on machine that already have another purpose. In this instance, we have a group of users that have their own cluster, and occasionally off load some computations onto the Grid. It would be ideal if they could submit to either their local cluster or the Grid from the same machine. Cluster head nodes aren't too portable, so the obvious approach is to turn their existing head node into a gLite UI.

Fortunately, the gLite developers forsaw this possibility, and the UI package is available in a single blob that can be installed for an individual user. So that's what I've done - but there's a few caveats, and a couple of bugs to work around.

The tar UI I used was the gLite 3.2.1 production release. This is still early in the 3.2 life cycle, and not all services are available at 3.2, so there might be a few teething issues here, interacting with the older services. At Glasgow we don't have any 3.0 services, which is good, as they're really unsupported.

On to the install: Download the two tarballs, and unpack into a directory (why 2 tarballs, one tarball aught to be enough for anyone). I then promptly fell of the end of the documentation - which assumes that you already know a lot about gLite.

What you have to do it produce a file (the site-info.def) that gives some high level details of what the UI needs to know to work. This file can be created anywhere (I put it in the same directory I unpacked the tarballs into), as you always gives it's path to yaim, the tool that uses it.

The first thing you need to put in is the 4 paths listed on the wiki page. Then you need a few other things:

BDII_HOST=svr019.gla.scotgrid.ac.uk
MON_HOST=svr019.gla.scotgrid.ac.uk
PX_HOST=lcgrbp01.gridpp.rl.ac.uk
WMS_HOST="svr022.gla.scotgrid.ac.uk svr023.gla.scotgrid.ac.uk"
RB_HOST=$WMS_HOST

The BDII host is where the UI gets it's information from - this should be a 'top level' BDII, not a site BDII. None of have the faintest clue why it needs the MON host - that's something I'll dig into later. The PX host is the MyProxy server to use by default. That one should be good for anywhere in the UK. The WMS host is the replacement for the deprecated (but still needed) RB hosts, and points to the WMS to be used for submission (by default).

One thing I found I needed that wasn't documented was a SITE_NAME. I just put the hostname in there - it doesn't appear to be used, but yaim complains if it's not there.

The last thing needed is a list of the VO's to be supported on the UI. When deploying a tar UI this will normally be a very small list - one or two I would expect. Therefore I choose to place them inline. There is a mechanism to put the VO specification in a separate directory, which is used for shared UI machines.

VOS="vo.scotgrid.ac.uk"

VO_VO_SCOTGRID_AC_UK_VOMS_SERVERS="vomss://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk"
VO_VO_SCOTGRID_AC_UK_VOMSES="'vo.scotgrid.ac.uk svr029.gla.scotgrid.ac.uk 15000 /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk vo.scotgrid.ac.uk'"
VO_VO_SCOTGRID_AC_UK_VOMS_CA_DN="'/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA'"

VO specification is in two parts - first we have to list the VO's (space separated list), and then , for each VO, give the VOMS server that defines the membership of the VO, and the certificate DN for the VOMS server. Note that the vo name gets translated to UPPER CASE and all the dots in it become underscores (a fact that's somewhat underdocumented, and results in a complaint about a syntactically invalid site-info.def, and no other message ...)

Once that's all in place, it's time to run yaim to configure things (from the dir I unpacked into):

./glite/yaim/bin/yaim -c -s site-info.def -n UI_TAR

Slight problem with installing certificates: By default these go into /etc/grid-security/certificates, but I'm not running as root. As a local user (for the initial testing), I need to tell yaim where to put them instead. In the site-info.def:

X509_CERT_DIR=${INSTALL_ROOT}/certificates

and make that directory, and re-run the yaim command. Chuntering along for a bit, and then finished with no errors - I did get a couple of warnings, but nothing that looked like a problem in this case.

Last step - testing. First, load up the installed software:

$GLITE_EXTERNAL_ROOT/etc/profile.d/grid-env.sh

and install my certificate on there.

lcg-infosites ... works
voms-proxy-* ... works
glite-wms-job-submit ... Boom!

glite-wms-job-submit: error while loading shared libraries: libboost_filesystem.so.2: wrong ELF class: ELFCLASS32

Hrm. Looks like a 32/64 bit problem. Some pokage later, and it turns out that the shell setup script supplied points only to the $GLITE_EXTERNAL_ROOT/usr/lib directory - and not the lib64, containing the needed libs. A quick hack onto the grid-env.sh, and that's rectified. Now:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl
glite-wms-job-submit: error while loading shared libraries: libicui18n.so.36: cannot open shared object file: No such file or directory

This turns out to be the International Components for Unicode (at least, I think so). The particularly interesting point about this is that the only references I can find to these libraries on SL include one from this very blog and they are all about Adobe Acrobat Reader... because that's the most common software that uses it.

I grabbed the RPM from http://linux1.fnal.gov/linux/scientific/5x/x86_64/SL/, and added it to $GLITE_EXTERNAL_ROOT/usr/lib64 by:

cd $GLITE_EXTERNAL_ROOT
rpm2cpio libicu-3.6-5.11.2.x86_64.rpm | cpio -i

And, finally:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

==========================================================================

Jobs submitted from a pure 64 bit SL5 system. Note that the separator character has changed, from being a line of '*' to a line of '='.

After chuntering away for a while,

[scotgrid@golem ~]$ glite-wms-job-output https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

Error - Operation Failed
Unable to retrieve the output

[scotgrid@golem ~]$ cd /tmp/jobOutput/
[scotgrid@golem jobOutput]$ ls
scotgrid_VW96yirZ4gG6jVXjx9UwBg

Which is a known problem - the UI reports that collecting job output failed, but it does succeed. If you don't need to get the OutputSandbox from the job (e.g. it's all written to an SE), then this isn't a problem.

Now to post some bug reports on the tar UI package... (Update: This is now bug number 52825 for the configuration and bug 52832 for the ICU package)

SL4/5 All bets are ON!

In the wake of Glasgow's upgrade to SL4 Andrew and I were quipping about when we would go to SL5. I jokingly said next year, but really thinking in 18 months.

However, after some discussion, Dr Paul Millar contends that more that 50% of UKI sites will still be running SL4 on the stroke of midnight, 1st January 2010.

Dr Millar - I take that bet. I think that more than 50% will be running something more recent that Sl4 on that date.

Further, Paul thinks that there will be at least one UKI site running SL4 in January 2013 (after the end of lifetime for RHEL4). I don't.

In each case the wager is a bottle of Veuve Clicquot Yellow Label.