Well we have had an sl5 test cluster for a while now but it has ever really seen any action other than the odd random hello world job from testing with cream and the like.
However, as the great sl5 debate raged on we put ourself forward as a test site for atlas along with Oxford, another site with an sl5 cluster for installs of SL4 and future SL5 versions of the atlas software.
However, in order to get my development CE visible to the real world I had to add it into our site bdii. This then attracted ops and dteam jobs as by default they were allowed through the CE. No great shakes and was actually good as it identified problems that had not been seen with simple hello world jobs from within ScotGrid.
The first mishap was a networking issue where the jobs could arrive but couldn't get their job wrapper and payload as most of our workers are NAT'd. Except my development one. A simple fix once we worked out what was wrong.
Two other problems were encountered. Firstly that CE-sft-lcg-rm-free test went into a warn state as the glite-WN package no longer pulls in ldapsearch. This is fixed by installing openldap-clients from sl-base.
Secondly, the many of the jobs that actually did run through the system encountered an error on CE-sft-brokerinfo with something like: error while loading shared libraries: libclassad_ns.so.0: cannot open shared object file: No such file or directory
After some googling, this bug is known about and has been fixed. The fix is adding gridpath_prepend "LD_LIBRARY_PATH" "/opt/classads/lib64/" to /etc/profile.d/grid-env.sh However, at Glasgow we control grid-env.sh though cfengine so I needed to make the appropriate change there too.
After going through this over the last few days I stumbled across Ewan's page as he had encountered the exact same issues. So take heed and do a spot of googling first!
There is also a metapackage available for Sl5 glite3.2 WN's this should hopefully contain all the required dependencies. This is located here. The gotcha with this is that you have to install it with yum localinstall or stick it in a yum repo as rpm -i doesn't work.
I have also just compared what is installed from this against the Atlas SL5 page and there were 4 packages missing: compat-gcc-34-g77, compat-libgcc-296, compat-libstdc++-296, ghostscript-8.15.2
So currently we have ops/dteam jobs running and passing. Atlas software jobs running, completing but not successfully working. More digging is required and I will keep you posted.
Tuesday, July 28, 2009
Wednesday, July 15, 2009
Rest in Peace gLite 3.0 ... finally
Today heralds a poignant day for the members of ScotGrid as we finally waved goodbye to the last gLite 3.0 Service (VOMS) and SL3.0 server in our cluster. The sombre mood was only broken by the arrival of the newborn gLite 3.1 VOMS server running on SL4. There was much flag waving and tears of joy as the first voms-proxy-init was issued and the shiny new web interface marvelled at.
Again Jpackage caused a little confusion as tomcat5 pulls jdk6 unless you exclude it or force an install of jdk5. This is preferred for all you firefox users out there. As if tomcat is running under jdk6 you have to remember to turn off TLS1.0 from the preferences menu in order to get the SSL handshaking to work or you get a nice fat error page! Not very useful for an admin screen let me tell you.
This upgrade was tried last year but was hampered by a lack of database migration scripts. This time around and with the help of these instructions it went swimmingly.
So although it was a sad day for gLite 3.0 and SL3 camp and a small victory for gLite 3.1/SL4, the war is not over. With gLite 3.2 and SL5 closing in on all fronts the battle is only just beginning.
p.s. we have an SL5 set-up so if you want to test, please let me know.
Again Jpackage caused a little confusion as tomcat5 pulls jdk6 unless you exclude it or force an install of jdk5. This is preferred for all you firefox users out there. As if tomcat is running under jdk6 you have to remember to turn off TLS1.0 from the preferences menu in order to get the SSL handshaking to work or you get a nice fat error page! Not very useful for an admin screen let me tell you.
This upgrade was tried last year but was hampered by a lack of database migration scripts. This time around and with the help of these instructions it went swimmingly.
So although it was a sad day for gLite 3.0 and SL3 camp and a small victory for gLite 3.1/SL4, the war is not over. With gLite 3.2 and SL5 closing in on all fronts the battle is only just beginning.
p.s. we have an SL5 set-up so if you want to test, please let me know.
Monday, July 06, 2009
Deflected Cosmic Rays...
This is the second short "when you're good..." post. During the RAL machine room move, we tested distributing ATLAS cosmics AOD and DPD data from CERN->GLASGOW->UK T2s. After some tweaking of the T2 FTS channels at CERN and tinkering in DDM this has worked a charm. Data distrubution in the UK has gone very well throughout the current combined cosmics data taking runs.
This is the first time that we tried circumventing the T1 for such an organised data distribution and it was a real success for the UK, ATLAS and Glasgow.
When you're good, you're Glasgow...
There hasn't been much time to write in the blog recently, STEP09 madness and all. However, it is wonderful to see that Glasgow was the top ATLAS T2 for analysis during the STEP09 challenges. We analysed more than 1.8B events, mostly through panda, with a 98% success rate.
We also took the largest fraction of data of any UK T2, 40%, and succeeded in getting all the data we were sent (we had little anxiety on the final weekend and want to increase our network heardroom for sure).
Sam and I wrote a full report on our experiences and how we used the opportunity to really probe the limits of the current cluster.
For the future, we really have to worry about how to maintain the i/o rate into the CPUs as the number of cores rises.
Installing (and fixing) a gLite Tar UI on SL5
First, a little background.
The UI machine is the gLite term for the machine from which you submit jobs (and monitor, receive output etc). This is analogous to the submit machine in Condor, and the head node for a local cluster - except that with the Grid, there is no reason that you can't submit on one UI, monitor from another and collect output on a third. No reason - except perhaps for keeping one's sanity.
Whilst most of the Grid servers are normally dedicated machines, occasionally given over to more than one Grid task, but only doing Grid tasks, the UI is a clear contender for being placed on machine that already have another purpose. In this instance, we have a group of users that have their own cluster, and occasionally off load some computations onto the Grid. It would be ideal if they could submit to either their local cluster or the Grid from the same machine. Cluster head nodes aren't too portable, so the obvious approach is to turn their existing head node into a gLite UI.
Fortunately, the gLite developers forsaw this possibility, and the UI package is available in a single blob that can be installed for an individual user. So that's what I've done - but there's a few caveats, and a couple of bugs to work around.
The tar UI I used was the gLite 3.2.1 production release. This is still early in the 3.2 life cycle, and not all services are available at 3.2, so there might be a few teething issues here, interacting with the older services. At Glasgow we don't have any 3.0 services, which is good, as they're really unsupported.
On to the install: Download the two tarballs, and unpack into a directory (why 2 tarballs, one tarball aught to be enough for anyone). I then promptly fell of the end of the documentation - which assumes that you already know a lot about gLite.
What you have to do it produce a file (the site-info.def) that gives some high level details of what the UI needs to know to work. This file can be created anywhere (I put it in the same directory I unpacked the tarballs into), as you always gives it's path to yaim, the tool that uses it.
The first thing you need to put in is the 4 paths listed on the wiki page. Then you need a few other things:
One thing I found I needed that wasn't documented was a SITE_NAME. I just put the hostname in there - it doesn't appear to be used, but yaim complains if it's not there.
The last thing needed is a list of the VO's to be supported on the UI. When deploying a tar UI this will normally be a very small list - one or two I would expect. Therefore I choose to place them inline. There is a mechanism to put the VO specification in a separate directory, which is used for shared UI machines.
Once that's all in place, it's time to run yaim to configure things (from the dir I unpacked into):
Last step - testing. First, load up the installed software:
lcg-infosites ... works
voms-proxy-* ... works
glite-wms-job-submit ... Boom!
I grabbed the RPM from http://linux1.fnal.gov/linux/scientific/5x/x86_64/SL/, and added it to $GLITE_EXTERNAL_ROOT/usr/lib64 by:
After chuntering away for a while,
Which is a known problem - the UI reports that collecting job output failed, but it does succeed. If you don't need to get the OutputSandbox from the job (e.g. it's all written to an SE), then this isn't a problem.
Now to post some bug reports on the tar UI package... (Update: This is now bug number 52825 for the configuration and bug 52832 for the ICU package)
The UI machine is the gLite term for the machine from which you submit jobs (and monitor, receive output etc). This is analogous to the submit machine in Condor, and the head node for a local cluster - except that with the Grid, there is no reason that you can't submit on one UI, monitor from another and collect output on a third. No reason - except perhaps for keeping one's sanity.
Whilst most of the Grid servers are normally dedicated machines, occasionally given over to more than one Grid task, but only doing Grid tasks, the UI is a clear contender for being placed on machine that already have another purpose. In this instance, we have a group of users that have their own cluster, and occasionally off load some computations onto the Grid. It would be ideal if they could submit to either their local cluster or the Grid from the same machine. Cluster head nodes aren't too portable, so the obvious approach is to turn their existing head node into a gLite UI.
Fortunately, the gLite developers forsaw this possibility, and the UI package is available in a single blob that can be installed for an individual user. So that's what I've done - but there's a few caveats, and a couple of bugs to work around.
The tar UI I used was the gLite 3.2.1 production release. This is still early in the 3.2 life cycle, and not all services are available at 3.2, so there might be a few teething issues here, interacting with the older services. At Glasgow we don't have any 3.0 services, which is good, as they're really unsupported.
On to the install: Download the two tarballs, and unpack into a directory (why 2 tarballs, one tarball aught to be enough for anyone). I then promptly fell of the end of the documentation - which assumes that you already know a lot about gLite.
What you have to do it produce a file (the site-info.def) that gives some high level details of what the UI needs to know to work. This file can be created anywhere (I put it in the same directory I unpacked the tarballs into), as you always gives it's path to yaim, the tool that uses it.
The first thing you need to put in is the 4 paths listed on the wiki page. Then you need a few other things:
BDII_HOST=svr019.gla.scotgrid.ac.ukThe BDII host is where the UI gets it's information from - this should be a 'top level' BDII, not a site BDII. None of have the faintest clue why it needs the MON host - that's something I'll dig into later. The PX host is the MyProxy server to use by default. That one should be good for anywhere in the UK. The WMS host is the replacement for the deprecated (but still needed) RB hosts, and points to the WMS to be used for submission (by default).
MON_HOST=svr019.gla.scotgrid.ac.uk
PX_HOST=lcgrbp01.gridpp.rl.ac.uk
WMS_HOST="svr022.gla.scotgrid.ac.uk svr023.gla.scotgrid.ac.uk"
RB_HOST=$WMS_HOST
One thing I found I needed that wasn't documented was a SITE_NAME. I just put the hostname in there - it doesn't appear to be used, but yaim complains if it's not there.
The last thing needed is a list of the VO's to be supported on the UI. When deploying a tar UI this will normally be a very small list - one or two I would expect. Therefore I choose to place them inline. There is a mechanism to put the VO specification in a separate directory, which is used for shared UI machines.
VOS="vo.scotgrid.ac.uk"VO specification is in two parts - first we have to list the VO's (space separated list), and then , for each VO, give the VOMS server that defines the membership of the VO, and the certificate DN for the VOMS server. Note that the vo name gets translated to UPPER CASE and all the dots in it become underscores (a fact that's somewhat underdocumented, and results in a complaint about a syntactically invalid site-info.def, and no other message ...)
VO_VO_SCOTGRID_AC_UK_VOMS_SERVERS="vomss://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk"
VO_VO_SCOTGRID_AC_UK_VOMSES="'vo.scotgrid.ac.uk svr029.gla.scotgrid.ac.uk 15000 /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk vo.scotgrid.ac.uk'"
VO_VO_SCOTGRID_AC_UK_VOMS_CA_DN="'/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA'"
Once that's all in place, it's time to run yaim to configure things (from the dir I unpacked into):
./glite/yaim/bin/yaim -c -s site-info.def -n UI_TARSlight problem with installing certificates: By default these go into /etc/grid-security/certificates, but I'm not running as root. As a local user (for the initial testing), I need to tell yaim where to put them instead. In the site-info.def:
X509_CERT_DIR=${INSTALL_ROOT}/certificates
and make that directory, and re-run the yaim command. Chuntering along for a bit, and then finished with no errors - I did get a couple of warnings, but nothing that looked like a problem in this case.Last step - testing. First, load up the installed software:
$GLITE_EXTERNAL_ROOT/etc/profile.d/grid-env.shand install my certificate on there.
lcg-infosites ... works
voms-proxy-* ... works
glite-wms-job-submit ... Boom!
glite-wms-job-submit: error while loading shared libraries: libboost_filesystem.so.2: wrong ELF class: ELFCLASS32Hrm. Looks like a 32/64 bit problem. Some pokage later, and it turns out that the shell setup script supplied points only to the $GLITE_EXTERNAL_ROOT/usr/lib directory - and not the lib64, containing the needed libs. A quick hack onto the grid-env.sh, and that's rectified. Now:
[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdlThis turns out to be the International Components for Unicode (at least, I think so). The particularly interesting point about this is that the only references I can find to these libraries on SL include one from this very blog and they are all about Adobe Acrobat Reader... because that's the most common software that uses it.
glite-wms-job-submit: error while loading shared libraries: libicui18n.so.36: cannot open shared object file: No such file or directory
I grabbed the RPM from http://linux1.fnal.gov/linux/scientific/5x/x86_64/SL/, and added it to $GLITE_EXTERNAL_ROOT/usr/lib64 by:
cd $GLITE_EXTERNAL_ROOTAnd, finally:
rpm2cpio libicu-3.6-5.11.2.x86_64.rpm | cpio -i
[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdlJobs submitted from a pure 64 bit SL5 system. Note that the separator character has changed, from being a line of '*' to a line of '='.
Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server
====================== glite-wms-job-submit Success ======================
The job has been successfully submitted to the WMProxy
Your job identifier is:
https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg
==========================================================================
After chuntering away for a while,
[scotgrid@golem ~]$ glite-wms-job-output https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg
Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server
Error - Operation Failed
Unable to retrieve the output
[scotgrid@golem ~]$ cd /tmp/jobOutput/
[scotgrid@golem jobOutput]$ ls
scotgrid_VW96yirZ4gG6jVXjx9UwBg
Which is a known problem - the UI reports that collecting job output failed, but it does succeed. If you don't need to get the OutputSandbox from the job (e.g. it's all written to an SE), then this isn't a problem.
Now to post some bug reports on the tar UI package... (Update: This is now bug number 52825 for the configuration and bug 52832 for the ICU package)
Thursday, July 02, 2009
NGS and EGEE Software Tags
After I reinstalled svr021 (CE) we lost some good work carried out by Andy Elwell to allow NGS software tags to be published along side glite ones through the BDII. I found the original post and re-installed the patch.
Wouldn't it be nice if this was available directly from glite well it is now!
The new script will correctly report information from the ngs-uee-gip-plugin plug-in that the NGS sites use to report applications installed under /usr/ngs.
Jason created the patch, I tested and Laurence Field at CERN has kindly merged the changes into gLite proper and has made an updated RPM available from...
http://etics-repository.cern.ch:8080/repository/download/registered/org.glite/glite-info-generic/2.0.2/noarch/glite-info-generic-2.0.2-5.noarch.rpm
so if you are an NGS affiliate site who runs a glite stack, the rpm above will allow both sets of software tags to be advertised through your BDII. If you try to install ngs-uee-gip-plugin without the new version of glite-info-generic your ngs tags will replace your glite software tags or vice-versa. Now they happily merge rather than replace.
Wouldn't it be nice if this was available directly from glite well it is now!
The new script will correctly report information from the ngs-uee-gip-plugin plug-in that the NGS sites use to report applications installed under /usr/ngs.
Jason created the patch, I tested and Laurence Field at CERN has kindly merged the changes into gLite proper and has made an updated RPM available from...
http://etics-repository.cern.ch:8080/repository/download/registered/org.glite/glite-info-generic/2.0.2/noarch/glite-info-generic-2.0.2-5.noarch.rpm
so if you are an NGS affiliate site who runs a glite stack, the rpm above will allow both sets of software tags to be advertised through your BDII. If you try to install ngs-uee-gip-plugin without the new version of glite-info-generic your ngs tags will replace your glite software tags or vice-versa. Now they happily merge rather than replace.
MPI really kicks off at Glasgow
It seems there is a real appetite for MPI codes on ScotGrid at the moment. First off there was Optics running Lumerical's FDTD FDTD and now we have UKQCD running Chroma. Next up is an MPICH install of CASTEP for Solid State Physics. So nearly 2 years after it was first enabled at ScotGrid it is finally seeing it's first tour of duty. Better late than never. We still have some kinks to iron out such as better scheduling of MPI on our cluster but early results from benchmarking are promising even for an ethernet based MPI solution!
Subscribe to:
Posts (Atom)