I was very keen to take up Stephen Child's offer to get MPI enabled
properly at Glasgow. (See link for my last attempts at this working,
where I cobbled together something that was far from satisfactory, but
at least proved it was possible in theory to get this to work.)
A problem for MPI at Glasgow is that our pool account home directories
are not shared and that jobs all wake up in /tmp anyway. For local
users we offer the /cluster/share area, which gets around this, but
what to do for generic MPI jobs? We decided it would be a very good
idea to offer some shared area for MPI jobs, and that the right
strategy would be to modify mpi-start to pick the job up by its
bootstraps and drop itself back down into a shared directory area in
/cluster/share/mpi. To do this we decided to generalise the
MPI_SHARED_HOME environment variable. Previously this had been "yes"
or "no", but in the new scheme if it points to a directory then the
script transplants the job to an appropriate subdirectory of this
area.
On the site side I had to make site all the MPI environment variables
were properly defined in the job's environment (which we do with
/etc/profile.d/mpi.sh) and advertise the right MPI attributes in the
information system.
It all went pretty well, until we had an issue with mpiexec not being
able to invoke the other job threads properly. (Mpiexec starts the
other job threads via torque, which means they get accounted for
properly and that we can disable passwordless ssh between the WNs
- which we did at the SL4 upgrade). There was a fear it was due to some
weird torque build problem, but in the end it was a simple issue with
server_name not being properly defined on the workers. A quick but of
cfengine and this was then fixed.
So, Glasgow now supports MPI jobs - excellent. (He will rebuild the
newly featured mpi-start and release it next week.)
Big thanks to Stephen for setting this up for us.
Monday, October 08, 2007
Lunch with glexec developers
We had a very interesting lunchtime meeting with the glexec developers
on Tuesday lunchtime (organised thanks to Alessandra's efforts in the TCG to convince the developers that the sites had serious issues with glexec). The developers very
happy to meet us and discuss the orgins of glexec and why they thought
that it was needed. It was clear that no-one in the meeting is at all
keen on generic pilot job frameworks. However, what's also clear is
that the LHC VOs are going to insist on having them and it's highly
unlikely that sites would ever be able to exert enough influence on
them to stop. However, what we can do is insist that if such
frameworks do exist, then they will have to use a glexec call and
respect its result. What glexec gives, in essence, is sudo like
abilities, but integrated into an X509 based authorisation scheme (and
it uses LCAS/LCMAPS plugins, which we are familiar with). The
advantages to the sites are that (1) there is a lot of control as to
who can call glexec in the first place, e.g., restricting this only to
production roles; (2) using LCMAPS single users can be banned, whereas
without glexec there only option with misbehaving pilot jobs is to ban
the pilot user on the CE (which is tantamount to banning the whole
VO); (3) there's an audit trail of who's payload has been executed
(which is crucial). glexec has safeguards to stop multiple calls
(i.e., the payload recalling glexec).
Then, to suid or not? We are assured that glexec will be distributed
in two flavours - one with the suid bit switched on in the RPM, the
other with it switched off. Of course, post-installation, one can
easily flip the bit using cfengine. The danger of not enabling suid is
that it will be possible for the payload job to access the submission
proxy certificate (danger for the VO) and that sorting out the payload
from the pilot is harder at the process level (danger for the
site). Of course, this has to be balanced by the danger of enabling
suid and risking a possible avenue of privilege escalation should
glexec turn out to have a security problem.
I'm still convinced that for ScotGrid deploying glexec in non-suid
form is best. We can run it for a while and then evaluate the
situation further. It's clear that sites like ECDF will never allow
glexec to be suid for them - so running it in non-suid mode will
always be an option.
The security vulnerabilities that Kostas identified have all been
fixed - along with more potential problems they spotted when going
through the code. They are very happy for other people to look at the
code and feedback problems to them.
Overall I was very impressed at the developers' openness. I still
don't think generic pilots are a good idea for the grid, but in a
world where they exist glexec is a definite help.
on Tuesday lunchtime (organised thanks to Alessandra's efforts in the TCG to convince the developers that the sites had serious issues with glexec). The developers very
happy to meet us and discuss the orgins of glexec and why they thought
that it was needed. It was clear that no-one in the meeting is at all
keen on generic pilot job frameworks. However, what's also clear is
that the LHC VOs are going to insist on having them and it's highly
unlikely that sites would ever be able to exert enough influence on
them to stop. However, what we can do is insist that if such
frameworks do exist, then they will have to use a glexec call and
respect its result. What glexec gives, in essence, is sudo like
abilities, but integrated into an X509 based authorisation scheme (and
it uses LCAS/LCMAPS plugins, which we are familiar with). The
advantages to the sites are that (1) there is a lot of control as to
who can call glexec in the first place, e.g., restricting this only to
production roles; (2) using LCMAPS single users can be banned, whereas
without glexec there only option with misbehaving pilot jobs is to ban
the pilot user on the CE (which is tantamount to banning the whole
VO); (3) there's an audit trail of who's payload has been executed
(which is crucial). glexec has safeguards to stop multiple calls
(i.e., the payload recalling glexec).
Then, to suid or not? We are assured that glexec will be distributed
in two flavours - one with the suid bit switched on in the RPM, the
other with it switched off. Of course, post-installation, one can
easily flip the bit using cfengine. The danger of not enabling suid is
that it will be possible for the payload job to access the submission
proxy certificate (danger for the VO) and that sorting out the payload
from the pilot is harder at the process level (danger for the
site). Of course, this has to be balanced by the danger of enabling
suid and risking a possible avenue of privilege escalation should
glexec turn out to have a security problem.
I'm still convinced that for ScotGrid deploying glexec in non-suid
form is best. We can run it for a while and then evaluate the
situation further. It's clear that sites like ECDF will never allow
glexec to be suid for them - so running it in non-suid mode will
always be an option.
The security vulnerabilities that Kostas identified have all been
fixed - along with more potential problems they spotted when going
through the code. They are very happy for other people to look at the
code and feedback problems to them.
Overall I was very impressed at the developers' openness. I still
don't think generic pilots are a good idea for the grid, but in a
world where they exist glexec is a definite help.
EGEE 07 Round-up
Here's a round-up of notes and interesting bits
EGI (European Grid Infrastructure) was a big theme during the conference. This is expected to be the permanent grid infrastructure replacement which arrives after the end of EGEE III. There's lots to be worked out, not least of which is the need for a sustainable national grid infrastructure (NGI) in each country or region.
NA4
Workbench Overview: Taverna; Review; There will always be a zoo of workflow engines. Which one to choose? No answer yet! Information system - need to engage GLUE. Pushes pain from workflow people to middleware providers?
Lots of portals and workflows out there - Taverna, P-Grade, Ganga/Diane.
Unfortunately because of SA3 and SA1 sessions didn't get to see the demos.
SA3
gLite release process: should patches be bundled or not? can we divide
patches into the simple RPM updates and more complex changes requiring
reconfiguration.
YAIM 4 is coming. Proper component based YAIM - heirarchy of
configuration files (site general -> component specific). Should be
much better. Has _pre and _post function hooks - much more flexible to
override aspects of YAIM, but not lose core functionality.
SA1
Ian Bird: Good overview of operations. Big scale up next year with the LHC. Transition to EGI and RGIs will be a very big challenge.
JRA1
gLite Overview:
See Claudio's presentation. Note that for highest performance should
separate LB and WMS. Well, for an initial trial service this is
probably not needed (not supporting 12000 jobs a day!). Time scale for
porting components to SL4 i386 is now (roughly) known.
x86_64 expected to be complete next spring (but DPM/LFC already
there).
OMII Europe:
I sat in on this as it was a project I had heard a lot about, but didn't know what they did. They have partners in US and Asia and do this:
Adding BES support to CREAM. (Might _eventually_ lead to dropping native CREAM interface, but not soon - WMS uses this). BES is the proposed OGF standard for job submission.
Adding SAML support to VOMS. (Security Assertion Markup Language: http://en.wikipedia.org/wiki/SAML).
Involved in Glue 2.0. Will help with schema production in XML and LDAP and assist standardisation efforts.
EGI (European Grid Infrastructure) was a big theme during the conference. This is expected to be the permanent grid infrastructure replacement which arrives after the end of EGEE III. There's lots to be worked out, not least of which is the need for a sustainable national grid infrastructure (NGI) in each country or region.
NA4
Workbench Overview: Taverna; Review; There will always be a zoo of workflow engines. Which one to choose? No answer yet! Information system - need to engage GLUE. Pushes pain from workflow people to middleware providers?
Lots of portals and workflows out there - Taverna, P-Grade, Ganga/Diane.
Unfortunately because of SA3 and SA1 sessions didn't get to see the demos.
SA3
gLite release process: should patches be bundled or not? can we divide
patches into the simple RPM updates and more complex changes requiring
reconfiguration.
YAIM 4 is coming. Proper component based YAIM - heirarchy of
configuration files (site general -> component specific). Should be
much better. Has _pre and _post function hooks - much more flexible to
override aspects of YAIM, but not lose core functionality.
SA1
Ian Bird: Good overview of operations. Big scale up next year with the LHC. Transition to EGI and RGIs will be a very big challenge.
JRA1
gLite Overview:
See Claudio's presentation. Note that for highest performance should
separate LB and WMS. Well, for an initial trial service this is
probably not needed (not supporting 12000 jobs a day!). Time scale for
porting components to SL4 i386 is now (roughly) known.
x86_64 expected to be complete next spring (but DPM/LFC already
there).
OMII Europe:
I sat in on this as it was a project I had heard a lot about, but didn't know what they did. They have partners in US and Asia and do this:
Adding BES support to CREAM. (Might _eventually_ lead to dropping native CREAM interface, but not soon - WMS uses this). BES is the proposed OGF standard for job submission.
Adding SAML support to VOMS. (Security Assertion Markup Language: http://en.wikipedia.org/wiki/SAML).
Involved in Glue 2.0. Will help with schema production in XML and LDAP and assist standardisation efforts.
Friday, October 05, 2007
knotty nat knowledge
hmm, thats odd, why aren't the NAT boxes visible on ganglia?
Seemed a simple enough problem - they used to be there, but for some reason fell off the plots late August.
Had boxes restarted and failed to start gmond? nope - good uptime. gmond running? yep. Telnet to gmond port? yep. hmm.
<CLUSTER NAME="NAT Boxes" ... >
</CLUSTER>
and no HOST or METRIC lines between them. Most odd. After some discussion with Dr Millar it turned out to be a probable issue with the Linux Multicast setup - the kernel wasn't choosing the same interface to listen and send on. Luckily this was patched in a newer version of ganglia - the config file supports the mcast_if parameter to allow explicit setting (in our case to the internal ones).
Sadly of course the out-of-the box RPM doesn't install on SL4 x86_64 - needs unmet dependencies (as normal....) so a quick compile on one of the worker nodes and some dirty-hackery-copying the binary over worked a treat. We now have natbox stats again..
Seemed a simple enough problem - they used to be there, but for some reason fell off the plots late August.
Had boxes restarted and failed to start gmond? nope - good uptime. gmond running? yep. Telnet to gmond port? yep. hmm.
<CLUSTER NAME="NAT Boxes" ... >
</CLUSTER>
and no HOST or METRIC lines between them. Most odd. After some discussion with Dr Millar it turned out to be a probable issue with the Linux Multicast setup - the kernel wasn't choosing the same interface to listen and send on. Luckily this was patched in a newer version of ganglia - the config file supports the mcast_if parameter to allow explicit setting (in our case to the internal ones).
Sadly of course the out-of-the box RPM doesn't install on SL4 x86_64 - needs unmet dependencies (as normal....) so a quick compile on one of the worker nodes and some dirty-hackery-copying the binary over worked a treat. We now have natbox stats again..
Monday, October 01, 2007
NFS does TCP (unexpectedly)
We got alerted by our users to that fact that svr020 was going very, very slowly. When I logged in it took more than a minute and a simple ls on one of the nfs areas was taking ~20s.
Naturally these damn things always happen when you're away and sitting with a 15% battery in a session which is actually of interest!
Anyway, I couldn't see what was wrong on svr020 - the sluggishness seemed symptomatic rather than tied to any bad process or user. There were clear periods when we hit serious CPU Wait. When I checked on disk037 I found many, many error messages like:
nfsd: too many open TCP sockets, consider increasing the number of nfsd threads
nfsd: last TCP connect from 10.141.0.101:816
When I checked on the worker nodes, sure enough, we seemed to have inadvertently switched to using tcp for nfs - I later found out this is the default mount option on 2.6 kernels (which we've obviously just switched to).
I decided to follow the advice in the error message, and run more nfsd threads, so I created a file /etc/sysconfig/nfs, with RPCNFSDCOUNT=24 (the default is 8). I then (rather nervously) restarted nfs on disk037.
Having done this, the load on svr020 returned to normal and the kernel error messages stopped on disk037. Whew!
Relief could clearly be seen, with the cluster load dropping back down to the number of running processes, instead of being "overloaded" as it had been.
We should use MonAMI to monitor the number of nfs tcp connections on disk037. (Just out of interest the number now is 343, which must have caused problems for 8 nfs server daemons; hence, 24 should scale to ~900 mounts.)
Of course, we could shift back to udp nfs mounts if we wanted. Reading through the NFS FAQ might help us decide.
Torque Queue ACLs
I got an email from Rod Walker on Friday night. He was having trouble submitting to the ATLAS queue on the cluster - again the infamously unhelpful "Unspecified gridmanager error".
I checked his mapping, and LCMAPS was correctly mapping him to one of the new atlas production accounts correctly. However, when I looked that the queue, the torque queue configuration had lost the ACL which allowed sgm and prd accounts to submit to it.
I corrected that and all was well again.
I think that probably these ACLs were never correctly set on the cluster as they seemed to be missing on most of the queues (notably the ops queue was not affected). The cfengine script to setup the queues had the correct ACL setup in it, but I guess it had never been run.
The effect on ATLAS services was notable - we ran an awful lot more ATLAS production this weekend (blue jobs).
I checked his mapping, and LCMAPS was correctly mapping him to one of the new atlas production accounts correctly. However, when I looked that the queue, the torque queue configuration had lost the ACL which allowed sgm and prd accounts to submit to it.
I corrected that and all was well again.
I think that probably these ACLs were never correctly set on the cluster as they seemed to be missing on most of the queues (notably the ops queue was not affected). The cfengine script to setup the queues had the correct ACL setup in it, but I guess it had never been run.
The effect on ATLAS services was notable - we ran an awful lot more ATLAS production this weekend (blue jobs).
Friday, September 28, 2007
Biomed Stalled Jobs
Since we came back after the upgrade to SL4 I had noticed a very large number of stalled biomed jobs on the cluster.
These were all jobs which had stalled running python ./get_task.py autodock AVIANFLUDC2_T02IAN3J1170 (or something very like it).
As the cluster hadn't actually been full, and I was very busy, I actually let this situation go for most of the week. However, today I emailed the user (using the CIC portal user look up). I got a very quick response that there was a known problem with an overloaded AMGA server, which was causing these stalls. I was given permission to kill the jobs, which I did.
Although it's a good thing (tm) to get in touch with users, following our stalled jobs guide, it is time consuming and I wish there was some form of automation we could apply.
DPM Dies
Our DPM died last night (sad!). It seemed that / got full and this then caused DPM and MySQL to get into a punch-up where all the CPU on the machine was consumed.
Investigating (with help from Paul - thanks!) the culprit seems mostly to be an innod db "auto-extending data file" called ibdata1. This has now reached 2.1GB in size.
There is some advice about how to configure innodb to control these sizes, but as the default MySQL install on SL3 has no default my.cnf file we'll have to create a sensible one of these before being able to customise this.
However, after some further investigation, it's now clear that 2.1GB is in fact the size of our DPM database (the gziped database dumps are now 800MB!). This with 14TB of data. Scaling up to 100TB and the DB will be > 10GB. Having looked at the tables, the obvious candidates to trim are dpm_put_filereq, dpm_get_filereq and dpm_req. These seem to contain historical data, but without timestamps it's pretty useless. These tables contain about 233MB, 460MB and 295MB respectively, which is about half that total DPM database size. [1]
Recovery strategy has been to move /var/lib/mysql and /var/log off root, to a larger partition (in our case /disk). Soft links point out from the original locations.
I shall put in a ticket to the developers about trimming these tables when the data has aged into uselessness. [2]
The warning for other sites is that /var/lib/mysql really needs to live in a relatively large disk partition.
We'll address this problem properly when we upgade to the gLite 3.1 version of DPM. In the meantime we urgently need to alarm on disk space usage on all the servers.
[1] Try:
mysql> user dpm_db; show table status like "dpm_%";
(Thanks Paul.)
[2] https://gus.fzk.de/ws/ticket_info.php?ticket=27385
Wednesday, September 26, 2007
Shaking down the user issues
The upgrade to gLite 3.1 on the UI has brought a couple of surprises, which we're gradually working our way around.
* First, the version of grid-proxy-init supplied at part of VDT seems to behave rather differently. A proxy initialised with this looks like:
Using voms-proxy-init (without VOMS extensions) gives a rather more normal proxy:
The fix for this was to ensure that /opt/glite/bin appears higher in the path than /opt/globus/bin and then to create a soft link from /opt/glite/bin/voms-proxy-init to grid-proxy-init.
* Second problem was with UKQCD client software. Craig reported he was getting errors because of a missing globus rft library (libglobus_rls_client_gcc32dbgpthr.so.0). gLite 3.1 is compiled against VDT 1.6 (as opposed to VDT 1.2) and this library is no longer built. The first attempt at fixing it was to copy across the missing library into /opt/globus/lib. This failed, though, because then it became clear that the 1.2 and 1.6 VDT libraries are not compatible with one another, so the new libraries were missing symbols the QCD software needed. So, in the end, the old libraries were copied, lock, stock and barrel, into /opt/globus-glite30/lib and, with a suitable LD_LIBRARY_PATH, the qcd application would run.
There is a new version of QCDGrid software being built, so hopefully that will be compatible with the new VDT.
* Finally (or the last issue which has come to light), is Dan having trouble with g++. The default gcc-c++ (3.4.1) doesn't play well with his version of NLOJET++. He's going to try again with the old 3.2.3 g++ (a.k.a. g++32). Hopefully this will fix things.
* First, the version of grid-proxy-init supplied at part of VDT seems to behave rather differently. A proxy initialised with this looks like:
svr020:~$ grid-proxy-infoAnd the lcg-RB does not like this proxy at all.
subject : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart/CN=892101086
issuer : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
identity : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
type : Proxy draft (pre-RFC) compliant impersonation proxy
strength : 512 bits
path : /tmp/x509up_u218012
timeleft : 11:59:57
Using voms-proxy-init (without VOMS extensions) gives a rather more normal proxy:
subject : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart/CN=proxyThis hits ganga particularly hard, which renews proxies for you, but uses grid-proxy-init.
issuer : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
identity : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
type : full legacy globus proxy
strength : 512 bits
path : /tmp/x509up_u218012
timeleft : 11:59:58
The fix for this was to ensure that /opt/glite/bin appears higher in the path than /opt/globus/bin and then to create a soft link from /opt/glite/bin/voms-proxy-init to grid-proxy-init.
* Second problem was with UKQCD client software. Craig reported he was getting errors because of a missing globus rft library (libglobus_rls_client_gcc32dbgpthr.so.0). gLite 3.1 is compiled against VDT 1.6 (as opposed to VDT 1.2) and this library is no longer built. The first attempt at fixing it was to copy across the missing library into /opt/globus/lib. This failed, though, because then it became clear that the 1.2 and 1.6 VDT libraries are not compatible with one another, so the new libraries were missing symbols the QCD software needed. So, in the end, the old libraries were copied, lock, stock and barrel, into /opt/globus-glite30/lib and, with a suitable LD_LIBRARY_PATH, the qcd application would run.
There is a new version of QCDGrid software being built, so hopefully that will be compatible with the new VDT.
* Finally (or the last issue which has come to light), is Dan having trouble with g++. The default gcc-c++ (3.4.1) doesn't play well with his version of NLOJET++. He's going to try again with the old 3.2.3 g++ (a.k.a. g++32). Hopefully this will fix things.
RB: "Rather Better"
One thing which went to pot during the upgrade, was the way that the higher UIDed pool accounts cascaded through to the RB. This, unfortunately, meant that any jobs which were running on svr023 were lost (there would have been very few, in fact, which is why we spent more efforts on the UI).
However, in our attempts to get the RB back it became clear that the edg-wl-ftpd service (yet another hacked version of GT2 gridftp) cannot handle UIDs > 16bit. This screwed things up for us, as all our new UIDs are in the range 200000+.
In the end I had to re-hack the perl passwd/group/shadow/users.conf generator, lowering all the UIDs specially for the RB. In fact this was not quite as awful as one might think, as the RB supports only a subset of the VOs we run jobs for on the main site. I also scripted up a "generator" for the RB's site-info.def, that strips down the VOS variable to those we support for job submission. In addition, communication between RB and the UI or the CE is of course mediated by certificate, so having a different pooll account or UID on the RB is not a problem.
There was a supporting tweak to cfengine to take passwd-rb (etc.) as the source passwd file for the RB.
Then the RB was blown away and rebuilt. It seems to have done it rather a lot of good, as now Steve Lloyd's dteam test jobs run properly (see his RB test page).
However, in our attempts to get the RB back it became clear that the edg-wl-ftpd service (yet another hacked version of GT2 gridftp) cannot handle UIDs > 16bit. This screwed things up for us, as all our new UIDs are in the range 200000+.
In the end I had to re-hack the perl passwd/group/shadow/users.conf generator, lowering all the UIDs specially for the RB. In fact this was not quite as awful as one might think, as the RB supports only a subset of the VOs we run jobs for on the main site. I also scripted up a "generator" for the RB's site-info.def, that strips down the VOS variable to those we support for job submission. In addition, communication between RB and the UI or the CE is of course mediated by certificate, so having a different pooll account or UID on the RB is not a problem.
There was a supporting tweak to cfengine to take passwd-rb (etc.) as the source passwd file for the RB.
Then the RB was blown away and rebuilt. It seems to have done it rather a lot of good, as now Steve Lloyd's dteam test jobs run properly (see his RB test page).
Tuesday, September 25, 2007
SL4 x86_64 UI now Availiable
I reinstalled the site's svr020 UI on Saturday. This involved an incredible amount of pain related to the bizarre inability of SL4 to properly install GRUB on a linux software RAID partition. Although the machine would install absolutely fine, on reboot it would just halt after the GRUB prompt.
In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an
That finally gave me a base SL4 install I could work with.
After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).
Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)
Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).
In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an
That finally gave me a base SL4 install I could work with.
After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).
Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)
Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).
Batch System Goes on Holiday?
When I started to fiddle with the UI and RB on Saturday night, I discovered that the site was failing SAM tests, with the, as usual, marvellously descriptive error "Unspecified gridmanager error".Further investigation showed that torque and maui servers were not running. When I restarted them the site recovered immediately. The very curious thing was, though, that torque logfile entries were still being written - so there was some part of torque running, but not enough to accept new jobs.
We need a nagios alarm on this. Paul tells me that there is a torque.available metric in the MonAMI sensor, so we should be able to passively monitor this - see the above graph which shows the dropout on Saturday afternoon.
Thursday, September 20, 2007
SL4/5 All bets are ON!
In the wake of Glasgow's upgrade to SL4 Andrew and I were quipping about when we would go to SL5. I jokingly said next year, but really thinking in 18 months.However, after some discussion, Dr Paul Millar contends that more that 50% of UKI sites will still be running SL4 on the stroke of midnight, 1st January 2010.
Dr Millar - I take that bet. I think that more than 50% will be running something more recent that Sl4 on that date.
Further, Paul thinks that there will be at least one UKI site running SL4 in January 2013 (after the end of lifetime for RHEL4). I don't.
In each case the wager is a bottle of Veuve Clicquot Yellow Label.
ATLAS jobs running properly
Chris submitted a sample ATLAS job to the cluster, using ppeui and the RAL RB.
It worked!
So in addition to passing the tests, we can also do real work. Tests and working for real jobs have a correspondence, but are not exactly the same (tests being a necessary, but not sufficient condition for doing real work, in gereral) so I am very pleased.
He'll now throw in 500 jobs, so we look forward to having the workload ramp up.
It worked!
So in addition to passing the tests, we can also do real work. Tests and working for real jobs have a correspondence, but are not exactly the same (tests being a necessary, but not sufficient condition for doing real work, in gereral) so I am very pleased.
He'll now throw in 500 jobs, so we look forward to having the workload ramp up.
Glasgow Upgraded to SL4
The upgrade is done! We started passing ops SAM tests at about 2230 last night, and I brought us out of downtime at 2300. That was 12 hours of total downtime. In addition the queues were closed from about 1600 the day before, so that meant we were unavailable for 31 hours. In the grand scheme of things I think, "not bad," for such a major upgrade.
Preparations for the upgrade were rushed, but certainly thorough enough for us to have a fair degree of confidence in the process. By Tuesday night I was able to reboot, rebuild and run jobs through a worker node successfully. Andrew was close to having the new pool account generator done, even if he had wimped out and used perl.
We had decided the plan was to upgrade the worker nodes and bring us out of downtime ASAP, then work on the UI and other less central services.
Here's my synopsis of what went wrong, or didn't behave quite as we expected:
However, we're working as a team now, rather than me playing Lone Ranger. This makes co-ordination, documentation and testing even more vital. Once Mike comes properly on board his first major task will be to understand and then document how the cluster is run.
Preparations for the upgrade were rushed, but certainly thorough enough for us to have a fair degree of confidence in the process. By Tuesday night I was able to reboot, rebuild and run jobs through a worker node successfully. Andrew was close to having the new pool account generator done, even if he had wimped out and used perl.
We had decided the plan was to upgrade the worker nodes and bring us out of downtime ASAP, then work on the UI and other less central services.
Here's my synopsis of what went wrong, or didn't behave quite as we expected:
- We initially tried to reboot the worker nodes in batches of 30. This overloaded dhcp or tftp on svr031, so in fact only 4 nodes were successful in that batch. Subsequently we did batches of 12, which worked fine. We could also put a larger stagger on the powernode reboot script (we had only used 1s).
Analysis: It was always going to be hard to know what level we could do this until we tried. It was easy to work around. Probably our rebuild time for the whole cluster is ~2-3 hours because of this node throughput limitation. - At the last minute I decided to just drop alice and babar to stop us from supporting VOs who just don't, or can't, use us (it's just clutter). However, that change was imperfectly expressed in site-info.def, so on the first batches of nodes YAIM just didn't run.
Analysis: This was a mistake. Andrew and I should have co-ordinated better and had more time to review the new user information files. - There were a few problems with the user information files: sgm and prd accounts weren't initially in the normal VO group. In addition local Glasgow users were in the wrong group. This was fixed pretty rapidly.
Analysis: As above. This aspect of the preparation was too close to the critical path - and it didn't work first time. - The new server certificates were botched initially. Although we were in downtime and it was relatively easy to correct, it was a distraction. Analysis: We need to document local procedures for certificate handling better.
- We'd been obsessing about the batch worker configurations, with the intention to leave the servers pretty much alone. However, we hadn't twigged that the change to pooled accounts for sgm and prd users would, of course, require the LCMAPS group and grid mapfiles to be updated. As no one on site is an sgm or a prd user this was not picked up during testing. It only came to light once I did a logfile analysis of why ops tests were failing (these are done as an sgm ops user). Later in the evening it became clear that this also had to be done for the DPM disk servers.
Analysis: If I'd been sharper I would have realised this in advance (but there was a lot on my mind). It would be useful of one of us had a special role to do this testing (gridpp VO would be ideal). However, it would actually have been a terribly hard thing to test, as the site was "live" during the testing phase and this problem's solution implied reconfiguring the CE as well as the pool accounts. Hopefully writing it down here will make us more cognoscent of this next time! - Running YAIM automatically if all well and good, but how do we know it's run successfully? We not only had nodes where YAIM jusy hadn't run, we also (and this was the last problem to be fixed), had two bad nodes where the directories in /opt ended up in mode 0700, so were unreadable.
Analysis: We need to develop a test and alarm system for which attempts to validate the YAIM run. At the moment we're pretty much flying blind. The two proxies which I ended up using yesterday were: - Look for files generated by YAIM, e.g., /opt/glite/etc/profile.d/grid-env.sh. There should be a nagios alarm or a cfengine warning if this file absent.
- Check permissions on directories such as /opt/glite/etc. If this is not readable to a pool account then something has gone wrong.
However, we're working as a team now, rather than me playing Lone Ranger. This makes co-ordination, documentation and testing even more vital. Once Mike comes properly on board his first major task will be to understand and then document how the cluster is run.
Wednesday, September 19, 2007
Yarr! Thats not line noise, it's Perl (me hearties)
Avast Ye Salty Sea-dogs - It be the glasgow upgrade day today (as well as international talk like a pirate day). Graeme gave me a minor task to do - come up with a new userid/groupid/passwd/shadow/yaim config generator script for the pool accounts. Simple enough, should only take half a day or so....
Much swearing at Python later, I gave up at 1AM this morning and resorted to trusty Perl. Done in 48 lines (including comments) and 2 hours. I think Python and I are going to take a looooong time to get aquainted properly.
I'm sure G will blog in more detail - Worker nodes went fairly smothly, few niggles - seems that about 24 simultaneous installs cause tftp timeouts. Discovered the Sandbox dirs on the RB needed their ownerships changing - again another Perl script to the rescue with the trick I learned from Steve Andrews - just get your script to print the command line you'd like to stdout, then once you've checked it looks reasonable, run it again piped through | /bin/sh
Much swearing at Python later, I gave up at 1AM this morning and resorted to trusty Perl. Done in 48 lines (including comments) and 2 hours. I think Python and I are going to take a looooong time to get aquainted properly.
I'm sure G will blog in more detail - Worker nodes went fairly smothly, few niggles - seems that about 24 simultaneous installs cause tftp timeouts. Discovered the Sandbox dirs on the RB needed their ownerships changing - again another Perl script to the rescue with the trick I learned from Steve Andrews - just get your script to print the command line you'd like to stdout, then once you've checked it looks reasonable, run it again piped through | /bin/sh
Friday, September 14, 2007
Glasgow upgrade to SL4 x86_64 next week
Issues have been gathered and a work plan is in place. The intention is still to upgrade next week, hopefully starting (and finishing) on Wednesday.
ECDF for Beginners
Basically, this is proving far more painful than anticipated. Although the MON/LFC box has been configured, the CE is proving seriously problematic. The ECDF team thought that SL3 was not a winner for GPFS, so Sam tried using the gLite 3.0 CE on top of SL4. This didn't work (not unexpectedly). Although we know that the lcg-CE has been built for gLite 3.1, it's not yet even been released to pre-production, so clearly there's nothing we can use for a production site. So Ewan reinstalled the CE with SL3, in order to install the old gLite 3.0 version. However, it then proved to be very difficult to get GPFS working on SL3, so this is still a work in progress. How long it will take to resolve is anyone's guess.
GPFS is necessary for the software area and the pool account home directories. At this point I would just buy a 500GB disk from PC world and run with that for a month while we wait for the gLite 3.1 CE, but we can't do that with machines other people are running.
Getting the site certified for the end of the month now looks challenging.
Hmmm....
GPFS is necessary for the software area and the pool account home directories. At this point I would just buy a 500GB disk from PC world and run with that for a month while we wait for the gLite 3.1 CE, but we can't do that with machines other people are running.
Getting the site certified for the end of the month now looks challenging.
Hmmm....
Durham News
Phil's procured a 15TB disk server for their DPM. This is now ready to go and we should get it configured and added to the Durham SRM next week.
WLCG Workshop
There's a lot on the agenda page, and Jamie's CHEP summary talk is a useful round up, but here are my highlights:
- GDB consider services necessary for WLCG operations to basically have been delivered, but not fully deployed (far less extensively tested and battle hardened). My own feeling is that in the integration and mutual interaction (e.g., VOMS and SRM v2.2) of all these services with WLCG production we've still a long way to go.
- In their pre-acceptance tests, the gLite-CE and the CREAM CE did about as well as one another. Given that CREAM has a simpler internal architecture and is more standards compliant it has been chosen over the gLite-CE (which is "no longer being developed"). It's anticipated that CREAM will be first delivered to production in the New Year, but that in total it's anticipated to take about a year from now before it's fully hardened, sites have experience in running it and the YAIM configuration is fully working - oh! just around data taking time ;-) However... this seems rather sensible to me to have settled on the one CE, rather than the unsatisfactory situation of having 2 on the go. Pity the poor folk who put effort into actually running gLite CE as a service!
- Operations: We still have trouble sharing knowledge. 2007 was the year of the grid blog, but only a fool would pretend that will solve all our problems. I proposed a wiki plugin, where articles less than a month old would be black on white, with the text gradually fading to paler shades of grey the longer the article was unrevised. When it reaches white the article is expired!
- SAM is the test framework for everyone. Experiments should publish their tests into SAM so that it's a one-stop-shop for sites' status.
- Monitoring: We look forward to a proper demo of the SAM/nagios framework at EGEE.
- SRM v2.2: Confusion still abounds. What spaces to ATLAS want at T2s? Not even they know yet...
- Dress Rehersals: Lots going on for every experiment. See their presentations for more details.
- Common Computing Readiness Challenge (CCRC): There's an urgent need to tests the whole of the Accelerator->T0->T1->T2 chain for all experiments before real data flows. This will probably happen in February (limited, because not everything will be ready) and in May (really, has to work!). I expect this to be quite a big deal for everyone, especially the May round.
- ATLAS Session: Good information on Production Dashboard, DDM dashboard and operations (with cool http queries).
The Midnight Blogger
Well, not quite midnight, but there's been so little time to write the blog this week it seems I have to do it late on a Friday night.
Are you sitting comfortably? Then I'll (grab a beer and) begin...
Are you sitting comfortably? Then I'll (grab a beer and) begin...
Wednesday, September 12, 2007
Nagios Acknowledements
We've been sucessfully using Nagios as one component of our site monitoring. The email and Jabber notification is great, but when someone acks a problem, we have to look at the webpage to see who did it and what comments they added.
One minor tweak to commands.cfg soon fixed that - thanks to the macros $SERVICEACKAUTHOR$, $HOSTACKAUTHOR$, $SERVICEACKCOMMENTS$ and $HOSTACKCOMMENTS$ - described in more detail in the documentation
I'll post the snippets onto the HepSysMan wiki soon...
One minor tweak to commands.cfg soon fixed that - thanks to the macros $SERVICEACKAUTHOR$, $HOSTACKAUTHOR$, $SERVICEACKCOMMENTS$ and $HOSTACKCOMMENTS$ - described in more detail in the documentation
I'll post the snippets onto the HepSysMan wiki soon...
Maximum Queable Jobs Bites Back
One of our local ATLAS users wanted to submit 2000 jobs onto the system, which I thought would be ok. Unfortunately he hit the 1000 max_queuable limit, and started having jobs fail. Worse, other ATLAS jobs could also not be queued and we failed quite a few of Steve's tests.
Another unexpected issue was that max_queuable seems to apply to running+queued, which was rather unexpected.
Reconsidering the issue I have decided to set the max_user_queuable parameter to 1000 on each queue instead.
This will prevent users from DOSing their entire VO, but should prevent accidents taking out the CE.
Another unexpected issue was that max_queuable seems to apply to running+queued, which was rather unexpected.
Reconsidering the issue I have decided to set the max_user_queuable parameter to 1000 on each queue instead.
This will prevent users from DOSing their entire VO, but should prevent accidents taking out the CE.
Tuesday, September 04, 2007
Glasgow goes dark
Glasgow computing service lost power to the main campus routers this morning - and the UPS didn't work. So Glasgow was down completely for 3 or so hours - failing everything. Durham and Edinburgh were failing RM tests because the Glasgow BDII could not be contacted.
Single points of failure, eh! It's long been a Glasgow complaint that only one BDII can be specified, with no fail over.
Single points of failure, eh! It's long been a Glasgow complaint that only one BDII can be specified, with no fail over.
Sunday, September 02, 2007
Durham Problems
Durham had a 2 day downtime last week as their network was down. Their central services were having a significant outage. A knock on effect from this seemed to be DNS issues which affected the site at the end of the week in a way that was not explained to me.
However, by the weekend the site was back up and running, so hopefully everything's ok now!
However, by the weekend the site was back up and running, so hopefully everything's ok now!
The Quiet Before CHEP...
Blog posts have been a bit thin on the ground this week, both because of GridPP 19 and the sense that CHEP was looming and the poster had to get done and the paper had to get written.
Well, after a few delays and some severe jet lag I did arrive and with a poster too. If you want a quick preview I put the poster's pdf on the web.
Weather is good, but my phone can't find a network, so don't try calling!
Well, after a few delays and some severe jet lag I did arrive and with a poster too. If you want a quick preview I put the poster's pdf on the web.
Weather is good, but my phone can't find a network, so don't try calling!
Ambleling in Ambleside
GridPP 19 took place last week in Ambleside. The Lake District looked beautiful, and although I couldn't stay for the Thursday afternoon ramble, I managed to walk on Loughrigg Fell on Wednesday morning and a quick hike up Orrest Head before getting the train back to Glasgow.
Helped, I think, by the pleasant environment, the meeting was very relaxed, but also very useful as the discussion sessions were very helpful with lots of good contributions.
My ScotGrid talk was a report on ScotGrid and a slightly humorous review of the life of ScotGrid.
Helped, I think, by the pleasant environment, the meeting was very relaxed, but also very useful as the discussion sessions were very helpful with lots of good contributions.
My ScotGrid talk was a report on ScotGrid and a slightly humorous review of the life of ScotGrid.
Friday, August 17, 2007
TPM Round Up
It was TPM week again. Probably not spend more than about 8 hours over the week dealing with things, so it remains a background task against which other things can be done. Between Pete and myself we probably had quite a reasonable response time. Spam tickets continue to be greatly annoying - they amounted to ~30-40% of all tickets (I also cleaned them up when they'd been submitted to VO support units, when I saw them).
Pheno Attacks!
Well, to round off a dreadful week in terms of users slapping the whole system about, the CE load spiked and we hit a System CPU storm again this afternoon. Fortunately this time I caught it within an hour and killed off the offending processes, and the system managed to recover ok.
Very oddly it was caused by a pheno user's gatekeeper processes stalling - gobbling CPU, but failing to submit any jobs onto the queue. I had to add their DN to the banned list and kill off these processes pending further investigation.
Action plan:
- Nagios alarm on cpu_system > 10%.
ScotGrid and AliEn
After a long hiatus, Dan and I sat down to look at getting the AliEn system submitting properly into ScotGrid. Battling through OO Perl (yuk!) we found the various points where the system seemed to lack configuration information - this was basically a value for the site BDII to query free slots, a CE queue name and the name of the VO (gridpp). When we'd hacked those values into the LCG.pm AliEn file then things started to look a lot rosier.
However, I think that AliEn is going to want a dedicated queue, because of the way it parses the BDII's output, which inclines us towards setting up a VO for panda sooner rather than later.
However, I think that AliEn is going to want a dedicated queue, because of the way it parses the BDII's output, which inclines us towards setting up a VO for panda sooner rather than later.
Glasgow and Phenogrid
We're trialling shell access for some Durham phenogrid users here at Glasgow, giving them the ability to rsync the latest builds of alpgen onto the cluster then fire off grid jobs against them.
Although in some ways this is a step backwards, the current EGEE/LCG software deployment mechanisms are too cumbersome for the kind of work that most of the Durham users want to do. In giving them access to a much larger tranche of ScotGrid resource we do help to promote grid work over local queues at Durham (which is a smaller older cluster). If it works then there's the motivation to find a more generic solution to the software problem over the whole grid.
Although in some ways this is a step backwards, the current EGEE/LCG software deployment mechanisms are too cumbersome for the kind of work that most of the Durham users want to do. In giving them access to a much larger tranche of ScotGrid resource we do help to promote grid work over local queues at Durham (which is a smaller older cluster). If it works then there's the motivation to find a more generic solution to the software problem over the whole grid.
Durham Network Outage
Durham suffered a network outage last night, losing all connectivity to the outside world. The grid systems themselves recovered well this morning, but we suffered about 7 hours of downtime.
SL4 Upgrade News
There seem to be enough outstanding issues with the SL4 upgrade that we have decided to hold off for now in ScotGrid. We will review the situation again after CHEP (w/o 10 Sept), and if there are no show stoppers at this time, then the week of the 17th will be the upgrade week for Glasgow and Durham.
At Edinburgh the ECDF resource is SL4 anyway, so getting this up and running will be the perfect way to iron out ScotGrid SL4 issues. It also makes more ScotGrid resource availiable, rather than risking currently functioning resources at Glasgow and Durham.
At Edinburgh the ECDF resource is SL4 anyway, so getting this up and running will be the perfect way to iron out ScotGrid SL4 issues. It also makes more ScotGrid resource availiable, rather than risking currently functioning resources at Glasgow and Durham.
Tuesday, August 14, 2007
Bad, bad, biomed....
A very flaky day - we had a biomed user throw jobs into the system which were trashing workernodes by filling up /tmp. This caused lots of nodes to get into a weird state where they seemed to run our of memory (ssh and ntp nagios alarms firing). Jobs couldn't start properly on these nodes, so they became black holes: one of our local users lost 47 of 50 jobs, another lost 124 of 150.
We then started to fail SAM tests, drop out of the ATLAS BDII, Steve's tests then couldn't resource match us, and so on. Bad day.
It took quite a few hours to sort out the mess, and a further few hours to stabalise the site.
Our GGUS ticket names and shames.
It's a very different ball game on the grid - we have 6000+ users who can submit jobs, and it's not hard to kill a worker node. The torque generally handles this badly and all hell breaks loose.
Action plan:
We then started to fail SAM tests, drop out of the ATLAS BDII, Steve's tests then couldn't resource match us, and so on. Bad day.
It took quite a few hours to sort out the mess, and a further few hours to stabalise the site.
Our GGUS ticket names and shames.
It's a very different ball game on the grid - we have 6000+ users who can submit jobs, and it's not hard to kill a worker node. The torque generally handles this badly and all hell breaks loose.
Action plan:
- Nagios alarms on WN disk space
- Group quotas on job stratch area
DPM Gridftp Resource Consumption
Durham was suffering from excessive resource consumption, from "hung" dpm.gsiftp connections from ATLAS transfers. Because of the way that gridftp v1 servers work huge buffers were being held in memory, leading to resource exhaustion on the machine and a subsequent crash.
Phil and I discussed this, and I noticed that the active network connections were to the RAL FTS server, not to the source SRM, so it looked like it was the control channel which was hung open, not the data channel.
Greig had a look on Glasgow's servers and discovered the same problem, but we were relatively unaffected due to the whopping 8GB of RAM we have in each disk server (and by having 9 disk servers, presumably). Cambridge also reported problems.
The issue is being looked at by the DPM developers, but for the moment Phil's had to write a cron script to kill off the hung ftps to keep gallow's head above water.
Maximum Queable Jobs
From our two phenogrid DOS attacks, it seems that the maximum number of queued jobs the system can cope with is about 2500. After this the system slides into a crisis, running out of CPU with too many gatekeeper processes active and a context switch storm starts - from which the system can rarely spontaneously recover, it seems.
So, I have set a max_queueable parameter on every queue of 1000, which seems a reasonable number for any single VO or queue.
It seems a limitation of torque that it cannot also have a global cap on queued jobs (at 2500, for instance), but this is only a parameter settable for queues.
Monday, August 13, 2007
MonAMI keeps on working...
Gosh, I tickled pink at how well MonAMI coped with Glasgow's job storm last Thursday.
To put this in context, we had a CE that was running at a constant 100% CPU usage: 50% in kernel (context-switching) and 50% in user-land (running Perl scripts). The ssh daemon wasn't working properly any more: the ssh-client would (almost always) time out because the ssh server would taking too long to fork. The machine's 1-min load average peaked at ~300!
All in all, this was an unhappy computer.
Despite all this, MonAMI just kept on going. As matters got progressively worse, it took longer and longer to gather data, particularly from Maui. From the normal value of less than a second, the Maui acquisition time peaked at around 15 minutes. Torque's faired better, peaking at around 30s (still far longer than normal).
Despite this, MonAMI didn't flood Torque or Maui. It only ever issuing one request at a time and enforced a 1-minute gap between successive requests. MonAMI also altered it's output to Ganglia to compensate for taking 15-times longer than normal. This prevented Ganglia from (mistakenly) purging the metrics.
So, although everything was running very, very slowly and it was difficult to log into the machine, the monitoring kept working and we have a record of what was happening to the jobs.
Incidentally, the failing ssh is why most (all?) of the jobs were going into wait-state: the worker node mom daemons couldn't stage-in, via scp, the files the jobs needed. This would fail the job being accepted by the WN, causing torque (or maui?) to reschedule the job for some time in the future, putting the job in to wait-state.
To put this in context, we had a CE that was running at a constant 100% CPU usage: 50% in kernel (context-switching) and 50% in user-land (running Perl scripts). The ssh daemon wasn't working properly any more: the ssh-client would (almost always) time out because the ssh server would taking too long to fork. The machine's 1-min load average peaked at ~300!
All in all, this was an unhappy computer.
Despite all this, MonAMI just kept on going. As matters got progressively worse, it took longer and longer to gather data, particularly from Maui. From the normal value of less than a second, the Maui acquisition time peaked at around 15 minutes. Torque's faired better, peaking at around 30s (still far longer than normal).
Despite this, MonAMI didn't flood Torque or Maui. It only ever issuing one request at a time and enforced a 1-minute gap between successive requests. MonAMI also altered it's output to Ganglia to compensate for taking 15-times longer than normal. This prevented Ganglia from (mistakenly) purging the metrics.
So, although everything was running very, very slowly and it was difficult to log into the machine, the monitoring kept working and we have a record of what was happening to the jobs.
Incidentally, the failing ssh is why most (all?) of the jobs were going into wait-state: the worker node mom daemons couldn't stage-in, via scp, the files the jobs needed. This would fail the job being accepted by the WN, causing torque (or maui?) to reschedule the job for some time in the future, putting the job in to wait-state.
Welcome Back! (sic): Picking Up The Pieces
Fool that I am, I opened my laptop after getting back from Gairloch on Saturday night. As I now have Paul's MonAMI torque plots on my Google homepage, I could see that the number of running jobs was down to almost zero. This was unexpected. A quick revision of the SAM pages and monitoring plots showed the pheno job storm on Thursday had killed the CE off, big time. Being throughly disinclined to engage in extensive debugging late on Saturday night, and knowing the machine needed a new kernel anyway, I rebooted the CE. Whatever the residual problem was this cleared it. Within minutes LHCb jobs were coming in and starting properly - indeed within about 6 hours they managed to fill the entire cluster again.
The remaining problem was then timeouts on the CE-RM test. This was puzzling, but not critical for the site, so I left things as they were at this point, pending further investigation. Then I recalled that this can happen if a period of intense DPM stress (Greig and Billy have been preparing the CHEP papers) causes threads in DPM to lock up or block, leaving few threads available to service requests. I restarted the DPM daemon and bingo! all was well again. The next time this happens we should look in MySQL for pending requests (which get cleared when DPM restarts), however at 10pm on Sunday night getting things working quickly is all I care about.
My car broke down yesterday too, so I'm off to Halfords to buy it some new brake pads; but at least that happened after I came back from holiday. If only the site had the good grace to do the same. I think I get to be grumpy about this for at least 2 days.
The remaining problem was then timeouts on the CE-RM test. This was puzzling, but not critical for the site, so I left things as they were at this point, pending further investigation. Then I recalled that this can happen if a period of intense DPM stress (Greig and Billy have been preparing the CHEP papers) causes threads in DPM to lock up or block, leaving few threads available to service requests. I restarted the DPM daemon and bingo! all was well again. The next time this happens we should look in MySQL for pending requests (which get cleared when DPM restarts), however at 10pm on Sunday night getting things working quickly is all I care about.
My car broke down yesterday too, so I'm off to Halfords to buy it some new brake pads; but at least that happened after I came back from holiday. If only the site had the good grace to do the same. I think I get to be grumpy about this for at least 2 days.
Thursday, August 09, 2007
Pheno goes bang, take two!
A problem started at about 02:45 this morning. The large number of pheno jobs that had accumulated in queued state started fail when run. Once failed, the job would go into waiting state, triggering maui to decide which job to run next.
With the current usage and fairshares, Maui's decision is to run the (apparently) broken pheno jobs. This keeps the server-load high and starves the cluster of long-running jobs (there's been 1-min avr load spikes of over 600!).
Look familiar? Here's a entry with very similar symptoms.
I'm in the process of trying to get to the bottom of what's actually happening, but I've started deleting the jobs as they clearly cannot run and are causing a detrimental effect on the cluster.
Wednesday, August 01, 2007
supernemo.vo.eu-egee.org ok
Gianfranco had tried to run some supernemo test jobs through Glasgow and had not succeeded last week. I addition I was concerned that some VO members (him included) were not in the grid-mapfile.
Of course, it turns out that Gianfranco, and others are also in other VOs - and as the grid-mapfile can only contain one entry, so they were in their other VOs. In addition, Gianfranco had been using a vanilla proxy to submit the job, the gatekeeper then mapped him to ATLAS and thus the job submission to the supernemo queue failed.
I emailed him the correct VOMS client files for supernemo, so that he could generate a VOMS proxy. After he did that his test jobs ran fine.
So, our first DNS VO, supernemo.vo.eu-egee.org works.
Of course, it turns out that Gianfranco, and others are also in other VOs - and as the grid-mapfile can only contain one entry, so they were in their other VOs. In addition, Gianfranco had been using a vanilla proxy to submit the job, the gatekeeper then mapped him to ATLAS and thus the job submission to the supernemo queue failed.
I emailed him the correct VOMS client files for supernemo, so that he could generate a VOMS proxy. After he did that his test jobs ran fine.
So, our first DNS VO, supernemo.vo.eu-egee.org works.
Monday, July 30, 2007
CIC Portal Reports
As I was on holiday on Friday, I tried to fill in the CIC portal report quite late on and it was locked.
Frankly the interface on the report section of the CIC is really rather rubbish (no aggregation, unreliable locking, difficult to review) and the 1 day time window has always been restrictive.
As it's clear now that the CIC "availability", where sites get to mark up failures as relevant/non-relevant/unknown, is a thing of the past (gridview will be used, warts and all...), the whole thing looks rather broken as a way of us telling Jeremy and Phillipa issues to report in the Ops meeting.
However, I checked the gridview page. Looks like a quiet week. Our one CE test failure was the infamous Globus 79...
Report: Quiet week. Ran lots of jobs ;-)
Monday, July 23, 2007
Job submission with DIRAC
In order to get some "real user" experience of performing physics analysis on the Grid, I have been doing a lot of reading and playing with the LHCb computing software. First of all, there's a lot of it so it takes a while to get to understand what each component does, how they can be linked together, how they are configured and built and how the applications can be run locally or on the Grid to do some real physics.
I was particularly interested in getting some basic jobs running on the Grid, so I quickly started playing with Ganga, the user interface for job configuration and submission. At first I was quite impressed. It was very simple to use Ganga to submit small jobs to the local system, CERN batch or the Grid (via the LHCb DIRAC workload management system). However, a few problems quickly appeared:
1. Jobs were continually failing on the Grid due to poorly configured software installations on the sites. Missing libraries was the main source of problems. It also seems that the latest version of Gauss v30r3 (LHCb MC generation) is a bit broken due to a mis-configured path. These things weren't a problem with Ganga as such, but using it meant that another layer had to potentially be debugged.
2. I found bulk job submission was very difficult in Ganga. Writing the python code to loop over the jobs is easy, but the client just couldn't handle the 100's of jobs going through it. It became very slow and eventually just hung. Even just starting up the client is slow. Maybe running on a non-lxplus machine would be better. There were also inconsistencies between the Ganga job monitoring and that reported by DIRAC.
As an alternative, I decided to bypass Ganga and use the DIRAC API directly. This proved to be quite successful, being much faster for bulk submission. I put together some notes on this, which can be found here:
http://twiki.cern.ch/twiki/bin/view/Main/LHCbEdinburghGroupDIRACAPI
Using DIRAC didn't help with the site mis-configurations (although it is easy to get the job output and check the log files for problems), but I found it a more efficient way of working. I'll try again with Ganga once I understand better the problems that keep on appearing on the Grid.
From my brief foray into running jobs on the Grid, it appears that Ganga/DIRAC do insulate users from malfunctioning middleware, however, there are still real problems when it comes to poorly installed software on the sites. From a deployment point of view, maybe this should be taken as encouragement, as the problem is at the application level and not so much with the middleware. I think we would need to do a more systematic study to find this out (much like Steve's ATLAS jobs).
What is needed is better testing of the sites by through VO-specific SAM tests. This information then has to be fed back into DIRAC (or whatever) so that mis-configured sites can be ignored until their problems are resolved. User will then find running jobs on the Grid a much easier and pleasant experience.
I was particularly interested in getting some basic jobs running on the Grid, so I quickly started playing with Ganga, the user interface for job configuration and submission. At first I was quite impressed. It was very simple to use Ganga to submit small jobs to the local system, CERN batch or the Grid (via the LHCb DIRAC workload management system). However, a few problems quickly appeared:
1. Jobs were continually failing on the Grid due to poorly configured software installations on the sites. Missing libraries was the main source of problems. It also seems that the latest version of Gauss v30r3 (LHCb MC generation) is a bit broken due to a mis-configured path. These things weren't a problem with Ganga as such, but using it meant that another layer had to potentially be debugged.
2. I found bulk job submission was very difficult in Ganga. Writing the python code to loop over the jobs is easy, but the client just couldn't handle the 100's of jobs going through it. It became very slow and eventually just hung. Even just starting up the client is slow. Maybe running on a non-lxplus machine would be better. There were also inconsistencies between the Ganga job monitoring and that reported by DIRAC.
As an alternative, I decided to bypass Ganga and use the DIRAC API directly. This proved to be quite successful, being much faster for bulk submission. I put together some notes on this, which can be found here:
http://twiki.cern.ch/twiki/bin/view/Main/LHCbEdinburghGroupDIRACAPI
Using DIRAC didn't help with the site mis-configurations (although it is easy to get the job output and check the log files for problems), but I found it a more efficient way of working. I'll try again with Ganga once I understand better the problems that keep on appearing on the Grid.
From my brief foray into running jobs on the Grid, it appears that Ganga/DIRAC do insulate users from malfunctioning middleware, however, there are still real problems when it comes to poorly installed software on the sites. From a deployment point of view, maybe this should be taken as encouragement, as the problem is at the application level and not so much with the middleware. I think we would need to do a more systematic study to find this out (much like Steve's ATLAS jobs).
What is needed is better testing of the sites by through VO-specific SAM tests. This information then has to be fed back into DIRAC (or whatever) so that mis-configured sites can be ignored until their problems are resolved. User will then find running jobs on the Grid a much easier and pleasant experience.
Return of Globus Error 79...
Glasgow suffered a bit from the infamous Globus Error 79 last week - the one we think might be the unexplained gatekeeper identity error. In fact it was bit of a flaky week altogether - Steve Lloyd's tests seemed to be suffering from some RB issues and most sites dropped into the 70% efficiency range last week.
Overall though, even the "bad" weeks are not so bad as the gridview availability plot shows (remember this is a "warts and all" plot - no excuses or chances to mark things as non-relevant) we seem to still be 95% plus.
However, the gatekeeper error is a real pest and I still don't have a good way of even trying to get a handle on it. I checked in the gatekeeper logs, against some known error events of this type (David found these on 11 June), but alas there's no clear signature.
Holiday Time: Durham Survives
Phil's been away for the last three weeks, with responsibility for Durham falling between myself (to respond to tickets and advise on grid problems) and Lydia (on the ground to press the buttons). This has worked pretty well - we have managed to deal with helmsley locking up (week 3 in the graph) and needing rebooted and a period of scheduled downtime (week 4, which revealed a problem in downtime synchronisation between SAM and the GOC).
A sterner test will come at Glasgow in a fortnight when Andrew and I are both on holiday and are more or less uncontactable. Time for icons and prayers?
ECDF Update
The procurement of our grid front end nodes for ECDF has been held up for about 2 weeks. Sam had tried to setup one of the old Edinburgh worker nodes as a trial CE to iron out any glitches in the gatekeeper scripts, job manager, accounting chain, but has hit an ACL in some router between ECDF and KB which prevents him from being able to even qsub.
The systems team are going to give him a requisitioned worker node to test with instead, which should happen this week - although as they are going live today and it's holiday time this has also been somewhat held up.
The systems team are going to give him a requisitioned worker node to test with instead, which should happen this week - although as they are going live today and it's holiday time this has also been somewhat held up.
Friday, July 20, 2007
SAM Gets Downtime Wrong?
Durham were in downtime from Wednesday -> Thursday, but SAM thinks they were in downtime from Thursday -> Friday. Doubtless this happened because I made the initial mistake (out by 1 day!), then edited the downtime to bring it forward. But SAM did not pick up the change.
I've raised a GGUS ticket - after all I'm always editing downtimes!
Glasgow to Edinburgh Lightpath Approved
After messing about for ages with application forms and revised procedures, our application, when finally submitted to JANET, was approved within 24 hours! They are now conducting a technical feasibility study, but no news on how long that might take.
Update: JANET people say this study will take "the shorter end of 'a few weeks'", which is good news.
Update: JANET people say this study will take "the shorter end of 'a few weeks'", which is good news.
GridICE Ate My CPU...
After upgrading the CE yesterday, the CPU and load were rather high. Re-running YAIM had re-enabled the GridICE monitoring system, which had merrily decided to swallow an entire CPU itself.
When I switched if off CPU load on the CE dropped from 70% to 20%. (See ganglia plot - the difference is pretty obvious.)
Although GridICE gives some interesting monitoring information and aggregation at the Grid level, it's a duplication of information elsewhere (like gstat) and consuming a whole CPU is absurd.
I put in a GGUS ticket about this, but for the moment GridICE is disabled on our CE.
Postscript:
I've added this to cfagent.conf:
processes:
ce::
"/opt/gridice/monitoring/bin/*" signal=term
"/opt/edg/sbin/edg-fmon-agent" signal=term
Thursday, July 19, 2007
Glasgow Updated to gLite 3.0.1 r27
Upgrade notes:
Basically I'm following
https://www.gridpp.ac.uk:443/wiki/UKI-SCOTGRID-GLASGOW_enabling_VO,
but being also aware that when services are updated YAIM needs rerun.
Preparations
------------
Added new groups for sgm and prd pool accounts - even when these will
not yet be enabled
Modified the poolacct.py script - now much improved and does the new
type of accounts (but can still do the old type!). Also now does
user.conf fragments as well.
Using this, added relevant entries to passwd, group, shadow and
users.conf.
Went through site-info.def again. Added supernemo stanzas and the
vo.d/supernemo.vo.eu-egee.org definitions.
All set!
Disk Servers
------------
Modify update.conf to clear cruft out of system (pruge=true)
Clear and restore yum.repos.d.
yum update
Remove local config_mkgridmap function
Run /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def
-f config_mkgridmap
Remove MySQL-server, which had bizarrely been installed on some of the
nodes.
Noticed that config_lcgenv is broken for DNS VOs. Submitted ticket
24917, with patch. However, for us this is controlled by cfengine, so
defined appropriate things here: e.g.,
VO_SUPERNEMO_VO_EU_EGEE_ORG_DEFAULT_SE.
Worker Nodes
------------
Changed passwd/group files have already triggered home directory
creation via cfengine. Thus no need to run YAIM.
Ran yum update on WNs (using pdsh). Removed glite-SE_dpm_disk,
DPM-gridftp-server and DPM-rfio-server RPMs - relic of !disk037
(woops). WNs will be blown and rebuilt at transition to SL4 anyway.
MON/Top Level BDII
------------------
Checked svr019 (MON + Top BDII). Nothing to update here. (Top level
BDII was done last week:
http://scotgrid.blogspot.com/2007/07/top-level-bdii-updated-to-glue-13yaim.html)
Site BDII/svr021
----------------
Did yum-update then reran YAIM. Got error
SITE_SUPPORT_EMAIL not set
and
chown: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
chmod: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
???
Odd entry has appeared:
GIP file:///opt/lcg/libexec/lcg-info-wrapper
Which is not active on a stand alone site BDII (would work on CE?)
Note that DPM not yet upgraded, so still polling GRIS on svr018 until
this is done.
UI / svr020
-----------
yum update
rerun yaim - discovered that i need to defined supernemo queue to be
snemo in $QUEUES (which is still used). Caused gLite python to throw
an exception (my mistake, but crap code nonetheless...)
Corrected site-info.def and reran YAIM.
CE / svr016
-----------
Checked list of RPMs to update. Potentially dangerous ones are
vdt_globus_jobmanager_pbs (we have patched pbs job manager).
Seems that there are patches to pbs jobmanager to support DGAS
accounting. I have commented out the cfengine job manager replacement
and will diff and repatch as necessary after configuration.
Created cdf and snemo queues using torque_queue_cfg script (note this
now adds access for sgm and prd groups, even if they are not used).
Ran YAIM.
Immediate sanity check:
GRIS is ok - information system is up.
gatekeeper dead
Help! Restarted.
globus.conf was rewritten, blowing away pbs job manager
re-enabled pbs job manager and restarted
OK, so I moved under the control of
cfengine, but again caveats about running YAIM
apply.
gatekeeper restarted again
Now tailing gatekeeper logs, everything looks ok.
Whew!
Diffing the pbs and lcgbps job managers, YAIM has added DGAS support
for them. I used these as new template modules and repatched the
"completed" job state (https://savannah.cern.ch/bugs/?7874). Helper.pm
was unchanged, so still has the correct patch for stagin via globus.
svr023 / RB
-----------
Ran yum update. Actual RB has not changed, so did not run YAIM
svr018 / DPM
------------
Upgraded YAIM to check on config_DPM_upgrade. Looks quite simple.
Booked downtime for 2pm to do this.
At 2pm: Stopped DPM
yum update
run config_DPM_upgrade YAIM function (updated db). Took ~8 minutes.
Start DPM again
run config_gip YAIM function (publish access details for
cdf/supernemo in info system)
run config_mkgridmap YAIM function (add additional certificates
into gridmap files)
run config_BDII to redo information system
servers)
Checked BDII was ok. It is.
Then went back to site BDII, changing URL to
BDII_DPM_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" -
restarted site BDII. Checked ldap info was ok.
Came up from downtime (took 15 minutes). Damn - we got SAM tested in the interval!
After Lunch
-----------
Wary of APEL changes I read Yves' notes in
http://www.gridpp.ac.uk/wiki/GLite_Update_27. I couldn't see the same
problems. Ran APEL publisher on CE and MON and things seem to be ok,
so let things lie here.
Finally able to lock atlas pool down to atlas members! When I looked
in the Cns_groupmap though, there had been rather an explosion of
atlas groups:
mysql> select * from Cns_groupinfo where groupname like 'atlas%';
+-------+------+-----------------------+
| rowid | gid | groupname |
+-------+------+-----------------------+
| 2 | 103 | atlas |
| 16 | 117 | atlas/Role=lcgadmin |
| 17 | 118 | atlas/Role=production |
| 55 | 156 | atlas/lcg1 |
| 57 | 158 | atlas/usatlas |
+-------+------+-----------------------+
Hmmm, I have given the pool to all these gids. Is this really
necessary?
Done! Whew!
Basically I'm following
https://www.gridpp.ac.uk:443/wiki/UKI-SCOTGRID-GLASGOW_enabling_VO,
but being also aware that when services are updated YAIM needs rerun.
Preparations
------------
Added new groups for sgm and prd pool accounts - even when these will
not yet be enabled
Modified the poolacct.py script - now much improved and does the new
type of accounts (but can still do the old type!). Also now does
user.conf fragments as well.
Using this, added relevant entries to passwd, group, shadow and
users.conf.
Went through site-info.def again. Added supernemo stanzas and the
vo.d/supernemo.vo.eu-egee.org definitions.
All set!
Disk Servers
------------
Modify update.conf to clear cruft out of system (pruge=true)
Clear and restore yum.repos.d.
yum update
Remove local config_mkgridmap function
Run /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def
-f config_mkgridmap
Remove MySQL-server, which had bizarrely been installed on some of the
nodes.
Noticed that config_lcgenv is broken for DNS VOs. Submitted ticket
24917, with patch. However, for us this is controlled by cfengine, so
defined appropriate things here: e.g.,
VO_SUPERNEMO_VO_EU_EGEE_ORG_DEFAULT_SE.
Worker Nodes
------------
Changed passwd/group files have already triggered home directory
creation via cfengine. Thus no need to run YAIM.
Ran yum update on WNs (using pdsh). Removed glite-SE_dpm_disk,
DPM-gridftp-server and DPM-rfio-server RPMs - relic of !disk037
(woops). WNs will be blown and rebuilt at transition to SL4 anyway.
MON/Top Level BDII
------------------
Checked svr019 (MON + Top BDII). Nothing to update here. (Top level
BDII was done last week:
http://scotgrid.blogspot.com/2007/07/top-level-bdii-updated-to-glue-13yaim.html)
Site BDII/svr021
----------------
Did yum-update then reran YAIM. Got error
SITE_SUPPORT_EMAIL not set
and
chown: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
chmod: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
???
Odd entry has appeared:
GIP file:///opt/lcg/libexec/lcg-info-wrapper
Which is not active on a stand alone site BDII (would work on CE?)
Note that DPM not yet upgraded, so still polling GRIS on svr018 until
this is done.
UI / svr020
-----------
yum update
rerun yaim - discovered that i need to defined supernemo queue to be
snemo in $QUEUES (which is still used). Caused gLite python to throw
an exception (my mistake, but crap code nonetheless...)
Corrected site-info.def and reran YAIM.
CE / svr016
-----------
Checked list of RPMs to update. Potentially dangerous ones are
vdt_globus_jobmanager_pbs (we have patched pbs job manager).
Seems that there are patches to pbs jobmanager to support DGAS
accounting. I have commented out the cfengine job manager replacement
and will diff and repatch as necessary after configuration.
Created cdf and snemo queues using torque_queue_cfg script (note this
now adds access for sgm and prd groups, even if they are not used).
Ran YAIM.
Immediate sanity check:
GRIS is ok - information system is up.
gatekeeper dead
Help! Restarted.
globus.conf was rewritten, blowing away pbs job manager
re-enabled pbs job manager and restarted
OK, so I moved under the control of
cfengine, but again caveats about running YAIM
apply.
gatekeeper restarted again
Now tailing gatekeeper logs, everything looks ok.
Whew!
Diffing the pbs and lcgbps job managers, YAIM has added DGAS support
for them. I used these as new template modules and repatched the
"completed" job state (https://savannah.cern.ch/bugs/?7874). Helper.pm
was unchanged, so still has the correct patch for stagin via globus.
svr023 / RB
-----------
Ran yum update. Actual RB has not changed, so did not run YAIM
svr018 / DPM
------------
Upgraded YAIM to check on config_DPM_upgrade. Looks quite simple.
Booked downtime for 2pm to do this.
At 2pm: Stopped DPM
yum update
run config_DPM_upgrade YAIM function (updated db). Took ~8 minutes.
Start DPM again
run config_gip YAIM function (publish access details for
cdf/supernemo in info system)
run config_mkgridmap YAIM function (add additional certificates
into gridmap files)
run config_BDII to redo information system
servers)
Checked BDII was ok. It is.
Then went back to site BDII, changing URL to
BDII_DPM_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" -
restarted site BDII. Checked ldap info was ok.
Came up from downtime (took 15 minutes). Damn - we got SAM tested in the interval!
After Lunch
-----------
Wary of APEL changes I read Yves' notes in
http://www.gridpp.ac.uk/wiki/GLite_Update_27. I couldn't see the same
problems. Ran APEL publisher on CE and MON and things seem to be ok,
so let things lie here.
Finally able to lock atlas pool down to atlas members! When I looked
in the Cns_groupmap though, there had been rather an explosion of
atlas groups:
mysql> select * from Cns_groupinfo where groupname like 'atlas%';
+-------+------+-----------------------+
| rowid | gid | groupname |
+-------+------+-----------------------+
| 2 | 103 | atlas |
| 16 | 117 | atlas/Role=lcgadmin |
| 17 | 118 | atlas/Role=production |
| 55 | 156 | atlas/lcg1 |
| 57 | 158 | atlas/usatlas |
+-------+------+-----------------------+
Hmmm, I have given the pool to all these gids. Is this really
necessary?
Done! Whew!
Wednesday, July 18, 2007
cfengine cruft
Updating the disk servers today, the older ones were coming up with bizarre errors and refusing to update themselves at all. Eventually I tracked this back to stale files in the cfengine cache on the disk servers themselves - old repo definitions which were in conflict with the newer mirrors.
I found that cfengine's copy stanza has a flag, purge, which needs to be set to remove files which are not present in the source. I have now set this in update.conf and the disk servers are busily crunching their way through the backlog of RPMs.
I found that cfengine's copy stanza has a flag, purge, which needs to be set to remove files which are not present in the source. I have now set this in update.conf and the disk servers are busily crunching their way through the backlog of RPMs.
Tuesday, July 17, 2007
Load on CE
I wish I understood what caused these periods of increased load on the CE. They can happen during a long period of job submission into the gatekeeper, as jobmanagers are forked off and then wilt, but that wasn't the case here. Odd...
Thursday, July 12, 2007
Queues Cut Back
I have now cut all the queues on UKI-SCOTGRID-GLASGOW to 36 hours of CPU and wall. The exceptions are:
- gridpp: Our bio user's code, which she didn't write, takes up to 6 days to run
- glee: The engineers claim they need a 28 day queue - we will have to talk to them about that, because it's ridiculous.
- dteam and ops: 6 hours - even that's a bit long...
Clearing Out The Gatekeeper Cruft
As part of the general clean up after this morning's pheno crisis, I cleaned out stuck gatekeeper processes on the cluster. There were about 50 of these processes, which were all in "T" state. This is "traced or stopped", and I presume stopped actually, as nothing would be tracing them. Most of them has an associated child in zombie state.
Wonder why that happens?
Wonder why that happens?
Pheno goes Bang!
Crisis on the cluster this morning. After a long night of job submission by a phenogrid user (putting in more than 1000 jobs) the cluster went into a spasm, where the pheno jobs started to hit wait state en masse. Then what I think happened was that as torque saw each pheno job hit wait, failing to start, it immediately picked the next pheno job, tried to start that, failed, tried to start the next, and so on. This resulted in a load storm within torque (loads >100), which was then not even able to answer normal client queries - so maui locked up and the gip plugin started to timeout.
When I realised what was happening (and the pheno jobs were still coming in) I added the user's DN to the LCAS ban_users.db file. I then carried out some debugging tests, restarting maui, clearing out maui stats files, etc. In the end I saw no option but to qdel the user's waiting jobs, to attempt to take the pressure off torque.
Once the jobs were flushed out the system torque quite quickly started to recover. Maui started to respond again and the GIP plugin could get sensible answers.
Why were the jobs going into waiting state? The error the user seemed to be getting back was "Globus error 158: the job manager could not lock the state lock file." This seems to be an error which crops up when the job is being cancelled. There was a strange mix of jobs from this user - some with VOMS extensions, some vanilla proxy. Was this a problem with proxy renewal and the gatekeeper trying to cancel jobs which it no longer had the right to? The problem kicked in at almost exactly the time that the user's original submission proxy expired and the RB would have renewed it from the RAL MyProxy server. The wrong proxy might well also have affected the ability of the jobs to start - hence the wait crisis being sparked.
After I had been satisfied that the cluster was stable again, I took the user out of the banned list. Their jobs are now flowing back into the cluster, interestingly all with the vanilla proxy now.
I will keep a close eye on things and check that things don't go wrong again.
Postscript: VOMS proxy renewal is broken: http://savannah.cern.ch/bugs/?func=detailitem&item_id=15208
Tuesday, July 10, 2007
Multiple VO Woes
Steve Lloyd and I sat down after lunch today to try and get to the bottom of why his dteam submitted jobs always fail. Strangely this seems to be a RB specific problem. IC always works, Glasgow always fails and RAL seems to come and go.
Using the Glasgow RB we submitted a job to Edinburgh, so that we could trace things through the batch system. The job arrived at Edinburgh, and ran through the batch system. However, it continued to be considered by the RB as
Current Status: Scheduled
Status Reason: Job successfully submitted to Globus
Clearly this was not the case.
We had a good look through the logs on the RB, but there's no particular sign of things going wrong there - although it must be said that the logs are both dense and impenetrable.
When it became clear that there was no easy solution I decided to try and reproduce the problem myself. Now, recall I had joined gridpp a while ago to help our local users and never had any trouble. However, now I can't seem to get a single job running through as a gridpp member - even on the Glasgow cluster. And things are in fact even worse than for Steve, because my gatekeeper process dies almost instantly, so the job never even goes into the batch system:
grep 2007-07-10.14:49:10.0000028268.0000113028 /var/log/messages
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 for /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart on 130.209.239.23
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 mapped to gridpp001 (17601, 10016)
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 has GRAM_SCRIPT_JOB_ID 1184075356:lcgpbs:internal_434559272:9672.1184075355 manager type lcgpbs
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 JM exiting
I'll now try and poke around inside the gatekeeper logs and see if I can come up with any indication why things are going wrong.
And what the hell's this got to do with the RB anyway? It's deeply puzzling and frustrating in equal measure.
Using the Glasgow RB we submitted a job to Edinburgh, so that we could trace things through the batch system. The job arrived at Edinburgh, and ran through the batch system. However, it continued to be considered by the RB as
Current Status: Scheduled
Status Reason: Job successfully submitted to Globus
Clearly this was not the case.
We had a good look through the logs on the RB, but there's no particular sign of things going wrong there - although it must be said that the logs are both dense and impenetrable.
When it became clear that there was no easy solution I decided to try and reproduce the problem myself. Now, recall I had joined gridpp a while ago to help our local users and never had any trouble. However, now I can't seem to get a single job running through as a gridpp member - even on the Glasgow cluster. And things are in fact even worse than for Steve, because my gatekeeper process dies almost instantly, so the job never even goes into the batch system:
grep 2007-07-10.14:49:10.0000028268.0000113028 /var/log/messages
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 for /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart on 130.209.239.23
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 mapped to gridpp001 (17601, 10016)
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 has GRAM_SCRIPT_JOB_ID 1184075356:lcgpbs:internal_434559272:9672.1184075355 manager type lcgpbs
Jul 10 14:49:16 svr016 gridinfo[9672]: JMA 2007/07/10 14:49:16 GATEKEEPER_JM_ID 2007-07-10.14:49:10.0000028268.0000113028 JM exiting
I'll now try and poke around inside the gatekeeper logs and see if I can come up with any indication why things are going wrong.
And what the hell's this got to do with the RB anyway? It's deeply puzzling and frustrating in equal measure.
New Disk Servers Deployed
We've now completed the deployment of our 5 additional disk servers for ATLAS. This takes our SRM space total to 84TB, with 77TB for ATLAS.
There is about 12TB used so far.
There is about 12TB used so far.
Monday, July 09, 2007
Top Level BDII Updated to Glue 1.3/yaim 3.0.1
I upgraded our top level BDII today, so it should now happily understand the Glue 1.3 schema. This was the deadline for the upgrade set in the EGEE operations meeting - BDIIs have to be upgraded from the top down, because they don't like information provided by lower levels to be in a schema they don't understand.
This was a more significant thing it might first seem, as I also went very carefully through our site-info.def file and used the latest 3.0.1 yaim to configure the node.
It turned out that there were significantly fewer changes than I had thought there would be. The main additions seemed to be exposing new BDII configuration variables, and changing the way that queues are configured, to allow for queues accessible to FQANs. Of course, much of this is not relevant to the BDII/MON box, which doesn't care about users or VOs, but it proves that there's no major errors in the new configuration.
I have noticed that the startup time for the BDII is slower, presumably as it's creating all of its index files; however, the performance should be much better.
This was a more significant thing it might first seem, as I also went very carefully through our site-info.def file and used the latest 3.0.1 yaim to configure the node.
It turned out that there were significantly fewer changes than I had thought there would be. The main additions seemed to be exposing new BDII configuration variables, and changing the way that queues are configured, to allow for queues accessible to FQANs. Of course, much of this is not relevant to the BDII/MON box, which doesn't care about users or VOs, but it proves that there's no major errors in the new configuration.
I have noticed that the startup time for the BDII is slower, presumably as it's creating all of its index files; however, the performance should be much better.
Friday, July 06, 2007
Success with MPI
Cracked it! I can now get MPI jobs running on the Glasgow cluster.
First thing to note is that the gatekeeper does not invoke mpirun for the job - this is very good, because it would be almost impossible to get this to work if it did.
The key file is the NODELIST file which the CE will generate and add to the executable's argument list. When given as the argument of the -p4pg option then mpirun will ssh to all of the "slave" nodes and start the binary which is given in the NODELIST file.
By default this breaks for 2 reasons:
There are, however, two problems which I can see.
(For more formal documentation, watch this wiki page....)
First thing to note is that the gatekeeper does not invoke mpirun for the job - this is very good, because it would be almost impossible to get this to work if it did.
The key file is the NODELIST file which the CE will generate and add to the executable's argument list. When given as the argument of the -p4pg option then mpirun will ssh to all of the "slave" nodes and start the binary which is given in the NODELIST file.
By default this breaks for 2 reasons:
- The gatekeeper only copies the job's sandbox into the working directory of the "master" worker node. So on the "slave" nodes the executable isn't present. (N.B. Even though we have a shared data area for our glaNNN accounts, the working directory is always in /tmp and local to the worker node.)
- The executable listed really needs to be a wrapper script, so it's the wrong thing for mpirun to be starting anyway.
- Change to a more sensible shared directory (like $CLUSTER_SHARED).
- Rewrite the NODELIST file so that the name of the correct mpi binary to run is given, instead of the wrapper script itself.
- Invoke mpirun, giving the new NODELIST file.
#! /bin/sh(Hmm, it's splitting that perl one liner in a really nasty way - no line breaks there.)
#
# Argument list is: BINARY -p4pg NODELIST -p4wd PATH
# What's really important for us is the NODELIST file, i.e., $3
cd $CLUSTER_SHARED/mpi
export MYBIN=$1
PGFILE=`pwd`/pgfile.`hostname -s`.$$
echo My Args: $@
echo "----"
echo "Original NODELIST file:
cat $3
echo "----"
cat $3 | perl -ne 'print "$1 $2 /cluster/share/gla012/mpi/$ENV{\"MYBIN\"}\n" if /^([\w\.]+)\s+(\d+)/;' > $PGFILE
echo "----"
echo "New NODELIST file:
cat $PGFILE
echo "----"
/opt/mpich-1.2.7p1/bin/mpirun $MYBIN -p4pg $PGFILE
There are, however, two problems which I can see.
- Accounting. Looking at the torque logs it's clear that only the master node's process is being accounted for. The slave node MPI processes are not accounted for. Do we multiply the master node's CPU and Wall by the node number as an interim measure?
- Orphaned and stray processes. As ssh is used to start the binary on the slave nodes, what happens if the code leaves them behind or they run away?
(For more formal documentation, watch this wiki page....)
Thursday, July 05, 2007
MPI Progress
I am making progress with MPI jobs. I can now get MPICH jobs into the batch system via edg-job-submit and they do get a batch system reservation.
It turned out I had to add MPICH as a GlueHostApplicationSoftwareRunTimeEnvironment in the information system. It's also essential to have GlueCEInfoLRMSType as pbs. It doesn't work if you put torque (it must be the only thing on the grid that actually cares!).
The job wrapper then adds some interesting arguments to the executable:
-p4pg NODELIST -p4wd PATH
Where NODELIST looks like this:
node067.beowulf.cluster 0 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node070 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node102 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node139 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
and PATH is just the working directory for the job. Note the magic number "0" seems to be the place where the job executable runs and "1" are all the nodes where other job slots are reserved for this job.
So clearly the NODELIST file then needs to be taken by mpirun and used to start all the mpi subprocesses. From the EGEE MPI Wiki, the standard method seems to like to use the i2g mpi-start command, so the arguments must be in a form appropriate for it. Open questions remain, though:
Further reading: EGEE-II-MPI-WG-TEC.doc.
It turned out I had to add MPICH as a GlueHostApplicationSoftwareRunTimeEnvironment in the information system. It's also essential to have GlueCEInfoLRMSType as pbs. It doesn't work if you put torque (it must be the only thing on the grid that actually cares!).
The job wrapper then adds some interesting arguments to the executable:
-p4pg NODELIST -p4wd PATH
Where NODELIST looks like this:
node067.beowulf.cluster 0 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node070 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node102 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
node139 1 /tmp/.mpi/https_3a_2f_2fsvr023.gla.scotgrid.ac.uk_3a9000_2fPSn7TiiAJeV6R6w-0vQjtA/./dummy.sh
and PATH is just the working directory for the job. Note the magic number "0" seems to be the place where the job executable runs and "1" are all the nodes where other job slots are reserved for this job.
So clearly the NODELIST file then needs to be taken by mpirun and used to start all the mpi subprocesses. From the EGEE MPI Wiki, the standard method seems to like to use the i2g mpi-start command, so the arguments must be in a form appropriate for it. Open questions remain, though:
- How to get i2g mpi-start to work. When I give it an MPI binary it seems determined to compile it - however this falls over, even though MPICH 1.2.7 is in the path.
- How do I ignore mpi-start and run a pre-prepared MPI binary, which will be what a Glasgow use wants to do.
- How on earth will torque account for all of this properly?
Further reading: EGEE-II-MPI-WG-TEC.doc.
LHCb Stuck Jobs
Coincidentally with drafting the stalled jobs document, we got 23 stalled LHCb jobs last Friday. These jobs had consumed about a minute of CPU then just stopped.
I reported them to lhcb-production@cern.ch and the response from LHCb was very swift and helpful. We did quite a bit of debugging on them - although in the end we had to confess that exactly why these ones had stalled was something of a mystery. At first LHCb thought that NFS might have gone wobbly at our end, so the jobs got stuck reading the VO software. From what I could see this was unlikely, and when NIKHEF, RAL and IN2P3 reported similar problems we were off the hook.
Some useful tools for stuck jobs:
When these jobs are killed it's helpful to poke the stalled process - that way information gets back to the VO. A qdel will see the outputs all lost and the job resubmitted elsewhere, which is far less helpful.
In the end, whatever the bug is, it's down at the 10^-6 level!
Thanks to LHCb for being so responsive.
I also must take my hat off to Paul and his MonAMI torque plugin. His live efficiency plots for the batch system queues made spotting this very easy. In the past this sort of thing would have been noticed on a very hit or miss basis.
I reported them to lhcb-production@cern.ch and the response from LHCb was very swift and helpful. We did quite a bit of debugging on them - although in the end we had to confess that exactly why these ones had stalled was something of a mystery. At first LHCb thought that NFS might have gone wobbly at our end, so the jobs got stuck reading the VO software. From what I could see this was unlikely, and when NIKHEF, RAL and IN2P3 reported similar problems we were off the hook.
Some useful tools for stuck jobs:
- lsof - see what file handles are open
- strace - what's the job doing right now
- gdb - attach a debugger to the code
When these jobs are killed it's helpful to poke the stalled process - that way information gets back to the VO. A qdel will see the outputs all lost and the job resubmitted elsewhere, which is far less helpful.
In the end, whatever the bug is, it's down at the 10^-6 level!
Thanks to LHCb for being so responsive.
I also must take my hat off to Paul and his MonAMI torque plugin. His live efficiency plots for the batch system queues made spotting this very easy. In the past this sort of thing would have been noticed on a very hit or miss basis.
Tuesday, July 03, 2007
Health and Efficiency
As part of investigating the problems of stalled jobs, I have plotted Wall vs. CPU time for ATLAS and LHCb on our cluster.
LHCb jobs are generally quite efficient (as evidenced by their 93% efficiency from the EGEE accounting pages). What's interesting is seeing the cluster of jobs at 11 and 22 hours of CPU time, with a smear in wall clock from prefect efficiency to ~50% (data management stikes again?).
ATLAS jobs have a far more variable profile, with many more short jobs of high efficiency, with a more general, and flatter line out to lower efficiencies. There's a very distinct line of problematic jobs (the spike on the tail).
It seems really that with our new fast CPUs our queue times are really much too long (inherited from the old cluster, if I remember). LHCb and ATLAS both seem happy for queues to be reduced from 96/100 hours to 36/36 hours.
ATLAS Software Week
Last week I was at ATLAS Software Week at CERN.
It was a useful meeting (as ever meeting people and chatting is most important!). Some issues I picked up for ATLAS sites were:
It was a useful meeting (as ever meeting people and chatting is most important!). Some issues I picked up for ATLAS sites were:
- Although 13.0.10 has been released there are quite a few things known broken (event generation, for instance). This means we are stuck with having a lot of "old" ATLAS software releases on our sites. At Glasgow we have 86GB of ATLAS software - more then 60% of the total for all VOs.
- Preparations for Computing System Commissioning and the Final Dress Rehearsal are underway. The start date seems to have slipped (was meant to start this week)? Actually, I must find out what the site involvement schedule actually is.
- The DQ2 data management system was upgraded to 0.3 last week. There were a few teething troubles, but the next release should handle many common problems much better.
- There's pressure not to run too many simulations as part of each job sent to a site - so keep the wallclock down (< 24 hours), but this reduced the file sizes. Small files are a big problem - they are inefficient to transfer and gunge up any tape system. So they should really be merged before any migration to tape. (Problem for CASTOR though, which even puts T0D1 stuff onto tape?)
- Event sizes keep going up. Computing TDR had ESD at 0.5MB, but currently this is 1.6MB (1.8 for MC). Probably a realistic target will be 1.3MB files.
- Memory footprints are rising too. 2GB necessary for simulation and probably a subset of reconstruction jobs too.
- To deal with merging and pile-up jobs worker nodes should now be speced with at least 20GB of disk space per core. At the moment, however, jobs will try and limit their ambitions to 10GB. However, this requirement also seems monotonic, so make sure it's accounted for in forthcoming purchases.
- Queues for ATLAS production should be around 24 to 36 hours of cpu and wall time (N.B. this is on modern CPUs). NIKHEF are currently at 24/36 and I'm going to cut Glasgow back to 36 hours.
- If you see stuck ATLAS jobs try and investigate the problem and report to atlas-comp-oper@cern.ch. This will help cut off the nasty tail in the ATLAS efficiency curve.
Subscribe to:
Posts (Atom)