Fun with 5!

So, the SL5 migration was done last week, and as promised at yesterday's dteam meeting I am posting our problems so that other sites can watch out for similar issues (although some of these are deeply related to the way we do things at Glasgow).

First though, the successes:

1. 1800 cores running SL5
2. DPM headnode upgraded to SL5
3. Torque server upgraded to SL5, running torque 2.3.6, maui 3.2.6p21
4. /atlas/uk voms group supported with a separate fairshare
   

Now, the list of problems:

1. We introduced a new python script, gridAccounts.py, to generate pool accounts, retiring the venerable, but incomprehensible, genaccts.pl script we had before (Andy Elwell wrote that and his comment was "OK - I give up with python as I need this NOW..."; my retort was "I HATE PERL SO MUCH. IT'S A SHIT LANGUAGE.", but I had never found the time to rewite it until now). The new script reads standard config files, so it's a lot easier to manage, understand and extend. However, all change is (a bit) dangerous and the new script initially had groups in the wrong order in yaim's user.conf, which caused the groupmapfile to be wrong. This then caused all jobs to fail - the uid/gid of the gridftp session did not match the uid/primary gid of the user and gridftp does not like that at all.

(The reason we have to write users.conf is because we still get yaim to do a lot, although we manage all accounts through cfengine; yaim relies on this file to configure various other aspects of the system, such as grid/group mapfiles.)

2. We were trying to support the /atlas/uk VOMS group as a separate entity. This is simple in theory (!), you're looking for the following entries in voms-grid-mapfile:

"/atlas/uk/Role=NULL/Capability=NULL" .ukatlas
"/atlas/uk" .ukatlas

and this in groupmapfile:

"/atlas/uk/Role=NULL/Capability=NULL" atlasuk
"/atlas/uk" atlasuk

If we were managing these files directly, it would have been no problem. However, convincing YAIM to do this was far from easy. This is not helped by the fact that YAIM is now utterly incomprehensible in many ways (have a look at yaim/utils/users_getvogroup if you don't believe me). Finally we hit on the correct recipe, which is to have these accounts in users.conf, with a new "special" defined:

201601:ukatlas001:201040,201000:atlasuk,atlas:atlas:uk:
201602:ukatlas002:201040,201000:atlasuk,atlas:atlas:uk:
201603:ukatlas003:201040,201000:atlasuk,atlas:atlas:uk:
...

with this line added to groups.conf:

"/VO=atlas/GROUP=/atlas/uk":::uk:

Aside: Sometimes I wonder if YAIM has outgrown its usefulness. From something we could understand and tweak easily it's now a sed|awk|cut|sort|tail black box monster, which uses a computerised format for configuration files. c.f. the configuration we have for our own scripts:

[someuser]
dn = /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=some user
uid = 4832
home = /clusterhome/home/someuser
group = atlas
tier25 = True
vo = gla

And trying to do grid configuration manipulations in a language which doesn't have dictionaries is just ridiculous.

Maybe we'll need to wean ourselves off it eventually?

3. Information publishing on svr018 was broken after the upgrade. There was a cryptic reference to a required 'dpminfo' user which Sam had made in his notes. Adding this user did seem to make things work, though it's not at all clear why. Hopefully Sam will enlighten us later. In passing, note that the resource BDII on the service nodes seems to be 'protected' now, so attempts to reach it from 'outside' fail. This is new behaviour and lost us some time in debugging.

4. Terrible trouble was caused by upgrading the torque server to SL5. Using the SteveT build of torque server (http://skoji.cern.ch/sa1/centos5-torque/) seemed to cause grave problems with moms crashing on the worker nodes. Downgrading the moms to the torque 2.3.0 didn't work as the jobs files (/var/spool/pbs/mon_priv/jobs) seemed to be in incompatible formats and led to crashing moms plus a very confused torque server. Cleaning out all jobs seemed to not work either. The final solution was to rebuild torque 2.3.6 on SL5 - this gave a consistent and compatible server/mon pairing.

A small side effect though, was the the rebuilt maui had a different 'secret' in it, so I have had to hack the info provider on the SL4 CEs to use the --keyfile= argument in the maui client commands. (That's such a stupid 'feature'.)

5. Once we were out of downtime, random transfers to the DPM were failing. Eventually we tracked to the reduction in the number of pool accounts for atlasprd. There was no sync between the passwd fila and to the /etc/grid-security/gridmapdir pool account list, of course, so gridftp was was throwing a "530 Login incorrect. : No local mapping". We realised that

1. /etc/passwd should be handled better on nodes which need to map pool accounts.
2. For the moment never reduce the number of accounts!
3. N.B. on the CEs the gridmapdir is shared, so maintenance probably needs to be delegated
4. If we remove a pool account mapping then you have to remove the link from any DNs to this mapping as well (look for DN filenames with only 1 hard link).
   

OK, that's it. We got there, though not without some anxious moments!

mmm, shiny (take 2)

It's here, It's grey and It's cubic. Oh and it has lots of blinkenlights. (aside: but not as many as here -- go and look at the arcade videos) - its the new cluster!

The Vendor install team took 3 days to get the basics in and on a burn-in, now its up to us to get the build and acceptance done. Mike has been working tirelessly on it and has been 'aquainted' with some of the rougher bits of YPF that we never really finished off (keygeneration, interface config scripts etc --- stuff that only needs to be done once per big group of new hardware delivery)

Anyway - we've now fired off a pile of installs and have node142-310 built and running stress (ta Mike) and I've been working on the disk servers -- they're now in a test pool and will be getting a pile of files written shortly.




what else -- The cold aisle containment seems good -- makes a huge difference to the airflow efficiency - hopefully reflected in the aircon load efficiency.

more news as we go and more pics soon, but you've gotta love the simple dpm-queryconf output of

CAPACITY 363.79T FREE 362.14T (99.5%)

Well Done Guys!

Well, I was waiting for Mike and Andrew to blog this, but they haven't. They very successfully upgraded Glasgow's DPM to the native 64bit version on Monday last week (when we had upgraded to SL4 only the 32 bit version was available). This was a significant step forwards but required the head node and all of the disk servers to have their OS rebuilt without losing data, and the database restored onto the head node.

It went very well and we were up and running again within 6 hours - no data lost!

We are also seeing an improvement in the SAM test results, with the spurious 'gSOAP' errors which were plaguing us now seemingly having gone (fingers crossed!).

It's terrible that the LHC is not running right now, but it does mean that interventions like this can be done.

Great work guys!

First among equals...

We got delivery of a sample WN from Viglen a week or so ago. Andrew and Mike did the cluster magic to integrate it into the system and I decided last night to open it up to some real jobs.

Results: 8/8 successful! (Snapshot from ATLAS panda.)

Unfortunately then ATLAS production dried up in the UK, but when the jobs come back, we're ready!

DPM Upgrade

Mike and I upgraded the DPM yesterday to 1.6.10. There are no schema changes in 1.6.7->1.6.10, so the upgrade involves 'downtime' of about 20s. We didn't put this into the CIC portal, but I suppose in retrospect we should have declared an 'at risk' period.

We're still running the i386 version of DPM (on top of x86_64). At some point it would be desirable to upgrade to x86_64; however, as i386 works just fine and this will involve real downtime, there is no urgent pressure to do so.

Greig has noted that dpm-updatespace in 1.6.10 has a bug in it: https://gus.fzk.de/pages/ticket_details.php?ticket=38330.

Upgrade to gLite 3.1 - Notebook

It was well over a year since we'd done a "from the top" install of the CE, so a few things were different:

• The information system has been re-branded. It's now configured in /opt/glite/etc/gip, although many of the plugins are still running from /opt/lcg.
• The CE information system is upgraded to use the BDII (on 2170).
• The site BDII also now uses a wrapper script to get all information, rather than coding the list of GRISs/BDIIs to query (GIP file:///opt/glite/libexec/glite-info-wrapper).
  
• LCAS and LCMAPS now also run out of /opt/glite.
• Pool account mappings are now done to a random pool account, rather than the "next" free one. In addition the hardlink syntax used for assigning a pool account to a DN has changed slightly (using colons to indicate VOMS attributes after the DN).

Funeral March for the Lost CE

So, here's the post mortem on the CE hard crash on Wednesday last. About 1700 the load on the CE started to ramp up and it quickly rose to almost 100. I could see this happening just as I was about to go home (typical!) so I started to indulge in a frantic bout of process killing to reduce load and bring the CE back under control. However, despite my best efforts, the CE crashed hard at 1800 (gap in the ganglia plot).

When the machine rebooted, the gatekeeper restarted and again the load began to rise. I then went through a frantic couple of hours trying to do everything I could to reduce the load and try an get the CE back on an even keel - this was made very hard by the fact that with load averages quickly rising to 60+ the machine was extremely sluggish.

I shut down R-GMA, turned off the mail server to no avail. I killed off queued jobs in the batch system, even got as far as disabling VOs, and banning users whose jobs I had cancelled. I even got so desparete as to firewall the gatekeeper from all but the ScotGrid RB! But although I coud slow down the load increase by doing this, by 10pm it became clear that something dreadful had happened to the gatekeeper. Every gatekeeper process which was forked stalled, consuming CPU and managing to do absolutely nothing. As there was no response, the RB then contected the CE again, forking off another gatekeeper and the march to death continued. If I reduced the number of users able to contact the CE this slowed down the rate of resource exhaustion, but could not stop it. Clearly something utterly evil had happened to the gatekeeper state.

At this point I became convinced that nothing could be done to save the remaining queued or running jobs and that the site was going down. I started to think instead about moving our March downtime forwards, to do the SL4 upgrades, and to prise the CE and the batch system apart. And of course, that is just what we did at the end of last week.

Thusday Night Status Update

Quick summary of where we are right now:

* YAIM configuration updated and rationalised.

* Batch system has been upgraded to Torque 2.1.9/Maui 3.2.6, running on SL4 x86_64.

* Queues have been reduced to 4, open to most VOs, with queue lengths of 30m, 6h, 3d and 7d.

* CE has been moved to svr021, again running SL4 x86_64.

* Information system has been reconfigured to new gLite versions. After a minor wobble on the CE seems to be working just fine.

* DPM headnode has been upgraded to SL4 x86_64.

And job submission works:

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
Submitted:          Thu Feb 21 23:22:30 2008 GMT
*************************************************************

...

================================================================================

   JOB GET OUTPUT OUTCOME

Output sandbox files for the job:
https://svr022.gla.scotgrid.ac.uk:9000/sx54e7252PGtxJtB4Y2mIg
have been successfully retrieved and stored in the directory:
/tmp/jobOutput/gla012_sx54e7252PGtxJtB4Y2mIg

================================================================================

And there was much rejoycing.

Significant work remains for tomorrow, but very good progress being made.

RB: "Rather Better"

One thing which went to pot during the upgrade, was the way that the higher UIDed pool accounts cascaded through to the RB. This, unfortunately, meant that any jobs which were running on svr023 were lost (there would have been very few, in fact, which is why we spent more efforts on the UI).

However, in our attempts to get the RB back it became clear that the edg-wl-ftpd service (yet another hacked version of GT2 gridftp) cannot handle UIDs > 16bit. This screwed things up for us, as all our new UIDs are in the range 200000+.

In the end I had to re-hack the perl passwd/group/shadow/users.conf generator, lowering all the UIDs specially for the RB. In fact this was not quite as awful as one might think, as the RB supports only a subset of the VOs we run jobs for on the main site. I also scripted up a "generator" for the RB's site-info.def, that strips down the VOS variable to those we support for job submission. In addition, communication between RB and the UI or the CE is of course mediated by certificate, so having a different pooll account or UID on the RB is not a problem.

There was a supporting tweak to cfengine to take passwd-rb (etc.) as the source passwd file for the RB.

Then the RB was blown away and rebuilt. It seems to have done it rather a lot of good, as now Steve Lloyd's dteam test jobs run properly (see his RB test page).

SL4 x86_64 UI now Availiable

I reinstalled the site's svr020 UI on Saturday. This involved an incredible amount of pain related to the bizarre inability of SL4 to properly install GRUB on a linux software RAID partition. Although the machine would install absolutely fine, on reboot it would just halt after the GRUB prompt.

In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an

That finally gave me a base SL4 install I could work with.

After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).

Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)

Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).

SL4/5 All bets are ON!

In the wake of Glasgow's upgrade to SL4 Andrew and I were quipping about when we would go to SL5. I jokingly said next year, but really thinking in 18 months.

However, after some discussion, Dr Paul Millar contends that more that 50% of UKI sites will still be running SL4 on the stroke of midnight, 1st January 2010.

Dr Millar - I take that bet. I think that more than 50% will be running something more recent that Sl4 on that date.

Further, Paul thinks that there will be at least one UKI site running SL4 in January 2013 (after the end of lifetime for RHEL4). I don't.

In each case the wager is a bottle of Veuve Clicquot Yellow Label.

ATLAS jobs running properly

Chris submitted a sample ATLAS job to the cluster, using ppeui and the RAL RB.

It worked!

So in addition to passing the tests, we can also do real work. Tests and working for real jobs have a correspondence, but are not exactly the same (tests being a necessary, but not sufficient condition for doing real work, in gereral) so I am very pleased.

He'll now throw in 500 jobs, so we look forward to having the workload ramp up.

Glasgow Upgraded to SL4

The upgrade is done! We started passing ops SAM tests at about 2230 last night, and I brought us out of downtime at 2300. That was 12 hours of total downtime. In addition the queues were closed from about 1600 the day before, so that meant we were unavailable for 31 hours. In the grand scheme of things I think, "not bad," for such a major upgrade.

Preparations for the upgrade were rushed, but certainly thorough enough for us to have a fair degree of confidence in the process. By Tuesday night I was able to reboot, rebuild and run jobs through a worker node successfully. Andrew was close to having the new pool account generator done, even if he had wimped out and used perl.

We had decided the plan was to upgrade the worker nodes and bring us out of downtime ASAP, then work on the UI and other less central services.

Here's my synopsis of what went wrong, or didn't behave quite as we expected:

1. We initially tried to reboot the worker nodes in batches of 30. This overloaded dhcp or tftp on svr031, so in fact only 4 nodes were successful in that batch. Subsequently we did batches of 12, which worked fine. We could also put a larger stagger on the powernode reboot script (we had only used 1s).
   Analysis: It was always going to be hard to know what level we could do this until we tried. It was easy to work around. Probably our rebuild time for the whole cluster is ~2-3 hours because of this node throughput limitation.
2. At the last minute I decided to just drop alice and babar to stop us from supporting VOs who just don't, or can't, use us (it's just clutter). However, that change was imperfectly expressed in site-info.def, so on the first batches of nodes YAIM just didn't run.
   Analysis: This was a mistake. Andrew and I should have co-ordinated better and had more time to review the new user information files.
3. There were a few problems with the user information files: sgm and prd accounts weren't initially in the normal VO group. In addition local Glasgow users were in the wrong group. This was fixed pretty rapidly.
   Analysis: As above. This aspect of the preparation was too close to the critical path - and it didn't work first time.
4. The new server certificates were botched initially. Although we were in downtime and it was relatively easy to correct, it was a distraction. Analysis: We need to document local procedures for certificate handling better.
5. We'd been obsessing about the batch worker configurations, with the intention to leave the servers pretty much alone. However, we hadn't twigged that the change to pooled accounts for sgm and prd users would, of course, require the LCMAPS group and grid mapfiles to be updated. As no one on site is an sgm or a prd user this was not picked up during testing. It only came to light once I did a logfile analysis of why ops tests were failing (these are done as an sgm ops user). Later in the evening it became clear that this also had to be done for the DPM disk servers.
   Analysis: If I'd been sharper I would have realised this in advance (but there was a lot on my mind). It would be useful of one of us had a special role to do this testing (gridpp VO would be ideal). However, it would actually have been a terribly hard thing to test, as the site was "live" during the testing phase and this problem's solution implied reconfiguring the CE as well as the pool accounts. Hopefully writing it down here will make us more cognoscent of this next time!
6. Running YAIM automatically if all well and good, but how do we know it's run successfully? We not only had nodes where YAIM jusy hadn't run, we also (and this was the last problem to be fixed), had two bad nodes where the directories in /opt ended up in mode 0700, so were unreadable.
   Analysis: We need to develop a test and alarm system for which attempts to validate the YAIM run. At the moment we're pretty much flying blind. The two proxies which I ended up using yesterday were:
1. Look for files generated by YAIM, e.g., /opt/glite/etc/profile.d/grid-env.sh. There should be a nagios alarm or a cfengine warning if this file absent.
2. Check permissions on directories such as /opt/glite/etc. If this is not readable to a pool account then something has gone wrong.

Summarising, I think a pretty good job was done yesterday. It was a major upgrade and our first significant downtime since last November. If we can keep these sorts of interventions down to the 1-2 day level then the site will continue to be considered a good one.

However, we're working as a team now, rather than me playing Lone Ranger. This makes co-ordination, documentation and testing even more vital. Once Mike comes properly on board his first major task will be to understand and then document how the cluster is run.

Yarr! Thats not line noise, it's Perl (me hearties)

Avast Ye Salty Sea-dogs - It be the glasgow upgrade day today (as well as international talk like a pirate day). Graeme gave me a minor task to do - come up with a new userid/groupid/passwd/shadow/yaim config generator script for the pool accounts. Simple enough, should only take half a day or so....

Much swearing at Python later, I gave up at 1AM this morning and resorted to trusty Perl. Done in 48 lines (including comments) and 2 hours. I think Python and I are going to take a looooong time to get aquainted properly.

I'm sure G will blog in more detail - Worker nodes went fairly smothly, few niggles - seems that about 24 simultaneous installs cause tftp timeouts. Discovered the Sandbox dirs on the RB needed their ownerships changing - again another Perl script to the rescue with the trick I learned from Steve Andrews - just get your script to print the command line you'd like to stdout, then once you've checked it looks reasonable, run it again piped through | /bin/sh

Glasgow Updated to gLite 3.0.1 r27

Upgrade notes:

Basically I'm following
https://www.gridpp.ac.uk:443/wiki/UKI-SCOTGRID-GLASGOW_enabling_VO,
but being also aware that when services are updated YAIM needs rerun.

Preparations
------------

Added new groups for sgm and prd pool accounts - even when these will
not yet be enabled

Modified the poolacct.py script - now much improved and does the new
type of accounts (but can still do the old type!). Also now does
user.conf fragments as well.

Using this, added relevant entries to passwd, group, shadow and
users.conf.

Went through site-info.def again. Added supernemo stanzas and the
vo.d/supernemo.vo.eu-egee.org definitions.

All set!


Disk Servers
------------

Modify update.conf to clear cruft out of system (pruge=true)
Clear and restore yum.repos.d.
yum update
Remove local config_mkgridmap function
Run /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def
-f config_mkgridmap
Remove MySQL-server, which had bizarrely been installed on some of the
nodes.

Noticed that config_lcgenv is broken for DNS VOs. Submitted ticket
24917, with patch. However, for us this is controlled by cfengine, so
defined appropriate things here: e.g.,
VO_SUPERNEMO_VO_EU_EGEE_ORG_DEFAULT_SE.


Worker Nodes
------------

Changed passwd/group files have already triggered home directory
creation via cfengine. Thus no need to run YAIM.

Ran yum update on WNs (using pdsh). Removed glite-SE_dpm_disk,
DPM-gridftp-server and DPM-rfio-server RPMs - relic of !disk037
(woops). WNs will be blown and rebuilt at transition to SL4 anyway.


MON/Top Level BDII
------------------

Checked svr019 (MON + Top BDII). Nothing to update here. (Top level
BDII was done last week:
http://scotgrid.blogspot.com/2007/07/top-level-bdii-updated-to-glue-13yaim.html)


Site BDII/svr021
----------------

Did yum-update then reran YAIM. Got error
SITE_SUPPORT_EMAIL not set
and
chown: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
chmod: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
???
Odd entry has appeared:
GIP file:///opt/lcg/libexec/lcg-info-wrapper
Which is not active on a stand alone site BDII (would work on CE?)

Note that DPM not yet upgraded, so still polling GRIS on svr018 until
this is done.


UI / svr020
-----------

yum update
rerun yaim - discovered that i need to defined supernemo queue to be
snemo in $QUEUES (which is still used). Caused gLite python to throw
an exception (my mistake, but crap code nonetheless...)

Corrected site-info.def and reran YAIM.


CE / svr016
-----------

Checked list of RPMs to update. Potentially dangerous ones are
vdt_globus_jobmanager_pbs (we have patched pbs job manager).
Seems that there are patches to pbs jobmanager to support DGAS
accounting. I have commented out the cfengine job manager replacement
and will diff and repatch as necessary after configuration.

Created cdf and snemo queues using torque_queue_cfg script (note this
now adds access for sgm and prd groups, even if they are not used).

Ran YAIM.

Immediate sanity check:
GRIS is ok - information system is up.
gatekeeper dead
Help! Restarted.
globus.conf was rewritten, blowing away pbs job manager
re-enabled pbs job manager and restarted
OK, so I moved under the control of
cfengine, but again caveats about running YAIM
apply.
gatekeeper restarted again
Now tailing gatekeeper logs, everything looks ok.
Whew!

Diffing the pbs and lcgbps job managers, YAIM has added DGAS support
for them. I used these as new template modules and repatched the
"completed" job state (https://savannah.cern.ch/bugs/?7874). Helper.pm
was unchanged, so still has the correct patch for stagin via globus.



svr023 / RB
-----------

Ran yum update. Actual RB has not changed, so did not run YAIM


svr018 / DPM
------------

Upgraded YAIM to check on config_DPM_upgrade. Looks quite simple.

Booked downtime for 2pm to do this.

At 2pm: Stopped DPM
yum update
run config_DPM_upgrade YAIM function (updated db). Took ~8 minutes.
Start DPM again

run config_gip YAIM function (publish access details for
cdf/supernemo in info system)
run config_mkgridmap YAIM function (add additional certificates
into gridmap files)
run config_BDII to redo information system
servers)

Checked BDII was ok. It is.

Then went back to site BDII, changing URL to
BDII_DPM_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" -
restarted site BDII. Checked ldap info was ok.

Came up from downtime (took 15 minutes). Damn - we got SAM tested in the interval!


After Lunch
-----------

Wary of APEL changes I read Yves' notes in
http://www.gridpp.ac.uk/wiki/GLite_Update_27. I couldn't see the same
problems. Ran APEL publisher on CE and MON and things seem to be ok,
so let things lie here.

Finally able to lock atlas pool down to atlas members! When I looked
in the Cns_groupmap though, there had been rather an explosion of
atlas groups:

mysql> select * from Cns_groupinfo where groupname like 'atlas%';
+-------+------+-----------------------+
| rowid | gid | groupname |
+-------+------+-----------------------+
| 2 | 103 | atlas |
| 16 | 117 | atlas/Role=lcgadmin |
| 17 | 118 | atlas/Role=production |
| 55 | 156 | atlas/lcg1 |
| 57 | 158 | atlas/usatlas |
+-------+------+-----------------------+

Hmmm, I have given the pool to all these gids. Is this really
necessary?

Done! Whew!

Urgent Updates Urges

The latest gLite update (r24) is labeled as urgent. Turns out that this is only because the VOMS certificate for lcg-voms.cern.ch is going to expire at the end of the month.

Forunately, we have the lcg-vomscerts RPM directly controlled by cfengine, so it was a simple matter to update this part of the system.

Of course, while we're languishing at r20 the world is marching on, so we'll have to play catch-up sometime. Currently I've sheduled next Thursday as our site upgrade day. I'm glad we have the cluster independent of any other authentication system - it means finding all of those new UIDs for sgm and prd accounts will not be a problem.

(I'm also glad we didn't break our DPM with the current fiasco over gridmap file paths!)

Torque/Maui Upgrade Lost Jobs?

Was the torque upgrade was not as smooth as I'd hoped?

It should have been just a minor upgrade, and thus pretty transparent, but some issues have arisen.

Firstly one of our local engineering users reported that the gatekeeper lost contact with all of his jobs - the jobs were still running in the batch queue, but globus-job-status reported them all done, and he couldn't get any output back.

Then I noticed that on the ATLAS production monitor page our 24 hour efficiency droped to the lowest ever level at 24%. This makes me rather worried that we lost all of our ATLAS jobs if the gatekeeper had a brain haemorrhage.

On the other hand, efficiency in the UK seems generally very low right now (ce02.tier2.hep.manchester.ac.uk, 26%; ce1.pp.rhul.ac.uk, 24%; fal-pygrid-18.lancs.ac.uk, 17%; lcgce01.gridpp.rl.ac.uk, 14%), so perhaps this is just a coincidence?

Doesn't explain the globus issues seen by our local user though.

Does anyone know the magic for getting into the guts of the gatekeeper and seeing which torque jobs it's connected to?

Glasgow Update to gLite 3.0r16

This was the first gLite update with significant component changes:

* DPM was upgraded to 1.6.3, with a schema change and a new SRM v2.2 daemon.
* Torque and Maui were upgraded to v2.1.6 and v3.2.6, respectively, from the previously ancient LCG versions.

The DPM upgrade I tackled first. This was fine and I blogged about it on the storage blog. Just be careful to take a db dump before you try it, just in case things go wrong. There's also a strong warning against running automatic updates on gLite server nodes - this is not supported and some people are reporting DPM database corruption on LCG-ROLLOUT.

The torque/maui upgrade I was a little nervous about as I don't feel I greatly understand these components. However, Steve T had said that minor upgrades are ok (we'd been using the Steve T build for the cluster since the start, so we were already on torque v2), so I took the plunge. First I did a single worker, and restarted pbs_mom, so make sure the 2.1.6 mom didn't have trouble talking to the 2.1.5 server - and it didn't. So then I updated all the WNs, before turning my attention to the server.

Here, I did the usual yum -y update first. Then I restarted pbs_server and maui. pbs_server didn't restart cleanly, claiming something was bound to the port. I had a look, but by the time I did there was nothing - I think it was the server being sluggish to exit. So pbs_server then (re)started fine, and I did an extra maui restart to be on the safe side.

A basic check of the batch system (pbsnodes, qstat, diagnose) looked ok.

I have commented Steve's repository out of by yum.conf - we'll now use the "official" gLite build on Steve's advice.

N.B. I still intend to manage to batch system using cfengine, not YAIM - it's a lot more flexible for us to do this, e.g., the new routing queue being the default one.