logs logs logs

Those of you who don't pour over the latest bug reports constantly may have missed that RedHat have fixed 208538 (see http://rhn.redhat.com/errata/RHBA-2008-0703.html)

"logrotate in Red Hat Enterprise Linux 4 did not support the maxage and dateext configuration parameters. Usage of these parameters has been backported and is now available to users of Red Hat Enterprise Linux 4."

basically logrotate-3.7.1-10 works as you'd expect from most other common non-stoneage linuxes and allows logs to be saved in .YYYYMMDD extensions, thus preventing huge renaming sessions nightly and forcing hard-link based backup systems (dirvish) to back up the whole log directory each night.

liblcas_lcmaps fix

There is a fix for the segfaulting bug I reported against the globus gatekeeper and gridftp server, https://gus.fzk.de/pages/ticket_details.php?ticket=35694.

If you download the patched version of liblcas_lcmaps_gt4_mapping.so.0.0.0 and install it to /opt/glite/lib then the problem is resolved.

I applied the patch last night and since then we haven't suffered a single segfault.

Maarten said that the official fix should reach production in the next couple of weeks, but I'm happier having it on site now.

Shaking down the user issues

The upgrade to gLite 3.1 on the UI has brought a couple of surprises, which we're gradually working our way around.

* First, the version of grid-proxy-init supplied at part of VDT seems to behave rather differently. A proxy initialised with this looks like:

svr020:~$ grid-proxy-info
subject  : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart/CN=892101086
issuer   : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
identity : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
type     : Proxy draft (pre-RFC) compliant impersonation proxy
strength : 512 bits
path     : /tmp/x509up_u218012
timeleft : 11:59:57

And the lcg-RB does not like this proxy at all.

Using voms-proxy-init (without VOMS extensions) gives a rather more normal proxy:

subject  : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart/CN=proxy
issuer   : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
identity : /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
type     : full legacy globus proxy
strength : 512 bits
path     : /tmp/x509up_u218012
timeleft : 11:59:58

This hits ganga particularly hard, which renews proxies for you, but uses grid-proxy-init.

The fix for this was to ensure that /opt/glite/bin appears higher in the path than /opt/globus/bin and then to create a soft link from /opt/glite/bin/voms-proxy-init to grid-proxy-init.

* Second problem was with UKQCD client software. Craig reported he was getting errors because of a missing globus rft library (libglobus_rls_client_gcc32dbgpthr.so.0). gLite 3.1 is compiled against VDT 1.6 (as opposed to VDT 1.2) and this library is no longer built. The first attempt at fixing it was to copy across the missing library into /opt/globus/lib. This failed, though, because then it became clear that the 1.2 and 1.6 VDT libraries are not compatible with one another, so the new libraries were missing symbols the QCD software needed. So, in the end, the old libraries were copied, lock, stock and barrel, into /opt/globus-glite30/lib and, with a suitable LD_LIBRARY_PATH, the qcd application would run.
There is a new version of QCDGrid software being built, so hopefully that will be compatible with the new VDT.

* Finally (or the last issue which has come to light), is Dan having trouble with g++. The default gcc-c++ (3.4.1) doesn't play well with his version of NLOJET++. He's going to try again with the old 3.2.3 g++ (a.k.a. g++32). Hopefully this will fix things.

SL4 x86_64 UI now Availiable

I reinstalled the site's svr020 UI on Saturday. This involved an incredible amount of pain related to the bizarre inability of SL4 to properly install GRUB on a linux software RAID partition. Although the machine would install absolutely fine, on reboot it would just halt after the GRUB prompt.

In the end, after tearing my hair out several times (I was working from home on Friday) and trying as many tricks as I could (I even DBANed the disks), I had to retreat from running software RAID1, and fallback to running only on one of the SCSI disks. (As an

That finally gave me a base SL4 install I could work with.

After that, the installation of the SL4 32bit UI was easy - running through cfengine (one little caveat was that the gsisshd restart would kill off the normal sshd on port 22, so that has been disabled).

Then I found that job submission didn't work, because it relies on a 32bit python/C module and the default python is 64bit now. The advice on ROLLOUT was to have a 32bit python higher in the path than /usr/bin/python. This seemed rather bad advice to me, as we'd like to really have 64 bit python - it is a 64 bit system after all! So, instead I decided to change the magic bang path to specifically reference /usr/bin/python32. Initially I tried to use cfengine's editfiles facility to do this. However, anything which is not a completely trivial modification is rather horrendous to do in cfengine (it reminded me of ed, actually), so I eventually abandoned this, and instead wrote a 3 line perl special in the cfengine script sources, and this is called after the RPMs are installed. (In addition to changing the python interpreter it disables the tk graphical interface, for which we don't have any users anyway.)

Finally, I upgraded ganga, and this went fine - ganga runs quite happily with 64 bit python (normally this wouldn't deserve special note, but in the grid world flowers and champagne are in order).

SL4/5 All bets are ON!

In the wake of Glasgow's upgrade to SL4 Andrew and I were quipping about when we would go to SL5. I jokingly said next year, but really thinking in 18 months.

However, after some discussion, Dr Paul Millar contends that more that 50% of UKI sites will still be running SL4 on the stroke of midnight, 1st January 2010.

Dr Millar - I take that bet. I think that more than 50% will be running something more recent that Sl4 on that date.

Further, Paul thinks that there will be at least one UKI site running SL4 in January 2013 (after the end of lifetime for RHEL4). I don't.

In each case the wager is a bottle of Veuve Clicquot Yellow Label.

Glasgow Upgraded to SL4

The upgrade is done! We started passing ops SAM tests at about 2230 last night, and I brought us out of downtime at 2300. That was 12 hours of total downtime. In addition the queues were closed from about 1600 the day before, so that meant we were unavailable for 31 hours. In the grand scheme of things I think, "not bad," for such a major upgrade.

Preparations for the upgrade were rushed, but certainly thorough enough for us to have a fair degree of confidence in the process. By Tuesday night I was able to reboot, rebuild and run jobs through a worker node successfully. Andrew was close to having the new pool account generator done, even if he had wimped out and used perl.

We had decided the plan was to upgrade the worker nodes and bring us out of downtime ASAP, then work on the UI and other less central services.

Here's my synopsis of what went wrong, or didn't behave quite as we expected:

1. We initially tried to reboot the worker nodes in batches of 30. This overloaded dhcp or tftp on svr031, so in fact only 4 nodes were successful in that batch. Subsequently we did batches of 12, which worked fine. We could also put a larger stagger on the powernode reboot script (we had only used 1s).
   Analysis: It was always going to be hard to know what level we could do this until we tried. It was easy to work around. Probably our rebuild time for the whole cluster is ~2-3 hours because of this node throughput limitation.
2. At the last minute I decided to just drop alice and babar to stop us from supporting VOs who just don't, or can't, use us (it's just clutter). However, that change was imperfectly expressed in site-info.def, so on the first batches of nodes YAIM just didn't run.
   Analysis: This was a mistake. Andrew and I should have co-ordinated better and had more time to review the new user information files.
3. There were a few problems with the user information files: sgm and prd accounts weren't initially in the normal VO group. In addition local Glasgow users were in the wrong group. This was fixed pretty rapidly.
   Analysis: As above. This aspect of the preparation was too close to the critical path - and it didn't work first time.
4. The new server certificates were botched initially. Although we were in downtime and it was relatively easy to correct, it was a distraction. Analysis: We need to document local procedures for certificate handling better.
5. We'd been obsessing about the batch worker configurations, with the intention to leave the servers pretty much alone. However, we hadn't twigged that the change to pooled accounts for sgm and prd users would, of course, require the LCMAPS group and grid mapfiles to be updated. As no one on site is an sgm or a prd user this was not picked up during testing. It only came to light once I did a logfile analysis of why ops tests were failing (these are done as an sgm ops user). Later in the evening it became clear that this also had to be done for the DPM disk servers.
   Analysis: If I'd been sharper I would have realised this in advance (but there was a lot on my mind). It would be useful of one of us had a special role to do this testing (gridpp VO would be ideal). However, it would actually have been a terribly hard thing to test, as the site was "live" during the testing phase and this problem's solution implied reconfiguring the CE as well as the pool accounts. Hopefully writing it down here will make us more cognoscent of this next time!
6. Running YAIM automatically if all well and good, but how do we know it's run successfully? We not only had nodes where YAIM jusy hadn't run, we also (and this was the last problem to be fixed), had two bad nodes where the directories in /opt ended up in mode 0700, so were unreadable.
   Analysis: We need to develop a test and alarm system for which attempts to validate the YAIM run. At the moment we're pretty much flying blind. The two proxies which I ended up using yesterday were:
1. Look for files generated by YAIM, e.g., /opt/glite/etc/profile.d/grid-env.sh. There should be a nagios alarm or a cfengine warning if this file absent.
2. Check permissions on directories such as /opt/glite/etc. If this is not readable to a pool account then something has gone wrong.

Summarising, I think a pretty good job was done yesterday. It was a major upgrade and our first significant downtime since last November. If we can keep these sorts of interventions down to the 1-2 day level then the site will continue to be considered a good one.

However, we're working as a team now, rather than me playing Lone Ranger. This makes co-ordination, documentation and testing even more vital. Once Mike comes properly on board his first major task will be to understand and then document how the cluster is run.

Glasgow upgrade to SL4 x86_64 next week

Issues have been gathered and a work plan is in place. The intention is still to upgrade next week, hopefully starting (and finishing) on Wednesday.

ECDF for Beginners

Basically, this is proving far more painful than anticipated. Although the MON/LFC box has been configured, the CE is proving seriously problematic. The ECDF team thought that SL3 was not a winner for GPFS, so Sam tried using the gLite 3.0 CE on top of SL4. This didn't work (not unexpectedly). Although we know that the lcg-CE has been built for gLite 3.1, it's not yet even been released to pre-production, so clearly there's nothing we can use for a production site. So Ewan reinstalled the CE with SL3, in order to install the old gLite 3.0 version. However, it then proved to be very difficult to get GPFS working on SL3, so this is still a work in progress. How long it will take to resolve is anyone's guess.

GPFS is necessary for the software area and the pool account home directories. At this point I would just buy a 500GB disk from PC world and run with that for a month while we wait for the gLite 3.1 CE, but we can't do that with machines other people are running.

Getting the site certified for the end of the month now looks challenging.

Hmmm....

SL4 Upgrade News

There seem to be enough outstanding issues with the SL4 upgrade that we have decided to hold off for now in ScotGrid. We will review the situation again after CHEP (w/o 10 Sept), and if there are no show stoppers at this time, then the week of the 17th will be the upgrade week for Glasgow and Durham.

At Edinburgh the ECDF resource is SL4 anyway, so getting this up and running will be the perfect way to iron out ScotGrid SL4 issues. It also makes more ScotGrid resource availiable, rather than risking currently functioning resources at Glasgow and Durham.