Disabling SAME/R-GMA in the Job Wrapper

Alessandra raised a GGUS ticket about the R-GMA client issues in the job wrapper. She got a rapid response, along with a recipe to disable them:

Unfortunately this is a known problem with SAM CE JobWrapper Tests and R-GMA. We are using R-GMA command line utility to publish a small piece of data from worker nodes but unfortunately sometimes R-GMA hangs for quite a lot of time.

We are planning a new release of JobWrapper tests without R-GMA publishing (replaced completely by our internal SAM/GridView transport mechanism). But for the time being the only solution is to disable JobWrapper tests on your site if you observe such a behaviour.

To do this you have to remove all the symlinks that appear in the following two directories on all WNs:

$LCG_LOCATION/etc/jobwrapper-start.d
$LCG_LOCATION/etc/jobwrapper-end.d

I have now put in the necessary cfengine stanza to delete the links and stop this nonsense:

disable:
        worker::
                # Disable the SAM wrapper which uses R-GMA
                /opt/lcg/etc/jobwrapper-start.d/01-same.start
                /opt/lcg/etc/jobwrapper-end.d/01-same.end

Now, how big a difference does it make? Quite a lot for short jobs - the wallclock time for a simple globus-job-run has gone down from 5 minutes to 3 seconds!

The total time for the jobmanager to handle the job has remained quite high - 1m30s c.f. 16s for the pbs jobmanager. However, at least no one is going to be "charged" for the time that the job is with the gatekeeper, unlike the time spent in the batch queue.

Last VO Enabling Twiddle

One last thing, making a VO Tag Directory.

Now it looks like it's all working fine.

GRIS Wobbles in Glasgow CE

Between 12 and 1pm yesterday the site BDII stopped reporting on the queue statuses. We suffered the classic problem of reporting 4444 queued jobs and 0 job slots available.

You can see from the plots from the sBDII that the amount of network traffic clearly dips. In fact the gstat graphs show clear a dip in the number of entries. So it was the CE's GRIS was misbehaving at this point.

I checked the CE for load, job floods, etc. There was nothing abnormal - we got 20 jobs in 1 minute, but we should be able to cope with that ok. There are no logs for the running slap daemon and nothing odd spotted in /var/log/messages.

We also aborted on a few of Steve Lloyd's tests. However, very weirdly the RB logs are showing an attempt to match a queue on host gla.scotgrid.ac.uk. Where did the name of the CE go? Is this related to the CE GRIS getting in a pickle?

So, one of these anomalous blips in the crappy information system again.

ScotGrid meets AstroGrid

I was invited to talk about data movement in GridPP/WLCG to the AstroGrid people today. It was really informal and a chance for GridPP and AstroGrid to learn a bit more about one another's problem domains and look at the solutions that each project has adopted.

AstroGrid have a nice virtual filesystem implementation called VOSpace, which allows astronomers to interact with IVOA databases and resources. They now want to extend the concept of these personal virtual spaces into the Astronomy data area. So, a database at the ROE could have a database for an instrument, with tables for different data, and an astronomer would be able to query and store their results in a virtual space within the ROE database (roe/user/graeme/myqueries/hotstars). These results could then be copied out, e.g., into a VOSpace area.

The cunning bit is can you combine query results between databases? i.e., do a join between a database view at ROE and one at Cambridge? Well, this requires a (possibly large) amount of data to be shipped between these two databases.

The question they wanted to ask me was, did I know an efficient way of transferring the data between the databases?

Now, this is quite a different from the LCG problem. So, instead of saying, "buggered if I know" (with my facetious streak, it was tempting ;-), I described the sorts of data flows associated with LCG, the software components we use and the achievements and limitations of our solutions: FTS is great - it ships PB of data around very reliably 24x7, balances VO and site requirements, etc.; FTS is rubbish - it needs Oracle and only talks to SRMs.

Actually, it became quite an interesting and wide ranging chat about grids - different methods of working and how to convince users to use it. They were quite heartened to see that in HEP the VOs really do now overwhelmingly use the grid for their activities.

In the end it seems they actually want to start this quite small - get the virtualised results spaces working first, and then probably tackle the bulk data shipping later. They were definitely interested in looking at an implementation of VOSpace which used an SRM as a backend store (at the moment they haven't really gone beyond the constraints of a single RAID array). And when they do come to look at bulk data movement, then they will look at FTS and RFT as possible methods for doing it.

Nice to make contact with other communities. They get a good view from the top of Blackford Hill...

ATLAS DPM Pool at Glasgow Set "World Writable"

When I was poking around looking for lost DPM files last week I found a problem with our large dedicated ATLAS DPM pool. The problem is that the pool is restricted to one GID only, but different VOMS roles are mapped to different GIDs by DPM. So in fact we have:

mysql> select * from Cns_groupinfo where groupname like 'atlas%';
+-------+------+-----------------------+
| rowid | gid | groupname |
+-------+------+-----------------------+
| 2 | 103 | atlas |
| 16 | 117 | atlas/Role=lcgadmin |
| 17 | 118 | atlas/Role=production |
+-------+------+-----------------------+
3 rows in set (0.00 sec)

So only "ordinary" ATLAS users were getting access to the big pool. ATLAS production was being relegated to the general pool! No wonder it had filled up so much:

POOL atlas
CAPACITY 27.22T FREE 23.04T ( 84.6%)
POOL generalPool
CAPACITY 6.81T FREE 2.68T ( 39.4%)

As the ATLAS data was split so evenly between regular and production use I decided to make the pool writable by all groups. Now, when DPM 1.6.4 is released we will want to switch the pool to be writable by all our ATLAS groups. This clearly leaves a lot of data misplaced. I have made a feature request to enhance dpm-drain to try and help sort the mess out.

Rare DPM srmPutDone Failures at Glasgow

Just a place holder really. We failed a SAM RM test this morning, and for the first time in I can recall it was a DPM failure, not an information system problem.

The error message was Setting SRM transfer to 'done' failed: Unregistering alias from catalog.+ result=1.

Looking in my SRM logs clearly something was badly wrong:

03/27 06:04:25 15848,0 put: request by /C=CH/O=CERN/OU=GRID/CN=Piotr Nyczyk 6217
from node139.beowulf.cluster
03/27 06:04:25 15848,0 put: SRM98 - put 389251 389251
03/27 06:04:25 15848,0 put: SRM98 - put 0 srm://svr018.gla.scotgrid.ac.uk/dpm/gl
a.scotgrid.ac.uk/home/ops/generated/2007-03-27/file92308f95-2e1e-43ba-b8bf-da8f1
c7c416e
03/27 06:04:27 15848,0 getRequestStatus: request by /C=CH/O=CERN/OU=GRID/CN=Piot
r Nyczyk 6217 from node139.beowulf.cluster
03/27 06:04:27 15848,0 getRequestStatus: SRM98 - getRequestStatus 389251
03/27 06:04:27 15848,0 getRequestStatus: returns 0
03/27 06:04:28 15848,0 setFileStatus: request by /C=CH/O=CERN/OU=GRID/CN=Piotr N
yczyk 6217 from node139.beowulf.cluster
03/27 06:04:28 15848,0 setFileStatus: SRM98 - setFileStatus 389251 0 Running
03/27 06:04:28 15848,0 setFileStatus: returns 0
03/27 06:04:29 15848,0 setFileStatus: request by /C=CH/O=CERN/OU=GRID/CN=Piotr N
yczyk 6217 from node139.beowulf.cluster
03/27 06:04:33 15848,1 setFileStatus: SRM98 - setFileStatus 389251 0 Done
03/27 06:04:33 15848,1 setFileStatus: dpm_putdone failed: Internal error
03/27 06:04:33 15848,1 setFileStatus: returns 12
03/27 06:04:33 15848,1 srmv1: SRM02 - soap_serve error : Internal error

So this is a bit worrying.

I looked back through the DPM logs and this is not the first time this has happened. The signature is the dpm_putdone failed: Internal error. There have been 10 of these in the last 90 days:

svr018:/var/log/srmv1# zgrep "dpm_putdone failed: Internal" *
log:03/27 06:04:33 15848,1 setFileStatus: dpm_putdone failed: Internal error
log.1:03/26 20:09:59 15848,0 setFileStatus: dpm_putdone failed: Internal error
log.20.gz:03/08 01:00:55 3442,0 setFileStatus: dpm_putdone failed: Internal error
log.21.gz:03/06 15:53:49 3442,1 setFileStatus: dpm_putdone failed: Internal error
log.21.gz:03/06 15:53:49 3442,5 setFileStatus: dpm_putdone failed: Internal error
log.21.gz:03/07 00:08:22 3442,2 setFileStatus: dpm_putdone failed: Internal error
log.53.gz:02/02 09:50:57 3442,0 setFileStatus: dpm_putdone failed: Internal error
log.59.gz:01/27 07:14:37 3442,0 setFileStatus: dpm_putdone failed: Internal error
log.62.gz:01/24 15:03:46 3442,0 setFileStatus: dpm_putdone failed: Internal error
log.83.gz:01/03 06:35:18 3442,1 setFileStatus: dpm_putdone failed: Internal error

The return code is always 12.

I shall raise this with the DPM people...

OK, done. See GGUS #20178. Just to put this in perspective, we've had ~500000 successful srmPutDone calls, so the failure rate is 2x10^-5.

Finalising New VOs on DPM

In the course of trying to debug some proxy problems for Hannah Cumming from Total, I discovered that new VOs were not properly enabled on our DPM.

A bit more investigation revealed another gridmap file which is used for the DPM and the LFC, /opt/lcg/etc/lcgdm-gridmap. It is also to be necessary to run the YAIM config_mkgridmap on DPM nodes. I have amended the wiki instructions appropriately.

I wasted quite a lot of time on this yesterday, because I was worried that there was some issue with pool account names longer that 8 characters. I'm very glad this turns out not to be the case as it would have been a real pain to redo the new VOs with short pool account names.

A very strange side effect is that I seem to have broken lcg-cr for me - and only when done on svr017. This is the error I get:

svr017:~$ lcg-cr -n 1 --vo dteam -v file://etc/group -d svr018.gla.scotgrid.ac.uk
Using grid catalog type: lfc
Using grid catalog : prod-lfc-shared-central.cern.ch
Source URL: file://etc/group
File size: 523
VO name: dteam
Destination specified: svr018.gla.scotgrid.ac.uk
Destination URL for copy: gsiftp://disk036.gla.scotgrid.ac.uk/disk036.gla.scotgrid.ac.uk:/gridstore0/dteam/2007-03-27/file0d9b09fa-9efd-4ee8-8c3e-b504f175f231.389985.0
# streams: 1
# set timeout to 0 seconds
Alias registered in Catalog: lfn:/grid/dteam/generated/2007-03-27/file-54f896cf-4e4c-408a-b23e-40cb9196256b
the server sent an error response: 550 550 disk036.gla.scotgrid.ac.uk:/gridstore0/dteam/2007-03-27: Permission denied.

Odd:

• I have compared my environment with Andrew's and there's no meaningful difference - yet he can lcg-cr fine from svr017.
  
• It works when I do the lcg-cr from ppeui.
  
• rfio, globus-url-copy and srmcp all work perfectly (for both gridpp and dteam VOs).
  
• I have tried using different streams in lcg-cr but that makes no difference at all.
  

I'm relegating this to the curiosity pile...

Gatekeeper Troubles at Durham

Durham went on the blink at about 1am today - suddenly failing JL. The error message was the usual erudite globus effort: Got a job held event, reason: Globus error 3: an I/O operation failed.

Well, it looked straight forward enough - it's an i/o error, right. I found a lot of hints on google that this was caused errors in transferring in the sandbox. So check gridftp, home directory quotas, etc. Mark and I spent lots and lots of time on this, checking different things, becoming more and more confused (ok, so gridftp of a file works, can I make a directory using edg-gridftp-mkdir? have we restarted the gatekeeper properly? what's bound to ports 2811 and 2119? etc., etc.).

In the end we just could not fathom what had gone wrong, so I suggested to Mark that he email LCG-ROLLOUT and TB-SUPPORT.

Maarten Litmath pointed us to a GOC Wiki article which also said that this i/o error could occur when the CE was short of memory. I have found the culprit code in the l_check_memory function in Helper.pm - it produces a failure if the free memory (swap + physical) on the CE is less than 20% of the total. However, this error is not passed up the stack properly (in fact the code in queue_submit() returns undef) and so an entirely misleading error is passed back which wasted hours of our time. Grrrr.

I was reminded of a Alice in Wonderland...

'When I use a error message,' Humpty Dumpty said, in rather a scornful tone, `it means just what I choose it to mean -- neither more nor less.'

`The question is,' said Alice, `whether you can make error messages mean so many different things.'

`The question is,' said Humpty Dumpty, `which is to be master -- that's all.'

Alice was too much puzzled to say anything; so after a minute Humpty Dumpty began again. `They've a temper, some of them -- particularly Globus errors: they're the proudest - batch system errors you can do anything with, but not Globus errors - however, I can manage the whole lot of them! Impenetrability! That's what I say!'

`Would you tell me please,' said Alice, `what that means?'

I have submitted a GGUS ticket - these things won't improve unless they are complained about: https://savannah.cern.ch/bugs/index.php?25048.

Local Top Level BDII now used at Glasgow

Somewhat related to the last post, I switched Glasgow to use our local top level BDII instead of the RAL one on Wednesday.

In fact, RAL now have 3 top level BDIIs, and have been much less problematic recently - so perhaps this isn't really necessary any more? However, after 2 weeks we'll count up the number of BDII failures at Glasgow and compare them to Edinburgh and Durham.

Gridview and Site Reliability

The reliability of Tier-2 sites was a major topic at GridPP 18. This is going to be measured in an automated way from the SAM tests. I was very surprised, though, when I compared the SAM CE tests (the test I look at every dayt) against gridview. I measured our test pass rate over the last 3 weeks to be 93% (Durham), 96% (Edinburgh) and 97% (Glasgow). However, GridView measured us down as low as 80% (see slide 8 of my talk).

John Gordon had given the availability formula in his talk: and it's CE & SE & Site BDII & SRM. So I had to check these tests as well. What I discovered was that it was failures in SE and SRM which were pulling us down (site BDII tests were 100%). However, investigating these further I discovered that it was in fact BDII failures in SRM and SE tests which were the problem. More that that, it was not failures in the site designated BDII, but that the BDII had been hardcoded to sam-bdii.cern.ch - and all the failures were from this component. It's used by both those tests.

I have raised a GGUS ticket asking to get this changed to the site defined BDII. There is absolutely nothing a site can do about the failure of a central component at CERN.

If one estimates a failure rate on information system lookups during replica management tests of ~2%, the fact that the information system is used for CE-RM, SE and SRM, i.e., 3 times, means a site just cannot get any better than 94%!

Clearly this identifies the information system as a major problematic component which needs to be addressed if we have a hope of reaching our 95% targets. See Laurence Field's GDB talk for some ideas in this area - retries and caching look to be essential.

I have one other serious issue, and one quibble, with gridview. The serious thing is the 10% quantisation on the plots, which is a nonsense when our target is 95%. The quibble is the stupid mapping of the GOC sitename to a quite unguessable 6 letter abbreviation (old scotgrid-gla site was "GLSGW", the new site is "SCOTG"). Clearly they should learn from the EGEE accounting pages and give us a heirarchical tree view with the correct site names.

Jobmanager Tweaks at Glasgow

I finally got around to applying the Cal Loomis patch to the gatekeeper which helps catch jobs in the torque Completed state. Instead of patching lcgpbs.in and then reconfiguring Globus, I patched the final perl module in /opt/globus/lib/perl/Globus/GRAM/JobManager/lcgpbs.pm and then added this to our cfengine configuration.

In addition I patched the pbs jobmanager in the same way.

I have also now enabled the pbs jobmanager in /etc/globus.conf. Change the jobmanagers line to read

jobmanagers="fork lcgpbs pbs"

and then add

[gatekeeper/pbs]
type=pbs
job_manager=globus-job-manager

before restarting the gatekeeper.

At first I thought that it was broken as I was getting GASS cache errors, but in fact this turns out to be because the pbs jobmanager cannot deal with non-shared home directories (the lcgpbs one can). We don't have shared directories for EGEE VO users - however, we do use shared homes for NGS and local users, so David reports that it works for him (he's in NGS).

In addition we now pass 14/16 of the NGS GITS tests. The two we don't pass are gsissh and gsiscp, because GITS naively assumes these are on the gatekeeper host. For us they are not as we maintain a gsi login host for NGS and local users on svr020 instead.

Good progress, though.

Contact with VOs

In my GridPP 18 talk I said we still felt we didn't have enough contact with VOs in general. A specific example of this is biomed. When I added the new VOs on Monday I found there was a problem with the biomed pool accounts, which seemed to have the wrong stub names (boimNNN instead of biomedNNN). As soon as I fixed this we got biomed jobs into the cluster.

So, we were a large site, with a biomed queue enabled, where things were broken from November until March - and we never got a ticket.

Of course, all the VOs are sorting out their operational procedures, and I'm sure that many other VOs would have been no different - but that's the point isn't it? Our contact with non-LHC VOs is poor right now.

ScotGrid Hosts GridPP 18

GridPP 18 was hosted at Glasgow this week. The weather was good, the wireless network more or less survived and lots of people came to see the machine room. I had a good time - it was nice to play host.

You can read my presentation on the Tier-2 in case you slept in or weren't at the meeting.

New VOs at Glasgow: camont, gridpp, totalep

I've enabled the camont, gridpp and totalep VOs on the Glasgow cluster.

A complete description of how to do this is on the wiki.

This stupidest task must have been to write a script to reverse engineer a YAIM users.conf file from our group/passwd files so that the YAIM utility functions like users_getvogroup work. There's surely an easier way of doing that?

This afternoon I'll redo the RB and try and run some jobs through as a gridpp VO member.

DPM Disk Server Controlled by cfengine

Working on enabling gridpp, camont and totalep VOs I realised that the DPM disk servers had never been brought under cfengine control. This was a little trickier as it was the first set of SLC4 servers which were done, but in 30 minutes they were all ticking along happily, automatically updating authentication and other files.

There's still something rotten on disk035, which consistently refuses to yum update, complaining that the fetch-crl package is required and conflicts with glite-SE_dpm_disk - despite being installed! Perhaps its RPM database is corrupted? The workaround is to forcibly uninstall fetch-crl and then yum update.

However, at some point we should probably drain this machine and re-install it.

Resource Broker (Beta) for ScotGrid

Through the wonders of YAIM and cfengine, I was able to setup an lcg-RB on svr023 in two cfengine lines: download metapackage, run configure_node.

And it works! I got output back from my first job:

ppepc62:~/jobs$ edg-job-status https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ


*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ
Current Status:     Done (Success)
Exit code:          0
Status Reason:      Job terminated successfully
Destination:        svr016.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-dteam
reached on:         Fri Mar 16 12:24:38 2007
*************************************************************

ppepc62:~/jobs$ edg-job-get-output https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ

Retrieving files from host: svr023.gla.scotgrid.ac.uk ( for https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ )

*********************************************************************************
                        JOB GET OUTPUT OUTCOME

 Output sandbox files for the job:
 - https://svr023.gla.scotgrid.ac.uk:9000/5hT0x7GZluDFMWJ6qT0KLQ
 have been successfully retrieved and stored in the directory:
 /tmp/jobOutput/graeme_5hT0x7GZluDFMWJ6qT0KLQ

*********************************************************************************

The jobs did, however, run really slowly as R-GMA managed to lock-up twice on me and need restarted. I'm really fed up with this, so I have started an Ops Logbook for the site to at least log these issues in a consistent way.

If anyone has a nagios/cfengine recipe for restarting R-GMA I'd be glad to use it.

Top Level BDII for ScotGrid

I setup a top level BDII for ScotGrid this morning. I have put it onto svr019 (the MON box) as these machines are very capable to running multiple services on this type (and svr019 might as well do something useful ;-).

I'll add a top level alias as bdii.scotgrid.ac.uk, and as load increases more machines can be added here.

Before I actively use it for ScotGrid, though, I will let it run for a week or so - I'm naturally nervous about MCE errors these days and although svr019 has not suffered any thus far, our experience suggests that loading up a machine can cause things to be more fragile in this respect.

Although the BDII is a simple node to install with YAIM, I'm really pleased how much value we get from running cfengine these days - just telling it the machine is in the grid class now does so much. So I only had to add two lines to cfengine to handle the whole install: one in packages, to get glite-BDII installed, one in shell commands to invoke the YAIM configure_node. So easy!

New Installer Christened: YPF is Born

Working on the new installer again today. With dnsmasq installed on svr031 the installer started to work fine. It is still using the text classes.conf and the move to SQLite might have to wait until svr031 is reinstalled - it's not clear it will work right now. (And today I was very pressed for time - just need to get things working!)

I felt we need a name for the installer. As it's written in python a flying circus name seemed to be appropriate and Andrew suggested "YAIM People's Front". Brilliant.

In the afternoon I added a new host to the cluster, using 2 cheepo netgear switches to connect to the internal and external cluster networks. The tools for manipulating the cluster database are very primitive right now - clearly there's a lot of work to be done here - but a minimal interface allowing an arbitrary SQL command to be issued suffices for now.

I have written some documentation in the wiki.

There's no real intention to make YPF a general installer project at the moment, but other sites might find aspects of it useful.

Minor Panic Over DPM Upgrade and GIP

I had a minor panic over the DPM GIP plugin today when I looked over Glasgow's storage stats on the storage accounting pages. I was worried that the DPM schema changes had made the new plugin fail. But when I looked in detail I saw that what has actually happened was that running YAIM had just replaced the new plugin with the old one - hence the publishing problems.

Fix is just to re-enable the new plugin, then things recovered right away.

svr031 Installer Almost Working

Spent more time on the new infrastructure for svr031 again today. I wrote the script to get a dhcpd.conf file out of the cluster database, which was quite straightforward to do. Then dhcpd could function again.

I tried a test install and found that tftp was, amazingly, still working even though xinetd had lost all of its configuration.

Then the install fell over slightly - the kickstart installer was using the old internal hostname of the svr031 server, master.beowulf.cluster, but setting the name server resolution to the university DNS servers. As an installing machine does not yet have the big cluster /etc/hosts file it could not resolve this name. So I changed the kickstart setup to code in only the IP address of the svr031, so no name resolution was needed. This allowed the install to proceed, however then I discovered some cleverness in the kickstart post install script which ensures that even on first boot the machine sets its hostname correctly (in particular, to the routed hostname for grid and disk servers) and that this relies on DNS to function.

So, I either have to rewrite the clever code, or go back to running DNS on svr031.

I decided that running DNS on svr031 was no bad thing. There's a lovely lightweight DNS server called dnsmasq - it loads up a DNS server with the contents of /etc/hosts, which is generated from the cluster database already, and can give out DNS during the install process.

After a machine has first booted, cfengine will copy in the global cluster /etc/hosts and set the DNS to the university servers to remove the single point of failure. But having internal DNS for the installs is nice.

Even better I found dnsmasq is built for RHEL4 x86_64 in the DAG repository, so installing it was a cinch.

Finally, dnsmasq also includes a dhcp server and a tftp server, so I will have a look at running these and possibly simplifying life even more.

Installs should be back up and running tomorrow.

Webcache Exemption For New Cluster

Finally, after months of asking, we've been exempted from the University's webcache. There are still some open networking issues, but this is a big step forwards.

New Fabric Management Schema Designed

I spent most of the day with svr031, desigining and implementing the new fabric management database system. Paul and Andrew reviewed my initial schema and suggested some improvements - the current schema is now described in the wiki.

Then I had to write a couple of scripts to take the extant information from the CVOS configuration files, and to add extra information for routed hosts, and then populate the database.

After that was done, I was able to write the first utility script, to extract an /etc/hosts file from the database. This is improved over the version recovered from DNS so I gave it to cfengine to distribute across the cluster.

Next stop, getting a working dhcpd.conf file, so that we can being new installations again.

This is useful and essential work, but bloody tedious at some level...

Torque/Maui Upgrade Lost Jobs?

Was the torque upgrade was not as smooth as I'd hoped?

It should have been just a minor upgrade, and thus pretty transparent, but some issues have arisen.

Firstly one of our local engineering users reported that the gatekeeper lost contact with all of his jobs - the jobs were still running in the batch queue, but globus-job-status reported them all done, and he couldn't get any output back.

Then I noticed that on the ATLAS production monitor page our 24 hour efficiency droped to the lowest ever level at 24%. This makes me rather worried that we lost all of our ATLAS jobs if the gatekeeper had a brain haemorrhage.

On the other hand, efficiency in the UK seems generally very low right now (ce02.tier2.hep.manchester.ac.uk, 26%; ce1.pp.rhul.ac.uk, 24%; fal-pygrid-18.lancs.ac.uk, 17%; lcgce01.gridpp.rl.ac.uk, 14%), so perhaps this is just a coincidence?

Doesn't explain the globus issues seen by our local user though.

Does anyone know the magic for getting into the guts of the gatekeeper and seeing which torque jobs it's connected to?

Torque default Queue

Problem - The NGS gits tests do not easily support naming the torque queue to be used.

Solution - Make the default queue "queue_type = route" and let it have all the VO based "queue_type=execution" queues as routing destinations

eg
set queue route2all route_destinations += dteam
set queue route2all route_destinations += atlas
set queue route2all route_destinations += alice
set queue route2all route_destinations += cms
set queue route2all route_destinations += lhcb
set queue route2all route_destinations += biom
.......... etc

It would seem that torque then tries the destination queues in turn and the first that matches the job accepts the job. Not only does this solve the NGS gits problem, but also seems more generally elegant as a scheme for the default queue.

svr031 recovery ongoing

There's been sadly little time to work on svr031. However, now work with our nuclear physicist community seems bound to involve the installation of new machines on the cluster, so things have to begin in earnest.

I have recovered the lftp mirror scripts, so that's a start here.

Next I have to look at making a proper SQLite database of machines and extracting a dhcpd.conf from that. Fortunately the tftp root was not lost, so that part of the boot process should be ok.

Blog Layout Updated

I've updated to the new blogger layout. This provides a much better archive option, using a collapsable tree view (so it's easier to find past posts) and <pre> text now wraps, instead of spilling into the sidebar.

The only thing I don't like is that the new default text style for posts is a little cramped - needs a little more line spacing IMO. If I knew about CSS I could probably fiddle with this in the tempate (but life's too short...)

Finally, safari doesn't quite work with the new layout editor, but firefox is fine.

Glasgow Update to gLite 3.0r16

This was the first gLite update with significant component changes:

* DPM was upgraded to 1.6.3, with a schema change and a new SRM v2.2 daemon.
* Torque and Maui were upgraded to v2.1.6 and v3.2.6, respectively, from the previously ancient LCG versions.

The DPM upgrade I tackled first. This was fine and I blogged about it on the storage blog. Just be careful to take a db dump before you try it, just in case things go wrong. There's also a strong warning against running automatic updates on gLite server nodes - this is not supported and some people are reporting DPM database corruption on LCG-ROLLOUT.

The torque/maui upgrade I was a little nervous about as I don't feel I greatly understand these components. However, Steve T had said that minor upgrades are ok (we'd been using the Steve T build for the cluster since the start, so we were already on torque v2), so I took the plunge. First I did a single worker, and restarted pbs_mom, so make sure the 2.1.6 mom didn't have trouble talking to the 2.1.5 server - and it didn't. So then I updated all the WNs, before turning my attention to the server.

Here, I did the usual yum -y update first. Then I restarted pbs_server and maui. pbs_server didn't restart cleanly, claiming something was bound to the port. I had a look, but by the time I did there was nothing - I think it was the server being sluggish to exit. So pbs_server then (re)started fine, and I did an extra maui restart to be on the safe side.

A basic check of the batch system (pbsnodes, qstat, diagnose) looked ok.

I have commented Steve's repository out of by yum.conf - we'll now use the "official" gLite build on Steve's advice.

N.B. I still intend to manage to batch system using cfengine, not YAIM - it's a lot more flexible for us to do this, e.g., the new routing queue being the default one.

Durham enables new VO's

Durham has enabled the following VO's -

• cedar
• ngs
• gridpp
• minos
• totalep
• camont
• mice

This brings the number of supported VO's at Durham up to 28.

Network Asymmetry

Any transfer tests including Glasgow show up assymmetry in the network. We can take data in twice as fast as we can spit it out. Note to self: Follow up with Compserv networks.

R-GMA is so...

R-GMA just keeps falling over on the new site. Inexplicably tomcat will lock up and no response will be got from the R-GMA servlet.

This is very frustrating - there's nothing interesting in the logs either. it just dies.

Today when I was trying to restart it, tomcat restarted OK, but the R-GMA servlet would not startup, announcing cryptically:

2007-03-06 16:49:21,601 [http-8443-Processor23] WARN  org.edg.info.ServletBase - Error processing servlet request.
org.glite.rgma.system.UnknownResourceException: Object has been closed: 77151190
        at org.edg.info.InstanceTracker.get(InstanceTracker.java:155)
        at org.glite.rgma.services.ResourceService.getInstance(ResourceService.java:179)
[...]
        at java.lang.Thread.run(Thread.java:534)

What on earth's going on there?

Paul saved the day, using the tomcat manager interface to stop and then restart R-GMA. After that is was a bit happier.

I'll have to work out some nagios alarm or cfengine test for tomcat though - as it just hangs simple checks on the daemon's presence won't work. Grrr...

Progress Towards Enabling Total at Glasgow

I've spent all afternoon working out how to enable VOs on the new Glasgow cluster, using the new "totalep" VO as my template VO.

As this is the first time we've added a new VO since the cluster was setup, naturally this needs to be done rather carefully to ensure that all the bits go into the right place.

My modus operandi is to try and run YAIM as little as possible - so if a complex YAIM function does lots and lots of things, and runs 2 commands per VO, then I will try and extract those two things and give "by hand" instructions. Hopefully the only thing I will need to re-run YAIM for is the information system, which is damn fiddly to get right by hand.

Progress so far is on the wiki.

Once this has been done once and documented it should be much faster for any subsequent VOs.

ALICE Queue Disabled at Glasgow

I have been regularly pruning the orphaned ALICE processes off the Glasgow CE. This morning I had to kill 371. We've had very little response from the user concerned, and I can't see how they will be motivated to fix the situation as ALICE have not, and at the moment will not, run any jobs at Glasgow. (We have a 5% share for ALICE as this is GridPP policy and, of course, we generally think grids work better if lots of VOs are enabled.)

I did check the local ALICE queue and it functions correctly.

However, I've now closed the queue in the hope that this will prevent the globus processes from spawning. We know from experience that if these processes are allowed to accumulate too much then they affect the performance of the CE and thus affect all users of the site

Devolution? cfengine style

In an attempt to allow the stragglers of the cluster to feel as if they are doing their own thing, we have let them lurk on the right hand side of the machine room, safe in their "slightly different hardware" setup to allow them to feel "Speshul". Time... for some "gentle persuasion" in the guise of cfengine.

First off, Not all disk servers are created equal. Once we'd Created and formatted the array, we then had to mount it. Cue the following snippet in the directories: stanza
disksvr::
# Mount points for Raid space
/gridstore0 mode=0755 owner=root group=root
/gridstore1 mode=0755 owner=root group=root
/gridstore2 mode=0755 owner=root group=root
/gridstore3 mode=0755 owner=root group=root
/gridstore4 mode=0755 owner=root group=root


and a new central /etc/fstab thrown out to all the nodes (in the copy: stanza)
disksvr::
$(skel)/disksvr/etc/fstab mode=644 dest=/etc/fstab type=sum


Finally I remembered that I should have reduced roots reserved space to zero on those chunks:

for i in `seq 1 5` ; do tune2fs -r 0 /dev/sdb$i ; done

SplayTime Finally Cracked

Finally I've got splaytime to work. Nothing to do with capitalisation; it needs to go into update.conf, not in cfagent.conf. Notice the hourly spikes dissapear after 1800 - that's when the splaytime of 25 minutes went into update.conf and started working.

LCG Gatekeeper On svr016 Patched For stagein

In our investigations for supporting NGS it became clear that the lcg-CE does not support GRAM stagein properly. There is a patch for this available in Savannah, which I have now applied to the CE.

Olivier and Alessandra were running this patch at IC and Manchester, so I was reassured enough to apply it to the live system. Checking the gatekeeper logs it doesn't seem to have broken anything, and hopefully it will make a difference for our local users attempting to get GRAM working properly.

Just in case the CE goes down in flames, the patched Helper.pm has been added to cfengine's skeleton files and a new line added to the copy stanza:

copy:
        grid.ce::
                # This applies patch https://savannah.cern.ch/bugs/?func=detailitem&item_id=4400 to the CE
                # which should allow for stagein with globus-job-{submit,run}. Also rumored to help
                # with ATLAS condor submissions...
                $(skel)/ce/gatekeeper/Helper.pm mode=0755 dest=/opt/globus/lib/perl/Globus/GRAM/Helper.pm type=sum

Finally, there's a rumor that this patch might help with ATLAS condor submissions. For the record svr016 currently is at 70% on the ATLAS Prod site efficiency pages. I'll keep an eye on this (although it mixes condor and RB submissions, so it's quite hard to track each method separately).

cfengine Running From Cron Too

I found another node on which cfengine (more precisely cfexecd) had died - so it hadn't run for sometime. I have now added a cfagent run from cron, which fires every half hour.

In addition, I've added a "processes" stanza to the cfengine configuration, which means that if any of the cfengine daemons are not running then the crontabbed cfagent run will restart them:

processes:
        any::
                "cfenvd" restart "/usr/sbin/cfenvd"
                "cfservd" restart "/usr/sbin/cfservd"
                "cfexecd$" restart "/usr/sbin/cfexecd"

Note the regexp on cfexecd - it's to stop the "cfexecd -F" run from cron from matching.

A couple of other things:

• I think I found the problem with my "splaytime" directive: it's "SplayTime", not "splaytime"!
  
• I added the time class Hr04 to the tidy stanza. From running cfagent by hand it was clear that this was the most expensive operation - and it really does only need run once a day.
  

Durham Queue reogranisation

Last week Durham implemented new queue structure, we now have three grid queues -

• ops (for ops jobs)
• pheno (for the phenogrid VO)
• grid (for all other grid jobs)

We have also implemented job suspension based upon queue priorities, this allows us to give local queues priority over grid jobs. How to configure preemption has been written up here http://www.gridpp.ac.uk/wiki/Maui_Preemption

Minor Cluster Updates

In anticipation of losing svr031 for a few days - and thus any easy way keeping the cluster up to date - I've been patching us up to the latest gLite release, r13.

There seems to be no good way of doing this other than a "yum update" - just updating the metapackage seems to not encapsulate all of the installed, but updated, RPMs, e.g.,

node001:~# yum update
[...]
I will do the following:
[update: edg-mkgridmap-conf 2.7.0-1_sl3.noarch]
[update: glite-config 1.8.4-0.noarch]
[update: glite-lb-common 3.0.6-1.i386]
[update: edg-mkgridmap 2.7.0-1_sl3.noarch]
[update: glite-rgma-command-line 5.0.4-1.noarch]
[update: glite-WN 3.0.13-0.noarch]
[update: glite-security-gsoap-plugin 1.2.5-0.i386]
[update: lcg-ManageSoftware 2.0-6.noarch]
[update: glite-rgma-api-python 5.0.10-1.noarch]
[update: glite-rgma-base 5.0.7-1.noarch]
[update: glite-wms-common 1.5.14-1.i386]
I will install/upgrade these to satisfy the dependencies:
[deps: lcg-tags 0.2.1-1.noarch]

But updating only the metapackage gives a palrty:

node001:~# yum update glite-WN
I will do the following:
[update: glite-WN 3.0.13-0.noarch]
I will install/upgrade these to satisfy the dependencies:
[deps: lcg-tags 0.2.1-1.noarch]

I think there's an argument for running a "yum update" nightly on the worker nodes, but it still seems far too dangerous a thing to do on the servers. Too much of a risk of daemons not restarting properly or java being arsed up.

I also added the new VOMS certificate for dzero - using cfengine this is easy. Unfortunately, not everything is yet cfenginified: machines currently not under cfengine control: svr019 (MON), svr021 (site BDII), disk servers. Should try and address this soon.

svr031 removed from active cluster role

It's taken a while, but svr031 has now been taken out of active service.

All machines have had /etc/hosts and /etc/resolv.conf files put on them, which takes care of internal cluster name resolution, and they have had their DNS servers pointed at the normal university ones.

In addition one of the nat hosts was setup as a gateway machine for the worker nodes. The workers were (carefully) told to use this new host as their default gateway.

So, nothing is relying on any services provided by svr031 and we can prepare for a reinstall next week.

I had to do a bit of resuscitation of svr031, so that one can at least login via ssh and scp files to and from it. I've restored some of the library paths to get auxiliary commands working.

The cluster itself has been remarkably untroubled by svr031 being in a tizzy - I thought about putting us into a precautionary downtime, but this has not been necessary. We've carried on passing the SAM tests without trouble.

Leaders? Who needs 'em?

The President's Brain Is Missing...

svr031, the Glasgow cluster headnode, is currently a bit FUBAR.

Grieg was mirroring the dcache repository from DESY, becasue the mandatory webcaching policy at Glasgow stuffs up yum big time, hence any repos need to be got locally for installation to work. However. his mirror script has went crazy and managd to wipe the whole of /etc. Arggg!

Backup, what backup? It's a RAID 5 disk, we didn't need a backup (woops)...

svr031's roles in the cluster are:

• NAT box for WNs
  
• DNS server for whole of the cluster
  
• Central syslogger
  
• DHCP server
  
• tftp server for kickstarting
  
• http server for installation, nagios and ganglia
  

The critical run-time services are NAT and DNS. Fortunately the DNS server on svr031 and the NATing are still working, so even though the president's brain is missing the organs of the state still function, for now.

The immediate things to be done are to remove svr031's run time functions from the cluster. This will consist of:

1. Generate and copy an /etc/hosts file with a complete set of entries for the internal cluster machines - so that the internal DNS is not required.
   
2. Update /etc/resolv.conf on the cluster to use the standard university DNS servers
   
3. Setup NAT via the dedicated NAT boxes
   

After those steps have been taken our dependancy on svr031 should be removed, and we can work on reinstalling it and re-establishing its other services.

Durham's DPM and lcg-rep problems

Problem with lcg-rep and the DPM at Durham has been fixed, it was tracked down to a permissions problem on /etc/grid-security/gridmapdir. This directory needs to be group owned by dpmmgr, it was group owned by edginfo. The DPM machine crashed a few weeks ago, the system has to fsck it's disks on the way back up and I suspect this is when the ownership change happened. The reason that globus-url-copy worked and non of the other DPM protocols didn't was because the dpm-ftpd runs as root, where as all the other dpm daemons run as dpmmgr.

Disk Servers

fdisk doesn't like "big" disks. GNU Parted is better and can do things like cope with GPT disk labels, but it needs some scripting to make it easier. enter one uber-nasty script. Yes I know it doesn't do *any* sanity checking that you've run it on the right host, but hey, sysadmins never make mistakes :-)

cat mkpart.sh

echo "Running script on $1"
PARTED=/sbin/parted
DISK=/dev/sdb
ssh $1 <<EOF
hostname
$PARTED $DISK print
$PARTED $DISK mkpartfs primary ext2 0.017 1812371.050
$PARTED $DISK mkpartfs primary ext2 1812371.050 3624742.100
$PARTED $DISK mkpartfs primary ext2 3624742.100 5437113.150
$PARTED $DISK mkpartfs primary ext2 5437113.150 7249484.200
$PARTED $DISK mkpartfs primary ext2 7249484.200 9061855.233
$PARTED $DISK print
EOF

Also discovered while I was doing this that I'd only created a 2.1TB partition on disk040 as I forgot to enable the 64-bit LBA mode. Done and reinitialising volume.

> snmpget -v1 -c <communityname> -Pu -m all areca040 ARECA-SNMP-MIB::volProgress.0
ARECA-SNMP-MIB::volProgress.0 = INTEGER: 313
ie, 31% done so far....

Glasgow CE slow march to death with alice?

I noticed last week, after coming back from Australia, that the load on the CE had been creeping up and up.

On further investigation I have found 1790 gatekeeper processes running as the user alice001 - but we have no alice jobs running. Indeed, the torque logs show that we've never run any alice jobs (well a few alicesgm jobs in December only).

In fact the CE has hit swap now, so it's time to get biblical:

To every thing there is a season, and a time to every purpose under the heaven:
A time to be born, and a time to die; a time to plant, and a time to pluck up that which is planted;
A time to kill...

# kill $(ps aux | perl -ne 'print "$1 " if /^alice001\s+(\d+)/')

In the spirit of grid operations I have raised the issue as a GGUS ticket rather than emailing the user directly. This will be an interesting experiment in how well sites can raise problems with VOs through GGUS. Ticket #18703.

Mass job exit on UKI-SCOTGRID-GLASGOW

The Glasgow site had a very odd, and severe, drop in the number of running jobs at 0547 yesterday. At first I thought there had been a networking outage, which had chopped the jobs, but on further examination it seems not. The 100s of jobs which exited were all owned by a single (non-production) ATLAS user and seemed to exit "naturally" - they hadn't exceeded any queue limits.

The jobs were also running with a high CPU efficiency (e.g., cpu/wallclock of resources_used.cput=07:19:17 resources_used.walltime=07:50:56) - but they all exited at once.

Either the user must have found a problem with the jobs and cancelled them all (I wonder how one would see that from the gatekeeper logs, there doesn't seem to be any obvious way) or the jobs had a suicide pact.

Unfortunately this must have caused sufficient load on the CE to have us drop out of the information system for a time. Even running a separate site BDII doesn't seem to be a cure for all information system ills.

This is a duck pond, right? I mean there's no way it's a lake...

Final presntation in Melbourne: "Grid Data Management and Storage (An EGEE-Centric View)".

This is organised by VeRSI (The Victorian eResearch Strategic Initiative). They have funding to setup multi-site storage of ~100TB to help scientists in Victoria share data and were very interested in the EGEE DM solutions. I ran though storage, catalogs, SRM 2.2, FTS, etc (presentation here). However, my conclusion was that operations were more important than technology choices - perhaps that is the real lesson from EGEE.

After learning a bit more about their project, I was inclined to recommend dCache for them - they have a muti-site storage problem, with dedicated networking between the data centres and dCache would seem to offer them the most flexible approach. Of course, as with many of the Australian grid projects, SRB seemed to be their de facto soution (needless to say, with shiboleth authentication), however in talking to their SRB expert about the LFC, I finally managed to get a number for the scaling of the SRB MCAT catalog - it can start to have problems when you have over 30,000 files. This seems terribly low - I know the LFC has been tested up to millions on even quite modest hardware (although the LFC is just a file catalog, where as MCAT is also a metadata calalog).

Of course, no software is a panacea, and they all have problems - perhaps the weakness of the EGEE solution is that there are so many bits to it - it would be quite a daunting thing to setup from scratch.

I'll be interested to see what they do decide in the end.

My fourth talk in Australia was the School of Physics Colloquium. Here was a chance to move away from the storage and data management focus, and deliver a much more general EGEE/LHC talk, which I entitled "Enabling Grids for eScience: How to build a working grid".

As ever, the RTM is a great start - it's such an attractive visualisation of the grid. This time I had no trouble getting it to work on my mac, although it only ever uses the primary display, so I had to put my laptop into mirror display mode, then flip back to two screens for the talk - a minor niggle.

I tried to speak about EGEE in a general way, introducing the project, the services delivered - even a slide on the "Life of a Job", problems commonly seen and operational aspects. The idea was to introduce the grid to potential users, rather than site admins or service providers. This seemed to go pretty well - no one obviously fell asleep (!) and there were a number of questions about EGEE and other grids, job efficiencies and data volumes. Later some of the graduate students complimented me on a good talk, so it must have been pitched at the right level.

I had intended to mention Byzantine Generals, but it slipped my mind. Single points of failure, eh...

Steve Lloyd was still having problems writing into Durham's DPM. I nabbed an ATLAS user and got them to help me, which has revealed an incredibly weird problem:

If the user is unknown to the DPM then lcg-cr will fail, with the cryptic error "transport endpoint not connected". Attempting to srmcp reveals an authentication failure: "SRMClientV1 : CGSI-gSOAP: Could not find mapping for: USERS_DN". Somehow the srm daemon is refusing to authenticate the user properly. It's actually quite a deep problem in the GSI chain, because the daemon doesn't even get as far as logging anything about the connection.

Drilling further down, to globus-url-copy, then revealed a bizarre work around: a globus-url-copy command will create the mapping from the user's DN to a pool account. After this is done everything starts to work properly.

We've checked to see if it's a problem with VOMS proxies and it isn't. It also doesn't seem to affect DPNS itself - even users who can't copy in files can do a dpns-mkdir and they get a new entry in Cns_userinfo fine.

Very mysterious.

Tuesday night: It's Linux Users Victoria.

These guys (and of the 50 or so there, 49 are guys) are uber-geeks. Talking to them is a lot of fun - they like the physics, they like the LHC, they even like the grid. Some of them want grid certificates. Personally I think we should give certificates to them - they'd probably fix lots of stuff!

The GridPP Real Time Monitor was a big hit.

Right, before the suntan gets washed off and all memory of antipodean adventure fades, better blog a bit about my vist to Melbourne Uni on the second week of the trip.

First Monday, it's VPAC. "Data Management and Storage in EGEE" is my talk.

VPAC is the Victorian Partnership for Advanced Computing. I'm giving much the same talk as I did at AUSGRID, but with some particle phyiscs at the beginning. This is partly to flesh things out, but mostly because the LHC is so cool it makes the storage and data management parts of the talk more interesting, having set the context for why we are doing what we're doing. This is my first talk pretending to be a particle physicist - remarkable what an afternoon browsing the Standard Model on wikipedia can produce ;-)

The audience are mostly coming from an HPC background, so they are more used to providing compute resources and fast interconnects, but now they have an interest in providing cross-site storage, rather and some dedicated disk hanging off their HPC. Oh, and they've heard about the grid.

Again, the scale of the EGEE project is generally appreciated - and by implication that the solutions we've adopted do scale. I'm specifically asked about why we ddn't use SRB. This is hard to give a definitive answer on as I don't know enough about SRB, however, the EGEE solution (usefully charactarised by Jens as component based) is that:

1. It allows poorly performing components to be replaced individually, e.g., the replacement of the RLS with the LFC.
   
2. It allows sites, whose storage is also serving local users and legacy applications, to layer an SRM interface on top of their storage, rather than have to implement a completely new system.
   
3. It makes it easier to deploy at a large number of sites, where agreements across the whole grid are often difficult to achieve.
   

In the discussion it also becomes clear that SRB doesn't offer any good methods of data localisation - there's no easy way to ensure that jobs at a site can access SRB data efficiently. There doesn't seem to be an SRB LAN access protocol, like rfio or dcap.

After the talk I discuss SRB a little more with David Bannon. He says that the SRB model of federations of storage is actually quite fragile and only seems to work if all the SRBs are running exactly the same release - so it's back to big bang upgrades to maintain.

Tada! GET /alt/echo.php?msg=cfengine%20post%20start HTTP/1.0

"so what?" I hear you ask - well, it finally means that kickstart is automagically[1] installing the cfengine rpms and doing the local post install voodoo.

The problem? I needed to create a "local packages" repo for our SLC4X tree - we had one for 3.x but I needed to go and get a copy of "createrepo". to create the magic YUM Metadata. Now to finsh setting up cfengine to customise the node.

Spent most of yesterday performing some housekeeping on the Edinburgh dCache. See my gridpp-storage post for the gory details. In summary, draining pools is slightly easier than it was before, but isn't quite for the faint of heart.

Also, I have a meeting today with the guys from EUCS. Still having problems with the SAN due to some weirdness that is going on with partition tables on some of the volumes that they have given us. For some reason fdisk and parted keep on barfing when I try to wipe everything and start again. It's strange since this is only happening with 3 of the volumes, the remainder seem fine. Also need the to configure the multipathing.

Wow! It's been a busy time here in Melbourne, but just time to blog before getting back on the plane to Scotland.

The first week I was at the ACSW conference in Ballarat. It turned out that AUSGRID was only a 1 day event as part of that, so in fact the primary reason (excuse?) for coming turned out to be the least interesting thing. There were a few good papers as part of that, though. The keynote was by Denis Caromel, from Nice, describing how they have grid "enabled" java, in a package called ProActive. Essentially it replaces all of those awful MPI calls with nice, easy to manipulate java objects. Looks like a splendid way of doing parallel code on the grid. Probably not of huge interest in LCG, where we're mostly embarassingly parallel, but for communities coming from a supercomputing background, or just starting out with grids, it might be a very valuable toolkit to use.

The paper I gave went down well and there was general appeciation of the scale of EGEE and the volume of work we are doing. There was some discussion on SRM vs. SRB, a theme that contunied into my second week.

I met with Lyndsay Hood, the acting program manager of the Australian Partnership for Advanced Computing (APAC), for lunch. We had an interesting chat about their storage issues, where the thing that seems to put them off the EGEE solution is the X509 authentication. Too difficult for normal users, they seem to think. This is something I've heard from people in the UK as well, but no one seems to have a better answer: portals are too limiting, username/password doesn't scale. Shiboleth gets talked about all the time, but it seems to be spoken of as a panecea, rather than a working product that people can use right now.

A general comment on computer science conferences: I've had enough graph theory to last me a lifetime.

This post is a bit late, but I figured people would be interested.

I started to re-organise Edinburgh's dCache at the end of last week. I've been meaning to do this since purchasing a qLogic HBA just before the holidays but have only now managed to find time to do the work. The new card is a QLA3242 (2Gb, dual port) and our intention is to use it to connect a dCache pool node
(pool2.epcc.ed.ac.uk) directly to the University SAN via fibre channel, rather than using NFS (which we saw led to lots of problems with stale file handles and terrible write rates).

The installation went fine, kudzu did it's job well and immediately picked up the new card after a reboot. We are using the qla2300 module, loading it at boot time. There were teething problems in seeing the new scsi devices from the SAN, but these have since disappeared. Still need to do a couple of things before getting the storage into dCache. I need to properly setup multipathing (any advice would be appreciated) and also to format the 1.6TB volumes. Each volume is it's own independent set of disks which eases the configuration issues slightly since I don't have to worry about simultaneous reading/writing to the same disk.

All of our IBM disk is now hanging off of pool1 again and this is our currently our only gridftp door. Strangely I've not seen any CLOSE_WAITs recently....

Spent some of the week doing sysadmin work on the new cluster. Discovered that not all raid cards are configured equal thanks to a set of scripts using Net-SNMP. Although Areca are kind enough to supply the MIB Files for the Raid cards, they're badly written and contain underscores in the definitions. I already discovered a workaround last week but as I've just installed Nagios on the master node (together with the magic mod-ssl incantation to make parts of the site only accessible to named DN's) I discovered that the check_snmp plugin doesn't work with it.

One quick hack of the source file to add in

asprintf (&command_line, "%s -Pu -t %d -r %d -m %s -v %s %s %s:%s %s",

and it works as well as can be expected. Raised a sf.net ticket to try and get something better propogated upstream.

Now working on a Makefile and a pile of shell scripts to automagically generate a set of nagios configs to check each node before Paul integrates it all with MonAMI. Crib notes and more info available on the GridPP Nagios Page

End of the week - more Transfers tests (and a quick Perl script to summarise logs). Colin's tweaks have vastly improved inbound speeds to UKI-SCOTGRID-GLASGOW from Edinburgh to over 600Mb/s. "Nice!"

Having problems transferring to/from RAL-T1 though. Suspect it's more likely to be typos in SRM URL's rather than system failures. Symptoms are

Reason: Failed on SRM put: Cannot Contact SRM Service. Error in srm__ping: SOAP-ENV:Client - CGSI-gSOAP: Error reading token data: Success; also failing to do 'advisoryDelete' on target.

Plan for next week - More of the same - check that size isn't an issue (hope not!)

Andrew has been pushing on with file transfer tests using the new site. The good news is that initial indicators are that we seem to be able to get much better rates: up to 650Mb/s incoming from Edinburgh.

However, he managed to provoke an MCE error twice on disk041 - so this server definately has bad memory.

Fortunately dpm-drain decided to start working (last time I tried to get data off this machine it failed with various bizarre errors) so it was very simple to move the 14GB of files off and remove this partition from the DPM. (Most of the data was dteam test dat, so fingers crossed there's been no corruption of the small number of pheno files.)

ClusterVision have been informed and David is working on a reliable way of triggering these errors so that we can test machines before they go back into production.

After updating to the latest gLite release (r11) on the old scotgrid site, the site started to fail JS with the much maligned message "Unspecified job manager error".

It turned out to be the classic WN cannot ssh back to CE to transfer job output. After a bit of poking around and playing I decided that for a site which has just 48 hours of run time left before it gets shut down it was just not worth the hassle of trying to debug this - so I have closed the queues and put the site into downtime, where it can languish until it is withdrawn on Friday.

Farewell, scotgrid-gla!

Happy New Year ScotGrid!

Despite running unattended over the Christmas holidays, ScotGrid sites were by-and-large available.

From 2006-12-20 to 2007-01-03 SAM status for each site was:

ScotGRID-Edinburgh:  SAM test pass rate 241/338 = 71.3%
                    (Failures: JS 3, RM 94)
scotgrid-gla:        SAM test pass rate 324/338 = 95.8%
                    (Failures: JS 4, RM 10)
UKI-SCOTGRID-DURHAM: SAM test pass rate 318/338 = 94.1%
                    (Failures: JS 6, RM 14)
UKI-SCOTGRID-GLA:    SAM test pass rate 328/338 = 97.0%
                    (Failures: JS 1; RM 9)

Looking at the JS and RM failures for everyone except Edinburgh these seem to be at the level of grid flakiness caused by higher level services failing or timing out (e.g., RM test times out, BDII fails to respond, etc.). So our only significant problem was issues with Edinburgh's dCache, which did seem to be unreliable.

Any comments Greig? Did you have to intervene?

I have finally fixed the problem where the BDII was refusing to publish the VO view for biomed on UKI-SCOTGRID-GLASGOW. This had been going on since the cluster was reconfigured. At this time the biomed torque queue was renamed biom (I'm sure I had a good reason for doing this, although I now cannot recall it...), which somehow was causing the dynamic scheduler wrapper to fail to publish for the biomed queue. I had manually hacked LDIF files, rerun YAIM, etc., etc., without success.

This afternoon I just renamed the biomed queue biomed, redefined the queue in site-info.def and reran YAIM's config_gip - finally, the information system started to do the right thing.

How annoying that I wasn't able to really understand the problem.

The old scotgrid production site (scotgrid-gla) has been failing on SAM rm tests for a few days because we have completely run out of storage! After the failure of the large RAID array we only had a single array with 1.6TB. It looks from the SRM logs that both ATLAS and ZEUS have been trying to add more files than we can cope with.

I have managed to secure a few 10s of GB by deleting some old dteam jobs. I have also reactivated a small pool on se2-gla itself and reserved it for ops.

I really need to do an EGEE broadcast today announcing the closure of the old site. Unfortunately the queue length is 200 hours - so we cannot close before Xmas.

First TPM Shift

Finished my first week on TPM duty. It was a far easier task than I had anticipated. It could just all be winding down for Xmas, but the volume of tickets needing processed was far less than I had anticipated.

I found it much easier to find tickets for processing via the web interface than trying to track things via email - by email every single ticket change is sent - I got well over 500 emails in the course of the week, most of which were changes made to tickes being processed and solved. Dealing with that volume would have been unmanegable, but via the web it's an absolute doddle to find the tickets currently assigned to TPM and process them.

I solved two tickets directly: one about DPM pool accounts and the other about DTEAM VOMS/LDAP servers.

We started failing SAM tests yesterday with the rather unhelpful "Unspecified job manager error". This usually means there was a problem starting the job at all. As usual, debugging this is a black art - nothing useful at all in globus-gatekeeper.log or the torque logs. Finally found emails from the batch system to the pool account users stating that there was no space in /home so their job could not be started. Poked around and it was clear it was /home on the WNs which was the problem.

Investigating I found that /home had been filled up with ATLAS jobs which had seemingly unpacked >3GB of software into their torque working directories - filled up /home and left the node dead to the batch system. Eventually we'd built up such a stack of these that SAM jobs found there way here and started to fail. It took quite some time to identify all the relevant WNs and clear out the mess.

Raised a ticket on ATLAS - got a quick response from Fredric and then from Rod. It seems that EDG_WL_SCRATCH was wong. Arggg, so partly my fault after all! I note that in the environment there were 4 environment variables pointing to /tmp (TMPDIR, EDG_TMP, GLITE_LOCATION_TMP, LCG_TMP) but EDG_WL_SCRATCH was pointing to the non-existant /local/glite (initially I had intended to put the large "scratch" partition on the WNs here, but later changed my mind to /tmp).

However, how can ATLAS jobs run with any sort of efficiency if they try and unpack GBs of software?

Things look a bit better for recent jobs, but there is a stack of ~400 ATLAS jobs which seem stuck in the cluster - 20s CPU time but up to days of wall time. I will email Fredric and see if I can delete them to unblock things.

I installed the beta DPM plugins onto scotgrid-gla and UKI-SCOTGRID-GLASGOW today. My instructions were really clear ;-)

Two minor problems:

1. On UKI-SCOTGRID-GLASGOW the DPMINFO user was added with a new style MySQL password, which is not supported by the SL3 version of perl-DBI. I had to rewrite the password hash using the old_password() function.
   
2. I edited the /opt/lcg/var/gip/plugin/lcg-info-dynamic-se with emacs, which left an editor backup file. Unfortunately the /opt/lcg/bin/lcg-info-generic master script will run editor backup files in the plugin directory, so I had two entries generated by plugins, both of which were published in the BDII. Deleting the backup files and restarting MDS cured the problem.
   

I have been working with a local ATLAS user who had to replicate 20,000 files of a PhD student's thesis onto ScotGrid for backup - it was about 1.8TB of data in total.

We discussed the various tools available and he opted for replicating the local directory structure into the ATLAS LFC, then manually generating the catalog entries for each of the files.

I helped him to write loops around the various lcg-utils and lfc client commands and we tried to do a bit of error catching and sanity checking. First of all it is indicative of the very poor state of data management end user tools that such loops have to be written - why isn't there a recurse option on lcg-cr? Why don't commands re-try properly?

Running these loops we have discovered that:

1. LFC is good - no commands failed here. It helps a lot that the server is given manually and not subject to information lookup.
   
2. File copying over SRM is pretty robust, but not 100% perfect. A fair few files failed to copy and had to force a catalog de-registration.
   
3. The information system is dreadful - forcing an LDAP lookup from the BDII for every single filecopy is madness. Many, many BDII timeout errors, connections refused, etc.
   

Why don't we have a decent information system, which at least can failover and/or retry, and why on earth is there no caching? Every lcg-cr requires a fresh lookup of the global LFC for the ATLAS VO - information which pretty much never changes. Think http queries, think squid and we would at least be in the right infrastructure area for robust information systems with failover (redirects) and caching (TTLs).

CLOSE_WAIT strikes again!

Since Wednesday, the number of tcp connections on one of our dCache door nodes that remain in a CLOSE_WAIT state have been steadily increasing. Once the number of CLOSE_WAITs reached the MaxLogin value (100) for the door, there was no further increase. For information, I upgraded to the v1.7.0-19 of dcache-server on Wednesday. UKI-NORTHGRID-LANCS-HEP are also experiencing the
CLOSE_WAIT problem.

I should also add that ~0409 this morning all of the dCache processes on one of my door nodes and the head node died simultaneously. You can see the sharp drop in CLOSE_WAITs in the ganglia plot as the door process stopped. Initially I thought this was connected to the CLOSE_WAIT issue, but deeper investigation showed that dcache-server v1.7.0-20 was released yesterday which was automatically updated on my nodes and subsequently caused the running services to stop working. I know that it must have been the update since the dCache services continued to run on the one node which did not have automatic updates enabled.

The log files contained references to runIO errors before stopping.

11/21 04:06:08 Cell(c-100@gridftp-pool1Domain) : runIO :
java.lang.InterruptedException: runIo has been interrupted
11/21 04:06:08 Cell(c-100@gridftp-pool1Domain) : connectionThread
interrupted

Automatic updates of dCache have been turned off. It is always the case that after upgrade you must re-run the YAIM configure_node function or run the dCache install.sh script. Otherwise tomcat will not start up due to a problem with the definition of the JAVA_HOME variable (even if it has not changed during the update).

So, in summary:

1. The CLOSE_WAIT issue is very much alive and kicking in v1.7.0 of dCache.

2. It is highly recommended to update your dCache by hand to make sure that everything comes back up.

We're back!

I had cfengine installing and configuring the CE and the batch system by early Wednesday, but neither the gatekeeper or the information system were working at all. Then at this point I had to leave things until Thursday afternoon (teaching, writing exam questions and having teeth removed (ouch!)), so I didn't get a chance to look at the gatekeeper until Thursday afternoon.

I couldn't find any significant difference between the files on the old production site and the new system, so what was wrong? Eventualy I attached an strace to the gatekeeper which revealed it could not open the jobmanager-fork file. Checked the permissions and they were 0600!

Turns out that cfengine runs shell scripts with a very aggressive umask of 077, so it had created many configuration files only readable by root. Gatekeeper forks the process as the pool account and the job falls flat on its face...

This is defintely a cfengine gotcha!

Judicious use of find for non-world readable files allowed me to fix things up.

The information system was suffereing the same problem as it was unable to execute the lcg-info-generic script.

It seems this was also causing job aborts for grid jobs on the WNs, as they too were running YAIM with the bad umask. I blasted them with kickstart and cfengine this morning and reinstalled the whole cluster in < 1 hour. Cool!

Subject: batch system completely screwed
Date: Mon, 13 Nov 2006 16:19:31 +0000

David, Tony

I cannot seem to get the batch system stable after this weekend's mess (see http://scotgrid.blogspot.com/2006/11/cluster-had-its-first-weekend-down.html).

torque and maui will run for some 10s of minutes and then lock up and need restarted. The they run for a while more and die again. I have suspicions that some of the worker nodes are bouncing jobs, but with the general unreliability of the system at the moment this is hard to demonstrate one way or the other,

Clearly there's something deeply screwed up here. The rate of dying nodes with MCE errors probably isn't helping anyone.

I don't seem to have any option but to put the site into downtime and completely gut the batch system.

However, I will also take the opportunity to use cfengine to redo the CE and incorporate local type accounts into the batch system as well.

Various deeply offensive words should be said about the unhelpfulness of torque and maui in this situation.

On the bright side, cfengine is now doing a splendid job on the worker nodes and this should be extended.

*sigh*

Graeme

The cluster had its first weekend down - / filled up on the CE so that jobs could not start or end properly as sandboxes could not be gridftped. The information system also went belly up, as the plugins could not write their data. Some processes were clearly hanging as ganglia was reporting a load average of 100+ by this morning, so the node needed power cycled to come back to life.

Since then I have spent the most frustrating time clensing the batch system from all of the dead/zombified batch jobs. This was made worse by the fact that torque seemed to hang when it detected a job which had exceeded its wallclock time. It seemed to try and delete it on the worker node, but of course the job was long since gone, and it did not return. I wasted a lot of time thinking it was maui which was hanging - as maui does hang if torque is not responding properly.

By the time I realised all of this I had rebooted several times and even put the site into downtime in an attempt to purge things.

There seems to be no good way to clear up this mess. Manually checking a few of the jobs which were in state 0 cpu used but running, it was clear they were no longer on the worker nodes, but there seemed to be no good way to automatically probe this status. In the end I had to pick all the old jobs (id < 26800) which were showing 0 cpu time used, and were in state "R", and force their deletion with "qdel -p".

Finally, there were a number of jobs in state "W" which must have come in this morning when things were flaky, which could not be started properly so they had to be qdeled too (see last week's post).

As remedial action, I have moved /var onto the large disk partition and made a soft link in /. Compression of logfiles has been enabled, and at the next downtime /home will also be moved.

For those of you who are interested, ScotGRID-Edinburgh is now consistently passing the SAM/SFT replica management tests. Looking at the the SAM results we have been 'green' for the past week. Digging into the logs it looks like the remote SE being used by the tests has been changing more frequently of late; RAL, CNAF, SARA as well as our old friend lxn1183.cern.ch are mentioned. Even for lxn1183 the tests are passing. Nothing has changed in the configuration of our dCache so (unfortunately) I can't take the credit for solving the mystery.

Hopefully this will be the end of the problem, however, with this being WLCG I won't be surprised if it rears its ugly head again. You have been warned...

Why do we need cfengine? I got an email from Rod Walker (ATLAS) alerting us to the fact that two workers had become job blackholes. Investogation showed they didn't have CA certificates installed, so they would not globus-url-copy a job's output back to the gatekeeper.

Now, I did run a distributed shell install of lcg-CA on these nodes, but with 100 batch workers I failed to notice that this had somehow failed on 2 of the nodes.

With cfengine we have built into the system:

packages:
        grid::
                lcg-CA action=install
                glite-yaim action=install elsedefine=runyaim

So it will check every hour if lcg-CA is properly installed and install it for us if it is not.

Note also how we can define the "runyaim" class when yaim is first installed - this will run yaim automatically after the metapackage is installed (and then triggers the switching on of the batch system).

More details in the wiki soon...

I have made very good progress with cfengine over the last 2 days. I am now able to:

• Manage cfengine from a central location (on the cluster master)
  
• Use cfengine in the usual way to control the copying of files across the cluster
  
• Trigger daemon reloads/restarts when their configuration files change
  
• Ensure that rpm packages are installed, and install them if they are not
  

The next step is to automate the running of YAIM for worker nodes, and the installation and setup of the batch system.

I should probably have a chat with Colin Morey at Manchester to find out how he does this - share best practice!

Some 20 atlas jobs went into a funny state in the batch system, being in "hold". I eventually figured that this was abnormal condition caused by the jobs not starting correctly.

Interestingly the fact that 20 atlas jobs were waiting, even though this was an abnormal wait, caused the GIP plugin to report a very high ERT/WRT for atlas. So there was nothing for it but to cancel these jobs (qdel). As soon as this was done the ERT returned to 0 and more atlas jobs arrived.

Didn't manage to get to the bottom of why they were failing to start though!

I have now installed a site level BDII on svr021. This was dead easy:

1. Use YAIM to setup a top level BDII (metapackage glite-BDII, node type BDII).
   
2. Copy the site level BDII configuration files from the CE.
   
3. Update the GIIS URL in the GOCDB.
   

Tomorrow I will shut down the CE's BDII.

We're suffering from information system dropouts. This seems to happen particularly when jobs come into the site - I can see that the load on the CE jumped to about 10 as the number of jobs ramped up last night. The corresponds exactly to the time that we start to drop out of the BDII.

So it's really urgent that we have a separate site BDII to stop this happening (we failed a few SFTs because of this).

It is a pretty poor show from the BDII to be so sensitive about the CE's load.

Access for "local users" is now being considered at the new cluster. After looking at the methods supported by NGS partner sites, it would seem that this is a reasonable level of "gridness" to ask of users.

This is then based around:

1. All of our users have a grid certificate.
   
2. gsissh login to svr020,to allow seeding of files and perparation of job environment.
   
3. globus-job-submit or edg-job-submit to the gatekeeper (from svr020 or from anyother UI type machine) to submit jobs - i.e., no qsub!
   

At the site level we then need to ensure:

• That certificates from this type of user are mapped to the same user account on the UI and CE.
  
• That home directories for this class of user are shared across the cluster.
  

We will dedicate one of the disk servers to this purpose - together with a suitable automount map.

Investigating our empty queues a little more, we have quite a large number of jobs which seem to exit immediately (0s CPU and walltime). However, may more have better usage patterns, e.g. 25h09m cput, 25h16m wallclock.

Everything has exit status 0, so I really can't see anything that the site is doing wrong.

Puzzling...

There seems to be a problem with publishing SAM/SFT tests on our site. Although we were passing every time until 2006-10-25, suddenly the publication of the tests on the SAM sebsite stopped.

Checking the torque logs it's clear that these tests are still running - just not being published. I have raised this as a (now urgent) GGUS ticket: https://gus.fzk.de/pages/ticket_details.php?ticket=14737.

This seems to be affacting the number of jobs running through the cluster seems to be dropping and dropping. After peaking at 200+ we have dropped to less than 20 runnng jobs after 10 days!