Development / PreProd : The CE

So on with the show and the creation of a development/preprod cluster. Next up the CE on dev machine 10.

Step 1. Again involved a quick search of the cfagent.conf. An additional entry was added in the ce groups stanza to add in the devce. cfagent -qv was used to pull down/install the node.

The first yum update updated all the certificates was successful. However, it looks again like glite-yaim-core was not successful first time around.

/opt/glite/yaim/bin/yaim: No such file or directory

A second run of cfengine caused the error below.

Transaction Check Error: file /usr/share/java/jaf.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp
file /usr/share/java/jaf_api.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp

This is a known issue and the fix is to disable the jpackage17 repo like so:

/usr/bin/yum -y install lcg-CE glite-TORQUE_utils --disablerepo=jpackage17-generic

This allowed a full installation of the relevant packages. This was run by hand in the first instance and then was added to the development configuration file.

Step 2. Running yaim by hand for the CE and then adding additional stanzas into the development configuration file. So installing the CE: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n lcg-CE -n TORQUE_utils

There appeared to be various quirks/warnings/errors with this. Here is a short summary.

There appeared to be an issue with line 36 of the config_apel_pbs.
Atfer inspecting the file it appears that the line separator in the file was not working as expected.

/opt/glite/yaim/functions/config_apel_pbs: line 36: APEL_DB_PASSWORD: command not found

This is a known bug https://savannah.cern.ch/bugs/index.php?39014 and was temporarily fixed by removing the additional whitespace before the /

These errors appeared next but did not seem to cause any problems.

/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32dbg.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32pthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32dbg.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32pthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32dbgpthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32dbgpthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_nog.so.0 is not a symbolic link

This is due to a known bug: https://savannah.cern.ch/bugs/?42481

INFO: Now creating the grid-mapfile - this may take a few minutes...
voms search(https://voms.gridpp.ac.uk:8443/voms/supernemo.vo.eu-egee.org/Role=lcgadmin/services/VOMSCompatibility?method=getGridmapUsers): /voms/supernemo.vo.eu-egee.org/Role=lcgadmin/services/VOMSCompatibility

voms search(https://voms.gridpp.ac.uk:8443/voms/ukqcd.vo.gridpp.ac.uk/Role=lcgadmin/services/VOMSCompatibility?method=getGridmapUsers): /voms/ukqcd.vo.gridpp.ac.uk/Role=lcgadmin/services/VOMSCompatibility

Exit with error(s) (code=2)

WARNING: It looks like /opt/globus/tmp/gram_job_state may not be on a local filesystem. WARNING: The test for local file systems is not 100% reliable. Ignore the below if this is a false positive.
WARNING: The jobmanager requires state dir to be on a local filesystem
WARNING: Rerun the jobmanager setup script with the -state-dir= option.Creating state file directory.
Done.

find-fork-tools: WARNING: "Cannot locate mpiexec"
find-fork-tools: WARNING: "Cannot locate mpirun"

find-lcgpbs-tools: WARNING: "Cannot locate mpirun"
checking for mpirun... no

Any clues to these errors would be greatly appreciated too.

Currently when a new CE is configured yaim attempts to run the function config_gip_vo_tag
This function attempts to create and change permission on the VO tags directory.
However, this directory is mounted on the Scotgrid cluster and comes pre-configured so to speak.

INFO: Executing function: config_gip_vo_tag
chmod: changing permissions of `/opt/edg/var/info/atlas': Operation not permitted
chmod: changing permissions of `/opt/edg/var/info/atlas/atlas.list': Operation not permitted
chmod: changing permissions of `/opt/edg/var/info/cms': Operation not permitted
ERROR: Error during the execution of function: config_gip_vo_tag
ERROR: Error during the configuration.Exiting. [FAILED]
ERROR: One of the functions returned with error without specifying it's nature !
INFO: Using locally defined function /opt/glite/yaim/functions/local/config_gip_vo_tag
cfengine controls config_gip_vo_tag. yaim function disabled.

The fix to this issue was to override the function gip_vo_tag in the yaim/function/local directory to make sure it did not try to change any of the NFS mounted directories.

Again the bdii issue reared it ugly head. https://savannah.cern.ch/bugs/index.php?40675

Starting glite-lb-interlogd ...chown: cannot access `/opt/bdii/var': No such file or directory
sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the /opt/bdii/doc/schemas.example file to /opt/bdii/etc/schemas and re-run yaim.
The re-run of yaim also fixes the chown of /opt/bdii/var.

All of the above workarounds have now been added to the cf.dev script that is included in the main cfagent.conf script. Thus all the temporary workarounds and development stanzas are kept out of the main script.

Step 3 - Testing:

First off I thought I could use the glite set of commands for the development cluster.

jdl extract:

Requirements = other.GlueCEUniqueID == "devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m";

submission extract:

-bash-3.00$ glite-wms-job-list-match -a hello.jdl
Connecting to the service https://devmachine9.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

==================== glite-wms-job-list-match failure ====================
No Computing Element matching your job requirements has been found!
==========================================================================

However, after some initial tests and some thought on the subject it became apparent that it would have to be entered into the site BDII for this to work.
This made sense since the WMS queries the site BDII to get information relating to the published queues from the CE.

Therefore, without setting up another siteBDII/BDII for the mini cluster, direct job submission via GLOBUS seemed like the way to go for intial testing.

-bash-3.00$ globus-job-submit devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs /bin/hostname -f
https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
PENDING
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
PENDING
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
DONE

However when I tried to obtain the job output: -bash-3.00$ globus-job-get-output https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/

--- Nothing, Nada! Doh!

After some investigation involving the logs on the CE and the logs on torque it became apparent that the torque was not allowing the job submission from the new CE. After some more investigation this seemed to be down to a file called hosts.equiv. This is file that holds the white list of hosts that torque will talk to.

Therefore, after added the new CE and restarting Torque:

-bash-3.00$ globus-job-run devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
node192.beowulf.cluster

Woohoo, we have a working CE that allows submission through Globus, Now to test the CE with WMS submission we need a dev site bdii. So I think I will install on the dev UI and make sure we can submit through glite-wms-job-submit.

Development / PreProd : The WMS

I thought I would continue my foray into grid middleware installations another quick blog on the workload management system or WMS as its affectionately known. . With the old development cluster very much moved/dead we now have dev008 -> dev013 as a sandbox for installs and upgrades. So after the UI install last year the next piece of the jigsaw was the slightly more heavy weight WMS.

Step 1. Was a quick search of the cfagent.conf. This returned the necessary files, links, packages to install. An additional entry was added in the wms groups stanza and cfengine was ran immediately using cfagent -qv

The first yum update in fact updated all the certificates from the lcg-CA, lcg-vomscerts package stanzas and these were successful. It then looked like it attempted to run the glite-yaim-core which could not have successful as it then through this error:

Executing script /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -n glite-LB...(timeout=0,uid=-1,gid=-1)
(Setting umask to 22)
cfengine:dev009:m/bin/yaim -c -: sh: /opt/glite/yaim/bin/yaim: No such file or directory

This looks like on the first run after a complete re-install it trys to runs yaim without installing yaim first or infact actually running the WMS package stanzas. Slightly bizarre.

However, another run of cfagent -qv seemed to work as this correctly ran through the WMS package stanzas of glite-WMS, glite-LB, rbwmsmon. Although in the spirit of grid wierdness there were a few warnings when it installed Condor just to make you wonder what was going on!

Installing: condor ##################### [ 58/105]WARNING: Multiple network interfaces detected. Condor might not work
cfengine:dev009: properly until you set NETWORK_INTERFACE =
cfengine:dev009: Unable to find a valid Java installation
cfengine:dev009: Java Universe will not work properly until the JAVA
cfengine:dev009: (and JAVA_MAXHEAP_ARGUMENT) parameters are set in the configuration file!
cfengine:dev009: Condor has been installed into:
cfengine:dev009: /opt/condor-6.8.4
cfengine:dev009: In order for Condor to work properly you must set your
cfengine:dev009: CONDOR_CONFIG environment variable to point to your
cfengine:dev009: Condor configuration file:
cfengine:dev009: /opt/condor-6.8.4/etc/condor_config
cfengine:dev009: before running Condor commands/daemons.
cfengine:dev009:

After some internet searching and logging in/out I could see that CONDOR_CONFIG was actually set correctly.

dev009:~# echo $CONDOR_CONFIG
/opt/condor-c/etc/condor_config

Step 2. Install yaim by hand. Then blow dev009 away and let cfengine do the whole lot.

Before running yaim I made sure that dev009 was included in the site-info.def. I first created a file in the node directory within yaim to override the current production WMS and this was okay to configure the WMS first time around. However, on successive runs of cfagent it overwrote the site-info.def - doh! Therefore, Mike suggested having a development site-info.def which is copied over the current production each time cfagent is run. This worked a treat.

So on with the show with yaim for the WMS: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -n glite-LB

Although yaim appeared to run successfully. There were a few warning/errors which had to be explained.

cfengine:dev009:m/bin/yaim -c -: WARNING: Only 1 pool account defined for tag 'sgm' of VO VO.PANDA.GSI.DE
cfengine:dev009:m/bin/yaim -c -: users_getspecialgroup: could not find 'sgm' user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialprefix: could not find 'sgm' prefix for BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialgroup: could not find 'prd' user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialprefix: could not find 'prd' prefix for BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: INFO: users_getspecialusers: could not find sgm user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: ERROR: Could not determine mapping for tag 'sgm' of VO BIOMED
cfengine:dev009:m/bin/yaim -c -: INFO: users_getspecialusers: could not find prd user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: ERROR: Could not determine mapping for tag 'prd' of VO BIOMED
cfengine:dev009:m/bin/yaim -c -: WARNING: No mapping found for "/biomed/Role=lcgadmin" in /tmp/yaim.vF5821
cfengine:dev009:m/bin/yaim -c -: WARNING: No mapping found for "/biomed/Role=production" in /tmp/yaim.vF5821

These are normal errors/warning and can be explained as the Panda VO does indeed have only one sgm account since there is only one panda account -Dan! The Biomed VO conversely has no sgm, prd or admin accounts and only run as plain users. Therefore, these messages are expected. The next warning was slightly more worrying since these undefined variables could be required.

cfengine:dev009:m/bin/yaim -c -: [Fri Dec 19 13:35:50 2008] [warn] PassEnv variable GLITE_WMS_WMPROXY_WEIGHTS_UPPER_LIMIT was undefined
cfengine:dev009:m/bin/yaim -c -: [Fri Dec 19 13:35:50 2008] [warn] PassEnv variable GLITE_SD_VO was undefined

After some searching on the web this is a warning and yaim will just use the default values. There also seemed to be some issue with the schema file in the WMS BDII section of yaim:

cfengine:dev009:m/bin/yaim -c -: Starting glite-lb-interlogd ...chown: cannot access `/opt/bdii/var': No such file or directory
cfengine:dev009:m/bin/yaim -c -: sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the /opt/bdii/doc/schemas.example file to /opt/bdii/etc/schemas and re-run yaim.
The re-run of yaim also fixes the chown of /opt/bdii/var.

Step 3. Testing from dev008 the development UI yielded:

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://dev009.gla.scotgrid.ac.uk:9000/d0nTvt6udraqpjs0Mx-eOw
Current Status: Done (Success)
Exit code: 0
Status Reason: Job terminated successfully
Destination: svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
Submitted: Thu Jan 15 17:19:59 2009 GMT
*************************************************************

Therefore, I conclude: we have a working development WMS on dev009

Trac now secured

I have been playing with Trac in my spare time off and on and Mike had noticed quite a few sensitive files lying around in the repo. So I have secured the Scotgrid Trac instance so you now required to sign in. This will then allow you to browse the repo as you normally would. The user/pass can be obtained from anyone here at Scotgrid.

now to get it externally accessible!

DNS goes wibble wobble...

Various funny things were happening today:

• General sickness in the atlas pilot factory.
• Quite a few BDII dropouts.
• SAM test failures from the above.
• Sluggish clients on our UIs.
• Very slow logins from CERN.
  

All things that pointed towards a slow/failing DNS. When I wrote a little test script with 30 forward/reverse DNS queries it was taking 20-50s on some servers and 0.5s on others.

The slow ones had been configured to look at a dnsmasq cache on our headnode, which for unknown reasons was going very slowly (even a restart did not help).

I reconfigured to take out the cache and suddenly all was rosy again across the cluster.

Curiously we had added the cache to overcome problems with campus DNS in the first place.

At least with things configured via cfengine this is a very easy change to make right across the cluster.

Source Control with Subversion Resurrection

With forthcoming SA4 tasks to update the scotgrid website and create a local registery of user information we have decided to resurrect the use of subversion as a source control system.

As we all know source control is basic software development practice and is jolly good idea for anything a team of developers want amend on a regular basis. So here it is being used once more on grid01. The current repo is located at /SVN.

The new project for the scotgrid website is scotgrid_www and can be checked out from the usual svn co svn+ssh command or any other svn gui tool. We plan to use this repo for all fabric management scripts. There is already a scotgrid repo which contains similar scripts but this is very out of date and will probably be blown away at some point.

Trac has been installed on grid01 (only accessible from the internal network at present - more to follow on this one ) and is running using the simple trac webserver. I have written a simple daemon to start/stop the service.

This gives a nice http interface to the repo to view files and trac changes. There is also a wiki/project management/ticket management facilities.

I will look to get the backing up of the repo automated in some way.

Confessions of a Data Management Systems Manager

After my cunningly timed arrival at Glasgow, barely two weeks before the start of Christmas Break (actually, I suspect they call it "Winter Break" now, although "Io Saturnalia!" would be both more fitting and more amusing), I've tried to hit the ground moving at a vaguely speedy pace on Storage / Data managementy things.

So, as the new Andrew Elwell, here's what I've managed to do so far:

dpm-sql-usage-by-vo-user
Partly as a means of getting myself better acquainted with the arcane mysteries of the DPM, I wrote this useful little tool which produces a pretty-printed output of all the storage used on a DPM, by VO and users within the VO.
Greig and I are planning to stick it in the next release of his DPM Admin Tools package but anyone who wants a beta release can have it if they ask.


DPM performance & xrootd
After the series of ATLAS Analysis Challenges made it increasingly clear that DPM can't produce an effective event rate of greater than about 12 Hz on any of the sites in the challenge, we decided this was worth some investigation. (Interestingly, Tokyo's cluster seems to be capable of getting upto 24 Hz, with DPM.)
At this rate, the DPM head node maxes out CPU, but the network rates from the head node and the pools are very low.

It appears from the DPM logs at Glasgow that the majority of the DPM's time is spent doing X509 authentication on each get request - since each authentication takes around 1.5 seconds, and we need two per request (one on the DPM and one on the disk pool), this is the majority of the time involved in the transfers for small files like the AODs (about 30Mb each).

We thought, therefore, that we'd try disabling X509 auth on the Glasgow DPM and getting another Challenge send to us. This involves some fairly dangerous settings in shift.conf on all the DPM nodes, which we did, and it seemed to work, with a noticeable speed increase, for rfcp on a node.
For some reason, though, the ganga jobs in the Analysis challenge did this:

which is clearly not expected.
We're still not sure why running DPM in "no X509", trusted mode breaks ganga submitted jobs in this way - it didn't break any of ATLAS Production, and rfcp and lcg-cp both worked when we tested them. In any case, we undid these changes sharpish...

The next avenue for testing is alternative transfer protocols other than rfio. Luckily, we have a "spare" DPM, svr025, which I've added xroot support to (thanks to some help from Greig), and will be using to test the benefits and efficiencies of the various DPM plugins vs rfio. Next year, we'll see how I've gotten on...

Am I seeing double site bdii?

With the imminent move of the development rack we need to move some of the important grid infrastructure out of the current dev rack and into a permanent production home in clustervision. To minimise site downtime we would like to create a temporary scotgrid BDII on svr027 (currently unused). So here goes.....

when running cfagent -qv it ran successfully on svr027 through the files, links, editfiles, packages including the correct glite-BDII packages and copy sections

All was going well until YAIM.

notes from configuring the UI

running yaim for a UI node will configure the UI, /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n BDII_site

this caused the following errors:

cfengine:svr027:m/bin/yaim -c -: INFO: Executing function: config_edgusers
cfengine:svr027:m/bin/yaim -c -: chown: cannot access `/opt/bdii/var': No such file or directory
cfengine:svr027:m/bin/yaim -c -: sed: can't read /opt/bdii/etc/schemas: No such file or directory
cfengine:svr027:m/bin/yaim -c -: INFO: Executing function: config_bdii_only
Stopping BDII27:m/bin/yaim -c -: [FAILED]
cfengine:svr027:m/bin/yaim -c -: Starting BDII [ OK ]

These errors were slightly puzzling but I realised that I had not changed anything in the site-info.def.
So I changed the SITE_BDII_HOST parameter from this:

SITE_BDII_HOST=svr030.$MY_DOMAIN

to this:

SITE_BDII_HOST="svr030.$MY_DOMAIN svr027.$MY_DOMAIN"

and re-ran /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n BDII_site

This time the only error was:

sed: can't read /opt/bdii/etc/schemas: No such file or directory

but the configurator still produced:

INFO: Configuration Complete. [ OK ]
NFO: YAIM terminated succesfully.

checking the /opt/bdii/etc on svr027 I had this:

svr027:/opt/bdii/etc# ls -la
total 64
drwxr-xr-x 2 edguser edguser 4096 Dec 17 16:17 .
drwxr-xr-x 6 root root 4096 Dec 17 15:55 ..
-rw-r----- 1 edguser edguser 503 Dec 17 16:17 bdii.conf
-rw-r--r-- 1 edguser edguser 2535 Oct 13 13:54 BDII.schema
-rw-r--r-- 1 edguser edguser 50 Oct 13 13:54 bdii-update.conf
-rw-r--r-- 1 edguser edguser 634 Oct 13 13:54 DB_CONFIG
-rw-r--r-- 1 edguser edguser 246 Oct 13 13:54 default.ldif
-rw-r--r-- 1 edguser edguser 1783 Oct 13 13:54 glue-slapd.conf

checking this against svr030 I had this:

svr030:/opt/bdii/etc# ls -la
total 48
drwxr-xr-x 2 edguser edguser 4096 Oct 8 10:35 .
drwxr-xr-x 6 root root 4096 Feb 10 2008 ..
-rw-r--r-- 1 edguser edguser 364 Oct 8 10:35 bdii.conf
-rw-r--r-- 1 edguser edguser 50 Feb 10 2008 bdii-update.conf
-rw-r--r-- 1 edguser edguser 377 Feb 10 2008 indexes
-rw-r--r-- 1 edguser edguser 268 Oct 8 10:35 schemas

very different!

I then decided to reboot and try again from scratch just to make sure there was nothing hanging around from the previous failure.
When I installed everything in the same way. The file structure still appeared different. So i decided to test the site level BDII to see if it actually worked.

svr027:/opt/glite/yaim/etc# ldapsearch -xLLL -b mds-vo-name=UKI-SCOTGRID-GLASGOW,o=grid -p 2170 -h svr027.gla.scotgrid.ac.uk > svr027.txt
svr027:/opt/glite/yaim/etc# ldapsearch -xLLL -b mds-vo-name=UKI-SCOTGRID-GLASGOW,o=grid -p 2170 -h svr030.gla.scotgrid.ac.uk > svr030.txt

this was then compared: cat svr027.txt | sort > ldapsvr027.txt;cat svr030.txt | sort > ldapsvr030.txt;diff -y ldapsvr027.txt ldapsvr030.txt | grep '>' | grep '.gla.scotgrid'

On comparing the output from an ldap search it apparent that something was missing as their output showed some missing servers. After a quick discussion with Sam we found the file /opt/glite/etc/gip/site-urls.conf and noticed the differences: the DPM2 and BDII_TOP i.e. svr025 and svr019

svr027:/opt/glite/etc/gip# cat site-urls.conf
CE ldap://svr021.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
CE2 ldap://svr026.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM ldap://svr018.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS ldap://svr022.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS2 ldap://svr023.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII ldap://svr027.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
VOBOX ldap://svr024.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid

svr030:/opt/glite/etc/gip# cat site-urls.conf
CE ldap://svr021.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
CE2 ldap://svr026.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM ldap://svr018.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM2 ldap://svr025.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS ldap://svr022.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS2 ldap://svr023.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII ldap://svr030.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII_TOP ldap://svr019.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
VOBOX ldap://svr024.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid

after updating svr027 and restarting /etc/init.d/bdii restart we now have a operational site BDII on svr027.

The question is, should these additional entries be in /var/cfengine/inputs/skel/yaim/services/glite-bdii?

On with the move!

Update: svr027 is currently the only SITE_BDII in the GOC DB

Analysis Challenge: Round 4

We re-run the analysis challenge yesterday with a better MySQL setup so that the higher number of dpns daemons could get db connections. However, the results were much the same as before and the conclusion seems to be that X509 sucks - it's killing the headnode with all of the simultaneous authentications.

We hope to prove later on that this is the real problem and then think about what we can do about it...

Development / PreProd : The UI

I thought my first foray into grid middleware installations deserved a quick blog so here goes. Apologies in advance if I am covering old ground.

With grid02 now defunct and dev008 very much part of the alive and kicking it was time to install the required packages/middleware and configure it to run as a UI.

First thing for me was to understand/create a cfagent script for the new host. After much deliberation of wishing to keep it all separate and out of the way of the main production script. I decided to add it into the main script to save duplication. Perhaps something to think about for the future may be to split this up into much smaller modules per host and import a few common modules. Although, at this stage I am inclined to go with the old adage, "don't fix it if it ain't broken". I have also heard/read much of puppet which is built on cfengine with bells and whistles. Perhaps something to look at? Anyway, on with the install.

Once the script was created I ran cfagent -qv . However, beginner's luck was thin on the ground and it failed to install the packages properly first time around.

First off there was a missing dependency:

cfengine:dev008: --> Processing Dependency: perl(URI::URL) for package: perl-libwwError: Missing Dependency: log4cpp >= 1.0 is needed by package glite-ce-cream-client-api-c
cfengine:dev008: Error: Missing Dependency: liblog4cpp.so.4 is needed by package glite-ce-cream-cli

The fix was to include the DAG repo onto dev008 to pull a later version of log4cpp.
However, there was some issues surrounding this as the UI is 386 and the grid machines we have are generally 64 bit machines.
So the DAG repo url in /etc/yum.repos.d/dag.repo had to be fudged to change the /$basearch variable to i386

After a yum clean all I ran cfagent -qv again. This resulted in a second error:


cfengine:dev008: Transaction Check Error: file /usr/share/java/jaf.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp
cfengine:dev008: file /usr/share/java/jaf_api.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp


This was a known error with the middleware install and the fix was to run yum install glite-UI --disablerepo=jpackage17-generic

After a third run of cfgent -qv it was good to go or so I thought. What I did see was that it was running YAIM and failing. Therefore, I opted to run YAIM manually. Using the normal UI command, /opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI I generated the following error:


INFO: Executing function: config_workload_manager_client_setenv
INFO: Executing function: config_workload_manager_client
ERROR: RB_HOST is not set
ERROR: One of the functions returned with error without specifying it's nature !


After a quick cat of the site-info.def, indeed RB_HOST is commented out as presumably the WMS is in there instead.


WMS_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
LB_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
#RB_HOST=svr023.$MY_DOMAIN


I managed to amend the local site-info.def before cfagent set it back to the original value and this allowed YAIM to get further. After reading some sites, I opted for this config as it appeared that you could actually have WMS_HOST and RB_HOST defined in the one file. Perhaps a WMS install will not like this setting? We will have to see.


WMS_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
LB_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
RB_HOST=$WMS_HOST


running yaim again: opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI now returned some errors when build the globus core:


gpt-build ====> Changing to /etc/grid-security/vomsdir/BUILD/globus_core-4.30/
gpt-build ====> BUILDING FLAVOR gcc32
GLOBUS_LOCATION=/opt/globus; export GLOBUS_LOCATION; GLOBUS_CC=gcc; export GLOBUS_CC; /etc/grid-security/vomsdir/BUILD/globus_core-4.30//configure --with-flavor=gcc32
Dependencies Complete
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... no
checking for cc... no
checking for cc... no
checking for cl... no
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.


It looked bizarrely like gcc is not installed by cfengine on a sl4.i386 version by default, so to fix: yum install gcc . After checking the cfagent.conf this does appear to be the case. There are lots of additional packages for sl4.x86_64 but not for i386. Should this be the case?

After another re-run of yaim: opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI


INFO: Configuration Complete. [ OK ]
INFO: YAIM terminated successfully.


This looked better and after sourcing the grid-env that had just been installed: source /etc/profile.d/grid-env.sh commands like: voms-proxy-init -voms vo.scotgrid.ac.uk were successful. In fact I was able to submit a job and retrieve its data from dev008. So installation successful. Or so I thought. I updated the cfagent.conf and ran it all from cfengine.

Cfengine appears to make two passes. The 1st pass install works correctly. It installs the UI, configures through YAIM. However, since some of the fileedits and links rely on the existence of a configured glite they actually fail on the first pass i.e


cfengine:dev008: Error while trying to link /opt/glite/bin/python2 -> /usr/bin/python32
cfengine:dev008: Error while trying to link /opt/glite/bin/grid-proxy-init -> voms-proxy-init
cfengine:dev008: Error while trying to link /opt/glite/bin/grid-proxy-info -> voms-proxy-info
cfengine:dev008: Couldn't stat /opt/glite/etc/glite_wmsui_cmd_var.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/edg/etc/edg_wl_ui_cmd_var.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/glite/etc/gaussian/glite_wms.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/glite/etc/gaussian/glite_wmsui.conf - no file to edit


I had expected these to be caught on the second pass as glite was installed and configured but that run of cfagent -qv does not pick them up on the second pass. When cfagent -qv is ran a second time it does update the files appropriately. Not sure this is the behaviour we want. Does anyone remember if this happened with the original UI? Currently the dev008 is using all the original classes for ui and clusterui at the moment and should be running in the same way as the original UI install.

So to summarise the questions:

1. Can you set RB_HOST and WMS_HOST in the same site-info.def?
2. Are there lots of packages missing for a sl4.i386install?
3. Does anyone remember from the original UI install what happens when it updates files on the second pass?

So a partial success, now onto a WMS.

Plots from last analyais challenge

Mostly confirmed the results which we saw at the end of last week's test. Load on the DPM headnode is our pressing concern - it's maxing its CPU out even at open rates of a little over 1Hz.

Analysis Challenge: Round 3

Last week's analysis challenge at Glasgow showed extreme load and sluggishness in the DPM (see the attached plots of awfulness). Although we managed a much better event rate we also suffered from incomplete processing and the DPM was a clear bottleneck.

I had a chat with JPB today who spotted the very high memory consumption of the dpm daemon - he thinks there's probably a memory leak and that this might be slowing things down. He also said it might be worth running more dpns daemons as these also do connection athentication.

So, to get ready for tomorrow I have:

1. Allowed core dumps for the DPM and DPNS daemons.
2. Increased the number of threads in the DPNS daemon to 60.
3. Restarted all the daemons.

That last operation freed up about 3GB of memory!

If we still see problems tomorrow then at least we should have some good information for the developers to chew on.

You want processing power?

Well we've got it at Glasgow; over 2,900,000 SI2k worth now, since we commissioned our cluster extension this week.

Following a period of, ahem, rigorous stress-testing (see Graeme's posting), we started releasing nodes to the Grid on Wednesday, and our first job (an ATLAS production task) arrived almost immediately. To date, the new nodes alone have handled over 14,000 jobs.

This now means that Glasgow are currently top of the UKI leaderboard in terms of raw processing power and, according to Gridmap, only behind RAL-LCG2 for the number of job-slots available.

For my next trick, I will make 400TB of storage appear...as if by magic...

Farewell and godspeed! Welcome!

Last week we bid a fond farewell to Andrew, who has moved on to a job in the gLite team at CERN. He did a great power of work of us, tweaking networks, pioneering regional nagios and generally being a smart and useful guy. Good luck to him... and we know where you live if it goes wrong :-) Doubtless we'll see you in R1 for a beer.

Andrew's replacement will be Sam, who moves along the M8 from Edinburgh to Glasgow and will start very soon (next week). We're very happy to have someone in the role who already knows grid so well.

Finally, I should bid welcome to Dug McNab, who started a few weeks ago as the ScotGrid EGEE T2 Co-ordinator. Welcome to him. Dug, among other things, has the task of teaching all the non-LHC people how to do data management properly!

All go at ScotGrid

This is a quick update to make up for the fact that we've been too busy to blog here in ScotGrid land - lack of activity in the blog rather indicates a frenzy of activity on the ground!

Glasgow:

• The new Viglen hardware arrived, was installed and passed its acceptance test without any problems. However, we did have severe air conditioning issues in the new computer room which prevented us from actually switching on the new kit in anger (we didn't want it to cook itself!). These were cured at the end of last week, when a failover between the two chilled water pumps was installed. Since then Mike has been proving the new worker nodes in the batch system and we're on the point of bringing the new nodes online.
• Meanwhile, in ATLAS land, I have been helping to organise UK Distributed Analysis Challenge. This has been hammering our system with 100s of ATLAS user analysis jobs. The first round we had inherited a bad setting of rfio readahead, so we delivered GB of data to the jobs which they did not want. Second time around this was cured, but it looked like we had serious load issues on the DPM headnode and some files could not be opened by jobs. What's worrying here is that we peaked at about 110 user analysis jobs running simultaneously, yet DPM really struggled to keep up with the rate of opens - to be investigated later.
• On the middleware front I installed a new CE (svr026) to provide redundant access to the batch system and a 'hot spare' DPM (svr025) which is there to (a) investigate peculiar client timeout errors we see with svr018 (do they repeat? initial answer seems to be no) and (b) provide a 'ready to go' DPM headnode if anything unfortunate happens to svr018.
  

Edinburgh:

• ECDF has been working much better using mw05, the new SL4 SGE CE. Also, thanks to continual pressure from Phil, we nailed the last of the VSZ problems (the sgm accounts had the low VSZ limits which caused the installation fo software to fall over in very peculiar ways). Since then ATLAS as run very well at ECDF.
• Continuing the CE improvements, Sam and Steve hope to introduce a second ECDF CE and retire the old SL3 CE very soon.

Durham:

• Durham's new kit (all 1MSI2K of it) should arrive very soon now, so they will revamp the whole cluster and dump the old kit. They will be in downtime for a while as this happens. They are taking the ScotGrid lead on virtulaising services which we see as a really important step to providing rapid recovery from equipment failures and lots of flexibility in deployment.

Finally, we have seen a welcome return of LHCb production jobs; had some serious gripes with biomed (I think they are disabled on all our SEs now) and seen some excellent SAM test figures for all the sites, despite generaly being full to the gunnels with jobs.

mmm, shiny (take 2)

It's here, It's grey and It's cubic. Oh and it has lots of blinkenlights. (aside: but not as many as here -- go and look at the arcade videos) - its the new cluster!

The Vendor install team took 3 days to get the basics in and on a burn-in, now its up to us to get the build and acceptance done. Mike has been working tirelessly on it and has been 'aquainted' with some of the rougher bits of YPF that we never really finished off (keygeneration, interface config scripts etc --- stuff that only needs to be done once per big group of new hardware delivery)

Anyway - we've now fired off a pile of installs and have node142-310 built and running stress (ta Mike) and I've been working on the disk servers -- they're now in a test pool and will be getting a pile of files written shortly.




what else -- The cold aisle containment seems good -- makes a huge difference to the airflow efficiency - hopefully reflected in the aircon load efficiency.

more news as we go and more pics soon, but you've gotta love the simple dpm-queryconf output of

CAPACITY 363.79T FREE 362.14T (99.5%)

Don't panic - it's only a test...

Hmm. We had a malicious user's DN on the glasgow system this morning. Am sure that other UKI sites may be affected too. Be careful with your cleanup processes as we missed something the 1st time round. Grr.

"Oh no! Not again..."

After being all enthusiastic that the gSOAP errors had been nailed, we failed two SE tests in the last 24 hours. Exactly the same issue as before.

As this error message is so vague it looks like lcg-rollout is our only hope.

I note in passing that Glasgow has one of the most reliable SEs in the UK for ATLAS (2.1% job loss, only beaten by Oxford who have 0.8%; UK average in Q3 was 8% loss) so this is particularly galling.

Shouldn't we be making the results as seen by our real customers rather more important than a once an hour stab in the dark from ops?

Death to gSOAP...

Even after the successful upgrade of DPM we started to get plagued again by SAM test failures with the generic failure message:

httpg://svr018.gla.scotgrid.ac.uk:8443/srm/managerv1:
CGSI-gSOAP: Error reading token data header: Connection closed

This time they came principally from the SE test, instead of from the CE-rm test.

For a while I wondered if there was a DNS problem, but this seemed unlikely for two reasons:

1. Durham use the .scotgrid.ac.uk domain, but they don't see errors.
2. We see the connection in the srmv1 logs, so the host can be resolved.

Then I started to wonder if there was a CRL problem as we occasionally get CRL warnings from SAM WN tests. We have an optimised CRL download system at Glasgow - the CE downloads CRLs as normal, then the remaining nodes mirror the CRLs from the CE. This means we make 1 outbound connection every 6 hours, instead of 150, which seems eminently sensible on a large cluster. However, the default crons for the nodes are 6 hours to process CRLs, which means that CRLs could be up to 12 hours old, in the worst case, on client nodes.

On this suspicion I changed the CE configuration to download CRLs every hour and for the clients do download these from the CE every 4 hours.

I made this change on Friday and, so far, we haven't seen the error again.

My eternal complaint with X509/openssl is why the error is reported as "CGSI-gSOAP: Error reading token data header: Connection closed" and not "CGSI-gSOAP: Error reading token data header: Connection closed [CRL for DN BLAH out of date]".

Is that so very hard to do?

ScotGrid Edinburgh progress

Finally we are green for the latest Atlas releases...

We've made a lot of progress this past week with ECDF. It all started on Friday 10th Oct when were trying to solve some Atlas installation problems in a somewhat ad hoc fashion.
We then incorrectly tagged/published having a valid production release. This then caused serious problems with the Atlas jobs, which resulted in us being taken out of the UK production and missing out on a lot of CPU demand. This past week we've been working hard to solve the problem and here are a few things we found:

1) First of all there were a few access problems to the servers for a few of us. So it was hard to see what was actually going on with the mounted atlas software area. Some of this has now been resolved.

2) The installer was taking ages and them timing (proxy and also SGE killing it off eventually). strace on the nodes linked this to a very slow performance while doing many chmod write to the file system. We solved this in a two fold approach
- Alessandro modified the installer script to be more selective regarding which files needs chmoding, but the system was still very slow.
- The nfs export was then changed to allow asynchronous write which helped speed up the tiny writes to the underlying LUN considerably. There is a worry now of possible data corruption, so should be borne in mind if the server goes down and/or we have edinburgh specific segv/problems with a release. Orlando may want to post later information about the nfs changes.

3) The remover and installer used ~ 3GB and 4,5 GB of vmem respectively and the 6GB vmem limit had only been applied to prodatlas jobs. The 3GB vmem default started causing serious problems for sgmatlas. This has now been changed to 6GB.

We're also planning in the ce to add "qsub -m a -M" SGE options to allow the middleware team to monitor better the occurence of vmem aborts. We also might add a flag to help better parse the SGE account logs for apel. Note: the APEL monitoring problem has been fixed. However, that's for another post (Sam?)...

Well done to Orlando, Alessandro, Graeme and Sam for helping us get to the bottom of this!

Well Done Guys!

Well, I was waiting for Mike and Andrew to blog this, but they haven't. They very successfully upgraded Glasgow's DPM to the native 64bit version on Monday last week (when we had upgraded to SL4 only the 32 bit version was available). This was a significant step forwards but required the head node and all of the disk servers to have their OS rebuilt without losing data, and the database restored onto the head node.

It went very well and we were up and running again within 6 hours - no data lost!

We are also seeing an improvement in the SAM test results, with the spurious 'gSOAP' errors which were plaguing us now seemingly having gone (fingers crossed!).

It's terrible that the LHC is not running right now, but it does mean that interventions like this can be done.

Great work guys!

chew 'em up, spit 'em out...

Failed SAM tests all day. When I checked the logs they'd all run on
node006. Logged in and...

Oct 11 16:58:40 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 17:28:40 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 17:58:41 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 18:00:14 node006 pbs_mom: Invalid argument (22) in mem_sum, 5754: get_proc_stat
Oct 11 18:13:23 node006 pbs_mom: Invalid argument (22) in resi_sum, 8121: get_proc_stat
Oct 11 18:28:40 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 18:58:40 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 19:28:40 node006 smartd[3163]: Device: /dev/hda, 1 Currently unreadable (pending) sectors
Oct 11 19:44:32 node006 pbs_mom: Invalid argument (22) in resi_sum, 9482: get_proc_stat

Took it offline and immediately we're back.

It's just amazing that one bad node in 142 can kill off a whole site for SAM... it took out 3626 jobs in less than 12 hours.

This is really torque's fault - it should have a bad node sensor at the batch system level.

(As an aside it didn't affect ATLAS production at all, because if a node is so bad that the pilot doesn't start then it never pulls in a real job.)

NGS - Your software is here too!

A long overdue action on me was to assist David to get the NGS Software published correctly. The WLCG software already has the architecture for .list files to be created by the SGM users, but the NGS relies on parsing the contents of /usr/ngs

I'll admit to being totally confused by the interactions of the various BDII components - they are horribly complex and interwoven collection of scripts / providers / plugins / programs. I understand that counselling is available for those who spend too long working with them.

anyway - attempts to get a new plugin to simply provide the NGS software failed horribly and I ended up patching /opt/lcg/libexec/lcg-info-dynamic-software

--- lcg-info-dynamic-software.orig 2007-11-22 14:25:02.000000000 +0000
+++ lcg-info-dynamic-software 2008-10-10 22:02:15.000000000 +0100
@@ -1,8 +1,10 @@
 #!/usr/bin/perl -w
 
 use strict;
+use IO::Dir;
 
 my $path="/opt/edg/var/info";
+my $ngspath="/usr/ngs";
 my @output;    # ldif output that is sent to std out.
 my @dirs;      # The contents of the path  
 my @ldif_file; # Content of the static ldif file
@@ -23,7 +25,7 @@
     exit 1
 }
 
-#Finds the installed software
+#Finds the installed software (glite)
 @dirs=`ls $path`;
 foreach(@dirs){
     chomp;
@@ -40,6 +42,13 @@
     }
 }
 
+# Do the same for the NGS software
+my @tags = sort grep { /^[A-Z0-9]+_?/ } ( IO::Dir->new($ngspath)->read );
+for my $t (@tags) {
+    push @exp_soft, "GlueHostApplicationSoftwareRunTimeEnvironment: NGS-$t\n";
+    push @exp_soft, "GlueHostApplicationSoftwareRunTimeEnvironment: $t\n";
+}
+
 #Produces the output from the static ldif file and the install software.
 for (@ldif_file){
     if(/dn:\s+GlueSubClusterUniqueID=/){

logs logs logs

Those of you who don't pour over the latest bug reports constantly may have missed that RedHat have fixed 208538 (see http://rhn.redhat.com/errata/RHBA-2008-0703.html)

"logrotate in Red Hat Enterprise Linux 4 did not support the maxage and dateext configuration parameters. Usage of these parameters has been backported and is now available to users of Red Hat Enterprise Linux 4."

basically logrotate-3.7.1-10 works as you'd expect from most other common non-stoneage linuxes and allows logs to be saved in .YYYYMMDD extensions, thus preventing huge renaming sessions nightly and forcing hard-link based backup systems (dirvish) to back up the whole log directory each night.

First among equals...

We got delivery of a sample WN from Viglen a week or so ago. Andrew and Mike did the cluster magic to integrate it into the system and I decided last night to open it up to some real jobs.

Results: 8/8 successful! (Snapshot from ATLAS panda.)

Unfortunately then ATLAS production dried up in the UK, but when the jobs come back, we're ready!

opendns to the rescue

Glasgow, Edinburgh and Durham suffered SAM failures today due to the scotgrid BDII going AWOL. Actually the BDII itself was OK, the problem was caused by the campus DNS servers taking ages to respond and the LDAP query timing out before they responded.

Cue one quick switchover to OpenDNS servers instead.

Worth scribbling on a sticky note - the 2 nameserver IPs are 208.67.222.222 and
208.67.220.220

Update to the above:
OpenDNS don't return NXDOMAIN for non-existent domains, such as .beowulf.cluster -- This can break your installer horribly (as we discovered at glasgow) if you're expecting things to check which is the right address)

However as we're using dnsmasq you can get round this by flagging the 'helpful' opendns guide addresses as bogus:

ie setup your /etc/dnsmasq.conf

no-resolv
server=208.67.222.222
server=208.67.220.220
bogus-nxdomain=208.69.34.132

This then gives the expected results:

svr031:~# dig www.flarble.co.uk

; <<>> DiG 9.2.4 <<>> www.flarble.co.uk
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 10483
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.flarble.co.uk.             IN      A

;; Query time: 105 msec
;; SERVER: 10.141.255.254#53(10.141.255.254)
;; WHEN: Fri Oct 31 09:52:00 2008
;; MSG SIZE  rcvd: 35


compared to...
svr031:~# dig www.flarble.co.uk @208.67.222.222

; <<>> DiG 9.2.4 <<>> www.flarble.co.uk @208.67.222.222
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24219
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.flarble.co.uk.             IN      A

;; ANSWER SECTION:
www.flarble.co.uk.      0       IN      A       208.69.34.132

;; Query time: 11 msec
;; SERVER: 208.67.222.222#53(208.67.222.222)
;; WHEN: Fri Oct 31 09:52:13 2008
;; MSG SIZE  rcvd: 51

mmmm. Shiny!

Yesterday we took delivery of one of the new workernodes that we're purchasing for the 'Phase 2' expansion of Glasgow.

Basically - Supermicro 6015TW-T servers with dual motherboards, each with 2* quad core 2.5GHz intel CPUs. Oh, and 2G RAM per core and a 500G HDD per motherboard.

Integration into the YPF installer was suprisingly painless - I generated a hundred or so new SSH keys and configs for the new boxes (still to do the cfengine ones - they're slightly fiddlier). updated the database of MAC addresses, wrote out the dnsmasq config and restarted the dnsmasq daemon. did a 'setboot' and lo, up n running.

Annoyingly I had to make some minor BIOS changes to these - we always want the nodes to power up in the event of a 'power failure' (ie, we shot them with the APC masterswitch) and there's no point them asking for a PXE boot off the second NIC (its not connected)

oh and the last bunch of workers had IDE disks not SATA (change kickstart to /dev/sda not /dev/hda for target).

So - status is the 2 machines are up n configured, now to get them into Torque (which is still playing sillybuggers wrt the gLite version - they package a pre-release 2.3.0 and it doesn't have the libtorque.0 for monami. Oh and diagnose -f truncates at 65k characters....

take that cfengine

We've had a long running problem with cfengine at glasgow - 2.2.3 (the latest DAG) didn't expand out HostRange properly on the non-workernodes (ie where we need it most - disksvr, gridsvr, natbox groups). today I spent far too long battling with both 2.2.8 and the latest svn release (don't go there - its far too fussy about the exact release of aclocal you use) and neither of them worked properly.

I finally got a minature testcase configuration file to work, then got *really* confused when I used our live config as a testcase file sucessfully, but not the normal incantation.

it turned out to be the fact we'd defined

domain = ( beowulf.cluster )
in update.conf

however, setting this broke the way cfengine handles FQDNs on the dual-homed nodes (which are gla.scotgrid.ac.uk and beowulf.cluster). Commented it out leaving cfengine to guess the right thing to do, and it all seems OK.

I have since upgraded uniformly to 2.2.3 across all the SL4 x86_64 machines and tested OK.

While doing this I noticed we hadn't defined the WMS as a mysqld node so we weren't monitoring it in nagios or backing up the database. Oops. Sorted.

nanocmos + lcas = FAIL

While working on an unrelated issue on svr021 I noticed an edg-mkgridmap error in the logfile

Aug 29 05:28:14 svr021 edg-mkgridmap[6693]: voms search(https://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk/services/VOMSCompatibility?method=getGridmapUsers): Internal Server Error

Mentioned to mike who promptly went and fixed the issue, only to discover 30 mins later we're failing SAM tests - LCAS voms plugin had once again gone fubar and caused globus-gatekeeper to segfault

Aug 29 12:14:41 svr021 GRAM gatekeeper[662]: Authenticated globus user: [DN REMOVED]
Aug 29 12:14:41 svr021 GRAM gatekeeper[663]: Authenticated globus user: [DN REMOVED] Aug 29 12:14:41 svr021 kernel: globus-gatekeep[662]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4
Aug 29 12:14:41 svr021 kernel: globus-gatekeep[663]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4


the globus gatekeeper log has a bit more info:
TIME: Fri Aug 29 12:14:41 2008
PID: 663 -- Notice: 5: Authenticated globus user: [DN REMOVED]
lcas client name: [DN REMOVED]
LCAS 0:
LCAS 1: Initialization LCAS version 1.3.7
allowing empty credentials
LCAS 2: LCAS authorization request
LCAS 0: lcas_userban.mod-plugin_confirm_authorization(): checking banned users in /opt/glite/etc/lcas/ban_users.db
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
JMA 2008/08/29 12:14:45 GATEKEEPER_JM_ID 2008-08-29.11:14:39.0000014519.0000000000 JM exiting

As before, commenting out the lcas_voms.mod in /opt/glite/etc/lcas/lcas.db allows it to work, at the expense of losing VOMS roles.

We've got it working using the voms_mod at the moment by altering the ACLs on the VOMS server (svr029) for nanocmos. Now to try and debug the lcas plugin failure

CE-sft-lcg-rm-rep fail

Ho Hum - After sorting out the gatekeeper, we still get a sam fail. Wait a minute...

Checking replication to Central SE (lxdpm101.cern.ch)

Replicate the file from the default SE to lxdpm101.cern.ch

+ lcg-rep -v --vo ops -d lxdpm101.cern.ch lfn:sft-lcg-rm-cr-node114.beowulf.cluster.080827052321.475531
Using grid catalog type: lfc
Using grid catalog : prod-lfc-shared-central.cern.ch
Source SE type: SRMv1
Destination SE type: SRMv1
httpg://lxdpm101.cern.ch:8443/srm/managerv1: No space left on device
lcg_rep: No space left on device
+ result=1
+ set +x

No space on device? grr - don't these people have monitoring ;-)

gatekeeper AWOL

Glasgow suffered 3-4 hours CE outage this evening as the globus-gatekeeper on svr021 had gone AWOL. we suffered a few SAM tests before I twigged that the 'connection refused' was coming from our end - 'service globus-gatekeeper restart' nobbled that but not until we'd failed 7 sam tests. Damn.

Glasgow on the move

Because of the current problems at RAL, Glasgow was nominated as a test peripatetic Tier-2, so see how agile ATLAS production was at moving Tier-2 resources in case of T1 downtime (note this test only works if the cloud services, FTS and LFC, are still running - if these are gone then it's almost impossible, today, to use any of the cloud's Tier-2s).

First off Glasgow was sent South East, into the NL cloud. Here we found a problem with the input datasets, because input datasets to T2s (which are subscribed without sources) only look for sources within the cloud (this follows the ATLAS computing model). However, the way around this is to specify the associated T1 (for production) as the source and then DQ2 does the work. The panda developers made the change on Friday, so that NIKHEF was specified as the source for inputs to Glasgow. Likewise, for output back to the NL T1, Glasgow's PRODDISK token was specified explicity as the source.

That done, Glasgow galloped through a couple of hundred jobs for the NL cloud, before they ran out of jobs:

Flushed with this success we've just shoved Glasgow into the FR cloud for a while, as they still have jobs left to run. Within an hour we're running a couple of 100 jobs.

SAM Failures across scotgrid: Someone else's problem

All 3 scotgrid sites have just failed the atlas SAM SE tests (atlas_cr, atlas_cp, atlas_del) as have quite alot of the rest of the UKI-* sites.

Once again this isn't a Tier-2 issue but an upstream problem with the tests themselves


ATLAS specific test launched from monb003.cern.ch
Checking if a file can be copied and registered to svr018.gla.scotgrid.ac.uk

   ------------------------- NEW ----------------
srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/
+ lcg-cr -v --vo atlas file:/home/samatlas/.same/SE/testFile.txt -l lfn:SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438 -d srm://svr018.gla.scotgrid.ac.uk/dpm/gla.scotgrid.ac.uk/home/atlas/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
Using grid catalog type: lfc
Using grid catalog : lfc0448.gridpp.rl.ac.uk
Using LFN : /grid/atlas/dq2/SAM/SE-lcg-cr-svr018.gla.scotgrid.ac.uk-1219649438
[BDII] sam-bdii.cern.ch:2170: Can't contact LDAP server
lcg_cr: Host is down
+ out_error=1
+ set +x
   -------------------- Other endpoint same host -----------

Supported VO Tweaks

I have enabled the 'gaussian' VO on the ScotGrid UIs at the request of a local user. This took a little longer than usual as the set-up is slightly unusual - we want to have voms and job submission correctly configured, but do not want to support the VO on our WMS (which we consider our fragile service).

YAIM 4 makes enabling a VO on a specific node type easy - we have an override stanza in services/glite-UI which adds gaussian (through 'VOS="$VOS gaussian"').

The trick bit is the cfengine magic to redefine the WMS and the LB services for gaussian only as YAIM sets all of these to be the same:

ui::
{ /opt/glite/etc/gaussian/glite_wms.conf
ReplaceFirst "https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server" With "https://rb1.cyf-kr.edu.pl:7443/glite_wms_wmproxy_server"
}
{ /opt/glite/etc/gaussian/glite_wmsui.conf
ReplaceAll "svr022.gla.scotgrid.ac.uk" With "lb.grid.cyf-kr.edu.pl"
}

Gaussian are a special VO who only seem to exist to allow access to a commercial software package: "Gaussian VO enables use of commercial chemical package Gaussian on EGEE Grid".

Grids and licensed software still have some serious paradigm issues when you have to setup an entire VO to use a piece of software.

Nagios, nagios, where are you?

Nagios seems not to be sending us alarms properly on SAM test failures (we had 2 yesterday - one for an internal SAM problem and the other for an SRMv1 timeout issue).

Andrew is investigating.

DPM Upgrade

Mike and I upgraded the DPM yesterday to 1.6.10. There are no schema changes in 1.6.7->1.6.10, so the upgrade involves 'downtime' of about 20s. We didn't put this into the CIC portal, but I suppose in retrospect we should have declared an 'at risk' period.

We're still running the i386 version of DPM (on top of x86_64). At some point it would be desirable to upgrade to x86_64; however, as i386 works just fine and this will involve real downtime, there is no urgent pressure to do so.

Greig has noted that dpm-updatespace in 1.6.10 has a bug in it: https://gus.fzk.de/pages/ticket_details.php?ticket=38330.

Alert! Alert!

'twas the night before holidays, when all through the servers not a pager was stirring...

Hmm lulled into a false sense of security by the appearance of [WLCG Nagios] entitled emails alerting me about proxy expiry on the shared nagios system I foolishly thought all was well. However we wern't getting any 'real' alerts from the system to the individual sites.

Turned out to be a configuration issue in the /etc/nagios/uki-scotgrid-*/contacts.cfg

We had

    service_notification_options    n
    host_notification_options       n

meaning no notifications were sent - changed this to

    service_notification_options    w,u,c,r,f
    host_notification_options       d,u,r,f,s

which means we get alerted on pretty much every state change - for more details see the manual

on a more annoying note - I left my macbook PSU back in the UK and there's a limited no of apple resellers here :-(

Space tokens aplenty!

Glasgow have deployed the ATLAS PROD, USER and GROUP disk space tokens, in line with the requirements of this GGUS ticket.

As we're running DPM, the procedure was fairly trivial, and is documented on the ScotGrid wiki.

We've also fixed and re-enabled the DPM information provider script, which Graeme reported broken in this blog posting.

An ldapsearch query shows that we're now advertising the new tokens correctly:


# atlas:ATLASPRODDISK:online, svr018.gla.scotgrid.ac.uk, resource, grid
dn: GlueSALocalID=atlas:ATLASPRODDISK:online,GlueSEUniqueID=svr018.gla.scotgri
d.ac.uk,Mds-Vo-name=resource,o=grid
objectClass: GlueSATop
objectClass: GlueSA
objectClass: GlueSAPolicy
objectClass: GlueSAState
objectClass: GlueSAAccessControlBase
objectClass: GlueKey
objectClass: GlueSchemaVersion
GlueSARoot: atlas:/dpm/gla.scotgrid.ac.uk/home/atlas
GlueSAPath: /dpm/gla.scotgrid.ac.uk/home/atlas
GlueSAType: permanent
GlueSALocalID: atlas:ATLASPRODDISK:online
GlueSAName: Replica online storage for VO atlas
.
.
.


All in all, a productive few minutes' work...I wonder how those with dCache are coping...

Attack of the Clones

So, I finally got round to booting the other three new servers we'd recently purchased (these are nice simple dual core, dual CPU boxes (Dual-Core AMD Opteron(tm) Processor 2216) on Tyan Thunder n3600M (S2932) Motherboards)

I'd always been exceptionally suspicious of the PXE MAC address thet the 1st one offered (60:50:40:30:20:10) rather than the imprinted one on the rj45 socket starting with 00:E0:81 but the fact that three out of four machines all had the same mac address meant a phone call to the vendor was called for.

The one that works has the 2.01 BIOS, the others were already shipped with 2.02 and downgrading didn't cure the amnesia. I don't see a utilty to reburn the mac address ont the machine, so have left that in the capable hands of the vendor for now. Googling seems to suggest that nvidia cards are somewhat prone to wierdisms like this. Bother.

Powercut at Durham

Ok, first blog post here...

On sunday, our machine room's UPS caused a brief power failure, which unfortunately tripped some breakers so we had to call out the electricians before we could start restoring service.

The UPS will take some time to repair, so Durham will be at risk for a while.

On the plus side, the changes involved with the SE rebuild have been proven to survive a reboot!

EDIT:
Just realised that I forgot to say that the site is back online, and has been since sunday evening, it is just currently at mercy of the power company.

ECDF - More VOs

Sam reports that:

"biomed, pheno, vo.scotgrid.ac.uk and vo.gridpp.ac.uk VOs have been enabled on the SL3 CE (ce.glite), the SRM (for storage), and the monitoring/accounting box (mon.glite)."

This seems to have produced a glitch on account records publication on the MON box, but a fix is in the pipeline.

ECDF SAM Tests Fixed

Sam managed to fix SAM last week. It turned out that the two CEs were interfering with each other in the samsgm account area and producing corrupted GASS caches. It's still a bit of a mystery as to why this happened - shared accounts for *sgm has been normal (if not best) practice for ages. However, it's being worked around and the site is up and stable again.

Durham SE Issues

Durham suffered a complete SE failure last week. A RAID card failure took down the old SE, gallows, and then an LVM metadata corruption took out the new disk server on se01.

The list of lost ATLAS files has been reported (https://savannah.cern.ch/bugs/?38037) and we're waiting for the catalog to be cleaned up to restart production here (well, when there are any jobs to run).

We took the opportunity to retire gallows and now se01 is the sole SE at Durham. It should suffice for ATLAS production where we only need a few TB cache anyway.

In the meantime there was a power outage in the Durham machine room over the weekend. David had to get the university to reset some breakers but things seem to be running well now.

Publish and be damned...

It all went pear shaped yesterday as information publishing fell over on the SE. It seems when I quickly "fixed" the DPM information provider script to get the correct hostname I forgot to chomp() the output from "hostname -f". So the hostname variable had a trailing newline which corrupted the information system. The BDII logs started to throw errors like "First line of LDIF entry does not begin with 'dn:' at /opt/glite/libexec/glite-info-generic line 17".

Unfortunately the BDII then considered the whole of the SE information package corrupt (rather than just that provider's output) and our SE promptly dissappeared from the information system with the attendent RM test failures.

This situation then persisted for most of the day until Andrew noticed it "by eye". So we had another failure - nagios didn't send an alarm properly when we started to fail. If that had happened it would have been fixed in a hour, but instead we were failing for 8 hours.

From the dizzy heights of SAM perfection we fell to 98% for the month, 95% for the week. It wasn't quite hubris, but it was ironic that I was blogging about Glasgow's reliability at the very moment we were broken.

At the moment I have removed the info provider for tokens, and I will more carefully put it back on Tuesday.

Problems up north

We have two major problems in ScotGrid right now:

ECDF: Have been failing SAM tests for over a week now. The symptom is that the SAM test is submitted successfully, runs correctly on the worker node, but then job outputs never seem to get back to the WMS, so eventually the job is timed out as a JS failure. As usual we cannot reproduce the problem with dteam or ATLAS jobs (in fact ATLAS condor jobs are running fine) so we are hugely puzzled. Launching a maual SAM test throught the CIC portal doesn't help because the test gets into the same state and hangs for 6 hours - so you cannot submit another one. Sam has asked for more network ports to be opened to have a larger globus port range, but the network people in Edinburgh seem to be really slow in doing this (and it seems it is not the root cause anyway).

Durham: Have suffered a serious pair of problems on their two SE hosts. The RAID filasystem on the headnode (gallows) was lost last week and all the data is gone. Then this week the large se01 disk server suffered an LVM problem and we can no longer mount grid home areas or access data on the SRM. Unfortunately Phil is on holiday, David is now off sick and I will be away on Monday - hopefully we can cobble something together to get the site running on Tuesday.

Thankfully, dear old Glasgow T2 is running like a charm right now (minor info publishing and WMS problems aside). In fact our SAM status for the last month is 100%, head to head with the T1! Fingers crossed we keep it up.

Much improved CE

Our bugbear in the past was always the lcg-CE, which was a service it was easy to overload and cause the site to hit meltdown (we have a lot of examples collected in http://scotgrid.blogspot.com/search/label/CE.).

A few months ago a new daemon, the globus cache marshal, was introduced which promised to substantially reduce the load on the old CE.

Recently we have had a few job spikes from local atlas and pheno users and I'm very happy to say that the CE seems much healtier than in the past. Having more than 1.5k jobs running and queued the load on the CE was modest and the CPU usage was < 20%.

This is a huge improvement over past performance and has removed a major source of site instability.

Who's crying now?

Our poor WMS was killed from the end of last week when a pheno user submitted about 20k jobs into it. Worse, they hadn't used a proxy renewal service, so their VOMS extension on the proxy expired and so the jobs they had submitted suffered shallow failures, promting further resubmission attempts and further load.

We have contacted the user, but they were having a great deal of trouble even cancelling the jobs. We've left things for 2 days now, but the situation is really not improving. I can't see much hope for the machine in its current state - we'll probably have to blow it away and start again next week.

For the moment we've asked our users to revert to using ye olde RB, which we were just about to switch off but hadn't actually decomissioned.

There's a serious question now about what level of WMS service we want to provide. It's a complex service and rather difficult to debug when it goes wrong. Certainly an upgrade to the SL4 version should be done, but do we want to have 2 WMS hosts and possibly even a separate LB service?

disinformation

We just got ticketed for a failing SE sam test. Most odd as Steve Lloyds SAM results were all green. Re-read the ticket and it turns out we were publishing info for svr018.beowulf.cluster rather than the external interface name. Despite this being noted before Graeme hacked it around and raised a Savannah Ticket

And don't do it again...

I re-enabled Heinz today on the cluster, our infamous RSA cracking biomed user.

Tony finally spoke to Cal and this seemed clarify that

• VOs now know they have a more serious responsibility to discipline their users
• Heniz knows he cannot run this work again under biomed
  

He'd been suspended for six months which seems like an appropriate punishment.

So, how was it for you...?

I made some comments on Glasgow's site availability from October last year until now. It was Q1 this year that was very hard on us (strictly last week of Feb until end of March) - we lost 5 days to the CE crash and reinstall of core nodes, then were pestered by minor networking and user level problems. All of these conspired to reduce our availability to 83% that quarter.

However, we seem to be very much on top of things now, with 95% so far for Q2 and 100% in the last week (we were lucky that our servers had certificates from the old CA though).

I suppose this also accounts for the lack of blog entries this month - things are going well.

Full steam ahead, Captain. The weather is fine...

liblcas_lcmaps fix

There is a fix for the segfaulting bug I reported against the globus gatekeeper and gridftp server, https://gus.fzk.de/pages/ticket_details.php?ticket=35694.

If you download the patched version of liblcas_lcmaps_gt4_mapping.so.0.0.0 and install it to /opt/glite/lib then the problem is resolved.

I applied the patch last night and since then we haven't suffered a single segfault.

Maarten said that the official fix should reach production in the next couple of weeks, but I'm happier having it on site now.

CRHell...

So, even after fixing our CE disk space problems we still were failing SAM tests. The error was one I'd never seen before: "7 an authentication operation failed".

The GOC wiki hinted at a few things which can cause this, but as it only seemed to affect SAM tests (Steve Lloyd jobs and my ATLAS production were both running fine) there wasn't a lot that could really be debugged locally - we had even had a successful SAM test from Rafa's SAM Admin interface at 11am.

One of the classic X509 errors though, is CRLs being out of date. When I checked the CRL files on the CE is was clear something was amiss. A few were dated today, but most were approaching 5 days old. When I ran the CRL updater by hand I got the error:

fetch-crl[19144]: 20080507T154251+0100 updating CRL 'CERN Trusted Certification Authority (1d879c6c)'
fetch-crl[19144]: 20080507T154251+0100 File /etc/grid-security/certificates//1d879c6c.r0 valid: no
fetch-crl[19144]: 20080507T154251+0100 Attempt to overwrite /etc/grid-security/certificates//1d879c6c.r0 failed since the original is not a valid CRL file

What's worse is that not only did this CRL fail to update, it caused the updater script to bomb and no CRL after this point even attempted to update!

In the end I had to delete all of the CRLs on the CE and re-run the script to get fresh copies.

What exactly happened I do not know, but the relevant command in fetch-crl is:

openssl crl -hash -in CERT_REVOKE_FILE -noout -inform PEM -text

This should produce, as a first line, the hash for the CRL, which is the same as the file name. however, testing one of the bad files I got instead:

unable to load CRL
12697:error:0906D066:PEM routines:PEM_read_bio:bad end line:pem_lib.c:731:

So clearly they had gone evil in some way.

I think this is poor behaviour from the script, so I have submitted a GGUS ticket (https://gus.fzk.de/pages/ticket_details.php?ticket=36191).

X509 error messages just suck so badly.

It's all gone 'orribly wrong

OK - My bad. I spotted we failed a sam test yeterday (got a mail from the automated alert) - didn't realise it doesn't send multiple ones if you keep failing....

Am sure Graeme will post more but we'd filled up / (as /var wasn't on a separate partition - it is now, and a nice healthy 30G) on the CE. Puzzlingly nagios hadn't bothered to alert that we'd gone warning at 8% free or critical at 4% free and was "OK - /0 free"

case sensitivity in check_disk: -w is for disk space, -W is for inodes. grr. Typo-tastic. I lowercased the offending config and let cfengine ripple it out. While it did so I noticed cfengine restarted ntpd on the 3 nat boxes (that also act as the timeservers for the cluster) - somehow it was copying both a standard then a local /etc/ntp.conf into place each time and restarting as planned on a new config file.

my bad - we use class 'natboxes' for the group and I'd specified any.!(master|nat):: changing it to any.!(master|natboxes) worked fine - no restart since and none of the workernodes are seeing any upstream timeservers on INIT or LOCAL

Holy lactating keyboards Batman, there goes the disk!

There's a pre-amble to this post which isn't really to do with ScotGrid directly, but has some bearing subsequent event. If I tell you the actors were a glass of milk, a three year old and my laptop keyboard then you can doubtless assemble the plot yourself. So yesterday some important keys were completely non-functional on my laptop - one of these keys was a character in my password so I wasn't able to login at all. Despite my best MacBook disassembling and cleaning efforts I couldn't recover the keyboard - there seems to be gunk etched onto the conducting keyboard sheet. Now that I am back in the office I can use a USB keyboard, but I need to get a new MacBook one quickly.

Anyway, upshot is that my online activities were squeezed into an ancient IBM R32 laptop running XP with 256MB of RAM. So, I was not watching (bookmarks were effectively unavaliable) when the CE ran out of disk space and we went belly up for SAM tests yesterday lunchtime.

Problem was easily identifed this morning and, thanks to the magic of LVM, corrected. When the machines were setup /var was not separate from /, so we've now added a separate /var logical volume with 30GB in it.

We looked into why this did not trigger and alarm and it seems that there was a mistake in the nagios configuration which was looking at free inodes, rather than free space. Corrected now.

I also took the opportinutity to re-run YAIM on the CE and fix the broken VOMS mappings. My atlas production role now maps correctly to an prdatlas account. This might cause a failure to clear from globus the stack of jobs running under my "local" gla012 account, but the actual production system doesn't require any outputs from globus so the real payload will run fine.

Sadly, the segfaulting on the CE still happens, however it seems the the problem was spotted earlier and that a fix is on the way: https://savannah.cern.ch/bugs/?35981.

Farewell ScotGrid-Edinburgh!

Not withstanding the problems this week at ECDF, in general the new Edinburgh resource has been working well. Consequently maintaining the old site was just a drag on our people, with no real purpose anymore. Sam broadcast the intention to close the site a couple of weeks ago and we changed its status in the GOC to "closed".

Greig and Sam have ensured that the site's old storage is accessible through ECDF, though we've advised VOs to move their data off this SE as the hardware is ageing and unreliable.

Edinburgh is dead! Long live Edinburgh!

ECDF down for the moment

ECDF have been having real trouble with GPFS in the last week, which gave us some miserable results (23% pass rate on SAM, c.f., UK average of 75%). For the moment the systems team have suspended job submission and the site went into downtime on Friday.

This may or may not be related to the problems we see with the globus job wrapper code on ECDF, where the GPFS daemon consumes up to 300% CPU due to a strange file access pattern in the job home directory. Sam is working on installing an SL4 CE (based on the GT4 code) to see if this improves matters.

Who's watching the watcher?

We had a problem on the DPM headnode with the new VOMS certificate for Zeus not being installed. When I checked I found it was in the repository, but had not been copied to the server. What gives? It turned out that the cfengine version we had on svr018 (2.2.3-1.el4) was not defining the gridsrv class properly (through the HostRange expansion), so then the grid class was not defined and consequently the vomsdir was not being checked.

I downgraded to 2.1.22 and this fixed the problem. But there is a mystery here - why does 2.2.3-1 work fine on the worker nodes?

I see 2.2.6 has been released. Maybe we should go back to rolling out own build of cfengine?

#include <documentation.h>

I quote from the DPM Developer Documentation

LFC/DPM Database schema
TO DO : describe non straight forward tables/fields....

so, with that in mind, I set about pulling out the number of SRM 2.2 requests vs the no of SRMv1 requests at the site. v1 should be constant (what with all the new users coming onboard) and SRM 2.2 being a rapid increase since we enabled it? well it's not easy to grep from the logs so I thought I'd poke the DB. First off in dpm_db.dpm_req r_type a char(1) field normally has g (get?) and p (put?) but we have just over 1500 rows where type is 'B' (broken?). hmm - all from flavia's DN and clienthost of lxdev25

my plots of the dpm usage are far too spikey to make sense of at the moment, but I'll work on presenting the info a bit clearer.
In the meantime I discovered that it's pretty obvious when we set torque to fill the jobslots in host order (made it easier to drain nodes off) and when we send nodes away to vendors.