Fun with 5!

So, the SL5 migration was done last week, and as promised at yesterday's dteam meeting I am posting our problems so that other sites can watch out for similar issues (although some of these are deeply related to the way we do things at Glasgow).

First though, the successes:

1. 1800 cores running SL5
2. DPM headnode upgraded to SL5
3. Torque server upgraded to SL5, running torque 2.3.6, maui 3.2.6p21
4. /atlas/uk voms group supported with a separate fairshare
   

Now, the list of problems:

1. We introduced a new python script, gridAccounts.py, to generate pool accounts, retiring the venerable, but incomprehensible, genaccts.pl script we had before (Andy Elwell wrote that and his comment was "OK - I give up with python as I need this NOW..."; my retort was "I HATE PERL SO MUCH. IT'S A SHIT LANGUAGE.", but I had never found the time to rewite it until now). The new script reads standard config files, so it's a lot easier to manage, understand and extend. However, all change is (a bit) dangerous and the new script initially had groups in the wrong order in yaim's user.conf, which caused the groupmapfile to be wrong. This then caused all jobs to fail - the uid/gid of the gridftp session did not match the uid/primary gid of the user and gridftp does not like that at all.

(The reason we have to write users.conf is because we still get yaim to do a lot, although we manage all accounts through cfengine; yaim relies on this file to configure various other aspects of the system, such as grid/group mapfiles.)

2. We were trying to support the /atlas/uk VOMS group as a separate entity. This is simple in theory (!), you're looking for the following entries in voms-grid-mapfile:

"/atlas/uk/Role=NULL/Capability=NULL" .ukatlas
"/atlas/uk" .ukatlas

and this in groupmapfile:

"/atlas/uk/Role=NULL/Capability=NULL" atlasuk
"/atlas/uk" atlasuk

If we were managing these files directly, it would have been no problem. However, convincing YAIM to do this was far from easy. This is not helped by the fact that YAIM is now utterly incomprehensible in many ways (have a look at yaim/utils/users_getvogroup if you don't believe me). Finally we hit on the correct recipe, which is to have these accounts in users.conf, with a new "special" defined:

201601:ukatlas001:201040,201000:atlasuk,atlas:atlas:uk:
201602:ukatlas002:201040,201000:atlasuk,atlas:atlas:uk:
201603:ukatlas003:201040,201000:atlasuk,atlas:atlas:uk:
...

with this line added to groups.conf:

"/VO=atlas/GROUP=/atlas/uk":::uk:

Aside: Sometimes I wonder if YAIM has outgrown its usefulness. From something we could understand and tweak easily it's now a sed|awk|cut|sort|tail black box monster, which uses a computerised format for configuration files. c.f. the configuration we have for our own scripts:

[someuser]
dn = /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=some user
uid = 4832
home = /clusterhome/home/someuser
group = atlas
tier25 = True
vo = gla

And trying to do grid configuration manipulations in a language which doesn't have dictionaries is just ridiculous.

Maybe we'll need to wean ourselves off it eventually?

3. Information publishing on svr018 was broken after the upgrade. There was a cryptic reference to a required 'dpminfo' user which Sam had made in his notes. Adding this user did seem to make things work, though it's not at all clear why. Hopefully Sam will enlighten us later. In passing, note that the resource BDII on the service nodes seems to be 'protected' now, so attempts to reach it from 'outside' fail. This is new behaviour and lost us some time in debugging.

4. Terrible trouble was caused by upgrading the torque server to SL5. Using the SteveT build of torque server (http://skoji.cern.ch/sa1/centos5-torque/) seemed to cause grave problems with moms crashing on the worker nodes. Downgrading the moms to the torque 2.3.0 didn't work as the jobs files (/var/spool/pbs/mon_priv/jobs) seemed to be in incompatible formats and led to crashing moms plus a very confused torque server. Cleaning out all jobs seemed to not work either. The final solution was to rebuild torque 2.3.6 on SL5 - this gave a consistent and compatible server/mon pairing.

A small side effect though, was the the rebuilt maui had a different 'secret' in it, so I have had to hack the info provider on the SL4 CEs to use the --keyfile= argument in the maui client commands. (That's such a stupid 'feature'.)

5. Once we were out of downtime, random transfers to the DPM were failing. Eventually we tracked to the reduction in the number of pool accounts for atlasprd. There was no sync between the passwd fila and to the /etc/grid-security/gridmapdir pool account list, of course, so gridftp was was throwing a "530 Login incorrect. : No local mapping". We realised that

1. /etc/passwd should be handled better on nodes which need to map pool accounts.
2. For the moment never reduce the number of accounts!
3. N.B. on the CEs the gridmapdir is shared, so maintenance probably needs to be delegated
4. If we remove a pool account mapping then you have to remove the link from any DNs to this mapping as well (look for DN filenames with only 1 hard link).
   

OK, that's it. We got there, though not without some anxious moments!

Installing (and fixing) a gLite Tar UI on SL5

First, a little background.

The UI machine is the gLite term for the machine from which you submit jobs (and monitor, receive output etc). This is analogous to the submit machine in Condor, and the head node for a local cluster - except that with the Grid, there is no reason that you can't submit on one UI, monitor from another and collect output on a third. No reason - except perhaps for keeping one's sanity.

Whilst most of the Grid servers are normally dedicated machines, occasionally given over to more than one Grid task, but only doing Grid tasks, the UI is a clear contender for being placed on machine that already have another purpose. In this instance, we have a group of users that have their own cluster, and occasionally off load some computations onto the Grid. It would be ideal if they could submit to either their local cluster or the Grid from the same machine. Cluster head nodes aren't too portable, so the obvious approach is to turn their existing head node into a gLite UI.

Fortunately, the gLite developers forsaw this possibility, and the UI package is available in a single blob that can be installed for an individual user. So that's what I've done - but there's a few caveats, and a couple of bugs to work around.

The tar UI I used was the gLite 3.2.1 production release. This is still early in the 3.2 life cycle, and not all services are available at 3.2, so there might be a few teething issues here, interacting with the older services. At Glasgow we don't have any 3.0 services, which is good, as they're really unsupported.

On to the install: Download the two tarballs, and unpack into a directory (why 2 tarballs, one tarball aught to be enough for anyone). I then promptly fell of the end of the documentation - which assumes that you already know a lot about gLite.

What you have to do it produce a file (the site-info.def) that gives some high level details of what the UI needs to know to work. This file can be created anywhere (I put it in the same directory I unpacked the tarballs into), as you always gives it's path to yaim, the tool that uses it.

The first thing you need to put in is the 4 paths listed on the wiki page. Then you need a few other things:

BDII_HOST=svr019.gla.scotgrid.ac.uk
MON_HOST=svr019.gla.scotgrid.ac.uk
PX_HOST=lcgrbp01.gridpp.rl.ac.uk
WMS_HOST="svr022.gla.scotgrid.ac.uk svr023.gla.scotgrid.ac.uk"
RB_HOST=$WMS_HOST

The BDII host is where the UI gets it's information from - this should be a 'top level' BDII, not a site BDII. None of have the faintest clue why it needs the MON host - that's something I'll dig into later. The PX host is the MyProxy server to use by default. That one should be good for anywhere in the UK. The WMS host is the replacement for the deprecated (but still needed) RB hosts, and points to the WMS to be used for submission (by default).

One thing I found I needed that wasn't documented was a SITE_NAME. I just put the hostname in there - it doesn't appear to be used, but yaim complains if it's not there.

The last thing needed is a list of the VO's to be supported on the UI. When deploying a tar UI this will normally be a very small list - one or two I would expect. Therefore I choose to place them inline. There is a mechanism to put the VO specification in a separate directory, which is used for shared UI machines.

VOS="vo.scotgrid.ac.uk"

VO_VO_SCOTGRID_AC_UK_VOMS_SERVERS="vomss://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk"
VO_VO_SCOTGRID_AC_UK_VOMSES="'vo.scotgrid.ac.uk svr029.gla.scotgrid.ac.uk 15000 /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk vo.scotgrid.ac.uk'"
VO_VO_SCOTGRID_AC_UK_VOMS_CA_DN="'/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA'"

VO specification is in two parts - first we have to list the VO's (space separated list), and then , for each VO, give the VOMS server that defines the membership of the VO, and the certificate DN for the VOMS server. Note that the vo name gets translated to UPPER CASE and all the dots in it become underscores (a fact that's somewhat underdocumented, and results in a complaint about a syntactically invalid site-info.def, and no other message ...)

Once that's all in place, it's time to run yaim to configure things (from the dir I unpacked into):

./glite/yaim/bin/yaim -c -s site-info.def -n UI_TAR

Slight problem with installing certificates: By default these go into /etc/grid-security/certificates, but I'm not running as root. As a local user (for the initial testing), I need to tell yaim where to put them instead. In the site-info.def:

X509_CERT_DIR=${INSTALL_ROOT}/certificates

and make that directory, and re-run the yaim command. Chuntering along for a bit, and then finished with no errors - I did get a couple of warnings, but nothing that looked like a problem in this case.

Last step - testing. First, load up the installed software:

$GLITE_EXTERNAL_ROOT/etc/profile.d/grid-env.sh

and install my certificate on there.

lcg-infosites ... works
voms-proxy-* ... works
glite-wms-job-submit ... Boom!

glite-wms-job-submit: error while loading shared libraries: libboost_filesystem.so.2: wrong ELF class: ELFCLASS32

Hrm. Looks like a 32/64 bit problem. Some pokage later, and it turns out that the shell setup script supplied points only to the $GLITE_EXTERNAL_ROOT/usr/lib directory - and not the lib64, containing the needed libs. A quick hack onto the grid-env.sh, and that's rectified. Now:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl
glite-wms-job-submit: error while loading shared libraries: libicui18n.so.36: cannot open shared object file: No such file or directory

This turns out to be the International Components for Unicode (at least, I think so). The particularly interesting point about this is that the only references I can find to these libraries on SL include one from this very blog and they are all about Adobe Acrobat Reader... because that's the most common software that uses it.

I grabbed the RPM from http://linux1.fnal.gov/linux/scientific/5x/x86_64/SL/, and added it to $GLITE_EXTERNAL_ROOT/usr/lib64 by:

cd $GLITE_EXTERNAL_ROOT
rpm2cpio libicu-3.6-5.11.2.x86_64.rpm | cpio -i

And, finally:

[scotgrid@golem ~]$ glite-wms-job-submit -a minimaltest.jdl

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

====================== glite-wms-job-submit Success ======================

The job has been successfully submitted to the WMProxy
Your job identifier is:

https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

==========================================================================

Jobs submitted from a pure 64 bit SL5 system. Note that the separator character has changed, from being a line of '*' to a line of '='.

After chuntering away for a while,

[scotgrid@golem ~]$ glite-wms-job-output https://svr023.gla.scotgrid.ac.uk:9000/VW96yirZ4gG6jVXjx9UwBg

Connecting to the service https://svr022.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

Error - Operation Failed
Unable to retrieve the output

[scotgrid@golem ~]$ cd /tmp/jobOutput/
[scotgrid@golem jobOutput]$ ls
scotgrid_VW96yirZ4gG6jVXjx9UwBg

Which is a known problem - the UI reports that collecting job output failed, but it does succeed. If you don't need to get the OutputSandbox from the job (e.g. it's all written to an SE), then this isn't a problem.

Now to post some bug reports on the tar UI package... (Update: This is now bug number 52825 for the configuration and bug 52832 for the ICU package)

activiting a VO on a WMS in four easy steps

We have been asked to add support for the camont VO to our WMS.
Since we only support a limited number of VO through our WMS, here is the recipe for adding a new one in the smallest number of steps.

1. edit the following file vim /opt/glite/yaim/etc/services/glite-wms
2. /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -f config_vomsmap
3. /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -f config_vomses
4. /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -f config_glite_wms

Development / PreProd : The CE

So on with the show and the creation of a development/preprod cluster. Next up the CE on dev machine 10.

Step 1. Again involved a quick search of the cfagent.conf. An additional entry was added in the ce groups stanza to add in the devce. cfagent -qv was used to pull down/install the node.

The first yum update updated all the certificates was successful. However, it looks again like glite-yaim-core was not successful first time around.

/opt/glite/yaim/bin/yaim: No such file or directory

A second run of cfengine caused the error below.

Transaction Check Error: file /usr/share/java/jaf.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp
file /usr/share/java/jaf_api.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp

This is a known issue and the fix is to disable the jpackage17 repo like so:

/usr/bin/yum -y install lcg-CE glite-TORQUE_utils --disablerepo=jpackage17-generic

This allowed a full installation of the relevant packages. This was run by hand in the first instance and then was added to the development configuration file.

Step 2. Running yaim by hand for the CE and then adding additional stanzas into the development configuration file. So installing the CE: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n lcg-CE -n TORQUE_utils

There appeared to be various quirks/warnings/errors with this. Here is a short summary.

There appeared to be an issue with line 36 of the config_apel_pbs.
Atfer inspecting the file it appears that the line separator in the file was not working as expected.

/opt/glite/yaim/functions/config_apel_pbs: line 36: APEL_DB_PASSWORD: command not found

This is a known bug https://savannah.cern.ch/bugs/index.php?39014 and was temporarily fixed by removing the additional whitespace before the /

These errors appeared next but did not seem to cause any problems.

/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32dbg.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32pthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32dbg.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32pthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_gcc32dbgpthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32dbgpthr.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsc_gcc32.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi.so.0 is not a symbolic link
/sbin/ldconfig: /opt/glite/lib/libvomsapi_nog.so.0 is not a symbolic link

This is due to a known bug: https://savannah.cern.ch/bugs/?42481

INFO: Now creating the grid-mapfile - this may take a few minutes...
voms search(https://voms.gridpp.ac.uk:8443/voms/supernemo.vo.eu-egee.org/Role=lcgadmin/services/VOMSCompatibility?method=getGridmapUsers): /voms/supernemo.vo.eu-egee.org/Role=lcgadmin/services/VOMSCompatibility

voms search(https://voms.gridpp.ac.uk:8443/voms/ukqcd.vo.gridpp.ac.uk/Role=lcgadmin/services/VOMSCompatibility?method=getGridmapUsers): /voms/ukqcd.vo.gridpp.ac.uk/Role=lcgadmin/services/VOMSCompatibility

Exit with error(s) (code=2)

WARNING: It looks like /opt/globus/tmp/gram_job_state may not be on a local filesystem. WARNING: The test for local file systems is not 100% reliable. Ignore the below if this is a false positive.
WARNING: The jobmanager requires state dir to be on a local filesystem
WARNING: Rerun the jobmanager setup script with the -state-dir= option.Creating state file directory.
Done.

find-fork-tools: WARNING: "Cannot locate mpiexec"
find-fork-tools: WARNING: "Cannot locate mpirun"

find-lcgpbs-tools: WARNING: "Cannot locate mpirun"
checking for mpirun... no

Any clues to these errors would be greatly appreciated too.

Currently when a new CE is configured yaim attempts to run the function config_gip_vo_tag
This function attempts to create and change permission on the VO tags directory.
However, this directory is mounted on the Scotgrid cluster and comes pre-configured so to speak.

INFO: Executing function: config_gip_vo_tag
chmod: changing permissions of `/opt/edg/var/info/atlas': Operation not permitted
chmod: changing permissions of `/opt/edg/var/info/atlas/atlas.list': Operation not permitted
chmod: changing permissions of `/opt/edg/var/info/cms': Operation not permitted
ERROR: Error during the execution of function: config_gip_vo_tag
ERROR: Error during the configuration.Exiting. [FAILED]
ERROR: One of the functions returned with error without specifying it's nature !
INFO: Using locally defined function /opt/glite/yaim/functions/local/config_gip_vo_tag
cfengine controls config_gip_vo_tag. yaim function disabled.

The fix to this issue was to override the function gip_vo_tag in the yaim/function/local directory to make sure it did not try to change any of the NFS mounted directories.

Again the bdii issue reared it ugly head. https://savannah.cern.ch/bugs/index.php?40675

Starting glite-lb-interlogd ...chown: cannot access `/opt/bdii/var': No such file or directory
sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the /opt/bdii/doc/schemas.example file to /opt/bdii/etc/schemas and re-run yaim.
The re-run of yaim also fixes the chown of /opt/bdii/var.

All of the above workarounds have now been added to the cf.dev script that is included in the main cfagent.conf script. Thus all the temporary workarounds and development stanzas are kept out of the main script.

Step 3 - Testing:

First off I thought I could use the glite set of commands for the development cluster.

jdl extract:

Requirements = other.GlueCEUniqueID == "devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m";

submission extract:

-bash-3.00$ glite-wms-job-list-match -a hello.jdl
Connecting to the service https://devmachine9.gla.scotgrid.ac.uk:7443/glite_wms_wmproxy_server

==================== glite-wms-job-list-match failure ====================
No Computing Element matching your job requirements has been found!
==========================================================================

However, after some initial tests and some thought on the subject it became apparent that it would have to be entered into the site BDII for this to work.
This made sense since the WMS queries the site BDII to get information relating to the published queues from the CE.

Therefore, without setting up another siteBDII/BDII for the mini cluster, direct job submission via GLOBUS seemed like the way to go for intial testing.

-bash-3.00$ globus-job-submit devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs /bin/hostname -f
https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
PENDING
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
PENDING
-bash-3.00$ globus-job-status https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/
DONE

However when I tried to obtain the job output: -bash-3.00$ globus-job-get-output https://devmachine10.gla.scotgrid.ac.uk:35001/25359/1232379693/

--- Nothing, Nada! Doh!

After some investigation involving the logs on the CE and the logs on torque it became apparent that the torque was not allowing the job submission from the new CE. After some more investigation this seemed to be down to a file called hosts.equiv. This is file that holds the white list of hosts that torque will talk to.

Therefore, after added the new CE and restarting Torque:

-bash-3.00$ globus-job-run devmachine10.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs "/bin/hostname -f"
node192.beowulf.cluster

Woohoo, we have a working CE that allows submission through Globus, Now to test the CE with WMS submission we need a dev site bdii. So I think I will install on the dev UI and make sure we can submit through glite-wms-job-submit.

Development / PreProd : The WMS

I thought I would continue my foray into grid middleware installations another quick blog on the workload management system or WMS as its affectionately known. . With the old development cluster very much moved/dead we now have dev008 -> dev013 as a sandbox for installs and upgrades. So after the UI install last year the next piece of the jigsaw was the slightly more heavy weight WMS.

Step 1. Was a quick search of the cfagent.conf. This returned the necessary files, links, packages to install. An additional entry was added in the wms groups stanza and cfengine was ran immediately using cfagent -qv

The first yum update in fact updated all the certificates from the lcg-CA, lcg-vomscerts package stanzas and these were successful. It then looked like it attempted to run the glite-yaim-core which could not have successful as it then through this error:

Executing script /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -n glite-LB...(timeout=0,uid=-1,gid=-1)
(Setting umask to 22)
cfengine:dev009:m/bin/yaim -c -: sh: /opt/glite/yaim/bin/yaim: No such file or directory

This looks like on the first run after a complete re-install it trys to runs yaim without installing yaim first or infact actually running the WMS package stanzas. Slightly bizarre.

However, another run of cfagent -qv seemed to work as this correctly ran through the WMS package stanzas of glite-WMS, glite-LB, rbwmsmon. Although in the spirit of grid wierdness there were a few warnings when it installed Condor just to make you wonder what was going on!

Installing: condor ##################### [ 58/105]WARNING: Multiple network interfaces detected. Condor might not work
cfengine:dev009: properly until you set NETWORK_INTERFACE =
cfengine:dev009: Unable to find a valid Java installation
cfengine:dev009: Java Universe will not work properly until the JAVA
cfengine:dev009: (and JAVA_MAXHEAP_ARGUMENT) parameters are set in the configuration file!
cfengine:dev009: Condor has been installed into:
cfengine:dev009: /opt/condor-6.8.4
cfengine:dev009: In order for Condor to work properly you must set your
cfengine:dev009: CONDOR_CONFIG environment variable to point to your
cfengine:dev009: Condor configuration file:
cfengine:dev009: /opt/condor-6.8.4/etc/condor_config
cfengine:dev009: before running Condor commands/daemons.
cfengine:dev009:

After some internet searching and logging in/out I could see that CONDOR_CONFIG was actually set correctly.

dev009:~# echo $CONDOR_CONFIG
/opt/condor-c/etc/condor_config

Step 2. Install yaim by hand. Then blow dev009 away and let cfengine do the whole lot.

Before running yaim I made sure that dev009 was included in the site-info.def. I first created a file in the node directory within yaim to override the current production WMS and this was okay to configure the WMS first time around. However, on successive runs of cfagent it overwrote the site-info.def - doh! Therefore, Mike suggested having a development site-info.def which is copied over the current production each time cfagent is run. This worked a treat.

So on with the show with yaim for the WMS: /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-WMS -n glite-LB

Although yaim appeared to run successfully. There were a few warning/errors which had to be explained.

cfengine:dev009:m/bin/yaim -c -: WARNING: Only 1 pool account defined for tag 'sgm' of VO VO.PANDA.GSI.DE
cfengine:dev009:m/bin/yaim -c -: users_getspecialgroup: could not find 'sgm' user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialprefix: could not find 'sgm' prefix for BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialgroup: could not find 'prd' user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: users_getspecialprefix: could not find 'prd' prefix for BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: INFO: users_getspecialusers: could not find sgm user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: ERROR: Could not determine mapping for tag 'sgm' of VO BIOMED
cfengine:dev009:m/bin/yaim -c -: INFO: users_getspecialusers: could not find prd user for VO BIOMED in /opt/glite/yaim/etc/users.conf
cfengine:dev009:m/bin/yaim -c -: ERROR: Could not determine mapping for tag 'prd' of VO BIOMED
cfengine:dev009:m/bin/yaim -c -: WARNING: No mapping found for "/biomed/Role=lcgadmin" in /tmp/yaim.vF5821
cfengine:dev009:m/bin/yaim -c -: WARNING: No mapping found for "/biomed/Role=production" in /tmp/yaim.vF5821

These are normal errors/warning and can be explained as the Panda VO does indeed have only one sgm account since there is only one panda account -Dan! The Biomed VO conversely has no sgm, prd or admin accounts and only run as plain users. Therefore, these messages are expected. The next warning was slightly more worrying since these undefined variables could be required.

cfengine:dev009:m/bin/yaim -c -: [Fri Dec 19 13:35:50 2008] [warn] PassEnv variable GLITE_WMS_WMPROXY_WEIGHTS_UPPER_LIMIT was undefined
cfengine:dev009:m/bin/yaim -c -: [Fri Dec 19 13:35:50 2008] [warn] PassEnv variable GLITE_SD_VO was undefined

After some searching on the web this is a warning and yaim will just use the default values. There also seemed to be some issue with the schema file in the WMS BDII section of yaim:

cfengine:dev009:m/bin/yaim -c -: Starting glite-lb-interlogd ...chown: cannot access `/opt/bdii/var': No such file or directory
cfengine:dev009:m/bin/yaim -c -: sed: can't read /opt/bdii/etc/schemas: No such file or directory

The file /opt/bdii/etc/schemas was missing. The fix is to copy the /opt/bdii/doc/schemas.example file to /opt/bdii/etc/schemas and re-run yaim.
The re-run of yaim also fixes the chown of /opt/bdii/var.

Step 3. Testing from dev008 the development UI yielded:

*************************************************************
BOOKKEEPING INFORMATION:

Status info for the Job : https://dev009.gla.scotgrid.ac.uk:9000/d0nTvt6udraqpjs0Mx-eOw
Current Status: Done (Success)
Exit code: 0
Status Reason: Job terminated successfully
Destination: svr021.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-q30m
Submitted: Thu Jan 15 17:19:59 2009 GMT
*************************************************************

Therefore, I conclude: we have a working development WMS on dev009

Am I seeing double site bdii?

With the imminent move of the development rack we need to move some of the important grid infrastructure out of the current dev rack and into a permanent production home in clustervision. To minimise site downtime we would like to create a temporary scotgrid BDII on svr027 (currently unused). So here goes.....

when running cfagent -qv it ran successfully on svr027 through the files, links, editfiles, packages including the correct glite-BDII packages and copy sections

All was going well until YAIM.

notes from configuring the UI

running yaim for a UI node will configure the UI, /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n BDII_site

this caused the following errors:

cfengine:svr027:m/bin/yaim -c -: INFO: Executing function: config_edgusers
cfengine:svr027:m/bin/yaim -c -: chown: cannot access `/opt/bdii/var': No such file or directory
cfengine:svr027:m/bin/yaim -c -: sed: can't read /opt/bdii/etc/schemas: No such file or directory
cfengine:svr027:m/bin/yaim -c -: INFO: Executing function: config_bdii_only
Stopping BDII27:m/bin/yaim -c -: [FAILED]
cfengine:svr027:m/bin/yaim -c -: Starting BDII [ OK ]

These errors were slightly puzzling but I realised that I had not changed anything in the site-info.def.
So I changed the SITE_BDII_HOST parameter from this:

SITE_BDII_HOST=svr030.$MY_DOMAIN

to this:

SITE_BDII_HOST="svr030.$MY_DOMAIN svr027.$MY_DOMAIN"

and re-ran /opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n BDII_site

This time the only error was:

sed: can't read /opt/bdii/etc/schemas: No such file or directory

but the configurator still produced:

INFO: Configuration Complete. [ OK ]
NFO: YAIM terminated succesfully.

checking the /opt/bdii/etc on svr027 I had this:

svr027:/opt/bdii/etc# ls -la
total 64
drwxr-xr-x 2 edguser edguser 4096 Dec 17 16:17 .
drwxr-xr-x 6 root root 4096 Dec 17 15:55 ..
-rw-r----- 1 edguser edguser 503 Dec 17 16:17 bdii.conf
-rw-r--r-- 1 edguser edguser 2535 Oct 13 13:54 BDII.schema
-rw-r--r-- 1 edguser edguser 50 Oct 13 13:54 bdii-update.conf
-rw-r--r-- 1 edguser edguser 634 Oct 13 13:54 DB_CONFIG
-rw-r--r-- 1 edguser edguser 246 Oct 13 13:54 default.ldif
-rw-r--r-- 1 edguser edguser 1783 Oct 13 13:54 glue-slapd.conf

checking this against svr030 I had this:

svr030:/opt/bdii/etc# ls -la
total 48
drwxr-xr-x 2 edguser edguser 4096 Oct 8 10:35 .
drwxr-xr-x 6 root root 4096 Feb 10 2008 ..
-rw-r--r-- 1 edguser edguser 364 Oct 8 10:35 bdii.conf
-rw-r--r-- 1 edguser edguser 50 Feb 10 2008 bdii-update.conf
-rw-r--r-- 1 edguser edguser 377 Feb 10 2008 indexes
-rw-r--r-- 1 edguser edguser 268 Oct 8 10:35 schemas

very different!

I then decided to reboot and try again from scratch just to make sure there was nothing hanging around from the previous failure.
When I installed everything in the same way. The file structure still appeared different. So i decided to test the site level BDII to see if it actually worked.

svr027:/opt/glite/yaim/etc# ldapsearch -xLLL -b mds-vo-name=UKI-SCOTGRID-GLASGOW,o=grid -p 2170 -h svr027.gla.scotgrid.ac.uk > svr027.txt
svr027:/opt/glite/yaim/etc# ldapsearch -xLLL -b mds-vo-name=UKI-SCOTGRID-GLASGOW,o=grid -p 2170 -h svr030.gla.scotgrid.ac.uk > svr030.txt

this was then compared: cat svr027.txt | sort > ldapsvr027.txt;cat svr030.txt | sort > ldapsvr030.txt;diff -y ldapsvr027.txt ldapsvr030.txt | grep '>' | grep '.gla.scotgrid'

On comparing the output from an ldap search it apparent that something was missing as their output showed some missing servers. After a quick discussion with Sam we found the file /opt/glite/etc/gip/site-urls.conf and noticed the differences: the DPM2 and BDII_TOP i.e. svr025 and svr019

svr027:/opt/glite/etc/gip# cat site-urls.conf
CE ldap://svr021.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
CE2 ldap://svr026.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM ldap://svr018.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS ldap://svr022.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS2 ldap://svr023.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII ldap://svr027.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
VOBOX ldap://svr024.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid

svr030:/opt/glite/etc/gip# cat site-urls.conf
CE ldap://svr021.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
CE2 ldap://svr026.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM ldap://svr018.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
DPM2 ldap://svr025.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS ldap://svr022.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
WMS2 ldap://svr023.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII ldap://svr030.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
BDII_TOP ldap://svr019.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid
VOBOX ldap://svr024.gla.scotgrid.ac.uk:2170/mds-vo-name=resource,o=grid

after updating svr027 and restarting /etc/init.d/bdii restart we now have a operational site BDII on svr027.

The question is, should these additional entries be in /var/cfengine/inputs/skel/yaim/services/glite-bdii?

On with the move!

Update: svr027 is currently the only SITE_BDII in the GOC DB

Development / PreProd : The UI

I thought my first foray into grid middleware installations deserved a quick blog so here goes. Apologies in advance if I am covering old ground.

With grid02 now defunct and dev008 very much part of the alive and kicking it was time to install the required packages/middleware and configure it to run as a UI.

First thing for me was to understand/create a cfagent script for the new host. After much deliberation of wishing to keep it all separate and out of the way of the main production script. I decided to add it into the main script to save duplication. Perhaps something to think about for the future may be to split this up into much smaller modules per host and import a few common modules. Although, at this stage I am inclined to go with the old adage, "don't fix it if it ain't broken". I have also heard/read much of puppet which is built on cfengine with bells and whistles. Perhaps something to look at? Anyway, on with the install.

Once the script was created I ran cfagent -qv . However, beginner's luck was thin on the ground and it failed to install the packages properly first time around.

First off there was a missing dependency:

cfengine:dev008: --> Processing Dependency: perl(URI::URL) for package: perl-libwwError: Missing Dependency: log4cpp >= 1.0 is needed by package glite-ce-cream-client-api-c
cfengine:dev008: Error: Missing Dependency: liblog4cpp.so.4 is needed by package glite-ce-cream-cli

The fix was to include the DAG repo onto dev008 to pull a later version of log4cpp.
However, there was some issues surrounding this as the UI is 386 and the grid machines we have are generally 64 bit machines.
So the DAG repo url in /etc/yum.repos.d/dag.repo had to be fudged to change the /$basearch variable to i386

After a yum clean all I ran cfagent -qv again. This resulted in a second error:


cfengine:dev008: Transaction Check Error: file /usr/share/java/jaf.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp
cfengine:dev008: file /usr/share/java/jaf_api.jar conflicts between attempted installs of geronimo-jaf-1.0.2-api-1.2-11.jpp5 and sun-jaf-1.1-3jpp


This was a known error with the middleware install and the fix was to run yum install glite-UI --disablerepo=jpackage17-generic

After a third run of cfgent -qv it was good to go or so I thought. What I did see was that it was running YAIM and failing. Therefore, I opted to run YAIM manually. Using the normal UI command, /opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI I generated the following error:


INFO: Executing function: config_workload_manager_client_setenv
INFO: Executing function: config_workload_manager_client
ERROR: RB_HOST is not set
ERROR: One of the functions returned with error without specifying it's nature !


After a quick cat of the site-info.def, indeed RB_HOST is commented out as presumably the WMS is in there instead.


WMS_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
LB_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
#RB_HOST=svr023.$MY_DOMAIN


I managed to amend the local site-info.def before cfagent set it back to the original value and this allowed YAIM to get further. After reading some sites, I opted for this config as it appeared that you could actually have WMS_HOST and RB_HOST defined in the one file. Perhaps a WMS install will not like this setting? We will have to see.


WMS_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
LB_HOST="svr022.$MY_DOMAIN svr023.$MY_DOMAIN"
RB_HOST=$WMS_HOST


running yaim again: opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI now returned some errors when build the globus core:


gpt-build ====> Changing to /etc/grid-security/vomsdir/BUILD/globus_core-4.30/
gpt-build ====> BUILDING FLAVOR gcc32
GLOBUS_LOCATION=/opt/globus; export GLOBUS_LOCATION; GLOBUS_CC=gcc; export GLOBUS_CC; /etc/grid-security/vomsdir/BUILD/globus_core-4.30//configure --with-flavor=gcc32
Dependencies Complete
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... no
checking for cc... no
checking for cc... no
checking for cl... no
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details.


It looked bizarrely like gcc is not installed by cfengine on a sl4.i386 version by default, so to fix: yum install gcc . After checking the cfagent.conf this does appear to be the case. There are lots of additional packages for sl4.x86_64 but not for i386. Should this be the case?

After another re-run of yaim: opt/glite/yaim/bin/yaim -c -s ../etc/site-info.def -n UI


INFO: Configuration Complete. [ OK ]
INFO: YAIM terminated successfully.


This looked better and after sourcing the grid-env that had just been installed: source /etc/profile.d/grid-env.sh commands like: voms-proxy-init -voms vo.scotgrid.ac.uk were successful. In fact I was able to submit a job and retrieve its data from dev008. So installation successful. Or so I thought. I updated the cfagent.conf and ran it all from cfengine.

Cfengine appears to make two passes. The 1st pass install works correctly. It installs the UI, configures through YAIM. However, since some of the fileedits and links rely on the existence of a configured glite they actually fail on the first pass i.e


cfengine:dev008: Error while trying to link /opt/glite/bin/python2 -> /usr/bin/python32
cfengine:dev008: Error while trying to link /opt/glite/bin/grid-proxy-init -> voms-proxy-init
cfengine:dev008: Error while trying to link /opt/glite/bin/grid-proxy-info -> voms-proxy-info
cfengine:dev008: Couldn't stat /opt/glite/etc/glite_wmsui_cmd_var.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/edg/etc/edg_wl_ui_cmd_var.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/glite/etc/gaussian/glite_wms.conf - no file to edit
cfengine:dev008: statcfengine:dev008: Couldn't stat /opt/glite/etc/gaussian/glite_wmsui.conf - no file to edit


I had expected these to be caught on the second pass as glite was installed and configured but that run of cfagent -qv does not pick them up on the second pass. When cfagent -qv is ran a second time it does update the files appropriately. Not sure this is the behaviour we want. Does anyone remember if this happened with the original UI? Currently the dev008 is using all the original classes for ui and clusterui at the moment and should be running in the same way as the original UI install.

So to summarise the questions:

1. Can you set RB_HOST and WMS_HOST in the same site-info.def?
2. Are there lots of packages missing for a sl4.i386install?
3. Does anyone remember from the original UI install what happens when it updates files on the second pass?

So a partial success, now onto a WMS.

YAIM stale...

Another problem I found with configuring the new VOs is that the reconfiguration of the information system on the CE failed. Mike had run config_gip, as has been done forever, but it did nothing. So queues and access rights for the VOs were not published.

There are new YAIM functions, like config_gip_ce, as well as a whole different way of structuring site-info.def. I fiddled with various options in the site-info.def file and even in the YAIM function itself, but I couldn't get it to work properly at all.

In some desperation I hacked the ldif file in the end, which is not a long term option; we really need now to look at the whole way that we structure YAIM for the site (as well as plan migrations to SL4 for BDII, CE and DPM...).

Glasgow Updated to gLite 3.0.1 r27

Upgrade notes:

Basically I'm following
https://www.gridpp.ac.uk:443/wiki/UKI-SCOTGRID-GLASGOW_enabling_VO,
but being also aware that when services are updated YAIM needs rerun.

Preparations
------------

Added new groups for sgm and prd pool accounts - even when these will
not yet be enabled

Modified the poolacct.py script - now much improved and does the new
type of accounts (but can still do the old type!). Also now does
user.conf fragments as well.

Using this, added relevant entries to passwd, group, shadow and
users.conf.

Went through site-info.def again. Added supernemo stanzas and the
vo.d/supernemo.vo.eu-egee.org definitions.

All set!


Disk Servers
------------

Modify update.conf to clear cruft out of system (pruge=true)
Clear and restore yum.repos.d.
yum update
Remove local config_mkgridmap function
Run /opt/glite/yaim/bin/yaim -r -s /opt/glite/yaim/etc/site-info.def
-f config_mkgridmap
Remove MySQL-server, which had bizarrely been installed on some of the
nodes.

Noticed that config_lcgenv is broken for DNS VOs. Submitted ticket
24917, with patch. However, for us this is controlled by cfengine, so
defined appropriate things here: e.g.,
VO_SUPERNEMO_VO_EU_EGEE_ORG_DEFAULT_SE.


Worker Nodes
------------

Changed passwd/group files have already triggered home directory
creation via cfengine. Thus no need to run YAIM.

Ran yum update on WNs (using pdsh). Removed glite-SE_dpm_disk,
DPM-gridftp-server and DPM-rfio-server RPMs - relic of !disk037
(woops). WNs will be blown and rebuilt at transition to SL4 anyway.


MON/Top Level BDII
------------------

Checked svr019 (MON + Top BDII). Nothing to update here. (Top level
BDII was done last week:
http://scotgrid.blogspot.com/2007/07/top-level-bdii-updated-to-glue-13yaim.html)


Site BDII/svr021
----------------

Did yum-update then reran YAIM. Got error
SITE_SUPPORT_EMAIL not set
and
chown: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
chmod: failed to get attributes of `/opt/lcg/var/gip/ldif': No such file or directory
???
Odd entry has appeared:
GIP file:///opt/lcg/libexec/lcg-info-wrapper
Which is not active on a stand alone site BDII (would work on CE?)

Note that DPM not yet upgraded, so still polling GRIS on svr018 until
this is done.


UI / svr020
-----------

yum update
rerun yaim - discovered that i need to defined supernemo queue to be
snemo in $QUEUES (which is still used). Caused gLite python to throw
an exception (my mistake, but crap code nonetheless...)

Corrected site-info.def and reran YAIM.


CE / svr016
-----------

Checked list of RPMs to update. Potentially dangerous ones are
vdt_globus_jobmanager_pbs (we have patched pbs job manager).
Seems that there are patches to pbs jobmanager to support DGAS
accounting. I have commented out the cfengine job manager replacement
and will diff and repatch as necessary after configuration.

Created cdf and snemo queues using torque_queue_cfg script (note this
now adds access for sgm and prd groups, even if they are not used).

Ran YAIM.

Immediate sanity check:
GRIS is ok - information system is up.
gatekeeper dead
Help! Restarted.
globus.conf was rewritten, blowing away pbs job manager
re-enabled pbs job manager and restarted
OK, so I moved under the control of
cfengine, but again caveats about running YAIM
apply.
gatekeeper restarted again
Now tailing gatekeeper logs, everything looks ok.
Whew!

Diffing the pbs and lcgbps job managers, YAIM has added DGAS support
for them. I used these as new template modules and repatched the
"completed" job state (https://savannah.cern.ch/bugs/?7874). Helper.pm
was unchanged, so still has the correct patch for stagin via globus.



svr023 / RB
-----------

Ran yum update. Actual RB has not changed, so did not run YAIM


svr018 / DPM
------------

Upgraded YAIM to check on config_DPM_upgrade. Looks quite simple.

Booked downtime for 2pm to do this.

At 2pm: Stopped DPM
yum update
run config_DPM_upgrade YAIM function (updated db). Took ~8 minutes.
Start DPM again

run config_gip YAIM function (publish access details for
cdf/supernemo in info system)
run config_mkgridmap YAIM function (add additional certificates
into gridmap files)
run config_BDII to redo information system
servers)

Checked BDII was ok. It is.

Then went back to site BDII, changing URL to
BDII_DPM_URL="ldap://$DPM_HOST:2170/mds-vo-name=resource,o=grid" -
restarted site BDII. Checked ldap info was ok.

Came up from downtime (took 15 minutes). Damn - we got SAM tested in the interval!


After Lunch
-----------

Wary of APEL changes I read Yves' notes in
http://www.gridpp.ac.uk/wiki/GLite_Update_27. I couldn't see the same
problems. Ran APEL publisher on CE and MON and things seem to be ok,
so let things lie here.

Finally able to lock atlas pool down to atlas members! When I looked
in the Cns_groupmap though, there had been rather an explosion of
atlas groups:

mysql> select * from Cns_groupinfo where groupname like 'atlas%';
+-------+------+-----------------------+
| rowid | gid | groupname |
+-------+------+-----------------------+
| 2 | 103 | atlas |
| 16 | 117 | atlas/Role=lcgadmin |
| 17 | 118 | atlas/Role=production |
| 55 | 156 | atlas/lcg1 |
| 57 | 158 | atlas/usatlas |
+-------+------+-----------------------+

Hmmm, I have given the pool to all these gids. Is this really
necessary?

Done! Whew!

Top Level BDII Updated to Glue 1.3/yaim 3.0.1

I upgraded our top level BDII today, so it should now happily understand the Glue 1.3 schema. This was the deadline for the upgrade set in the EGEE operations meeting - BDIIs have to be upgraded from the top down, because they don't like information provided by lower levels to be in a schema they don't understand.

This was a more significant thing it might first seem, as I also went very carefully through our site-info.def file and used the latest 3.0.1 yaim to configure the node.

It turned out that there were significantly fewer changes than I had thought there would be. The main additions seemed to be exposing new BDII configuration variables, and changing the way that queues are configured, to allow for queues accessible to FQANs. Of course, much of this is not relevant to the BDII/MON box, which doesn't care about users or VOs, but it proves that there's no major errors in the new configuration.

I have noticed that the startup time for the BDII is slower, presumably as it's creating all of its index files; however, the performance should be much better.