LSC files and emailAddress redux

This post involves a very complicated journey to get to a simple place.

The fundamental problem is around the catchy titled OID 1.2.840.113549.1.9.1

No, wait, let me take a step back. On the Grid, we use certificates for authentication. An X509 certificate is, as with most certificates, a signed set of assertions, and a public key. As with the rest of the X500 standards, it's native language is something called ASN.1 (Abstract Syntax Notation 1) (aka X208, and the later revision X680), held in files encoded by the DER (Distinguished Encoding Rules).

The fundamental takeaway from that tech-dump is that X509 certificates are not in plain text, and there are multiple standards required in order to understand their contents.

So when someone says their certificate Distinguished Name is '/O=SomeUni/OU=SomeDept/L=group/CN=JohnSmith' ... that's not quite accurate. What they really mean is that there certificate DN is some set of objects that can be unambiguously matched to that ASCII text.

That happens because there are universally agreed mappings between the actually stored OID and the text representation of them (e.g. CN is OID 2.5.4.3).

Unfortunately, the agreement breaks down a bit for the emailAddress field; with some software mapping it to Email, and others to emailAddress. By the PKCS#9 standard, one could argue that it should be emailAddress - but that doesn't help us get software working.

Fortunatly, all of this is not a problem unless we want to store certificate DN's in ASCII, _and_ want to have email addresses in the DN.

Yeah, you can see where this is going, can't you?

In the UK, Jens has been working to allow us to not have them in DN's. However, in the short term, they are present.

One particular case where ASCII representations of the DN are used is in LSC files - which are used to authenticate VOMS servers. What happens is if the VOMS server DN matches the DN in the LSC file, and the cert was signed by the CA DN in the LSC file, _and_ the certificate chain is signed by a trusted root, then it's valid. This process means that we don't need to distribute lots of VOMS server certs, just the root CA's, and a small note (that shouldn't change over renewals) of the server DN.

I've been tidying up our ARC install here, and during the process managed to break things. Not unusual for me, (one of the reasons I avoid tiding at all costs!), but this one was quirky. I'd put the vomsdir under CFEngine control, so that it was sync'd with all the other servers, and suddenly it stopped accepting the scotgrid VO.

Root cause, as if you can't guess by now, LSC file, and the emailAddress. Looks like the gLite stack expects it one way, and ARC the other. Of course, by the time you read this, that's probably been fixed somewhere, but not in the version we had installed.

It turns out that there's one trick in LSC files that saves this case. Let me put the LSC file in here:

/C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk
/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA
------ NEXT CHAIN ------
/C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/emailAddress=grid-certificate@physics.gla.ac.uk
/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA

The 'NEXT CHAIN' line lets one put multiple entries in the file. However, it appears that ARC isn't reading multiple, only the first one. So, in this case, I put the ARC friendly one first, so it matches fine - and the gLite stack tries again, finds the second, and thus suceeds.

Imporant notes: I can't find anyone else with a field report of NEXT CHAIN working in the gLite stack. This is such a field report. It doesn't appear to work with ARC.

local users before pool users

Further to the original post by Graeme 'to voms or not to voms'. The Nikhef documentation has been thoroughly overhauled and I have now been able to switch lcmaps in CREAM and SCAS over to use local unix group mappings before pool accounts, if they exist.

The main changes are changing localaccount to pull in the glasgow centric grid-mapfile.

localaccount = "lcmaps_localaccount.mod"
               " -gridmapfile /usr/local/etc/grid-mapfile-local"
              # " -gridmapfile /etc/grid-security/grid-mapfile"

Some small tweaks are required to move localaccount from the last check to the first check. If this is successful it uses that account, otherwise it moves to check voms and pool accounts.

glexec_get_account:
proxycheck -> localaccount
localaccount -> good | vomslocalgroup
#proxycheck -> vomslocalgroup
vomslocalgroup -> vomspoolaccount | poolaccount
vomspoolaccount -> good | vomslocalaccount
vomslocalaccount -> good | poolaccount
poolaccount -> good #| localaccount

glexec_verify_account:
proxycheck -> localaccount
localaccount -> good | vomslocalgroup
#proxycheck -> vomslocalgroup
vomslocalgroup -> vomspoolaccount | poolaccount
vomspoolaccount -> good | vomslocalaccount
vomslocalaccount -> good | poolaccount
poolaccount -> good #| localaccount

SCAS is works in the same way and all that is required is to change the localaccount setting to pull in our Glasgow local grid-mapfile a'la

localaccount = "lcmaps_localaccount.mod"
                " -gridmapfile /usr/local/etc/grid-mapfile-local"
#               " -gridmapfile /etc/grid-security/grid-mapfile"

Job done. I can now flit between gla or pool accounts depending on my existence in /usr/local/etc/grid-mapfile-local

Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
2013.svr008               cream_441636610  ssp001                 0 R q2d            
2014.svr008               cream_963867097  gla057                 0 Q q2d

redundancy in grid services proves it worth

From about midway through 2008 there has been a second lcg-CE, svr026 at Glasgow. This has proved useful for dealing with intermittent failures through the course of normal operations. However, some time last year the original lcg-CE, svr021 started to seg fault when the VOMS plugin was activated. Therefore, VOMS was turned off and the second lcg-CE took on the role of offering VOMS mapping to group accounts whilst the original CE dealt with mapping local users to local accounts with no VOMS. So in an effort to fix svr021 and build a fully redundant system some changes have been scheduled. Namely, server rebuild of svr021 and NFS mounting of the gridmapdir to share it between all CE's. However, taking down key grid services in a running production cluster full of jobs is not a trivial matter. So here is an overview of the experience for our lcg-CE.

Of the tasks that we had to do the easiest by far was to NFS mount the gridmapdir (used to track the VOMS pool accounts). This was created on disk037 and a cron'd rsync set up from second lcg-CE, svr026 which had working VOMS. This allowed a mirror of the currently running gridmapdir to be created. After some testing on development and during a quiet moment the gridmapdir on svr026 was blown away and remounted from NFS. This was successuful as far as I could tell. For resilancy another rsync and a cron'd script was setup from the NFS to a backup dir on svr026 so that should the NFS fail. It will revert back to a local gridmapdir automatically. This will also be done on svr21 as part of the rebuild.

svr26 at this time although supporting VOMS did not support the "don't map local accounts to pool accounts" addition that svr021 had been retrofitted with. This was outlined here: http://scotgrid.blogspot.com/2008/02/to-voms-or-not-to-voms-that-is-question.html . This was applied to svr026. Again this was tested thoroughly on development and was applied successfully to production. So now svr026 was nearly in a state to become the primary lcg-CE. What it did not have was support for all the available queues at Glasgow. As historically some queues were shared and some were specific to svr021. This was fixed by re-running yaim on svr026 and setting the appropriate $QUEUES in the site-info.def. Having a development CE to test this was invaluable since the last time yaim had been run a CE was sometime last year and the behaviour of one of the functions this time round actually caused yaim to fail. With a fix in place svr026 now supported all queues.

This meant that svr021 was ready for draining. The documented procedure for draining a lcg-CE relies on disabling queues in the batch system, pbs in this case. However, in our case we have shared queues so disabling svr021's queues would have disabled svr026's. Not what we wanted at all. So a workaround was found to modify the gip plugin on svr021 to always set the CEStatus values to Draining. This was then picked up automatically by LDAP and our site posted svr021 as draining. This should have been enough if it were not for direct submission to a CE as GStat and other tools using LDAP to determine available resources checked the CEStatus and knocked out svr021 but custom JDL and Ganga scripts can just select a CE in the script. After some investigation with running jobs it became apparent that in fact many of the local ScotGrid users use the Grid in this way. Doh. So after some experimentation on development we found that you could remove the CE from the hosts.equiv file on the batch system even when there were running jobs. This effectively stopped submission to the CE dead in its tracks and rather handly allowed running jobs to finish successfully. The only other thing to remember to do was downtime the node in the GOCDB as SAM would start failing as we were no longer advertising or accepting jobs on our svr021 lcg-CE.

With svr021 now draining the last piece of the puzzle is to rebuild it but since we support 7 day queues at Glasgow its going to be a long wait. Once its drained the plan is to make sure we have our 90 days worth of logs, run APEL to publish our final results, rebuild, apply the local-novoms patch and mount the gridmapdir from the NFS share. All going well we should have two fully operational lcg-CE's by next week with no lost jobs in sight.

Conclusions
Grid service redundancy allowed our Glasgow cluster to operate for sometime in a semi-broken state without the immediate requirement for a rebuild.
Having a test/dev server with which to test changes before applying them prod was invaluable.
Running mirrored grid services to perform maintenance tasks is a perfect way to keep your cluster accessible and running during downtime.
7 Days is a long time to wait for a queue to drain!

nanocmos + lcas = FAIL

While working on an unrelated issue on svr021 I noticed an edg-mkgridmap error in the logfile

Aug 29 05:28:14 svr021 edg-mkgridmap[6693]: voms search(https://svr029.gla.scotgrid.ac.uk:8443/voms/vo.scotgrid.ac.uk/services/VOMSCompatibility?method=getGridmapUsers): Internal Server Error

Mentioned to mike who promptly went and fixed the issue, only to discover 30 mins later we're failing SAM tests - LCAS voms plugin had once again gone fubar and caused globus-gatekeeper to segfault

Aug 29 12:14:41 svr021 GRAM gatekeeper[662]: Authenticated globus user: [DN REMOVED]
Aug 29 12:14:41 svr021 GRAM gatekeeper[663]: Authenticated globus user: [DN REMOVED] Aug 29 12:14:41 svr021 kernel: globus-gatekeep[662]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4
Aug 29 12:14:41 svr021 kernel: globus-gatekeep[663]: segfault at 0000000000000046 rip 0000000000b86259 rsp 00000000ffff9d98 error 4


the globus gatekeeper log has a bit more info:
TIME: Fri Aug 29 12:14:41 2008
PID: 663 -- Notice: 5: Authenticated globus user: [DN REMOVED]
lcas client name: [DN REMOVED]
LCAS 0:
LCAS 1: Initialization LCAS version 1.3.7
allowing empty credentials
LCAS 2: LCAS authorization request
LCAS 0: lcas_userban.mod-plugin_confirm_authorization(): checking banned users in /opt/glite/etc/lcas/ban_users.db
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
LCAS 0: lcas_plugin_voms-plugin_confirm_authorization_from_x509(): Did not find a matching VO entry in the authorization file
LCAS 0: 2008-08-29.12:14:41 : lcas_plugin_voms-plugin_confirm_authorization_from_x509(): voms plugin failed
LCAS 0: lcas.mod-lcas_run_va(): authorization failed for plugin /opt/glite/lib/modules/lcas_voms.mod
LCAS 0: lcas.mod-lcas_run_va(): failed
JMA 2008/08/29 12:14:45 GATEKEEPER_JM_ID 2008-08-29.11:14:39.0000014519.0000000000 JM exiting

As before, commenting out the lcas_voms.mod in /opt/glite/etc/lcas/lcas.db allows it to work, at the expense of losing VOMS roles.

We've got it working using the voms_mod at the moment by altering the ACLs on the VOMS server (svr029) for nanocmos. Now to try and debug the lcas plugin failure

To VOMS or not to VOMS? That is the question (for LCMAPS...)

Our advice to local users of the cluster has traditionally been to not use VOMS credentials. This is to ensure that they are mapped in the batch system to their local account, rather than to a pool account from their VOMS attributes (mappings to local accounts are maintained by the grid-mapfile-local file). In the default configuration of LCMAPS VOMS pool account mappings are made before the grid-mapfile, which is now just a fall back.

However, I could not simply reverse the order of the LCMAPS plugins as this would undo all the good which VOMS brings and move everyone back to a single fixed or pool account mapping no matter what their VOMS credentials (this would probably have affected me worse than anyone as I flit between atlas, atlas/Role=production, vo.scotgrid.ac.uk and dteam!).

So, for local users grid-proxy-init seemed to be the way to go, even if I knew this would come back and be a problem later. However, later became earlier as soon as I started to test the gLite-WMS - here it turns out you must use a VOMS proxy. Simple grid proxies just don't work anymore.

Finally, puzzling over the very poor LCMAPS documentation, and staring at the configuration script I managed to solve the problem by:

1. First running a local account plugin against a grid-mapfile which only contains our local user accounts.
2. Then running the VOMS plugins as usual.
3. Finally, running the grid-mapfile plugin, against the usual /etc/grid-security/grid-mapfile.

This was almost too easy to be true - and indeed it turns out not to quite be that simple as you hit a bug in LCMAPS that you cannot use a module twice - so having lcmaps_localaccount.mod twice is not possible. However, it turns out that one can do it if the module is renamed and physically copied. This works, so we now have an lcmaps_localaccount.mod and a lcmaps_localuseraccount.mod - exactly the same bytes, different names! (To be strictly accurate we have two copies of liblcmaps_localaccount.so.0.0.0, to which these links point.)

And, in the end, I was able to keep myself out of the local user grid-mapfile, so I have the full array of VOMS roles for myself, while the local users are cosily tucked up in their local account areas.

SAM tests changed VOMS Role (without warning!)

We, and a large fraction of the rest of the grid, started to fail replica management tests late on Friday night. At first I thought it must be a catalog problem at cern, so I raised a ticket. However, it turned out that what had actually happened was that the VOMS role used to submit the SAM tests had changed. This caused DPM to map the SAM tester DN into a different group - who then did not have permission to write into the default generated directory for lcg-cr.

This change was made completely unannounced, and I suspect without any real thought as to the implications for sites using DPM 1.6.3 and earlier.

Maarten Litmath helpfully posted a fix-up script on LCG-ROLLOUT, which uses ACLs to grant suitable privileges to lcgadmin and production roles for each supported VO, which I applied to Glasgow and Durham at about midnight last night (I was surely violating cardinal rules of sysadmining, but I couldn't see how it would cause harm - and this time I got away with it). This did fix the problem.

I'm really annoyed that this though. Changes like this should never, ever be made on a Friday! (It seemed the change actually came through at ~10am, but didn't break until midnight, when the next YYYY-MM-DD directory needed to be created.) In addition several people have commented that the fix is to upgrade to DPM 1.6.4 - despite the fact that this is broken in gLite 3.0r25 in two significant ways!

Grrrr. I just hope they don't ask us to explain these SFT failures - they shall have a piece of my mind... (I sound just like my Mum, when she was annoyed - see what the grid's doing to me!).

VOMS is a Good Thing (tm)

Spent some of yesterday playing with VOMS and the static grid-mapfile mappings we put into the system. The point of these was to support users who were not yet in any VO and might be using globus-job-* commands to submit jobs to the cluster. For these users it was obviously important that they mapped to the same account on the batch system as on the UI. (What actually happens is that the UI mirrors its grid-mapfile from the CE.)

However, we now have a different class of user, one who might use gsissh to access the cluster, but use the RB to submit jobs and be in a VO. For this user we need a local mapping in the grid-mapfile, but we need to ensure that job submission is made with a normal VO mapped pool account.

To test this I mapped myself to a local account (gla012) on the UI, using gsissh to login. Then I initalised a VOMS proxy and submitted a job. The job correctly ran as dteam.

This means we can support our multiple classes of users, and also users in multiple VOs, in a relatively straightforward way.

One point for the local account, though, is that we will put the local account into the correct group for the user's expected VO. This means that job submission with a vanilla proxy will work as expected.

(An outstanding problem might well be data write access - at the moment the data users put onto nfs using their local account is fine for writing, but the area cannot be written to. This may be an issue for some people.)

Fiddly old VO configurations

Got an email from a Zeus VO member who couldn't use the SE. Poking around revealed that we hadn't updated the VO membership list to be obtained from VOMS, instead of the old LDAP system.

As YAIM is now in a mess, I had to modify /opt/edg/etc/edg-mkgridmap.conf and /opt/lcg/etc/lcgdm-mkgridmap.conf by hand and then rebuild the mapfiles.

The should fix things, but I so, so wish we had a one-stop-shop for configuring VOs. It's too easy to leave VOs half cocked on the system, and often it's months before anyone notices or complains.

Total Checks Out

The problem with VOMS was that the DN of the gridpp VOMS server was wrong in the setup.

Once I realised that, it all started to work.

Results were that it all works! lcg-cr and lcg-del function perfectly, so it seems the problem is with Hannah's UI.

Graeme Becomes an Oil Man

To help Hannah at Total with LFC issues I have joined the totalep VO.

Unfortunately things then fell over at the first hurdle - voms-proxy-init fails for this VO:

ppepc62:~$ vpi -voms totalep
Enter GRID pass phrase:
Your identity: /C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=graeme stewart
Creating temporary proxy .................................... Done
Contacting voms.gridpp.ac.uk:15026 [/C=UK/O=eScience/OU=Manchester/CN=voms.gridpp.ac.uk/Email=hostmaster@hep.man.ac.uk] "totalep" Failed

Error: Could not establish authenticated connection with the server.
GSS Major Status: Unexpected Gatekeeper or Service Name
GSS Minor Status Error Chain:

an unknown error occurred

None of the contacted servers for totalep were capable of returning a valid AC for the user.

I have put a ticket in (Footprints ISSUE=1207 PROJ=14), but no news yet.

Biomed and VOMS

We got a ticket from Biomed about their jobs not running properly.

I had a suspicion that this was the case, and when I went to the yaim tool site and found that not only had they changed their VOMS certificate, which I had updated, but they had also changed their VOMS server's DN.

So, I changed their VOMS DN in our site-info.def and re-ran the config_mkgridmap function on the CE, which also configures LCMAPS. After that things seemed fine.

One to watch out for, though, when updated VOMS certificates come through.