LSC files and emailAddress redux

This post involves a very complicated journey to get to a simple place.

The fundamental problem is around the catchy titled OID 1.2.840.113549.1.9.1

No, wait, let me take a step back. On the Grid, we use certificates for authentication. An X509 certificate is, as with most certificates, a signed set of assertions, and a public key. As with the rest of the X500 standards, it's native language is something called ASN.1 (Abstract Syntax Notation 1) (aka X208, and the later revision X680), held in files encoded by the DER (Distinguished Encoding Rules).

The fundamental takeaway from that tech-dump is that X509 certificates are not in plain text, and there are multiple standards required in order to understand their contents.

So when someone says their certificate Distinguished Name is '/O=SomeUni/OU=SomeDept/L=group/CN=JohnSmith' ... that's not quite accurate. What they really mean is that there certificate DN is some set of objects that can be unambiguously matched to that ASCII text.

That happens because there are universally agreed mappings between the actually stored OID and the text representation of them (e.g. CN is OID 2.5.4.3).

Unfortunately, the agreement breaks down a bit for the emailAddress field; with some software mapping it to Email, and others to emailAddress. By the PKCS#9 standard, one could argue that it should be emailAddress - but that doesn't help us get software working.

Fortunatly, all of this is not a problem unless we want to store certificate DN's in ASCII, _and_ want to have email addresses in the DN.

Yeah, you can see where this is going, can't you?

In the UK, Jens has been working to allow us to not have them in DN's. However, in the short term, they are present.

One particular case where ASCII representations of the DN are used is in LSC files - which are used to authenticate VOMS servers. What happens is if the VOMS server DN matches the DN in the LSC file, and the cert was signed by the CA DN in the LSC file, _and_ the certificate chain is signed by a trusted root, then it's valid. This process means that we don't need to distribute lots of VOMS server certs, just the root CA's, and a small note (that shouldn't change over renewals) of the server DN.

I've been tidying up our ARC install here, and during the process managed to break things. Not unusual for me, (one of the reasons I avoid tiding at all costs!), but this one was quirky. I'd put the vomsdir under CFEngine control, so that it was sync'd with all the other servers, and suddenly it stopped accepting the scotgrid VO.

Root cause, as if you can't guess by now, LSC file, and the emailAddress. Looks like the gLite stack expects it one way, and ARC the other. Of course, by the time you read this, that's probably been fixed somewhere, but not in the version we had installed.

It turns out that there's one trick in LSC files that saves this case. Let me put the LSC file in here:

/C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/Email=grid-certificate@physics.gla.ac.uk
/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA
------ NEXT CHAIN ------
/C=UK/O=eScience/OU=Glasgow/L=Compserv/CN=svr029.gla.scotgrid.ac.uk/emailAddress=grid-certificate@physics.gla.ac.uk
/C=UK/O=eScienceCA/OU=Authority/CN=UK e-Science CA

The 'NEXT CHAIN' line lets one put multiple entries in the file. However, it appears that ARC isn't reading multiple, only the first one. So, in this case, I put the ARC friendly one first, so it matches fine - and the gLite stack tries again, finds the second, and thus suceeds.

Imporant notes: I can't find anyone else with a field report of NEXT CHAIN working in the gLite stack. This is such a field report. It doesn't appear to work with ARC.

Batch system juggling

We've been a bit quiet up here recently. This is normally a sign of either nothing interesting happening, or entirely too many interesting things happening. Opinions on that may divide, but I think it's closer to the latter...

One of the recent bits of fun that occurred was with our batch server. This story actually starts a long time ago; about this time last year. At that point, we started to get intermittent memory errors from the Torque server - corrected by ECC - but that's generally a sign that the RAM's about to fail. Given that the batch server is single point of failure for a site, that's not a good thing.

So I spent some time preparing a spare box, and being ready to move the batch system over, in case it failed over the winter break. Which, after all that prep, it didn't, and the errors stopped. On the expectation that the current hardware was nearing end of life, we ordered a new box early this year, and have had it sitting in a machine room for a while.

Unfortunately we didn't get time to have it running a tested batch system until our power supply started to ... well, insert colourful metaphor here, describing the 8 months where we were affected by lack of power.

Power got to stable supply in September, and so to catch up on things. One of the things we got around to was software versions. Whilst we didn't intent to update the Torque version, and managed to avoid it for a bit, the gLite developers eventually managed to sneak the update past us as part of an ordinary gLite update. Strictly, this didn't affect the batch server, just all the CE's, making them incompatible with the previous version of Torque.

Whilst a clever manoeuvre, reminiscent of Odysseus' Pony, it did leave us with a conundrum of either reverting the gLite update, or running forward with it. Neither were options of good character, but running forward did have some actual documentation; hence it was full speed ahead.

Which worked out well enough. The Torque 2.5.7 packages were set to use Munge, so getting that installed and tested as a first step helped it go smoothly. To preserve compatability in file locations, we used /etc/sysconfig/pbs_mom to put the pbs working directories in the same place as previously - meaning we didn't have to reconfigure any other tools.

What didn't go so smoothly was the memory leak in the server.

Which gave it a runtime of around 36 hours between crashes. Actually, not even crashes - we found that the pbs_server process hit either

12/05/2011 10:19:12;0080;PBS_Server;Req;req_reject;Reject reply code=15012(PBS_Server System error: No child processes MSG=could not unmunge credentials), aux=0, type=AlternateUserAuthentication, from tomcat@svr021.gla.scotgrid.ac.uk

or

10/29/2011 18:11:24;0001;PBS_Server;Svr;PBS_Server;LOG_ERROR::Cannot allocate memory (12) in send_job, fork failed

and then sat around moaning. Had it crashed hard, then the auto-restart would have caught it. Ho, hum, one for the Fast Fail philosophy there.


By this point, my proof reader is pointing out that I started off talking hardware, and now talking software. Punchline is that the new server that we never got a chance to use has a lot more RAM than the old server. Therefore we wanted to move the server from the old hardware to the new, to give it a lot more RAM space. That won't fix the memory leak, but will mitigate the problem a bit.

Conventionally, this would involve draining the cluster, repositioning the CE's and then starting up everything again. Had we done that, this blog post would be over now.

Instead, we did a rolling update. This let us move things over without having to do a full drain. The biggest problem with a full drain is that, while most of the jobs finish within a shorter period of time that then limit, there are always some that take the full duration. This leaves us with an empty cluster, doing nothing, for 24 hours or so, wainting on a couple of jobs to finish.

So, instead, by moving things in small batches, then we can keep most of the nodes working, and thus get more work out of things. Step zero is to disable cfengine, otherwise it tends to try and 'fix' things part way through.

Step one is to drain a CE, which we did over a weekend, and a small number of nodes, which we put offline on the Sunday morning.

Come Monday, I set up and tested basic operations with the new batch server, and then moved the freed up nodes across to it. Once those were tested (which shook out a couple of issues about versioning of some libs), point the CE at the new batch server, and then run a test job though it. (It turns out that Atlas are fast enough to sneak some pilots through a 2 minute window for a test job. However, only a few, so they actually functioned as effective tests, without compromising the site if they failed).

After that, it's time to offline another CE, and then some more nodes, and start moving nodes over when they were empty. In the end I scripted this:

#!/bin/sh

NODE=$1
RUNNING=$(qstat -n -1 | grep $NODE | wc --lines)

if [ "x${RUNNING}" != "x0" ]
then
echo $NODE: Still $RUNNING jobs going, skipping
exit 2
fi

CORES=$(qmgr -c "print node ${NODE}" | grep "np = " | cut -d= -f2)

FROM=svr666
TO=svr999

echo $NODE: Moving to ${TO} with ${CORES} cores

ssh ${TO} "~/addNode.sh ${NODE} ${CORES}"

ssh ${NODE} "service pbs_mom stop"
scp config.mom.svr666 ${NODE}:/var/spool/pbs/mom_priv/config
ssh ${NODE} "service pbs_mom start"

ssh ${FROM} "~/deleteNode.sh ${NODE}"

In theory one can run qmgr remotely, rather than ssh-ing to the batch servers and running a script. In practice, with the different versions of Torque, I couldn't get that to work. Note the automation of the mom config switch as well; and that this script checks that the node is empty.

This reduced the gradual move of nodes to a process of croning the script, and offlining nodes occasionally.

The net result was that we were operating at around 80% capacity for 48 hours, and it was all rather uneventful - in a good way. The final step was to update cfengine config and re-enable it.

One of the plus points of the above script is that it should be simple to adapt to two distinct batch systems; which means if we end up moving away from Torque, we should be able to do that without downtime too.

Scotgrid goes South

Last we week attended the bi-annual GridPP Collaboration meeting.
The venue this time was CERN itself and the meeting was, as ever, incredibly useful.

We were lucky enough to have presentations from the Experiments, the LHC, EGI and the WLCG community as well as presentations from across the UK collaboration.

A full programme of the meeting is available here:

http://www.gridpp.ac.uk/gridpp27/

Above is a picture of our own Dr Crooks presenting on the Glasgow Security Model

Circuits, Circuits everywhere but not a drop to switch

Since the late afternoon of the 26th of July we have been working to resume service on the Cluster at Glasgow.
We were put into unexpected downtime by our old friend; the power cut.

The root cause of this appears to be that the local mains supply into the site failed and was sub-sequentially re-instated. However, we decided to restart the cluster on Wednesday morning, to ensure that there was a clean and stable supply into the site. So off to the Gocdb, announce the unscheduled downtime and proceed.

While normally we would have immediately started on getting the cluster back online, as it turned out we couldn't have got ourselves back into production any sooner due to the residual issues caused by the power outage. As we have had several power interruptions at the site over the last 10 months, we have now got a reasonably robust restart procedure and we started this on Wednesday morning.

Initially, we had absolutely no issues surrounding the reset of both rooms, bar the loss of a rather expensive 10 Gig Ethernet interface on one of the new Dell Switches and the loss of the switch configuration files, which was caused by yours truly not running a copy run start on the switch after configuring a LAG group and QOS. We reconfigured the switch and all connectivity across the cluster was confirmed as good.

We then proceeded to rebuild our one of our internal stacks to free up the 10 Gig Interfaces on a Nortel 5530, which we had planned to move to our lower server room to build out the second 10 Gig link, mentioned in a previous post. This too went surprisingly well, but Dave and myself had pretested building the stack and adding and removing devices and inserting new base units on older test equipment.

We then retested again Stacking, LAGs were working fine, Spanning tree was happy and the Cluster's network was in good shape. We then moved to phase 2 of the upgrade which was to insert the 5530 switch into the switch stack in the downstairs server room. After we inserted the switch in the stack, it came up and the entire stack stabilised and then started to forward traffic.

However, about 3 minutes later we started to see the latency in the network rise and hosts fail to contact one another. Ping, SSH and normal cluster network traffic such as NFS, NTP and DNS also started to experience issues. We reduced the load on the network by detaching hosts from it but to no avail. We then removed the 5530 from the stack but the problem remained. Over the next 4 hours we tried a variety of tests which were all ending with either the dreaded Host Unreachable or 142 millisecond response times. To make matters worse (confusing), the switches were reporting an internal response time between room of 0.50 milliseconds via ping but telnet and ssh between devices was also timing out.

As we were unable to ascertain the exact root cause, we called a break and went and got some air.

20 minutes and one pizza slice later, it occurred  to me that if no device on the network was generating traffic at the volume required to generate a 94% packet loss scenario across multiple 10 Gig connections, then it has to be the network itself. Or rather what is attached to it.

The 10 Gig Interface being cooked wasn't the cause as it was dead at this point, but the power cut had left another present:

Damaged Ethernet Cables.

As the Cluster is too large to manually go round and check every cable individually with a line tester, we did something that I, as a former telco engineer, don't like doing. We rebooted the switches in numbered sequence. Starting with Stack01.

The purpose of this test is to isolate as quickly as possible the damaged cable, device or interface by pinging across the cluster from one room to another and intra switch if need be.

So Ping from Svr001 (upstairs) to Node141 (downstairs).
Destination Host Unreachable.
Leave the ping running.
Reboot Stack 01.
Ping response time of 0.056 miliseconds
Stack01 reloads.
Destination Host Unreachable.

We repeated this test twice. And got the same result.

So onto Stack01. The partner switch which trunks into this stack to affect an uplink onto the core of our network did not report any errors on the multi-link trunk but also very little traffic. Neither did Stack01, until I tried to ping its loopback address from the partner switch. The error rate on the interfaces increased and CRC counters were recorded. So we systematically disabled the multi-link trunk link by link until the stack interconnect stablised.

This reduced the trunk's capacity substantially but it also stabilised the network. So we added the 5530 back into the Stack downstairs, turned on the partner ports upstairs and were awarded with a 20 Gig backbone which is now operational at the Glasgow site.

As for the old LAG connection it was stripped out completely this morning and by early afternoon we had re-instated a 6 Gig connection to Stack01 which is working happily. From here we brought the site out of downtime and are back on the Grid.

We are putting in place an  internal tftp process for backing up switch configurations each night.

The main lesson from this is that on a large layer 2 environment, the smallest issue can become a major one and plans are well advanced on the next set of configuration changes to the network at Glasgow, to get around this and other potential issues in the future.

A switch port too far

As part of the ongoing upgrades surrounding the recent issues that the CEs have had when communicating with svr016, we decided to upgrade the core backbone link to 20 Gigabits. Presently, we have one 10 Gigabit trunk link between 141 and 243d, which is occasionally saturating with traffic.

As previously posted, we disabled the 10 gigabit link into Stack01 and used the XFP GBIC recovered from it to facilitate this new link. Sam and I laid new fiber optic patch leads in both rooms to the patch panels and connected these to spare ports on the Core Dell 8024F and Stack02's 5530.

However, the link refused to come up. After several hours investigation we acquired a fiber optic line tester which proved that light was coming through the new link. We then tested the ports on both switches with a fiber optic loop.

While the port and GBIC in the 8024F looped correctly, you get a rather re-assuring green link light on the transmit and receive port, it failed on the port in Stack02. We retested the XFP in its old unit, stack01 and it came up correctly using the loop.

While we are using 62.5 um patch leads which, under the standards can't be driven as far as 50 um,  we thought this may have been the issue, we confirmed that this wasn't the case through the re-testing of all the components end to end with the fiber optic meter.

We cleaned out the interface slot on the Stack02 5530 with compressed air and isopropyl alcohol,  the port, while recognising the gbic correctly, did not bring up the link.  We fear that the on board optical interface is damaged, however we would need to put the site into downtime to confirm this, so we have come up with a Plan B.

As we have successfully built a LAG between 141 and 243d which is in place and did not impact service at all during its commissioning, and have laid in the fiber interconnect,  we have decided to investigate moving our second 5530 into Stack02 from Stack01 to give us the 20 Gigabit uplink that we require within the core of the network.

More on this after the move. 

As an aside, you never know how windy cold aisles are, until you lift a floor tile. Sam is on the floor in this image and not glued to the ceiling as his hair direction may imply.

And after studying its behaviour, objectively and critically, we believe we have a reliable method (With apologies to Neil Fallon)

Since the last post on the blog we have implemented a series of measures on the network which were planned to be deployed during the next Cluster refresh.

Primarily, we have migrated elements of our core servers such as svr020, svr001 and svr008 to the new Dell switch infrastructure and have introduced a series of Link Aggregation Groups (LAGS) across the Dell estate to raise their backbone to a full 20 Gigabits per second intra switch. This has led to the decommissioning of the core 10 Gigabit interconnect into our old Nortel gateway, stack01and this has been replaced with another LAG between the Dell's and stack01. The reason behind this will become clear in the next post.

The main upshot of this part of the network upgrade is that we now can have greater control over the network services and monitoring running out of these servers such as SNTP and Gangli respectively. These can be fine tuned to a greater degree on the Dell environment to minimise the broadcast and Layer 2 multicast impact of these services.

However, that is not to say that the Nortel's are on the way out quite yet. Our Torque and Maui Server, svr016, still resides on older Nortel equipment in Stack02 which is currently connected to the new Dell infrastructure by a 10 Gig fibre. This link is occasionally saturating; we have decided to upgrade the link to 20 Gigabits by running a new multimode fibre between the two computer rooms, 141 and 243d. We also decided to implement Layer 2 QOS for Server016 to ensure that it got priority over all other cluster traffic within the stack and through the core network switches.

Therefore, we embarked on the re-configuration on the QOS parameters on Stack02. The complexity behind this lies not in the actual end configuration: effectively the mac address of svr016 is tracked across VLAN's 1 and 2 respectively to ensure that a Gold Quality of Service is met for any device wishing to speak to or be spoken to by Svr016. The real complexity is implementing this so that you don't disable the entire cluster attached to the network stack.

Earlier implementations of the Nortel OS had a nasty tendency to drop all non-specified traffic within the network, and the QOS policy generation, while incredibly granular in its ability to tag and filter traffic, involves 6 different stages to ensure that traffic is correctly tagged and forwarded.

Added to the fact that if the MAC address do not have the correct MAC address mask  all traffic generated by Svr016 will be dropped, effectively disabling the cluster for a period of time, a general picture of the care required to implement this feature developed on our part.

Sam and myself rechecked the configurations twice before attempting to implement them. However, when we attempted to commit  we discovered that the Nortel GUI is a lot more thorough in its checks than we could ever have imagined. Due to a mis-configuration of the MAC address mask the system refused to commit it to the switches. It even supplied an error message which identified that the mask was wrong.

Once the mask had been corrected the configuration was loaded onto stack02 and immediately started to work. The image below shows the packet matching since the 30th of June 2011.

Now for the real test. How would it cope under increased DPM traffic loads?

Surprisingly well: it turns out as now all traffic to and from svr016 has a low drop status and high precedence value across the network.

The images below show the system performance during one of this recent event.

As can be seen, there is no real increase in activity now as the QOS mappings for svr016 now mean that, while it is still part of the production and external VLANs it always travels 1st class.

The next phase of QOS development is to start to investigate the corralling of network broadcasts for services such as NFS to see if we can reduce the background chatter on the network without impacting service.

The Grid is a hungry, hungry beast....

... and it eats networks. From here begins a long, convoluted story, ending, as these often do, in something that has something that seems like it should have been obvious.

We've been noticing some 'blips', during which Maui fights bravely but ultimately fails to schedule jobs. This is generally considered rather sub-optimal.

The root of it was Maui was failing with an error:

ERROR:    cannot get node info: Premature end of message

That Maui error results in Maui taking a break for 15 minutes, before trying to schedule anything again. Which is fair enough, in the face of communication errors. Only ... Maui doesn't speak to anything except the Torque server. Which is running on the same host.

So what's actually happening here is that Torque can't talk to some node or other, and reporting that to Maui, which is then breaking. It didn't seem right that a communication failure to a single node once should stop jobs from starting elsewhere, which prompted some deeper investigation.

Looking for obvious correlations, we noticed that the scheduling blips happened right when we're running lots of analysis jobs - exactly when we don't want scheduler blips! However, it wasn't an obvious correlation, in that sometimes running 1000 jobs at once was fine, other times 400 caused things to gum up.

More worry-some than sub-optimal scheduling was that during the same time period we got occasional errors from the CE's, of the form:

BLAH error: submission command failed (exit code = 1) 

(stdout:) 

(stderr:pbs_iff: cannot read reply from 

pbs_server-No Permission.-qsub: 

cannot connect to server svr016.gla.scotgrid.ac.uk 

(errno=15007) Unauthorized Request

Dissecting that down, the BLAH part is CREAM saying it can't submit the job, so we're looking at the pbs_iff part. The purpose of pbs_iff is to authenticate the current user to the Torque server, so that the job is run with the correct user id (and can be checked with the ACL's on the server, if appropriate). The next part with qsub is just reporting that it's not able to talk to the server.

The root problem is pbs_iff not able to communicate, after which the rest of the qsub is failing for lack of authentication. This is a problem, because these are jobs that are already accepted by the CREAM CE, and shouldn't be failed here. (If a site can't cope with the jobs, the CE should be disabled, so it never accepts the jobs - that's the signal to the submitter/WMS to try elsewhere.)

How does all this link back to the network issues? Well, our cluster is split into two rooms - liked by a couple of fibres.

During analysis, we can see 2 GB per second (yes, that's in bytes) in traffic leaving the disk servers. Roughly half the disk and about half of the CPUs [see later!] are in each room; that implies that given a random distribution half that traffic has to pass through the fibre link.

And, yep, that's the problem right there. The Torque server unable to shout loud enough to talk to the nodes when the link is full, or be heard from some of the CE's. Digging into the stats shows that the link is running at 83% average utilisation, over the past month. So when analysis hits, it wipes out any other traffic.

For the moment, then, I've put a cap on the number of analysis jobs until we can resolve this, as mitigation. And sent Mark off to find some more fibre and ports on the switches!

Some interesting sums: Turns out we have nearer 1/3 the CPU upstairs, and 2/3 (1200 job slots) downstairs. Disk is close to 1/2 each. Matching this up with the planning number of 5 MB per second 'disk spindle to analysis cpu' bandwidth suggests that we need 3 GB per second, or 24 Gbs-1 bandwidth between the rooms to run at full capacity. Compared to 10 Gbs-1 at the moment.

Hrm. No wonder we were having difficulty! On the other hand, it's probably been this link that's the limiting factor in our analysis throughput, so we should be able to roughly double our peak throughput of analysis jobs once that link is upgraded.

That, and not have the scheduler taking a wee nap during peak times.

Side Effects may include ...

On Wednesday, the 25th, the Glasgow Scotgrid site was part of the wider SSC5 Security Challenge and during the course of the challenge we encountered several issues with the network security configuration on our core switch.

The configuration changes which caused issued are specifically:
1) Access List Configuration for inbound services
2) ICMP dos-control settings

The Access List Configuration (ACL) did not accept a global default permit with a wild card mask for both IP address ranges and subnets. The key issue here is that when the Access List was applied  on an access port for inbound traffic the Access List worked correctly. However, when applied to the primary egress port onto our network switch it disabled remote connectivity into the cluster, while not impacting internal  machine to machine traffic on the cluster.  The access list was removed and remote access was restored. The root cause for this failure was traced to an incorrectly set ACL ANY permit within the list, however on further investigation each network requiring access to and from the cluster will require its own unique entry rather than a default network range with a series of denied services.  The central IT group at the University also run a series of access lists and fire walls within the edge routing and switching network to the JANET environment which can be adapted to fit our requirements within the cluster setup at Glasgow.

A secondary issue;

A dos-control setting which controls the maximum payload for ICMP also caused unusual network behaviour after it was implemented. Effectively by limiting the payload to 512 bytes, this caused Maui and Torque to encounter issues when attempting to communicate with one another which then impacted other services within the cluster environment, while this slowed down Torque and Maui it did not completely stop the cluster, however its removal immediately improved data connectivity within the cluster. This issue is being referred back to the manufacturer as the payload incrementation only increases to 1023 bytes presently.

Once we have an update on this issue we will post it up on the blog.

The CE is dead. Long live the CE. Nos paenitet incommodo

As part of the on-going developments to the Scot Grid cluster at Glasgow, we have decommissioned our final LCG-CE, which resided on SVR021. The removal of this CE allows us to concentrate the support and development of two CE platforms; Cream and ARC. We are planning to conduct a series of tests around the three CREAM CE's we have deployed at Glasgow in an attempt to gain a better understanding of their maximum loading potential for running jobs and how to tweak them to gain the maximum efficiency from this service.

Additionally, we will be testing our availability metrics over the next month as the LCG-CE was one of the corner stones of Steve Lloyd's tests of our overall availability. This will now be monitored primarily through our SRM availability.

The reasons for decommissioning the LCG-CE are that we would be removing it at some point in the near future, all the big VO's do not have issues with submitting to Cream CEs and it simplifies our internal support requirements.

The new servers running Cream are svr008, svr014 and svr026.

Thank you LCG-CE and goodnight.

My God; it's full of data-transfers!

The Great ATLAS Spacetoken Migration of 2011 kicked off yesterday evening, and with 47TB of data sitting in MCDISK at Glasgow, Brian and We decided to take the opportunity to see how fast we could push it across to DATADISK.
So, since ATLAS Data Management on this case happens over FTS (even though the vast majority of the transfers are internal to a site), we turned up the number of slots for STAR-GLASGOW a bit, from 20 (our default) to 50 (which was fun) up to 80 (although we peaked at around 65 used).
With effectively no limit from FTS, our data rates were... impressive. Although it's an unfair comparison (everyone else was limited by FTS, and we were mostly moving things over the internal network), we managed to hit a peak transfer rate of 1.5GB/s internally (yes, that's 12Gbit/s), and sustain at around 8Gbits/s. That equated to around 2/3s of the total UK data movement over STAR channels, or roughly 2/3s of ATLAS's total traffic in this migration. At that rate, none of our disk servers were stressed, and the network switches were intensely relaxed.

Some exciting graphs follow:

A baffling spot of localised cooling

How do you keep your cool in this sort of weather? Well, there's various options, but I'll bet one you've not tried is wrapping up in lots of insuating foam.

And yet, that's been just the ticket for some worker nodes up here; despite it being one of the warmer days (23° C outside). Have a look at the temperature graph, and see if you can spot when something changed:
(The peak at midnight was due to a sneak attack Hammercloud; it was just before 12 when I put in the insulation.)

I'd discovered that there's some empty head space at the top of the racks. In those racks were there's a network switch at the top, this wasn't doing much, but where there were worker nodes, the top node was a lot hotter than the node two down from it. That's a lot sharper change than I'd expected - it was noticeable by touching the metal cover on the front of the nodes. The theory was that hot air out the back of the nodes was being sucked forward over the top of the highest node (through the headspace), and then recirculated round, getting hotter, until the steady state of it was about 5 K hotter that the others.

So, it was time to do something about that. First couple attempts at stopping up the gap didn't have much effect, until I dug out a few bits of packing foam (that the nodes were shipped in). Being, of course, the correct width, and jut a bit taller than 1U, they fit snugly into the headspace.


And that foam baffle reduced the temperature; to the point that the node at the top of the racks are now at the lowest temperature since records began! (i.e. they were installed.) Counter intuitive, but that's the way air/heat flow goes sometimes.

Although these worker nodes are due for replacement, we're going to be reusing the racks themselves, so little things like this are good to know. It may be that this won't be a problem with the new worker nodes - or it might be the case that it'd be worse. Either way, forewarns is fore armed (and cooler).

So long and thanks for all the fish

I would just like to say thanks to everyone who I have worked with at ScotGrid, GridPP and EGEE. I couldn't have picked a better time to be working on grid, LCG and WLCG. I have learned a lot, accomplished most of the things I set out to do and hopefully contributed to the project in some small way. I will always be on the other end of an email should you wish to get in touch. So long and thanks for all the fish.

gLite Virtual Box Image takes off

A new user today was looking for the download of our pre-built UI Virtual box image for usage with two VOS: vo.iscpif.fr and vo.complex-systems.eu

The ISC-PIF (Institut des Systèmes Complexes, Paris Île-de-France) is a multidisciplinary research and training center promoting the development of French, European and international strategic projects on complex adaptive systems, construed as large networks of elements interacting locally and creating macroscopic collective behaviour.

Hopefully we will get some feedback on the image and any improvements that could be made.

SGE and Lustre

On my list of things to do was install (Sun/Oracle) Grid Engine and get a CREAM CE submitting to it on my development cluster. So far I have SGE installed and running qsub jobs. I am documenting the experience for those who are interested here. I have opted for Lustre rather than NFS 3 as it is painfully ill-equipped for the task and we have a test Lustre instance to play with so why not go the whole hog.

the positives ...
1. The wealth of documentation on the Oracle page.
2. The interactive install is very easy to do.

and the negatives ...
1. The rpms default install location is /gridware & I can't seem to get my yum repo to use a --prefix like option. Something you can do with rpm. Ideas welcome?
2. The automatic install scripts having no debugging on them at all. When they fail they just fail silently with no output or logs. I have only managed an interactive install so far but I will try to set /bin/sh -x and see if it makes a difference. Hopefully I can get the automatic script running so that cfengine can deal with the install of the execution hosts rather than by hand.

The return of LHCb at Glasgow

After weeks of investigating and debugging our LHCb transfer issue at Glasgow we have finally fixed it. So ..... spill the beans I hear you cry.

Well in short, we had an iptables rule on the INPUT filter of the NAT that was dropping strangely behaving gridftp connections. This was relaxed and allowed inbound connections to be established. This has solved the issue and we still have the protection of the campus firewall for security.

Strangely behaving gridftp connections, what does that mean? Well, transfers that had failed to work first time seemed to get into an unknown state and transfer no bytes, with many RETRY packets and no FIN packet. It appears that these connections were trying to establish inbound connections. These were then dropped by a REJECT within our iptables.

Moral of the story is, if you can get external IP's for your worker nodes, use them. NAT'ing just adds complexity especially when dealing with GLOBUS.

The full story if you are interested ....

Problem: LHCb don't use FTS. They use direct outbound gridftp transfers of job outputs. Jobs on WN's transfer results, using the lcg-utils tools, at the end of the job to CERN and failover to various T1's if there is an issue with the CERN transfer. LHCb have seen a large failure rate with around 50% of gridftp/lcg-cp transfers failing at Glasgow. Brunel, Sheffield and Lancaster have been affected with the same issue although to a lesser extent. Failure rates at the other sites are much less at around 2-3%. We see the initial transfer timing out, failing over to a T1, this sometimes works and sometimes fails over to another T1 and so on. Why has this not been seen sooner? Well this has actually been there since day dot but DIRAC masked the return code of the failure. A new version of DIRAC catches the fail-overs and are killed by their watchdog. Thus bringing this issue to the surface.

Investigation: Glasgow looks like this WN's-> NAT->CAMPUS FIREWALL->WORLD. We managed to recreate the issue with a simple transfer test from varying amounts of WN's to test SRM end-points. This recreated the issue and we saw a 50% failure rate across various SRM implementations, in particular CASTOR, DCACHE, STORM. However, DPM transfers were 100% successful. Failed transfers manifested themselves are lcg-cp: timed out or lcg-cp: error on send. We repeated these tests using various VO's and got similar results so we did not think it was VO related. We monitored the connections though our NAT and asked the firewall team to check if any outbound ports were blocked, they were not. The GLOBUS_TCP_PORT_RANGE at Glasgow was set to a specific known open port range for inbound connections but this does not matter in this case of outbound connections. To be on the safe side we set GLOBUS_TCP_SOURCE_RANGE for outbound connections through our NAT. As we expected this did not make a difference. After discussion with other sites we checked client libraries, OS and network. One thing that did crop up was the use of NAT.

The final test was 100 simultaneous transfers from one node via the NAT. We saw a 50% failure rate. We repeated this test but this time with an external address and no NAT routing. This was 100% successful over 3 attempts. Quickly repeated tests did show some failures but this was probably the firewall dropping connections. Therefore, we were able to clearly identify the NAT as being the issue. We tried tweaking TCP settings on the NAT i.e. tcp_fin_timeout, tcp_tw_reuse, tcp_tw_recycle, tcp_keepalive_time with no success. The iptables rules themselves seemed sensible but we were still dropping 50% of the connections.

We then moved to tcpdumping the tcp packets (SYN and FIN) from the internal (eth0) device and compared it to a tcpdump of the external (eth1) device. You could clearly see the control channels opening, data channels opening, transfers and then around 50% of the transfers sending retry packets and never sending a FIN. It looked like something was being blocked.

A closer look at the iptables rules identified an entry on the INPUT filter that could be the culprit. Further up the chain we were allowing RELATED,ESTABLISHED as you would expect. Then we had a -A INPUT -i eth1 -p tcp -m tcp -j REJECT --reject-with tcp-reset. It appears this entry caused attempts to re-establish the connection to fail (possibly by blocking the initial packet from the destination, erroneously considering it not to count as ESTABLISHED any more). Very strange behaviour indeed. In the plus side we generally use the campus firewall to protect us from unwanted traffic rather than our own iptables rules, so we have relaxed the INPUT filter and guess what, near 100% transfer success.

CREAM thickens

CREAM at Glasgow has been upgraded to the latest glite3.2 release 3.2.5-0.sl5 (or INFN version 1.6). This brings lots of enhancements like

self limiting behaviour a'la WMS
A new proxy purger to clean the delegationdb and from the file system the expired proxies
a new way to customize the job wrapper
an improved proxy renewal mechanism
and one of my favourites, support for ISB/OSB transfers from/to gridftp servers run using user credentials rather than server host certificates. This will work well with users running gridftp servers on their own machines for example (as long as they don't turn them off when they go home at night!)

you can find out more about them all here. Now back to draining the WMS.

batch migrations

A week ago we finally migrated our batch system to better hardware. This had been on the cards for a while but was expedited as we need to start a series of server moves from an old rack which will be removed come the installation of our new kit. We also took this opportunity to upgrade the pbs server version inline with our mom version which was a little out of step. If that wasn't enough things changing at once we also built the latest MAUI 3.3 to test how it performs. So far so good.

Next up, will be the two WMS. Both will be put in downtime, drained and then moved out of the old rack.

serious multi core

who would have thought it possible ....



well with an large SSD, 24 cores, file stager analysis and it survives! More soon on our testing with some cutting edge equipment.

The User Forum in Uppsala continues with lots of interesting talks today. More user focussed today with sessions from Bioinformatics, Earth Science and Computational Chemistry. Again the buzz words of cloud, EC2, Eucalyptus and Open Nebula continue to mentioned during the Novel Technologies and Architectures sessions.


The cathedral in Uppsala.

Take my outputs, damn you...

We recently ran up a very large backlog of production output files waiting to go from Glasgow back to the Tier-1 (reminder, panda doesn't consider a job finished until the outputs are safely stored at the T1). This is clearly seen in the red line on the panglia plot above, which reaches very high values. As we recently cut the timeout for the UK cloud to 2 days for transferring jobs, to improve the responsiveness of the production system, we started to leak out failed jobs (light green line) as panda gave up and decided to rerun.

Fortunately we got a big boost in the number of FTS slots from Glasgow to RAL, increasing from 10 to 25 active transfers (see the bottom FTS monitoring plot). Even so it clearly takes 24 hours for all the backlogs to drain down.

One of the problems here is that the output files are small from simulation (a tiny log file and a 20-50MB HITS file), so the overheads of FTS + SRM are very considerable and the actual bandwidth achieved is quite low. One possibility we are considering in ATLAS is introducing a pre-merge of outputs on the T2, which will allow us to send much bigger files back to the T1 (although a final "super-merge" will probably still be necessary). For this we are waiting for the generic Athena merge transform and then we will need to test integrating this into the mainline production workflow.

Until then we just have to take the operational load of tweaking the FTS settings when necessary.