Farewell ScotGrid-Edinburgh!

Not withstanding the problems this week at ECDF, in general the new Edinburgh resource has been working well. Consequently maintaining the old site was just a drag on our people, with no real purpose anymore. Sam broadcast the intention to close the site a couple of weeks ago and we changed its status in the GOC to "closed".

Greig and Sam have ensured that the site's old storage is accessible through ECDF, though we've advised VOs to move their data off this SE as the hardware is ageing and unreliable.

Edinburgh is dead! Long live Edinburgh!

Glasgow to Edinburgh Lightpath Approved

After messing about for ages with application forms and revised procedures, our application, when finally submitted to JANET, was approved within 24 hours! They are now conducting a technical feasibility study, but no news on how long that might take.

Update: JANET people say this study will take "the shorter end of 'a few weeks'", which is good news.

Edinburgh and ATLAS FCR

Edinburgh got bumped by ATLAS for failing JS for quite a while, but as their 5 jobs slots were tied up with Zeus jobs the ATLAS SAM tests couldn't run so they couldn't get back in the ATLAS BDII. Of course this meant that Steve Lloyd's jobs were aborting (no matching resources).

So, Sam has allowed the atlassgm user access to the reserved ops/dteam job slot. This will be fine as long as nothing happens to stick that job in the queue.

However, it's all getting a bit silly - more and more tweaks in the batch system to rush through the test jobs.

Of course, to some extent it's an Edinburgh problem - they are so strapped for CPU that getting anything to run through in a guaranteed time is hard.

Intervention At Edinburgh

I had to intervene at Edinburgh 2 weeks ago (14th, just before I went to London for the T2 review). They had been failing JS since the Friday night. Logging on I could see a stack of ops jobs, but nothing running on several WNs.

I tried starting the oldest ops job using runjob -cx, but that didn't work, giving the error:

04/14/2007 16:54:30;0080;PBS_Server;Req;req_reject;Reject reply code=15057(Cannot execute at specified host because of checkpoint or stagein files), aux=0, type=RunJob, from root@ce.epcc.ed.ac.uk

Not at all clear to me what was going on. I tried running different ops jobs and they all started and ran properly, so in the end I deleted that job from the queue and that seemed to ungunge things.

torque seems to produce rather unhelpful information in these sort of cases, unless I'm just looking in the wrong places.

Edinburgh finally got pushed over the edge by the LHCb jobs - the CE became unstable and the BDII failed.

The GGUS ticket is https://gus.fzk.de/pages/ticket_details.php?ticket=11192&from=allt.

There's no evidence that Edinburgh ever published an annomolously low ERT which would have sucked these jobs in, so I think it must have been an LHCb problem: