I had to intervene at Edinburgh 2 weeks ago (14th, just before I went to London for the T2 review). They had been failing JS since the Friday night. Logging on I could see a stack of ops jobs, but nothing running on several WNs.
I tried starting the oldest ops job using runjob -cx, but that didn't work, giving the error:
04/14/2007 16:54:30;0080;PBS_Server;Req;req_reject;Reject reply code=15057(Cannot execute at specified host because of checkpoint or stagein files), aux=0, type=RunJob, from root@ce.epcc.ed.ac.uk
Not at all clear to me what was going on. I tried running different ops jobs and they all started and ran properly, so in the end I deleted that job from the queue and that seemed to ungunge things.
torque seems to produce rather unhelpful information in these sort of cases, unless I'm just looking in the wrong places.
Showing posts with label support. Show all posts
Showing posts with label support. Show all posts
Monday, April 30, 2007
Wednesday, April 11, 2007
Edinburgh storage woes
Edinburgh has been suffering lately from a number of failures in the SAM replica management tests. Looking at the dCache webpages reveals the reason:
http://srm.epcc.ed.ac.uk:2288/usageInfo
Rather than displaying the usual pool usage information, the table contained entries like this:
pool1_04 pool1Domain [99] Repository got lost
The "Repository got lost" error can be explained as follows. The dCache periodically runs a background process which attempts to write a small test file onto each pool in an attempt to check that it is still operational. If this fails then the above error message will be generated. According to the dCache developers this will occur if there is a filesystem problem or a disk is not responding quickly enough.
What is strange, however, is that the recent problems have resulted in all of our dCache pools as being marked with the above error. It seems strange that the same filesystem or disk issue would simultaneously affect all of the pools. I have submitted a ticket to dCache support in an attempt to get more information.
The problem can eaily be fixed by restarting the pool process on the affected dCache pool node. In the above case it is pool1.epcc.ed.ac.uk since the pool in question is named pool1_04.
service dcache-pool restart
I have started to add material to this page
http://www.gridpp.ac.uk/wiki/Edinburgh_dCache_troubleshooting
to desribe our dCache setup in more detail. This should give people a better idea of how a working system should be configured. I will add more information when I have time.
http://srm.epcc.ed.ac.uk:2288/usageInfo
Rather than displaying the usual pool usage information, the table contained entries like this:
pool1_04 pool1Domain [99] Repository got lost
The "Repository got lost" error can be explained as follows. The dCache periodically runs a background process which attempts to write a small test file onto each pool in an attempt to check that it is still operational. If this fails then the above error message will be generated. According to the dCache developers this will occur if there is a filesystem problem or a disk is not responding quickly enough.
What is strange, however, is that the recent problems have resulted in all of our dCache pools as being marked with the above error. It seems strange that the same filesystem or disk issue would simultaneously affect all of the pools. I have submitted a ticket to dCache support in an attempt to get more information.
The problem can eaily be fixed by restarting the pool process on the affected dCache pool node. In the above case it is pool1.epcc.ed.ac.uk since the pool in question is named pool1_04.
service dcache-pool restart
I have started to add material to this page
http://www.gridpp.ac.uk/wiki/Edinburgh_dCache_troubleshooting
to desribe our dCache setup in more detail. This should give people a better idea of how a working system should be configured. I will add more information when I have time.
Subscribe to:
Posts (Atom)