We got a ticket on Tuesday because an ATLAS user couldn't get their files back from one disk server which had run out of its 600 (!) ATLAS mapped pool accounts.
I did a bit of a hacky clean up, but actually this is very safe because, unlike a CE, there are no files involved to be mis-inherited by a subsequent user. The only issue would occur at the very moment that a user tried to transfer files.
The clean up removed the oldest mappings, and even the busiest server was down to ~150 mappings and ~450 free slots, so adequate breathing room was gained.
Sam is going to think about this in the storage group concept and write a more general tidier-upper for all VOs.
Showing posts with label DPM. Show all posts
Showing posts with label DPM. Show all posts
Thursday, November 25, 2010
Thursday, February 12, 2009
HammerCloud 135---A Load Shared is a Load Halved, to a point.
We performed our splitting of the DPM across two hosts just in time for the most recent HammerCloud test on UK sites:
http://gangarobot.cern.ch/st/test_135/
So, we already have some metrics to compare the old arrangement with the new.
For reference, Graeme blogged about the last big HammerCloud UK test here, where we were getting an event rate of around 10Hz, at the cost of the DPM head node running at an unsustainable load.
Since then, a couple of Hammerclouds have come by, generally coincident with ATLAS production and other stresses on the DPM, and it has just utterly failed to cope.
After our surgery, we did a lot better:
with an event rate of about 14 Hz, a 50% improvement, almost.
and, the load on the DPM head node was very much more acceptable, given the increased power of the hardware:
However, we're still not close to maxing out the pool nodes:
probably because we've hit another, higher, performance bottleneck on the new svr015 "MySQL server" machine:
that orangish stuff is the CPU in I/O Wait state, waiting for seeks within the DB.
We're currently looking at ways of tuning MySQL, or the disk, to improve this performance, since it looks like there's another 30 to 40% of performance there, at least.
Some ideas we've had include splitting the dpm_db and cns_db across different filesystems (since they have very different access patterns for this kind of use), tweaking MySQL settings (although they look generally fine...), or even getting Faster Disks. Roll on solid state drives, we say!
http://gangarobot.cern.ch/st/test_135/
So, we already have some metrics to compare the old arrangement with the new.
For reference, Graeme blogged about the last big HammerCloud UK test here, where we were getting an event rate of around 10Hz, at the cost of the DPM head node running at an unsustainable load.
Since then, a couple of Hammerclouds have come by, generally coincident with ATLAS production and other stresses on the DPM, and it has just utterly failed to cope.
After our surgery, we did a lot better:
with an event rate of about 14 Hz, a 50% improvement, almost.
and, the load on the DPM head node was very much more acceptable, given the increased power of the hardware:
However, we're still not close to maxing out the pool nodes:
probably because we've hit another, higher, performance bottleneck on the new svr015 "MySQL server" machine:
that orangish stuff is the CPU in I/O Wait state, waiting for seeks within the DB.
We're currently looking at ways of tuning MySQL, or the disk, to improve this performance, since it looks like there's another 30 to 40% of performance there, at least.
Some ideas we've had include splitting the dpm_db and cns_db across different filesystems (since they have very different access patterns for this kind of use), tweaking MySQL settings (although they look generally fine...), or even getting Faster Disks. Roll on solid state drives, we say!
DPM improvements!
Ever since ATLAS analysis has been enabled at Tier 2 sites (and the relevant sheaves of AOD files have arrived at our DPM), the Glasgow DPM has been looking increasingly strained.
This first became obvious during the HammerCloud tests for analysis in December, but over January it became increasingly clear that the access patterns of normal analysis jobs, en-mass, are quite enough to make the storage unreliable for other users.
In particular, we had one period where chunks of ATLAS production work died because the DPM was so overloaded.
Looking at the DPM during these periods, it looked like it was a combination of I/O waits and, more significantly, the dpm and srmv2.2 daemons maxing out the CPU.
Last Friday, we tried "optimising" the DPM MySQL backend by taking the dpm offline, and then exporting, dropping, and reimporting the dpm_db and cns_db databases. The InnoDB engine has an issue that it sometimes becomes fragmented, increasing the size of the physical DB file and reducing performance - reimporting from a logical backup usually reduces this fragmentation in the restored DB.
Unfortunately, this reimporting process took far longer than we anticipated---on the order of 5 hours!---and, in the end, resulted in a distinctly unimpressive 10% size reduction in the physical DB.
After bringing things back up again, however, it became clear that the performance hadn't changed much, and that it was most likely that we just needed to give the DPM processes more room to breathe.
Our DPM is considerably underspecced compared to our new worker nodes (which are lovely 8-core machines, at higher clock rates), but, of course, has the benefit of RAIDed storage to give our DB a bit more reliability. So, we decided to take the big step of splitting the DPM across two nodes - the old DPM being moved to a role as "MySQL backend server", and the "new" DPM being a repurposed worker node hosting all the DPM services.
Thanks to cfengine, and the arcane workings of YPF, it isn't too hard to make a node into any other kind of node that we want---the tricky bit, in this case, is swapping the hostnames, so that the "new" DPM still gets to be svr018, while the old DPM gets moved to svr015 (and also hosts our DPM monitoring stuff now).
The new svr018 used to be node310 - the last node in our pool of new worker nodes - which I'd previously taken offline and allowed to drain over the weekend in anticipation of this.
However, thanks to some synchronized administration by Mike and myself, things seemed to go relatively smoothly with the move on Monday, with only an hour of downtime and barely a failed job in sight, despite being full of ATLAS production at the time.
It looks like this also improved our HammerCloud performance, about which more in a later post.
This first became obvious during the HammerCloud tests for analysis in December, but over January it became increasingly clear that the access patterns of normal analysis jobs, en-mass, are quite enough to make the storage unreliable for other users.
In particular, we had one period where chunks of ATLAS production work died because the DPM was so overloaded.
Looking at the DPM during these periods, it looked like it was a combination of I/O waits and, more significantly, the dpm and srmv2.2 daemons maxing out the CPU.
Last Friday, we tried "optimising" the DPM MySQL backend by taking the dpm offline, and then exporting, dropping, and reimporting the dpm_db and cns_db databases. The InnoDB engine has an issue that it sometimes becomes fragmented, increasing the size of the physical DB file and reducing performance - reimporting from a logical backup usually reduces this fragmentation in the restored DB.
Unfortunately, this reimporting process took far longer than we anticipated---on the order of 5 hours!---and, in the end, resulted in a distinctly unimpressive 10% size reduction in the physical DB.
After bringing things back up again, however, it became clear that the performance hadn't changed much, and that it was most likely that we just needed to give the DPM processes more room to breathe.
Our DPM is considerably underspecced compared to our new worker nodes (which are lovely 8-core machines, at higher clock rates), but, of course, has the benefit of RAIDed storage to give our DB a bit more reliability. So, we decided to take the big step of splitting the DPM across two nodes - the old DPM being moved to a role as "MySQL backend server", and the "new" DPM being a repurposed worker node hosting all the DPM services.
Thanks to cfengine, and the arcane workings of YPF, it isn't too hard to make a node into any other kind of node that we want---the tricky bit, in this case, is swapping the hostnames, so that the "new" DPM still gets to be svr018, while the old DPM gets moved to svr015 (and also hosts our DPM monitoring stuff now).
The new svr018 used to be node310 - the last node in our pool of new worker nodes - which I'd previously taken offline and allowed to drain over the weekend in anticipation of this.
However, thanks to some synchronized administration by Mike and myself, things seemed to go relatively smoothly with the move on Monday, with only an hour of downtime and barely a failed job in sight, despite being full of ATLAS production at the time.
It looks like this also improved our HammerCloud performance, about which more in a later post.
Wednesday, December 17, 2008
Analysis Challenge: Round 4
We re-run the analysis challenge yesterday with a better MySQL setup so that the higher number of dpns daemons could get db connections. However, the results were much the same as before and the conclusion seems to be that X509 sucks - it's killing the headnode with all of the simultaneous authenticati
ons.
We hope to prove later on that this is the r
eal problem and then think about what we can do about it...
ons.
We hope to prove later on that this is the r
eal problem and then think about what we can do about it...
Wednesday, December 10, 2008
Plots from last analyais challenge
Mostly confirmed the results which we saw at the end of last week's test. Load on the DPM headnode is our pressing concern - it's maxing its CPU out even at open rates of a little over 1Hz.
Monday, December 08, 2008
Analysis Challenge: Round 3
Last week's analysis challenge at Glasgow showed extreme load and sluggishness in the DPM (see the attached plots of awfulness). Although we managed a much better event rate we also suffered from incomplete processing and the DPM was a clear bottleneck.
I had a chat with JPB today who spotted the very high memory consumption of the dpm daemon - he thinks there's probably a memory leak and that this might be slowing things down. He also said it might be worth running more dpns daemons as these also do connection athentication.
So, to get ready for tomorrow I have:
- Allowed core dumps for the DPM and DPNS daemons.
- Increased the number of threads in the DPNS daemon to 60.
- Restarted all the daemons.
If we still see problems tomorrow then at least we should have some good information for the developers to chew on.
Tuesday, October 21, 2008
"Oh no! Not again..."
After being all enthusiastic that the gSOAP errors had been nailed, we failed two SE tests in the last 24 hours. Exactly the same issue as before.
As this error message is so vague it looks like lcg-rollout is our only hope.
I note in passing that Glasgow has one of the most reliable SEs in the UK for ATLAS (2.1% job loss, only beaten by Oxford who have 0.8%; UK average in Q3 was 8% loss) so this is particularly galling.
Shouldn't we be making the results as seen by our real customers rather more important than a once an hour stab in the dark from ops?
As this error message is so vague it looks like lcg-rollout is our only hope.
I note in passing that Glasgow has one of the most reliable SEs in the UK for ATLAS (2.1% job loss, only beaten by Oxford who have 0.8%; UK average in Q3 was 8% loss) so this is particularly galling.
Shouldn't we be making the results as seen by our real customers rather more important than a once an hour stab in the dark from ops?
Saturday, October 11, 2008
Well Done Guys!
Well, I was waiting for Mike and Andrew to blog this, but they haven't. They very successfully upgraded Glasgow's DPM to the native 64bit version on Monday last week (when we had upgraded to SL4 only the 32 bit version was available). This was a significant step forwards but required the head node and all of the disk servers to have their OS rebuilt without losing data, and the database restored onto the head node.
It went very well and we were up and running again within 6 hours - no data lost!
We are also seeing an improvement in the SAM test results, with the spurious 'gSOAP' errors which were plaguing us now seemingly having gone (fingers crossed!).
It's terrible that the LHC is not running right now, but it does mean that interventions like this can be done.
Great work guys!
It went very well and we were up and running again within 6 hours - no data lost!
We are also seeing an improvement in the SAM test results, with the spurious 'gSOAP' errors which were plaguing us now seemingly having gone (fingers crossed!).
It's terrible that the LHC is not running right now, but it does mean that interventions like this can be done.
Great work guys!
Wednesday, July 23, 2008
DPM Upgrade
Mike and I upgraded the DPM yesterday to 1.6.10. There are no schema changes in 1.6.7->1.6.10, so the upgrade involves 'downtime' of about 20s. We didn't put this into the CIC portal, but I suppose in retrospect we should have declared an 'at risk' period.
We're still running the i386 version of DPM (on top of x86_64). At some point it would be desirable to upgrade to x86_64; however, as i386 works just fine and this will involve real downtime, there is no urgent pressure to do so.
Greig has noted that dpm-updatespace in 1.6.10 has a bug in it: https://gus.fzk.de/pages/ticket_details.php?ticket=38330.
We're still running the i386 version of DPM (on top of x86_64). At some point it would be desirable to upgrade to x86_64; however, as i386 works just fine and this will involve real downtime, there is no urgent pressure to do so.
Greig has noted that dpm-updatespace in 1.6.10 has a bug in it: https://gus.fzk.de/pages/ticket_details.php?ticket=38330.
Monday, June 23, 2008
Durham SE Issues
Durham suffered a complete SE failure last week. A RAID card failure took down the old SE, gallows, and then an LVM metadata corruption took out the new disk server on se01.
The list of lost ATLAS files has been reported (https://savannah.cern.ch/bugs/?38037) and we're waiting for the catalog to be cleaned up to restart production here (well, when there are any jobs to run).
We took the opportunity to retire gallows and now se01 is the sole SE at Durham. It should suffice for ATLAS production where we only need a few TB cache anyway.
In the meantime there was a power outage in the Durham machine room over the weekend. David had to get the university to reset some breakers but things seem to be running well now.
The list of lost ATLAS files has been reported (https://savannah.cern.ch/bugs/?38037) and we're waiting for the catalog to be cleaned up to restart production here (well, when there are any jobs to run).
We took the opportunity to retire gallows and now se01 is the sole SE at Durham. It should suffice for ATLAS production where we only need a few TB cache anyway.
In the meantime there was a power outage in the Durham machine room over the weekend. David had to get the university to reset some breakers but things seem to be running well now.
Saturday, June 14, 2008
Publish and be damned...
It all went pear shaped yesterday as information publishing fell over on the SE. It seems when I quickly "fixed" the DPM information provider script to get the correct hostname I forgot to chomp() the output from "hostname -f". So the hostname variable had a trailing newline which corrupted the information system. The BDII logs started to throw errors like "First line of LDIF entry does not begin with 'dn:' at /opt/glite/libexec/glite-info-generic line 17".
Unfortunately the BDII then considered the whole of the SE information package corrupt (rather than just that provider's output) and our SE promptly dissappeared from the information system with the attendent RM test failures.
This situation then persisted for most of the day until Andrew noticed it "by eye". So we had another failure - nagios didn't send an alarm properly when we started to fail. If that had happened it would have been fixed in a hour, but instead we were failing for 8 hours.
From the dizzy heights of SAM perfection we fell to 98% for the month, 95% for the week. It wasn't quite hubris, but it was ironic that I was blogging about Glasgow's reliability at the very moment we were broken.
At the moment I have removed the info provider for tokens, and I will more carefully put it back on Tuesday.
Unfortunately the BDII then considered the whole of the SE information package corrupt (rather than just that provider's output) and our SE promptly dissappeared from the information system with the attendent RM test failures.
This situation then persisted for most of the day until Andrew noticed it "by eye". So we had another failure - nagios didn't send an alarm properly when we started to fail. If that had happened it would have been fixed in a hour, but instead we were failing for 8 hours.
From the dizzy heights of SAM perfection we fell to 98% for the month, 95% for the week. It wasn't quite hubris, but it was ironic that I was blogging about Glasgow's reliability at the very moment we were broken.
At the moment I have removed the info provider for tokens, and I will more carefully put it back on Tuesday.
Friday, June 13, 2008
disinformation
We just got ticketed for a failing SE sam test. Most odd as Steve Lloyds SAM results were all green. Re-read the ticket and it turns out we were publishing info for svr018.beowulf.cluster rather than the external interface name. Despite this being noted before Graeme hacked it around and raised a Savannah Ticket
Friday, April 25, 2008
#include <documentation.h>
I quote from the DPM Developer Documentation
so, with that in mind, I set about pulling out the number of SRM 2.2 requests vs the no of SRMv1 requests at the site. v1 should be constant (what with all the new users coming onboard) and SRM 2.2 being a rapid increase since we enabled it? well it's not easy to grep from the logs so I thought I'd poke the DB. First off in dpm_db.dpm_req r_type a char(1) field normally has g (get?) and p (put?) but we have just over 1500 rows where type is 'B' (broken?). hmm - all from flavia's DN and clienthost of lxdev25
my plots of the dpm usage are far too spikey to make sense of at the moment, but I'll work on presenting the info a bit clearer.
In the meantime I discovered that it's pretty obvious when we set torque to fill the jobslots in host order (made it easier to drain nodes off) and when we send nodes away to vendors.
LFC/DPM Database schema
TO DO : describe non straight forward tables/fields....
so, with that in mind, I set about pulling out the number of SRM 2.2 requests vs the no of SRMv1 requests at the site. v1 should be constant (what with all the new users coming onboard) and SRM 2.2 being a rapid increase since we enabled it? well it's not easy to grep from the logs so I thought I'd poke the DB. First off in dpm_db.dpm_req r_type a char(1) field normally has g (get?) and p (put?) but we have just over 1500 rows where type is 'B' (broken?). hmm - all from flavia's DN and clienthost of lxdev25
my plots of the dpm usage are far too spikey to make sense of at the moment, but I'll work on presenting the info a bit clearer.
In the meantime I discovered that it's pretty obvious when we set torque to fill the jobslots in host order (made it easier to drain nodes off) and when we send nodes away to vendors.
Friday, February 22, 2008
Acrobat ate my disk servers!
Glasgow is finally out of downtime. GS worked his grid-fu and managed to upgrade lots to SL4 - Admittedly some things (RGMA) weren't a goer. APEL Accounting could be broken for a while as we've now split the CE (new home = svr021) and the Torque server (still on svr016). My 'simple' job was to take care of the DPM servers...
Simple enough, we hacked into the YAIM site-info.def stuff and separated things out into services/ and vo.d/ - easy. Few gotchas as cfengine was once again reluctant to create the symlinks on the target nodes (however creating the symlinks on the master and replicating those works fine) which we thought might be fixed by an upgrade of cfengine from 2.1.22 to 2.2.3. Big mistake. it broke HostRange function of cfengine.
so we have
but cfengine complained that
now I'm not sure if this is due to the problem of short hostname vs FQDN - I've hit a similar issue when I want to copy iptables configs off -
Anyway, this all seems trivial compared to the hassle with the latest SLC 4X that got mirrored up to the servers overnight (the disk servers run SLC4 rather than SL4 as the Areca raid card drivers are compiled in) - dpm-queryconf kept failing with
The really frustrating thing was that the server that I installed from home while munching breakfast (all hail laptops and broadband) worked fine, but those I installed (and reinstalled) later in the office were broken. [hmm. is this a sign that I should stay at home in the mornings and have a leisurely breakfast?]
Puzzling was the fact that the broken servers had more rpms installed than the working ones. - I eventually resorted to installing strace on both boxes and diffing the output of 'strace dpm-qryconf'
the failing one had a big chunk of
whereas the working one didn't call this at all.
I was also bemused as to why acroread had been installed on the server and more annoyingly why I couldn't uninstall,
Yep - someone (step up to the podium Jan Iven) had mispackaged the SLC acroread 8.1.2 update...
Yep, thats right - RPM had decided that acroread was a dependency. Grr. Workaround - remirror the CERN SLC repo, (no they hadn't updated since), manually remove the offending rpm, and rebuild the metadata with 'createrepo'
Then make sure that the nodes were rebuilt and only ever looked to our local repository rather than the primary cern / dag ones. (thanks to kickstart %post and cfengine)
Finally we got yaim to *almost* run - it was failing on lcmaps and grid-mapfile creation (fixed by unsupporting a certain VO)
Easy fix in comparison. Anyway - DPM up n running and seems OK. Roll on the next SAM tests.... (or real users)
Phew.
Grid Middleware should not be this hard to install!
Simple enough, we hacked into the YAIM site-info.def stuff and separated things out into services/ and vo.d/ - easy. Few gotchas as cfengine was once again reluctant to create the symlinks on the target nodes (however creating the symlinks on the master and replicating those works fine) which we thought might be fixed by an upgrade of cfengine from 2.1.22 to 2.2.3. Big mistake. it broke HostRange function of cfengine.
so we have
dpmdisk = ( HostRange(disk,032-036) HostRange(disk,038-041) )
but cfengine complained that
SRDEBUG FuzzyHostParse(disk,032-041) succeeded for disk033.gla.scotgrid.ac.uk
SRDEBUG FuzzyHostMatch: split refhost=disk033.gla.scotgrid.ac.uk into refbase=disk033.gla.scotgrid.ac.uk and cmp=-1
SRDEBUG FuzzyHostMatch(disk,032-041,disk033.gla.scotgrid.ac.uk) failed
now I'm not sure if this is due to the problem of short hostname vs FQDN - I've hit a similar issue when I want to copy iptables configs off -
$(skel)/nat/etc/iptables.$(host) mode=0600 dest=/etc/iptables define=newiptables type=sumneeds iptables.host.gla.scotgrid.ac.uk not just iptables.host on the master repo.
Anyway, this all seems trivial compared to the hassle with the latest SLC 4X that got mirrored up to the servers overnight (the disk servers run SLC4 rather than SL4 as the Areca raid card drivers are compiled in) - dpm-queryconf kept failing with
send2nsd: NS002 - send error : No valid credential foundand yet the certificates were there and valid - openssl verify ... returned OK, ddates were valid, NTP installed etc. The dpm log showed
dpm_serv: Could not establish security context: _Csec_recv_token: Connection dropped by remote end !
The really frustrating thing was that the server that I installed from home while munching breakfast (all hail laptops and broadband) worked fine, but those I installed (and reinstalled) later in the office were broken. [hmm. is this a sign that I should stay at home in the mornings and have a leisurely breakfast?]
Puzzling was the fact that the broken servers had more rpms installed than the working ones. - I eventually resorted to installing strace on both boxes and diffing the output of 'strace dpm-qryconf'
the failing one had a big chunk of
open("/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/lib/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/i686", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/i686/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/i686", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/sse2/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/sse2", 0xffff9028) = -1 ENOENT (No such file or directory)
open("/usr/lib/libstdc++.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
whereas the working one didn't call this at all.
I was also bemused as to why acroread had been installed on the server and more annoyingly why I couldn't uninstall,
Yep - someone (step up to the podium Jan Iven) had mispackaged the SLC acroread 8.1.2 update...
rpm -qp ./acroread-8.1.2-1.slc4.i386.rpm --provides
warning: ./acroread-8.1.2-1.slc4.i386.rpm: V3 DSA signature: NOKEY, key ID 1d1e034b
2d.x3d
3difr.x3d
ADMPlugin.apl
Accessibility.api
AcroForm.api
Annots.api
DVA.api
DigSig.api
EFS.api
EScript.api
HLS.api
MakeAccessible.api
Multimedia.api
PDDom.api
PPKLite.api
ReadOutLoud.api
Real.mpp
SaveAsRTF.api
SearchFind.api
SendMail.api
Spelling.api
acroread-plugin = 8.1.2-1.slc4
checkers.api
drvOpenGL.x3d
drvSOFT.x3d
ewh.api
libACE.so
libACE.so(VERSION)
libACE.so.2.10
libACE.so.2.10(VERSION)
libAGM.so
libAGM.so(VERSION)
libAGM.so.4.16
libAGM.so.4.16(VERSION)
libAXE8SharedExpat.so
libAXE8SharedExpat.so
libAXE8SharedExpat.so(VERSION)
libAXSLE.so
libAXSLE.so
libAXSLE.so(VERSION)
libAXSLE.so(VERSION)
libAdobeXMP.so
libAdobeXMP.so
libAdobeXMP.so(VERSION)
libAdobeXMP.so(VERSION)
libBIB.so
libBIB.so(VERSION)
libBIB.so.1.2
libBIB.so.1.2(VERSION)
libBIBUtils.so
libBIBUtils.so(VERSION)
libBIBUtils.so.1.1
libBIBUtils.so.1.1(VERSION)
libCoolType.so
libCoolType.so(VERSION)
libCoolType.so.5.03
libCoolType.so.5.03(VERSION)
libJP2K.so
libJP2K.so
libJP2K.so(VERSION)
libResAccess.so
libResAccess.so(VERSION)
libResAccess.so.0.1
libWRServices.so
libWRServices.so(VERSION)
libWRServices.so.2.1
libadobelinguistic.so
libadobelinguistic.so
libadobelinguistic.so(VERSION)
libahclient.so
libahclient.so
libahclient.so(VERSION)
libcrypto.so.0.9.7
libcrypto.so.0.9.7
libcurl.so.3
libdatamatrixpmp.pmp
libextendscript.so
libextendscript.so
libgcc_s.so.1
libgcc_s.so.1(GCC_3.0)
libgcc_s.so.1(GCC_3.3)
libgcc_s.so.1(GCC_3.3.1)
libgcc_s.so.1(GCC_3.4)
libgcc_s.so.1(GCC_3.4.2)
libgcc_s.so.1(GCC_4.0.0)
libgcc_s.so.1(GLIBC_2.0)
libicudata.so.34
libicudata.so.34
libicui18n.so.34
libicuuc.so.34
libicuuc.so.34
libpdf417pmp.pmp
libqrcodepmp.pmp
librt3d.so
libsccore.so
libsccore.so
libssl.so.0.9.7
libssl.so.0.9.7
libstdc++.so.6
libstdc++.so.6(CXXABI_1.3)
libstdc++.so.6(CXXABI_1.3.1)
libstdc++.so.6(GLIBCXX_3.4)
libstdc++.so.6(GLIBCXX_3.4.1)
libstdc++.so.6(GLIBCXX_3.4.2)
libstdc++.so.6(GLIBCXX_3.4.3)
libstdc++.so.6(GLIBCXX_3.4.4)
libstdc++.so.6(GLIBCXX_3.4.5)
libstdc++.so.6(GLIBCXX_3.4.6)
libstdc++.so.6(GLIBCXX_3.4.7)
nppdf.so
prcr.x3d
tesselate.x3d
wwwlink.api
acroread = 8.1.2-1.slc4
Yep, thats right - RPM had decided that acroread was a dependency. Grr. Workaround - remirror the CERN SLC repo, (no they hadn't updated since), manually remove the offending rpm, and rebuild the metadata with 'createrepo'
Then make sure that the nodes were rebuilt and only ever looked to our local repository rather than the primary cern / dag ones. (thanks to kickstart %post and cfengine)
Finally we got yaim to *almost* run - it was failing on lcmaps and grid-mapfile creation (fixed by unsupporting a certain VO)
Easy fix in comparison. Anyway - DPM up n running and seems OK. Roll on the next SAM tests.... (or real users)
Phew.
Grid Middleware should not be this hard to install!
Tuesday, February 19, 2008
DPM and ATLAS Space Tokens
So, here's the definitive guide to enabling space tokens for ATLAS:
- Assign the token to atlas/Role=production using the dpm-reservespace command. (Example here.)
- Create the directory where DDM will put the files, which is the normal path for ATLAS, plus the name of the spacetoken in lower case. e.g.
- dpns-mkdir /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk
- Now change group ownership of this directory to atlas/Role=production, chmod it to 775 and finally add two ACLs which will mean the entire tree will be writable by production roles:
- dpns-chgrp atlas/Role=production /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk
- dpns-chmod 775 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk
- dpns-setacl -m d:g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk
- dpns-setacl -m g:atlas/Role=production:7,m:7 /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk
- Enable the space token publisher (this only needs to be done once - subsequent tokens are picked up automatically). Instructions here.
Thursday, February 14, 2008
DPM SRMv2 Tweaks
Got back from holiday to find that FDR transfers to Glasgow were failing with a "permission denied" error. Looking through the logs it seemed that the srmMkDir call was failing (unlike SRMv1 you need to make the path before transferring data into the SE).
However, it seems the default umask for DPM srmMkDir is 022, which leaves the directory unwritable. As I had tested the /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk path I left it unwritable by Mario's certificate.
I fixed this with a dpns-chmod, but then I wrote a script to patch up the space token directory tree area using DPM ACLs to ensure the directories are group writable by the production role.
You can get the script here: http://www.physics.gla.ac.uk/~graeme/atlas/scripts/atlas-dpm-token-fix.sh.
However, it seems the default umask for DPM srmMkDir is 022, which leaves the directory unwritable. As I had tested the /dpm/gla.scotgrid.ac.uk/home/atlas/atlasdatadisk path I left it unwritable by Mario's certificate.
I fixed this with a dpns-chmod, but then I wrote a script to patch up the space token directory tree area using DPM ACLs to ensure the directories are group writable by the production role.
You can get the script here: http://www.physics.gla.ac.uk/~graeme/atlas/scripts/atlas-dpm-token-fix.sh.
Tuesday, February 05, 2008
ATLAS Space Token Roles
Following a discussion with ATLAS people I've now clarified that SRMv2 writing will happen using the atlas/Role=production VOMS role.
Therefore sites should restrict access to the ATLASMCDISK and ATLASDATADISK space tokens to this role. To do this, release and then recreate the space reservation:
# dpm-releasespace --token_desc ATLASDATADISK
# dpm-reservespace --gspace 10T --lifetime Inf --group atlas/Role=production --token_desc ATLASDATADISK
Greig's python snippet to list spaces is very helpful.
Therefore sites should restrict access to the ATLASMCDISK and ATLASDATADISK space tokens to this role. To do this, release and then recreate the space reservation:
# dpm-releasespace --token_desc ATLASDATADISK
# dpm-reservespace --gspace 10T --lifetime Inf --group atlas/Role=production --token_desc ATLASDATADISK
Greig's python snippet to list spaces is very helpful.
Friday, February 01, 2008
DPM Storage Token Information Published
It was a bit fiddly, but with 4 fixes to Michel's script, Glasgow are publishing the space token information for ATLASDATADISK.
I documented the workarounds in the LCG Twiki.
I documented the workarounds in the LCG Twiki.
Wednesday, January 30, 2008
SRM2.2 Configuration for FDR/CCRC
I configured Glasgow's DPM for the SRM 2.2 space tokens required by ATLAS for FDR/CCRC:
svr018:~# dpm-reservespace --gspace 10T --lifetime Inf --group atlas --token_desc ATLASDATADISK
ab0f1a60-59d6-4099-82fa-a17711678860
Easy, eh!
I notice that there is no dpm-listspaces command, which means that one has to grub around inside the database to find out what spaces are currently defined.
Two additional notes for other T2s:
svr018:~# dpm-reservespace --gspace 10T --lifetime Inf --group atlas --token_desc ATLASDATADISK
ab0f1a60-59d6-4099-82fa-a17711678860
Easy, eh!
I notice that there is no dpm-listspaces command, which means that one has to grub around inside the database to find out what spaces are currently defined.
Two additional notes for other T2s:
- Transfers from DDM are done using a vanilla atlas proxy for now (belonging to Mario Lassnig), so make sure the token is writable by the atlas group, not, e.g., atlas/Role=production.
- All that is needed for CCRC is 3TB, however this is based on a 1 week cleaning cycle. If, like Glasgow, you have lots of space, then making the area bigger means the cleaning is not critical. (The space can be updated later with dpm-updatespace.)
Tuesday, January 29, 2008
Keeping up with the Jones'
Well we recently had an incident with our NFS server for the cluster (home / software) locking up and needing a cold power cycle. Due to $vendors setup this takes aaaages (in the order of 20 mins) to go through the BIOS selfcheck (hangs at 053C). $vendor would like to poke around system and perhaps perform bios upgrade. Hmm. Oh well, all 10 disk servers are identical so we'll just drain one down and play - it also gives us chance to upgrade (from 1.6.5) to the latest 1.6.7-mumble DPM.
... or so we thought.
"Thats odd - Graeme have you updated these?" nope - Turns out that yum.nightly cron was auto updating on both the disk servers and some of the grid servers... Gaaah. clickity click and we're all ready to play.
In the meantime, dpm-drain migrated most of the data off the server to the other stash of disks but there were still 69 files that failed with 'Internal error' - Am looking through the DB to try and see if I can pull any more info out
... or so we thought.
disk032:~# rpm -qa | grep DPM
DPM-gridftp-server-1.6.7-1sec
DPM-rfio-server-1.6.7-2sec.slc3
DPM-client-1.6.7-2sec.slc3
"Thats odd - Graeme have you updated these?" nope - Turns out that yum.nightly cron was auto updating on both the disk servers and some of the grid servers... Gaaah. clickity click and we're all ready to play.
In the meantime, dpm-drain migrated most of the data off the server to the other stash of disks but there were still 69 files that failed with 'Internal error' - Am looking through the DB to try and see if I can pull any more info out
Subscribe to:
Posts (Atom)