As part of the on-going developments to the Scot Grid cluster at Glasgow, we have decommissioned our final LCG-CE, which resided on SVR021. The removal of this CE allows us to concentrate the support and development of two CE platforms; Cream and ARC. We are planning to conduct a series of tests around the three CREAM CE's we have deployed at Glasgow in an attempt to gain a better understanding of their maximum loading potential for running jobs and how to tweak them to gain the maximum efficiency from this service.
Additionally, we will be testing our availability metrics over the next month as the LCG-CE was one of the corner stones of Steve Lloyd's tests of our overall availability. This will now be monitored primarily through our SRM availability.
The reasons for decommissioning the LCG-CE are that we would be removing it at some point in the near future, all the big VO's do not have issues with submitting to Cream CEs and it simplifies our internal support requirements.
The new servers running Cream are svr008, svr014 and svr026.
Thank you LCG-CE and goodnight.
Showing posts with label cream. Show all posts
Showing posts with label cream. Show all posts
Monday, February 21, 2011
Covering up problems with CREAM
For some days now, ScotGrid Glasgow has been operating with only CREAM CEs, having turned our final lcg-CE off around the 14th. I'll let Mark cover the details of this in his later post, but I wanted to briefly mention one of the minor configuration details that caused some problems for us initially.
The gridmapdir (usually in /etc/grid-security/gridmapdir ) is a somewhat integral part of the pool account mapping system in LCG/gLite services. It contains one (empty) file for each pool account, plus hard-links to them from each DN(+VOMS Role) mapped to them. Basically, it's a cheap way to ensure that you don't get multiple mapped DNs to the same account (as you can always count the number of hard-links to an inode).
We share our gridmapdir, over NFS, to all of our CEs, to ensure that any incoming job from a given user is consistently mapped. Unfortunately, this lead to our minor configuration gaffe (which I just fixed).
The lcg-CE, you see, is configured to set the ownership and permissions on the gridmapdir to 0755 root:root. This is fine for it, since lcg-CEs do strange things like running their services with root permissions, and it prevents anything else from messing up the mappings.
CREAM CEs (using glexec), need to have their gridmapdir as 0775 root:glexec, a change which we hadn't made when we installed them (and which probably YAIM couldn't have done for us). This meant that, for the time the CREAM CEs were installed, they've never been able to create a new mapping in the gridmapdir, as they try to do that as members of the glexec group.
We never really noticed this problem while we had lcg-CEs which were busy, as the lcg-CE would almost always have also received jobs from the user previously and already performed the mapping.
Now that we don't have an lcg-CE, however, it started to cause some odd problems when we enabled new VOs, as the configuration seemed perfectly fine for the VO itself, but jobs would bounce off the CREAM CEs with "Failed to get the local userid with glexec" errors.
Obviously, this was trivially solved once we worked out what the issue was (by setting the gridmapdir's group-ownership and permissions to glexec g+w), but identifying it was a little tricky, as the default logging level for LCMAPS doesn't give many clues as to what problem it's having.
Turning the debug level up to 3 (in /opt/glite/etc/glexec.conf ) was sufficient to get it to log errors with gridmapdir_newlease(), however, and then, after some poking (and manual creation of DN links to see what happened), the problem became clear.
So, this is a cautionary tale about moving from a mixed CE environment to a monoculture (ignoring Stuart's ARC installation) - sometimes a misconfiguration in one service can be hidden by the correct functioning of the service you're just about to remove.
Tuesday, December 14, 2010
Clotted CREAM
Last time I was blogging, I mentioned some problem with our CREAM CE, and too many jobs in the Blah Registry.
Unlike my initial theory, the all_done interval problem turned out to not be the culprit; instead it was down to the Blah Registry.
CREAM splits the whole deal with being a Compute Element into two main parts: the interaction with the wider world, which is handled with some Java code using Tomcat; and the direct interaction with the batch system, called BLAH, and written in C and shell script.
The Java code, which I'll refer to as CREAM, as distinct from the BLAH parts, keeps it's state in the MySQL database. BLAH, on the other hand, uses a hand rolled indexed file, with C functions for accessing and writing data.
The BLAH registry is updated by the command blah_job_registry_add after the qsub is complete; to record the mapping between the CREAM job ID and batch system job id. This is the step were we ran into problems. The version of CREAM we were running was set to purge jobs after about two months - and in two months we were putting just over half a million jobs through it.
With that many jobs in the registry, it was taking a noticeable time to add any job. Further, the locking done effectively serialises access to the registry (i.e. Table locking in RDBMS parlance). Couple that with the Atlas pilot factory's favourite habit of dumping jobs in batches of 10 to 20 at a time, and you can see how some jobs ended up taking longer than the timeout to register.
Just before we'd encountered this, there was a new version of CREAM released (glite-CREAM-3.2.8) that cut the default time before purging to about one month, and put the indices in a mmaped file; both should mitigate this problem. We limped along with some workarounds for a bit [0], before doing that update earlier this week. The update from 3.2.7 to 3.2.8 went very quickly, by the way; took us about 5 minutes; although we did have to manually tidy up /etc/sudoers.
As it stands now, with about quarter of a million jobs in the registry, it's taking about a couple of seconds to register a job; but with occasional pauses when there are many jobs pending. Thus far it's prevented a recurrence of large number of blocked jobs, but I'll be keeping an eye on it.
[0] The other CE's were having hardware issues, and we didn't want to have all the CE's down at once...
Unlike my initial theory, the all_done interval problem turned out to not be the culprit; instead it was down to the Blah Registry.
CREAM splits the whole deal with being a Compute Element into two main parts: the interaction with the wider world, which is handled with some Java code using Tomcat; and the direct interaction with the batch system, called BLAH, and written in C and shell script.
The Java code, which I'll refer to as CREAM, as distinct from the BLAH parts, keeps it's state in the MySQL database. BLAH, on the other hand, uses a hand rolled indexed file, with C functions for accessing and writing data.
The BLAH registry is updated by the command blah_job_registry_add after the qsub is complete; to record the mapping between the CREAM job ID and batch system job id. This is the step were we ran into problems. The version of CREAM we were running was set to purge jobs after about two months - and in two months we were putting just over half a million jobs through it.
With that many jobs in the registry, it was taking a noticeable time to add any job. Further, the locking done effectively serialises access to the registry (i.e. Table locking in RDBMS parlance). Couple that with the Atlas pilot factory's favourite habit of dumping jobs in batches of 10 to 20 at a time, and you can see how some jobs ended up taking longer than the timeout to register.
Just before we'd encountered this, there was a new version of CREAM released (glite-CREAM-3.2.8) that cut the default time before purging to about one month, and put the indices in a mmaped file; both should mitigate this problem. We limped along with some workarounds for a bit [0], before doing that update earlier this week. The update from 3.2.7 to 3.2.8 went very quickly, by the way; took us about 5 minutes; although we did have to manually tidy up /etc/sudoers.
As it stands now, with about quarter of a million jobs in the registry, it's taking about a couple of seconds to register a job; but with occasional pauses when there are many jobs pending. Thus far it's prevented a recurrence of large number of blocked jobs, but I'll be keeping an eye on it.
[0] The other CE's were having hardware issues, and we didn't want to have all the CE's down at once...
Thursday, November 25, 2010
Stale CREAM and Maui partitioning
Nothing terribly exciting; but we've done a bit of an update on our Maui configuration, and CREAM problem has been cleared a-whey.
Previously, we've had our different era's of compute nodes annotated with the key speed notifier; so that the scheduler understands the fact that some are faster. For the vast majority of grid jobs, this is an utterly irrelevant distinction - so long as the job gets the time it expects (and Maui is scaling requests that go to the slower nodes so they get longer).
However jobs that use more than one process (i.e. MPI jobs) are a different case - if they get scheduled with different classes of nodes, then you get sub-optimal resource useage. So it's useful to keep some distinction between them. Previously we've been using reservations to restrict where jobs can go - but there's an (ill-defined) upper bound to the maximum number of overlapping reservations on a single job slot at once; too many breaks things.
So we had a look at Partitions in Maui, which is really the proper way to handle these. The downside is that you're limited to 3 partitions - there's a compiled in limit of 4, one of which is the special [ALL] partition, and one is DEFAULT. Fortunately, we have 3 era's of kit - and as long as we're happy calling one 'DEFAULT', it all works. And Maui understands never to schedule a job to more than one partition at a time.
So we ended up with a lot of lines like:
Not terribly exciting Maui tweaks, but sometimes that's the way of it.
The CREAM problem manifested itself as a set of jobs that have gone sour. There's a known problem in the current versions of CREAM where if the job is queueing on PBS for over an hour, CREAM (rather, the BLAH parser) thinks it's dead, and kills the job, with reason=999.
What's not made explicit is that you need to have no spaces in that! i.e. you must have
because if you put alldone_interval = 7200, then CREAM doesn't understand that. So fixed that, and it was all hunky dory for a while. Then we started getting lots of blocked jobs in Torque again; all from CREAM.
Cue more digging.
Eventually found this in the CREAM logs (after a slight reformatting):
JOB CREAM942835851 STATUS CHANGED: PENDING => ABORTED
[failureReason=BLAH error: no jobId in submission script's output
(stdout:) (stderr:/opt/glite/etc/blah.config: line 81: alldone_interval: command not found-
execute_cmd: 200 seconds timeout expired, killing child process. )
So, two things here. Firstly, alldone_interval with spaces had crept back in, along with the correct version - in our case via cfengine directives, probably down to the Double CREAM plan. More interesting was that having the invalid part of the BLAH config present slows down BLAH (BUpdaterPBS was pegging a CPU at 100%), sufficent that it hits another timeout at 200 seconds to respond at all. And then CREAM kills the job, but doesn't actually tell Torque, sand box is blown away, so it can't finish (nowhere to put output), or, if not started, can't start.
Removing the second (wrong) version of the alldone_interval fixed that - CPU use in the parser dropped to trival levels, and all appears to be happy again. This one's not really CREAMS fault, but it's always good to have an idea of what misconfigured services end up doing, otherwise it's hard to fix those 'not enough coffee' incidents. Hence, this one for Google...
UPDATE: Oops! Spoke too soon. That's the problem defined, but clearly not the solution - as it's happened again. Gonna leave this here as a reminder to self to give it a bit longer before considering something fixed...
Previously, we've had our different era's of compute nodes annotated with the key speed notifier; so that the scheduler understands the fact that some are faster. For the vast majority of grid jobs, this is an utterly irrelevant distinction - so long as the job gets the time it expects (and Maui is scaling requests that go to the slower nodes so they get longer).
However jobs that use more than one process (i.e. MPI jobs) are a different case - if they get scheduled with different classes of nodes, then you get sub-optimal resource useage. So it's useful to keep some distinction between them. Previously we've been using reservations to restrict where jobs can go - but there's an (ill-defined) upper bound to the maximum number of overlapping reservations on a single job slot at once; too many breaks things.
So we had a look at Partitions in Maui, which is really the proper way to handle these. The downside is that you're limited to 3 partitions - there's a compiled in limit of 4, one of which is the special [ALL] partition, and one is DEFAULT. Fortunately, we have 3 era's of kit - and as long as we're happy calling one 'DEFAULT', it all works. And Maui understands never to schedule a job to more than one partition at a time.
So we ended up with a lot of lines like:
NODECFG[node060] SPEED=0.94 PARTITION=cvoldBut in order to make them used by all jobs, we had to adjust the default partitions available to include them all:
SYSCFG PLIST=DEFAULT:cvoldwhich gives all users equal access to all the partitions.
Not terribly exciting Maui tweaks, but sometimes that's the way of it.
The CREAM problem manifested itself as a set of jobs that have gone sour. There's a known problem in the current versions of CREAM where if the job is queueing on PBS for over an hour, CREAM (rather, the BLAH parser) thinks it's dead, and kills the job, with reason=999.
What's not made explicit is that you need to have no spaces in that! i.e. you must have
alldone_interval=7200
because if you put alldone_interval = 7200, then CREAM doesn't understand that. So fixed that, and it was all hunky dory for a while. Then we started getting lots of blocked jobs in Torque again; all from CREAM.
Cue more digging.
Eventually found this in the CREAM logs (after a slight reformatting):
JOB CREAM942835851 STATUS CHANGED: PENDING => ABORTED
[failureReason=BLAH error: no jobId in submission script's output
(stdout:) (stderr:/opt/glite/etc/blah.config: line 81: alldone_interval: command not found-
So, two things here. Firstly, alldone_interval with spaces had crept back in, along with the correct version - in our case via cfengine directives, probably down to the Double CREAM plan. More interesting was that having the invalid part of the BLAH config present slows down BLAH (BUpdaterPBS was pegging a CPU at 100%), sufficent that it hits another timeout at 200 seconds to respond at all. And then CREAM kills the job, but doesn't actually tell Torque, sand box is blown away, so it can't finish (nowhere to put output), or, if not started, can't start.
UPDATE: Oops! Spoke too soon. That's the problem defined, but clearly not the solution - as it's happened again. Gonna leave this here as a reminder to self to give it a bit longer before considering something fixed...
Wednesday, May 05, 2010
CREAM thickens
CREAM at Glasgow has been upgraded to the latest glite3.2 release 3.2.5-0.sl5 (or INFN version 1.6). This brings lots of enhancements like
self limiting behaviour a'la WMS
A new proxy purger to clean the delegationdb and from the file system the expired proxies
a new way to customize the job wrapper
an improved proxy renewal mechanism
and one of my favourites, support for ISB/OSB transfers from/to gridftp servers run using user credentials rather than server host certificates. This will work well with users running gridftp servers on their own machines for example (as long as they don't turn them off when they go home at night!)
you can find out more about them all here. Now back to draining the WMS.
self limiting behaviour a'la WMS
A new proxy purger to clean the delegationdb and from the file system the expired proxies
a new way to customize the job wrapper
an improved proxy renewal mechanism
and one of my favourites, support for ISB/OSB transfers from/to gridftp servers run using user credentials rather than server host certificates. This will work well with users running gridftp servers on their own machines for example (as long as they don't turn them off when they go home at night!)
you can find out more about them all here. Now back to draining the WMS.
Tuesday, April 06, 2010
CREAM gets an upgrade
The CREAM instance at Glasgow has now been upgraded to the latest SL5 version. This continues the push to migrate those services that can be moved from SL4 to SL5 and should also make it easier to upgrade to the new 1.6 instance when it is released. The only hitch to a relatively painless upgrade was cfengine tweaking LCAS and replacing 64 bit path names with 32 bit paths.
Tuesday, February 16, 2010
cream sours
Well we now know how much it takes to kill our CREAM instance. Yesterday it stoppped working completely and it appeared to be caught in a tailspin with the Lease and Proxy Renew processes within CREAM. Grepping the logs indicated that most of the Renewals and Lease Manager entries were all related to condor submission from ATLAS.
From speaking to Massimo at INFN it was described how Proxy and Lease renewals are operations which are executed with higher priorities wrt other commands. One hypothesis might be that the CREAM CE was so overloaded doing these commands that it was unable to deal with basic job submission since all the test jobs I submitted never made it out out the REGISTERED state.
It looked bad on Ganglia:
The first course of action was to disable job submission using the command line tool: glite-ce-disable-submission and try to deal with the renewals. This worked for a time but they reoccurred later on that evening.
The timestamps on these ATLAS cream jobs seems to be very old and hinted at stale jobs so the next course of action was to manually purge the database using the tool provided by the CREAM developers: here. The easiest way I could see to do this was to connect to the creamdb, select out the id's and create a script that called the purger for each id. Note: you need jdk 1.6 in order to run the purger!
This ended up removing around 3000 CREAM entries.
Ganglia looked much happier:
So I think you have to be careful when getting submissions from Condor at the moment as it looks to be quite easy to denial of service your CREAM CE.
Roll on CREAM 1.6
- That proxy renewal is not very efficient in the release now in production (already addressed in the coming CREAM CE: see here)
- When there are too many pending commands, new job submissions will be disabled by the limiter: see here
From speaking to Massimo at INFN it was described how Proxy and Lease renewals are operations which are executed with higher priorities wrt other commands. One hypothesis might be that the CREAM CE was so overloaded doing these commands that it was unable to deal with basic job submission since all the test jobs I submitted never made it out out the REGISTERED state.
It looked bad on Ganglia:
The first course of action was to disable job submission using the command line tool: glite-ce-disable-submission and try to deal with the renewals. This worked for a time but they reoccurred later on that evening.
The timestamps on these ATLAS cream jobs seems to be very old and hinted at stale jobs so the next course of action was to manually purge the database using the tool provided by the CREAM developers: here. The easiest way I could see to do this was to connect to the creamdb, select out the id's and create a script that called the purger for each id. Note: you need jdk 1.6 in order to run the purger!
This ended up removing around 3000 CREAM entries.
Ganglia looked much happier:
So I think you have to be careful when getting submissions from Condor at the moment as it looks to be quite easy to denial of service your CREAM CE.
Roll on CREAM 1.6
- That proxy renewal is not very efficient in the release now in production (already addressed in the coming CREAM CE: see here)
- When there are too many pending commands, new job submissions will be disabled by the limiter: see here
Monday, May 18, 2009
Cream in Action : Local Users & Glexec
At Glasgow have now rolled out a production Cream instance open to only dteam, ops, vo.scotgrid.ac.uk and our newly created vo.optics.ac.uk (to support optics user community and Lumerical's FDTD software). This is svr014 and it looks like CMS are now looking for production Cream instances too. So it may see further action.
One thing that we have done in the past with our local user community is tweak LCMAPS such that specific local users do not use a pool account for their jobs. This was documented in a previous blog post. With cream I thought we should at least attempt to follow the same model for local users.
However, Cream uses glexec with LCMAPS and unfortunately the current version of glexec that comes with the cream CE to map to local users does not work correctly. Thanks to Oscar and Mischa at Nikhef for getting me the right versions of glexec. Here are the versions required to do the following mapping in LCMAPS:
these are all in pre-production, so should be out soon in a full cream update.
When these rpm's are installed take care to set the setuid bits as these are lost during the update.
With these installed the following lcmaps policy can be added/amended to
This policy when moved to be executed first in the list will map any users in the
This 'tweak' seems to work but as I discovered Cream does not really like you doing this and you have to be very careful about the primary group of the user that glexec transforms you to. In Cream
Note that these are all owned by the tomcat user and the group is in effect the grid group. So when not using any customised local users when glexec maps you via your voms extension e.g. vo.scotgrid.ac.uk to scotg001 a member of the scotg group and you end up in the scotg directory. Also note the permission of the directory named after your proxy: 700. Meaning only no group read/write permissions on the files contained within the directories.
When using a local user 'tweaked' LCMAPS and my vo.scotgrid.ac.uk proxy gla057/scotg it attempts to stage the input files to scotg but fails like this:
This was very confusing at first but when you actually try to do a globus-url-copy or an uberftp which I presume was the CREAM UI is trying to do. You see that it is in fact using your proxy on the client side to map you to a pool account and gsiftp the files to CREAM. From what I could see it was using scotg094. On the server side after applying the local user 'tweak' what it meant was that glexec was actually interacting with cream to build the sandbox directories with a different user. This interaction can be seen here in
So the gsiftp could not write as the user was no longer the pool user and there are no group write permission on the directories contained within the sandbox. I was able to get round this by relaxing the permissions from 700 to 770 so that members of the same group could effectively read/write/execute to the sandbox directory by patching
Now this all worked because my local user gla057 still has a primary group that matches the pool accounts primary group of scotg. However, we have other local users that have a unix group glee. This does not match any of primary groups of the accounts pool available to the VO that they are a member of: nanocmos. I thought the quick win would be to add the nano pool accounts to have an additional group of glee.
But it turns out that globus-url-copy and uberftp etc do not understand the concept of secondary groups when gsiftp'ing. So no luck there.
I think the only possible solution is to create another local VO which can be supported properly through the middleware. A hassle but less of a hack.
One thing that we have done in the past with our local user community is tweak LCMAPS such that specific local users do not use a pool account for their jobs. This was documented in a previous blog post. With cream I thought we should at least attempt to follow the same model for local users.
However, Cream uses glexec with LCMAPS and unfortunately the current version of glexec that comes with the cream CE to map to local users does not work correctly. Thanks to Oscar and Mischa at Nikhef for getting me the right versions of glexec. Here are the versions required to do the following mapping in LCMAPS:
glite-security-glexec-0.6.8-2.slc4.i386.rpm
glite-security-lcmaps-1.4.7-1.slc4.i386.rpm
glite-security-lcmaps-plugins-basic-1.3.10-2.slc4.i386.rpm
these are all in pre-production, so should be out soon in a full cream update.
When these rpm's are installed take care to set the setuid bits as these are lost during the update.
-rwsr-sr-x 1 root glexec 65620 Apr 30 15:56 /opt/glite/sbin/glexecWith these installed the following lcmaps policy can be added/amended to
/opt/glite/etc/lcmaps/lcmaps-suexec.db
localuseraccount = "lcmaps_localuseraccount.mod -gridmapfile /usr/local/etc/grid-mapfile-local"
glexec_get_account:
proxycheck -> localuseraccount
localuseraccount -> good | vomslocalgroup
vomslocalgroup -> vomspoolaccount | poolaccount
vomspoolaccount -> good | vomslocalaccount
vomslocalaccount -> good | poolaccount
poolaccount -> good
This policy when moved to be executed first in the list will map any users in the
grid-mapfile-local to their local user accounts rather than a pool account.This 'tweak' seems to work but as I discovered Cream does not really like you doing this and you have to be very careful about the primary group of the user that glexec transforms you to. In Cream
/opt/glite/var/cream_sandbox is the directory where the sandbox files are staged on the CREAM CE. This contains a set of directories, created I believe by yaim, named after each of the user/role combination. For example
drwxrwx--- 2 tomcat scotg 4096 Apr 28 12:42 scotg
drwxrwx--- 2 tomcat scotgprd 4096 Apr 28 12:42 scotgprd
drwxrwx--- 2 tomcat scotgsgm 4096 Apr 28 12:42 scotgsgm
dev011:/opt/glite/var/cream_sandbox/scotg# ls -la
total 24
drwxrwx--- 3 tomcat scotg 4096 May 18 14:14 .
drwxrwxr-x 81 tomcat tomcat 4096 May 18 14:10 ..
drwx------ 3 scotg094 scotg 4096 May 18 14:14 C_UK_O_eScience_OU_Glasgow_L_Compserv_CN_douglas_mcnab_vo.scotgrid.ac.uk_Role_NULL_Capability_NULL
Note that these are all owned by the tomcat user and the group is in effect the grid group. So when not using any customised local users when glexec maps you via your voms extension e.g. vo.scotgrid.ac.uk to scotg001 a member of the scotg group and you end up in the scotg directory. Also note the permission of the directory named after your proxy: 700. Meaning only no group read/write permissions on the files contained within the directories.
When using a local user 'tweaked' LCMAPS and my vo.scotgrid.ac.uk proxy gla057/scotg it attempts to stage the input files to scotg but fails like this:
2009-05-18 14:20:18,983 INFO - Sending [/clusterhome/home/gla057/lumerical/paralleltest.fsp] to [gsiftp://dev011.gla.scotgrid.ac.uk/opt/glite/var/cream_sandbox/scotg/C_UK_O_eScience_OU_Glasgow_L_Compserv_CN_douglas_mcnab_vo.scotgrid.ac.uk_Role_NULL_Capability_NULL/CREAM679019987/ISB/paralleltest.fsp]...
2009-05-18 14:20:18,984 DEBUG - ftpclient::put() - dst=[gsiftp://dev011.gla.scotgrid.ac.uk/opt/glite/var/cream_sandbox/scotg/C_UK_O_eScience_OU_Glasgow_L_Compserv_CN_douglas_mcnab_vo.scotgrid.ac.uk_Role_NULL_Capability_NULL/CREAM679019987/ISB/paralleltest.fsp]
2009-05-18 14:20:19,761 ERROR - data_cb() - globus_ftp_client: the server responded with an error
2009-05-18 14:20:19,761 ERROR - done_cb() - globus_ftp_client: the server responded with an error
2009-05-18 14:20:19,764 FATAL - Error sending file [/clusterhome/home/gla057/lumerical/paralleltest.fsp]
This was very confusing at first but when you actually try to do a globus-url-copy or an uberftp which I presume was the CREAM UI is trying to do. You see that it is in fact using your proxy on the client side to map you to a pool account and gsiftp the files to CREAM. From what I could see it was using scotg094. On the server side after applying the local user 'tweak' what it meant was that glexec was actually interacting with cream to build the sandbox directories with a different user. This interaction can be seen here in
/opt/glite/etc/glite-ce-cream/cream-glexec.sh
drwx------ 3 gla057 scotg 4096 May 18 14:20 C_UK_O_eScience_OU_Glasgow_L_Compserv_CN_douglas_mcnab_vo.scotgrid.ac.uk_Role_NULL_Capability_NULL
So the gsiftp could not write as the user was no longer the pool user and there are no group write permission on the directories contained within the sandbox. I was able to get round this by relaxing the permissions from 700 to 770 so that members of the same group could effectively read/write/execute to the sandbox directory by patching
/opt/glite/etc/glite-ce-cream/cream-glexec.sh. Although I am not entirely happy about this as this could be a security concern. Now this all worked because my local user gla057 still has a primary group that matches the pool accounts primary group of scotg. However, we have other local users that have a unix group glee. This does not match any of primary groups of the accounts pool available to the VO that they are a member of: nanocmos. I thought the quick win would be to add the nano pool accounts to have an additional group of glee.
But it turns out that globus-url-copy and uberftp etc do not understand the concept of secondary groups when gsiftp'ing. So no luck there.
I think the only possible solution is to create another local VO which can be supported properly through the middleware. A hassle but less of a hack.
Cream in Action : Consumable Resources
I am not sure if you remember this previous post but I said stated that some experimenting was required in order to get consumable resources working with the glite middleware stack.
The reason for this requirement was that for some licensed software (FDTD by Lumerical) that we have installed on our cluster. The documented way to 'consume' a license is to qsub directly to the batch system and pass #software -l FDTD. Not much good when you have an lcg-CE in front of it! After some further investigation it appeared that the only way to get this information through the lcg-CE would be to 'patch' the job manager, so that it added this into the generated PBS script based on RSL that could be sent to it. Unfortunately, from what I could see the RSL schema did not have anything that could be used to fit this software attribute out of the box and patching the job manager was not an ideal going forward.
This looked to only leave the option of creating a specific queue for the software and only allowing members of the new VO to run in this queue. However, it finally struck me to look at the capabilities of cream. With the help of Massimo Sgaravatto and David Rebatto I was able to pass this batch system requirement through the wms, cream and finally end up on the batch system correctly with very little customisation.
in summary:
- set in your JDL (the one used for the glite-ce-job-submit command):
- Create in the CREAM CE node the file:
/opt/glite/bin/pbs_local_submit_attributes.sh
which has to properly manage the added attribute ("software" in your
case). E.g. for this specific use case it could be something like:
So for any special CE requirements your can handle them by adding them into the
As for WMS submission, well when the ice component worked if only for a brief time...
the CErequirements attribute in the JDL sent to CREAM is supposed to be filled by the WMS. This value should basically take into account what it is specified in the Requirements attribute of the JDL and the value specified as CeForwardParameters in the WMS configuration file.
For example, if in your JDL you have:
and if the conf file of the WMS there is:
CeForwardParameters = {"GlueHostMainMemoryVirtualSize","GlueHostMainMemoryRAMSize","GlueCEPolicyMaxCPUTime"};
The JDL sent by ICE to CREAM should be:
Unfortunately this doesn't work because of this bug
What you can do now, as a workaround, is specify in the JDL used in the submission to the WMS this cerequirements, e.g.:
This will be forwarded as it is to CREAM.
This has now been written up in more detail on the cream page.
So to sum it up: Thumbs up for cream.
The reason for this requirement was that for some licensed software (FDTD by Lumerical) that we have installed on our cluster. The documented way to 'consume' a license is to qsub directly to the batch system and pass #software -l FDTD. Not much good when you have an lcg-CE in front of it! After some further investigation it appeared that the only way to get this information through the lcg-CE would be to 'patch' the job manager, so that it added this into the generated PBS script based on RSL that could be sent to it. Unfortunately, from what I could see the RSL schema did not have anything that could be used to fit this software attribute out of the box and patching the job manager was not an ideal going forward.
This looked to only leave the option of creating a specific queue for the software and only allowing members of the new VO to run in this queue. However, it finally struck me to look at the capabilities of cream. With the help of Massimo Sgaravatto and David Rebatto I was able to pass this batch system requirement through the wms, cream and finally end up on the batch system correctly with very little customisation.
in summary:
- set in your JDL (the one used for the glite-ce-job-submit command):
cerequirements = "software==\"FDTD\"";- Create in the CREAM CE node the file:
/opt/glite/bin/pbs_local_submit_attributes.sh
which has to properly manage the added attribute ("software" in your
case). E.g. for this specific use case it could be something like:
#!/bin/sh
if [ "$software" == "FDTD" ]; then
echo "#PBS -l software=FDTD"
fi
So for any special CE requirements your can handle them by adding them into the
submit_attributes.sh file. Cream also has similar capabilities for other batch systems.As for WMS submission, well when the ice component worked if only for a brief time...
the CErequirements attribute in the JDL sent to CREAM is supposed to be filled by the WMS. This value should basically take into account what it is specified in the Requirements attribute of the JDL and the value specified as CeForwardParameters in the WMS configuration file.
For example, if in your JDL you have:
Requirements= "other.GlueHostMainMemoryRAMSize > 100 && other.GlueCEImplementationName==\"CREAM\"";and if the conf file of the WMS there is:
CeForwardParameters = {"GlueHostMainMemoryVirtualSize","GlueHostMainMemoryRAMSize","GlueCEPolicyMaxCPUTime"};
The JDL sent by ICE to CREAM should be:
CeRequirements= "other.GlueHostMainMemoryRAMSize > 100";Unfortunately this doesn't work because of this bug
What you can do now, as a workaround, is specify in the JDL used in the submission to the WMS this cerequirements, e.g.:
cerequirements = "software==\"FDTD\"";This will be forwarded as it is to CREAM.
This has now been written up in more detail on the cream page.
So to sum it up: Thumbs up for cream.
Wednesday, April 08, 2009
cream broken pipes
I was just updating our pre-production cream set-up for testing it with a newly installed pre-production torque instance and it ceased to submit any jobs.
So if you ever find that when submitting a job to cream you see the following....
It looked like re-running yaim on the node had re-configured something incorrectly. Checking /var/log/messages it actually looked like glexec could no longer write to a log file.
It appears that cream has a default glexec log location set in the glexec.conf which is either in
This must directory must exist or else cream will not start! Something to remember in future!
So if you ever find that when submitting a job to cream you see the following....
2009-04-07 16:11:31,265 FATAL - MethodName=[jobRegister] Timestamp=[Tue 07 Apr 2009 16:11:31]
ErrorCode=[0] Description=[system error] FaultCause=[cannot write the job wrapper (jobId = CREAM600033116)!
The problem seems to be related to glexec which reported: Broken pipe]
It looked like re-running yaim on the node had re-configured something incorrectly. Checking /var/log/messages it actually looked like glexec could no longer write to a log file.
dev011:/var/log# tail -f messages
Apr 7 16:40:55 dev011 glexec[11697]: Error in LCAS/LCMAPS, rc = 107
Apr 7 16:40:55 dev011 glexec[11697]: LCAS failed, see '/var/log/glite/glexec_lcas_lcmaps.log' for more info.
Apr 7 16:43:33 dev011 glexec[12065]: glexec pid: 12065
Apr 7 16:43:33 dev011 glexec[12065]: lcas_log_open(): Cannot open logfile /var/log/glite/glexec_lcas_lcmaps.log
It appears that cream has a default glexec log location set in the glexec.conf which is either in
/opt/glite/var/log/glexec_lcas_lcmaps.log or /var/log/glite/glexec_lcas_lcmaps.log. This must have changed!This must directory must exist or else cream will not start! Something to remember in future!
Wednesday, March 11, 2009
ice cream anyone ...
We now have a functional cream CE in our preproduction mini cluster designated dev011. So what does this give us I hear you cry. Well it has been recently reported that an update to the glite packages allows the WMS to submit directly to Cream through the ice component. anyone for ice cream! So I thought, why not give this a shot! The updates were successfully installed on the UI (dev008), WMS (dev009) and Cream CE (dev011). When I say successfully installed, I actually mean with some minor jpackage voodoo. It seems that this repo is just plain broken and there are all sorts of clashes between the jpackage 5 and 1.7. In fact, the advice on lcg rollout seems to be to remove 1.7 from the repo definition altogether. It would be nice if we could get a standard build of the java that worked and distribute it along with the middleware. Since we haven't seen it. I'm guessing that is not possible! Anyway, on with the ice cream.
I was going to post all the fun I had trying to install the cream CE but for brevity I have moved that to a ScotGrid wiki page and will just show it working with the WMS. In order to test it working on our mini cluster I installed a site bdii and changed LCG_GFAL_INFOSYS such that lcg-infosites on the UI picked up the mini cluster CE's.
a whipped cream example:
submission to cream ce through a WMS:
can we see the job in torque? Yes we can.
Has it worked through the cream ce? Yes!
It also appears there is no need to ask for the job output as this is automatically gsiftp'd to your output sandbox dir specified in the jdl.
One point to note is that you now have to run a gridftp server to stage successful output from the cream CE. This is also useful for staging files in especially if you want to bypass WMS inputSandbox size limitations imposed by sites. For a more in-depth account of the install you can check out the ScotGrid wiki. This may help if you encounter anything weird.
I was going to post all the fun I had trying to install the cream CE but for brevity I have moved that to a ScotGrid wiki page and will just show it working with the WMS. In order to test it working on our mini cluster I installed a site bdii and changed LCG_GFAL_INFOSYS such that lcg-infosites on the UI picked up the mini cluster CE's.
-bash-3.00$ lcg-infosites --vo vo.scotgrid.ac.uk ce
#CPU Free Total Jobs Running Waiting ComputingElement
----------------------------------------------------------
.....
1912 5 4 0 4 dev011:8443/cream-pbs-q30m
1912 8 4 0 4 dev010:2119/jobmanager-lcgpbs-q30m
a whipped cream example:
-bash-3.00$ cat whippedcream.jdl
Type = "Job";
JobType = "Normal";
Executable = "double.sh";
StdOutput = "hw.out";
StdError = "hw.err";
InputSandbox = {"double.sh"};
OutputSandboxBaseDestURI = "gsiftp://dev008/clusterhome/home/gla057/cream/job_output";
OutputSandbox = {"hw.out", "hw.err"};
Requirements = other.GlueCEUniqueID == "dev011:8443/cream-pbs-q30m";
submission to cream ce through a WMS:
-bash-3.00$ glite-wms-job-submit -a --vo vo.scotgrid.ac.uk --debug -r dev011:8443/cream-pbs-q30m whippedcream.jdl
can we see the job in torque? Yes we can.
svr016:~# qstat | grep sco
2214311.svr016 cream_034614244 scotg001 0 W q30m
Has it worked through the cream ce? Yes!
-bash-3.00$ glite-wms-job-status https://dev009:9000/l4-RXjbtZbk1g00moK2IWA
*************************************************************
BOOKKEEPING INFORMATION:
Status info for the Job : https://dev009:9000/l4-RXjbtZbk1g00moK2IWA
Current Status: Done (Success)
Logged Reason(s):
- job completed
- Job Terminated Successfully
Exit code: 0
Status Reason: Job Terminated Successfully
Destination: dev011:8443/cream-pbs-q30m
Submitted: Fri Mar 6 16:28:15 2009 GMT
*************************************************************
It also appears there is no need to ask for the job output as this is automatically gsiftp'd to your output sandbox dir specified in the jdl.
-bash-3.00$ glite-wms-job-output https://dev009:9000/l4-RXjbtZbk1g00moK2IWA
Connecting to the service https://dev009:7443/glite_wms_wmproxy_server
Error - Output not Allowed
Output files already retrieved
One point to note is that you now have to run a gridftp server to stage successful output from the cream CE. This is also useful for staging files in especially if you want to bypass WMS inputSandbox size limitations imposed by sites. For a more in-depth account of the install you can check out the ScotGrid wiki. This may help if you encounter anything weird.
Subscribe to:
Posts (Atom)