lightening testing of glexec with SCAS

Well since it is looking increasingly lightly that we will be moving to some form of identity switching at our sites to give us more information about who is running their jobs via their pilot frameworks. I thought I would give it whirl.

So in some lightening tests, a phrase I am stealing from lightening talks sometimes given at technical conferences, I am trialling glexec for identity switching coupled with SCAS for centralised allow/deny decisions.

Here is what was tested:

an install of SCAS
and install and test GLEXEC with SCAS on LCG-CE
and install and test GLEXEC with SCAS on CREAM [1]
and install and test GLEXEC on WN (SL4)
and install and test GLEXEC on WN (SL5)

Detailed Instructions and Results can be found here

The short and long of it is that it is very easy to set-up SCAS and use it on whatever service you want. So easy infact that once you SCAS server is up and running you cn direct calls to it from your CE's in a matter of minutes. glexec on the WN is just as easy, all that remains would be for someone to use it.

We currently have not rolled any of this into production but I am confident that it could be done quickly and safely. Since we are into real data taking, safely is the keyword. We want no unnecessary downtimes, which I think is achievable.

Thanks to Oscar at Nikhef for answering questions.

1: there appeared to be a certificate permission issue when calling SCAS from CREAM that prevented job submission. It looks like you need to copy the hostcert/key by hand to another cert owned by the tomcat user.


-rw-r--r-- 1 tomcat tomcat 2187 Dec 4 10:44 tomcathostcert.pem
-r-------- 1 tomcat tomcat 1863 Dec 4 10:44 tomcathostkey.pem


Lunch with glexec developers

We had a very interesting lunchtime meeting with the glexec developers
on Tuesday lunchtime (organised thanks to Alessandra's efforts in the TCG to convince the developers that the sites had serious issues with glexec). The developers very
happy to meet us and discuss the orgins of glexec and why they thought
that it was needed. It was clear that no-one in the meeting is at all
keen on generic pilot job frameworks. However, what's also clear is
that the LHC VOs are going to insist on having them and it's highly
unlikely that sites would ever be able to exert enough influence on
them to stop. However, what we can do is insist that if such
frameworks do exist, then they will have to use a glexec call and
respect its result. What glexec gives, in essence, is sudo like
abilities, but integrated into an X509 based authorisation scheme (and
it uses LCAS/LCMAPS plugins, which we are familiar with). The
advantages to the sites are that (1) there is a lot of control as to
who can call glexec in the first place, e.g., restricting this only to
production roles; (2) using LCMAPS single users can be banned, whereas
without glexec there only option with misbehaving pilot jobs is to ban
the pilot user on the CE (which is tantamount to banning the whole
VO); (3) there's an audit trail of who's payload has been executed
(which is crucial). glexec has safeguards to stop multiple calls
(i.e., the payload recalling glexec).

Then, to suid or not? We are assured that glexec will be distributed
in two flavours - one with the suid bit switched on in the RPM, the
other with it switched off. Of course, post-installation, one can
easily flip the bit using cfengine. The danger of not enabling suid is
that it will be possible for the payload job to access the submission
proxy certificate (danger for the VO) and that sorting out the payload
from the pilot is harder at the process level (danger for the
site). Of course, this has to be balanced by the danger of enabling
suid and risking a possible avenue of privilege escalation should
glexec turn out to have a security problem.

I'm still convinced that for ScotGrid deploying glexec in non-suid
form is best. We can run it for a while and then evaluate the
situation further. It's clear that sites like ECDF will never allow
glexec to be suid for them - so running it in non-suid mode will
always be an option.

The security vulnerabilities that Kostas identified have all been
fixed - along with more potential problems they spotted when going
through the code. They are very happy for other people to look at the
code and feedback problems to them.

Overall I was very impressed at the developers' openness. I still
don't think generic pilots are a good idea for the grid, but in a
world where they exist glexec is a definite help.