I've made some significant steps towards being able to improve the security of installation in the cluster.

I have adopted a "watcher" method, where a process running on the master node looks for a signal that a client is ready to receive secrets, then checks an authorisation database (sqlite) to see if this is allowed. If it's not, then it igmores it. If it is it will push the node's ssh keys and restart its ssh server, then push out its grid certificate (if applicable).

This is easy enough to patch in to CVOS, which can send the signal (the signal is in fact just a side effect of requesting "firstboot.php" from the master's web server). However, after the initial install we still want thre rsh server turned off!