A tale of two job managers

A while back I posted about supporting SL4 and SL5 OS's through the same batch system. Our solution was to use torque submit filters to add additional node properties to the jobs as they passed through the job managers on the CE's. This coupled with specific node properties on all nodes on the cluster worked quite well until I noticed that we were leaking CREAM jobs that should have requested SL5 running on the SL4 nodes.

After some investigation it appeared that when I was testing the filter and running the cream pbs qsub submit by hand I was always setting the number of nodes, even if I only required 1 i.e.

 
as a pool account .... 
/opt/glite/bin/pbs_submit.sh -c /bin/hostname -q q1d -n 1

This meant that there was always a #PBS -l nodes= in the submission script. However, if you call pbs_submit without the -n you get behaviour where no #PBS -l nodes= line appears in the final submission script. This then relies on the default behaviour that if no number of nodes is specified you get 1 node. This meant that my pbs filter did not catch the number of nodes and did not add the node property at all!

As it turns out on deeper investigation into the CREAM pbs_submission script. That when 1 node is required it uses the pbs default behaviour and does not specify a number of nodes. Only when there is more than one does it specify this i.e. MPI. This is a change from the lcg_CE job manager which always specifies a number of nodes be it 1 or more. Something to remember.

To get round this I have added an additional line to the cream pbs submit script to always default to 1 node if not MPI. Not the best but it's a short lived tweak until we get rid of our SL4 support. This should be very soon.

/opt/glite/bin/pbs_submit.sh

[ ! -z "$bls_opt_mpinodes" ] || echo "#PBS -l nodes=1" >> $bls_tmp_file