NATs Maxing Out

During our investigation of our LHCb failures we noticed that our number of conntrack entries on our two NAT hosts were in fact being totally used up i.e. all 43200! By looking at /proc/net/ip_conntrack we noticed that most of the connections were in fact udp DNS lookups by Camont jobs. We also noticed that we had not changed the default timeouts, 32768 for tcp and 3600 for udp. This was probably the reason they were being used up. So we have tweaked the timeouts and increased the maximum.
So our new NAT settings look like this:

/etc/sysctl.conf
original values of 43200, 32768, 3600 respectively.
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 21600
net.ipv4.netfilter.ip_conntrack_max = 65536
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30

Now out NAT's look much healthier. Only problem - it didn't help with LHCb productions jobs not being able to upload their results back to CERN. Back to the drawing board.

My name is total (well, maybe...)

I heard from Hannah today that the problems using the LFC at RAL for totalep were down to DNS problams (globus being very picky about hostnames). She's managed to overcome them by hard coding the LFC's hostname into /etc/hosts, or by using a different machine on a subnet with better DNS. Hopefully we'll see some jobs soon.

Spaeking of jobs from minor VOs, I noticed that camont sent quite a few jobs in the last week. In total they've done nearly 500 jobs and consumed about 450 CPU hours (812 kSI2k hours).

New VOs at Glasgow: camont, gridpp, totalep

I've enabled the camont, gridpp and totalep VOs on the Glasgow cluster.

A complete description of how to do this is on the wiki.

This stupidest task must have been to write a script to reverse engineer a YAIM users.conf file from our group/passwd files so that the YAIM utility functions like users_getvogroup work. There's surely an easier way of doing that?

This afternoon I'll redo the RB and try and run some jobs through as a gridpp VO member.