Grid-Monitoring Nagios tests

Finally (after a long interval) I reinvestigated getting the LCG Grid Service Monitoring Working Group nagios tests installed at Glasgow.

I had tried once before, but it needed nagios on our UI. This time I added a UI to our nagios host (nice n simple - simply added the hostname into the relevant UI group in cfengine). Works fairly well - I've got it installed and polling the SAM tests via the sam-API and picking up the results. I still need to get certificate proxy renewals working, and merge the records together with our existing definitions for the hosts (we use non-qualified names, the wlcg.cfg used FQDNs)

As the screen shot shows - We've already got a lot of green - and if we can nail the cert problems I'll switch it over to use the normal notification system

Scalpel, Mr Elwell please...

We started the re-install of svr031 this morning shortly before 11. So far the patient is doing well, coming around after surgery before 1pm.

cfengine is running again. ganglia is installed and Andrew's hacking in the ram disk for the rrds.

So far, so good...

New Installer Christened: YPF is Born

Working on the new installer again today. With dnsmasq installed on svr031 the installer started to work fine. It is still using the text classes.conf and the move to SQLite might have to wait until svr031 is reinstalled - it's not clear it will work right now. (And today I was very pressed for time - just need to get things working!)

I felt we need a name for the installer. As it's written in python a flying circus name seemed to be appropriate and Andrew suggested "YAIM People's Front". Brilliant.

In the afternoon I added a new host to the cluster, using 2 cheepo netgear switches to connect to the internal and external cluster networks. The tools for manipulating the cluster database are very primitive right now - clearly there's a lot of work to be done here - but a minimal interface allowing an arbitrary SQL command to be issued suffices for now.

I have written some documentation in the wiki.

There's no real intention to make YPF a general installer project at the moment, but other sites might find aspects of it useful.

svr031 Installer Almost Working

Spent more time on the new infrastructure for svr031 again today. I wrote the script to get a dhcpd.conf file out of the cluster database, which was quite straightforward to do. Then dhcpd could function again.

I tried a test install and found that tftp was, amazingly, still working even though xinetd had lost all of its configuration.

Then the install fell over slightly - the kickstart installer was using the old internal hostname of the svr031 server, master.beowulf.cluster, but setting the name server resolution to the university DNS servers. As an installing machine does not yet have the big cluster /etc/hosts file it could not resolve this name. So I changed the kickstart setup to code in only the IP address of the svr031, so no name resolution was needed. This allowed the install to proceed, however then I discovered some cleverness in the kickstart post install script which ensures that even on first boot the machine sets its hostname correctly (in particular, to the routed hostname for grid and disk servers) and that this relies on DNS to function.

So, I either have to rewrite the clever code, or go back to running DNS on svr031.

I decided that running DNS on svr031 was no bad thing. There's a lovely lightweight DNS server called dnsmasq - it loads up a DNS server with the contents of /etc/hosts, which is generated from the cluster database already, and can give out DNS during the install process.

After a machine has first booted, cfengine will copy in the global cluster /etc/hosts and set the DNS to the university servers to remove the single point of failure. But having internal DNS for the installs is nice.

Even better I found dnsmasq is built for RHEL4 x86_64 in the DAG repository, so installing it was a cinch.

Finally, dnsmasq also includes a dhcp server and a tftp server, so I will have a look at running these and possibly simplifying life even more.

Installs should be back up and running tomorrow.

New Fabric Management Schema Designed

I spent most of the day with svr031, desigining and implementing the new fabric management database system. Paul and Andrew reviewed my initial schema and suggested some improvements - the current schema is now described in the wiki.

Then I had to write a couple of scripts to take the extant information from the CVOS configuration files, and to add extra information for routed hosts, and then populate the database.

After that was done, I was able to write the first utility script, to extract an /etc/hosts file from the database. This is improved over the version recovered from DNS so I gave it to cfengine to distribute across the cluster.

Next stop, getting a working dhcpd.conf file, so that we can being new installations again.

This is useful and essential work, but bloody tedious at some level...

svr031 recovery ongoing

There's been sadly little time to work on svr031. However, now work with our nuclear physicist community seems bound to involve the installation of new machines on the cluster, so things have to begin in earnest.

I have recovered the lftp mirror scripts, so that's a start here.

Next I have to look at making a proper SQLite database of machines and extracting a dhcpd.conf from that. Fortunately the tftp root was not lost, so that part of the boot process should be ok.

Minor Cluster Updates

In anticipation of losing svr031 for a few days - and thus any easy way keeping the cluster up to date - I've been patching us up to the latest gLite release, r13.

There seems to be no good way of doing this other than a "yum update" - just updating the metapackage seems to not encapsulate all of the installed, but updated, RPMs, e.g.,

node001:~# yum update
[...]
I will do the following:
[update: edg-mkgridmap-conf 2.7.0-1_sl3.noarch]
[update: glite-config 1.8.4-0.noarch]
[update: glite-lb-common 3.0.6-1.i386]
[update: edg-mkgridmap 2.7.0-1_sl3.noarch]
[update: glite-rgma-command-line 5.0.4-1.noarch]
[update: glite-WN 3.0.13-0.noarch]
[update: glite-security-gsoap-plugin 1.2.5-0.i386]
[update: lcg-ManageSoftware 2.0-6.noarch]
[update: glite-rgma-api-python 5.0.10-1.noarch]
[update: glite-rgma-base 5.0.7-1.noarch]
[update: glite-wms-common 1.5.14-1.i386]
I will install/upgrade these to satisfy the dependencies:
[deps: lcg-tags 0.2.1-1.noarch]

But updating only the metapackage gives a palrty:

node001:~# yum update glite-WN
I will do the following:
[update: glite-WN 3.0.13-0.noarch]
I will install/upgrade these to satisfy the dependencies:
[deps: lcg-tags 0.2.1-1.noarch]

I think there's an argument for running a "yum update" nightly on the worker nodes, but it still seems far too dangerous a thing to do on the servers. Too much of a risk of daemons not restarting properly or java being arsed up.

I also added the new VOMS certificate for dzero - using cfengine this is easy. Unfortunately, not everything is yet cfenginified: machines currently not under cfengine control: svr019 (MON), svr021 (site BDII), disk servers. Should try and address this soon.

svr031 removed from active cluster role

It's taken a while, but svr031 has now been taken out of active service.

All machines have had /etc/hosts and /etc/resolv.conf files put on them, which takes care of internal cluster name resolution, and they have had their DNS servers pointed at the normal university ones.

In addition one of the nat hosts was setup as a gateway machine for the worker nodes. The workers were (carefully) told to use this new host as their default gateway.

So, nothing is relying on any services provided by svr031 and we can prepare for a reinstall next week.

I had to do a bit of resuscitation of svr031, so that one can at least login via ssh and scp files to and from it. I've restored some of the library paths to get auxiliary commands working.

The cluster itself has been remarkably untroubled by svr031 being in a tizzy - I thought about putting us into a precautionary downtime, but this has not been necessary. We've carried on passing the SAM tests without trouble.

Leaders? Who needs 'em?

The President's Brain Is Missing...

svr031, the Glasgow cluster headnode, is currently a bit FUBAR.

Grieg was mirroring the dcache repository from DESY, becasue the mandatory webcaching policy at Glasgow stuffs up yum big time, hence any repos need to be got locally for installation to work. However. his mirror script has went crazy and managd to wipe the whole of /etc. Arggg!

Backup, what backup? It's a RAID 5 disk, we didn't need a backup (woops)...

svr031's roles in the cluster are:

• NAT box for WNs
  
• DNS server for whole of the cluster
  
• Central syslogger
  
• DHCP server
  
• tftp server for kickstarting
  
• http server for installation, nagios and ganglia
  

The critical run-time services are NAT and DNS. Fortunately the DNS server on svr031 and the NATing are still working, so even though the president's brain is missing the organs of the state still function, for now.

The immediate things to be done are to remove svr031's run time functions from the cluster. This will consist of:

1. Generate and copy an /etc/hosts file with a complete set of entries for the internal cluster machines - so that the internal DNS is not required.
   
2. Update /etc/resolv.conf on the cluster to use the standard university DNS servers
   
3. Setup NAT via the dedicated NAT boxes
   

After those steps have been taken our dependancy on svr031 should be removed, and we can work on reinstalling it and re-establishing its other services.