One XOS, Great Big Purple Packet Eater. Sure looks good to me.

So we haven't been blogging a great deal since December and for good reason. We found ourselves in the exciting position of being given additional funding to enhance our network capability and also we had additional equipment to install into the cluster.

First things first however, as you may have read we have had no end of issues with the older network equipment. We had a multi-vendor environment which, while adequate for 800 analysis jobs and 1200 production jobs, wasn't quite up to cutting the mustard as we couldn't expand from there.

The main reason was the 20 Gig link between the two computing rooms which was having real capacity issues. Also, add in issues between the Dell and Nortel LAG and associated back flow problems, sprinkled with a buffer memory issue on the 5510s and you get the picture. In addition to this we were running out of 10 Gig ports and therefore couldn't get much bigger without some investment.

Therefore, the grant award was a welcome attempt to fix this issue. After going to tender we decided upon equipment from Extreme Networks. The proposed solution allowed for a vast 160 Gigabit interconnect between the rooms broken into two resilient link bundles in the Core and an 80 Gigabit Edge layer. In addition to this connection we also installed a 32 core OM4 grade fiber optic network for the cluster which will carry us into the realms of 100 Gigabit connections, when it becomes available and cheap enough to deploy sensibly.

We now have 40 x 40 gigabit port, 208 x 10 gigabit ports and 576 1 x Gigabit ports available for the Cluster.

 There is quick and clever and here it is

The new deployment utilises X670s  in the Core and X460s at the Edge.

The magic of the new Extreme Network is that it uses EAPS, so bye bye Spanning Tree and good riddance as well as MLAG which allows us to load share traffic across the two rooms so having 10 Gigabit connections for disk servers in one room is no longer an issue.

Then it got a bit better. Due to the Extreme OS we can now write scripts to handle events within the network which ties in with the longer term plan for a Cluster Expert System (ARCTURUS) which we are currently designing for test deployment. More on this after August.

Finally, it even comes with its own event monitoring software, Ridgeline which gives a GUI interface to the whole deployment.

We stripped out the old network installed the new one and after some initial problems with the configuration, which were fixed in a most awesome fashion by Extreme got the new one up and running. What we can say is that the network isn't a problem anymore, at all.

This has allowed us to start to concentrate upon other issues within the Cluster and look at the finalised deployment of the IPV6 test cluster which has benefited in terms of hardware from the new network install. Again, more on this soon.

Right, so now to the rest of the upgrade we have also extended our cold isle enclosure to 12 racks, have a secondary 10 Gig link onto the Campus being installed and have a UPS. In Addition to this we refreshed our storage using Dell R510s and M1200s as well as buying 5 Interlagos boxes to augment the worker node deployment.

 The TARDIS just keeps growing

We also invested in an experimental user access system with wi-fi and will be trying this out in the test cluster to see if a wi-fi mesh environment can support a limited number of grid jobs.  As you do.

In addition to this we improved connectivity for the research community in PPE at Glasgow and across the Campus as a whole, with part of the award being used to deliver the resilient second link and associated switching fabrics.

It hasn't been the most straight forward process as the decommissioning and deployment work was complex and very time consuming in an attempt to keep the cluster up and running as long as possible and to minimise down times.

We didn't quite manage this as well as expected due to the configuration issues on the new network but we have now upgraded the entire network and have removed multiple older servers from the cluster to allow us to enhance the entire batch system for the next 24 - 48 months.

As we continue to implement additional upgrades to the cluster we will keep you informed.
For now it is back to the computer rooms.

Attack of the Clones

So, I finally got round to booting the other three new servers we'd recently purchased (these are nice simple dual core, dual CPU boxes (Dual-Core AMD Opteron(tm) Processor 2216) on Tyan Thunder n3600M (S2932) Motherboards)

I'd always been exceptionally suspicious of the PXE MAC address thet the 1st one offered (60:50:40:30:20:10) rather than the imprinted one on the rj45 socket starting with 00:E0:81 but the fact that three out of four machines all had the same mac address meant a phone call to the vendor was called for.

The one that works has the 2.01 BIOS, the others were already shipped with 2.02 and downgrading didn't cure the amnesia. I don't see a utilty to reburn the mac address ont the machine, so have left that in the capable hands of the vendor for now. Googling seems to suggest that nvidia cards are somewhat prone to wierdisms like this. Bother.

p p p pick up a pakiti

We've been using pakiti at Glasgow for some time now for keeping an eye on which nodes are out of date. One minor niggle is that it doesn't keep track of the grub default kernel (ie what should come in on reboot) compared to the running kernel

We already had a v simple shell script that did that:

 pdsh -w node[001-140] chkkernel.sh | dshbak -c
----------------
node[001,005,007,014,016-020,022-023,025,028,031-061,063-085,087-090,092,095-096,098-101,103-104,106-107,109-110,113,115,118-120]
----------------
 Running: 2.6.9-67.0.7.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status OK
----------------
node[062,091,093-094,097,102,105,108,111-112,114,116-117,121-127,129,131,133-134,136-139]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[003,009,011,013,015,021,027,029]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[128,130,132]
----------------
 Running: 2.6.9-55.0.12.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status error
----------------
node[002,004,006,010,012,024,026,030,140]
----------------
 Running: 2.6.9-55.0.9.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error
----------------
node[086,135]
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.4.ELsmp, Status OK
----------------
node008
----------------
 Running: 2.6.9-67.0.4.ELsmp, Grub: 2.6.9-67.0.7.ELsmp, Status error

but I finally got it integrated with some patching - see http://www.scotgrid.ac.uk/wiki/index.php/Pakiti

result - pretty green / red status on the "default kernel' column.

The patches have been emailed to Romain so may well appear upstream eventually

Brain transplant leftovers

Bah! yum update has failed to get the headers (404) when tested on a client node and the pxe/dhcp/tftp part of the build process still needs working on. We've also got 2 nodes that require a re-image to bring them under the full control of the build system. Looks like today is going to be spent making sure the YPF is working properly.

Stuff that needs to be done:
* clusterdb to dnsmasq - it should spit out the MAC addressess to file ready to use.
* Check all the apache aliases for build
* check the autokick stuff.

then reimage machines. Hopefully that'll reduce the no of red bits in nagios down to a less boss-scaring amount. Once thats done we can work on removing the false alarms (where we monitor non-existing services that we know are't configured yet)

Server Spread

Scotgrid-Glasgow continues to grow. Not only do we have the vobox for a user group, but we now have a brace of test Scientific Linux 5 servers kickstarted and ready to go. Changes that we needed to make to our existing setup were minimal, the main one being the need to add noipv6 to the PXE command line, otherwise the installer hunts (and fails) for an IPv6 address by DHCP.

I have also added a "sl5" flavour to our local repository for Misc Stuff RPMs (such as cfengine) in both i386 and x86_64. One niggle is that you have to run createrepo once in each location if you use the $basearch variable in the repo config source.

Next step - install apache onto one of the boxes, tweak config and see if it has all the correct mirroring arrangements and replace the brain dead svr031 versiom.

New Installer Christened: YPF is Born

Working on the new installer again today. With dnsmasq installed on svr031 the installer started to work fine. It is still using the text classes.conf and the move to SQLite might have to wait until svr031 is reinstalled - it's not clear it will work right now. (And today I was very pressed for time - just need to get things working!)

I felt we need a name for the installer. As it's written in python a flying circus name seemed to be appropriate and Andrew suggested "YAIM People's Front". Brilliant.

In the afternoon I added a new host to the cluster, using 2 cheepo netgear switches to connect to the internal and external cluster networks. The tools for manipulating the cluster database are very primitive right now - clearly there's a lot of work to be done here - but a minimal interface allowing an arbitrary SQL command to be issued suffices for now.

I have written some documentation in the wiki.

There's no real intention to make YPF a general installer project at the moment, but other sites might find aspects of it useful.

svr031 Installer Almost Working

Spent more time on the new infrastructure for svr031 again today. I wrote the script to get a dhcpd.conf file out of the cluster database, which was quite straightforward to do. Then dhcpd could function again.

I tried a test install and found that tftp was, amazingly, still working even though xinetd had lost all of its configuration.

Then the install fell over slightly - the kickstart installer was using the old internal hostname of the svr031 server, master.beowulf.cluster, but setting the name server resolution to the university DNS servers. As an installing machine does not yet have the big cluster /etc/hosts file it could not resolve this name. So I changed the kickstart setup to code in only the IP address of the svr031, so no name resolution was needed. This allowed the install to proceed, however then I discovered some cleverness in the kickstart post install script which ensures that even on first boot the machine sets its hostname correctly (in particular, to the routed hostname for grid and disk servers) and that this relies on DNS to function.

So, I either have to rewrite the clever code, or go back to running DNS on svr031.

I decided that running DNS on svr031 was no bad thing. There's a lovely lightweight DNS server called dnsmasq - it loads up a DNS server with the contents of /etc/hosts, which is generated from the cluster database already, and can give out DNS during the install process.

After a machine has first booted, cfengine will copy in the global cluster /etc/hosts and set the DNS to the university servers to remove the single point of failure. But having internal DNS for the installs is nice.

Even better I found dnsmasq is built for RHEL4 x86_64 in the DAG repository, so installing it was a cinch.

Finally, dnsmasq also includes a dhcp server and a tftp server, so I will have a look at running these and possibly simplifying life even more.

Installs should be back up and running tomorrow.

New Fabric Management Schema Designed

I spent most of the day with svr031, desigining and implementing the new fabric management database system. Paul and Andrew reviewed my initial schema and suggested some improvements - the current schema is now described in the wiki.

Then I had to write a couple of scripts to take the extant information from the CVOS configuration files, and to add extra information for routed hosts, and then populate the database.

After that was done, I was able to write the first utility script, to extract an /etc/hosts file from the database. This is improved over the version recovered from DNS so I gave it to cfengine to distribute across the cluster.

Next stop, getting a working dhcpd.conf file, so that we can being new installations again.

This is useful and essential work, but bloody tedious at some level...