A baffling spot of localised cooling

How do you keep your cool in this sort of weather? Well, there's various options, but I'll bet one you've not tried is wrapping up in lots of insuating foam.

And yet, that's been just the ticket for some worker nodes up here; despite it being one of the warmer days (23° C outside). Have a look at the temperature graph, and see if you can spot when something changed:
(The peak at midnight was due to a sneak attack Hammercloud; it was just before 12 when I put in the insulation.)

I'd discovered that there's some empty head space at the top of the racks. In those racks were there's a network switch at the top, this wasn't doing much, but where there were worker nodes, the top node was a lot hotter than the node two down from it. That's a lot sharper change than I'd expected - it was noticeable by touching the metal cover on the front of the nodes. The theory was that hot air out the back of the nodes was being sucked forward over the top of the highest node (through the headspace), and then recirculated round, getting hotter, until the steady state of it was about 5 K hotter that the others.

So, it was time to do something about that. First couple attempts at stopping up the gap didn't have much effect, until I dug out a few bits of packing foam (that the nodes were shipped in). Being, of course, the correct width, and jut a bit taller than 1U, they fit snugly into the headspace.


And that foam baffle reduced the temperature; to the point that the node at the top of the racks are now at the lowest temperature since records began! (i.e. they were installed.) Counter intuitive, but that's the way air/heat flow goes sometimes.

Although these worker nodes are due for replacement, we're going to be reusing the racks themselves, so little things like this are good to know. It may be that this won't be a problem with the new worker nodes - or it might be the case that it'd be worse. Either way, forewarns is fore armed (and cooler).

Phew, what a scorcher!

Yes, summer has arrived, even in Glasgow, and with it the people of this great city (myself included) are transformed from a whiter-shade-of-pale, into something that can only be described as lobster-esque. We Celts do not tan well.

Alas, it is not all fun and games in the sunshine, because the arrival of fine weather heralds the inevitable air-conditioning problems.

Despite regular love and attention (serviced 3 times a year and recently hosed through with nitrogen) one of our roof-mounted compressors is particularly troublesome. This is most likely a combination of age (~12 years is the best guess) and 24 x 7 load; it serves the warmest corner of our original machine room.

For this reason, I have a site meeting on Monday to discuss options, one of which will hopefully involve the replacement of said compressor before we take delivery of new hardware later this year.

Also under consideration is a home-brew cold-aisle containment system. This will almost certainly be less sophisticated (and cheaper) than our excellent Knuerr racks in the basement, but should result in more intelligent use of the available chilled air.

Until a solution arises, we shall continue to nurse the existing system through the summer months, and take comfort from the fact that there are clearly worse air-conditioning failures a site can suffer...