On Friday the 29th of October, the ScotGrid, Glasgow site was impacted by two power outages at 15:25 and 15:40. These power cuts weren't localised to just the ScotGrid Glasgow site but also impacted other parts of the west end of Glasgow. These outages resulted in the site being placed in unscheduled downtime as we wanted to ensure that the power feed into the site was stable prior to returning the site to full production.
On Monday the 1st of November we re-checked all essential core services, boosted our UPS capability and then re-checked all services were functioning correctly prior to the site re-entering full production.
By 17:15 on Monday night we were expecting ATLAS jobs and the site is now back to a normal functioning basis.
Interestingly enough our new 10 Gig Core re-acted as planned and rebooted in full operational mode minutes after each outage and was completely stable over the weekend, the new cluster equipment was also functioning correctly after both outages. In addition to this the older cluster equipment was not badly affected by these power losses either.
The site is now getting back to a normal functioning status.
No comments:
Post a Comment