Ctg sit23 hub banner.jpg

Google forced to switch off London data centre to save it during heatwave

Google wasn't the only company affected - Oracle also switched off servers in the heat

Image:

Google wasn't the only company affected - Oracle also switched off servers in the heat

Redundant cooling systems packed up, forcing the shutdown

The UK's recent heatwave, where temperatures hit 40°C, was behind the outage that affected Google's services in its London-based europe-west2-a zone last month.

The company's was forced to shut its London data centre during the record-breaking temperatures to prevent further downtime, as its cooling systems had failed.

Although the incident report doesn't specify why the cooling systems malfunctioned, it says Google first learned of problems with two cooling systems on Tuesday 19th July. On that day temperatures in several regions of the UK reached record highs of 40°C, with London hitting 39°C in the late afternoon.

Due to the simultaneous failure of several redundant cooling systems and the very high outdoor temperatures, one of the data centres was unable to maintain a safe operating temperature.

Google engineers worked to fix the failed system, but were unsuccessful. With temperatures remaining above 35°C into the evening, Google decided to shut part of the europe-west2-a zone to avoid an even longer outage or equipment damage.

From that point on, a number of services were essentially inaccessible across the Europe West 2 zone. All virtual machines running on Google's Compute Engine, Persistent Disk (PD) and Google Cloud Storage were affected, which led to instance terminations, networking problems and service degradation.

Google also appears to have botched its attempt to provide redundancy. It unintentionally changed routing for internal services at the beginning of the problem such that traffic avoided all three zones in the europe-west2 region, rather than just the impacted europe-west2-a zone.

That meant users were no longer able to access data from regional storage services like GCS and BigQuery.

Google managed to bring the cooling system came back online at about 10PM. The total impact on cloud services was estimated at 18 hours and 23 minutes.

The company has vowed to take measures to insure against this type of failure again, which include fixing and re-testing failover automation to ensure stronger resilience.

Google was not the only company that was forced to close its data centre during the UK heatwave; Oracle servers also experienced cooling-related failures as a result of the record-breaking temperatures.

Modern data centres are designed to avert potent disasters, but existing infrastructure in the UK has not been built to withstand such intense temperatures.

In 2020, Microsoft tested an undersea data centre near the Orkney Islands in the North Sea as part of an experiment to use natural resources as passive cooling solutions.

Scientists have warned that intense heatwaves could occur more often, so tech firms are scrambling to discover greener alternatives that use less energy and produce less heat.

Computing says:

Google is doing what it can to protect its data centres in the short term - but as temperatures continue to climb around the world due to climate change, the existing tech will only go so far. New data centre innovations (like Microsoft's sunken facility in Orkney) will be necessary to insulate - both literally and metaphorically - against the rising heat.

Google forced to switch off London data centre to save it during heatwave

Redundant cooling systems packed up, forcing the shutdown

Computing says:

Green

Cloud big three sign open letter urging datacentre kit suppliers to step up

Public Sector

National Grid is turning analogue to digital - Ctrl Alt Lead podcast

Law

Indian antitrust regulator finds Apple's App Store abuses market power