Microsoft reveals three causes behind global Microsoft 365 outage

Legacy software bug, traffic surge and reduced server capacity combined to trigger cascading failure

Microsoft has completed its preliminary investigation into the global Microsoft 365 outage last week.

The hyperscaler issued an initial report which identifies three issues which combined to create the outage: a legacy software defect, increased network traffic, and reduced server capacity. Microsoft executives said that whilst each of these problems could have been dealt with in isolation, the combination of the three was much harder to mitigate.

The incident locked Microsoft users out of a wide range of cloud services, including AI assistant Copilot, Exchange Online, Teams, OneDrive, SharePoint, and the Office suite of productivity applications.

In a detailed post on X, Charles Lamanna, EVP Copilot, Agents, and Platforms at Microsoft, set out the details of the three main problems which combined to cause such chaos for which Microsoft was “deeply sorry.”

Three issues combined to cause the Microsoft 365 outage

The first was a hidden bug in gateway software which caused servers to lock up under heavy load. It came from a fix written specifically for Microsoft 365’s gateways (not Windows more broadly) that rolled out in July and caused no trouble for three months. Microsoft’s testing missed it because labs tested on older, lower-spec hardware. On newer, multi-core servers the bug started causing problems.

The second problem was a sudden surge in traffic. Microsoft runs two separate gateways so services can switch between them if one struggles. About fifteen hours before the outage, Teams failed over to the main gateway – which, Lamanna’s post explains, is exactly what is supposed to happen. The problem is that Exchange Online relied on the same gateway, piling extra load onto it at exactly the wrong time.

The final issue was that fewer servers were available than usual. Around 14% of the gateway’s capacity was offline after an unrelated problem the day before and was on its way back. There was plenty of headroom, so on its own it wasn’t a remarkable event.

These three conditions combined during peak US traffic hours and resulted in requests piling up until servers just stopped responding.

Failover systems triggered a cascading failure

Normally Microsoft’s traffic management systems would redirect requests to normally functioning servers, but the redirected traffic overloaded those failover servers and triggered a cascading failure reaching up to 80% of Microsoft’s networking gateway servers at its peak.

Microsoft also experienced a failure of the partitioning which should have contained the failure to a single region. According to Lamanna:

“In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.”

Why Exchange Online took longer to recover

Email took longer to restore than other Microsoft services because whilst most services such as Teams and Office applications could fail over successfully, Exchange Online could not. As soon as servers managed to get back online they were overwhelmed with traffic to such an extent that they fell over again. As Lamanna says:

“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state.”

Microsoft says its final Post-Incident Review will provide more details later this week.

The outage has triggered another bout of soul searching over the extent of reliance of US-based cloud services. The UK's public sector is particularly reliant on Microsoft, having spent the last 15 –20 years pursuing a 'cloud first' strategy.

James Stewart, Former UK Government Deputy CTO, and CTO at Public Digital said:

“The outage revealed how little some large organisations understood their dependence on Microsoft. They arrived there one small decision at a time: renewing a contract or taking up another service bundled into the package, without stepping back to consider how much of their business now rested on a single provider.

“Sovereignty starts with knowing what you rely on, how exposed you are when it fails and what options you have. The imperative for organisations now is to take stock and act.”