At the lowest point of the Microsoft 365 outage on Thursday, November 12, more than 80% of the servers in the networking gateway that sits in front of Exchange Online had stopped responding, and Exchange Online, which has no way around that gateway, stayed down while Teams meetings largely carried on. Microsoft Executive Vice President of Copilot, Agents and Platform Charles Lamanna wrote on X on Monday that three main factors "together caused a global outage for the networking gateway that fronts many Microsoft 365 services," and his account traces the failure back to a defect first deployed in July.
Shortly before, the Microsoft 365 Status account said Microsoft had released its preliminary post-incident report in the Microsoft 365 admin center under incidents MO1550821 and EX1550543. Lamanna, whose post gives an abridged version of the root cause and points to the admin center for more detail, opened with an apology.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
He then set out how the outage began months earlier, with a change made specifically for the gateway's own servers.
A defect that ran from July
The change began deploying in July and, by Lamanna's account, ran worldwide for more than three months without impact. It came from a fix prepared specifically for Microsoft 365 networking gateway servers, and he said neither the defect nor the fix applies to broader Windows deployments. Inside it was a latent defect that caused servers "to become overwhelmed and unresponsive under certain load conditions," because of lock contention inside an internal process. Normal load testing did not detect it, Lamanna explained, because the contention is worse on newer machines with more cores, whereas Microsoft's load testing tended to focus on weaker hardware SKUs with fewer cores.
A Teams failover and missing capacity
The load conditions that exposed the defect built up in the hours before the outage. Microsoft runs two independent gateway systems within Microsoft 365 so that services can move between them, and about fifteen hours before the outage Teams failed over from the alternative gateway to the "primary" gateway that also supports Exchange Online. Lamanna said the move increased the load on that gateway and changed its traffic patterns, and he described this type of failover as "part of normal operations."
At the same time, nearly 14% of the gateway's server capacity was out of service because of an unrelated issue from the previous day. Lamanna said that reduction had no impact on its own because the service had enough headroom, but combined with the extra load it brought the July bug to the surface during peak US traffic.
How the failure spread across the fleet
Microsoft's traffic management normally routes users away from servers that lose connectivity, but this time the redirected traffic made the same failure more likely on the servers that remained. Lamanna described a cascading failure from server to server, in which the defect was encountered across the fleet in less than two hours. At the lowest point, more than 80% of servers were unresponsive. The bulkheads between the gateway's "capacity units" would normally contain such a failure, but he said a separate issue stopped them doing so, which let the impact spread globally.




