NEWS
Microsoft Tested Its Gateway Fix on the Wrong Machines
A July Microsoft 365 gateway fix, tested on weaker machines, collapsed Exchange Online when Teams failed over and capacity dropped.
A July software fix for Microsoft 365 gateway servers took down Exchange Online on November 12 after load tests on weaker, fewer-core machines missed a lock-contention defect.
At its lowest point, more than 80 percent of those servers stopped responding. Most users could not open a mailbox for just under 24 hours, from 4:20 p.m. UTC on November 12 to 4:10 p.m. UTC on November 13.
Three Conditions, One Global Mail Outage
Charles Lamanna, executive vice president for Copilot, Agents, and Platform, posted an abridged root-cause note on November 16. The same morning, the Microsoft 365 Status account said a preliminary post-incident report for admins was available under incidents EX1550543 and MO1550821.
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on… https://t.co/p8mUHxDe67
— Charles Lamanna (@clamanna) November 16, 2026
The November 15 report names three factors that hit the networking gateway in front of many Microsoft 365 services during the peak traffic window in the United States.
THE THREE FACTORS MICROSOFT NAMED
| Factor | What Microsoft said | Why it looked safe alone |
|---|---|---|
| Latent July defect | Lock contention inside an internal Windows process exhausted CPU and memory | Passed load testing and ran worldwide for more than three months |
| Teams failover | About fifteen hours earlier, Teams moved onto the primary gateway that also serves Exchange Online | Described as part of normal operations |
| Lost capacity | Nearly 14 percent of gateway server capacity was already out after an unrelated issue the previous day | Microsoft said the service still had enough headroom |
Microsoft said each factor looked safe on its own. The Teams move is part of normal operations, and the missing capacity had enough headroom. The July change had already run worldwide for more than three months with no observed impact.
The fix was prepared specifically for Microsoft 365 networking gateway servers, then pushed through safe deployment over several weeks starting in July. Other Microsoft 365 workloads that talk to Exchange, including Copilot, Defender, Purview, Fabric, Intune, Planner, Power Apps, To Do, Power BI and Universal Print, felt knock-on effects even when their own apps stayed up.
Microsoft Tested the Fix on Weaker Hardware
The defect caused lock contention inside an internal Windows process, which burned CPU and memory until the machine stopped answering. More cores made that wait worse, because more threads piled onto the same lock. Microsoft’s labs had been exercising a different traffic mix on weaker hardware SKUs with fewer cores.
The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores.
Microsoft, Preliminary Post-Incident Report, November 15, 2026
A lab that runs quieter, smaller-core machines cannot show a bug that appears when more cores fight over the same lock. That is how a fix can look clean in July and still take Exchange Online down in November, on the newer kit that actually answers customer connections.
Lamanna wrote that neither the underlying defect nor the fix applies to broader Windows deployments. The code path lived on Microsoft’s own gateway fleet, which is why a rollback later could not be a simple Windows patch for customers.
A Cascade Across Thousands of Capacity Units
At 1:07 a.m. UTC on November 12, automated failover moved Teams traffic onto the front-end servers shared with Exchange Online. Load rose through the day, and isolated server failures began in some regions, with automated mitigations already firing. At 4:03 p.m. UTC a connectivity monitor flagged more end-user failures, and at 4:17 p.m. automation began sending traffic to other endpoints that still looked healthy.
Those endpoints carried the same July code. Redirected load made the contention more likely on the machines that remained, so the same amount of work landed on fewer servers. In less than two hours the defect had been hit across the fleet.
The gateway is built from thousands of capacity units, shards of hardware and networking grouped into regions, with bulkheads that are supposed to keep a failure inside one partition. A separate issue stopped those bulkheads from holding, which is how a local contention problem became a global gateway outage.
Servers that were brought back did not stay up. Waiting traffic hit them at once and they stopped responding again. Microsoft said its normally automated recovery processes were not designed to restore service from this broad unresponsive state, so the system that usually reroutes around a bad box became the mechanism that spread the failure.
Why Exchange Online Could Not Fail Away
Exchange Online cannot leave the primary networking gateway, Microsoft said, for scale reasons. Teams, OneDrive, SharePoint and Office apps did leave, which is why chats and meetings worked for most of the outage while Outlook did not. Copilot, Defender and Purview were hit where they depend on Exchange, even when their own apps stayed reachable.
SERVICES THAT LEFT THE GATEWAY
| Service | Could leave the faulty gateway | What users saw |
|---|---|---|
| Teams | Yes | Chats and meetings worked for most of the outage |
| OneDrive | Yes | Restored earlier than mail |
| SharePoint | Yes | Restored earlier than mail |
| Office apps | Yes | Restored earlier than mail |
| Exchange Online | No, for scale reasons | Most users could not open mailboxes |
Microsoft 365 keeps two independent network gateway systems so many services can move from one to the other during a disruption. That design helped Teams get out. It did not help Exchange Online, which had to sit on the broken path until engineers could change the code under it.
At 4:20 p.m. UTC a high-priority alert for Exchange Online opened EX1550543 on the Service Health Dashboard. Five minutes later Microsoft confirmed trouble across several ways of reaching a mailbox, and the first customer support case arrived at 4:33 p.m. UTC. At 6:27 p.m. UTC it published MO1550821 for the wider set of affected services, including Teams, Copilot and Defender.
The July Change Was Too Old to Undo
Because the defective code had been live since July, normal rollback, flighting and feature flags could not quickly revert it. Engineers had to write an update that returned the Windows code path to its previously known good state, then take groups of servers out of service, apply the change, restart them and put them back, often 50 percent or more of a region at a time, starting with places that were in or entering business hours.
At 7:00 p.m. UTC on November 12 the company paused all ongoing and planned deployments and configuration changes. Extra capacity was added. Traffic was shifted. The boxes kept falling over. Fifteen hours after the Exchange alert, at 7:20 a.m. UTC on November 13, a mitigation that reverted the defective change finally went onto part of the fleet.
THE OUTAGE CLOCK IN UTC
- July 2026: A gateway fix begins deploying worldwide and later proves to carry the latent defect.
- November 12, 2026, 1:07 a.m.: Teams traffic moves onto the gateway shared with Exchange Online.
- November 12, 2026, 4:20 p.m.: Exchange Online incident EX1550543 opens after a high-priority alert.
- November 12, 2026, 7:00 p.m.: All ongoing and planned deployments and configuration changes are paused.
- November 13, 2026, 7:20 a.m.: An update that reverts the defective change is applied to part of capacity.
- November 13, 2026, 11:00 a.m.: Recovery actions complete across portions of Asia Pacific.
- November 13, 2026, 11:35 a.m.: Availability in Europe returns to healthy levels.
- November 13, 2026, 2:17 p.m.: Global webmail availability returns to expected levels.
- November 13, 2026, 4:10 p.m.: Microsoft records no further active customer impact.
- November 14, 2026, 4:15 a.m.: Final customer updates go out on EX1550543 and MO1550821.
Asia Pacific came back first, then Europe at 11:35 a.m. UTC, then global webmail at 2:17 p.m. UTC. The incident clock stops at 4:10 p.m. UTC on November 13. Microsoft dated the preliminary report November 15.
REPAIRS MICROSOFT SAYS ARE DONE
- Code path: An update restores the earlier connection-handling path in Windows on the gateway servers.
- Spare capacity: More capacity was added as a proactive step while the fleet was still unstable.
- Watching redirects: Monitoring of traffic-redirection patterns and connection-handling health was increased.
- Guardrails: New limits on traffic distribution were added to reduce the chance of another cascade.
Longer-term repairs are promised in the final post-incident review, which Microsoft scheduled for November 20, 2026, five business days after the impact window closed.
Azure Front Door Lost Its Rollback Snapshot in 2025
The November mail outage is not a copy of last year’s edge failure, but the undo problem rhymes. After an Azure Front Door outage on 29 October 2025, Microsoft’s own review said valid customer configuration changes, made across two control-plane builds, produced metadata that exposed a latent data-plane bug. Health signals stayed green long enough for the bad config to roll worldwide, and the last-known-good snapshot was updated with that same config, so a clean rollback was no longer sitting on the shelf.
Impact on that incident ran from 15:41 UTC on 29 October 2025 to 00:05 UTC on 30 October 2025. Engineers blocked further config changes, edited the snapshot by hand, and reloaded edge sites in stages so recovering nodes were not crushed by returning traffic. Microsoft later published lessons from those October outages, including longer bake time between rollout stages and a push to keep last-known-good actually good.
In November 2026 the stale object was not a snapshot of config. It was a July binary. Feature flags and flighting could not wind it back, so recovery again meant taking capacity down on purpose, patching it, and easing traffic back so the cure did not knock the patient over.
How Phishers Used the Outage Number
While mail queues were still catching up, attackers treated the incident ID as a prop. Microsoft Threat Intelligence described a phishing campaign exploiting the disruption on November 12, with messages impersonating Microsoft 365 service-health notices and citing the real Exchange Online incident EX1550543.
PHISHING MARKERS FROM THE OUTAGE WEEK
- Lure: Some mails claimed messages held during the outage would be deleted within 24 hours unless the recipient clicked to restore them.
- Subject line: Observed subjects followed the pattern Outage, then a six-character code.
- From names: Display names used organization-specific prefixes plus Outage Notification or Service Health.
- Domains: Senders used moments-notice.com, planetmaison.com and newbooker.com, with SendGrid click-tracking that redirected to reallifeaccounting.com, including the path /themelist0.html.
Microsoft classed the activity as credential phishing, though it had not confirmed stolen passwords or malware delivery. The practical check it asked of security teams was simple: confirm a real disruption in the Microsoft 365 admin center, and do not use a restore link that arrived by mail during an Exchange incident.
The final review is due on November 20. That document still has to say how the bulkheads failed, and whether the next load test will run on the same class of machines that carry live mail.
Frequently Asked Questions
Which Outlook Connection Methods Failed During the Outage?
At 4:25 p.m. UTC on November 12 Microsoft confirmed customers were hitting errors across Outlook, Outlook on the web, Exchange ActiveSync, and Messaging and REST APIs, so the break was not limited to one desktop client. Severity still varied by region, connection method and time, and some people saw only intermittent access while the gateway was degraded.
How Long Did Microsoft Hold Delayed Mail?
Microsoft kept an extended 48-hour email timeout window in place so delayed messages were not bounced as undeliverable while delivery queues cleared. The preliminary report also lists a longer email delivery window among completed repairs, and it says queued mail was processed once the known-good code path was back.
Does the Gateway Bug Affect Ordinary Windows PCs?
No. Microsoft said neither the underlying defect nor the fix applies to broader Windows deployments, because the change was prepared specifically for Microsoft 365 networking gateway servers. A company PC or another Windows server that is not part of that gateway fleet was not running this code path.
Was the Microsoft 365 Outage a Cyberattack?
Microsoft said the outage was a technical issue not related to cyber activity or to AI and agents, and the preliminary report describes a software defect combined with load and lost capacity. That finding sits beside a separate fact from the same week: criminals sent fake service-health mail that cited the real incident number while recovery was still underway.
-
NEWS3 months agoBoston Scientific Cyberattack Leaves Hospitals Waiting on Implants
-
GAMING4 weeks agoXbox Sells Cloud Playtime Packs Ahead of November Caps
-
NEWS3 months agoGuizhou Cave Cafes Spread When Each Cavern Fills
-
GAMING3 months agoThe Second Ned Stark Kit Lands When a Banner Dies
-
NEWS2 months agoSatellites Saw Nepal’s Ice Slope Move Before the Flood
-
NEWS3 months agoOpenAI’s Jalapeño Chip Turns Watts Into Faster Agent Steps
-
NEWS3 months agoMLB Quiet Periods Copy a Freeze Other Leagues Use
-
NEWS3 months agoRoman Space Telescope Launches After a Decade-Long Budget Bet
