A corrected update existed within roughly an hour and a quarter. It did no good on any machine that had already taken the bad one, because those machines were no longer completing a boot and could not fetch anything.
Remediation guidance published at the time required per-machine intervention. Recovery consequently ran for days, and in large estates longer.
Automation Reverses At The Worst Moment
Central management is what makes an estate of a hundred thousand endpoints tractable: one console, one policy, one push. It is also what delivered the faulty file to a hundred thousand endpoints at once.
And when the failure prevented boot, the same tooling was unavailable for the fix. The mechanism scaled the damage and did not scale the repair.
A Fleet Is Measured By How Fast You Can Touch All Of It
That is a capability almost nobody plans for or tests. Disaster exercises assume systems can be reached — the recovery runbook starts from a network that works and a management plane that answers.
This corpus records the fallback theme as manual procedures inherited from a pre-automation era that nobody tracks. Here the fallback needed was not a paper form; it was enough people to walk to every device, and headcount is exactly what the automation was bought to reduce.
The Distribution Asymmetry Generalises
Anything that can push a change to a fleet in minutes can push a bad change to a fleet in minutes, and cannot necessarily pull it back. That applies to endpoint agents, configuration management, firmware and mobile device management alike.
Graded medium: the sequence and the need for per-machine work are consistently reported, but this desk has not reviewed the vendor’s own remediation documentation and cites no recovery duration as established.
Compiled from contemporaneous reporting, listed below. The characterisation of remediation as requiring per-machine intervention is as reported; this desk has not reviewed the vendor’s remediation guidance directly and gives no figure for recovery duration. The generalisation about push-based management is this desk’s argument. Graded medium. Corrections: corrections@forensicpost.com.