PostPublius
U.S. national editionReporting checked

Technology · Column

Run the Failure Before It Runs You

Three big systems faltered in five days this month. Their operators say none was attacked, and two of the failures began with a change to a live system.

By Victor Julian Hale · Written

Three big systems faltered in five days this month. None of them, their operators say, was attacked, and that’s the more useful news.

On November 12, Microsoft’s email service went down long enough that Maine’s courts told anyone who had emailed a protection-order request after 12:30 p.m. to send it again or file a paper copy. Microsoft’s post-incident report blames a defect in a server update it began rolling out in July. The defect sat quietly until load and traffic shifts woke it.

On November 13, the FAA stopped departures nationwide. The Transportation secretary said a contractor was replacing copper with fiber at a remote site when a router overloaded the national network. The FAA says data flooded its network from a radar site in Amarillo, Texas, and a radio site in El Dorado, Arkansas, where its network contractor had installed a new router. It has paused installing that model of router while the root cause is investigated.

On November 16, Chase blamed “a temporary issue related to a software bug” for payment delays. All three said they found no sign of a cyberattack.

With three failures in one week, it was tempting to look for one attacker. The causes disclosed so far give no reason to assign them a common one. Two of them, by their operators’ accounts, began with a change to a live system: a server update Microsoft started rolling out in July, and a new router a contractor installed at a remote FAA site. Chase has said only that a software bug was to blame.

That’s a habit worth breaking, and the fix is old and unglamorous. Break your own systems on purpose before someone or something else does. Pull the cable in a rehearsal. Switch off the gateway during a planned drill and watch what fails. Then publish what you learned, as Microsoft did for its customers, so they know the next fix is real.

There’s a fair objection. Rehearsing failure costs money and risks the outage you’re trying to prevent. That’s true, and it’s why a drill should be scheduled, small and staffed. The FAA itself now says it will rethink when it schedules routine replacement work. Unscheduled outages land at 3:30 on a Friday afternoon, with about 3,500 planes in the air.

The cheapest outage is the one you cause yourself, on a quiet Tuesday, with the right people in the room.