Nobody Outside The Room Knew When The Outage Ended

The technical side of that morning was one of our better performances. An engineer spotted the failing component within four minutes of the first alert, understood it within ten, and had service restored twenty minutes after the page. By any measure we publish, it was a short outage handled well.

The business remembers it as a two hour outage, and they are not being unfair.

Nobody told them. The people who could have sent an update were the same people holding the problem, and while you are holding a problem you do not stop to write to four hundred users. The service desk took ninety calls from staff who had no idea whether the system was broken, their laptop was broken, or they had done something wrong. Two department heads made decisions based on a rumour that the outage would last all day, one of which sent a delivery team home. And when service came back at twenty past eight, we did not say so, because from inside the room it was obvious. People kept trying, failing on cached pages, and assuming it was still down for another hour and a half.

The gap was not a tooling gap. We had a status page. It had not been updated in fourteen months because updating it was nobody's job during the only occasion it mattered.

What we changed is small and it has held. Any incident above the lowest severity now has two named people from the first minute: one who owns the fix and one who owns the telling, and the second is not allowed to also be the first. The communicator sends something within ten minutes whether or not there is anything to say, because silence gets filled with a worse story. Updates go out on a fixed cadence, in the channel people actually read rather than the one we would prefer. And recovery is announced explicitly, with a note about what users may need to do, because the end of an incident is not self-evident to anyone outside the room.

Restoring the service is half the job. The other half is making sure the organisation knows it has been restored, and it is the half we were measuring least.

– Serguey Shinder