SysAdmin Weekly - 050 - How Do You Run a Blameless Incident Postmortem?

[asyrewicze]

Hey everyone!

This week Eric Siron and I tackle a topic that is not the most glamorous in IT but might be one of the most important once you have been in the trenches long enough: the incident postmortem process. Something breaks, you scramble, you get it back online, and then comes the part nobody really prepares you for. We talk through how to run that process so it actually makes things better instead of turning into a blame game.

What we get into:

• What kind of incident earns a postmortem and what does not (the SQL service that blips for two seconds probably does not)
• How the process scales from a sixty-person war room down to a one-person writeup for a nervous boss
• Who belongs in the room, including the political and visibility reasons people get pulled in
• Keeping it blameless without letting it become unaccountable, and what to do when human error really is the root cause
• The information to gather first: timeline, logs, who was impacted, and the dollar-impact the bean counters will ask for
• Why follow-through is the entire point, and how to keep action items from rotting
• Bonus: using the same framework to hold your software vendors accountable when their support team blames the first error in the log

I also put together a free Markdown postmortem template. No login, no catch, grab it and make it your own. It lives in the show repo now, so it has proper version history and you can suggest changes with a pull request.

Relevant Links Below!

Postmortem Template (GitHub)
Podcast Episode - YouTube
Podcast Episode - Spotify

Question for the board: what is the one field or section your postmortem template has that you wish every other team used? Or if you have lived through a real witch-hunt postmortem, how did you handle it?