// Leadership
Most postmortems end with "be more careful". Here is how to write ones that find real contributing factors and stop incidents from repeating.
Md Nayeem HossainJun 9, 20253 min readMost postmortems I read early in my career followed the same arc: a timeline, a root cause that boiled down to "someone made a mistake", and an action item to "be more careful". Three months later the same incident happened again.
A good postmortem is not paperwork. It is the cheapest way an engineering team has to learn from expensive failures. Here is how I run them.
"Blameless" means we assume everyone acted reasonably given what they knew at the time. If a capable engineer could run a command that deleted production data, the problem is the system that made that command easy to run, not the engineer.
This is not about being nice. It is about getting accurate information. People who fear punishment hide details, and a postmortem built on half the facts produces fixes for the wrong problem.
Start the document within a day of the incident, while chat logs, dashboards and memories are still available. Assign one owner to drive it, but collect input from everyone who was involved.
Two or three sentences a busy executive can read: what broke, for how long, and who was affected.
Between 14:05 and 14:47 UTC, 38% of checkout requests failed after a configuration change exhausted the database connection pool. Roughly 1,200 customers saw errors; no payments were double-charged.
Be specific and quantitative: duration, number of users affected, failed requests, revenue at risk, support tickets, SLO budget consumed. Numbers make it possible to compare incidents and prioritize fixes.
Timestamped, factual, and including the moments of confusion:
114:02 Config change merged: pool size 50 → 20 (intended for staging)214:05 Error rate alert fires for checkout-api314:11 On-call acknowledges, suspects payment provider outage414:24 Provider status page is green; investigation moves to our side514:39 Connection pool saturation spotted on database dashboard614:44 Config reverted714:47 Error rate back to baselineThe gap between 14:11 and 14:39 is the most valuable part of this timeline. It shows that detection worked but diagnosis was slow — which leads to different fixes than a detection problem would.
Avoid the single "root cause". Real incidents come from several factors lining up. Ask "why" repeatedly, and keep going past the first human action:
Record what helped: the alert fired within three minutes, the rollback was one command. This reinforces practices worth keeping.
Every action item needs an owner, a due date and a ticket. Vague items like "improve monitoring" never get done. Compare:
Prioritize fixes that remove a whole class of failure over fixes that patch one instance. Automated guardrails beat checklists, and checklists beat good intentions.
Publish postmortems where the whole engineering organization can read them. Review the open action items in a regular meeting. A postmortem whose action items quietly die in the backlog is a story, not an improvement.
A small, consistent practice: once a quarter, read back through recent postmortems and look for patterns. Three incidents caused by manual deploy steps is a strong argument for investing in automation.
The value of an incident is entirely in what you learn from it. Write the postmortem quickly, keep it blameless so the facts come out, look for contributing factors instead of a single culprit, and treat action items as real work with real owners. That is how the same incident stops happening twice.