Postmortems and Learning
Every incident contains lessons. Postmortems — also called retrospectives or incident reviews — extract those lessons systematically. Done well, they prevent recurrence and improve your systems. Done poorly, they become blame sessions that discourage honesty.
The Blameless Approach
Blameless postmortems focus on systems and processes, not individuals. When someone makes a mistake, the question isn't "who screwed up?" but "why did our systems allow this mistake to cause an outage?"
Humans make errors. That's inevitable. Well-designed systems catch errors before they cause damage. If an engineer's typo brought down production, the postmortem should ask: Why didn't code review catch it? Why didn't tests fail? Why didn't the deployment process include safeguards?
This approach requires psychological safety. People must feel safe admitting mistakes and sharing what really happened. If postmortems lead to punishment, engineers will hide information, and you'll never learn the real causes.
Postmortem Structure
A good postmortem document captures everything needed to understand and prevent recurrence:
# Incident: [Descriptive Title]
## Summary
Brief description of what happened and the impact.
## Impact
- Duration: How long users were affected
- Users affected: Scope of impact
- Revenue/business impact: If applicable
## Timeline
Chronological record of events:
- 10:00 - Alert fired for elevated error rates
- 10:05 - On-call engineer paged
- 10:15 - Root cause identified as database connection exhaustion
- 10:30 - Fix deployed, connections restored
- 10:45 - Verified system healthy
## Root Cause
The underlying reason this happened. Go deep — "human error" is never a root cause.
## Contributing Factors
What made the incident worse or delayed resolution?
## Action Items
Specific, assigned, time-bound improvements.
## Lessons Learned
What did we learn that applies beyond this incident?
Effective Action Items
Action items transform postmortems from documentation into improvement. Each item should be:
- Specific: "Add connection pool monitoring" not "improve monitoring"
- Assigned: Someone owns it
- Time-bound: Has a due date
- Tracked: Reviewed until complete
Prioritize actions that prevent recurrence over actions that improve detection. Catching problems faster is good; not having problems is better.
Building Learning Culture
Postmortems work best when they're routine, not exceptional. Review every significant incident, not just catastrophic ones. Small incidents often reveal the same systemic issues that cause big ones.
Share postmortems broadly. Other teams learn from your incidents, and transparency builds trust. Some organizations publish postmortems publicly, demonstrating their commitment to reliability and learning.