Enterprise engineering

What is the process for incident management and on-call?

Goal. Detect quickly, contain the impact, restore service, then learn.

Process

  1. Detect — SLO-based alerts and clear severity levels (for example SEV1 full outage, SEV2 major degradation, SEV3 minor)
  2. Triage and declare — assign an incident commander, a communications lead and technical responders
  3. Mitigate first, diagnose second — roll back, disable a feature flag, fail over or scale; restore service before finding root cause
  4. Communicate — regular updates to stakeholders and a public status page for customer-facing outages
  5. Resolve and verify against the SLOs
  6. Blameless postmortem — timeline, root and contributing causes, what went well, and tracked action items with owners and dates

Foundations: a sustainable on-call rotation, up-to-date runbooks, escalation paths, break-glass access and regular game-day drills. Track MTTD and MTTR and review recurring incidents quarterly.

All guides