What are SLIs, SLOs and the golden signals?
Give each critical user journey an SLI (what you measure), an SLO (the target), an error budget, an owner, a dashboard and an incident process.
What to measure:
- Latency — p50/p95/p99 per journey (client vs origin vs database time)
- Traffic — RPS, read/write mix, unique users, connections, burst rate
- Errors — 5xx and domain failure rate, separated from safe 4xx
- Saturation — CPU/memory, DB pool use, locks, IOPS, Redis evictions, queue lag
- Correctness — duplicate payouts, booking conflicts, reconciliation mismatches
- Security and cost — auth anomalies, abuse flags, secret access, spend spikes
Alerts should describe user-impacting failures, not noisy CPU thresholds. Hold blameless postmortems and track time-to-detect and time-to-recover. Use correlation IDs across logs and traces, and avoid high-cardinality metrics and unbounded log retention.