What is chaos engineering and resilience testing?
Chaos engineering deliberately injects controlled failures to prove a system tolerates them.
Method
- Define the steady state with measurable SLIs
- Form a hypothesis: "if X fails, users still succeed"
- Inject the fault with a limited blast radius — kill an instance, add latency, drop a dependency, fill a disk, expire a certificate, partition the network
- Observe, compare against the steady state, and fix weaknesses
- Automate recurring experiments and run game days with the on-call team
Rules: begin in staging, always have an abort switch, announce experiments, and never inject failure in production without approval and monitoring. The aim is confidence and learning, not breaking things for sport.