Day 113: Chaos engineering: kill the DB, saturate the queue
Chaos engineering: finding weaknesses before an incident does
Chaos engineering deliberately injects failure into a system (ideally in a controlled way, with a hypothesis and a rollback plan) to discover weaknesses before they surface as a real, unplanned incident. This isn't recklessness — it's the same 'shift left' philosophy from Phase 14, applied to reliability instead of security: find the failure mode on your terms, not an attacker's or an outage's.
Three real chaos experiments to run on your own stack
Using everything built in this course: (1) kill your Postgres pod and time how long the app takes to recover, verifying data integrity afterward; (2) saturate your RabbitMQ/Kafka queue with a burst of messages and watch how consumers, prefetch limits, and autoscaling (Phase 24) respond; (3) forcibly drop a Kubernetes node and confirm workloads reschedule correctly, with no silent data loss.
kubectl delete pod -l app=db --grace-period=0 --force
# then watch: does the StatefulSet recover with data intact? How long did it take?Every experiment here is a future STAR story
The roadmap's Track A explicitly calls this out: every self-inflicted outage, documented as a blameless postmortem (Day 111), becomes real interview material — 'I killed my own database on purpose and here's what I learned' is a genuinely strong story precisely because it's specific and true.
The Four Questions: chaos engineering (or blameless postmortems)
Worked example for Docker: dependency hell → consistent runtime environments → VMs too heavy → shared kernel, weaker isolation. Apply it to chaos engineering: what problem (discovering failure modes only during real incidents) did it solve, why couldn't passive monitoring alone solve it, and what trade-off (real risk of causing an actual incident if done carelessly) does it introduce?
Phase 18 complete — you should now be able to
Why is deliberately killing your own database pod in a controlled experiment more valuable than just hoping it never fails in production?
