Practical Chaos Engineering
A few notes from the session Practical CHAOS Engineering held at AWS Loft 2019; we dove into the practices, common challenges to exercise the practice as well as tooling usually found when conducting experiments.
The slides aren now available on AWS EMEA site.
The practice
We are encouraged to interpret CHAOS engineering as a practice that allow us to build confidence in our systems by conducting experiments under a controlled environment, additionally, we could at an early stage pinpoint weaknesses in our systems. Monitoring activity and assessing results constantly are practices part of the lifecycle as a direct feedback loop.
Ultimately, this becomes a cycle of experimenting, reviewing results and improving; in practice, it is divided in the following stages:
- Figure out the systems ready state, in order words, its normal behavior.
- Produce an hypothesis, start with what if?.
- Design and run the experiment.
- Verify and quantify the results, i.e.: time to detect, notify, self-heal, et cetera.
- Improve.
One instrumental part when designing the experiments is how to deliberately create failure scenarios, in practice, this is called failure injection and it can be applied at the following levels (perturbation models):
- Application
- Host
- Resources (CPU, Mem)
- Network
- AZ / Region
- People
We also need to have a clear understanding of the blast radius of our experiment and have mechanisms to stop it, i.e. an emergency button; it is also advised to start simple and local, unlike common belief, experiments in productions should be rare and highly controlled, some dimensions to factor in as part of the practice:
- Software (versioning, memory leaks, certificate expiration, etc.)
- Infrastrucure (redundancy, self-healing, bulkheads and cells, etc.)
- Application (resiliency patterns, load shedding, etc.)
- Operations (observability, monitoring, etc.)
CHAOS Engineering will not elimimate failure entirely, rather handle it more gracefully; we shouldn’t think of the practie as a replacement for other mechanisms such as unit or integration tests as they address other things like quality.
Products/Tooling
The following tools were shwon by the speaker, some of them are good for local experimentation and others meant to be used in live/deployed environments.
- Stress CPU: stress-ng
- Network Delay/Latency: tc
- Block DNS resolution: iptables could be used to block DNS queries
- DDoS: wrk is an HTTP benchmarking tool we could to generate concurrent calls.
- The CHAOS Toolkit (experiment definitions)
- Gremlin
- ToxiProxy
- chaos-lambda
Overall remarks by the speaker
- Currently there is a challenge to inject failure in AWS Managed Services.
- AWS Aurora supports mechanisms such as: disk congestion, disk failure, replica failure, master instance crash.