Chaos Engineering
This is a small extract from a previous post regarding Patterns for a Resilient Architecture in which we reviewed good practices and design to ensure a high degree of resliency in our systems; this was too part of a session during the past AWS Loft in Stockholm on 2018.
The Principle
To experiment on a System (typically distributed) in order to build confidence in regard to its capability to withstand abnormal conditions. One may think that tests performed on the system are done just at random, but on the contrary,they are designed with specific scenarios in mind and performed on a Controlled Environment.
The Practice
The practice is seen as not causing issues but rather reavealing them, we can re-phrase issue as weaknesses as part of the adoption of the mindset; here are some of the areas where we can Inject Faults:
- Application level.
- Host Failure.
- Resource attacks (CPU, memory, disk, etc.).
- Network attacks (dependencies, latency, etc.).
- Region attacks.
It is typically suggested to start small and proceed gradually.
Phases
1) Ready State
The normal behavior of your system as well as its state, it is not uncommon that systems provide a mechanism to determine its pulse, it is also complemented by defining some sort of metric; this way we can detect if the system is undergoing an anomaly. This would turn to be our baseline for comparison and experimentation.
2) Hypotesis
This is where we formulate the questions around the system and its dependencies that give way to the various experiments we decide to conduct.
The What if…?
3) Design and Run experiment
| Design Experiment | Rules of Thumb |
|---|---|
| - Pick Hypotesis. | - Start small. |
| - Scope the experiment. | - Similarity to Production as close as possible. |
| - Identify the metrics. | - Minimize and contain the blast radius. |
| - Notify the organization. | - Have a kill switch. |
Canary Release could be an option to conduct an experiment on a live environment, especially if this happens to be Production.
4) Verify and Learn
This is where we quantify the results of our experiment; the time elapsed until:
- Detection.
- Interested parties are notified.
- Graceful degradation takes place.
- System recovered completely.
The later bullet point is where we could certainly assure everything is stable. Practices such as Post-Mortem take place in order to assess the outcome and working towards finding out a solution, it is good to keep in mind the following:
- Failure usually requires multiple faults.
- Multiple contributing factors to a fault rather than isolated.
5) Fix
A one liner during the presentation, which I personally think is golden
Never let a good crisis go to waste.
The silver lining of a disaster, the given opportunity to fix underlying issues and work towards improvement.
Challenges to put all this in practice were also addressed, being mostly around work culture:
- The practice is neglected due to lack of time allocation and priority.
- Team members fully allocated.
- Controversy or negative discussions.
- Addressing it can become overly political.