Patterns for a Resilient Architecture
Yet another long overdue post with the notes from the session “Patterns for Resilient Architecture” I attended at AWS Loft last year in Stockholm.
Despite being a short session done in a two-parter, the speaker Adrian Hornsby went through quite a few architectural patterns backed up by some basic scenarios to exemplify them (and of course how AWS can help you here); given that we are talking about patterns for resiliency, before anything else, I’d better start by giving a definition of for Resiliency, the one I always have in my head is:
The ability of a System to handle abnormal crap and eventually recover, or at least do our best to get there.
Usually, such circumstances will be of the “I-have-no-control-over” kind, our systems must embrace and accept that truth and “take them like a champ” when they happen. Another key aspect is that systems should, as much as possible, degrade gracefully rather than fail beautifully when facing an operational problem, at least until the system recovers.
Patterns
They were wrapped in four different contexts:
- Infrastructure
- Network and Data
- Application
- People
Infrastructure
System availability addresses the different techniques and concepts were addressed such as Availability Zones and Mult-Regional Redundancy, present in almost every vendor, these practices when applied give us High-Availability and Low-Latency, they also lie at the foundation for things such as Disaster Recovery. Some techniques based on routing were mentioned:
- Latency based routing.
- Geo-based routing.
- Weighted Round-Robin.
- DNS Failover.
Auto-Scaling was among the topics too as a mechanism to deal with:
- Node Failure.
- Traffic Spikes.
- Performance.
Ultimately, improve Computing Efficiency.
Immutable Infrastructure is practiced to ensure that updates on an operating environment are as less intrusive as possible and without causing an interruption on the service, the key principle is that the change is applied on new infrastructure rather than the live one, you never touch a live environment. Some take aways:
- No updates on live streams.
- Always start with newly provisoned resources.
- DNS cutovers / routing.
- Keep old verisones up and running in case we need to switch back (fast rollback upon failure)
Network and Data
As our network topology grows and we spread our data out, a few bits here and others there, we will likely face situations where we need to choose between high availability and consistency, you cannot have both it is said, you know “99.99% uptime… oh and real-time data.”
When we handle errors during an operation, we either handle it or let it fail if the former is not valid; when we say we handle an issue, we typicalle rely on our cache store, fallback to configured as defaults, etc. at least until the system fully recovers, in short Eventual Consistency. By properly designing and laying out our components and knowing their interactions we set expectations and build components with a preventive approach, below some known techniques to achieve this:
- Perform workloads asynchronously.
- Prioritize with Queues.
- Read/Write sharding.
- Database Federation.
- Database Sharding.
Another highlight externed is that databases should not handle Transient States.
Application
At the application layer, issues are approached based on different aspects: traffic, how critical it is to the business, workloads, etc. In a distributed system it is not uncommon to face cascading failures, the causes usually being some component in the network slowing down or failing, this usually has a domino effect on the rest of the mesh. In order to address the issues, we have at our disposal different known patterns to ensure a desired level of resliency, we list below some of the most common:
- Timeouts.
- Degradation and Fallback.
- Exponential Backoff (with Jitter).
- Circuit Breaker.
- Bulkhead isolation.
These techniques are applicable when issues arise during the executing flow; another side to this is to find out whether a service is in a good state of not or healthy. Perfoming a system healthcheck is usually either Shallow and Deep, the difference being that the latter will trasverse a whole vertical, this is, a component from start to end, most of the times the result is more trustworthy that the former.
People
Lastly but not least important is the people factor, the lesson learned here is that the more we seggregate responsibilities and roles within a single team or several, the better its scaling maps to that of the system, this aligns with Conway’s law.
Teams, their functions, practices and process should scale along with the system they are supporting, it is almost symbiotic.