Designing for failure and elasticity

Werner Vogels, Amazon's chief technology officer, is often quoted as saying that everything fails, all the time. Gwen writes the line on the whiteboard at the start of the reliability session, under the four high-risk findings from the Well-Architected review. "In the server room," she says, "you tried to make one machine never fail. In the cloud, you assume machines will fail and design so that nobody notices." This lesson follows how she redesigns VetLedger and BookVet around that idea, and how the same design lets BookVet grow and shrink with demand.

Single points of failure

A single point of failure is any component whose failure stops the whole system. Cascade Ridge's current setup has several: one server room, one storage array, one database server, one internet connection at head office, one person who knows how everything works. The July outage happened because one component, the air conditioner, failed and nothing else could take over.

The cure is redundancy: running more than one copy of each critical component, in places that do not fail together, with a way to switch between them. AWS makes the expensive part of redundancy, separate buildings with separate power and cooling, available to every customer through Availability Zones. The customer's job is to use them.

Spreading across Availability Zones

Gwen's VetLedger design spreads every layer across two Availability Zones in the primary Region. Two application servers run on EC2, one in each zone. An Elastic Load Balancing load balancer sits in front of them, checks their health every few seconds and sends each request only to healthy servers. The SQL Server database runs on Amazon RDS in a Multi-AZ deployment: RDS keeps a standby copy in the second zone, copies every change to it synchronously, and if the primary fails it switches to the standby automatically, usually within a minute or two.

Images go into Amazon S3, which already stores every object redundantly across several Availability Zones in the Region, so no extra design is needed for that layer. AWS states that S3 Standard is designed for very high durability, often quoted as eleven nines, meaning the chance of losing a stored object in a year is extraordinarily small. The exam uses the phrase durability for not losing data and availability for being able to reach it when needed; the two are different.

Recovering from a Regional problem

Multi-AZ protects against the failure of a zone, which is the most common large-scale event. A failure of a whole Region is far rarer but not impossible. For that, the design keeps a recovery copy in the second Region chosen in lesson 5: database backups and S3 images are copied there automatically, and templates can rebuild the servers there if needed. Recovering in the second Region would take hours rather than minutes, which Dr. Kaur accepts for a once-in-many-years event.

Two measures express how much protection a design gives. The recovery time objective (RTO) is the longest acceptable time to restore service after a disruption. The recovery point objective (RPO) is the largest acceptable amount of data loss, measured in time: an RPO of fifteen minutes means losing at most the last fifteen minutes of changes. The business sets these objectives, and they drive the design and its cost. Gwen agrees an RTO of five minutes and an RPO of near zero for a zone failure, and an RTO of four hours and an RPO of one hour for a Regional disaster.

# Check a design against recovery objectives (all times in minutes; illustrative).
objectives = {"zone failure": {"rto": 5, "rpo": 1}, "region failure": {"rto": 240, "rpo": 60}}
design = {
    "zone failure":   {"recovery_time": 2,   "data_loss": 0},    # RDS Multi-AZ failover
    "region failure": {"recovery_time": 180, "data_loss": 30},   # rebuild from copies every 30 min
}
for event, goal in objectives.items():
    d = design[event]
    ok = d["recovery_time"] <= goal["rto"] and d["data_loss"] <= goal["rpo"]
    print(f"{event:15} recovers in {d['recovery_time']:3} min, loses {d['data_loss']:2} min of data ->",
          "meets objectives" if ok else "does NOT meet objectives")

zone failure    recovers in   2 min, loses  0 min of data -> meets objectives
region failure  recovers in 180 min, loses 30 min of data -> meets objectives

Loose coupling

Components that depend tightly on each other fail together. Loose coupling means components interact through well-defined interfaces and can keep working, or at least fail gracefully, when another component is slow or missing. A common technique is to put a queue between parts of a system. Kwame plans to use Amazon SQS (Simple Queue Service) so that when BookVet takes a booking it places a message in a queue, and a separate process sends the confirmation email. If the email service is slow for ten minutes, bookings keep working and the emails catch up afterwards.

Elasticity and Auto Scaling

The same redundancy that protects against failure also makes elasticity easy. If BookVet runs on several identical servers behind a load balancer, adding or removing servers changes capacity without changing the design. Amazon EC2 Auto Scaling manages a group of instances and keeps its size between a minimum and a maximum that the team sets. A scaling policy adds instances when a measure such as average processor use rises above a target and removes them when it falls.

Kwame sets BookVet's group to a minimum of two instances, one per zone, so that one failure never leaves the app without a server. The maximum is eight, enough for the spring vaccination rush with room to spare. A target-tracking policy aims to keep average processor use around a set level. When reminders go out in March, the group grows within minutes; in summer it sits at two.

# Simulate a target-tracking Auto Scaling group through a busy day.
import math
hourly_requests = [40, 30, 30, 30, 60, 150, 420, 900, 1300, 1500, 1400, 1200,
                   1100, 1150, 1250, 1100, 900, 700, 500, 350, 220, 120, 80, 50]
per_instance = 250           # requests per hour one instance handles at the target load
minimum, maximum = 2, 8
running = []
for r in hourly_requests:
    want = math.ceil(r / per_instance)
    running.append(min(maximum, max(minimum, want)))
print("Instances by hour:", running)
print("Instance-hours used:", sum(running), "versus", maximum * 24, "if sized for the peak all day")

Instances by hour: [2, 2, 2, 2, 2, 2, 2, 4, 6, 6, 6, 5, 5, 5, 5, 5, 4, 3, 2, 2, 2, 2, 2, 2]
Instance-hours used: 80 versus 192 if sized for the peak all day

Scaling up and scaling out

There are two ways to add capacity. Vertical scaling, or scaling up, means moving to a bigger server with more processors or memory. It is simple, but it has limits, usually needs a restart and leaves a single point of failure. Horizontal scaling, or scaling out, means adding more servers of the same size. It needs an application that can run on several servers at once, but it has no practical ceiling and adds redundancy at the same time. The reliability pillar's principle of scaling horizontally to increase aggregate availability says exactly this.

VetLedger's database will scale vertically, because SQL Server is a single primary database; RDS makes changing the instance size a few clicks and a short maintenance window. VetLedger's application servers and BookVet scale horizontally. Gwen notes that many cloud designs mix both, as this one does.

Testing failure on purpose

A redundant design that has never been tested is a hope, not a plan. The reliability pillar asks teams to test recovery procedures, and the general principles recommend game days. Before the spring cutover, Gwen schedules a game day: during a quiet evening, the team will reboot the primary database with failover to force RDS to switch zones, stop one application server, and restore a backup in the recovery Region, timing each step against the RTO and RPO. AWS Fault Injection Service can run controlled experiments like these, but for a first game day Gwen prefers simple, scripted steps.

Health checks and self-healing

Redundancy works only if failed components are noticed and replaced. The load balancer's health checks send a small request to each server every few seconds; a server that stops answering is taken out of service. An Auto Scaling group can use the same health checks: when an instance is marked unhealthy, the group terminates it and launches a replacement from the same template. The system heals itself without anyone logging in, which is the reliability pillar's principle of recovering automatically from failure.

Self-healing changes how Mateo thinks about servers. In the server room, each machine had a name and a history, and he nursed it back to health when it misbehaved. In an Auto Scaling group, instances are interchangeable and disposable; any one of them can be replaced at any time. That only works if no important data lives on an instance's own disk, so BookVet will store everything that matters in the database or in S3, and the instances will hold nothing that cannot be rebuilt.

Gwen adds that health checks should test what users actually need. A check that only confirms the server is switched on can pass while the application is broken. BookVet's check therefore loads a small page that reads from the database, so that a server with a broken database connection is taken out of service too, and Kwame can see in the logs which check failed and why.

Dr. Kaur asks to be invited to the game day. Mateo adds her to the calendar and writes RTO and RPO at the top of a new page. Lesson 8 steps back from technology to the organizational side of the move.