Capstone: deploying a web application to the cloud

Over eleven lessons, Jordan and Sam have collected every piece of the wholesale site's design. This capstone puts them together. It follows the site from the final architecture, through the first deployment and the launch checklist, to the first busy Monday morning and the first month's review. Along the way it revisits each lesson's ideas in the order a real deployment uses them. The goal is not to memorize Ridgeline's particular choices, but to see how the decisions connect, so that you can make your own for a different application.

The requirements

Rosa's requirements, agreed in the spring, were short. Wholesale customers must be able to sign in, see their own prices, place orders and download invoices from a phone or laptop. The site must cope with the Monday morning rush and the December peak. It may be down for at most four hours at a time, an RTO of four hours, and lose at most fifteen minutes of orders, an RPO of fifteen minutes. Customer data must stay in the country. And it should cost no more than about 150 dollars a month, roughly what the old ordering system cost in Theo's time spent retyping orders from emails and texts.

Sam added technical requirements of his own: nothing administrative exposed to the internet, MFA for every person, no long-term secrets in code, everything defined in code and reviewed before it changes, every resource tagged, and backups in a second region.

The architecture

Reading the diagram from the customer inward shows how each lesson contributed.

The service models are mixed, as lesson 2 predicted: SaaS for invoicing in the accounting package and for customer sign-in, PaaS for the container platform and database, functions for the event handlers, and a single IaaS virtual machine where control of the operating system was unavoidable. The deployment model is public cloud with a small hybrid link to the roastery.

Building it

Jordan built the site in stages, each one a pull request with a plan that Sam reviewed. First came the foundations: the network, the IAM groups and roles, the budget and the tag policy. Then the data layer: the database, the buckets and the secrets entry. Then the application: the container registry, the container service, the load balancer and the DNS record. Finally the extras: the CDN, the message service and the functions, and the VPN to the roastery.

Building a test environment first was the single most useful decision. Because the infrastructure was code, the test environment was the same configuration with smaller sizes and a different name, created in fifteen minutes. Jordan found three problems there that would have been embarrassing in production: the database rule from lesson 7's worked example, a container that ran out of memory when generating large invoices, and a missing permission for the invoice function to write to the bucket. Each was fixed in code, so production never had them.

The application itself is packaged as a container image by an automated pipeline whenever Jordan merges code. The pipeline runs the tests, builds the image, scans it for known vulnerabilities, pushes it to the registry, and deploys it to the test environment. Deploying to production is a separate, approved step.

Deploying safely

Replacing a running application with a new version is a risky moment, and there are several strategies for doing it without downtime. A rolling deployment replaces copies one or a few at a time, so some copies run the old version while others run the new one. A blue/green deployment runs a complete new copy of the application, green, beside the old one, blue, tests it, then switches all traffic at once, keeping blue ready for an instant switch back. A canary deployment sends a small share of traffic, say one request in twenty, to the new version, watches the error rate and response times, and increases the share only if all is well, rolling back automatically if not.

Serverless container platforms make canary and blue/green deployments easy by keeping each version as a revision and letting you split traffic between revisions. Jordan's pipeline sends one request in ten to a new revision for fifteen minutes, compares its error rate with the current revision, and either promotes it to all traffic or rolls it back.

def canary_decision(old_errors, old_requests, new_errors, new_requests, max_ratio, min_requests):
    return "promote"

print(canary_decision(12, 9000, 2, 1000, 1.5, 500), canary_decision(12, 9000, 9, 1000, 1.5, 500))

The launch checklist

Before switching the old email address to an auto-reply pointing at the new site, Sam worked through a launch checklist. It is a good model for any cloud launch, because each line maps to a lesson and to a real failure that has happened to someone.

One item deserves emphasis: a restore tested from backup. Backups that have never been restored are a hope, not a plan. Sam restored the test database from a point-in-time backup to a new instance, checked the orders were there, timed it, about twenty minutes, and deleted the copy. That test proved the RPO and RTO targets could be met.

resources = [
    {"name": "orders-db", "kind": "database", "public": False, "encrypted": True, "tags": {"project": "wholesale"}},
    {"name": "invoices", "kind": "bucket", "encrypted": True, "tags": {"project": "wholesale"}},
    {"name": "jordan", "kind": "user", "mfa": True, "tags": {"project": "wholesale"}},
    {"name": "old-test-vm", "kind": "vm", "public": True, "tags": {}},
]

def blockers(resources):
    return []

print(blockers(resources))

The first Monday

The site went live on a Thursday, so that any problems would surface before the first real rush. On Monday morning, orders started at six o'clock and peaked just after eight at about two orders a minute. The container service scaled from one copy to four, then back down to two by ten o'clock and one by the afternoon. The database's processor use never passed a third. One customer could not sign in because their password reset email went to a spam folder, which was solved by asking them to check it and, later, by improving the email's sender settings. Theo spent the morning roasting instead of typing orders, which was the whole point.

There was one real incident a few weeks later. A new version that changed how prices were rounded produced errors for orders with more than twenty lines. The canary step sent one request in ten to the new revision; within four minutes its error rate was several times the old one, and the pipeline rolled it back automatically. Two cafes saw an error and retried successfully. Jordan fixed the bug with a test that reproduced it, and the next deployment passed. Without the canary, every customer would have hit the bug.

The first month's review

At the end of the first month, Sam, Jordan, Anita and Rosa reviewed the site against its requirements. Availability was complete apart from a nine-minute window during a provider maintenance event on the database, during which the standby took over as designed. No data was lost. The bill was 131 dollars, under the budget, after Sam removed a forgotten test load balancer worth 18 dollars that the untagged-resources report caught. Customers liked the site, and two new cafes signed up because ordering was easier.

The review also produced a list of improvements, which is normal: add a status page for customers, try a cheaper database size during the quiet summer months, move older roast logs to the archive tier with a lifecycle rule, and write a runbook, a step-by-step guide, for restoring the database, so that someone other than Sam could do it. Cloud systems are never finished; they are measured, reviewed and improved, which is exactly what the operate phase of FinOps and good service management, covered in the ITSM courses, are about.

# The capstone in numbers: availability and cost for the final design
def series(*p):
    out = 1.0
    for x in p:
        out *= x
    return out

def parallel(*p):
    fail = 1.0
    for x in p:
        fail *= 1 - x
    return 1 - fail

availability = series(0.9999,                     # CDN and DNS
                      0.9999,                     # load balancer
                      parallel(0.995, 0.995),     # application copies in two zones
                      0.9995)                     # multi-AZ database
minutes = (1 - availability) * 30 * 24 * 60

costs = {"containers": 33.0, "database and standby": 49.6, "storage": 1.2,
         "CDN and egress": 6.1, "functions": 0.0, "roast logger VM": 5.4,
         "VPN": 27.0, "secrets and DNS": 2.4, "logs and monitoring": 6.3}

print(f"Estimated availability {availability:.5f}, about {minutes:.0f} minutes down in a 30-day month")
print(f"Estimated monthly cost {sum(costs.values()):.2f} dollars (budget 150)")
print("Largest items:", ", ".join(k for k, v in sorted(costs.items(), key=lambda kv: -kv[1])[:3]))

Estimated availability 0.99928, about 31 minutes down in a 30-day month
Estimated monthly cost 131.00 dollars (budget 150)
Largest items: database and standby, containers, VPN

def review(costs, budget, availability, max_minutes_down):
    return {}

print(review({"containers": 33.0, "database": 49.6, "vpn": 27.0}, 150, 0.9993, 60))