Cloud storage and databases

Every application needs somewhere to keep data, and the cloud offers far more choices than a server's hard drive. The wholesale site alone needs three kinds of storage: a database for customers, prices and orders; a place to keep invoice PDFs and product photos; and a disk for the roast logger's virtual machine. Kai's roast logs, which go back six years, need a fourth: somewhere cheap to keep years of records that are rarely read. This lesson explains the three basic kinds of cloud storage, the storage tiers that trade price against access speed, and the main kinds of managed database.

Three kinds of storage

Cloud storage comes in three basic forms, which differ in how data is organized and how it is reached.

Block storage behaves like a hard drive attached to one virtual machine. The operating system sees a raw disk, formats it with a file system and reads and writes blocks of data at high speed. Block storage is what a server's operating system and databases run on. The providers' block storage services are Amazon Elastic Block Store, Azure Managed Disks and Google Persistent Disk. A block volume usually lives in one availability zone and attaches to one instance at a time, although some services allow limited sharing.

File storage presents a shared file system that many machines can mount at once over the network, using the NFS or SMB protocols familiar from office file servers. It suits applications that expect shared folders, such as content management systems or groups of servers that all need the same files. Examples are Amazon EFS and FSx, Azure Files and Google Filestore.

Object storage keeps data as objects, each a file of any size together with metadata and a unique key, in flat containers called buckets at AWS and Google Cloud and containers within storage accounts at Azure. Objects are reached over HTTPS through an API rather than mounted as a disk. Object storage is enormously scalable, very durable and cheap, which makes it the default home for documents, images, backups, logs and data for analysis. The services are Amazon S3, Azure Blob Storage and Google Cloud Storage.

Durability and availability

Two different promises describe storage. Durability is the likelihood that data, once stored, is not lost. Availability is the likelihood that you can reach it when you ask. Object storage services are designed for extraordinary durability by keeping several copies across several availability zones: Amazon S3 Standard, for example, is designed for eleven nines of durability, 99.999999999. At that level, if you stored ten million objects, you would expect to lose one about every ten thousand years. Availability is lower, typically three or four nines, because a network or service problem can make data briefly unreachable without losing it.

High durability protects against hardware failure, not against people. If someone deletes a file or a program overwrites it with garbage, the provider faithfully keeps the deleted or damaged version across all its copies. That is why object storage offers versioning, which keeps previous versions of each object, and object lock or immutability settings, which prevent deletion for a set period. Ridgeline's cloud backup copy uses immutable versions kept for thirty days, as NET-101 described, so that even ransomware that reached the office could not destroy the backups.

Storage tiers

Not all data is used equally. Product photos on the wholesale site are read many times a day; invoices from last month are opened occasionally; roast logs from 2021 are almost never opened but must be kept. Object storage services therefore offer tiers, often called storage classes, that trade storage price against retrieval cost and speed. Hot tiers cost the most to store but nothing or very little to read. Cool tiers cost less to store but charge for each read and expect data to stay for at least a month. Archive tiers cost a small fraction to store but charge more to read, may take hours to return data, and expect data to stay for months.

Choosing a tier by hand for every object would be tedious, so the services offer lifecycle rules: policies that move objects to cheaper tiers as they age and delete them when they are no longer needed. Ridgeline's rule for invoices, for example, keeps them in the hot tier for 30 days, moves them to cool storage until they are a year old, then to archive storage for the rest of the seven years the tax authorities expect records to be kept, and then deletes them. Some services can also choose tiers automatically based on observed access, such as S3 Intelligent-Tiering and Azure's smart tiering options.

# Monthly cost of storing and reading data in each tier (illustrative prices)
tiers = {
    "hot":     {"store_gb": 0.023,  "read_gb": 0.00},
    "cool":    {"store_gb": 0.0125, "read_gb": 0.01},
    "cold":    {"store_gb": 0.004,  "read_gb": 0.03},
    "archive": {"store_gb": 0.001,  "read_gb": 0.10},
}

def monthly_cost(tier, stored_gb, read_gb):
    t = tiers[tier]
    return stored_gb * t["store_gb"] + read_gb * t["read_gb"]

for stored, read in [(500, 1000), (500, 40), (500, 0.5)]:
    costs = {name: monthly_cost(name, stored, read) for name in tiers}
    best = min(costs, key=costs.get)
    print(f"{stored} GB stored, {read} GB read: cheapest tier is {best} ({costs[best]:.2f} dollars)")

500 GB stored, 1000 GB read: cheapest tier is hot (11.50 dollars)
500 GB stored, 40 GB read: cheapest tier is cold (3.20 dollars)
500 GB stored, 0.5 GB read: cheapest tier is archive (0.55 dollars)

tiers = {
    "hot":     {"store_gb": 0.023,  "read_gb": 0.00, "min_months": 0},
    "cool":    {"store_gb": 0.0125, "read_gb": 0.01, "min_months": 1},
    "cold":    {"store_gb": 0.004,  "read_gb": 0.03, "min_months": 3},
    "archive": {"store_gb": 0.001,  "read_gb": 0.10, "min_months": 6},
}

def cheapest_tier(stored_gb, read_gb, months_kept):
    return "hot"

print(cheapest_tier(500, 40, 12), cheapest_tier(500, 0.5, 2))

Protecting stored data

The most damaging cloud storage incidents are not lost disks but data accidentally made public. Over the years, misconfigured storage buckets have exposed voter records, medical files and millions of customer details, almost always because someone set a bucket or object to allow public access while testing and never changed it back. The providers have responded by blocking public access by default on new buckets, and Sam's rule for Ridgeline is that nothing in object storage is public. Product photos for the wholesale site will be served through the CDN with access controlled by the CDN, and invoices will be downloaded through short-lived signed links that the application creates for each signed-in customer.

Other protections are standard. Data at rest is encrypted by default at all three providers, with keys managed by the provider or, for more control, by the customer in a key management service, which lesson 8 mentions. Versioning protects against accidental overwrites and deletions. Access logs record who read or changed what. Snapshots copy block volumes at a point in time, and backups should be copied to a second region, as lesson 4 explained.

Managed databases

Running a database server well is hard work: installation, tuning, patching, backups, replication, failover and monitoring. Managed database services take most of that work away. The customer chooses the database engine and size, and the provider runs the servers, applies patches during a maintenance window, takes automatic backups with point-in-time recovery, and keeps a standby copy in another zone if asked. As lesson 2 explained, managed databases are usually counted as PaaS.

Relational databases store data in tables with fixed columns and relationships between them, and are queried with SQL. They suit structured business data such as customers, orders and invoices, and they guarantee that related changes happen together or not at all, which matters when money is involved. The managed services are Amazon RDS and Aurora, Azure SQL Database and Azure Database for PostgreSQL and MySQL, and Google Cloud SQL and AlloyDB, supporting engines such as PostgreSQL, MySQL, MariaDB, SQL Server and Oracle. PYTH-215 teaches SQL and relational design in depth.

NoSQL databases relax the fixed table structure in exchange for flexibility or scale. Key-value and document databases, such as Amazon DynamoDB, Azure Cosmos DB and Google Firestore, store items looked up by key, scale to enormous traffic and are popular for sessions, shopping carts and mobile apps. In-memory caches such as Amazon ElastiCache, Azure Cache for Redis and Google Memorystore keep frequently read data in memory for very fast access. Data warehouses such as Amazon Redshift, Microsoft Fabric's warehouse and Google BigQuery store large volumes of historical data for analysis and reporting.

For the wholesale site, the choice was easy: orders and invoices are structured, related and involve money, so a managed PostgreSQL database it is, in a small size, with a standby in a second zone, automatic backups kept for fourteen days and point-in-time recovery. That design meets Rosa's recovery point objective of fifteen minutes from lesson 4, because point-in-time recovery can restore the database to any second within the backup window.

def tier_for_age(age_days):
    return "hot"

def count_by_tier(ages):
    return {}

print(tier_for_age(45), count_by_tier([3, 45, 400, 3000]))