Storage networks and data protection

Every application needs somewhere to keep its data, and in a data centre that storage is almost always reached over a network. Storage traffic behaves differently from ordinary application traffic: it is sensitive to loss and latency, comes in large bursts, and grows whenever drives fail and data is rebuilt or copied to another site. This lesson explains the kinds of storage, the networks that carry them, the protocols from Fibre Channel to NVMe over Fabrics, and the replication and backup traffic that protects data against failure and disaster.

Block, file and object storage

Storage is presented to applications in three main ways:

• Block storage presents a raw disk (a volume or LUN) made of numbered fixed-size blocks. The server's operating system puts its own file system on it. Databases and virtual machine disks use block storage because it gives the lowest latency and the most control.
• File storage presents a shared file system with folders and files, reached over a network protocol such as NFS (common on Linux) or SMB (common on Windows). Many servers can share the same files.
• Object storage stores data as objects, each with a unique key and metadata, in a flat namespace reached over HTTP with an API such as Amazon S3. It scales to billions of objects and exabytes, and holds backups, media, logs, data lakes and AI training data.

These map onto three architectures: direct-attached storage (DAS), with drives inside or cabled directly to one server; storage area networks (SANs), dedicated networks carrying block storage from shared arrays; and network-attached storage (NAS) and object stores reached over the ordinary IP network. Hyperscale operators increasingly use software-defined storage: ordinary servers full of drives, with software spreading data across many of them over the Ethernet fabric.

Fibre Channel

Fibre Channel (FC) is the traditional SAN technology in enterprise data centres. Despite its name, it runs over copper or fibre. Its defining feature is that it is lossless: a sender may transmit only when the receiver has granted it buffer credits, so frames are never dropped for lack of buffer space. Storage protocols were designed to expect a perfectly reliable link, and FC provides one.

• Speeds are marketed as 16GFC, 32GFC, 64GFC and 128GFC; the 128GFC generation (Gen 8) was introduced in late 2025.
• Each device has a 64-bit World Wide Name, and switches assign 24-bit fabric addresses.
• Zoning on the switches controls which servers can see which storage ports, and LUN masking on the array controls which volumes each server can use.
• Best practice is two completely separate fabrics, A and B. Every server has a connection to each, and every array has ports on each, so a failure or misconfiguration of one entire fabric leaves the other running.

FC networks remain common in banks, governments and large enterprises because of their predictability and isolation, but they require separate switches, adapters and skills. Most new capacity, especially in clouds, runs storage over Ethernet.

Storage over Ethernet

Several protocols carry block storage over Ethernet and IP:

• iSCSI carries SCSI commands over TCP/IP. It works on any IP network, and TCP's retransmission handles loss, but latency and CPU use are higher than with FC.
• FCoE (Fibre Channel over Ethernet) carries FC frames directly in Ethernet frames on a lossless Ethernet class. It was promoted for converging LAN and SAN but is now rarely deployed.
• NVMe over Fabrics (NVMe-oF) extends the NVMe protocol, designed for fast flash drives with many parallel queues, across a network. It runs over RDMA (RoCE or InfiniBand), over plain TCP (NVMe/TCP), or over Fibre Channel (FC-NVMe). NVMe/TCP has become popular because it needs no special network configuration and delivers latency close to local flash for many workloads.

Flash drives respond in tens of microseconds, compared with milliseconds for spinning disks, so the network and protocol stack now make up a large share of storage latency. That is why storage traffic drives demand for low-latency fabrics, RDMA and DPUs that handle storage protocols in hardware (Lesson 4).

Distributed storage and rebuild traffic

Software-defined and object storage systems protect data by spreading it across many servers, so that losing a drive, a server or even a rack loses no data. Two methods are common:

• Replication: each piece of data is stored as two or three complete copies on different servers or racks. Writing 1 TB with three-way replication sends 3 TB across the network and uses 3 TB of capacity.
• Erasure coding: data is split into \(k\) data fragments and \(m\) parity fragments, any \(k\) of which can rebuild the original. An 8+3 scheme survives the loss of any three fragments while using only \(11/8 = 1.375\) times the raw data in capacity.

Erasure coding saves capacity but costs network traffic when something fails, because rebuilding a lost fragment requires reading \(k\) surviving fragments from other servers.

Replication between sites

To survive the loss of a whole data centre, data is copied to a second site. Two figures define the requirement:

• Recovery point objective (RPO): how much recent data the business can afford to lose, measured in time. An RPO of zero means no committed transaction may be lost.
• Recovery time objective (RTO): how quickly service must be restored after a disaster.

Synchronous replication writes every change to both sites before acknowledging it to the application, giving an RPO of zero. But each write waits for a round trip between sites, so distance adds directly to write latency: at about 5 µs per kilometre each way, every 100 km adds about 1 ms per write. Synchronous replication is therefore normally limited to sites within about 100 km of each other, and is one reason cloud availability zones are built within a metro area.

Asynchronous replication acknowledges writes locally and sends changes to the remote site a few seconds or minutes later. It works over any distance but the RPO is greater than zero: whatever was in transit when disaster struck is lost.

Backup and the 3-2-1 rule

Replication protects against site failure but faithfully copies mistakes and malicious changes: if ransomware encrypts the primary data, replication encrypts the copy too. Backups keep point-in-time copies that can be restored to a moment before the damage. The traditional 3-2-1 rule calls for three copies of data, on two different kinds of media, with one copy off site; a common modern extension (3-2-1-1-0) adds one immutable or offline copy that cannot be altered and zero errors on restore tests.

Backups move large volumes of data on a schedule, so they are a major source of network traffic. Backup networks are often separated from production traffic, or carried in a low-priority QoS class (Lesson 2), and backup windows are sized so that full and incremental backups finish before the business day.

Canadian organizations often must keep certain data in Canada: some provinces restrict or attach conditions to storing personal information held by public bodies outside Canada (Nova Scotia's rules are among the strictest), and federal institutions follow Treasury Board direction on data residency. Replication and backup sites must respect these rules.

Storage for AI

AI training (Lesson 11) creates distinctive storage traffic. Training data is read repeatedly from object or parallel file systems at high bandwidth, and the model state is saved periodically as a checkpoint so that training can resume after a failure. Large models produce checkpoints of many terabytes, and the cluster is idle while it writes them unless writes are made asynchronous.