Storage: drives, RAID, network storage and the cloud

Image makers accumulate data relentlessly. A busy photographer may add several terabytes a year; a small video production company may add tens or hundreds. Storage choices affect how fast you can work, how safe your files are, and how much you spend. This lesson explains the main storage technologies (hard drives, solid-state drives and memory cards), interfaces and real-world speeds, RAID arrays and what they do and do not protect against, network-attached storage and shared storage for teams, cloud storage, and how to plan capacity and tiers for working, near-line and archive data. Lesson 11 builds on this to design backup and archiving systems.

Hard disk drives

Hard disk drives (HDDs) store data magnetically on spinning platters read by heads on moving arms. They offer the lowest cost per terabyte for large capacities, with desktop and enterprise drives now exceeding 20 TB each. Sustained transfer rates are typically 150 to 280 MB/s, adequate for still photography and many video codecs, but random access is slow because the heads must move, so HDDs feel sluggish for catalogs and for editing many small files or multiple streams. Drives are designed for different duties: desktop drives for personal use, NAS drives for continuous operation in multi-drive enclosures, and enterprise drives for heavy workloads with higher workload ratings and longer warranties. Mechanical parts make HDDs vulnerable to drops, especially while running; portable HDDs carried to shoots are among the most common sources of data loss.

Solid-state drives

Solid-state drives (SSDs) store data in flash memory with no moving parts. They are much faster, especially for random access, quieter, lighter and far more resistant to shock. SATA SSDs reach about 550 MB/s, limited by the SATA interface; NVMe SSDs, connected directly to PCI Express, reach 3,000 to over 10,000 MB/s depending on the generation. SSDs are ideal for operating systems, applications, catalogs, active projects, cache files and fast video editing. Their cost per terabyte is higher than HDDs, although falling. Flash cells wear out after a finite number of writes, specified as terabytes written (TBW) in the drive's warranty; for typical imaging use this is rarely a limitation, but heavy cache use can consume it faster. SSDs left unpowered for long periods can gradually lose data, so they are not ideal for long-term cold archives.

Interfaces and real-world speed

A fast drive is only as fast as its connection. USB 3.2 Gen 1 (5 Gbps) limits transfers to about 400 MB/s in practice, USB 3.2 Gen 2 (10 Gbps) to about 1,000 MB/s, and USB4 and Thunderbolt 3, 4 and 5 (40 to 80 Gbps or more) to several thousand MB/s. A fast NVMe SSD in a USB enclosure on an older port is therefore limited by the port. Network storage depends on network speed: gigabit Ethernet delivers about 110 MB/s, enough for still photography and some video, while 10 Gb Ethernet delivers about 1,100 MB/s, suitable for shared video editing. When calculating transfer times, use the slowest link in the chain: card reader, cable, port, drive or network.

RAID

RAID (redundant array of independent disks) combines several drives into one volume for more speed, more capacity, protection against drive failure, or a combination:

• RAID 0 (striping) splits data across drives for speed and full capacity, but if any drive fails, all data is lost. Useful only for scratch or cache volumes with data stored elsewhere.
• RAID 1 (mirroring) writes identical copies to two drives; either can fail without data loss, at the cost of half the raw capacity.
• RAID 5 stripes data with distributed parity across three or more drives; one drive can fail without data loss, with the capacity of one drive used for parity.
• RAID 6 uses double parity across four or more drives, surviving two drive failures, which matters with large drives because rebuilding after a failure takes many hours, during which another failure would be fatal in RAID 5.
• RAID 10 combines mirroring and striping for speed and redundancy, using half the raw capacity.

Usable capacity with \(n\) drives of capacity \(C\):

RAID is not a backup. It protects against one kind of failure, a drive breaking, so that work can continue. It does not protect against accidental deletion, file corruption, ransomware, software bugs, controller failure, theft, fire or flood, because all of these affect every drive in the array at once. A RAID array must still be backed up.

Network-attached and shared storage

Network-attached storage (NAS) devices are small computers with multiple drive bays, usually running RAID, that connect to the network and share storage with several computers. Popular NAS systems offer snapshots (read-only point-in-time copies of folders that protect against accidental deletion and ransomware, discussed in Lesson 11), automated backups to other NAS units or the cloud, remote access, media servers and user permissions. For video teams, shared storage systems with 10 Gb or faster networking, or specialized SAN (storage area network) systems, let several editors work on the same media simultaneously, with project sharing features in editing software. A NAS should have an uninterruptible power supply so it can shut down cleanly during an outage, and drive health monitoring with alerts so failing drives are replaced promptly.

Cloud storage

Cloud storage services keep data in remote data centers operated by providers such as Backblaze, Amazon, Google, Microsoft, Dropbox and Wasabi, or specialist photography and video services. The cloud offers off-site protection, access from anywhere, sharing with clients and collaborators, and no hardware to maintain. Limitations include upload and download speed (uploading 5 TB at an upload speed of 50 Mbps takes about nine days of continuous transfer), ongoing subscription costs that grow with the archive, fees for downloading large amounts of data from some services, dependence on the provider's continued operation and terms, and privacy and data residency considerations for client material. Many workflows use the cloud for off-site backup and delivery, while keeping working data on local drives.

In the United States, there is no general federal data residency requirement, but contracts with clients, sector rules and state privacy laws can impose obligations on how personal information, including identifiable images, is stored and secured in the cloud, and some government and enterprise clients require data to stay within the country or within approved services.

Planning capacity and tiers

A practical storage plan separates data into tiers by how often it is used:

• Working (online) storage: fast SSDs or a fast RAID or NAS volume holding current projects and catalogs.
• Near-line storage: larger, slower HDD arrays holding recent and frequently referenced projects.
• Archive: completed projects on HDDs, tape or cloud archive tiers, kept for the long term with several copies.

Estimating growth is straightforward: multiply expected shoots per year by data per shoot, add derivatives and project files, and multiply by the number of copies kept. Plan for growth over the life of the hardware, typically three to five years for drives, and leave headroom, because full drives slow down and RAID rebuilds need space.

Drive health and lifespan

Drives give warning signs before many failures, and monitoring them prevents surprises. SMART (Self-Monitoring, Analysis and Reporting Technology) data reported by HDDs and SSDs includes reallocated sectors, pending sectors, error counts and, for SSDs, wear indicators; NAS systems and utilities display these values and send alerts when they change. Drives that start reallocating sectors, making unusual noises or slowing down should be replaced promptly, after their data has been copied to healthy storage. Failure rates follow a bathtub curve: elevated in the first weeks (infant mortality), low for several years, then rising with age. Many professionals retire working drives after three to five years, moving them to secondary duties or recycling them securely.

Drives that are retired or sent for repair can still contain client data. Secure disposal means wiping drives with tools that overwrite data or use the drive's secure erase function, using full-disk encryption from the start so that discarded drives reveal nothing, or physically destroying drives that held sensitive material.

Choosing storage for a new studio

For a small studio starting out, a common and cost-effective set-up is a fast internal or external NVMe SSD for current projects and catalogs, a four-bay or larger NAS with large HDDs in RAID 5 or 6 for working and near-line storage, a second backup target, and a cloud backup service. As the business grows, faster networking, more NAS capacity and an archive tier are added. Whatever the scale, the principles are the same: fast storage where you work, large affordable storage for everything else, and independent backups of all of it.

Removable media for transport

Data still travels physically. Shuttle drives carry footage from sets to post-production, and couriers deliver drives to clients and archives. Rugged portable SSDs survive drops that would destroy HDDs, and hardware-encrypted drives protect sensitive material if lost. Each shuttle drive should carry a verified copy, never the only copy, and be labeled with project, contents, date and contact information, with checksum reports included so the recipient can verify the transfer.

Whatever storage is chosen, label every drive clearly, keep an inventory listing drives, contents, purchase dates and locations, and review it regularly. An unlabeled drawer of old drives is a common sight in creative studios and a common source of lost work.