hardware.co · AI hardware for on-prem LLM deploymentsVendor-neutral · Quote onlyBuilt to spec
hardware.co · November 18, 2026

NVMe Storage Planning for On-Prem AI Servers

How to size and lay out NVMe storage for on-prem AI servers running model weights, vector databases, and KV caches, plus RAID choices that fit each tier.

By Timothy Carter

Rows of NVMe SSDs in a server backplane with visible heatsinks and PCIe routing behind them.

A single open-weight model, Meta's Llama 3.1 405B, needs about 810 GB on disk at BF16, before any vector index, KV cache spill, or checkpoint. That number anchors the NVMe decision on a cstmAI Rack or Cluster more than any vendor spec sheet does.

Treating storage as the last line item means buying the wrong drives twice. Too little endurance for checkpoints, the wrong tier mix for retrieval, and a RAID mode that halves usable capacity for reliability the GPUs never actually benefit from.

What NVMe Carries on an AI Server

An on-prem LLM server loads four different kinds of data from NVMe, and each one bends the drive choice in a different direction.

Model weights stream sequentially from disk into GPU memory at startup or when a model swaps in. Reads are large and burst-shaped. Vector databases, used for retrieval-augmented generation, serve random reads with low tail latency. KV cache offload, where attention state spills from VRAM to NVMe during long-context inference, mixes reads and writes with tight latency targets. Fine-tuning checkpoints flush sustained write bandwidth every few minutes, which chews through drive endurance faster than any other workload on the same box.

PCIe lanes feeding these drives are also feeding the GPUs, so this is not a separate decision from GPU count. PCIe lane bottlenecks on four-GPU builds decide how many Gen5 drives can run at line rate.

Capacity Math for Weights, Vectors, and KV Offload

Weight storage scales linearly with parameter count and precision. At BF16, a 70B model needs roughly 140 GB, a 180B model around 360 GB, and Llama 3.1 405B lands at the 810 GB cited above. Most on-prem deployments keep three or four model variants on disk for A/B testing and fallback, so planning for four to eight times the largest model is a reasonable floor. Running a 70B model at multiple quantizations also means storing INT8 and INT4 copies alongside BF16.

Vector databases add capacity proportional to document count, embedding dimension, and index type. A million 1024-dim float32 embeddings is about 4 GB raw; with HNSW overhead and metadata, plan two to three times that. KV cache offload is bursty: a 128K context, batch 16 run at 70B parameters can need tens of GB of transient NVMe scratch per inference process.

On a Rack serving 50 to 100 concurrent users with a few active models, 20 to 40 TB of usable NVMe is a reasonable first estimate.

Cutaway illustration of an AI server with GPUs, a PCIe switch, and an NVMe storage tier.

Fine-Tuning Turns Storage Into a Write Workload

Fine-tuning changes the shape of the problem. Instead of streaming weights to GPUs, drives absorb sustained writes as checkpoints flush from GPU memory to local or parallel storage every few minutes. MLPerf Storage v2.0 collected more than 200 results from 26 submitting organizations across seven countries, with tested systems serving roughly twice the accelerators of the previous round, which tells you how fast the write side of the workload is growing.

Two implications for a buyer. First, read-optimized drives at 1 DWPD will reach end of life early if the same box is asked to train. Spec at least 3 DWPD on the checkpoint tier. Second, the sustained write path has to land somewhere with real bandwidth rather than an afterthought array of consumer drives behind a RAID 5 group.

The chart below shows how much of a representative 800-billion-parameter training cycle goes to checkpoint I/O versus productive compute.

Where Training Memory Goes Per Billion Parameters
Where Training Memory Goes Per Billion ParametersFP16 weights: 2 GB; FP16 gradients: 2 GB; Adam optimizer states: 12 GB; Total per 1B params: 16 GB+2 GBFP16 weights+2 GBFP16 gradients+12 GBAdam optimize…+16 GBTotal per 1B…32 GBTotal model state
Each bar is the delta a mixed-precision Adam setup adds per 1B parameters; the final bar is the total that must fit in HBM or spill to NVMe. Note: the source describes GPU memory, which is what fine-tuning writes to disk when offloaded. Source: Shreyans, 2025

RAID 0, 1, 10 or None Per Tier

RAID decisions follow the workload, not the drive. A weight tier that can be re-pulled from an object store in minutes does not need mirroring; a drive failure costs a download, not data. Scratch tiers for KV offload are ephemeral by definition and belong on RAID 0 or no RAID at all, since redundancy adds latency without protecting anything the model needs after the request completes.

Vector databases and fine-tuning checkpoints are the opposite case. They are slow to rebuild, and the write pattern punishes parity schemes. RAID 10 doubles the drive count but also roughly doubles effective write bandwidth and keeps rebuilds short, since each replacement only resilvers from its mirror partner rather than reading every drive in the group. RAID 5 on NVMe looks cheaper on paper, but each write requires reading old data, reading parity, computing new parity, and committing both, which taxes the same drives that serve inference requests alongside the training run.

Usable Capacity vs Mirror or Parity Overhead
Usable Capacity vs Mirror or Parity OverheadRAID 0: 100%; RAID 1: 50%; RAID 5 (4 drives): 75%; RAID 10: 50%Usable capacityMirror or parity overheadRAID 0100%RAID 150%50%RAID 5 (4 drives)75%25%RAID 1050%50%
RAID 5 on four drives yields 75% usable; RAID 10 always sits at 50% regardless of drive count. Source: Western Digital, 2025

GPUDirect Storage and the PCIe Path

A direct DMA path from NVMe to GPU memory, bypassing the CPU bounce buffer, is the main reason storage decisions cannot be made in isolation from the motherboard layout. The path performs best when drives and GPUs share a PCIe switch rather than routing every transfer through the CPU root complex. That decision happens at the chassis level, not at the drive bay.

The Micron 9550 Gen5 reaches about 14 GB/s sequential reads and 10 GB/s sequential writes at up to 30.72 TB per drive, enough to saturate a single x4 Gen5 slot. Four of these behind a shared PCIe switch feeding a pair of H100s can keep weight streaming off the CPU entirely, provided the slot assignment is right.

If the rack is already short on lanes for GPUs, adding Gen5 drives behind the wrong switch yields little of the benefit.

Specifying Drives for a Rack or Cluster

At the Rack scale, a practical split is a small boot and OS pair, a hot tier of Gen5 drives for weights and vector indexes, and a warmer Gen4 tier for logs, checkpoints and archive. Capacity, bandwidth and endurance each need to be specified separately rather than left to vendor defaults.

Clusters add parallel file systems — Lustre, Weka, VAST — which change the math because capacity and bandwidth scale across nodes rather than inside one chassis. Local NVMe on each node still matters for scratch, KV offload and checkpoint staging. Llama 3.1 was pretrained on 15.6 trillion tokens across more than 16,000 H100 GPUs, which sets the ceiling of the market; on-prem deployments rarely approach that, but the tier structure is the same shape.

For a mid-size build, the sizing estimator gives a first-pass storage number alongside GPU and VRAM output. The chart below walks the order of decisions.

NVIDIA's AI Storage Hierarchy, Top to Bottom
  1. 11. GPU memory (vRAM)Fast and directly connected to the GPU · Fixed capacity, cannot be upgraded
  2. 22. Data fabricPCIe bus, network cards, DPUs — anything in the datapath between storage and GPU memory · Connects the tiers; where GPUDirect Storage pays off
  3. 33. Storage devicesHard drives, flash drives, and hybrid arrays combining the two · Scalable in capacity but slower, especially on latency
Three tiers from fastest to most scalable; each drive decision has to earn its place in this stack. Source: NVIDIA, 2025

Three Starting Points by Workload

If you are buying a Rack for inference only, start with an 8 to 16 TB Gen5 hot tier mirrored in RAID 1, a smaller OS pair, and a warm Gen4 tier for logs. Keep DWPD modest; writes are limited to vector index updates and logs.

If the same box also fine-tunes, add a RAID 10 checkpoint tier with 3+ DWPD drives, sized for two or three consecutive checkpoints of your largest model plus headroom for async drain.

For a multi-node Cluster, the drive decision moves up a level: pick the parallel file system first, then spec local NVMe on each node for scratch and KV offload. Before any of that, comparing Rack and Cluster configurations against the model and user count is cheaper than reversing the decision mid-deployment.

Get a quote

Spec your system.

Tell us the models you want to run, how many people will use them and where the hardware should live. An engineer replies with a first configuration and the questions that decide the quote.

Form CSTM-Q · Quote only