Most teams asking about 70B models expect a tidy VRAM number. The honest answer is that the weights are the easy part. What actually decides whether a Llama 3.3 70B or Qwen 2.5 72B deployment fits on one GPU, one server, or a cluster is a stack of smaller line items: the precision you pick for the weights, the per-token KV cache multiplied by every concurrent user, activation scratch, and the slack your serving stack reserves by default.
So what does the full memory bill look like, and where does it push you from a workstation to a rack?
Weights Are the Floor, Not the Budget
The arithmetic for weights is straightforward. One parameter in FP16 or BF16 is two bytes, in INT8 or FP8 is one byte, and in INT4 is half a byte. A 70B model therefore lands at roughly 140 GB at FP16, 70 GB at FP8, and 40 GB at INT4 before anything else is loaded. Qwen 2.5 72B is a hair larger, around 144 GB at FP16, and behaves the same way under quantization.
Those numbers describe the model sitting idle. A production estimator that includes modest activations and KV cache at a 4,096-token context puts Llama 3.3 70B closer to 154 GB at FP16, 77 GB at INT8, and 38 GB at INT4. That is a reasonable starting point for a single-user chat workload with short prompts. It is not what you need to run the model for a team.
Quantization is the single biggest lever you have. Dropping from FP16 to INT8 roughly halves the weight footprint; INT4 cuts it again. The quality trade-off depends on the method (AWQ, GPTQ, GGUF variants, FP8 with calibrated scales) and the task, but for most retrieval, extraction, drafting and classification work on a 70B base, INT8 or FP8 weights are a defensible default and INT4 is where you go when you need the model to fit a single card.
The KV Cache Is Where the Budget Goes
Attention needs to remember every token it has already seen. That memory is the key-value cache, and it grows linearly with context length and linearly again with the number of concurrent requests. The per-token formula is 2 x num_layers x (num_kv_heads x head_dim) x bytes, which for a Llama 3 70B configuration (80 layers, 8 KV heads thanks to grouped-query attention, head dim 128, BF16) comes out to 320 KB per token, so 1.31 GB at 4K context and 42 GB at 128K.
Multiply by users. Eight concurrent sessions at 32K context on Llama 3 70B is roughly 84 GB of KV cache alone, before the model is anywhere in memory. One hundred sessions at 32K approaches a terabyte. The cache is also why vendors ship with aggressive defaults: TensorRT-LLM allocates 90% of whatever VRAM remains after weights and activations to KV, so a system that looks "full" on nvidia-smi is often just the serving stack reserving headroom it has not yet used.
Two architectural details change the math a lot. Grouped-query attention, which Llama 3 and Qwen 2.5 both use, cuts the cache by the ratio of query heads to KV heads: for Llama 3 70B that is an 8x reduction, moving the per-token figure from about 2.6 MB on a comparable MHA model to around 320 KB. KV quantization to FP8 or NVFP4 then halves or quarters the cache on top of that, which is often the highest-impact lever for long-context or reasoning workloads.
Activations, Framework Overhead and Honest Headroom
Beyond weights and KV, every inference engine reserves memory for activation tensors, CUDA graphs, allocator fragmentation, paged-attention block tables, and the usual grab bag of temporary buffers. For a 70B model in production this is typically another 5 to 15 GB per GPU, with the higher end on vLLM and TRT-LLM configurations tuned for throughput.
A good rule when sizing is to leave 10 to 15% of VRAM unallocated once weights, KV at your target concurrency, and activation overhead are counted. That is the slack that absorbs a traffic spike or a longer-than-expected prompt without an out-of-memory error that drops the request. If your plan requires the model to use 100% of a card, your plan does not have enough card.
The practical sizing question is therefore not "how big is the model" but "what context length and concurrency do I need to support, at what quantization, with what headroom." The AI hardware estimator walks through those variables and returns a VRAM target you can map to a system.
How This Maps to Real GPUs
Current data-center cards give you four meaningful landing zones for a 70B model:
- Single 80 GB card (H100 SXM or PCIe). The H100 SXM provides 80 GB of HBM3 at roughly 3.35 TB/s, the PCIe variant 80 GB of HBM2e at about 2 TB/s. Fits INT4 or FP8 70B weights with modest context and single-digit concurrency.
- Single 141 GB card (H200). The H200 is the first GPU with 141 GB of HBM3e at 4.8 TB/s, roughly 1.4x the bandwidth of H100. Fits FP8 70B comfortably with real context, or INT4 with large batch.
- Single 192 GB card (MI300X). AMD's MI300X carries 192 GB of HBM3 at 5.3 TB/s per chip, enough to hold FP16 70B weights on one GPU with KV room for a small concurrent pool.
- Multi-GPU server. Two to eight 80 to 192 GB cards, tensor-parallel. This is where you go when you need FP16 quality, long context for many users, or both. An MI300X 8-GPU platform pools 1.5 TB of HBM3, enough for models up to 680B parameters, which is a lot of room for a 70B at long context.
Bandwidth matters as much as capacity. Decode is memory-bound: every generated token re-reads the active weights and KV from HBM. A card with more GB but slower memory can hold the model and still produce fewer tokens per second than a smaller, faster card. When comparing options, pay as much attention to TB/s as to GB. The piece on NVIDIA GPU alternatives walks through where AMD and others actually compete.
Mapping to a cstmAI Desk, Rack or Cluster
The sizing bands translate cleanly to the four cstmAI form factors. A cstmAI Desk with one or two professional GPUs in the 48 to 96 GB range is the right home for a 70B at INT4 or FP8 serving a small team on moderate context, or for a pilot before anyone commits to a rack. It is also where most fine-tuning experiments start, since LoRA on a quantized 70B is tractable at this scale.
A cstmAI Rack with four to eight data-center GPUs is the typical production answer for a 70B model used by a department or across an organization. This is where you can run FP8 or FP16 with real context windows, keep KV cache at FP16 for quality-sensitive tasks, and still serve dozens of concurrent sessions without eviction. For a 72B Qwen or 70B Llama handling document AI, agent workloads, or search over an enterprise corpus, a rack is the honest default.
A cstmAI Cluster is for the cases where one rack is not enough: many hundreds of concurrent users, 128K-plus context across most sessions, multiple 70B-class models hosted simultaneously, or a 70B serving tier alongside a larger reasoning model. Edge sits off to the side, for the ruggedized or offline deployments where a quantized 70B on a single accelerator is the whole system. The sister piece on sizing by team size and active use covers the concurrency assumptions in more detail.
A Worked Example You Can Reuse
Assume Llama 3.3 70B at FP8 weights, 16K context, serving 12 concurrent users with FP16 KV cache and 10% headroom. The math:
- Weights: ~70 GB.
- KV cache: 16,384 tokens x 0.32 MB/token x 12 users ≈ 63 GB.
- Activations and overhead: ~10 GB.
- Subtotal: ~143 GB.
- With 10% headroom: ~158 GB target.
That rules out a single 141 GB H200 if you want slack, and it fits comfortably on a single 192 GB MI300X or on two 80 GB H100s with tensor parallelism. Drop KV to FP8 and the subtotal falls to around 112 GB, which fits one H200 with room to spare. Push context to 128K at the same concurrency and KV alone becomes roughly 500 GB, which is a rack-class problem, not a workstation one.
That is the shape of every real sizing exercise: weights set the floor, KV scales with your usage pattern, and the serving stack takes a cut off the top. Nail the three inputs - precision, context, concurrency - and the GPU answer stops being a guess.