hardware.co · AI hardware for on-prem LLM deploymentsVendor-neutral · Quote onlyBuilt to spec
hardware.co · November 16, 2026

Kubernetes or Bare Metal for On-Prem LLM Deployment

Compare Kubernetes and bare-metal approaches for running on-prem LLMs on GPU servers, with honest notes on cost, throughput, and operational load.

By Timothy Carter

A rack GPU server and a single workstation separated by a divider with cables arcing between them.

Wrapping a single-node vLLM deployment in a full Kubernetes platform rollout burns the first six months of an on-prem AI budget on infrastructure the model never needed. Running an 8-GPU server for 60 users with no scheduler at all breaks the first time one Python process segfaults.

The choice between Kubernetes with the NVIDIA GPU Operator and a leaner bare-metal install under systemd comes down to three things: how many GPUs are being shared, how heterogeneous the workloads are, and how much platform engineering the organization can carry long term. For a single cstmAI Desk or small Rack, the answer is almost always bare metal. For a multi-tenant cluster running agents, search, and fine-tuning jobs together, it bends toward Kubernetes.

The Two Patterns in One Install

Bare metal means the CUDA driver installed directly on the host, model servers like vLLM, Triton, or Ollama running as systemd units, and a reverse proxy in front for routing. The NVIDIA Container Toolkit is optional for workloads that want containers. Nothing above the kernel cares about scheduling across more than one machine, and updates are one package per node, done out of hours.

Kubernetes replaces those units with Deployments and StatefulSets scheduled against the nvidia.com/gpu resource or, on recent releases, a Dynamic Resource Allocation claim. The NVIDIA GPU Operator installs and reconciles the driver, container runtime, device plugin, and DCGM exporter on every node. Multi-node tensor parallel for vLLM is wired through the LeaderWorkerSet API, so a 70B model that spans two nodes still looks like one service to callers. For the serving layer itself, our vLLM, Ollama, and Open WebUI stack walkthrough covers which runtime fits which workload on either pattern.

Adoption makes Kubernetes the default assumption for container workloads. Adoption is an input to a buying decision, not the decision itself.

Share of organizations hosting generative AI that use Kubernetes for inference
Share of organizations hosting generative AI that use Kubernetes for inferenceUse Kubernetes for inference: 66%; Use another orchestrator or bare metal: 34%Use Kubernetes for…66%Use another orchest…34%
Two in three generative AI hosts run some or all inference on Kubernetes. Source: CNCF, 2026

What Containers Cost the GPU

The throughput cost of putting Kubernetes between vLLM and the GPU is close to zero on a single node and real on large multi-node jobs. A single containerized vLLM pod on one H100 sees overhead inside measurement noise. Where it bites is overlay networking under distributed inference: the moment Flannel or a default CNI sits between pods doing tensor-parallel synchronization, every kernel that crosses nodes pays for the extra hops.

vLLM itself is doing the heavy lifting on either pattern. Its PagedAttention and continuous batching deliver up to 24x higher throughput than stock HuggingFace Transformers, and 3.5x more than HuggingFace TGI. That gain dwarfs the orchestration overhead on a single box. It is also what gets given back when a scheduling mistake splits NVLink-paired GPUs across nodes, as covered in our note on NVLink, SXM, and PCIe for inference.

The Ops Team Each Pattern Needs

A bare-metal install needs one competent Linux admin who knows how to pin driver and CUDA versions, write systemd restart policies, manage model weight storage, and read DCGM metrics. Monitoring is Prometheus scraping the exporter directly. For a single server running a known workload, this is a two-person on-call rotation, nothing more.

Kubernetes needs a platform engineer who already runs the control plane elsewhere, plus an SRE who can diagnose CNI, CSI, and GPU Operator issues at 2am. The Operator reconciles against each kernel and container-runtime update, and the fabric choice matters more under Kubernetes than under bare metal. Our comparison of InfiniBand, RoCE, and 400GbE covers why. Teams already running Kubernetes for CPU workloads carry very little marginal cost on top. Teams standing it up for the LLM alone carry the full cost of a platform team plus its on-call burden.

A single GPU illustrated with multiple colored partitions across its die resting on a motherboard.

GPU Sharing Is Where Kubernetes Earns Its Rent

Bare metal gives each model server a whole GPU unless MIG partitions get scripted by hand. Kubernetes with the GPU Operator exposes MIG partitions directly to the scheduler, so a single H100 can host several small models through isolated 1g.10gb or 2g.20gb partitions, each carved by a pod spec rather than a sysadmin. That matters once a shop runs internal tools, agent backends, and embeddings on the same hardware.

CUDA 13.4 added memory-enforced GPU sharing through an open-source GPU Fractioning Operator, letting per-workload memory limits apply via Pod annotation. The scheduling flexibility shows up at the fleet level: reports of 35% higher usage for Kubernetes-managed GPU fleets versus hand-managed bare metal are common enough to take seriously, even stripping out the vendor enthusiasm in them.

H100 80GB MIG profiles: memory per instance against maximum instances
H100 80GB MIG profiles: memory per instance against maximum instances1g.10gb: 10; 1g.20gb: 20; 2g.20gb: 20; 3g.40gb: 40; 4g.40gb: 40; 7g.80gb: 80Memory per instance (GB) →Max concurrent instances →1g.10gb1g.20gb2g.20gb3g.40gb4g.40gb7g.80gb
The scheduler picks a point on this curve; bare metal only picks the top-right or bottom-right corner. Source: NVIDIA MIG User Guide, 2025

Where Bare Metal Still Wins

Single-node servers running one or two dedicated models remain the clean win for bare metal. An air-gapped deployment in a SCIF or industrial site, where every binary is reviewed before install, is easier to pass through security on systemd than on a stack that pulls thirty operator images on each reconcile. The same applies at the edge: a ruggedized box in a substation or clinic needs a service that restarts cleanly, not a scheduler.

Disaster recovery is simpler too. A bare-metal server can be reimaged from a Ventoy stick and have its models pulling from an S3-compatible store inside 90 minutes. Rebuilding a Kubernetes control plane plus a GPU Operator on a replacement box is a longer path, and the people who can do it unassisted are more expensive to keep on staff.

Kubernetes releases that shifted GPU and accelerator scheduling
Kubernetes releases that shifted GPU and accelerator schedulingv1.20 Third-Party Device Metrics GA: 2,020; v1.26 Device Manager GA and DRA alpha: 2,022; v1.27 In-Place Pod Resize alpha: 2,023; v1.32 DRA promoted to beta with v1beta1 API: 2,0242020v1.20 Third-PartyDevice Metrics GA2022v1.26 DeviceManager GA and DRAalpha2023v1.27 In-Place PodResize alpha2024v1.32 DRA promotedto beta withv1beta1 API
Each milestone moves accelerator handling a step closer to first-class scheduler support. Source: Kubernetes, 2024

Matching the Pattern to the System You Buy

A cstmAI Desk or small Rack serving one team runs bare metal. A multi-node GPU cluster serving tenants across legal, finance, and engineering teams runs Kubernetes, because the alternative is a growing pile of shell scripts that eventually stops being auditable. The threshold where the trade-off flips is usually around 8 to 16 GPUs, and it is more about workload diversity than raw count. If every GPU serves the same model to the same team, the orchestrator is overhead. If each GPU hosts a different pipeline for a different tenant, Kubernetes pays for itself in the first on-call rotation that does not need to page a human.

Where DRA Shifts the Trade-off

The change most likely to affect this decision over the next year is Dynamic Resource Allocation moving to general availability alongside memory-enforced GPU fractioning in the driver stack. Together they shrink the cluster-tax argument against Kubernetes for mid-sized deployments by making GPU allocation first-class in the scheduler rather than something patched over by the device plugin. For a buyer committing hardware this quarter, choosing bare metal today does not foreclose adding Kubernetes in year two. Choosing Kubernetes now for a workload that does not need it still costs platform engineering that cannot easily be recovered. Size the ops team that can actually be hired, then pick the pattern that fits them.

Get a quote

Spec your system.

Tell us the models you want to run, how many people will use them and where the hardware should live. An engineer replies with a first configuration and the questions that decide the quote.

Form CSTM-Q · Quote only