Air-gapped does not mean no network. It means no routed path to untrusted networks, and every model, container or file crossing the boundary moves through a reviewed workflow with a signed artifact on the other side.
Configuration is the part people skip. Standing up an air-gapped AI server means wiring together network isolation, offline mirrors for models and packages, key handling, structured logging and an update process that survives an audit. The frame for why you would build this way is covered in the air-gapped AI servers overview; the configuration itself is less often written down.
Draw the Isolation Boundary First
Start with a one-page network diagram. On it: the LLM server or cluster, the inference endpoint, the management VLAN, any storage targets and the exact port or physical interface where the enclave ends. If a system is not on that page, it should not be on the network.
Egress gets a default-deny rule. For Kubernetes-hosted stacks that is a NetworkPolicy with policyTypes Egress and no allow rules; for a bare-metal build the switch or firewall enforces it. Management access comes in through one jump host with hardware MFA, never through the inference pods themselves.
Isolation is a spectrum. A partial separation keeps the LLM on a VLAN with no route to the internet but allows internal ERP and identity traffic. A full separation means no network at all, with data moving by signed media. Pick the point on that spectrum that matches your threat model, not the one that reads best in a slide. The direction of travel across enterprises is clear: a February 2026 Cloudian survey of 203 IT decision-makers found 93% have already repatriated some AI workload, are moving one, or are evaluating the move.
Build Offline Mirrors for Models, Packages and Containers
The enclave needs its own registry, its own package mirror and its own model store. Standing these up is the single largest piece of work in an air-gapped LLM configuration, and the piece that most often gets quietly deferred until production.
Use a connected staging host to pull everything the stack needs: container images, Python wheels, OS packages, model weights. Transfer the signed artifacts over on fixed media. NVIDIA's NIM documentation spells out a clean version of the pattern. Run download-to-cache or create-model-store on the connected host, move the resulting directory, then on the air-gapped side set NIM_MODEL_PATH and leave NGC_API_KEY unset.
| Component | Connected staging host | Air-gapped host |
|---|---|---|
| Container images | Pull from NGC | Tagged in registry.airgap.example.com |
| Model weights | download-to-cache with NGC_API_KEY | Mounted at NIM_MODEL_PATH |
| Credentials | NGC_API_KEY or HF_TOKEN set | NGC_API_KEY and HF_TOKEN unset |
| Proxy and TLS | HTTP_PROXY, SSL_CERT_FILE set | None required |
| Kubernetes egress | Open for pulls | NetworkPolicy deny-egress |
The pattern generalises. For a vLLM or Ollama build on open-weight models, the equivalent is a local Hugging Face mirror, a private container registry and a Debian or RHEL mirror behind the firewall. Pin exact versions. Attach a signed manifest to every transfer. If the ops team cannot tell you which SHA256 is running in production, the mirror has become a liability rather than a control.
Key Material and Identity Belong Inside the Enclave
Secrets never leave the enclave. The identity provider sits inside it or terminates at the boundary. Model-encryption keys, TLS certificates for internal services and signing keys for the update workflow live in an HSM or sealed Vault instance on the air-gapped side, not in a cloud KMS.
For user identity, keep the SSO stack inside too. SAML or OIDC against a private IdP, with no dependency on an internet-facing authentication provider, keeps login flows working when upstream outages or compromised third parties would otherwise block the whole platform. Rotate secrets on a schedule the audit team can see, and log every rotation to the same local store as the rest of the telemetry.
Logging That Survives Without an Internet Connection
Every call to the inference service generates a prompt, a response, a user, a model version and a timestamp. All of it lands in a local log store. SIEM shipping goes to an internal collector; cloud SaaS log pipelines do not belong anywhere in this design, because the reason the server is air-gapped is the sensitivity of exactly that data.
Covert channels are the reason the telemetry matters at all. The RAMBO attack from Dr. Mordechai Guri's lab moved a 4,096-bit RSA key off an air-gapped machine in roughly 42 seconds using radio signals from the RAM bus, at about 1,000 bits per second. Research since has added acoustic, thermal and power-line channels to the catalogue, and each one depends on malware that got in somehow and ran without being spotted.
Local telemetry catches the "got in somehow" part. Process creation, container pulls from the mirror, unexpected outbound DNS attempts, USB mount events, firmware updates to the BMC — all of it belongs in the same store, correlated with inference logs. Keep twelve months hot and ship older data to WORM storage inside the enclave. A deeper walk through hardware-side defences sits in securing AI hardware against tampering and firmware attack.
The Update Workflow Is Where Most Builds Fail
Updates are the one point where the isolation boundary is deliberately crossed. Treat every crossing as a cross-domain transfer: fixed media, dual approval, and independent verification of every inbound file. NSA's Raise the Bar guidance for Cross Domain Solutions enforces a Rule of Three, requiring three independent verification mechanisms per file and mandating Content Disarm and Reconstruction on any CDS attached to a US National Security System. For a commercial deployment the same pattern is right-sized by risk, not abandoned.
Model updates demand the hardest treatment. New weights are large, opaque and expensive to inspect. Build a staging enclave that mirrors production, test the update there for drift and for secrets that may have been baked into the weights during fine-tuning, then promote. The Stuxnet case is the reference point every security team still reaches for: a USB drive crossed Iran's Natanz isolated network and destroyed roughly 1,000 centrifuges, about 10% of capacity, between late 2009 and early 2010. Removable media is still the most common way a perimeter like this is breached.
Patch and firmware updates follow the operational-technology controls in NIST SP 800-82 Rev. 3, published September 2023, which align with SP 800-53 Rev. 5 and the Cybersecurity Framework. A quarterly patch window with a signed bundle and an attached SBOM is a reasonable floor; monthly is closer to right where the risk profile justifies the ops cost.
Hardware Choices Shape What You Can Isolate
The system you buy constrains how clean the isolation can be. A rack-mounted GPU server inside a secured data hall is easy to isolate at switch level. A workstation under a desk is not. Remote management interfaces (BMC, IPMI, vPro) are an often-missed egress path, and they need their own management VLAN with no route out, their own credentials and their own patch cadence.
For sites where even an internal management network is a problem, a compact edge appliance with no wireless radios and physically disabled external ports is often the right form factor. Physical attack surface matters as much as the network diagram: locking bezels, port blockers, tamper-evident seals on chassis screws, and a documented chain of custody from factory to rack. These are the controls that keep a USB-based intrusion from being trivial, and they belong in the configuration spec alongside the firewall rules.
Where This Decision Moves Next
The change most likely to affect an air-gapped AI build over the next twelve months is the next revision of NIST SP 800-82. A Rev. 4 initial public draft is out for comment through the end of November 2026, and its treatment of AI and LLM workloads inside OT environments will shape the baseline auditors ask you to show against. Build to Rev. 3 now, document the configuration against the headings Rev. 3 uses, and the delta when Rev. 4 lands will be a diff rather than a rebuild. The rest of the work above is the same either way.