Most on-prem AI procurement goes wrong in the two weeks before signature, not after. The hardware is picked, the finalists are down to two, and the pressure is to close before the quarter ends. That is exactly when the questions that would have changed the decision get skipped, and the contract that lands on the desk is the vendor's standard paper with your logo on the cover page.
The point of a final-stage vendor conversation is not to re-litigate the architecture. It is to pin down what the vendor will actually do when a GPU fails on a Saturday night, when a model you depend on gets deprecated, when your auditor asks for logs, and when you want to leave. The question list below is what to put in writing before anyone signs.
So what should you actually ask?
Pin Down What the Hardware Really Is
"NVIDIA-based" is not a specification. A finalist should be able to tell you, in one page, the GPU model and count, VRAM per GPU, interconnect (NVLink, PCIe generation, or Ethernet fabric), CPU, system memory, storage tier, chassis vendor, and power draw at sustained inference load. If the answer arrives as a glossy brochure, that is the answer.
Ask who integrates the board. NVIDIA HGX is a GPU baseboard that OEMs such as Dell, HPE, Supermicro, Lenovo, and Gigabyte integrate into their own servers with chosen CPUs, memory, storage, networking, chassis, and software, which is a different product from a turnkey DGX. The CPU, memory ratio, cooling design, and firmware stack change the behavior of the box under real workloads, and a vendor-neutral integrator should be able to show you two or three configurations and explain the trade-off between them. The four cstmAI sizes exist for that reason.
Then ask about supply. Enterprise GPU lead times stretched to 36 to 52 weeks through 2025 and into 2026, with non-priority buyers facing 30+ week waits and HBM memory manufacturers having effectively booked their 2026 production capacity. A vendor who promises an eight-week delivery on a specific GPU SKU is either carrying stock, has a committed allocation, or is guessing. You want to know which.
Make the Software Stack Concrete
A hardware vendor who ships a box and leaves the inference layer to you is not selling you a system. You are integrating it. Ask the finalist to name the serving runtime (vLLM, TensorRT-LLM, llama.cpp, SGLang), the orchestration layer, the model gateway, the observability stack, and the UI, and to say which parts they wrote, which parts are open source, and which parts require a separate license.
That license question is the one that gets skipped. NVIDIA AI Enterprise is licensed per GPU at $4,500 for a one-year subscription or $18,000 for five years, with a perpetual option at $22,500 per GPU including five years of Business Standard support. On an eight-GPU server that is real money, and it is often not in the hardware line of the quote. If the stack depends on it, you want that line itemized before signing.
If the vendor's answer is their own closed platform, read the clause that governs what happens when the subscription ends. Can you keep running the models you deployed? Do the fine-tuned weights come with you? If the inference runtime is open source, this problem is smaller, which is one reason an open stack built on vLLM, Ollama and Open WebUI is worth asking about by name.
Read the Support SLA Line by Line
Every finalist will claim 24x7 support. The useful question is what "response" means. NVIDIA's Business Critical Support provides 24x7 service and a one-hour response time for Severity 1 cases, which is the kind of commitment to measure others against. A vendor who promises "next business day" on a production inference cluster is telling you how they will be staffed on Sunday morning.
Separate response time from resolution time. A one-hour acknowledgment that leads to a 72-hour hardware swap still leaves you down for 72 hours. Ask for both numbers, and ask what the remedy is when they are missed. Service credits that cap at a token percentage of the monthly figure are not a remedy, they are a fee schedule for breaking the contract.
Then ask what the SLA covers. Hardware availability, GPU performance under load, software stack uptime, and model availability are four different things, and a cheap SLA covers only the first. Push for a definition that ties to usable service rather than "powered on." The companion question is what happens on the second week of the first ninety days, when the stack is still settling in and most of the issues the operations team will ever hit are about to surface, documented in detail in first 90 days of on-premises AI.
Nail Down the Model Lifecycle
Open-weight models move fast. Llama, Qwen, Mistral, DeepSeek, Gemma and gpt-oss all ship new generations every few months, and a vendor who deploys a 70B model today will need to deploy its successor within a year. Ask who does that work, how it is priced, and how long you have to run the previous model after a new one is validated.
A few specific questions worth putting in writing:
- Which model versions are covered under the base contract and which trigger a change order.
- Who owns the fine-tuned weights, the evaluation sets, and the prompt templates. If the vendor runs training, this is not optional.
- What the acceptance test looks like when a new model is deployed, and who signs it off.
- How long the vendor will support a model after a replacement is live, so you are not forced to re-validate on someone else's clock.
If training or adaptation is in scope, ask the vendor to walk through their fine-tuning and model training process, including how they handle data pipelines, evaluation, and the rollback path if the new model regresses on your task.
Write the Exit Terms Before You Need Them
The exit clause is written at signature or it is not written. On-prem makes this easier than SaaS because the hardware is already yours, but the software, configurations, prompts, and fine-tuned weights are the part that matters, and they are the part that is often silently retained.
Ask for the full handover list in the contract. The template is well-covered in a 14-clause AI contract checklist that specifies the code repository with full history, prompts and agent configurations, evaluation sets with latest test results, architecture notes and runbooks, a list of every account and API key with keys rotated on the last day, data exports in a named format, and a transition period during which the vendor answers questions from you or your next provider.
- 1Code repository with full history
- 2Prompts, system instructions, agent configurations
- 3Evaluation set and latest test results
- 4Architecture notes and runbooks
- 5Accounts, API keys and sub-processors, keys rotated
- 6Data export in a named format, plus logs
- 7Transition period for your next provider
Hold Compliance Claims to the Actual Standard
"HIPAA compliant" is not a thing a vendor can be. A vendor can support your HIPAA, SOC 2, and GDPR programs, and the right question is which controls they implement, which they document, and which audit evidence they will provide on request. Ask for the audit report, not the badge.
If any part of your use case touches the EU, the EU AI Act is now the binding floor. Deployers of high-risk AI systems must use systems in accordance with provider instructions, assign trained human oversight, and retain automatically generated logs for at least six months. Your vendor needs to tell you how their stack produces those logs, where they are stored, and how you retrieve them for a regulator. If the answer is a shrug, that is a procurement decision.
Air-gapped deployments change the compliance posture in ways worth asking about explicitly. Model updates, telemetry, license checks and remote support all need an offline story, and the trade-offs are specific enough to deserve their own conversation about air-gapped AI servers before the contract is drafted.
One Question That Separates Finalists
Ask each finalist to describe, in writing, what happens on the day a GPU fails. Who notices, how fast, who dispatches the replacement, who installs it, who revalidates the stack, and what the user-visible impact looks like end to end. The Llama 3 405B pretraining run on 16,384 H100 GPUs recorded 419 unexpected interruptions over 54 days, roughly one failure every three hours, with GPU failures accounting for 30.1% and HBM3 memory failures 17.2% of unplanned outages. Your cluster is smaller, but the failure rate per GPU is not fundamentally different. A vendor who cannot narrate the response path in detail will narrate it badly when it happens.
None of these questions require you to be an expert in GPU architecture or contract law. They require the vendor to be specific, in writing, before the money moves. The finalists who answer them cleanly are the ones worth signing with. The ones who dodge, defer, or redirect you to a sales deck are telling you how the next three years will go. If you want a second opinion on what a vendor-neutral answer sounds like, that is what a scoping conversation is for.