Most financial services buyers approach on-prem AI as a hardware question: how many GPUs, which rack, what power budget. The regulators they answer to see something different. They see a data flow with a jurisdiction, an encryption state at every hop, a key with a custodian, and a human role attached to every query. Hardware is only the floor those controls stand on.
This piece walks through what that floor looks like when you build it as a single system. Residency boundaries, encryption at rest, in transit and in use, key custody models, and tiered access control, each one mapped to concrete choices across the cstmAI product line. If you are a CTO at a bank, asset manager or insurer deciding what to actually buy, this is the specification sheet behind the sizing conversation.
Residency Is a Boundary, Not a Preference
Under DORA Article 30(2)(b), ICT contracts have to spell out the locations, regions or countries where contracted or subcontracted functions and services are provided and where data is stored and processed. That is not a disclosure requirement. It is a design constraint that reaches into every component you buy, every log you keep and every third party you call.
On-prem makes the boundary easy to draw. The physical rack sits in a building you can point at on a map. The harder part is keeping the boundary intact once the system is running. Model weights, embeddings, vector indexes, prompt logs, evaluation sets, fine-tuning corpora and operator traces all count as processed data. A system that keeps inference local but sends traces to a hosted observability tool has crossed the line.
For a bank operating in multiple EU member states, that typically means a GPU cluster per residency zone rather than one central deployment with regional caches. For a US insurer with policyholder data under state insurance codes, it often means pinning each business unit's embeddings and logs to a specific site, with no cross-site replication of raw content. cstmAI Rack and Cluster builds support this by keeping the Platform, Search, Agents and Govern components inside the same physical boundary as the model and the index.
Penalties make the stakes concrete. DORA entered into force on January 16, 2023, with compliance mandatory from January 17, 2025, and penalties reaching EUR 5 to 10 million or 5 to 10% of annual turnover.
Encryption Has Three States and You Owe All of Them
Financial services buyers tend to think of encryption as a storage question. Regulators treat it as a lifecycle question with three distinct states.
At rest. AES-256 on the system drives and the storage array holding model weights, vector databases and document corpora. cstmAI systems ship with self-encrypting drives and dm-crypt or LUKS at the block layer, with keys released at boot from a hardware security module rather than stored on the box.
In transit. TLS 1.3 on every network hop, including the ones inside the rack. This is the state regulators have tightened most recently. NYDFS Part 500's Second Amendment required mandatory encryption of nonpublic information in transit over external networks effective November 1, 2024, with at-rest encryption still permitting CISO-approved compensating controls. The asymmetry tells you where auditors expect no excuses.
In use. This is the state most on-prem AI deployments ignore. During inference, prompts, intermediate activations and generated tokens all sit in GPU memory in cleartext. An operator with host access can scrape them. Hardware TEEs close that gap. NVIDIA Confidential Computing is built into the Hopper architecture, making the H100 the first accelerator with hardware-based TEE capabilities for isolating entire AI workloads, and the Blackwell generation extends that further.
The honest trade-off is performance. Benchmark testing of NVIDIA H100 TEE mode for LLM inference showed overhead below 5% for most typical queries, with larger models and longer sequences experiencing near-zero overhead. For a 70B model serving long-context retrieval augmented generation to a treasury desk, you can run in confidential mode and most users will not notice. For latency-sensitive agentic chains on an 8B model, the overhead is more visible and worth measuring against your actual workload before committing.
GDPR Article 32 reinforces the pattern. It mandates pseudonymisation and encryption, systems to ensure ongoing confidentiality and integrity, the ability to restore availability after incidents, and regular testing of security effectiveness. Each of those is a system property, not a feature flag.
Key Custody Is the Decision Behind the Decision
Encryption without key custody is a label on a datasheet. The question that matters is who can decrypt, under what quorum, and whether that person works for you.
Three custody models cover most financial services deployments:
- Vendor-held. The platform generates and holds keys. Convenient, uncommon in regulated settings, and effectively off the table for a bank's core data.
- BYOK (bring your own key). You generate the master key in your own KMS or HSM. The system receives wrapped data encryption keys for operational use. Common baseline.
- HYOK or KYOK (hold or keep your own key). The master key never leaves a customer-controlled HSM. The system calls out for unwrap operations and never caches plaintext key material beyond a short TTL.
For on-prem AI, HYOK against a dedicated HSM in the same facility is usually the right pattern. The crypto officers hold the quorum, typically a 3-of-5 smartcard set, and the HSM sits on the same floor as the GPUs. cstmAI Govern integrates with PKCS#11-compatible HSMs from the common enterprise vendors, with mTLS between the Platform and the HSM and workload identity rather than shared secrets.
Separate encryption domains matter more than people expect. Logs, vector indexes, model artifacts and fine-tuning corpora should sit under different keys with different rotation schedules. When an incident forces you to revoke access to a prompt log, you do not want to take down inference along with it.
Access Control Is Tiered, Not Flat
Flat RBAC does not survive contact with a trading floor. A junior analyst, a portfolio manager, a compliance officer and an auditor need the same assistant to behave differently, see different documents, and leave different traces. Tiered access control sits above the model and inside the retrieval layer, not just at the login screen.
Three layers are worth building separately:
- Identity. SSO through SAML or OIDC with SCIM provisioning, backed by MFA on every session. As of November 1, 2025, NYDFS now requires MFA for all user access to all information systems, expanding the prior rule that only required MFA for remote access. For teams putting this together, the mechanics are covered in the enterprise SSO walkthrough.
- Authorization. Role and attribute-based filtering at the retrieval layer so a user's query only sees documents their identity allows. The model never gets the forbidden text in context, which also closes the prompt-injection exfiltration path.
- Audit. Immutable logs of prompts, retrieved chunks, tool calls and completions, exported to your SIEM. For an auditor, the useful artifact is not a transcript but a reproducible chain from identity to answer.
Break-glass access for operators should require a second approver and should be loud. Quiet operator access to inference traffic is the pattern that turns into a disclosure event.
Mapping It to Desk, Rack and Cluster
The size of the system follows the use case, not the firm. A compliance team drafting policy memos needs very different infrastructure from a sell-side desk running retrieval against ten years of research.
The cstmAI Desk is a 1 to 2 GPU workstation and fits a single team working with sensitive data behind a locked door, with a local HSM USB token for key custody and no network inference path at all. Suitable for internal audit, small quant teams, or a legal group doing document review on deal data that cannot leave a room.
cstmAI Rack, typically 4 to 8 GPUs, is the common shape for a departmental deployment: one or two mid-sized open-weight models serving a few hundred users, HYOK against a rack-mounted HSM, TLS between every service, confidential computing enabled on the GPUs for workloads where the overhead is tolerable.
cstmAI Cluster is where you land when a line of business needs tens of thousands of users, multiple model sizes, document AI pipelines running overnight, and residency zones per jurisdiction. Here the Govern plane does real work, enforcing per-tenant key separation, per-business-unit retrieval boundaries and per-role audit retention. Sizing conversations usually start with the hardware estimator and firm up during the two-week discovery sprint.
What This Looks Like in Production
Adoption is no longer the question. By 2025, 65% of financial services institutions reported actively deploying or using AI, with capital markets firms leading at 68%, and 61% reported active use of generative AI, up from 52% in 2024. The question is whether the deployment survives the first serious audit.
A system that keeps the residency boundary physical, encrypts in all three states, puts key custody in your HSM, and ties every query to an identity and a role is the one that does. The hardware, the stack and the integration work are the same project. Treating them as one is the point of buying on-prem rather than renting a tenant.
- 11. Rack and powerHardware lands in the residency zone, physical access logged · Facilities + infra
- 22. Disk and boot encryptionSelf-encrypting drives bound to HSM-released keys at boot · Platform engineering
- 33. Key custody handoverHYOK quorum ceremony, master key never leaves customer HSM · Crypto officers
- 44. TLS and in-use encryptionmTLS between services, confidential computing enabled on GPUs · Security engineering
- 55. Identity and SSOSAML or OIDC with SCIM, MFA enforced on every session · IAM team
- 66. Retrieval authorizationAttribute-based filtering wired into the index before models go live · Data platform
- 77. Audit export to SIEMImmutable prompt, retrieval and tool-call logs streaming off-box · SecOps
- 88. First audited workloadOne business unit cut over, controls evidenced end to end · Business + risk