Ordering one oversized GPU cluster before you know the workload mix, then bolting identity and audit on after go-live, is the mistake this piece helps a banking buyer avoid: six-figure capital tied up in idle silicon while examiners ask for audit records you cannot produce.
The fix is to design the full stack together. Workloads first, then GPU sizes, then the software layers those workloads touch, then identity, audit and change control. An IT leader at a mid-market or enterprise bank can then picture what they would own end to end.
Earlier posts walked through data residency for financial services and on-prem LLM security for banks. A blueprint pulls those into a single procurement shape.
The Workloads a Banking Stack Needs to Carry
Start with an honest list of what runs on the stack: contact-center summarization and complaint review; document AI across KYC files, loan packets and vendor contracts; fraud and AML triage over mixed structured and narrative data; credit memos, model documentation and regulatory responses drafted by analysts; policy search for compliance and risk staff; and back-office agents that reach into core banking, the ERP and the CRM to draft work for a human reviewer.
These workloads do not share a GPU profile. Summarization against a 7B model fits different hardware than vision-heavy document extraction, which in turn fits different hardware than a 70B assistant serving a whole credit department. Lumping them into one cluster is where sizing goes wrong.
Each workload also carries its own data boundary. KYC files hold PII and transaction history. Trading desks hold positions and counterparty data. Complaint queues hold customer identifiers. A stack that cannot enforce workload-level access on retrieval exposes more than it should by the first week of use.
The stakes concentrate in this sector. Financial services carries one of the highest average breach costs of any industry, which is why controls bolted on after go-live cost more in banking than in most places.
Sizing GPUs to the Workload Mix
Split the estate into tiers, then pick a system size for each tier rather than one blended cluster.
A single team piloting document AI or a credit-memo assistant fits a workstation in the cstmAI Desk range, typically one or two GPUs. A branch or regional office that has to run offline, such as a remote treasury site or an inspector workroom, fits the cstmAI Edge. The main bankwide stack, carrying document AI, chat and search for 25 to 100 concurrent users, generally sits on a cstmAI Rack with four to eight GPUs. Heavier concurrency, a large frontier-weight assistant, or agent workloads that fan out across teams push into a cstmAI Cluster across multiple nodes.
Open-weight models fit each tier. A quantized Llama or Qwen 70B runs for a credit-memo assistant on a well-provisioned Rack; a Mistral 7B or a Gemma-class model fits document extraction and classification; a smaller specialized model sits behind fraud triage. Typical VRAM ranges guide the fit. A 70B assistant needs serious headroom per concurrent session; document AI with layout and OCR expects more; smaller models leave enough margin to co-host a secondary model on a Rack. The free sizing tool prints a VRAM and GPU count estimate against a chosen model, concurrency and context, which is where most buyers should start before quoting.
The Software Layers a Bank Workload Touches
Hardware alone does not run a banking workload. The cstmAI software stack ships on the box with six layers, each doing distinct work.
- Platform. Model serving, routing, logging, user and workspace management. Throughput and cost per response are controlled here.
- Search. Enterprise retrieval over policy manuals, product sheets, examiner guidance, prior credit memos and internal procedures. Document access rights are enforced on retrieval, not after the fact.
- Agents. Scripted workflows that read a case, pull from a system of record, draft an answer and route to a reviewer. KYC file assembly and complaint response fit this layer.
- Studio. Prompt, workflow and evaluation authoring for the staff building internal tools.
- Govern. Model registry, prompt and response logging, evaluation sets and approval records.
- Connect. Integrations into the core banking system, ERP, CRM, data warehouse and ticket queues.
Mapping workloads onto the layers tightens procurement. A credit-memo assistant is Platform plus Search plus Govern. A KYC refresh workflow is Platform plus Search plus Agents plus Connect. A fraud triage queue adds document AI to Agents and a smaller, specialized model behind Platform.
Connect carries the identity and residency rules outward. A ticket written back to the core banking system carries the originating user, the model version and the retrieval set as attributes, which is how a reviewer traces a decision later and how a regulator traces it years later.
Where those workloads run today matters for the sizing conversation. Hybrid deployments dominate financial-services AI, with on-prem and edge carrying a smaller but non-trivial share for the workloads that cannot leave the perimeter.
Identity, Audit and Change Control Belong in the Blueprint
These three decide whether the stack holds under examination, which is why they belong in the build, not a later remediation. Four pieces earn their place here.
Identity. SSO over SAML or OIDC against Active Directory or Entra ID, role-based access aligned to business lines, and multi-factor at any entry point into the AI systems. The 2023 amendment to NYDFS Part 500 added an MFA requirement for any individual accessing any information system and created a Class A entity category for the largest covered institutions.
Audit. Every prompt, retrieval, generation and approval logged with user, model version, retrieved documents and timestamp, kept inside the same residency boundary as the model itself. A practical scope covers prompts, retrievals, generation, reviewer edits, approvals and model overrides, keyed by user ID and session, with retention matched to the document rules the bank already runs for emails and customer records. The EU's Digital Operational Resilience Act, which has applied across EU financial entities since January 17, 2025, expects ICT risk management, major incident reporting and a Register of Information covering third-party ICT dependencies.
Change control. A model registry with signed versions, held-out evaluation sets per use case, and a promotion gate from pre-prod to prod. Credit-scoring use cases sit under Annex III Point 5(b) of the EU AI Act as high-risk, with only fraud detection carved out. US bank model risk historically sat under the Federal Reserve's SR 11-7 guidance, which remained the reference until SR 26-2 replaced it in April 2026.
Third-party risk. The 2023 Interagency Guidance treats on-prem vendors as third parties with a full lifecycle, not a questionnaire at purchase. That lifecycle sets the rhythm for every renewal, model swap and integration change.
- 1Planning
- 2Due diligence and third-party selection
- 3Contract negotiation
- 4Ongoing monitoring
- 5Termination
What You Own at Go-Live
A banking buyer who follows this shape ends up owning one coherent system: a tiered hardware estate matched to workloads, the cstmAI stack with identity and logging on from the first login, a model registry with signed versions and evaluations attached, and connectors into the systems that actually hold the bank's data. Two-week discovery, then 8 to 12 weeks to production, is the typical shape of the delivery.
The question worth putting to a vendor is this: show the audit record of a single prompt, end to end, from user login through model version, retrieved documents, reviewer approval and response, with the examiner-grade trail attached. A good answer is one query in the Govern layer and a complete record. If the answer involves exports to spreadsheets or stitched logs from three systems, the stack has not been built for a bank.