The costliest mistake is pasting a 50-page loan committee memo into a public chatbot to summarize the covenants, then learning that your compliance officer has to log it as a reportable incident, that the vendor's terms let the text sit in a training queue, and that the names in the memo now live outside the bank's four walls. Samsung lived a version of this in 2023, when engineers pasted confidential source code into ChatGPT in three separate incidents inside a 20-day window, and the company responded with a hard ban on public generative AI across its devices and networks. For a bank, the equivalent paste is a KYC file, a draft SAR, a client's holdings, or the minutes of a loan committee. That is the moment the on-prem LLM for banks conversation starts for real.
What a Banking LLM Does All Day
Big-bank deployments are no longer pilots. JPMorgan Chase is rolling out its internal LLM Suite assistant to roughly 140,000 employees, and Morgan Stanley made its advisor assistant generally available in September 2023, with a corpus of about 100,000 research reports behind it. Strip the headlines and the real work is six workflows that read a lot of private text and write short private text back.
- AML narrative drafting. A model reads transaction clusters, prior SARs, and policy, then drafts the narrative an investigator edits before filing.
- KYC adverse media and sanctions screening. A model reads news, filings, and watchlist hits and summarizes the match in a form a reviewer can defend.
- Trade surveillance explanations. A model reads chat, voice transcripts, and order history and proposes why an alert fired, in the language the lookback review uses.
- Credit and loan memo drafting. A model assembles a first draft from spreads, borrower filings, and the policy manual.
- Call summarization. A model produces structured call notes, action items, and disclosures that feed CRM and compliance.
- Policy and procedure Q and A. A model answers "what does our policy say about X" against the controlled version of the policy, with cited paragraphs.
Every one of those pipes a document the bank cannot afford to leak into another document the bank cannot afford to leak. OWASP ranks sensitive information disclosure as the number two risk in its 2025 Top 10 for LLM Applications. The regulator wants to see where the paragraph came from. The customer wants to know why the SAR mentions them. Both answers are inside the bank's four walls, or they are not credible.
Why Private Cloud Is Not the Same as On-Prem
A model endpoint hosted in your tenant on a hyperscaler is still a tenant. The weights are not yours. The logs transit someone else's control plane. The audit evidence is a contract clause rather than a cable you can trace. On-premises LLM security starts from a different place: GPUs, model weights, vector store, gateway, and audit log sit on hardware the bank owns or leases inside its own data center or colocation cage, with inference traffic that never crosses a public network. That is the configuration that lets the second line say in writing that no customer record entered a third-party inference service.
Matching GPUs to Each Workflow
Model size drives everything. A 7B model will summarize a call acceptably and quantize onto a single workstation GPU. A 70B model is the point at which drafting a loan memo stops reading like a toy, and still needs serious VRAM to serve concurrent users at low latency. A 405B class model is a cluster conversation, not a workstation conversation.
In practice, banks run several models in parallel. A smaller fast model handles call notes and routing. A mid-sized model handles AML narratives and KYC summaries. A larger model is reserved for memo drafting and policy work where quality matters more than tokens per second. That is why vendor-neutral AI systems matter more than any single GPU spec sheet. The right box for AML batch work is a rackmount inference server. The right box for a satellite branch running offline KYC is a small ruggedized node. Buying one shape for all of them is the second costliest mistake after the public-chatbot paste.
The Software Layer Keeps Data Inside the Perimeter
GPU server for banks is a very expensive paperweight without an opinionated stack on top. The pieces banks need are a model-serving gateway with per-user tokens and per-tenant key wrap, a vector index over the bank's own documents, an evaluation harness that can replay prompts against last quarter's data, an orchestration layer for agents that call internal APIs, and an audit log that records every prompt, retrieval, and response in a form the examiner can read. That is the argument for a packaged on-prem software stack instead of a drawer of open-source parts a platform team wires up on their own time.
Document AI is the quietest beneficiary. The same cluster that drafts a loan memo can be the one that extracts covenants from a signed PDF, classifies incoming correspondence, and surfaces the clauses a reviewer flags. The sector-level view sits in a separate piece on AI hardware for financial services. The point here is that the workflows above all share a software spine, and that spine is the thing that enforces the controls your examiners already ask for.
The Risk Model the Regulator Already Expects
Banking AI compliance is not a new idea built around LLMs. It is why the Bank of England PRA's Supervisory Statement SS1/23 on model risk management, which took effect on 17 May 2024, applies model-agnostically to AI and machine learning. It is why the EU's Digital Operational Resilience Act became applicable on 17 January 2025, mandating ICT risk management, resilience testing, and third-party oversight across more than 20 types of financial entities. And it is why the final set of amendments to NYDFS Part 500 took effect on 1 November 2025, requiring MFA for every user accessing any information system and a complete written asset inventory.
Those obligations push in the same direction. Know every model you run. Know every user who touches it. Know every document the model has read. Custom AI for financial services makes that evidence straightforward to produce because the full stack sits on hardware the bank controls. A hyperscaler endpoint can produce most of it, but not all of it, and the missing evidence is where examiners spend their time.
Where the Real Trade-off Lives
The honest giveback is speed and flexibility. On-prem means you buy, rack, cool, and operate the hardware. You cannot double capacity by changing a tier in a console. A 70B refresh means an engineering change rather than a dropdown. You will build vendor-neutral abstractions so you are not locked to one GPU line, and that work is not free. In exchange you get inference traffic that never leaves the bank, model weights your legal team can point at, an audit log you own, and a cost curve that stops bending upward with every token. For the six workflows above, where the input is customer data and the output is read by regulators, that trade is usually the one that lets you ship.