A model risk team at a large US bank kept a printed copy of a 2011 letter on the wall through every machine learning project it approved, and the letter, SR 11-7 from the Federal Reserve, still sets the tempo for how ML reaches production in American finance. This guide walks through how machine learning in finance is actually built, governed, and shipped at US banks and fintechs, from the feature store to the audit trail. The mechanics matter because the same data pipeline that approves a credit card application in seconds also has to defend itself in a fair lending exam.
The data and feature pipeline that comes before the model
Most production ML at US banks starts not with a model but with a feature store. A feature is a computed signal, such as the rolling thirty-day count of declined transactions on a card or the standardized debt-to-income ratio of an applicant. The feature store materializes those signals from raw transaction, account, and bureau data, and exposes them through a low-latency API for online scoring and a batch interface for offline training. Feast and Tecton are the two open or commercial feature stores most common in US fintech, with custom in-house equivalents at JPMorgan, Capital One, and Goldman Sachs.
The pipeline upstream of the feature store is where most of the engineering work lives. Raw data flows in from card networks, ACH and FedNow rails, deposit ledgers, credit bureaus including Experian, Equifax, and TransUnion, and identity vendors. The data lands in a lake, often on S3 or GCS, governed by a metadata layer such as Unity Catalog or AWS Lake Formation. A medallion-style structure is now standard: bronze for raw landing, silver for cleansed and joined, gold for production features. The Federal Reserve’s payment systems hub publishes guidance that pushes banks to document lineage end to end, and the medallion pattern makes that documentation natural.
Quality controls run at every stage. Schema validation, freshness checks, drift detection on input distributions, and PII redaction all run inline. Stripe Radar, the firm’s fraud machine learning system, has publicly described feature pipelines that ingest hundreds of signals per transaction in tens of milliseconds, with the same pipeline reused across training and serving so the model sees the same shape of data in both modes. The pattern matters because feature skew between training and serving is the most common cause of silent model failure in US fintech, and the medallion structure paired with a shared feature store removes the most obvious sources of that skew.
The model training loop and the experiments around it
The training loop at a US bank looks similar to other industries on the surface and very different underneath. The framework choices are familiar. XGBoost and LightGBM dominate for tabular credit and fraud problems, PyTorch and TensorFlow run for sequence and language tasks, and Hugging Face transformers have entered the picture for document understanding in lending and compliance. The training environment is usually a managed service like AWS SageMaker, Databricks, or Vertex AI, with a Git-tracked notebook or pipeline that records the exact data slice, hyperparameters, and code commit.
The difference shows up in the experiments around the loop. Every model has to be reproducible from a recorded random seed, a frozen feature snapshot, and a pinned dependency list. The model registry, often MLflow or a vendor equivalent, stores the artifact along with its training data hash, evaluation metrics, and a written model card. The model card is not a marketing document. It is the input to a model risk management review, and at a US bank that review can take longer than the model build itself.
Cross-validation choices matter for regulatory reasons. A credit risk model evaluated on a random shuffle of historical data tends to overstate accuracy, because future borrowers do not behave like past borrowers. US banks now use time-based validation, out-of-time test windows, and economic stress scenarios drawn from the Federal Reserve’s annual stress test design. TechBullion’s AI in financial services coverage has documented case studies showing that this discipline cuts model performance degradation in production by roughly half over an eighteen-month horizon.
Model risk governance and SR 11-7
SR 11-7 is the supervisory letter the Federal Reserve and the OCC published in 2011 on model risk management. It defines a model broadly, requires independent validation, demands documentation of conceptual soundness, and asks for ongoing monitoring. The letter predates modern machine learning but applies to it directly. Every US bank has a model risk management function that signs off on new ML models, recertifies them annually, and tracks them in a central inventory. At the largest US banks the inventory contains thousands of models, with each one tied to a documented owner, validator, and use case.
For ML, SR 11-7 has been operationalized through a set of explainability and fairness controls. SHAP values are the standard tool to explain why a model produced a specific score. Adverse action notice generation, required by the Equal Credit Opportunity Act and Regulation B, has to produce reasons in plain English drawn from those explanations. The Consumer Financial Protection Bureau’s research reports library reinforces that creditors cannot hide behind algorithmic complexity. The reasons have to be specific, accurate, and complete.
Fair lending controls run alongside. Models are tested for disparate impact across protected classes using statistical tests such as the standardized mean difference and the four-fifths rule. A model that fails the test does not go to production, regardless of its overall accuracy. The pattern that TechBullion’s regtech compliance overview tracks is a steady shift toward automated fair lending pipelines that run before every model promotion, rather than as a one-time pre-launch check.
The MLOps stack that ships and watches the model
Shipping a model to production at a US bank is a discrete event, not a continuous deployment. The deploy passes through change management, with a recorded approver and a rollback plan. The serving layer, often Seldon, KServe, SageMaker endpoints, or Vertex AI endpoints, exposes the model behind an internal API with the same latency, observability, and security baseline as any other service. Inference latency targets are tight. A fraud model in a card authorization path has roughly fifty to one hundred milliseconds end to end, which leaves a model perhaps twenty milliseconds of compute time.
Monitoring is where the most engineering effort goes after launch. Input drift, output drift, and outcome drift are tracked continuously. Champion-challenger setups run multiple models in parallel, with traffic routed by policy. When drift is detected the model risk management team is notified, and a documented response runs, ranging from retraining to a temporary rule-based fallback. Champion-challenger setups also feed the next round of model development, because the production traffic seen by the challenger is the cleanest training signal a US bank can collect on its own customers.
What the next two years change
Three shifts will reshape US financial ML through 2027. The first is on-policy adoption of large language models for narrow, defensible use cases such as document understanding, complaint triage, and analyst research assistance, with retrieval-augmented generation and tightly bounded outputs. The second is real-time feature engineering on streaming data, with Apache Flink and Kafka Streams pipelines feeding fraud and credit decisions on data that is seconds old rather than days. The third is regulatory machine-readability, where the same pipeline that ships a model also produces the evidence package a federal examiner needs. TechBullion’s digital banking trends coverage tracks how these shifts move from lab to live traffic at the major US neobanks and the digital arms of incumbent banks. The teams that ship the cleanest model governance pipelines now are writing the documentation that will set the audit bar for US financial ML by 2028, and the firms that treat that documentation as a product rather than a project will define the operating standard for the rest of the decade.
