At an engineering all-hands at a US regional bank in March 2025, the data platform lead walked his team through the new architecture diagram and stopped at the box labeled feature store, where a junior engineer asked the obvious question, namely whether the firm could finally retire the four overlapping Hadoop clusters that had been running since 2017. The answer was a qualified yes, conditional on a 14-month migration to a lakehouse, a streaming bus, and a managed feature store. That conversation is now playing out in data engineering teams at almost every US financial institution, and it is the working version of the big data finance guide most architects are trying to write. Deloitte’s 2025 financial services outlook puts US bank spending on data platform modernization at more than $30 billion a year.
From data warehouses to lakes and lakehouses
The starting point in every US bank is the relational data warehouse, often Teradata or Oracle Exadata, that anchored the firm’s analytics for two decades. The warehouse remains the system of record for regulatory and financial reporting, because its consistency model is exactly what an examiner expects. What changed in the 2010s is everything else, namely the customer 360 view, the marketing data, the fraud feature set, and the machine learning training data, all of which outgrew the warehouse on cost and schema flexibility.
The data lake on cloud object storage absorbed that overflow. Most US banks now run Amazon S3 or Microsoft Azure Data Lake Storage as the raw storage layer, with the same data accessible through multiple compute engines. The lakehouse, the next iteration, layers a transactional table format on top of the lake. The two dominant players in US finance are Databricks, with the Delta Lake format, and Snowflake, with its Iceberg and native table support. Both let an engineer run SQL, Python, and Scala against the same data without copying it, and both add ACID guarantees that the older lake architectures lacked.
The practical impact for a US bank is that a single petabyte-scale dataset can serve regulatory reports in the morning, machine learning training in the afternoon, and live dashboards through the trading day, without three separate copies. The economics matter, because cloud object storage at one to two cents per gigabyte per month is roughly an order of magnitude cheaper than warehouse storage.
Batch and streaming in a single pipeline
The second technical shift is the convergence of batch and streaming. Batch processing, the workhorse of every US bank since the mainframe era, still runs the nightly close, the regulatory reports, and the back-dated reconciliations. The tooling has changed from COBOL on JCL to Apache Spark on Databricks or Snowflake, but the rhythm is the same, with files arriving overnight and reports landing by 6 a.m.
Streaming has come in alongside it. Apache Kafka, originally built at LinkedIn and now operated by Confluent and by every major US bank’s in-house platform team, has become the standard event backbone for US finance. A single Kafka cluster at a large US bank routinely moves 10 to 50 gigabytes per second across thousands of topics. Apache Flink, also widely adopted, processes the same events for stateful operations like windowed aggregations, fraud scoring, and intraday risk metrics.
The hard architectural decision is which workloads belong in batch and which in streaming. The rule that has settled in US banks through 2025 is that any business decision with a same-day deadline lives on the stream, and any process where consistency matters more than freshness lives in batch. A fraud score has to be in stream, because the merchant is waiting. A month-end financial close has to be in batch, because every transaction has to land.
Feature stores and the machine learning data layer
The third technical shift is the feature store. A feature store is a system that computes, stores, and serves the input variables, called features, that machine learning models consume. The pattern was developed at Uber, popularized by the open source Feast project, and is now in production at every large US bank, either through Tecton, Feast, Databricks Feature Store, or Snowflake’s native equivalent.
The point of the feature store is to compute a given feature once and serve it identically to the training pipeline and to the live model. Without that guarantee, US banks ran into the textbook training-serving skew problem, where a model that performed well in test failed in production because the feature was computed differently. A top-five US bank typically operates 5,000 to 15,000 features across hundreds of models, and the feature store doubles as the inventory and the access control layer for model risk purposes.
For US compliance teams, the feature store also closes an audit gap. Every feature has an owner, a definition, a refresh cadence, and a lineage record. When an examiner asks how a credit decision was made, the feature store can replay the exact feature values that were served to the model at the time of decision. That capability is now a standard ask in US Federal Reserve and OCC exam letters that touch model risk.
Governance, security, and the audit trail
Data governance has caught up with the new stack only recently. The dominant tools in US financial firms are Apache Ranger, Apache Atlas, and the native governance layers built into Databricks Unity Catalog and Snowflake Horizon. The governance layer enforces access at the column and row level, masks sensitive personally identifiable information, and produces the audit log that satisfies the Gramm-Leach-Bliley Act and the New York Department of Financial Services cybersecurity rule.
Encryption is table stakes. US bank examiners expect data encrypted in transit and at rest, with keys managed in a hardware security module that is itself audited. The cloud providers offer the building blocks, and US banks typically add their own key management layer on top so that the cloud provider cannot read customer data on its own. The pattern is documented in CISA’s financial services cybersecurity guidance and is reviewed in every US federal exam cycle.
The audit trail is where the platform earns its keep with regulators. Every read, write, schema change, model run, and feature pull is logged with the user identity, the source system, and a timestamp. The logs themselves are managed records under the Securities Exchange Act and the Bank Secrecy Act, and they have to be retrievable for seven years. The volumes are large, often hundreds of millions of events per day at a top-ten US bank, which means the audit log itself has become a streaming pipeline of its own.
What US data teams are building next
Three engineering bets will shape the next two years of big data analytics in US finance. The first is the unification of the operational data store and the analytical lakehouse. Several US banks have started experimenting with hybrid transactional analytical processing systems, sometimes called HTAP, including SingleStore, Databricks photon, and the operational table features inside Snowflake. The goal is to read the same row a teller updated 200 milliseconds ago without copying it through a change data capture pipeline.
The second is the move toward open table formats across vendors. Apache Iceberg has become a de facto standard, supported by Snowflake, Databricks, AWS, Google Cloud, and Microsoft Azure. US banks that adopt Iceberg avoid vendor lock-in and can move workloads between engines based on price and performance. TechBullion’s cloud finance modernization page and the AI in financial services hub track these architectural shifts as they happen.
The third is the convergence of big data analytics with the AI stack, because the same lakehouse now serves the training data, the feature store, and the retrieval index for retrieval augmented generation. Coverage of the resulting buildouts sits on the TechBullion fintech news hub. The next round of US Federal Reserve and OCC examinations, scheduled through 2026, will tell US data architects how aggressively the regulators expect the new stack to document and explain every data movement that ends in a customer decision.
