Modernizing Core Banking Analytics: Migrating a Legacy Data Warehouse to Apache Iceberg on AWS with Dagen

Financial services institutions run some of the most complex analytical estates in any industry. At the center of most sits an enterprise data warehouse fed by the core banking system: thousands of ETL mappings built over decades, loading nightly batches into proprietary warehouses or appliances such as Oracle Exadata or Teradata that carry heavy licensing costs and hard scaling limits. At the same time, analytics infrastructure built solely for humans fails to serve agentic systems because they lack fine-grained access control for non-human identities, semantic context attached to the data, and queryable lineage — the three properties agentic workloads require. A bank migrated its core banking data warehouse to an Apache Iceberg data lake on Amazon Simple Storage Service (Amazon S3), AWS's object storage service that provides virtually unlimited scalability for data lake storage, and on-premises storage, working with Dagen, an AWS Partner, to automate the translation of 1,012 ETL mappings.

Dagen's AI agents translated the warehouse's proprietary transformation logic into open, testable Spark pipelines that run on Amazon EMR, AWS's managed cluster service for running Apache Spark and other big-data frameworks at scale, and, where data residency requirements apply, on the bank's existing on-premises Spark environment. Because Apache Iceberg is an open table format that data warehouses, data lakes, hadoop clusters and different analytics engines can read and write across multiple engines including Spark, Amazon Athena, and Trino, the bank operates a single set of analytics stack across both on-premises and cloud. In this post, you will learn how a bank migrated its core banking data warehouse to an Apache Iceberg lakehouse on Amazon S3, the migration methodology Dagen used, and the measured results in pipeline velocity, data availability, and validation effort.

The business problem

This blog references an anonymous core banking customer operating in Türkiye. In the legacy environment, transactions generated by the core banking system were transformed using Oracle Data Integrator and loaded into an Oracle Exadata enterprise data warehouse, where Power BI supported regulatory, finance, risk, and management reporting. At the start of the modernization program, the bank’s analytical estate contained 1,290 GB of data across 26,740 warehouse tables and 1,012 ETL mappings, representing more than 7.85 billion records. For decades, this pattern remained largely unchanged. The data warehouse is one of the largest single line items: appliance hardware refreshes, per-core database licensing, and ETL tool renewals, all coupled together so that the cost of any one component cannot be reduced without migrating the whole.

Four pressures pushed the bank to act.

1. Regulatory frameworks require demonstrable, point-in-time lineage from the core banking source through every transformation to the regulatory report, which the legacy warehouse could only reconstruct manually. This includes BCBS 239 principles and the corresponding Turkish supervisory requirements from the BDDK.
2. The bank operates in a regulated environment with data residency and sovereignty requirements, so certain datasets must remain on infrastructure under the bank's direct control, while its AI roadmap required open, queryable data that the proprietary warehouse format could not serve without expensive extract layers.
3. The bank’s agentic AI roadmap required a data foundation that machine consumers can use: governed access for non-human identities, semantic definitions attached to the data itself, and lineage that explains every transformation.
4. An approaching license renewal put a deadline on the build-versus-renew decision.

The destination was clear. Because Iceberg decouples data from any single vendor's engine, it also gave the bank a format it could run both on AWS and on premises where sovereignty rules require, without maintaining two codebases. The hard part was never choosing Iceberg: it was migrating thousands of interdependent ETL mappings that encode the bank's business logic, and manual migration of a single complex mapping takes days of scarce specialist time. Most warehouse takeout programs stall at this point. The bank brought in Dagen to automate the translation.

Solution architecture

The target architecture replaces the proprietary warehouse and ETL stack with an open lakehouse on AWS. In this architecture, an open lakehouse means that analytical data is stored in the open Apache Iceberg table format on Amazon S3 and can be accessed by multiple processing and query engines without requiring proprietary format conversion or dependence on a single technology vendor:

• The core banking system's Oracle databases feed the lake through two paths: Oracle GoldenGate provides log-based change data capture for continuous incremental changes, while bulk extraction handles historical loads. For teams without existing GoldenGate licensing, AWS Database Migration Service (AWS DMS), a fully managed service for migrating databases to AWS, provides a managed CDC alternative.
• Amazon S3 serves as the landing zone and the storage layer for all Apache Iceberg tables, organized in raw, curated, and consumption zones.
• Spark pipelines generated by Dagen run on Amazon EMR. For the subset of datasets subject to data residency requirements, the same pipeline definitions run on the bank's on-premises Spark environment, writing Iceberg tables to storage under the bank's direct control. One codebase, two runtimes.
• Amazon Athena, a serverless query service that runs SQL directly on data in Amazon S3 with no infrastructure to manage, offers regulatory reporting, ad hoc analysis, and BI consumption directly on Iceberg tables, with no extract layer.

Why Iceberg for core banking analytics

Iceberg's features map directly to banking requirements. Time travel and snapshot isolation give the bank point-in-time reproducibility: any regulatory report can be regenerated exactly as the data stood at the reporting date, which turns lineage reconstruction under BCBS 239 and BDDK supervisory review from a manual project into a query. ACID transactions allow concurrent CDC writes and reporting reads during month-end windows without contention. Schema evolution absorbs changing regulatory templates without table rewrites. Hidden partitioning keeps query performance predictable on time-series transaction data without analysts needing to know the partition scheme. And because Iceberg is an open specification, the same tables serve EMR, Athena, and the bank's on-premises engines (Spark, Trino, and others) without format conversion.

Building the semantic layer for AI-ready consumption

Migrating tables is necessary but not sufficient for the bank’s agentic roadmap. A machine consumer retrieving a column named TXN_ST_CD has no way to know it holds a settlement status, whose values are terminal, or that its meaning changed after a 2019 core banking upgrade. As part of the curated-to-consumption promotion, the team attached business definitions, ownership, and quality-rule metadata to the Iceberg tables through AWS Glue Data Catalog, the centralized metadata repository that stores table schemas and makes them discoverable across analytics engines, so that the same governed definitions serve BI analysts and AI agents alike. Combined with AWS Lake Formation’s fine-grained access control, which enforces permissions at the table, column, row, and cell level, this gives non-human identities the three things the legacy warehouse could not provide: governed access, semantic meaning, and queryable lineage. The lakehouse costs less than the warehouse it replaced and supports the agentic workloads the bank is building next.

The migration methodology: Assess, Mobilize, Move

Dagen structured the migration around the three phases of the AWS migration methodology: Assess, Mobilize, and Move. Dagen's agents automated the analytical and engineering work in each phase and the bank's domain experts owned correctness at every gate.

Phase 1: Assess. Before moving a single byte, Dagen's assessment agents built a complete, evidence-based picture of the source estate. The agents parsed the legacy ETL tool's XML mapping repository, the warehouse's data dictionary, and the query history, and produced four assessment artifacts:

• Structural discovery: A full inventory of tables and views with schema-level type mapping to Iceberg, recommended partition and sort specifications per table, and an activity classification from query history that separates hot, warm, and cold objects, so rarely touched tables could be archived rather than migrated.
• Lineage and dependency analysis: A complete dependency graph across staging, dimension, fact, and mart layers, including cross-dataset dependencies and downstream consumers. The dependency analysis catches broken objects early — views referencing sources that no longer exist — before they cause production failures after cutover.





• Access control and security analysis: The existing warehouse permission model mapped to an AWS Lake Formation equivalent, with sensitive and PII columns flagged for column-level permissions. Because the legacy warehouse had no fine-grained access control, the assessment turned the migration into a security upgrade, not just a like-for-like port.
• Migration readiness report: A readiness score across schema compatibility, dependency complexity, security, and operational readiness; a wave plan sequenced by the dependency graph; a risk register with concrete mitigations; and an effort estimate per phase.

Each mapping received a complexity score, which drove the wave plan. The bank's data, risk, and finance teams then reviewed the generated dependency graph, validated the business-critical data flows, and prioritized the waves against regulatory reporting deadlines, business impact, and operational risk. Automated analysis replaced months of consultant discovery, with expert review — and every downstream phase inherited its plan from this evidence base.

Phase 2: Mobilize. With the assessment as the blueprint, the team built the lakehouse foundation and rebuilt the estate on it. The landing zone structure, Iceberg table designs, AWS Glue Data Catalog configuration, and Amazon EMR runtime baseline (Amazon EMR 7 (Spark 3.5)) came directly from the structural discovery, and the Lake Formation permission model was implemented from the security analysis — including the PII column protections the legacy warehouse never had. A hybrid historical load moved decades of history: parallel JDBC reads for tables under 10 GB and native bulk export to compressed Parquet staging for larger tables, loaded into partitioned Iceberg tables on Amazon S3. Then, wave by wave, Dagen's agents translated each legacy mapping's transformation logic into declarative PySpark. Every generated pipeline is human-readable, version-controlled, and ships with generated unit tests, so the bank's engineers review and own the code rather than inheriting a black box. Residency-scoped datasets ran the same pipeline definitions on the bank's on-premises Spark environment — the same pipeline definitions ran without modification.

Phase 3: Move. Validation gated the cutover. Every migrated table went through automated reconciliation against the legacy warehouse at four tiers (schema conformance, row counts, column-level aggregates, and row-level comparison), checked against the baselines captured during Assess. Dagen generated the reconciliation results and highlighted material differences; the bank's subject-matter experts reviewed the exceptions, validated the underlying business rules, and approved each data flow for production. The validation reports double as audit evidence for the bank's regulators. Change data capture then switched each subject area to continuous incremental loading, with legacy and lakehouse pipelines running in parallel for full reporting cycles before decommissioning. Post-cutover, Dagen monitors the production pipelines and automatically repairs common failure modes, such as schema drift from core banking system patches.

Results at a glance

Quantified outcomes:

• 86% of legacy mappings translated automatically
• Migration timeline: 1.5 months with 4 data engineers versus the plan of 6 months with 8 data engineers manually
• 100% of migrated tables reconciled at four validation tiers

Conclusion

The bank replaced a proprietary warehouse and ETL stack with an open lakehouse: Apache Iceberg on Amazon S3, pipelines on Amazon EMR, consumption through Amazon Athena, and a single codebase spanning AWS and residency-scoped on-premises infrastructure. The migration pattern generalizes to any institution facing a warehouse renewal decision: automate the assessment and translation with Dagen's agents, anchor banking correctness with an experienced delivery partner, and let Iceberg's open format eliminate the lock-in that created the problem.

About the Authors