Case Study - Data lake migration takes dashboards from hours behind to minutes
A fintech running payables and receivables automation had outgrown its on-prem analytics stack. We moved it to a cloud data lake with change-data-capture replication, and took BI freshness from roughly three hours behind to roughly three minutes — without pausing financial close reporting.
- Client
- BILL
- Year
- Service
- Data engineering, cloud migration
Overview
BILL automates payables and receivables. The analytics behind that ran on an on-prem cluster fed by Oracle and SQL Server, and it had stopped keeping up — data volume and query concurrency had grown past what the hardware could serve, so BI dashboards ran minutes behind during exactly the hours people wanted them.
The ordinary fix was more hardware, and more hardware meant a procurement cycle measured in months. Engineering had no route to capacity on the timescale the business was asking questions on.
The constraint that shaped the work was simpler than any of that: financial close reporting could not pause. There is no maintenance window in which a finance organisation agrees to stop closing the books.
The constraint
Migrate underneath a running business. Reporting had to stay correct and available throughout, which rules out any plan whose safety depends on a quiet period.
Prove parity, don't assert it. Moving a warehouse is easy; convincing a finance team the numbers still tie out is the actual job. Verification had to be mechanical and checkable, not a spot inspection.
Fixed window. The mandate was twelve weeks, and the value of finishing in twelve weeks was mostly that it removed the argument for buying hardware instead. Architecture was held in the US and engineering ran from our team in Vadodara, which is how a twelve-week window absorbs a migration of this size without a large onshore bill.
What we built
- AWS DMS change data capture
- S3 data lake
- Glue and Lambda ETL
- Athena and Redshift Spectrum
- Step Functions
- OAuth 2.1
Continuous replication rather than a cutover
The historical load matters less than the ongoing one. We ran change-data-capture tasks per source schema into a partitioned, versioned S3 landing zone, tuned until replication lag sat in single-digit seconds. Once the lake is continuously current, the cutover stops being an event — the new platform is simply already correct, and switching readers over is a decision rather than a risk.
Transformation is event-driven: an object landing in the raw zone triggers the work that curates it into columnar files for query. Orchestration runs on Step Functions, which keeps the whole pipeline serverless and makes the cost of an idle pipeline close to zero — the point being that engineers stopped waiting on hardware to try things.
Parity by checksum
Every migrated dataset was verified by checksum against its source rather than sampled. That is unglamorous and it is the reason the finance organisation signed off.
Results
Replication settled at a median CDC lag of 6.8 seconds. Data freshness for BI dashboards went from roughly three hours behind to roughly three minutes behind — the difference between a dashboard you consult and a dashboard you can act on.
200 TB was migrated and verified with zero checksum variance. Financial close reporting ran uninterrupted throughout.
A separate strand replaced a home-grown SAML login flow, which a security audit had flagged against refreshed SOC 2 controls, with a standards-based OAuth 2.1 proof of concept supporting Google Workspace, Microsoft 365 and Okta.