I build the pipelines other people's dashboards quietly depend on. Four years of it, all at Infocepts — Glue and PySpark on AWS, Terraform underneath, and enough production incidents to have opinions about error handling. Given the choice I take the unglamorous work: the job that runs at 3am and nobody thinks about, because that is the one that has to be right. Lately I have been pointing the same instincts at applied AI, where the interesting problem is still data movement — just with a more expensive consumer at the end of the pipe.

Four years of moving data around have left me convinced the hard part of applied AI is not the model. It is the same problem it has always been: getting the right data to the right place, on time, in a shape someone can trust.

The problem. The platform pulled from several upstream systems with nothing in common — different shapes, different schedules, different definitions of "late". Every new source meant another bespoke script, and because each one handled its own failures, none of them handled failures well. The batch jobs were fine. The near-real-time paths were where things quietly went wrong.

The architecture. Everything lands in S3 first, raw and unmodified, before anything is allowed to interpret it. Batch sources come in through Glue jobs written in PySpark; event sources arrive over API Gateway into Lambda, buffered through SQS so a slow consumer creates a queue rather than a data loss. Jenkins schedules and gates the batch side. The boundary that matters is the landing zone: upstream of it, anything can be true; downstream of it, the shape is guaranteed, which is what makes the transformation layer possible to reason about at all.

The interesting decision. Buffering the event path through SQS rather than writing straight through from Lambda. It costs a hop and it costs latency, and for a while it looked like unnecessary machinery. It stopped being unnecessary the first time a downstream dependency went slow instead of going down — the queue absorbed it, and the alternative would have been partial writes we would have found out about days later from a business user. The general lesson I keep re-learning: the failure mode you should design for is not "it broke", it's "it half-worked and nobody noticed."

The real-time path failed in ways the batch path never did. A downstream service being slow or briefly unavailable did not just delay a message — it left records half-applied, so the failure showed up later as inconsistent data rather than as an error anyone was paged for.

Built a real explicit error-handling layer across the streaming path: retries with backoff, dead-letter queues on the SQS consumers, and failures surfaced as failures instead of as silence. System failures and data inconsistencies dropped by around 25%.

The problem. Ingestion solved getting data in. It did not solve anyone being able to use it. Analysts were querying close to raw tables, which meant every dashboard encoded its own quiet interpretation of what a metric meant, and two dashboards could disagree without either being wrong. Query cost and query time were both climbing for reasons nobody could point at.

The architecture. Curated data lands in Redshift as modelled tables rather than as mirrors of the source systems. Denodo sits above that as a virtualization layer, so consumers hit one consistent surface instead of learning which physical table is the current one. QuickSight reads from there. Storage stays tiered across S3 and Redshift by how often something is actually touched, which is where most of the cost reduction came from — roughly 30% lower storage cost, without deleting anything anyone still needed.

The interesting decision. Putting a virtualization tier in front of the warehouse instead of just publishing more views. Views would have been simpler and one fewer system to run. The argument for Denodo was that it decouples what consumers depend on from what the platform is physically doing, so a migration underneath does not become a coordination problem with every dashboard owner in the company. That is a bet on future change being likely — a reasonable engineer could have taken the other side of it and been right for a couple of years.

Analysts were querying close to raw tables, so every dashboard encoded its own quiet interpretation of what a metric meant — two of them could disagree without either being wrong. Query times and storage costs were both climbing, and nobody could point at the reason.

Modelled the curated layer in Redshift and put Denodo in front of it, so consumers depend on one surface instead of learning which physical table is the current one. Performance tuning brought query execution times down by about 35%; tiering storage across S3 and Redshift by how often something is actually touched cut storage cost by about 30%, without deleting anything anyone still needed.

The problem. The pipelines were code and the infrastructure they ran on was not. Environments were assembled by hand, which meant they drifted, which meant "works in dev" carried no information. Onboarding a new environment was an exercise in remembering, and the only real documentation was whoever had done it last.

The architecture. Terraform for the long-lived account-level infrastructure, AWS CDK where the resources are tightly coupled to application code, and a shared library of reusable modules so that a bucket with the right encryption and lifecycle rules is the default thing you get rather than something you have to remember to configure. Jenkins and GitHub Actions run plan on every pull request and apply on merge. Manual effort per deployment fell by about 40%, but the number that mattered more was the one I cannot put a percentage on: an environment change became something two people had looked at.

The interesting decision. Running Terraform and CDK side by side rather than standardising on one. Two IaC tools is a genuine cost — two mental models, two state stories, two ways to be wrong. I kept both because they are good at different things: Terraform for infrastructure with a long life and a slow change rate, CDK where the infrastructure and the code that uses it change in the same pull request. Forcing everything into one would have meant either writing application-shaped resources in HCL or managing account foundations from a language runtime, and both of those are worse than owning the seam.

The pipelines were code and the infrastructure under them was not. Environments were assembled by hand, so they drifted, so "works in dev" stopped carrying information. The only real documentation of how an environment got built was whoever had built one last.

Moved account-level infrastructure into Terraform and application-coupled resources into AWS CDK, behind a shared library of modules so that a correctly configured resource is the default you get rather than something you have to remember. Plan runs on every pull request, apply on merge — roughly 40% less manual effort per deployment, and every change seen by a second person.