Modernizing ETL Architecture for an AI-Driven Food & Beverage Analytics Platform

Introduction

The client runs an AI-driven analytics platform for the food and beverage industry. Its customers depend on the platform for AI-driven insights and reporting, and both depend on the data pipelines behind them.

 

Those pipelines ran on a legacy ETL architecture built on Amazon ECS. Jobs ran periodically, processed large datasets stored in Amazon S3, and loaded the transformed data into an Amazon Redshift data warehouse. As data volumes grew, the setup threatened the platform's scalability and delayed the AI analytics and customer reporting built on top of it.

 

The client needed an architecture that was faster, cheaper to run, and ready for more data, and a partner who could redesign it rather than patch it.

The Challenge

The core problem was that the ETL layer repeated the same heavy work on every run, with little visibility into how that work was going. The existing implementation led to a set of problems that built on each other:

• Day-long pipelines: ETL pipelines ran for 24 to 48 hours, depending on the data source.
• High operating costs: ETL processing averaged about USD 5,000 a month.
• Full reloads on every run: Each execution processed the full dataset, regardless of how much new data had arrived.
• Limited observability: Little visibility into pipeline execution made failures hard to identify and partial job runs hard to recover from.
• Growth pressure: Increasing data volumes threatened the platform's scalability and delayed downstream AI analytics and customer reporting.

The Approach

We modernized both the processing layer and the way data is ingested, replacing long, full-dataset runs on ECS with distributed processing that only touches what has changed, and adding the monitoring the old setup lacked. The redesign had three parts:

Distributed Processing with Apache Spark on AWS Glue

We re-engineered the legacy ECS-based ETL jobs to run as Apache Spark jobs on AWS Glue. Spark distributes the work across multiple workers and processes it in parallel. The key improvements:

• Transformation workloads run in parallel
• Partition-based processing keeps each unit of work efficient
• Compute resources are used better and scale automatically
• AWS Glue keeps operations serverless and simple

Processing Only What Changed

Rather than reprocessing the complete dataset on every run, we introduced a Change Data Capture (CDC) architecture. The storage layer was redesigned to:

• Detect newly arrived or modified records
• Process only incremental changes
• Eliminate unnecessary full-table scans
• Reduce compute time and storage I/O
• Minimize data movement throughout the pipeline

Each execution now processes significantly less data, which improved the efficiency of the platform as a whole.

Pipelines You Can See and Recover

The modernized architecture also made the pipelines easier to operate, with:

• Centralized job monitoring
• Enhanced execution logging
• Better failure visibility
• Simplified troubleshooting
• Retries for failed jobs without reprocessing successful workloads

These changes made the platform more reliable and reduced the effort needed to maintain it.

The Technical Architecture

The diagram below shows how data moves from the client's operational systems into Amazon S3, through CDC detection and AWS Glue running Apache Spark, and into Amazon Redshift, with monitoring and observability across the pipeline.

The Outcome

The modernization improved both performance and cost efficiency.

 

Before the redesign, a pipeline run took 24 to 48 hours, ETL processing cost about USD 5,000 a month, and every execution reloaded the full dataset. Visibility into pipeline runs was limited, so failures were hard to spot and partial runs were hard to recover from.

 

After the redesign, runs finish in 2 to 4 hours, up to 12 times faster, and the monthly ETL cost is USD 800, down 84%. Data is loaded incrementally through CDC, processed on distributed, serverless compute, and monitored centrally, with failed jobs retried without reprocessing completed work.

Metric	Before	After
ETL processing time	24 to 48 hours	2 to 4 hours (up to 12x faster)
Monthly ETL cost	USD 5,000	USD 800 (84% lower)
Processing strategy	Full data reload	Incremental (CDC), with significantly lower compute overhead
Compute framework	Amazon ECS	Apache Spark on AWS Glue (distributed, serverless)
Operational visibility	Limited	Centralized monitoring and retries

Data that used to take days to become available now takes a few hours.

Value Delivered

The results go beyond infrastructure optimization. Each one is measurable business value for the platform and its customers.

• Data availability latency dropped from days to a few hours, which brings the platform's analytics close to real time
• Monthly ETL operating costs fell by about 84%, from roughly USD 5,000 to USD 800, improving the platform's overall cloud cost efficiency
• The architecture can support future growth in customer data volumes without infrastructure costs growing in proportion
• Failures are now visible, and failed jobs can be retried without reprocessing successful workloads, which makes the platform more reliable
• AI-driven insights and reporting reach customers in the food and beverage industry faster, improving their experience of the platform

The ETL layer stopped being the bottleneck. The same platform now gets its data in hours instead of days, at a fraction of the cost, and with room to grow.

Technology Used

The stack behind the modernized pipeline:

• Cloud Platform: AWS
• Storage: Amazon S3
• Processing Engine: Apache Spark
• Serverless ETL: AWS Glue
• Data Warehouse: Amazon Redshift
• Monitoring: AWS CloudWatch