Scaling a Corporate Travel Platform with an Apache Iceberg Data Lakehouse

Introduction

The client is a leading corporate travel technology platform. As its customer base and transaction volume grew, so did the demands on its data: more reporting, more analytical workloads running at the same time, and a growing set of AI and machine learning initiatives.

 

The legacy data warehouse could not keep up. It used an OLTP database for analytical processing, which degraded performance as demand grew.

 

The client needed a modern architecture that could support today's reporting and tomorrow's AI-driven applications on the same foundation, and a partner who could build it.

The Challenge

The core problem was that the data warehouse was built on a transactional database and was being asked to do analytical work. The result was a set of problems that built on each other:

• Slow ETL: ETL jobs consistently took 7 to 8 hours to complete.
• Resource contention: Concurrent analytical workloads competed for resources and were throttled.
• Analytics on an OLTP database: Using the transactional database for analytical processing degraded performance.
• Coupled compute and storage: With the two tightly linked, they could not scale independently.
• Repeated transformations: Multiple downstream consumers kept transforming the same datasets, which increased compute costs.
• Pressure from AI and ML: Growing AI and machine learning initiatives required a scalable, analytics-optimized data platform.

The Approach

We designed a cloud-native lakehouse architecture on Apache Iceberg to replace the legacy OLTP-based warehouse, and moved from repeated ETL to an ELT model: raw data is loaded once, then transformed with distributed Apache Spark jobs. Four architectural decisions define the platform:

Apache Iceberg as the Data Lakehouse

Apache Iceberg became the organization's central analytical storage layer. It brings:

• Open table format
• ACID transactions
• Schema evolution
• Time travel
• Partition evolution
• Efficient metadata management

This let the platform scale without the operational limits of the previous OLTP database.

Separating Compute and Storage

Storage moved to Amazon S3, while compute runs independently through Apache Spark on AWS Glue. The benefits:

• Independent scaling of storage and compute
• Lower infrastructure costs
• Elastic processing capacity
• Improved workload isolation
• Better support for concurrent analytics

Load Once, Transform Many

Instead of repeatedly extracting data for different downstream systems, the new architecture ingests raw data a single time into Iceberg tables. It is then transformed into curated datasets that many consumers can reuse. This significantly reduced duplicate processing and keeps business logic consistent across the organization.

Distributed Processing with Apache Spark

Apache Spark powers the ELT pipeline. It:

• Cleanses raw operational data
• Applies business transformations
• Builds curated Gold datasets
• Processes large datasets in parallel
• Produces analytics-ready data for reporting and AI

Spark's distributed execution improved scalability dramatically compared to the legacy environment.

The Technical Architecture

The diagram below shows how data moves from the client's operational systems (booking, travel inventory, CRM and loyalty, and payments) through AWS Glue ingestion into Iceberg tables on Amazon S3, through a Bronze, Silver, and Gold Spark ELT, and out to Amazon QuickSight dashboards and AI and ML workloads.

The Outcome

The new architecture delivered clear operational improvements in speed, scale, and concurrency.

• Faster data processing: Long-running analytical workloads caused by OLTP limitations are gone, and distributed Spark processing significantly reduced ETL execution times.
• Improved scalability: The platform scales with growing bookings, customers, and historical data, and storage expands without affecting compute performance.
• Better concurrency: Multiple analytical workloads run at the same time without database contention, with less throttling and fewer resource conflicts.
• A foundation for AI: The Gold data layer is now a trusted source for business intelligence dashboards, machine learning feature engineering, predictive analytics, AI model training and inference, and future generative AI applications.

Area	Before	After
Analytics workloads	OLTP database used for analytics	Analytics-optimized lakehouse architecture
Scaling	Compute tightly coupled with storage	Independent compute and storage scaling
ETL processing	Long ETL execution windows	Distributed Spark-based ELT processing
Concurrency	High concurrency contention	Parallel analytics workloads
Transformations	Repeated data transformations	Load Once, Transform Many architecture
AI readiness	Limited support for AI	Unified platform for BI and AI

By modernizing its analytics platform with an Apache Iceberg lakehouse, the client now has a scalable, cloud-native data foundation that supports both enterprise reporting and advanced AI initiatives. The new architecture improved performance, reduced operational complexity, and lets the organization process growing volumes of travel data efficiently, while accelerating insights for business users and data scientists alike.

Technology Used

The stack behind the lakehouse:

• Table Format: Apache Iceberg
• Processing Engine: Apache Spark
• Storage: Amazon S3
• Serverless ETL: AWS Glue
• Business Intelligence: Amazon QuickSight