Optimizing Petabyte-Scale Parquet Storage for a Travel Data Solutions Provider

Arbisoft benchmarked and tuned the Parquet storage of a petabyte-scale data lake. Zstandard compression and larger, fewer files cut storage by about 25% and file counts by over 80%, saving approximately USD 350,000 a year.

Client/Partner

Confidential

Industry

Travel

Service Offered

Data Engineering & Storage Optimization

Tech Stack Used

The client is a travel data solutions provider whose data lake holds more than 1 PB of Parquet data in Amazon S3. Default settings had left that data expensive to store: Snappy compression used more space than it needed, and Trino writes had produced millions of small files.

Arbisoft's data engineering team built a structured benchmarking framework and tested storage configurations on production-scale datasets, instead of relying on engine defaults. Only changes that proved themselves were promoted into production.

The result is about 25% less storage with Zstandard compression, over 80% fewer Parquet files, and estimated savings of approximately USD 350,000 per year, with query performance maintained.

Introduction

The client is a travel data solutions provider. Its data lake holds more than 1 PB of Parquet data in Amazon S3. Parquet is an efficient columnar storage format, but the way it was being used had become increasingly expensive and operationally inefficient.

The client needed to reduce storage costs without disrupting the analytical pipelines that depend on the data, and without compromising data quality or write and query performance.

The client needed a partner who could show, on production-scale data, which changes were safe to make.

The Challenge

The core problem was that default settings were still in place at petabyte scale. This led to a set of problems that built on each other:

High storage consumption: The default Snappy compression used more storage than necessary.
Millions of small files: Trino write operations had generated millions of small Parquet files.
Rising S3 costs: Amazon S3 storage costs kept increasing.
Poor file layout: Fragmented files hurt downstream analytics workloads.
Tight constraints: Storage had to be optimized while preserving data quality and acceptable write and query performance.

The Approach

We designed a structured benchmarking and optimization framework to find the most efficient Parquet configuration, and validated every option on production-scale datasets before it reached production. The work had four parts:

Benchmarking Compression

We evaluated Snappy and Zstandard (ZSTD) to find the best balance between storage savings and processing performance. Each configuration was measured for:

Storage footprint
Write throughput
Query performance
CPU utilization

The benchmarks showed that ZSTD significantly reduced storage requirements while keeping read and write performance acceptable for analytical workloads.

Reorganizing the Parquet File Layout

The existing datasets contained a very large number of small Parquet files created during Trino writes. We tuned the Parquet writer configuration to:

Increase the Parquet block size to generate larger files
Reduce file fragmentation
Improve row group organization

This gave downstream analytics a much healthier data layout.

Tuning the Trino Writers

We tuned Trino's write pipeline to minimize unnecessary file generation, optimizing:

Writer scaling
Writer task count
Output parallelism

The new configuration significantly reduced the number of output files while keeping write performance acceptable.

Validating Before Rollout

Every optimization was benchmarked against production-scale datasets before it was rolled out. The evaluation framework measured:

Storage size
Number of generated files
Write latency
Query execution performance

Only validated improvements were promoted into production, which kept operational risk to a minimum.

The Technical Architecture

The diagram below shows the optimized pipeline: travel data from operational databases, files, third-party providers, and streams is written by Trino into optimized Parquet files in Amazon S3, cataloged through the Hive metastore, and consumed by downstream analytics and ML workloads.

Architecture diagram: travel data sources feed Trino writers on Kubernetes, which write optimized Parquet files to Amazon S3, cataloged by the Hive metastore, for downstream analytics, BI, and ML workloads.

The Outcome

The optimization delivered substantial operational and financial benefits.

Metric	Result
Storage reduction	About 25% using ZSTD compression
File count reduction	Over 80% fewer Parquet files
Storage cost savings	Approximately USD 350,000 per year
Analytics performance	Improved data layout with maintained query performance
Operational efficiency	Reduced file fragmentation and simplified downstream processing