Partitioning, Compaction and the Small-File Problem

The lake's two enemies: too many tiny files, wrong partitions

Partitioning physically separates rows by a column (event_date=2026-05-27/...). Done right, scans skip 99% of the data. Done wrong (high-cardinality column like user_id) you create millions of microscopic files and the engine spends its life opening them.

Compaction (OPTIMIZE in Delta, rewrite_data_files in Iceberg) merges those microscopic files into ~128MB–1GB chunks — the sweet spot for object storage + columnar reads.

Rules of thumb: partition on a low-cardinality column the queries actually filter on (usually a date); run compaction on a schedule; never partition on something with millions of distinct values.

Partitioning is how you file paper in a filing cabinet. File by month (low cardinality) and finding 'everything from May' means pulling one drawer. File by customer's full name (high cardinality) and you get a million one-sheet folders — finding anything means opening every drawer in the building. Compaction is the office junior who comes in at night and merges those thousands of one-page folders into tidy hundred-page binders, so tomorrow's search touches a handful of binders instead of a blizzard of loose sheets. Skip the junior's night shift and the cabinet silently grinds to a halt.