Day 24 of DataBrickster
Liquid Clustering
A new way of organising data in Delta Lake tables that automatically optimises data layout without requiring static partitioning.
Unlike traditional partitioning (where data is rigidly divided by column values), Liquid Clustering allows:
- Smarter data skipping
- Dynamic data reorganisation
- Fewer small files
- Better performance for a wide range of queries
How is Liquid Clustering different from Traditional Clustering?
Physical Changes — Traditional clustering rewrites the files to sort them by specified columns, whereas Liquid clustering DOES NOT write files immediately.
Instead of it,
- assigns logical cluster metadata to new files (soft tags).
- allows Delta Lake to prune and scan efficiently based on metadata.
- performs incremental physical clustering behind the scenes when it’s optimal to do so.
Imagine a library where books are returned randomly and dumped on desks. Liquid Clustering is like the librarian tagging each book with a genre as it’s returned and later shelving them in order at a convenient time — meanwhile, users can still search efficiently by genre without needing all books to be perfectly arranged yet.
An issue that the Liquid Clustering targets is ‘Small Files’ generations. For distributed systems such as Apache Spark and Databricks, datasets are written into many tiny files instead of fewer large ones, and this causes performance and cost problems.
What is a Small File?
In Spark and Delta Lake, data is usually stored in Parquet files. Ideally, each file should be:
- 100 MB to 1 GB in size (depending on workload)
- Large enough to process efficiently across multiple nodes
But when each write creates many small files (like 1 MB or 2 MB), you get the small file problem.
Why are they bad?
Metadata Overload: Each file has metadata (stats, min/max values, etc.) in the Delta log. More files = more metadata = slower reads.Too Many Tasks: Each file becomes a task. If you have 1000 small files, Spark launches 1000 tasks; the Overhead increases; The Cluster gets overwhelmed with task management.Increases Costs: Cloud storage (like S3) and compute resources are stressed. It results in higher I/O costs, lower job times.
How Liquid Clustering Solves this?
- When writing data, files are tagged with clustering metadata (e.g., clustering ID or clustering key).
- Delta’s query engine uses this metadata to prune files logically, even if they’re not physically sorted.
- As the system gains idle time, it incrementally rewrites and merges files behind the scenes.
How does it work?
- Choose Clustering Columns:
You define which columns matter for clustering (e.g.,customer_id,event_date). - No Folders Created:
Unlike static partitions, no folder structure is created — everything happens internally. - Delta Engine Optimises It:
Background processes (like OPTIMIZE ZORDER BY) reorganise data intelligently using clustering metrics during idle time. - Adaptive Execution:
As data grows, Databricks learns access patterns and improves layout.
Use Case Examples
Event tracking logs
Clustering by user_id, timestamp Improves range queries on time windows.
IoT sensor data
Clustering by device_id and location_id for quick filtering on those keys.
Retail transactions
Cluster on store_id, purchase_date for fast reporting and dashboarding.
How to Enable Liquid Clustering
ALTER TABLE my_table SET TBLPROPERTIES (
'delta.autoOptimize.optimizeWrite' = 'true',
'delta.autoOptimize.autoCompact' = 'true'
);Optionally, you can define clustering columns:
ALTER TABLE my_table SET TBLPROPERTIES (
'delta.dataSkippingNumIndexedCols' = '3',
'delta.clusteringColumns' = 'customer_id,event_date'
);Liquid Clustering reshapes how we think about organising large datasets — no more rigid rewrites or expensive overheads. By dynamically clustering data in the background, it elegantly balances performance, scalability, and storage efficiency. Just like marbles settling naturally into their jars, Liquid Clustering lets your data find its optimal place — fluidly and intelligently.
