← All casesCase 1013 min

Case 10 · Partitioning and clustering

You asked for one day. It read five years.

The query was right. The table was right. Nobody did anything wrong. It still read several terabytes to hand you a few thousand rows, and somebody paid for all of it.

The default

A filter does not stop the reading. It happens after.

Here is the bit that surprises people. You write this:

What you wrote

Looks precise

select count(*)
from events
where event_date = '2026-06-15'

One day. A few thousand rows. It reads like a small question.

What happened

Was not

read all 5 years
check each row
keep the ones that match
throw away 99.9%

The filter is applied to rows that have already been read. It cannot stop something being read that has not been looked at yet.

So the engine pulled every row off storage, checked each one, and kept almost none of them. The answer is correct. The work was not.

The only cheap read is the one that never happens.

Both techniques in this case are the same idea wearing different clothes: arrange the data so the engine can tell, before opening something, that it cannot possibly contain what you asked for.

And this matters twice over. Where you are billed for every byte scanned, reading less is cheaper straight away. Where you rent a cluster by the hour, reading less is faster — which is the same saving, arriving by a different route.

The first move

Split the table into separate piles.

Partitioning means physically splitting one table into chunks, based on the value of one column. Almost always a date.

One chunk holds everything from 13 June. Another holds 14 June. And so on, back five years.

Now your filter has something to work with. The engine looks at where event_date = '2026-06-15', sees that a single chunk is the only one able to hold matching rows, opens it, and leaves the rest alone.

The name for skipping whole chunks like this is partition pruning. It is the single biggest win available in this case, and it usually costs you one line when you create the table.

A month of post, sorted into dated envelopes. You want Tuesday’s letters, so you pick up one envelope.

You met a version of this in Case 01, where blocks got skipped because of the notes kept beside them. This is the same instinct, one level up: skip entire folders because of what they are named.

Try it

Five years of events. Choose how to split it, then ask a question.

Five terabytes, five years, two million customers. Pick the column to split the table by, pick a query, and watch how much gets read — and how much of that was needed.

split the table by
the question
—
what actually gets read
—read off storage
—you actually needed
—times more than necessary

Illustrative figures. Real numbers depend on your data, row sizes, file layout, engine and how well the data compresses.

What that showed

The wrong column is worse than no column.

Go back and split by customer_id, then run three of the four queries. Watch what happens.

One query gets dramatically faster. Looking up a single customer now opens one tiny chunk. That part works.
The other three get no help at all. Ask for one day and there is nothing to prune, because the chunks are not organised by date. You are back to reading five years.
And you have created two million chunks. Each one a couple of megabytes. The engine now has to keep track of all of them, and listing them starts to cost more than reading them.

That is the trap, and it catches people who have just learned the word. Partitioning is not a thing you do to a table to make it faster. It is a bet on which question gets asked most, and a wrong bet costs you more than not betting.

How many is too many

The whole rule comes down to one number.

How many different values does the column have? Drag it and watch what happens to the size of each chunk.

—
—chunks created
—average chunk size
—a column like this
This is the failure people actually hit. Thousands of tiny chunks means the engine spends its time opening and closing files instead of reading them. It can genuinely end up slower than not splitting the table at all.

The word for “how many different values a column has” is cardinality. Low means few — a handful of countries, a dozen statuses, a few hundred dates. High means many — two million customers.

And that one number tells you which tool to reach for. Low cardinality, split the table by it. High cardinality, do not — use the other technique instead, which is next.

The second move

Sort the rows inside each pile.

You cannot split the table by customer. But customer is exactly what half your queries filter on. So what do you do?

You keep the rows for each customer sitting next to each other inside the chunk, instead of scattered through it at random.

Scattered

One day, unsorted

c_8821  c_104   c_55012
c_377   c_8821  c_9
c_55012 c_41    c_8821

Customer 8821 appears three times, in three different places. To find them all you have to look everywhere.

Grouped

One day, sorted by customer

c_9     c_41    c_55
c_104   c_377   c_8821
c_8821  c_8821  c_55012

Now they are together. Every block of rows can carry a note saying which customers are in it, so most blocks get skipped.

People call this clustering, and depending on the system you will also hear sorting, bucketing, or a Z-order. The names differ. The instinct — put things that get asked for together, together — does not.

Note the difference in what gets skipped. Partitioning skips whole chunks without opening them. Clustering opens the chunk, then skips most of what is inside. The first is a bigger saving. The second is the one that still works when you cannot split by that column.

Together

One picks the pile. The other finds the row.

They are not alternatives. They work at different scales, and the useful thing is to use both.

The query asks for customer 8821 on 15 June.
↓
Partitioning rules out 1,824 days without opening any of them.
↓
Clustering opens that one day and skips most of the blocks inside it.
↓
A few megabytes read, out of five terabytes.

Go back to the widget and try it: event_date for the split, customer_id for the sort, then ask about one customer on one day. You will see why that pairing turns up everywhere. It happens to answer the two questions people ask most.

It is also why clustering alone is rarely the answer. Without partitioning, a clustered table still has to open every chunk before it can skip anything inside them.

Deciding

Two questions, in this order.

firstWhat does nearly every query filter on?

Look at what people actually run, not what you imagine they run. If almost everything has a date filter, split by date. That is the common answer because it is usually true.

thenHow many different values does that column have?

A few hundred to a few thousand chunks is healthy. Tens of thousands is worth a think. Millions means you have picked the wrong column.

and thenWhat is the next most common filter?

If it is something with a lot of distinct values — customer, product, device — cluster by it. That is what clustering is for.

One thing to check before you change anything. Partitioning only helps if queries actually filter on that column, and filter on it plainly. Wrapping the column in a function, or comparing it to something the engine cannot work out in advance, can quietly stop the pruning from happening. The query still returns the right answer, which is why nobody notices. If you have partitioned a table and it got no faster, this is the first thing to look at.

Three things people say

And the one that costs money.

“It is slow, add an index.” That instinct comes from app databases, where an index points you at individual rows. Warehouses mostly do not work that way. What you have instead is the layout: which chunk a row is in, and whereabouts it sits inside that chunk.
“More partitions, more speed.” Up to a point, then sharply the other way. Each chunk has to be tracked and opened. Past a few tens of thousands, the bookkeeping starts to cost more than the reading it saves.
“We partitioned it, so it is fast now.” Only for queries that filter on that column, plainly. Everything else reads the whole table exactly as before, and the table being partitioned gives everyone false confidence that it has been dealt with.

If you get asked

“This query is slow. What would you do?”

Resist saying “add an index”. In a warehouse it is usually the wrong answer, and it is the first thing most candidates say.

Start by asking what it filters on. You cannot answer this without knowing. Saying so is a better opening than any technique.

Then name the mechanism, not just the word. That you would split the table by whichever column nearly every query filters on — a date, nine times out of ten — so whole chunks get ruled out before anything is opened. Saying partition pruning and then explaining it in plain words lands better than saying it alone.

Then the second layer. Cluster by the high-cardinality column that queries filter or join on, so blocks inside the chunk get skipped too.

Then show you know the failure. That splitting by something like customer id makes millions of tiny chunks, helps one query shape and nothing else, and can be slower overall. Volunteering the trap is what separates this from a memorised answer.

Three questions

Did it land?

Why does a one-day query read five years by default?

You split a table by customer_id, for two million customers. What is the likely outcome?

A table is partitioned by date, but the daily dashboard is still slow. First thing to check?

Your playbook · Case 10

Eight things to keep

  1. A filter does not prevent readingIt is applied to rows already pulled off storage. The saving has to come from the layout, not the query.
  2. Partitioning splits the table into pilesUsually one per day. Whole piles get ruled out before anything is opened. The name for that is partition pruning.
  3. Clustering sorts rows inside a pileRows for the same customer sit together, so most blocks inside the chunk get skipped.
  4. One picks the pile, the other finds the rowUse both. Dates for the split, customer for the sort, and a one-customer one-day question barely reads anything.
  5. Split by a column with few distinct valuesA few hundred to a few thousand chunks is healthy. Millions means you picked the wrong column.
  6. The wrong column is worse than noneIt helps one query shape, leaves the rest unchanged, and leaves you managing far too many small chunks.
  7. Pruning only happens if the filter is plainWrap the column in a function and the skipping can quietly stop, while the answer stays correct.
  8. Fewer bytes read is the whole gameCheaper when you are billed by the byte, faster when you are billed by the hour. Same win, two different invoices.

Your review

How was this case?

Loading ratings…

Next · Case 11 · drops soon

The whole map, finally.

Case 11 steps back from the pieces and lays out the landscape — where every tool you have met fits, and which ones you can safely ignore for now. Leave your email and we’ll tell you the day it goes live.

← Case 09All cases