Case 08 · Schema evolution

Somebody added one field. Three jobs died.

Nobody did anything unreasonable. A team added a field they needed. The whole problem is that you and they could not possibly press deploy at the same second.

The shape

Every table has an agreed shape.

An order arrives looking like this. Same fields, same types, every time.

What the app sends

The shape today

order_id   A-1
city       Mumbai
amount     120

Three fields. Everyone who reads this has been built around those three.

What they want to send

The shape tomorrow

order_id   A-1
city       Mumbai
amount     120
coupon     DIWALI20

One more field. A perfectly sensible request from a team doing their job.

People call this agreed shape the schema. Changing it over time, without hurting anybody, is the whole of this case.

And it is worth saying who “everybody” is. One app writes those orders. A nightly job reads them. A dashboard reads them. A model reads them. None of those four teams sit together.

The real problem

You cannot all press deploy at the same second.

Here is the thing that makes this hard. It is not the change itself.

The team that writes the data updates their code on Tuesday morning. The nightly job gets updated on Thursday. The dashboard, whenever somebody gets to it.

So for two days, old code and new code are both running at once. Old code is reading new data, or new code is reading old data, or both.

The gap is where everything breaks.

If every team could deploy together, at the same instant, none of this would matter. They never can. Someone is on holiday. A job is mid-run. A deploy fails and gets rolled back.

So the question is never “is this change good?”. It is: who can safely go first?

Try it

Pick a change. Pick who upgrades first. Watch what falls over.

One app writes the orders, three jobs read them. Choose what to change and who moves first, then roll it out and watch the middle step. That middle step is Tuesday to Thursday.

the change

who upgrades first

—

writes the orders

orders going past, in the old shape

reads the orders

Press Roll it out and step through it.

What that showed

Two changes. Opposite answers.

Go back and compare just two of them.

Adding a field that must be there. Readers must not go first. A new reader demands a field that old orders simply do not have. So the app has to move first, and even then, every order written before today is still missing it.

Removing a field. Writers must not go first. The moment the app stops sending it, every job still asking for it falls over. So the readers have to stop needing it first.

That is the entire reason people have two different words for this. One kind of change needs writers to move first. The other needs readers to move first. Get the order backwards and you get a 3am phone call.

And then there are the two that have no safe order at all. Changing a type breaks everyone whichever way round you do it, because the old value and the new value are different things wearing the same name.

The list

Safe, and not safe.

✓	Add a field that may be missing. Old readers ignore what they do not recognise. New readers fill in a blank when it is not there. Safe in both directions, which is why almost everything should be done this way.
✓	Remove a field nobody reads. Fine, once you have actually checked that nobody reads it. Lineage from Case 05 is how you check.
✕	Add a field that must be there. Old orders do not have it and never will. Readers cannot move first, and old data stays broken.
✕	Remove a field someone still reads. Their job stops working the moment you stop sending it.
✓	Move the fields around. Putting amount before city changes nothing. Readers look fields up by name, not by position. People worry about this one and they do not need to.
✕	Take a field that was allowed to be missing and make it required. This feels like tidying up, which is why it catches people. Old orders where it was missing are now unreadable.
✕	Change a field’s type. A reader expecting a number gets text. There is no order of deploys that saves you.
✕	Rename a field. To everyone reading, this is two changes at once: a field vanished, and a field they have never heard of appeared.

Grow the shape. Do not shrink it.

That one line covers most of it. Adding is nearly always fine. Taking away and renaming are where people get hurt.

One extra line

Say what to use when it is not there.

The most common breaking change is adding a field with nothing to fall back on. What repairs it is almost too small to believe.

Breaks

No fallback

coupon  text
        (must be there)

A reader opens an order from last month. There is no coupon. It has nothing to use and stops.

Works

A fallback

coupon  text
        may be empty
        if missing → none

Same reader, same old order. No coupon, so it uses none and carries on. Nothing else changes.

That is it. One extra line in the definition, and a change that would have broken three jobs breaks nothing.

Some companies run a tool that checks every schema change before it is allowed through — a schema registry, if you meet the term. This is the change it rejects more than any other. It is also the one people argue with it about, and the tool is right.

Renaming

You cannot rename a field. You can replace one.

Renaming looks like one small change. To everybody reading, it is two: something they used disappeared, and something they have never seen turned up.

So you do not rename. You run both for a while.

step 1 · this weekSend both names.

The app writes user_name and full_name, with the same value in each. Nothing reading it notices anything at all.

step 2 · the next few weeksMove the readers over, one at a time.

Each job switches to full_name whenever its team gets to it. No deadline, no coordination, nobody blocked.

step 3 · when nobody is leftStop sending the old name.

Check that nothing still reads user_name. Then drop it. If something does still read it, you have not finished step 2.

It takes weeks instead of minutes, and nobody is ever woken up. That trade is almost always worth it.

Some formats let you record an old name alongside the new one, so a reader recognises both. That is the same idea, done for you, and it is worth checking whether yours does.

When there is no safe version

Build the new one next to the old one.

Sometimes the change really is breaking. A type has to change. The shape is genuinely wrong and no amount of adding fixes it.

Do not fight it. Make a second table, with the new shape, and let people move across in their own time.

Write to both for a while. The app fills the old table and the new one. More work for you, no work for anybody else.

Let readers move when they can. Each team switches when it suits them. Nobody is blocked by anybody.

Then check, and stop. When nothing reads the old table, stop writing it. Keep it around a little longer before you delete it.

It costs a few weeks of writing everything twice. It replaces a flag day where forty people have to deploy together and someone always does not.

The two words

Now the names will make sense.

You have met both of these already. Here they are with their proper names, which are confusing enough that most people get them the wrong way round.

Backward compatible

New code reads old data

new reader ← yesterday's orders

Means your readers can go first. They understand what is already out there, so they are safe to upgrade before the app changes.

Forward compatible

Old code reads new data

old reader ← tomorrow's orders

Means your writers can go first. The app can start sending the new shape and nothing reading it falls over.

If you only remember one translation, make it this one. Each word is really telling you who is allowed to deploy first.

And for anything long-lived you usually want both at once, so either side can move whenever they like. That is what adding a field with a fallback buys you, and it is why it is the default advice.

Three things people believe

And one that causes most of the damage.

“We will just coordinate it.” This is the one that hurts. Coordination means a plan where eleven people deploy in order on a Thursday. Someone is ill. A job is already running. A rollback puts old code back at 2am. Design it so nobody has to be told, and then it does not matter who read the message.

“The two words mean the same thing.” They are mirror images. One lets readers move first, the other lets writers move first. Mixing them up is how people choose exactly the wrong rollout order.

“A version number is documentation.” “We are on v3” tells nobody anything. What changed, when, and what should somebody do about it — that is the part people need, and it is the part that never gets written.

If you get asked

“A new field needs adding. Walk me through it.”

Ask one question first. Does this field have to be there, or can it be missing? That single question decides everything else, and asking it makes you sound like you have done this.

Then give the safe version. Add it so it is allowed to be missing, with something to use when it is. Old jobs ignore it. New jobs cope with old data. Nobody has to deploy in any particular order.

Then show you know about the gap. Say that old code and new code run side by side for days, so the change has to be safe in both directions during that window.

Then handle the hard version. If it genuinely must be required, or a type has to change, build the new table beside the old one and let people move across. Do not ask forty people to deploy together.

If you have room, add that you would check who actually reads the field before removing anything — which is lineage, from Case 05, doing real work.

Two questions

Did it land?

You are adding a field that every order must have. Who should upgrade first?

Someone on another team wants a column renamed. What do you suggest?

Your playbook · Case 08

Eight things to keep

The gap is the problemOld code and new code run at the same time for days. The change has to survive that window, not just the end of it.
Grow the shape, do not shrink itAdding is nearly always safe. Taking away and renaming are where jobs die.
Let it be missing, and say what to use insteadOne extra line turns a breaking change into a safe one. It is the change tools reject most often.
Required fields mean writers go firstReaders cannot demand something that old data does not have.
Removing means readers go firstThey have to stop needing it before you stop sending it.
You cannot rename, only replaceSend both names, move readers one at a time, then drop the old name once nothing reads it.
If it must break, build beside itA second table with the new shape beats a day where everybody has to deploy together.
Never rely on everyone rememberingSomeone is on leave, a job is mid-run, a rollback puts old code back. Design so it does not matter.

Your review

How was this case?

Loading ratings…

Next · Case 09

Where all of this actually sits.

Case 09 is about data lakes and file formats — what the files on disk really look like, and why the choice decides what your queries cost.

Open Case 09 →← Case 07