The dashboard says zero. You have fifteen tables and no idea which one.

03:40

The pager goes off. Now what?

A revenue figure that leadership opens every morning is sitting at zero. Orders definitely came in last night — you can see them in the app. So it is the figure that is broken, not the business.

You now have three questions and no obvious way to answer any of them. Where did it go wrong? How far back does the damage start? And who else is about to open something built on the same broken thing?

Somebody on a comment thread described exactly this: a revenue dashboard showing nothing, a top-priority incident raised, half the engineering team woken up — and the cause turned out to be an upstream source arriving two hours late. Nothing was broken. Something was late, and nobody could see that from where they were standing.

Two things would have shortened that night. A map of what feeds what, and something watching the data that would have said “payments are late” at 02:15 instead of leaving you to guess at 03:40.

The map

Lineage is just: what feeds what.

Every table in your warehouse was built from something, and feeds something else. Write all of those arrows down and you have lineage. That is the entire concept — the value is not in the idea, it is in actually having it when you need it.

And you walk it in two directions, for two completely different jobs.

dashboard
   ↑
revenue_daily
   ↑
payments_clean
   ↑
raw_payments   ← here

raw_payments
   ↓
payments_clean
   ↓
revenue_daily
   ↓  ↓  ↓
dash  export  model

The second direction is the one people forget, and it is the one that saves you from the worst version of the morning: fixing the dashboard, going back to bed, and finding out at eleven that a finance export built on the same table went out to someone at nine.

What that was worth

Three looks, or fifteen.

Following the arrows, the cause is three inspections from the symptom. Without the map you are opening tables in whatever order occurs to you, and on average you will be about halfway through before you land on it — call it eight looks, twenty four minutes, at four in the morning, while a number is wrong in public.

That gap is the entire business case for lineage, and it gets wider every time somebody adds a table.

Switch the widget to the second mode and look at what the broken table feeds. The dashboard that woke you up is one of several things on that path — and it is not the expensive one.

The part people skip

Fixing it is half the job. Telling people is the other half.

The finance export is the one that should worry you. It left the building at nine o’clock carrying a zero, to someone outside your team who has no idea the pipeline had a bad night. Nobody is going to come and tell you about it — they will simply believe it.

So when something breaks, lineage gives you the list of people to warn before they act on it, and the list is usually longer than you would guess from the thing that alerted you.

There is a second use for the same arrows, on a much calmer day. Before you rename a column or change a transformation, walk forwards first. “This touches twenty three things, including the board pack” is the sentence that stops an ordinary Tuesday from becoming another 03:40.

The alarm side

Four things worth watching, and no more than that.

Lineage helps once you know something is wrong. Observability is what tells you. Case 03 covered the checks that stop a bad batch from being served; this is the layer that watches tables continuously, including ones nobody is loading right now.

You can measure almost anything, which is exactly the problem. These four catch most of what actually goes wrong:

Notice all four are about the table, not the job. That is the whole distinction, and it is why a green run never settles the question.

Two checks, two moments

Look before you start, and look after you finish.

There is a useful distinction here that almost nobody is taught, and it comes up the first time you have a pipeline that is expensive to run.

is there a file for today?
does it have roughly
the rows we expect?

no duplicate keys
no nulls where there
should not be
totals within range

The first one saves you money and noise. The second one protects the people reading the numbers. They are not alternatives — most mature pipelines have both, and the cheap one at the front is the one teams usually add second, after a few nights of carefully transforming an empty file.

Starting from nothing

What a three-person team should build first.

Everything above describes what a large company runs. Here is the honest minimum, in the order it pays off, and none of it needs a paid tool.

That is a few hours of work on top of whatever you already run, and it covers most of what the expensive platforms are sold for. The time to buy something is when you genuinely cannot keep the map in your head any more — not before.

The names

What you will hear these called.

As with every tool list on this site: the map matters, not the thing that drew it. A graph on a whiteboard that is correct beats a generated one nobody trusts.

Three things that sound sensible

And quietly are not.

If you get asked

“How do you find out when something has gone wrong?”

Almost everyone answers with the job status. Separate the two layers and you are instantly in a different conversation.

First, split watching the job from watching the table. A run can finish perfectly and load nothing, so the exit code is not the question. Say you watch the table itself: how recent it is, how many rows arrived, whether the columns are still what you expected, and whether the values are sane.

Then bring in the map, and say both directions. When something fires you step backwards through what feeds what until you reach the first place the data is already wrong — and then forwards, to find every dashboard, export and model standing on it, so you know who to warn before they act on it.

Then say the thing almost nobody says. That you would keep the number of alerts small enough that people still read them, because a hundred alerts and no response is worse than ten that get acted on.

Naming a tool at the end is fine. Naming one at the start is what the weak answer sounds like.

Two questions

Did it land?