The rider moves every second. The accountant counts once a night.

Two jobs, two clocks

Both of these are data pipelines.

There is a little bike crawling across a map on somebody’s phone right now. It moves because a rider’s position is being sent onward the moment it is known, and nothing in that chain is allowed to pause and think.

In the same company, a finance team adds up yesterday. Every order, every refund, every commission, every tax line. That job starts at two in the morning, runs for forty minutes, and produces one set of numbers that nobody will question for the rest of the day.

Neither team is doing it wrong. They want different things. The rider’s dot is worthless if it is an hour old. Yesterday’s revenue is worthless if it changes every time someone refreshes.

Most people arriving at this topic expect to learn which one is more modern. That is not the useful part, and it is not what gets asked in interviews either. The useful part is what goes wrong once you pick the fast one.

Why anyone still does this

Batch is not the old way. It is the calm way.

It is tempting to read “runs once a night” as outdated. It is worth seeing what you get in exchange.

This is why the most trusted tables in most companies are still built by batch jobs — the ones finance signs off, the ones sent to the government. It is not that nobody thought of anything better. It is that “this number will be the same tomorrow” is very hard to get any other way.

The other approach

Never stop. React to each thing as it turns up.

Streaming does the opposite. There is no scheduled run, because the job never stops. Events are produced continuously — a tap in an app, a row changing in a database, a sensor reading, a card being swiped — and something is always listening.

In the middle sits a queue. It holds the events in the order they happened, and it remembers how far each reader has got through them. Your job reads the next event, does its work, and sends the result somewhere a person or another system can see it. Then it reads the next one. It never finishes.

Written down like that it sounds like batch with a smaller gap, and that is the mistake almost everyone makes on the way in. It is not a faster batch job. Closing that gap takes away something you did not know you were leaning on.

What you gave up

A batch job knows when the data is complete. A streaming job never does.

This is the whole difficulty, in one sentence, and it is worth reading twice.

When the batch job starts at 02:00, the day it is summarising is over. Everything that was going to happen has happened, and everything that was going to arrive has arrived. The job can count with confidence because there is nothing left to come.

A streaming job has no such moment. At 11:00 it has been handed some of the ten o’clock hour. It has no way of knowing whether that is all of it. Somewhere there is a phone in a lift with an unsent order on it.

The uncomfortable bit

There is no cut-off that is both fast and complete.

Go back and try all three cut-offs. You will find there is no setting that wins. Stop at 11:00 and you are quick and wrong. Wait until 11:30 and you are right and late. Every streaming system in the world is sitting somewhere on that line, and somebody chose the spot.

That choice has a name — people call it a watermark, which sounds far more mysterious than it is. It is just the system saying: I will assume nothing older than this is still on its way. It is a guess. A careful guess, and one you are allowed to change.

Notice what batch got for free here. When the finance job runs at two in the morning, the phone in the lift has long since reconnected. Batch does not fix the late-data problem. It waits so long that the problem fixes itself.

The rest of the bill

Three more things that only exist once the job never stops.

Late data is the interesting problem. These next three are the ones that eat a normal working day.

None of this is an argument against streaming. It is an argument against picking it without thinking. Every item on that list becomes somebody’s job at two in the morning, on the night it breaks.

Decide it in one question

What breaks if this answer is an hour old?

Not “how fast can we make it”. Anything can be fast. Ask what actually goes wrong while you wait — that is the thing you are paying for, and streaming is expensive.

Four of those eight want streaming. That ratio is roughly honest for a real company, and it is much lower than the internet implies.

Two questions after that

And then the ones nobody asks until it is too late.

Who is awake when it breaks? A batch job that fails at 02:00 can be rerun at 09:00 by whoever gets in first, and nothing is lost. A streaming job that fails at 02:00 falls further behind every minute, and the pile of unread events keeps growing while somebody looks for the instructions. If the answer to “who is on call” is “nobody, really”, you have your answer about which system to build.

Will you need to redo history? If the rules are new and likely to change, batch will save you months. If everyone already agrees on exactly what you are counting, that argument gets weaker.

These two questions matter more than any speed target, because they are about the team, not the technology. You can swap the tools. You cannot swap the team.

What actually gets built

Almost nobody picks one.

The interview answer people expect is a choice. The real answer, in most companies of any size, is both — with a clear division of labour.

events → queue → stream job
       → alert / live tile

same events → storage
       → nightly job → warehouse

Both read the same events. The difference is when they read them, and what they promise. The live tile might say ₹4.1 lakh at nine in the evening; the next morning’s table says ₹4.06 lakh and that is the number that goes in the deck. Nobody is upset, because everyone was told which one to trust for what.

If you can explain that split in an interview, you are answering a level above the question that was asked.

Same query, two meanings

The code barely changes. The promise does.

Here is an hourly order count. It is the same idea in both worlds.

SELECT hour_of(placed_at) AS hr,
       COUNT(*) AS orders
FROM   orders
WHERE  placed_at >= yesterday
GROUP  BY 1;

SELECT hour_of(placed_at) AS hr,
       COUNT(*) AS orders
FROM   order_stream
GROUP  BY 1
-- and then: when is an hour done?
-- what about the ones that come late?
-- what if we already published it?

Remember this when a tool tells you it lets you “write streaming pipelines in plain SQL”. That is true, and it does help. It does not make the hour finish any sooner.

Three things people say

That sound right and are not.

The third one matters more than it looks, because it means “batch versus streaming” is not a hard border. Run a batch job often enough and it starts to behave like a stream. Make a stream wait long enough and you have built batch again. The real difference is what each one promises about having all the data — not how big the chunks are.

If you get asked

“Design a pipeline for live order tracking.”

The trap is to start naming tools. Tools are the least interesting part of your answer, and the interviewer already knows them all.

Start with the delay. Say what goes wrong during it. “A rider’s position is useless once it is a minute old, so this one has to be a stream.” You have now explained why, instead of just saying so.

Then show you know what it costs. Say that you would bucket by when the event happened rather than when it landed, and that some of it will turn up out of order, or long after the hour it belongs to. That one sentence separates people who have run a streaming pipeline from people who have only read about one.

Then split it. Live path for the map, nightly path for the numbers finance uses. Say plainly that the two will not always match and that this is intended.

Then say what you would not stream. Offering the batch half yourself is the strongest thing you can do, because everyone else waiting outside is arguing for streaming everywhere.

Two questions

Did it land?