Streaming has a cost nobody puts in the business case: the weekly tax of running it after the demo works.
I once built streaming infrastructure for data the business looked at once a day. Months of engineering for freshness nobody used. That’s the scar I check every real-time request against now.
Real-time has two costs. The build, which everyone estimates. And the operating tax, which almost nobody does.
The operating tax, week after week:
- On-call. Streaming fails at 3am, not at 3pm. Someone owns that pager now.
- Retries and idempotency. Every consumer has to handle duplicates and out-of-order events, or you get silent double-counting.
- Backfills get hard. Reprocessing a day of history in a streaming system is a project. In batch it’s a rerun.
- Debugging. State spread across a running system is harder to inspect than a table you can query.
For maybe 70% of SME analytics I see, a daily or hourly refresh answers the same questions at a fraction of the ongoing cost. The dashboard gets opened in the morning. The model retrains nightly. Nobody’s making sub-minute decisions on it.
Real-time earns its tax when a decision genuinely depends on fresh data: fraud checks, live inventory, anything where a stale number costs money in minutes. Those exist. They’re rarer than people assume.
The useful question before you build: how stale can this data be before someone makes a worse decision? Get the honest answer, in minutes or hours, and size the system to that. Usually it’s hours.
If the answer really is seconds, build the stream. And budget the on-call with it.
What’s the freshness you actually need on your most-watched dashboard, in minutes? And what did you build it to?