Slow pipelines are a symptom. The disease is poor architectural decisions nobody revisited.
This one keeps coming back. A team complains about slow pipelines, flaky jobs, queries that take forever. Leadership’s first instinct: throw more compute at it. Or hire another engineer.
But the pipelines aren’t the problem. They’re the symptom.
The actual problem is a schema designed for three data sources that now handles thirty. An orchestration layer built when the team was two people. A “temporary” staging table that became load-bearing infrastructure because nobody had time to replace it.
I’ve been guilty of this myself - patching symptoms instead of tracing back to the decision that caused them. It feels productive. It’s not.
The fix isn’t more resources. It’s an honest audit. Pick your three worst pipelines. For each one, ask:
What decision created this mess? When was it made? What’s changed since then?
Most of the time, the answer is obvious. The hard part isn’t finding it - it’s admitting the original decision no longer fits.
Which pipeline does your team route around instead of fixing?