FractionalDataArchitect
Book a discovery call

Pipeline Full-Time Owner Trap (Year 2 Update)

Pipeline Full-Time Owner Trap (Year 2 Update)
Pipeline Full-Time Owner Trap (Year 2 Update)

One pipeline quietly consumed a full data engineer. On a team of 5, that’s 20% of capacity feeding one job.

Nobody had decided this. The pipeline was old, central, and brittle: manual reruns after upstream hiccups, a nightly check “just in case”, silent failures that only showed in the numbers. Maria had inherited it, and keeping it alive had slowly become her actual job. The roadmap items with her name on them rolled over, sprint after sprint.

The trap has a tell: ask each engineer what percentage of their week goes to keeping existing things alive versus building. Answers above half, concentrated on one person and one system, mean a pipeline has hired itself an operator.

The refactor took about 6 weeks, part-time, and was standard reliability work: idempotent steps so reruns are safe and boring, retries with backoff for the upstream flakiness, alerts on the data itself rather than on task success, and a one-command backfill.

The pipeline still fails sometimes. It recovers without Maria, which was the goal. She shipped 2 roadmap projects the next quarter.

I look for this pattern in every platform review now. Most teams above 4 engineers have at least one of these, and nobody planned it.

Which pipeline on your team has a de facto full-time operator, and do they know it?

Written by Thomas Nys

Fractional Data Architect helping startups and scaleups build data platforms that scale.

More about Thomas Nys →

Recognise the problem? Let's talk about it.