Pipeline Novelty Implementation

Novel data projects tend to die between the team that built the prototype and the team stuck running it.

A product squad or a data scientist builds something new: the first streaming feed, a recommendation feature, an LLM step inside a pipeline. The demo works. Then it’s handed to the platform team, who’ve never seen it, and it sits in a “productionising” ticket for a quarter.

I set up the bridge before the prototype starts:

1. One engineer from the team that’ll run it joins the prototype from week one. A day a week is enough.
2. The definition of done includes the boring list up front: alerting, backfill, an owner, a rollback.
3. Someone asks the freshness question early. How stale can this get before anyone notices?

I learned to ask about freshness the hard way, after building streaming infrastructure for data people looked at once a day. It worked beautifully, and a nightly batch would’ve done the job.

It does delay the demo. In the projects where I’ve insisted on it, the handover afterwards was a lot less painful.

What’s stuck in “productionising” at your company right now? Tell me and I’ll send back how I’d sequence it.