Cross-Functional Data Ownership

A backend deploy broke the data pipeline for the third time this quarter. Both teams had done their jobs correctly.

The backend team shipped a field rename that passed their tests. The data job failed at 3am on a column that no longer existed. Both sides had named owners. The thing between them didn’t.

We’ve gotten reasonably good at putting a name next to a table. The contract between two teams still lives in a conversation, so nobody’s name is on it and nobody gets told when it changes.

Three things that stopped it at one client:

1. The interface is recorded like any other asset. The event schema has a version, a producer, and a consumer list you can query.
2. One owner with a decision right, not two stakeholders with opinions. Usually the producing team, because they hold the change.
3. The consumer list is visible to the person making the change. An engineer who can see the 4 jobs downstream behaves differently from one who can’t.

I’ve changed my mind on where the owner sits. I used to argue for the consuming team, since they feel the breakage. In practice the producer holds the change, so that’s where the decision has to live.

None of this is glamorous, and it moves delivery further than another hire usually does.

Who owns the schema between your backend and data teams, by name?