Real-Time Failure Modes

Duplicate invoices came from the recovery, not the outage. The retry replayed 40 minutes into a double-counting sink.

That incident shape repeats across most streaming platforms I review. The outage itself was handled fine. The recovery did the damage, because the pipeline was built for the forward path and nobody designed the replay path.

Streaming failure modes cluster in 3 places:

1. Redelivery. Brokers promise at-least-once in practice; “exactly-once” holds only inside narrow boundaries and dissolves at the edges of your system, where the side effects live. Every consumer will eventually see a duplicate.
2. Replay. Recovery means reprocessing a window of events. Sinks that append instead of upsert turn every recovery into data corruption.
3. Ordering. Events arrive late and out of order, and downstream logic that assumes arrival order works until the first rebalance.

The design property that defuses all three is idempotency: stable event keys end to end, upserts or merges at the sinks, dedup windows where the sink can’t upsert, and a dead-letter queue with a replay tool that’s been tested on purpose, before the incident.

The test worth running quarterly: replay yesterday into staging and diff the results. Zero diff means the recovery path works. Anything else means your next outage has a second act.

Have you ever replayed a day into staging just to see what changes?