Why notification systems are a nightmare for engineers
A shop sends you five messages after you place an order. For each one I found a company that got it wrong in production, and below are those incidents and the techniques that prevent them.
sendEmail() and called it done.I'll follow one order. Ana lives in Lisbon. On Tuesday afternoon she orders a ceramic lamp from Halden, a small homewares shop with a warehouse in Ohio. Over the next week, Halden sends her five messages:
- ReceiptSent twice · Salesloft, Zoho
- ShippedArrives in the middle of the night · Asana
- Out for deliveryThe rate limiter breaks · Buttondown
- DeliveredFor an order nobody placed · Starbucks
- Review requestA test reaches real people · HBO Max, Everbridge
The build list (on the left, or the counter at the bottom on a phone) fills up as you read with everything you'd have to write to get these messages right.
The receipt that arrives twice
Checkout asks for the receipt. The request times out, so the code retries. But a timeout only means you didn't hear back. The email may already have gone out.
An idempotency key built from the event (order-8812-confirmed,
not a random UUID) turns the retry into a no-op. A send lock, such as a Redis
SET NX with a lease longer than the send timeout, stops two workers sending the same
message. Saved step results let a restarted worker skip what it already did.
None of that would have saved Salesloft. If the provider reports a failure that didn't happen, every sender retries. Only the provider can close that gap, by accepting your key.
"Your order has shipped," at 2:40 in the morning
The night shift in Ohio scans the box in the evening, which is the middle of the night in Lisbon, and Ana's phone is on her bedside table.
Store an IANA zone (Europe/Lisbon), not a UTC offset, because clocks change twice a year. Park the message in a durable scheduler so it survives a restart. At 08:00, check her preferences again in case she unsubscribed overnight. Fraud alerts need a critical bypass.
Out for delivery, and so is everyone else's
Every European order leaves the depot at once, so thousands of pushes want to go out in the same minute. Providers cap how fast you can send, so you add a rate limiter, and the limiter is new code that can have its own bugs, as Buttondown found.
A common bug is telling every throttled message to come back in one second, so they all return at once. Jitter fixes it: each waits a random time within the window. The limiter must also be shared across processes, usually in Redis, or each worker stays under the limit while together they blow through it.
The jam forms again every second.
A queue that moves at the speed the provider allows.
When a provider is actually down, the message moves along a chain until something works:
A circuit breaker skips a failing provider for a while instead of waiting on a timeout for every message. Priority lanes keep a newsletter from holding up a password reset, and a per-user cap limits the damage when your own code loops.
"Delivered," for a parcel that wasn't
The courier scans the parcel and calls Halden's webhook, and Halden sends whatever the webhook says. If the event is wrong, the message is wrong too. A notification system can't catch that, but it can tie each message to the right event for the right order by running the whole sequence as one workflow:
✕ not this order, ignored
✓ matched, continue
Events wake only the matching order, "delivered" can't go out before "shipped", and if Ana cancels, nothing more is sent. That takes a workflow engine with durable waits and cancellation, a project in its own right.
The email that should never have gone out
Marketing sends review requests in bulk, to all of last month's customers at once. That's also how the most famous notification incident happened.
Here are the guards between a bulk send and an inbox:
Topic preferences let Ana mute review requests and keep receipts. Gmail and Yahoo require one-click unsubscribe on bulk mail. A suppression list fed by bounce and complaint webhooks blocks every send, critical ones included. Delivery logs and campaign tracking show who got what, and a kill switch stops a bad send halfway.
All of those check whether Ana wants the message, and none of them checks whether you meant to send it. HBO Max and Everbridge both had a test reach a live channel. What helps there is keeping test sends on staging credentials that can't reach a real inbox or phone.
Why it's a nightmare
Five order messages need all of this, plus a dead-letter queue and alerts so you can answer "why didn't Ana get her email?"
- idempotency keys
- per-send lock
- saved step results
- time-zone quiet hours
- durable scheduler
- re-check before send
- critical bypass
- shared rate limiter
- jitter
- circuit breakers
- provider failover
- channel fallback
- priority lanes
- per-user cap
- workflow engine
- cancellation
- topic preferences
- one-click unsubscribe
- suppression list
- delivery logs
- kill switch
- campaign tracking
22 pieces behind five order messages.
Each piece is manageable alone, but they interact. The quiet-hours check has to respect the suppression list, the retry has to respect the idempotency key, and the workflow has to survive a deploy halfway through. Most of the companies in this article shipped one of these bugs themselves.
Oftentimes things that seem like they should be simple end up being way more complicated when you get into the details.
Sophie Alpert, on Slack's notification logic · "Everyone is wrong about that Slack flowchart"
What I'm building
notifkit, the open-source library I've been building, is built around this list. It runs next to your own Postgres and Redis. It's a young project and I'm working on it actively, so if something here is missing or breaks for you, open an issue at github.com/devkitshq/notifkit and I'll look at it.