Notifications Are a Distributed Systems Problem in Disguise

Meet Priya. She lives in Melbourne, and yesterday she ordered running shoes from Kestrel, a small online shop whose warehouse is in California. Kestrel sends its emails through notifkit, our open-source notification engine. Here's the trip her "Your order has shipped" email takes:

1. 9:04 am California · 3:04 am MelbourneThe warehouse scans Priya's box. Its Wi-Fi hiccups, so the shop asks for the email twice.
2. a few milliseconds laternotifkit says "got it", notices the second request is a repeat, and looks up who Priya is.
3. same secondIt's 3am where Priya is. The email is put to bed until 7:00.
4. 1:00 pm California · 7:00 am MelbourneThe email wakes up, checks Priya still wants it, and heads out.
5. 7:00 amThe shop's main email provider is down. The email takes the backup route.
6. 7:15 amPriya reads it with her coffee. One email, nothing strange.

Let's follow it stop by stop. Each one hides a classic computer-science problem. If you're an engineer, open the For engineers boxes to see exactly how notifkit handles it.

The request that arrived twice

A scanner beeps, and Kestrel's system asks notifkit to send Priya her shipping email. The warehouse Wi-Fi drops for a second, the answer never comes back, and Kestrel's code does the sensible thing: it tries again.

Now there are two requests. As far as notifkit can tell, they could be two different emails. Kestrel doesn't want to send Priya two "your order has shipped" emails, so how does notifkit know?

Kestrel tells it, with a cloakroom ticket. Each request carries a label that Kestrel makes up, something like order-5531-shipped. The first time notifkit sees that label, it does the work and remembers the label. If it sees the same label again within 24 hours, it says "already got this one" and does nothing. Engineers call this idempotency: doing something twice has the same effect as doing it once.

The other thing notifkit does here is answer quickly. It doesn't send the email while Kestrel waits. It writes the job down, says "got it", and does the work in the background. It's how a restaurant works: the waiter takes your order to the kitchen and comes back to your table, and doesn't stand at the stove until your food is ready.

curl -X POST https://notifkit.example.com/v1/notify \
  -H "X-Idempotency-Key: order-5531-shipped" \
  -d '{"user":"priya","template":"order-shipped","data":{"order":5531}}'

Conveyor belts, and why nothing falls off

Inside notifkit, the job travels along a series of conveyor belts. At each belt sits a worker with one job: one looks up who Priya is, one decides whether and when she should get the email, one actually sends it.

The important rule is about when a job leaves the belt. When a worker picks up a job, it stays on a "being worked on" list. It's only crossed off once the worker says "done". If the worker crashes halfway (a server restarts, a deploy goes out, the power blinks), the job is still on the list. After a while, another worker notices it has been sitting there too long and picks it up.

So jobs don't get lost. The catch is that sometimes a job gets done twice: the first worker might have finished the work and crashed just before saying "done". This guarantee has a name, at-least-once delivery, and the rest of this story is mostly about making those repeats harmless.

One more safety net: some jobs crash every worker that touches them, perhaps because the data is broken. Those are retried a few times, waiting longer each time, and then moved to a "problem pile" for a human to look at, instead of crashing workers forever.

Priya is asleep

The first worker looks Priya up: her email address, her phone number, her preferences. Two things stand out. Her time zone is Australia/Melbourne, and she has set quiet hours from 10pm to 7am.

It's 9:04 in the morning in California, but 3:04 in the morning in Melbourne. A shipping update isn't urgent, so the email is put to bed until 7:00 am her time. It isn't thrown away, just delayed.

That sounds simple. Time zones make sure it isn't:

• Clocks change. Melbourne moved its clocks forward on the first Sunday of October. If notifkit had stored Priya's quiet hours as "noon to 9pm UTC" back in winter, they'd now be an hour off. So it stores what Priya actually meant ("10pm to 7am, Melbourne time") and works out the real moment fresh every time.
• The window crosses midnight. 10pm to 7am starts on one day and ends on the next. The naive check "after 10pm and before 7am" is never true. It has to be "after 10pm or before 7am".
• Check the time the email will go out, not the time now. If Kestrel had scheduled the email for Friday at 3am, the check has to look at Friday at 3am, not at whatever time it is when the request arrives.
• Some things can't wait. A password reset or a fraud alert should arrive at 3am, because that's when Priya is trying to log in. Messages marked as critical skip quiet hours.

let utcMs = Date.UTC(y, mo - 1, d, h, mi);          // guess: the zone is UTC
for (let i = 0; i < 3; i++) {
  const p = partsIn(timezone, new Date(utcMs));       // how the guess looks locally
  const diff = Date.UTC(y, mo - 1, d, h, mi)
             - Date.UTC(p.year, p.month - 1, p.day, p.hour, p.minute);
  if (diff === 0) break;
  utcMs += diff;                                     // shift by the error, look again
}

The waiting room

For the next four hours, Priya's email sits in notifkit's waiting room, the scheduler. It's where every "not yet" email goes: quiet hours, emails scheduled for later, and retries.

The waiting room keeps two records. There's an alarm clock that's very fast at answering "who's due right now?" (that's Redis), and a notebook that's slower but doesn't forget (that's Postgres, the database). If the alarm clock ever gets wiped, notifkit notices and rebuilds it from the notebook, so a server hiccup at 5am doesn't mean Priya never gets her email.

At 7:00, the alarm goes off. But before the email leaves, notifkit checks again. Four hours is a long time. Priya could have unsubscribed, or her email address could have started bouncing. The check at 3:04am was correct at 3:04am, and that's not good enough at 7:00. She hasn't changed anything, so off it goes.

If you've ever had a "reminder" email arrive after you'd already cancelled, this is the check that was missing.

local tasks = redis.call('ZRANGE', key, 0, now, 'BYSCORE', 'LIMIT', 0, limit)
for i, task in ipairs(tasks) do
  redis.call('ZADD', key, now + visibilityTimeout, task)
end
return tasks

One key to the room

Remember the conveyor belts: sometimes a job gets picked up twice. For most steps that's harmless, but this is the step that actually sends the email. Two workers grabbing Priya's email at the same moment would mean two emails.

So the sending step works like a hotel room with one key. Before sending, a worker has to take the key for this email. While it holds the key, the door says in progress, and any other worker that turns up walks away and checks back later. When the send succeeds, the door sign changes to done, and anyone who turns up later knows to do nothing.

What if the worker holding the key crashes? The key only lasts 60 seconds, then it's returned automatically, so a crashed worker can't lock an email away forever. To make sure a slow worker doesn't lose the key mid-send, notifkit gives up on any send that takes longer than 25 seconds and tries again later instead.

// Phase 1: a short lease. SET NX means only one worker can hold it.
const ok = await redis.set(`idem:${taskId}`, "L", "EX", 60, "NX");
if (ok === "OK") return "acquired";             // our turn: send
const v = await redis.get(`idem:${taskId}`);
return v === "1" ? "completed" : "locked";      // done already / someone is on it

// Phase 2, after the provider accepted: swap the lease for a long marker.
await redis.set(`idem:${taskId}`, "1", "EX", ttlSeconds);

The provider is having a bad morning

notifkit doesn't deliver email itself. It hands each email to an email provider, a company whose job is getting mail into inboxes. Kestrel has set up two: a main one and a backup.

This morning the main provider isn't answering. The emails sent a few minutes before Priya's all timed out after ten seconds each. After five failures in a row, notifkit did what you'd do with a friend who isn't picking up: it stopped calling for a while. Engineers call this a circuit breaker, after the one in your house. When too much goes wrong, it trips, and nothing goes through until it's reset.

So when Priya's email arrives, notifkit doesn't spend ten seconds waiting on a provider it already knows is down. It goes straight to the backup, which accepts the email in a fraction of a second.

After 30 seconds, notifkit tries the main provider again, with exactly one email, like sending one scout over the bridge before the whole army. If the scout gets through, normal service resumes. If notifkit let every waiting email through at once instead, a provider that had only just recovered would get knocked over again.

What if the backup had failed too?

Then notifkit would have tried again later: after 30 seconds, then 2 minutes, then 10 minutes. It doesn't wait around in the meantime. The email goes back to the waiting room, so a server restart in those 10 minutes doesn't lose it.

Not every failure is worth retrying, though. "This address doesn't exist" won't fix itself in 10 minutes, so that kind of failure stops immediately. And if Kestrel had said "if email fails, send a text", notifkit would hand the message back to the decision step to try SMS, which has its own rules. Priya might have texts turned off, or no phone number on file.

The 7am rush

Here's something we didn't see coming. Priya isn't the only Kestrel customer in Melbourne with quiet hours until 7am. Overnight, 50,000 emails went to the waiting room, and they all wake up at the same minute.

Email providers limit how fast you can send, a bit like a motorway on-ramp with a traffic light. Say the limit is 100 a second. Here's how notifkit used to handle it: 100 went through, and the other 49,900 were told "come back next second". So they all came back next second, together. 100 got through, 49,800 bounced, and so on. It was a traffic jam that formed again every second.

The fix is the one you'd use for people at a door: don't tell everyone to come back at the same time. Each email that gets turned away picks a random moment to return, and the longer it has been waiting, the wider the spread. The crowd thins into a queue that moves at the speed the provider allows.

Waiting in line also doesn't count as failing. An email can be turned away by the rate limit up to 50 times without losing any of its three real attempts. Being stuck in traffic isn't the same as the road being closed.

// Spread over more windows the longer a task has waited.
const spreadMs = Math.random() * windowMs * Math.min(task.throttleAttemptCount, 10);
await this.reschedule(task, new Date(Date.now() + retryAfterMs + spreadMs), "throttle");

The one duplicate we can't rule out

We've made a lot of repeats harmless. There's one we can't fully prevent, and you should be suspicious of anyone who claims they can.

Picture posting a letter, then fainting on the way home before you tick it off your to-do list. When you wake up, the list says "not posted". Do you post it again? You have no way of knowing whether it went, because the answer is inside the post office.

It's the same for notifkit. Here's every moment the sending worker could crash during Priya's email:

The red row lasts a few milliseconds per email. It takes a crash at exactly that moment, or a timeout where the provider got the email but the answer didn't make it back. At millions of emails a month, it happens occasionally.

The real fix has to come from the post office: give the provider your own label for the email, and let it refuse a second letter with the same label. Some providers support this. Notifkit already has a stable label for every email, but it doesn't pass it to providers yet. That's next on our list.

Priya reads her email

Priya picks up her phone. "Your order has shipped." One email, at a reasonable hour. She doesn't know about the Wi-Fi, the 3am wake-up call that didn't happen, the provider outage or the 50,000 emails queued at 7am, and she shouldn't have to.

That's the joke in the title. Notifications look like the easy part of an app. Underneath, they meet the same problems as banks, payment systems and databases: things happen twice, machines crash at bad moments, clocks disagree, and other companies' servers go down. You don't avoid these problems; you design for each one ahead of time.

The cheat sheet

Notifkit is free, MIT-licensed, and runs next to your own Postgres and Redis. The quickstart takes about ten minutes. Curious how many Priyas one small server can handle? We load-tested it on a $25 server.