Operations

What to watch, what to alert on, and what to do at 3am when notifications stop.

Health surface

A worker with no heartbeat reports { status: "unknown", message: "No report received from worker" }. In a split deployment that's expected, since the API can't see workers you didn't deploy. In a services: ["all"] process it means the worker died.

Metrics

Default Node.js process metrics (heap, event loop lag, GC) are exported alongside these.

Alerts worth having

When nothing is arriving

The dead-letter queue

A message that exhausts its attempts moves to notifkit:stream:dlq. Nothing there is retried automatically; a poison payload would just loop.

# Most recent 50
curl -s http://localhost:3000/v1/dlq -H "Authorization: Bearer $NK_KEY" | jq

# Put one back at the front of the pipeline
curl -X POST http://localhost:3000/v1/dlq/replay \
  -H "Authorization: Bearer $NK_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id":"1737054981234-0"}'

# Discard one
curl -X DELETE http://localhost:3000/v1/dlq/1737054981234-0 -H "Authorization: Bearer $NK_KEY"

A replay republishes the entry to the inbound stream at its original priority and removes it from the DLQ. Fix the cause first (a bad template, missing transport, expired credential), or it lands right back.

Configuration

Constructor options beat environment variables, which beat .env, which beats defaults.

Per-project limits

Each project has its own API request ceiling: 600 requests per minute by default, as a sliding window in Redis. Over it, callers get 429 with Retry-After: 60.

curl -X PATCH http://localhost:3000/v1/projects/{projectId} \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"rateLimitRpm": 3000, "throttleLimit": 50, "throttleWindowHours": 24}'

Requests made with the admin key are allowed 6 000 per minute.

Changes apply on the next notification: the engine caches these per project for 60 seconds, and a PATCH publishes an invalidation so every worker drops its copy at once. throttleLimit of 0 stops all non-critical sends for the project, a usable kill switch; critical priority still bypasses the throttle.

Key rotation

# Mint the replacement
curl -X POST http://localhost:3000/v1/projects/{projectId}/keys \
  -H "Authorization: Bearer $ADMIN_API_KEY" \
  -H "Content-Type: application/json" -d '{"role":"admin"}'

# List what exists — returns id, role, and createdAt. Never the key itself.
# Take the id of the key you are retiring; that is the {keyId} below.
curl -s http://localhost:3000/v1/projects/{projectId}/keys -H "Authorization: Bearer $ADMIN_API_KEY"

# Revoke the old one
curl -X DELETE http://localhost:3000/v1/projects/{projectId}/keys/{keyId} \
  -H "Authorization: Bearer $ADMIN_API_KEY"

Deploy the new key, then revoke the old one. Revocation invalidates the cache in every API process, so it takes effect immediately, not after the 60-second auth cache expires.

Give read_only keys to anything that only reads: dashboards, status pages, reporting jobs. They get 403 on any non-GET request.

Suppressions and deliverability

The suppression list records people who asked you to stop, plus addresses that no longer exist. It's checked before every dispatch, and priority: "critical" doesn't override it.

# Everything suppressed, newest first
curl -s "http://localhost:3000/v1/suppressions?limit=200" -H "Authorization: Bearer $NK_KEY"

# Only the hard bounces — a spike here usually means a stale or bought list
curl -s "http://localhost:3000/v1/suppressions?reason=bounced" -H "Authorization: Bearer $NK_KEY"

# Someone asked to be removed by phone or reply
curl -X POST http://localhost:3000/v1/suppressions -H "Authorization: Bearer $NK_KEY" \
  -H "Content-Type: application/json" \
  -d '{"channel":"email","target":"[email protected]","reason":"manual"}'

Watch two numbers per campaign, both from GET /v1/campaigns/{campaign}/stats:

Shutdown

SIGINT and SIGTERM trigger a graceful stop: the HTTP server closes, consumers stop reading, in-flight messages get up to 30 seconds to finish, buffered log rows are flushed, and connections close. Anything unfinished at the deadline stays unacked and another worker reclaims it, so nothing is lost.

Give your orchestrator a termination grace period above 30 seconds so it doesn't SIGKILL mid-flush.

// Shutting down explicitly, e.g. from a test harness.
await server.stop();

Production checklist

• NODE_ENV=production, with REDIS_URL and DATABASE_URL set explicitly.
• autoMigrate: false everywhere except the one deployment that owns migrations.
• /health and /metrics unreachable from the public internet.
• ADMIN_API_KEY unset on deployments that do not manage projects.
• Postgres max_connections sized for services × processes × DB_MAX_CONNECTIONS.
• Recommended: Redis persistence (AOF) on. In-flight messages and pending lists live there, so without it a restart loses whatever's queued. Managed Redis without AOF, such as ElastiCache, works with that trade-off.
• Termination grace period above 30 seconds.
• notifkit 0.1.15 or later, so a large backlog is never trimmed. On older versions, QUEUE_MAX_LEN above your worst expected backlog.
• TRUST_PROXY=true if, and only if, notifkit sits behind a reverse proxy.
• Alerts on stream depth, delivery failure ratio, and DLQ length.
• Transports registered on every delivery deployment, and on api if you use provider webhooks.