Design a Notification System

One pipeline, several channels: eligibility, deduplication, quiet hours, and what to do when a provider or a device token dies.

SyntaxPOST https://fcm.googleapis.com/v1/projects/{project_id}/messages:send | POST https://api.push.apple.com/3/device/{deviceToken}

A notification system turns product events into messages on channels you do not control. The design work is almost entirely in the decisions before the send: is this user eligible, has this already been sent, and is now an acceptable time.

The pipeline

Event in, then: load preferences, decide channels, deduplicate, render per channel, hand to a provider adapter, record the outcome. Each stage is a queue boundary so that a slow SMS provider cannot stall in-app delivery, and each stage is keyed by the producer's event id so a redelivered queue message is harmless.

Deduplication

Two layers, and you want both. Yours: a claim key per (event, user, channel) with a TTL, written with a conditional set — the worker that creates the key sends, a duplicate does not. The provider's: APNs apns-collapse-id (64 bytes maximum) and FCM's Android collapse_key replace an earlier undelivered notification with the same id instead of stacking three badges for the same chat.

Preferences and quiet hours

Store the user's IANA timezone name, never a UTC offset, and treat quiet hours as a schedule: compute the next permitted send time and persist a delayed job, rather than sleeping in a worker. Decide up front which classes ignore quiet hours — security and payment failures normally do — because that is a product rule, not an implementation detail.

Provider failure

Separate three outcomes that all look like errors: a dead token (delete it), a throttle or 5xx (retry with backoff, trip a breaker), and a rejected payload (alert a human; retrying never fixes it).

Example

Example · javascript
// One pipeline stage: decide, deduplicate, schedule, send.
// event = { id, userId, type, payload }  - id is supplied by the producer and is
// stable across redeliveries, which is what makes the dedup key meaningful.

const CRITICAL = new Set(['security.login', 'billing.payment_failed']);

async function handle(event) {
  const prefs = await preferences.get(event.userId);

  // Absent preference rows fall back to the product default, so shipping a new
  // notification category does not require a backfill for every user.
  const channels = (prefs.channels[event.type] ?? DEFAULTS[event.type] ?? [])
    .filter((c) => c !== 'email' || prefs.emailVerified)
    .filter((c) => c !== 'sms' || prefs.phoneVerified);

  for (const channel of channels) {
    // SET NX is the whole dedup mechanism: the first worker to claim the key
    // sends. A retried queue message, or the same event emitted twice, does not.
    const key = `notif:${event.id}:${event.userId}:${channel}`;
    const claimed = await redis.set(key, '1', { NX: true, EX: 7 * 24 * 3600 });
    if (!claimed) continue;

    const when = nextAllowedTime(new Date(), prefs, event.type);
    if (when > new Date()) {
      // Quiet hours are a schedule, not a sleep: persist the decision so a
      // worker restart neither drops nor duplicates the notification.
      await scheduler.enqueueAt(when, { ...event, channel });
      continue;
    }

    try {
      await adapters[channel].send(event, prefs);
    } catch (err) {
      // Release the claim only for retryable failures, or a transient provider
      // error would silently suppress the notification forever.
      if (err.retryable) { await redis.del(key); throw err; }
      await outcomes.record(event, channel, err);
    }
  }
}

// Intl is the timezone database already in the runtime. Reading the local hour
// through it is what makes 22:00-08:00 correct across a DST change.
function localHour(date, timeZone) {
  return Number(new Intl.DateTimeFormat('en-GB',
    { hour: 'numeric', hour12: false, timeZone }).format(date));
}

function nextAllowedTime(now, prefs, type) {
  if (CRITICAL.has(type) || prefs.quietStart == null) return now;
  const inQuiet = (d) => {
    const h = localHour(d, prefs.timezone);
    return prefs.quietStart > prefs.quietEnd          // window crosses midnight
      ? (h >= prefs.quietStart || h < prefs.quietEnd)
      : (h >= prefs.quietStart && h < prefs.quietEnd);
  };
  if (!inQuiet(now)) return now;
  // Step forward in real time rather than doing arithmetic on local hours; this
  // stays correct on the days that have 23 or 25 hours.
  const next = new Date(now);
  while (inQuiet(next)) next.setUTCMinutes(next.getUTCMinutes() + 15);
  return next;
}

When to use it

  • Stopping a retrying queue from sending the same "order shipped" push four times.
  • Honouring a 22:00-08:00 quiet window correctly across timezones and DST changes.
  • Adding SMS as a fallback when a push has not been opened within ten minutes.
  • Deciding which notification classes are allowed to override a user's quiet hours.

More examples

The FCM shorthand next to its per-platform longhand

The top-level notification is the shorthand: FCM expands one title and body into each platform's own form. android.notification and apns.payload.aps.alert are that longhand written out, and where both are present the platform block's title and body override the common ones - which is exactly how you give Android a tag and iOS a thread-id from one request. Reach for the shorthand for simple alerts, and drop to the longhand the moment a platform needs a field the common block has no name for: collapse_key on Android, apns-collapse-id (64 bytes maximum) on iOS, and apns-expiration 0 to say discard rather than store if the device is offline.

Example · json
{
  "message": {
    "token": "DEVICE_TOKEN",

    "notification": { "title": "3 new messages", "body": "from Priya" },

    "android": {
      "collapse_key": "chat-42",
      "ttl": "3600s",
      "notification": { "title": "3 new messages", "body": "from Priya", "tag": "chat-42" }
    },

    "apns": {
      "headers": {
        "apns-push-type": "alert",
        "apns-priority": "10",
        "apns-collapse-id": "chat-42",
        "apns-expiration": "0"
      },
      "payload": {
        "aps": { "alert": { "title": "3 new messages", "body": "from Priya" }, "thread-id": "chat-42" }
      }
    }
  }
}

Three different meanings of a provider error

Conflating these three is the most common reason a notification service degrades badly: dead tokens burn quota, deterministic rejections fill the retry queue, and an un-breakered outage turns one provider's bad hour into your own. Reach for this shape whenever you add a channel, and give each channel its own breaker and its own queue so SMS being down does not delay push.

Example · javascript
// APNs is HTTP/2: the status plus the JSON reason decide whether the token
// survives, whether we retry, and whether the provider is still healthy.
async function sendApns(token, payload, headers) {
  if (breaker.isOpen()) throw new RetryableError('breaker-open'); // fail fast, keep the queue

  const res = await apns.post(`/3/device/${token}`, payload, headers);

  if (res.status === 200) { breaker.recordSuccess(); return 'sent'; }
  const reason = res.body?.reason;

  // 1. The device is gone. Delete the token; nothing you send will ever arrive.
  if (res.status === 410 || reason === 'Unregistered' || reason === 'BadDeviceToken') {
    await tokens.delete(token);
    return 'dead-token';
  }

  // 2. Transient. Retry with backoff, and count it against the breaker so a
  //    provider outage does not turn into a million in-flight retries.
  if (res.status === 429 || res.status >= 500) {
    breaker.recordFailure();
    throw new RetryableError(reason ?? `status ${res.status}`);
  }

  // 3. Our fault. A malformed payload or wrong topic is deterministic: park it
  //    in the dead-letter store and page someone instead of looping.
  await deadLetters.put({ token, reason, status: res.status });
  return 'rejected';
}

A preference matrix that survives new categories

Shows preferences as sparse overrides rather than a dense row per user per category, which is what keeps adding a notification type from becoming a migration. The hourly digest query is also a load-shaping trick: evaluating the user's local hour spreads the send naturally across timezones, so the email provider sees a steady rate instead of one spike.

Example · sql
-- Per-user settings that every channel needs, including the timezone that
-- quiet hours and digests are evaluated in.
CREATE TABLE user_notification_settings (
  user_id     BIGINT PRIMARY KEY,
  timezone    TEXT     NOT NULL,          -- IANA name, 'Europe/Lisbon' - never an offset
  quiet_start SMALLINT,                   -- local hour, NULL = no quiet hours
  quiet_end   SMALLINT,
  digest_hour SMALLINT NOT NULL DEFAULT 9
);

-- Sparse overrides: a missing row means "whatever the product default is".
-- A new category therefore ships without touching a single user row.
CREATE TABLE notification_prefs (
  user_id  BIGINT NOT NULL REFERENCES user_notification_settings (user_id),
  category TEXT   NOT NULL,               -- 'mention' | 'follow' | 'billing' | 'security'
  channel  TEXT   NOT NULL,               -- 'push' | 'email' | 'sms' | 'inapp'
  mode     TEXT   NOT NULL,               -- 'instant' | 'digest' | 'off'
  PRIMARY KEY (user_id, category, channel)
);

-- The digest sweep runs every hour and selects only the users whose local hour
-- is now their digest hour, which spreads a global send over 24 hours instead
-- of firing every email at once.
SELECT s.user_id
FROM user_notification_settings s
WHERE s.digest_hour = EXTRACT(HOUR FROM now() AT TIME ZONE s.timezone)
  AND EXISTS (SELECT 1 FROM notification_prefs p
              WHERE p.user_id = s.user_id AND p.mode = 'digest');

Discussion

  • Be the first to comment on this lesson.