Enterprise Notification Platform Design

Design a scalable enterprise notification platform that delivers email, SMS, push, and in-app notifications across multiple channels. Covers requirements, capacity estimation, high-level architecture, routing engine, template service, delivery pipeline, rate limiting, preference management, retry logic, observability, and trade-offs.

1-Hour Interview Roadmap

Time	Topic
0 – 5 min	Requirements clarification
5 – 10 min	Capacity estimation
10 – 18 min	High-level architecture + core components
18 – 28 min	Notification routing + template engine
28 – 38 min	Delivery pipeline — channel workers + provider abstraction
38 – 46 min	Reliability — retry, deduplication, rate limiting
46 – 52 min	Database design + user preference management
52 – 58 min	Observability + failure handling
58 – 60 min	Trade-offs + common interview mistakes

What Are We Building?

An enterprise-grade notification platform that:

Accepts notification requests from any internal service and delivers them to users across email, SMS, push, and in-app channels
Applies user preference management — users opt out, set quiet hours, and choose preferred channels
Renders dynamic templates with per-user personalization
Routes to the best available channel when a preferred channel is unavailable
Guarantees at-least-once delivery without sending duplicates to the user
Handles third-party provider failures with retry, fallback, and circuit breaking

Scale reference: Enterprises like Uber, Airbnb, and Slack each deliver hundreds of millions of notifications per day across mobile, email, and SMS. Design for 50 million notifications per day as our target scale.

Key unique challenges:

Multi-channel fan-out — one notification event may result in delivery across several channels simultaneously
Provider unreliability — email, SMS, and push providers each have their own SLAs, rate limits, and failure modes
User preferences — delivering a notification the user has opted out of is a compliance and trust failure
Deduplication at scale — retries must not result in duplicate messages reaching the user
Rate limiting — sending too many notifications too fast triggers provider blocks and user fatigue

Step 1 — Requirements

Functional Requirements

#	Requirement
1	Internal services send notification requests specifying recipient, template, channel preference, and payload data
2	Platform supports four channels: Email, SMS, Push (iOS/Android), and In-App
3	Users can manage notification preferences: opt-out per channel, set quiet hours, and set preferred channel
4	Notifications are rendered from templates with per-user dynamic variable substitution
5	Platform routes to the best available channel when the preferred channel is unavailable or opted-out
6	Notifications are delivered at-least-once — no notification is silently dropped
7	Duplicate prevention — same logical notification is not delivered more than once per user per channel
8	Platform exposes delivery status: queued, sent, delivered, failed
9	Supports scheduled notifications — deliver at a specific future time
10	Supports batch notifications — broadcast to millions of users simultaneously

Non-Functional Requirements

#	Requirement
1	API-triggered notifications delivered to provider within 5 seconds (p99)
2	Batch/broadcast notifications completed within 30 minutes for 10M recipients
3	High availability — 99.9% uptime for the notification API
4	At-least-once delivery with deduplication to prevent user-facing duplicates
5	Horizontally scalable — throughput added by scaling workers, not re-architecting
6	Provider-agnostic — swap email/SMS/push providers without code changes to callers
7	Audit trail — every notification attempt and outcome is permanently logged

Out of Scope

Two-way SMS or email (replies, inbound handling)
Real-time chat (WebSocket message delivery)
Marketing campaign builder UI
A/B testing and notification analytics dashboards
GDPR data deletion pipeline (assume handled by a separate service)

Step 2 — Capacity Estimation

Traffic Estimates

Daily notifications:          50 million
Average TPS:                  50M / 86,400 sec  = ~580 TPS
Peak TPS (5× spike):          ~2,900 TPS

Channel breakdown (approximate):
  Push (iOS/Android):         60%  → 30M/day
  Email:                      25%  → 12.5M/day
  In-App:                     10%  → 5M/day
  SMS:                        5%   → 2.5M/day

Batch broadcasts:             10 events/day, avg 1M recipients each
Batch throughput needed:      1M / (30 min × 60 sec) = ~555 sends/sec per batch

Storage Estimates

Per notification record:
  Notification log entry:          ~600 bytes
  Template-rendered content:       ~2 KB (email/SMS body, stored compressed)
  Delivery status event:           ~200 bytes per attempt
  Total per notification:          ~3 KB

Daily storage:
  50M notifications × 3 KB = ~150 GB/day

Annual raw storage (90-day hot, archive rest):
  Hot (90 days):       150 GB × 90 = ~13.5 TB
  Archive (1 year):    150 GB × 365 = ~55 TB

User preference records:
  100M users × 500 bytes = ~50 GB  (fits in a replicated relational DB)

Template storage:
  500 templates × 50 KB avg = 25 MB  (trivial — fully in-memory cache)

Key Insights

Write-heavy — every notification creates records; reads are mostly status queries
Push is the dominant channel — 60% of volume; push worker fleet must be the largest
Batch is a distinct workload — broadcast jobs require a separate fan-out pipeline, not the real-time path
Storage is large but manageable — tiered storage with 90-day hot retention keeps query costs acceptable
Preference checks must be fast — every notification hits the preference service; it must be low-latency (Redis-backed)

Step 3 — High-Level Architecture

flowchart TD
    Callers["Internal Services\n(Order, Auth, Billing, ...)"] --> AG["API Gateway\nAuth + Rate Limiting"]

    AG --> NS["Notification Service\n(Orchestrator)"]

    NS --> PS["Preference Service\n(Redis-backed)"]
    NS --> TS["Template Service\n(Render engine)"]
    NS --> NDB[("PostgreSQL\nNotification DB")]
    NS --> KF["Kafka\nNotification Queue"]

    KF --> EW["Email Worker"]
    KF --> SW["SMS Worker"]
    KF --> PW["Push Worker"]
    KF --> IW["In-App Worker"]

    EW --> EP["Email Providers\n(SES / SendGrid)"]
    SW --> SP["SMS Providers\n(Twilio / SNS)"]
    PW --> PP["Push Providers\n(FCM / APNs)"]
    IW --> NDB

    EP --> DT["Delivery Tracker"]
    SP --> DT
    PP --> DT

    DT --> NDB
    DT --> KF2["Kafka\nDelivery Events"]

    KF2 --> WH["Webhook Service\n(caller callbacks)"]

Component Responsibilities

Component	Responsibility
API Gateway	Authenticate callers (service-to-service JWT), rate-limit per caller, validate request schema
Notification Service	Orchestrate: check preferences, resolve channels, render template, persist record, enqueue
Preference Service	User opt-out, quiet hours, preferred channel — Redis-backed with PostgreSQL source of truth
Template Service	Load template by ID, substitute variables, return rendered content per channel format
Notification DB	PostgreSQL — notification records, delivery attempts, audit log, scheduled jobs
Kafka (Notifications)	Decouple notification acceptance from channel delivery; per-channel topics
Channel Workers	Email/SMS/Push/In-App — consume queue, call provider, report outcome
Provider Abstraction	Adapter layer over external providers; handles provider-specific API differences
Delivery Tracker	Receive provider webhooks and callbacks; update delivery status in DB
Kafka (Delivery Events)	Publish delivery outcomes to callers via webhooks or event subscriptions

Step 4 — Notification Routing and Preference Engine

Before any message is sent, two questions must be answered: which channels should receive this notification? and does this user want it?

Routing Decision Flow

flowchart TB
    Request["Notification Request\n{user_id, type, channels: ['push','email'], data}"]

    Request --> LoadPrefs["Load user preferences\n(Redis, 1ms latency)"]
    LoadPrefs --> QuietHours["Check quiet hours\n(Is current time in user's\ndo-not-disturb window?)"]

    QuietHours -->|"Yes"| Schedule["Reschedule for\nend of quiet window"]
    QuietHours -->|"No"| FilterChannels["Filter channels\nby user opt-out status"]

    FilterChannels --> NoChannel{"Any channel\nremaining?"}
    NoChannel -->|"No"| Drop["Drop with status:\nSUPPRESSED_BY_PREFERENCE\n(logged, not retried)"]
    NoChannel -->|"Yes"| Rank["Rank channels by\nuser preference order"]

    Rank --> FanOut["Fan-out: enqueue to\neach eligible channel"]

User Preference Schema

Every user has a preference record that controls what they receive, when, and how.

-- User notification preferences
CREATE TABLE user_notification_preferences (
    user_id          UUID          NOT NULL,
    channel          VARCHAR(20)   NOT NULL,  -- 'email' | 'sms' | 'push' | 'in_app'
    notification_type VARCHAR(50)  NOT NULL,  -- 'order_update' | 'security_alert' | 'marketing' | ...
    is_enabled       BOOLEAN       NOT NULL DEFAULT TRUE,
    updated_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    PRIMARY KEY (user_id, channel, notification_type)
);

-- Global quiet hours per user
CREATE TABLE user_quiet_hours (
    user_id          UUID          NOT NULL PRIMARY KEY,
    start_time       TIME          NOT NULL,  -- e.g., '22:00'
    end_time         TIME          NOT NULL,  -- e.g., '08:00'
    timezone         VARCHAR(50)   NOT NULL,  -- e.g., 'America/New_York'
    updated_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW()
);

Preference Caching Strategy

Preference checks happen on every single notification — at 2,900 peak TPS, even 5ms database latency adds 14.5 seconds of total stall time per second of throughput. Preferences must be served from Redis.

Cache key:   prefs:{user_id}
Cache value: Compressed JSON of all preferences for this user
TTL:         15 minutes (reloaded from PostgreSQL on miss or update)

On preference update:
  1. Write to PostgreSQL (source of truth)
  2. Invalidate Redis key immediately
  3. Next notification for this user reloads from DB

Notification Priority Levels

Not all notifications are equal. Security alerts cannot be delayed. Marketing messages can wait.

Priority	Examples	Quiet Hours	Behavior
CRITICAL	Password reset, account locked, fraud alert	Bypassed	Delivered immediately regardless of preferences
HIGH	Order shipped, payment received	Respected	Delivered now if outside quiet hours
NORMAL	Reminders, activity updates	Respected	Delivered now if outside quiet hours
LOW	Marketing, newsletters, tips	Respected	Batched and delivered at scheduled time

Critical notifications bypass quiet hours. This must be used sparingly — abuse results in users disabling all notifications entirely.

Step 5 — Template Service

The Template Service renders channel-specific content for each notification. A single "Order Shipped" notification type requires four different rendered outputs: an HTML email, an SMS body under 160 characters, a push title+body within character limits, and a structured JSON payload for the in-app feed.

Template Structure

{
  "template_id":   "order_shipped_v2",
  "notification_type": "order_shipped",
  "channels": {
    "email": {
      "subject":  "Your order {{order_number}} has shipped!",
      "html_body": "<h1>Hi {{first_name}},</h1><p>Your order is on the way...</p>",
      "text_body": "Hi {{first_name}}, your order {{order_number}} has shipped."
    },
    "sms": {
      "body": "Hi {{first_name}}, order {{order_number}} shipped! Track: {{tracking_url}}"
    },
    "push": {
      "title": "Your order has shipped",
      "body":  "Order {{order_number}} is on its way to {{delivery_city}}",
      "deep_link": "myapp://orders/{{order_id}}"
    },
    "in_app": {
      "title":   "Order {{order_number}} Shipped",
      "message": "Your order has shipped and will arrive by {{estimated_delivery_date}}",
      "action_url": "/orders/{{order_id}}"
    }
  }
}

Rendering Pipeline

sequenceDiagram
    participant NS as Notification Service
    participant TS as Template Service
    participant Cache as Redis Template Cache
    participant DB as Template DB

    NS->>TS: RenderTemplate(template_id="order_shipped_v2", channel="email", data={order_number, first_name, ...})
    TS->>Cache: GET template:order_shipped_v2
    Cache-->>TS: Cache HIT — return template JSON
    TS->>TS: Substitute variables using Mustache/Handlebars
    TS->>TS: Validate rendered output (length checks, required fields)
    TS-->>NS: {subject: "Your order 12345...", html_body: "...", text_body: "..."}

Template rendering is synchronous and happens in the Notification Service before enqueuing. This ensures that if template rendering fails (missing variable, template not found), the error is returned immediately to the caller — not discovered later by a worker.

Template Versioning

Templates are versioned. A new template version can be staged alongside the existing version and gradually rolled out. Notifications always reference a specific version, so in-flight notifications are unaffected by template updates.

CREATE TABLE notification_templates (
    template_id      VARCHAR(100) NOT NULL,
    version          INT          NOT NULL,
    channel          VARCHAR(20)  NOT NULL,
    subject          TEXT,
    html_body        TEXT,
    text_body        TEXT,
    is_active        BOOLEAN      NOT NULL DEFAULT TRUE,
    created_at       TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    PRIMARY KEY (template_id, version, channel)
);

Step 6 — Delivery Pipeline

Kafka Topic Design

Each channel has its own Kafka topic. This provides independent scaling, independent consumer lag monitoring, and isolation — an email backlog does not delay push delivery.

Topic Name	Producers	Consumers	Partitions
`notifications.push`	Notification Service	Push Workers	32
`notifications.email`	Notification Service	Email Workers	16
`notifications.sms`	Notification Service	SMS Workers	8
`notifications.inapp`	Notification Service	In-App Workers	8
`notifications.scheduled`	Scheduler Service	Notification Service	4
`notifications.delivery_events`	Channel Workers	Webhook Service	16

Partitioning by user_id % num_partitions ensures all notifications for the same user go to the same partition, maintaining ordering for the in-app feed.

Channel Worker Architecture

All four channel workers follow the same pattern — only the provider client differs.

flowchart LR
    KF["Kafka Topic\nnotifications.push"]

    subgraph Worker["Push Worker (per instance)"]
        Consumer["Kafka Consumer"]
        Dedup["Deduplication Check\n(Redis SET NX)"]
        Pref["Re-check Preference\n(stale-safe)"]
        Provider["FCM / APNs Client"]
        Retry["Retry with\nExp Backoff"]
        DLQ["Dead Letter Queue\n(after max retries)"]
    end

    Tracker["Delivery Tracker\n(update DB)"]

    KF --> Consumer
    Consumer --> Dedup
    Dedup -->|"Duplicate"| Drop["Skip — already sent"]
    Dedup -->|"New"| Pref
    Pref -->|"Opted out"| Suppress["Mark SUPPRESSED"]
    Pref -->|"Eligible"| Provider
    Provider -->|"Success"| Tracker
    Provider -->|"Transient failure"| Retry
    Retry -->|"Max retries exceeded"| DLQ
    DLQ --> Alert["PagerDuty Alert\n+ Manual review"]

Email Worker — Provider Abstraction

The provider abstraction layer allows swapping email providers (SendGrid → SES → Mailgun) without changing the worker logic.

EmailProviderAdapter interface:
  - Send(to, subject, html_body, text_body, metadata) → (provider_message_id, error)
  - GetDeliveryStatus(provider_message_id) → (status, timestamp, error)

Implementations:
  - SendGridAdapter
  - AWSESAdapter
  - MailgunAdapter

Active provider selected by feature flag / circuit breaker state
Fallback chain: SendGrid → SES → Mailgun

Push Worker — FCM and APNs

Push delivery requires maintaining separate connections to Google FCM (Android) and Apple APNs (iOS), with different authentication and payload formats.

flowchart TB
    PW["Push Worker"]

    PW --> DeviceDB["Load device tokens\nfor user_id\n(PostgreSQL)"]
    DeviceDB --> DeviceList["user has\n2 iOS devices, 1 Android device"]

    DeviceList --> iOS1["APNs → iOS Device 1"]
    DeviceList --> iOS2["APNs → iOS Device 2"]
    DeviceList --> Android1["FCM → Android Device 1"]

    iOS1 -->|"410 Gone"| Cleanup["Remove stale token\nfrom DB"]
    iOS2 -->|"200 OK"| Delivered["Mark delivered"]
    Android1 -->|"200 OK"| Delivered

Stale token management: When APNs returns 410 Gone or FCM returns NotRegistered, the device token is immediately removed from the database. Keeping stale tokens wastes push throughput and skews delivery metrics.

Step 7 — Reliability: Retry, Deduplication, and Rate Limiting

Retry Strategy

Not all failures should be retried the same way. A clear retry policy prevents flooding a recovering provider while ensuring eventual delivery.

Failure Type	Retry?	Strategy
Network timeout	Yes	Exponential backoff: 1s → 2s → 4s → 8s → 16s
Provider 429 (rate limited)	Yes	Respect `Retry-After` header; pause consumer
Provider 5xx (server error)	Yes	Exponential backoff with jitter
Provider 4xx (bad request, invalid token)	No	Mark FAILED immediately; log error
Opted-out after enqueue	No	Mark SUPPRESSED; do not retry
Max retries exceeded	No	Move to Dead Letter Queue; alert on-call

Max retry attempts:   5
Base delay:           1 second
Max delay:            5 minutes
Jitter:               ±20% of computed delay (prevents thundering herd)
DLQ retention:        7 days (manual review + re-process if needed)

Deduplication

At-least-once delivery from Kafka means a worker may process the same notification message more than once — during rebalances, restarts, or slow consumer commits. Deduplication prevents the user from receiving duplicate messages.

Dedup key:   dedup:{notification_id}:{channel}
Store:       Redis SET NX with TTL = 48 hours
Check:       Worker checks before calling provider
On hit:      Skip delivery, log "DUPLICATE_SKIPPED", commit Kafka offset
On miss:     Proceed with delivery, set Redis key

Why 48-hour TTL? Kafka consumer lag can theoretically cause a message to be reprocessed hours later if a consumer was down. 48 hours covers all practical replay windows.

Idempotency with providers: For email (SES, SendGrid), pass the notification_id as the provider's idempotency key where supported. For push (FCM/APNs), collapse keys can prevent duplicate visible notifications at the device level.

Rate Limiting

Three layers of rate limiting protect the system and its providers:

flowchart TB
    Request["Notification Request"]

    L1["Layer 1: Caller Rate Limit\nPer-service API rate limit\n(API Gateway)\nPrevents one service from flooding the platform"]
    L2["Layer 2: Per-User Rate Limit\nMax N notifications per user per hour\n(Redis sliding window)\nPrevents spamming a single user"]
    L3["Layer 3: Provider Rate Limit\nRespect provider throughput limits\n(worker-level token bucket)\nPrevents provider bans"]

    Request --> L1 --> L2 --> L3

Per-user rate limit example:

Key:   ratelimit:user:{user_id}:{channel}:{hour_bucket}
Type:  Redis counter with INCR
TTL:   1 hour (expires automatically)
Limit:
  Push:   10 per hour, 50 per day
  Email:  5 per hour, 20 per day
  SMS:    3 per hour, 10 per day
  In-App: 20 per hour (no daily limit)

If a rate limit is exceeded, the notification is either deferred (scheduled to next window) or dropped with status RATE_LIMITED, depending on the notification's priority level.

Step 8 — Database Design

Core Tables

-- Central notification record — one row per notification request
CREATE TABLE notifications (
    notification_id  UUID          NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY,
    caller_service   VARCHAR(50)   NOT NULL,
    user_id          UUID          NOT NULL,
    notification_type VARCHAR(50)  NOT NULL,
    template_id      VARCHAR(100)  NOT NULL,
    template_version INT           NOT NULL,
    priority         VARCHAR(20)   NOT NULL DEFAULT 'NORMAL',
    idempotency_key  VARCHAR(255),              -- caller-provided; optional dedup
    payload_data     JSONB         NOT NULL,    -- raw variables for template
    scheduled_at     TIMESTAMPTZ,               -- NULL = deliver immediately
    status           VARCHAR(30)   NOT NULL DEFAULT 'PENDING',
    created_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW()
);

-- One row per channel per notification
CREATE TABLE notification_channel_deliveries (
    delivery_id      UUID          NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY,
    notification_id  UUID          NOT NULL REFERENCES notifications(notification_id),
    channel          VARCHAR(20)   NOT NULL,
    status           VARCHAR(30)   NOT NULL DEFAULT 'QUEUED',
    provider         VARCHAR(50),               -- 'ses' | 'sendgrid' | 'fcm' | 'apns' | 'twilio'
    provider_message_id VARCHAR(255),           -- provider-side reference
    rendered_subject TEXT,
    rendered_body    TEXT,
    attempt_count    INT           NOT NULL DEFAULT 0,
    last_attempted_at TIMESTAMPTZ,
    delivered_at     TIMESTAMPTZ,
    failed_reason    TEXT,
    created_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW()
);

-- Immutable attempt log — one row per delivery attempt
CREATE TABLE delivery_attempts (
    attempt_id       UUID          NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY,
    delivery_id      UUID          NOT NULL REFERENCES notification_channel_deliveries(delivery_id),
    attempted_at     TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    outcome          VARCHAR(20)   NOT NULL,   -- 'SUCCESS' | 'FAILURE' | 'SKIPPED'
    provider_response JSONB,
    error_code       VARCHAR(50),
    duration_ms      INT
);

-- User device tokens for push notifications
CREATE TABLE user_device_tokens (
    token_id         UUID          NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY,
    user_id          UUID          NOT NULL,
    platform         VARCHAR(10)   NOT NULL,   -- 'ios' | 'android'
    token            VARCHAR(500)  NOT NULL UNIQUE,
    device_name      VARCHAR(100),
    app_version      VARCHAR(20),
    is_active        BOOLEAN       NOT NULL DEFAULT TRUE,
    registered_at    TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    last_used_at     TIMESTAMPTZ
);

Notification Status State Machine

stateDiagram-v2
    [*] --> PENDING : Notification record created

    PENDING --> SUPPRESSED : All channels opted-out or rate-limited
    PENDING --> SCHEDULED : Delivery deferred (quiet hours or scheduled_at)
    PENDING --> QUEUED : Enqueued to Kafka for each eligible channel

    SCHEDULED --> QUEUED : Scheduler fires at delivery time

    QUEUED --> IN_FLIGHT : Worker picks up and calls provider
    IN_FLIGHT --> DELIVERED : Provider confirms delivery
    IN_FLIGHT --> FAILED : Provider error, max retries exhausted
    IN_FLIGHT --> QUEUED : Transient error — re-enqueued with backoff

    SUPPRESSED --> [*]
    DELIVERED --> [*]
    FAILED --> [*]

Key Indexes

-- Query by user (notification history, in-app feed)
CREATE INDEX idx_notifications_user_id         ON notifications(user_id, created_at DESC);

-- Query pending + scheduled deliveries (scheduler polling)
CREATE INDEX idx_deliveries_status_scheduled   ON notification_channel_deliveries(status, created_at)
    WHERE status IN ('QUEUED', 'SCHEDULED');

-- Query by notification_id (status lookups by caller)
CREATE INDEX idx_deliveries_notification_id    ON notification_channel_deliveries(notification_id);

-- Dedup on caller-provided idempotency key
CREATE UNIQUE INDEX idx_notifications_idempotency
    ON notifications(caller_service, idempotency_key)
    WHERE idempotency_key IS NOT NULL;

Step 9 — In-App Notification Feed

The in-app channel is fundamentally different from the others. Email, SMS, and push are delivered to external providers and then read by the user on their device. In-app notifications are stored in the platform's own database and fetched by the client on demand.

In-App Delivery Flow

sequenceDiagram
    participant App as Mobile/Web App
    participant NS as Notification Service
    participant IW as In-App Worker
    participant DB as Notification DB
    participant WS as WebSocket Server

    NS->>IW: Enqueue in-app notification
    IW->>DB: INSERT in_app_notifications {user_id, title, message, is_read: false}
    IW->>WS: Push real-time event to connected clients

    App->>NS: GET /v1/notifications?user_id=xxx&page=1
    NS->>DB: SELECT notifications WHERE user_id = xxx ORDER BY created_at DESC
    DB-->>NS: [{notification_id, title, message, is_read, created_at}, ...]
    NS-->>App: Paginated notification list

    App->>NS: POST /v1/notifications/{id}/read
    NS->>DB: UPDATE in_app_notifications SET is_read=true WHERE notification_id = id

In-App Schema

CREATE TABLE in_app_notifications (
    notification_id  UUID          NOT NULL REFERENCES notifications(notification_id),
    user_id          UUID          NOT NULL,
    title            VARCHAR(200)  NOT NULL,
    message          TEXT          NOT NULL,
    action_url       VARCHAR(500),
    is_read          BOOLEAN       NOT NULL DEFAULT FALSE,
    read_at          TIMESTAMPTZ,
    expires_at       TIMESTAMPTZ,           -- notifications removed from feed after expiry
    created_at       TIMESTAMPTZ   NOT NULL DEFAULT NOW(),
    PRIMARY KEY (notification_id, user_id)
);

CREATE INDEX idx_inapp_user_unread ON in_app_notifications(user_id, is_read, created_at DESC);

The in-app feed is the safest channel from a deliverability perspective — there is no external provider, no rate limit from a third party, and no token invalidation to worry about. It is also the channel most users check least frequently, making it the lowest-urgency channel for time-sensitive notifications.

Step 10 — Batch and Broadcast Notifications

Sending a marketing notification to 10 million users is a fundamentally different problem from sending a single transactional notification. The real-time API path is not designed for this load — it would require generating 10 million Kafka messages in a single request.

Batch Job Architecture

flowchart TB
    Caller["Caller Service\n(Marketing, Product)"]
    Caller --> BJ["Create Batch Job\nPOST /v1/batch-notifications\n{template_id, segment_query, scheduled_at}"]

    BJ --> BatchDB["Batch Jobs Table\n(status: QUEUED)"]
    BatchDB --> Scheduler["Batch Scheduler\n(polls for ready jobs)"]

    Scheduler --> FanOut["Fan-Out Worker\n(reads user segment in chunks)"]
    FanOut --> UserDB["User DB\n(SELECT user_ids WHERE segment_query)"]
    UserDB --> Chunk1["Chunk 1: 10K users\n→ enqueue to notifications.push"]
    UserDB --> Chunk2["Chunk 2: 10K users\n→ enqueue to notifications.push"]
    UserDB --> ChunkN["Chunk N: 10K users\n→ enqueue to notifications.push"]

    Chunk1 --> PW["Push Worker Fleet\n(scales independently)"]
    Chunk2 --> PW
    ChunkN --> PW

Batch job flow:

Caller creates a batch job with a user segment query and a template
Batch job record is created with status QUEUED and a scheduled delivery time
Batch Scheduler scans for due jobs every minute
Fan-Out Worker reads matching user IDs in chunks of 10,000
For each chunk, individual notification records are created and enqueued to Kafka
Standard channel workers process the queue — the same infrastructure handles batch and real-time

This architecture means batch notifications use the same workers as real-time notifications. To avoid batch jobs starving real-time traffic, separate Kafka consumer groups are used with priority-based processing.

Batch Rate Shaping

Sending 10 million push notifications in under a minute would hit FCM rate limits. Rate shaping spreads delivery over a configured window:

Batch target:      10M notifications over 30 minutes
Target throughput: 10M / 1800 sec = ~5,555 sends/sec
Token bucket:      FCM allows ~500K sends/sec per project
Partition count:   Increase to 64 partitions for large broadcasts
Worker count:      Auto-scale to match target throughput

Step 11 — Scheduled Notifications

Scheduled notifications are a special case where delivery must happen at a precise future time — OTP expiry warnings, subscription renewal reminders, and appointment reminders.

Scheduler Design

flowchart LR
    API["POST /v1/notifications\n{scheduled_at: '2025-03-01T09:00:00Z'}"]
    API --> DB["INSERT notification\nstatus=SCHEDULED\nscheduled_at=2025-03-01T09:00:00Z"]

    Scheduler["Scheduler (cron — every 60s)"]
    Scheduler --> Query["SELECT * FROM notifications\nWHERE status = 'SCHEDULED'\nAND scheduled_at <= NOW()\nLIMIT 1000"]
    Query --> Process["For each due notification:\n1. Re-check user preferences\n2. Re-check rate limits\n3. Enqueue to Kafka\n4. Update status → QUEUED"]

Important: Preferences and rate limits are re-checked at delivery time, not at scheduling time. A user who opts out after a notification was scheduled will not receive it. This is the correct behavior for marketing and informational notifications.

Clock skew and distributed scheduling: To avoid multiple scheduler instances processing the same notification, use PostgreSQL SELECT ... FOR UPDATE SKIP LOCKED. This provides atomic job claiming without distributed locks.

-- Atomic job claiming with row-level locking
WITH due_notifications AS (
    SELECT notification_id
    FROM notifications
    WHERE status = 'SCHEDULED'
      AND scheduled_at <= NOW()
    ORDER BY scheduled_at ASC
    LIMIT 500
    FOR UPDATE SKIP LOCKED
)
UPDATE notifications
SET status = 'PROCESSING', updated_at = NOW()
WHERE notification_id IN (SELECT notification_id FROM due_notifications)
RETURNING *;

Step 12 — Observability

Key Metrics

Metric	Description	Alert Threshold
`notifications.queued_total`	Total notifications enqueued per channel	— (trend metric)
`notifications.delivered_total`	Successful deliveries per channel	— (trend metric)
`notifications.failed_total`	Failed deliveries per channel	> 5% failure rate
`notifications.e2e_latency_p99`	Time from API call to provider send	> 10 seconds
`kafka.consumer_lag`	Kafka consumer lag per channel topic	> 50,000 messages
`notifications.dlq_size`	Messages in Dead Letter Queue per channel	Any increase
`provider.error_rate`	Error rate per external provider	> 2%
`ratelimit.suppressed_total`	Notifications dropped by rate limiter	Sudden spike
`preferences.cache_hit_rate`	Redis cache hit rate for preference lookups	< 90%

Distributed Tracing

Every notification carries a trace_id from the originating caller's request. This trace ID propagates through the entire pipeline — from the API call, through Kafka, into the worker, and into the provider call. When a notification fails to arrive, the trace ID lets engineers see the exact path and failure point across all services.

Trace: trace_id=7f3a9b2c
  ├── [Notification Service] received request → 1ms
  ├── [Preference Service]   preferences loaded (cache hit) → 0.8ms
  ├── [Template Service]     template rendered → 2ms
  ├── [PostgreSQL]           record inserted → 3ms
  ├── [Kafka]                message produced → 0.5ms
  └── [Push Worker]          FCM called → 120ms ← SLOW
        └── FCM response: delivered

Delivery Rate Dashboard

A per-channel delivery rate dashboard shows the health of each provider in real time:

Channel	Sent (last hour)	Delivered	Failed	Suppressed	Avg Latency
Push	1,250,000	1,218,500 (97.5%)	12,500 (1.0%)	19,000 (1.5%)	145ms
Email	520,000	509,600 (98.0%)	5,200 (1.0%)	5,200 (1.0%)	820ms
SMS	104,000	100,360 (96.5%)	3,120 (3.0%)	520 (0.5%)	950ms
In-App	208,000	208,000 (100%)	0 (0.0%)	0 (0.0%)	12ms

Step 13 — Failure Handling

Provider Outage — Fallback Chain

When a primary provider is unavailable, the circuit breaker opens and the worker automatically routes to the fallback provider.

stateDiagram-v2
    [*] --> SendGrid_Primary

    SendGrid_Primary --> SES_Fallback : Circuit opens\n(>5% error rate, 30-sec window)
    SES_Fallback --> SendGrid_Primary : SendGrid circuit resets\n(after 60-sec probe)

    SES_Fallback --> Mailgun_Tertiary : SES circuit also opens
    Mailgun_Tertiary --> SES_Fallback : SES circuit resets

    Mailgun_Tertiary --> DLQ : All providers down\n(message to DLQ, alert on-call)

Circuit breaker parameters:

Error rate threshold: 5% of requests in a 30-second sliding window
Half-open probe: one request every 60 seconds
Metric tracked: provider error rate per worker instance

Kafka Consumer Lag — Backpressure

If workers fall behind (provider is slow, spike in volume), Kafka consumer lag grows. The system auto-scales workers based on consumer lag:

Trigger: consumer lag > 10,000 messages per partition
Action:  Scale out workers (add pods)
Scale-in: consumer lag < 1,000 messages for 10 minutes
Max workers: 50 per channel (limits concurrent provider connections)

This keeps delivery latency bounded even during traffic spikes, without over-provisioning workers at steady state.

Data Loss Prevention

flowchart LR
    Step1["1. Notification Service\nINSERT into PostgreSQL\nBEFORE producing to Kafka"]
    Step2["2. Kafka message produced\nafter DB commit"]
    Step3["3. Worker commits Kafka offset\nONLY after delivery attempt\n(success OR failure)"]
    Step4["4. Outcome written to DB\nregardless of success/failure"]

    Step1 --> Step2 --> Step3 --> Step4

If the service crashes between steps 1 and 2, a recovery job (Transactional Outbox pattern) scans for notifications with status PENDING older than 30 seconds and re-produces them to Kafka.

Step 14 — API Design

Create Notification

POST /v1/notifications

{
  "idempotency_key": "order-shipped-12345-user-789",
  "recipient": {
    "user_id": "user_789abc"
  },
  "notification_type": "order_shipped",
  "template_id": "order_shipped_v2",
  "channels": ["push", "email"],
  "priority": "HIGH",
  "data": {
    "order_number": "ORD-12345",
    "first_name": "Alex",
    "tracking_url": "https://tracking.example.com/abc123",
    "delivery_city": "San Francisco",
    "estimated_delivery_date": "March 5"
  }
}

202 Accepted response:

{
  "notification_id": "notif_9f8e7d6c5b",
  "status": "QUEUED",
  "channels": ["push", "email"],
  "created_at": "2025-03-01T10:00:00Z"
}

Query Notification Status

GET /v1/notifications/{notification_id}

{
  "notification_id": "notif_9f8e7d6c5b",
  "status": "DELIVERED",
  "channels": [
    {
      "channel": "push",
      "status": "DELIVERED",
      "provider": "fcm",
      "delivered_at": "2025-03-01T10:00:02Z",
      "attempt_count": 1
    },
    {
      "channel": "email",
      "status": "DELIVERED",
      "provider": "ses",
      "delivered_at": "2025-03-01T10:00:05Z",
      "attempt_count": 1
    }
  ]
}

Register Device Token

POST /v1/devices

{
  "user_id": "user_789abc",
  "platform": "ios",
  "token": "apns_device_token_here",
  "app_version": "4.2.1"
}

Update Preferences

PUT /v1/users/{user_id}/preferences

{
  "channels": {
    "sms": {
      "marketing": false,
      "order_updates": true,
      "security_alerts": true
    },
    "email": {
      "marketing": false,
      "order_updates": true,
      "security_alerts": true
    }
  },
  "quiet_hours": {
    "start_time": "22:00",
    "end_time": "08:00",
    "timezone": "America/New_York"
  }
}

Step 15 — Scaling

Scaling Strategy by Component

Component	Scaling Approach	Bottleneck
Notification Service	Horizontal — stateless, behind load balancer	None above 50 instances
Kafka Brokers	Add brokers + increase partitions	Partition count sets max parallelism
Push Workers	Horizontal — scale by consumer lag	FCM/APNs connection limits
Email Workers	Horizontal — scale by consumer lag	Provider sending rate
SMS Workers	Horizontal — scale by consumer lag	Twilio TPS limits
PostgreSQL (writes)	Vertical for primary; read replicas for reads	Single-writer bottleneck
Redis (preferences)	Redis Cluster — shard by user_id	Memory per shard
Redis (deduplication)	Redis Cluster — shard by notification_id	Memory per shard

Read vs Write Separation for the Notification DB

At 580 TPS writes and 5× read ratio, the notification DB sees ~2,900 QPS. A single PostgreSQL primary handles this comfortably, but status queries from dashboards and caller services should route to read replicas to protect write performance.

flowchart LR
    NS["Notification Service\n(writes)"]
    API["Status Query API\n(reads)"]
    Dashboard["Internal Dashboard\n(reporting reads)"]

    Primary["PostgreSQL Primary\n(all writes)"]
    Replica1["Read Replica 1\n(status queries)"]
    Replica2["Read Replica 2\n(dashboard + analytics)"]

    NS --> Primary
    API --> Replica1
    Dashboard --> Replica2
    Primary -->|"Streaming replication"| Replica1
    Primary -->|"Streaming replication"| Replica2

Design Trade-offs

Trade-off 1: Synchronous Template Rendering vs Asynchronous

Approach	Pros	Cons
Synchronous (chosen) — render before enqueue	Caller gets immediate error if template fails; rendered content stored for audit	Adds latency to the synchronous API path
Asynchronous — enqueue raw data, render in worker	Faster API response	Template errors discovered late; harder to debug; no rendered content at creation time

Decision: Render synchronously. Template errors are configuration bugs that callers need to know about immediately. The latency cost (< 5ms) is acceptable.

Trade-off 2: Per-User Preference Re-check in Workers

Approach	Pros	Cons
Check only at enqueue time	Simpler workers; no extra Redis calls in hot path	Notification can be delivered after user opted out (between enqueue and processing)
Re-check at worker time (chosen)	Respects preference changes in near-real-time	Extra Redis call per message in worker; slight latency

Decision: Re-check in workers. The delay between enqueue and worker processing can be seconds to minutes under load. A user who opts out should not receive a notification that was queued before the opt-out. Compliance requirements reinforce this.

Trade-off 3: Separate Topics per Channel vs Single Topic

Approach	Pros	Cons
Single topic, channel in message	Simpler producer logic	Push backlog blocks email delivery; shared consumer lag; harder to scale individual channels
Per-channel topics (chosen)	Independent scaling; isolated lag monitoring; channel-specific retention	More topics to manage; slightly more complex producer

Decision: Per-channel topics. The operational benefits of isolation and independent scaling far outweigh the administrative overhead of managing additional topics.

Trade-off 4: At-Least-Once vs Exactly-Once Delivery

The platform guarantees at-least-once delivery with deduplication, not true exactly-once.

True exactly-once delivery to a user requires the provider to guarantee it — and no email, SMS, or push provider offers exactly-once. FCM may deliver a push notification twice if there is a network acknowledgment failure. The correct approach is deduplication at every layer to make duplicates extremely unlikely, while accepting they are not impossible.

For critical notifications (password reset, OTP), callers should include a deduplication token in the notification content ("Your OTP: 123456, valid for 5 minutes") so that even if a duplicate is delivered, the user understands the context.

Common Interview Mistakes

Mistake	Why It Matters
Designing a single "notification table" without channel-specific delivery records	Cannot independently track retry state per channel; status queries become complex joins
Ignoring user preferences and quiet hours	Notification platforms are regulated in many jurisdictions; opt-out compliance is not optional
Using HTTP callbacks from workers instead of Kafka for delivery events	HTTP is synchronous and fragile; Kafka decouples outcome processing from delivery
Not separating batch broadcast from real-time delivery	A 10M-user broadcast should never block a transactional alert; they are different workloads on different paths
Designing retry in the API layer instead of the worker layer	The API must return fast; retries happen asynchronously in workers
Forgetting stale device token cleanup	Keeping stale push tokens wastes throughput, skews metrics, and can cause provider rate limiting
Missing the transactional outbox pattern for Kafka	Without it, a crash between DB commit and Kafka produce causes silent notification loss

Summary

The enterprise notification platform is an asynchronous, multi-channel delivery system built around three core principles:

Decouple acceptance from delivery — the API returns immediately; workers handle the complexity of provider communication, retries, and failures asynchronously via Kafka
Respect user intent at every layer — preferences are checked at enqueue time and re-checked at delivery time; quiet hours and opt-outs are never bypassed except for critical notifications
Design for provider failure — external providers are unreliable; circuit breakers, fallback chains, retry with backoff, and dead letter queues ensure notifications reach users even when a provider is degraded

flowchart TB
    Core["Enterprise Notification Platform"]

    Core --> Accept["Accept fast\nAPI validates + enqueues\n< 50ms response"]
    Core --> Route["Route smart\nPreferences + priority\n+ channel availability"]
    Core --> Deliver["Deliver reliably\nRetry + fallback\n+ deduplication"]
    Core --> Observe["Observe completely\nPer-channel metrics\n+ distributed tracing"]
    Core --> Scale["Scale independently\nPer-channel workers\n+ auto-scale on lag"]