System Design: Building a Real-Time Notification System at Scale

When I was asked to design a real-time notification engine capable of handling 100,000 concurrent WebSocket connections, my first instinct was to reach for a ready-made solution. But the requirements were specific enough — custom delivery guarantees, multi-tenant isolation, hybrid push/pull — that we ended up building our own. This is what we designed, why, and what I’d do differently.

Requirements First

Before any architecture discussion, I established clear requirements:

Functional:

• Deliver notifications to connected clients within 500ms
• Support multiple notification types: in-app alerts, chat messages, system events
• Persist notifications for offline users (deliver on reconnect)
• Mark notifications as read with acknowledgment

Non-functional:

• Handle 100k concurrent WebSocket connections per cluster
• Horizontally scalable (stateless notification servers)
• At-least-once delivery (no dropped notifications)
• Message ordering within a user’s notification stream

These requirements immediately ruled out some approaches and mandated others.

Choosing the Transport: WebSocket vs SSE vs Polling

I evaluated three transport options:

Long polling was off the table immediately. At 100k connections, even 30-second poll intervals meant roughly 3,300 HTTP requests per second just for heartbeats — before any actual notifications. The connection overhead and server resource usage was too high.

Server-Sent Events (SSE) is excellent for one-way server-to-client updates. It’s simpler than WebSockets, works over HTTP/2, and has automatic reconnection built in. But our system needed bidirectional communication: clients send read receipts and typing indicators. SSE would have required a separate HTTP channel for client-to-server messages, adding complexity.

WebSockets won. True full-duplex communication, efficient binary framing, long-lived connections with low per-message overhead. The main operational consideration is that WebSocket connections are stateful — a client is connected to a specific server instance — which we handled with Redis.

Architecture Overview

The system has three main tiers:

Clients (browser/mobile)
        ↕ WebSocket
┌──────────────────────────────────┐
│   Notification Servers (N nodes) │
│   Node.js + ws library          │
│   In-memory connection registry  │
└───────────┬──────────────────────┘
            │ pub/sub
┌───────────▼──────────────────────┐
│   Redis Cluster                  │
│   Pub/Sub channels per user      │
│   Sorted sets for message queues │
└───────────┬──────────────────────┘
            │
┌───────────▼──────────────────────┐
│   PostgreSQL                     │
│   Notification records           │
│   Delivery receipts              │
└──────────────────────────────────┘

Each notification server maintains an in-memory map of userId → WebSocket connection. When a notification needs to be delivered, it’s published to a Redis channel. Every server subscribes to every channel; the server that has that user connected delivers the message locally. Servers that don’t have the user connected simply ignore it.

The Redis Pub/Sub Layer

Redis pub/sub is the backbone of horizontal scaling. Without it, a notification generated by Server A couldn’t reach a user connected to Server B.

// publisher (called when a notification event occurs)
async function publishNotification(notification: Notification): Promise<void> {
  const channel = `user:${notification.userId}:notifications`;
  await redis.publish(channel, JSON.stringify(notification));

  // Also persist to PostgreSQL for offline delivery
  await notificationRepo.save(notification);
}

// subscriber (runs on every notification server)
const subscriber = redis.duplicate();
await subscriber.pSubscribe('user:*:notifications', (message, channel) => {
  const notification = JSON.parse(message) as Notification;
  const ws = connectionRegistry.get(notification.userId);

  if (ws && ws.readyState === WebSocket.OPEN) {
    ws.send(JSON.stringify({
      type: 'notification',
      payload: notification,
    }));
  }
});

For user presence and connection tracking, I use Redis hashes:

// On WebSocket connect
await redis.hset(`connections`, userId, serverId);

// On WebSocket disconnect
await redis.hdel(`connections`, userId);

Message Persistence and Offline Delivery

Pub/sub is fire-and-forget. If a user is offline when a notification is published, they miss it. That’s fine for some use cases (live cursor positions) but not for ours (unread message counts, critical alerts).

We solve this with PostgreSQL. Every notification is persisted before being published:

CREATE TABLE notifications (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     UUID NOT NULL,
  type        TEXT NOT NULL,
  payload     JSONB NOT NULL,
  read_at     TIMESTAMPTZ,
  delivered   BOOLEAN DEFAULT false,
  created_at  TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_notifications_user_unread
  ON notifications (user_id, created_at DESC)
  WHERE read_at IS NULL;

On WebSocket connect, the client sends its last-seen timestamp, and the server fetches any missed notifications:

ws.on('message', async (data) => {
  const msg = JSON.parse(data.toString());

  if (msg.type === 'sync') {
    const missed = await notificationRepo.fetchSince(userId, msg.since);
    ws.send(JSON.stringify({ type: 'sync_response', notifications: missed }));
  }

  if (msg.type === 'ack') {
    await notificationRepo.markDelivered(msg.notificationIds);
  }
});

Delivery Guarantees

We implemented at-least-once delivery with client-side deduplication:

1. Server assigns each notification a UUID before publishing
2. Client acknowledges receipt with the UUID
3. Server marks delivered = true in PostgreSQL
4. On reconnect, client fetches undelivered notifications by UUID
5. Client ignores duplicates by checking a local seen-set

This is simpler than exactly-once delivery (which requires distributed transactions) and sufficient for our use case.

Connection Management at Scale

Each Node.js process can handle roughly 10,000-15,000 WebSocket connections comfortably before memory and event loop pressure builds up. At 100k connections, we run 8-10 server instances behind a load balancer.

The load balancer must use sticky sessions (or hash by user ID) to ensure a reconnecting client hits the same server if possible — though since we recover from Redis, any server can handle reconnections.

We set connection limits and implement backpressure:

const wss = new WebSocketServer({
  maxPayload: 64 * 1024, // 64KB max message size
  perMessageDeflate: {
    zlibDeflateOptions: { level: 1 }, // fast compression
  },
});

// Heartbeat to detect dead connections
const HEARTBEAT_INTERVAL = 30_000;
setInterval(() => {
  wss.clients.forEach((ws: ExtWebSocket) => {
    if (!ws.isAlive) return ws.terminate();
    ws.isAlive = false;
    ws.ping();
  });
}, HEARTBEAT_INTERVAL);

Monitoring

For a real-time system, monitoring is critical. Key metrics I track:

• Active WebSocket connections per server (Prometheus gauge)
• Message delivery latency — time from publish to client ack (histogram)
• Undelivered notification queue depth per user (alerts if > threshold)
• Redis pub/sub subscriber lag
• PostgreSQL notification table growth

The most useful alert: if a user’s undelivered count exceeds 100, something is wrong with their connection or our delivery pipeline.

Building this system taught me that real-time is as much about recovery and persistence as it is about speed. The “real-time” part is easy; the “reliable” part is where the engineering lives.