Designing a scalable notification system sounds straightforward until - A 10M marketing blast delaying a single login OTP by 4 minutes will kill your checkout conversion faster than a database outage.

The naive architecture of dumping every notification into a shared message broker with a priority header guarantees head-of-line blocking the second a surge hits.

Here is how I design a notification engine to enforce a strict <5s p99 OTP SLA at 16M+ daily events without dropping a single packet:

1. The Ingestion & Isolation Path: Upstream Service ➔ Stateless Gateway ➔ Redis Dedup Shield (atomic SETNX) ➔ HTTP 202 Accepted ➔ Physical Kafka Priority Topics

2. The Smart Core & Egress Path: Dedicated Worker Pool ➔ Smart Core (Quiet Hours + Redis Debounce + Hydration) ➔ Distributed Token Bucket Limiter ➔ Vendor APIs (APNs / Twilio / SendGrid)

When scaling this to production, watch out for these 3 traps:

1. The Priority Flag Illusion Relying on priority headers inside a shared Kafka topic fails because consumer threads spend all their cycles de-queueing bulk marketing payloads while urgent OTPs sit starved in partition lag. We solve this with physical isolation: dedicated, high-priority Kafka topics consumed by autonomous worker pools provisioned for zero queuing, keeping OTP p99 delivery latency under 3s.

2. The Vendor Throttling Meltdown Calling external provider APIs (Twilio, SendGrid, APNs) synchronously inside your consumer workers will exhaust thread pools the millisecond an upstream vendor experiences a latency spike or throws HTTP 429s. You must decouple egress using distributed Token Bucket rate limiters in Redis and enforce message custody in Kafka before external invocation, allowing bounded exponential backoff instead of silent drops.

3. The "Double-Tap" Resend Storm Querying a relational database to verify if an alert was already dispatched collapses under load when anxious users tap "Resend Code" 6 times in 5 seconds during an outage. We drop a Redis Deduplication Shield directly at the API Gateway using atomic SETNX with a sliding 60-second TTL on userId:eventType, returning an immediate cached HTTP 202 without
   generating downstream broker traffic.

I broke down the complete architecture - including the sub-5s OTP failure latency budget, the Two-Stage Kafka topic pipeline, and how chronological Kafka delay topics replace database retry polling - in my full system design write-up here:
https://engineering.by-tech.workers.dev/articles/scalable-notification-system-design/

The Golden Rule for Notification Egress: If an authentication OTP doesn't arrive in under 5 seconds, the user assumes your entire platform is down. If a promotional discount notification arrives 30 minutes late, nobody notices.

Never let bulk traffic share the same connection pool, Kafka partition, or egress rate limiter as auth traffic.

SystemDesign #DistributedSystems #BackendEngineering #Kafka #Redis