System Design Cases
Kafka vs RabbitMQ vs Pulsar
Comparison of three messaging brokers side-by-side: Kafka (log-based, partitioned, dumb broker + smart consumer with offset tracking), RabbitMQ (smart broker with topic exchange routing to multiple queues, push delivery, DLX for rejects), and Pulsar (stateless broker + Apache BookKeeper segmented storage with E=3/W=2/A=2 quorum, tiered S3 offload for cold segments, multiple subscription types). Includes 4 scenarios: Kafka log fan-out via consumer groups with replay, RabbitMQ topic exchange routing with DLX, Pulsar segmented storage and broker failover, and ADR decision matrix walkthrough.
Kafka, RabbitMQ and Pulsar: logs, queues, subscriptions and acknowledgements
Нельзя сравнивать brokers одной строкой «Kafka = streaming, RabbitMQ = queue, Pulsar = оба». Нужно сопоставлять exact workload contract: routing, replay, consumer coordination, storage, ack и failure recovery.
Корректная модель
- Kafka order is per partition; retention is independent of whether a particular consumer read a record.
- RabbitMQ publisher confirms and consumer acknowledgements are orthogonal.
- Pulsar brokers use BookKeeper for persistent storage and named subscriptions keep cursors.
- acks=all, durable queue or persistent topic are configurations within a larger disk/quorum/application contract, not fsync magic by name.
Границы и компоненты
| Компонент | Ответственность |
|---|---|
| Publisher | Выбирает message key/id и ждёт documented broker acknowledgement. |
| Kafka Partitioned Log | Хранит ordered offsets per partition и consumer-group positions. |
| Kafka Group Consumer | Подключается к broker, обрабатывает assigned partitions и commits offsets. |
| RabbitMQ Exchange and Queue | Exchange routes; queue delivers with manual/auto acknowledgement policy. |
| RabbitMQ Consumer | Подключается к queue, ACK/NACK и готов к redelivery. |
| Pulsar Broker and Topic | Обслуживает producers/consumers и subscription cursors. |
| BookKeeper Storage | Durable ledger storage behind Pulsar brokers. |
| Pulsar Subscription Consumer | Подключается к named subscription with exclusive/shared/failover/key_shared semantics. |
Сценарии
Kafka partition replay
Records are ordered within each partition. A group assigns a partition to one consumer at a time; processing then offset commit yields at-least-once on crash unless a scoped transaction/idempotent sink closes the boundary.
Проверяемый исход: Retention and auto.offset.reset are configured; auto.offset.reset is only used when no valid committed offset exists.
RabbitMQ routing and acknowledgements
Publisher confirms cover publisher-to-broker acceptance; consumer acknowledgements independently cover delivery processing. Exchanges route into queues; unacked messages can be requeued on connection loss.
Проверяемый исход: Consumer handles redelivery idempotently and caps poison-message loops with dead-letter/backoff policy.
Pulsar subscription and storage separation
A broker appends to BookKeeper and tracks a named subscription cursor. Subscription type determines distribution/ordering behavior; no invented E/W/A quorum formula is treated as a universal application guarantee.
Проверяемый исход: A broker failover can reconnect clients to durable ledgers; consumers still handle redelivery and cursor semantics.
Outage and workload-driven choice
All three paths can redeliver or expose ambiguous publish outcomes depending on where failure happens. The application needs stable message IDs, bounded retries and reconciliation.
Проверяемый исход: The choice table records workload/failure assumptions, not universal latency or durability rankings.
Failure, concurrency и replay checklist
- Stable message ID plus idempotent side effect at every consumer.
- Bound redelivery and poison loops with retry count/backoff/dead-letter policy.
- Monitor lag/backlog, unavailable partitions/queues, disk and acknowledgement latency.
- Run broker/node/network fault tests using the actual replication and durability configuration.
Формулы, units и допущения
- Backlog drain time = backlog messages / (sustained consume rate − sustained produce rate), only when consume rate is greater.
- Kafka parallelism is bounded by partitions per consumer group; Rabbit/Pulsar queue-like parallelism has ordering trade-offs.
- Retention bytes ≈ ingress bytes/s × retention seconds × replication/ledger overhead, then add indexes/segments/headroom.
Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.
Связанные темы
[CONCEPT]exactly-once-semantics
Первичные источники
- https://kafka.apache.org/41/design/design/
- https://kafka.apache.org/41/configuration/consumer-configs/
- https://www.rabbitmq.com/docs/confirms
- https://www.rabbitmq.com/tutorials/amqp-concepts
- https://pulsar.apache.org/docs/4.2.x/concepts-messaging/
- https://pulsar.apache.org/docs/next/concepts-architecture-overview/
Scope note
Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.