Kafka, RabbitMQ and Pulsar: logs, queues, subscriptions and acknowledgements

Нельзя сравнивать brokers одной строкой «Kafka = streaming, RabbitMQ = queue, Pulsar = оба». Нужно сопоставлять exact workload contract: routing, replay, consumer coordination, storage, ack и failure recovery.

Корректная модель

• Kafka order is per partition; retention is independent of whether a particular consumer read a record.
• RabbitMQ publisher confirms and consumer acknowledgements are orthogonal.
• Pulsar brokers use BookKeeper for persistent storage and named subscriptions keep cursors.
• acks=all, durable queue or persistent topic are configurations within a larger disk/quorum/application contract, not fsync magic by name.

Границы и компоненты

Сценарии

Kafka partition replay

Records are ordered within each partition. A group assigns a partition to one consumer at a time; processing then offset commit yields at-least-once on crash unless a scoped transaction/idempotent sink closes the boundary.

Проверяемый исход: Retention and auto.offset.reset are configured; auto.offset.reset is only used when no valid committed offset exists.

RabbitMQ routing and acknowledgements

Publisher confirms cover publisher-to-broker acceptance; consumer acknowledgements independently cover delivery processing. Exchanges route into queues; unacked messages can be requeued on connection loss.

Проверяемый исход: Consumer handles redelivery idempotently and caps poison-message loops with dead-letter/backoff policy.

Pulsar subscription and storage separation

A broker appends to BookKeeper and tracks a named subscription cursor. Subscription type determines distribution/ordering behavior; no invented E/W/A quorum formula is treated as a universal application guarantee.

Проверяемый исход: A broker failover can reconnect clients to durable ledgers; consumers still handle redelivery and cursor semantics.

Outage and workload-driven choice

All three paths can redeliver or expose ambiguous publish outcomes depending on where failure happens. The application needs stable message IDs, bounded retries and reconciliation.

Проверяемый исход: The choice table records workload/failure assumptions, not universal latency or durability rankings.

Failure, concurrency и replay checklist

• Stable message ID plus idempotent side effect at every consumer.
• Bound redelivery and poison loops with retry count/backoff/dead-letter policy.
• Monitor lag/backlog, unavailable partitions/queues, disk and acknowledgement latency.
• Run broker/node/network fault tests using the actual replication and durability configuration.

Формулы, units и допущения

• Backlog drain time = backlog messages / (sustained consume rate − sustained produce rate), only when consume rate is greater.
• Kafka parallelism is bounded by partitions per consumer group; Rabbit/Pulsar queue-like parallelism has ordering trade-offs.
• Retention bytes ≈ ingress bytes/s × retention seconds × replication/ledger overhead, then add indexes/segments/headroom.

Числа выше — учебные inputs или размерностные формулы. Их нельзя выдавать за benchmark или SLA конкретного продукта.

Связанные темы

[CONCEPT]stream-processing

[CONCEPT]exactly-once-semantics

Первичные источники

• https://kafka.apache.org/41/design/design/
• https://kafka.apache.org/41/configuration/consumer-configs/
• https://www.rabbitmq.com/docs/confirms
• https://www.rabbitmq.com/tutorials/amqp-concepts
• https://pulsar.apache.org/docs/4.2.x/concepts-messaging/
• https://pulsar.apache.org/docs/next/concepts-architecture-overview/

Scope note

Диаграмма показывает причинные границы и recovery contracts, а не скрытую реализацию конкретного managed-сервиса. Любая stronger guarantee действует только в явно названной transaction/checkpoint/acknowledgement boundary.