System Design Cases
Message Queues vs Task Queues
Очереди и брокеры: work queue vs pub/sub, back-pressure, redelivery, at-least-once. Kafka vs RabbitMQ vs SQS.
Queues, topics and delivery contracts
A broker separates arrival from processing, but it does not remove failures. Define the durability boundary, acknowledgement point, retry policy, ordering scope and duplicate behavior before choosing a product.
Consumers initiate and maintain their connection or poll to the broker. The diagram therefore connects each consumer to the queue or topic; animation may travel in reverse when the broker delivers a record.
Work queue
A work queue distributes deliveries among active consumers. “One worker receives it” is not “exactly once”: if the worker or connection fails before acknowledgement, the broker can redeliver to another worker. Acknowledge only after the durable effect. The effect transaction uses a unique event/idempotency key, so repeated delivery does not repeat money movement, email intent, inventory decrement, or another side effect.
Visibility timeout is an Amazon SQS mechanism. RabbitMQ uses unacknowledged deliveries and channel/connection recovery; Kafka tracks offsets and group assignments. Treat these as product-specific implementations of a delivery contract, not interchangeable vocabulary.
Publish/subscribe
Each independent subscription or consumer group receives its own logical copy. Multiple workers inside one group divide that group's partitions or deliveries; every process is not guaranteed to see every event. Ordering is normally scoped to a queue, stream partition, or key, not the whole system.
Producer atomicity
A database commit followed by a separate publish can lose an event; publishing first can create an event for a rolled-back transaction. A transactional outbox stores business state and event intent together. The relay publishes with a stable event id and marks progress. A crash between broker confirmation and that mark creates a duplicate, which consumers must tolerate.
Retry and dead letters
Retry only transient classes and retain the same event identity. Use bounded exponential backoff with jitter and a maximum attempt or age budget. Immediate requeue can form a hot loop. Permanent validation errors and exhausted messages move to a dead-letter/quarantine path with enough metadata to diagnose, replay safely, or discard under an audited policy.
A DLQ is not completion. Alert on it, control access to sensitive payloads, define retention, and make replay idempotent.
Backpressure
A finite queue absorbs a finite mismatch between arrival rate and service rate. If arrival stays above departure, backlog and delay grow until retention/storage limits fail. Bound publisher admission, message size, consumer prefetch/in-flight count and concurrency. Monitor oldest-message age, not only record count. Apply load shedding or degrade producers before the broker becomes the next outage.
Little's Law relates steady-state average items, effective arrival rate and average time in the system: L = λW. It is useful for checking measurements, not a promise that a burst will be safe.
Scenarios
Atomic business/outbox commit followed by confirmable, repeatable publication.
Lost acknowledgement, redelivery and an idempotent durable effect.
Two independent subscriptions receive one topic record.
Classified, delayed retry followed by quarantine after a bounded budget.
Finite buffering, consumer prefetch and backlog-age monitoring.
“Exactly once” scope
Some broker/database combinations can atomically consume, transform and publish inside a defined transaction boundary. Apache Kafka documents exactly-once processing for Kafka read-process-write workflows when the appropriate transactional APIs are used. That does not automatically include an arbitrary external API, email provider, filesystem or separate database. End-to-end correctness still needs an idempotent endpoint, a shared transaction, or reconciliation.
Primary sources
- RabbitMQ consumer acknowledgements and publisher confirms: https://www.rabbitmq.com/docs/confirms
- RabbitMQ reliability guide: https://www.rabbitmq.com/docs/reliability
- Amazon SQS visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- Amazon SQS standard queues and at-least-once delivery: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html
- Apache Kafka design and delivery semantics: https://kafka.apache.org/41/design/design/
- Little, A Proof for the Queuing Formula L = λW: https://pubsonline.informs.org/doi/10.1287/opre.9.3.383