Message Queue
An asynchronous buffer between services: producers enqueue work, consumers process at their own pace — decoupling that turns traffic spikes into backlogs instead of outages.
What is a message queue?
A message queue (SQS, RabbitMQ, or the log-structured Kafka family) lets services communicate without being simultaneously available: producers append messages and move on; consumers pull and process at their own rate. That temporal decoupling changes failure behavior fundamentally — a payment-processor slowdown becomes a growing queue and a delay, rather than cascading timeouts through every upstream caller. Spikes become backlogs; backlogs drain.
The contract you must design for
Practical queues deliver at-least-once: after a crash between processing and acknowledgment, the message returns. Duplicates are therefore not an edge case but a specification — every consumer must be idempotent (dedupe keys, upserts, “already processed” checks), or you will double-charge someone during your first bad deploy. The second obligatory pattern is the dead-letter queue: messages failing N attempts route aside for inspection instead of poisoning the queue in an infinite retry loop. A queue without a DLQ is an outage scheduled for later.
Queue or log? The fork that matters
Classic queues (SQS/RabbitMQ) delete on consumption — one message, one processing, work-distribution semantics. Log systems (Kafka) retain messages; consumers track positions, multiple independent consumers replay the same stream. Choose queues for task distribution, logs for event streams that several systems consume or that need replay. Teams that pick Kafka “for scale” and use it as a task queue inherit its operational weight for none of its benefits.
What people get wrong
- Assuming ordering — standard queues don’t guarantee it; FIFO variants do, at throughput cost per ordering key.
- No backpressure story: if producers permanently outpace consumers, the queue is a slow-motion failure; autoscale consumers on queue depth and alert on age-of-oldest-message.
- Payloads as blobs — unversioned message schemas make every producer deploy a potential consumer-crash event.
Primary source: AWS SQS documentation — best practices
Historical figures and technical concepts for informational purposes only. Not technical, professional, legal, or financial advice. Sources: Official Documentation.