The Problem: Tight Coupling
Without a queue, Service A must call Service B directly and wait for a response. If Service B is slow, Service A is blocked. If Service B is down, Service A fails. If Service B cannot keep up with the rate of requests, both services degrade.
A message queue breaks this dependency: Service A writes a message to the queue and moves on. Service B reads from the queue at its own pace.
Core Concepts
Producer
The component that creates and publishes messages to the queue. Producers do not know or care which consumer will process the message.
Consumer
The component that reads and processes messages from the queue. Consumers can be scaled horizontally: add more consumers to process messages faster.
Message
A unit of data placed on the queue. Contains a payload (the data to process) and metadata (timestamp, priority, headers). Messages are typically serialized as JSON, Protobuf, or Avro.
Queue / Topic
The channel that holds messages between producer and consumer. A queue delivers each message to exactly one consumer (point-to-point). A topic delivers each message to all subscribers (publish-subscribe).
Messaging Patterns
Point-to-Point (Work Queue)
Each message is processed by exactly one consumer. Multiple consumers can compete for messages: this is called a competing consumers pattern. Used for task distribution: email sending, image processing, order fulfillment.
Publish-Subscribe (Pub/Sub)
A message is broadcast to all subscribers of a topic. Each subscriber gets its own copy. Used for event notification: when a user signs up, notify the email service, analytics service, and recommendation service simultaneously.
Delivery Guarantees
| Guarantee | Meaning | Trade-off |
|---|---|---|
| At-most-once | Message is delivered zero or one times. Might be lost. | Simplest, fastest. No deduplication needed. |
| At-least-once | Message is delivered one or more times. Might be duplicated. | Consumer must be idempotent (handle duplicates). |
| Exactly-once | Message is delivered exactly one time. | Hard to achieve. Requires coordination between queue and consumer. |
Backpressure
When producers publish faster than consumers can process, the queue grows. If unchecked, the queue can exhaust memory or disk. Backpressure mechanisms prevent this:
- Queue size limits: Reject or block producer when the queue is full.
- Rate limiting on producers: Throttle how fast producers can publish.
- Consumer autoscaling: Spin up more consumers as queue depth increases.
- Dead-letter queues (DLQ): Messages that fail processing N times are moved to a separate queue for investigation, preventing them from blocking the main queue.
Dead-Letter Queues
A DLQ captures messages that cannot be processed successfully after a configured number of retries. Instead of losing the message or retrying forever, it is moved to the DLQ where engineers can inspect it, fix the issue, and reprocess.
Popular Message Queue Systems
| System | Model | Best For |
|---|---|---|
| Apache Kafka | Distributed log (pub/sub + replay) | High-throughput event streaming, log aggregation, real-time pipelines |
| RabbitMQ | Traditional message broker (AMQP) | Task queues, complex routing, moderate throughput |
| Amazon SQS | Managed queue (point-to-point) | Simple, serverless task processing on AWS |
| Amazon SNS | Managed pub/sub | Fan-out event notifications on AWS |
| Apache Pulsar | Multi-tenant pub/sub + queue | Combines Kafka's streaming with traditional queuing |
| Redis Streams | In-memory stream | Lightweight streaming when Redis is already in the stack |
Event-Driven Architecture
Message queues are the backbone of event-driven architecture (EDA). In EDA, services communicate by publishing and subscribing to events rather than calling each other directly. Benefits:
- Services are loosely coupled: they can be developed, deployed, and scaled independently.
- New consumers can be added without modifying existing producers.
- Events create a natural audit log of everything that happened in the system.
- Complex workflows can be orchestrated by chaining events.
Key Takeaways
- Message queues decouple producers and consumers, enabling independent scaling and fault isolation.
- Use point-to-point for task distribution; use pub/sub for event broadcasting.
- Design consumers to be idempotent: at-least-once delivery is the practical standard.
- Always configure dead-letter queues for failed messages.
- Monitor queue depth. Growing queues indicate consumers are not keeping up.
- Kafka is the go-to for high-throughput event streaming. RabbitMQ for complex routing and moderate loads. SQS/SNS for serverless AWS architectures.