Message Queue
A durable buffer from which consumers claim messages, allowing producers and workers to operate at different times and rates.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/message-queue/
01What it means
A message queue stores units of work or communication until a consumer receives and processes them. Producers enqueue messages without waiting for the work to finish; one of one or more competing consumers typically handles each message and acknowledges completion. Delivery and ordering guarantees vary by queue.[1][2]
- 01ProducerEnqueues work
- 02QueueStores until eligible
- 03ConsumerClaims and processes
- 04AcknowledgeConfirms completion
- 05Retry or failureHandles unsuccessful work
Acknowledgement, retry, ordering, and durability depend on delivery policy.
02Why it matters
Queues can keep slow or bursty work out of user-facing requests, absorb temporary demand spikes, and allow workers to scale independently. They also make backlog age, duplicate delivery, poison messages, and eventual completion part of the product's behavior.
03When to use it
Use a queue when work can complete asynchronously, producers and consumers need temporal decoupling, or demand must be buffered. Avoid it when the caller requires an immediate result, the operation is trivial, or the team cannot operate retries, idempotency, monitoring, and recovery.
04How it works
- A producer writes a message with the minimum data and identifiers needed to perform the work.
- The queue stores the message durably and exposes it to an eligible consumer.
- A consumer claims the message, processes it, and acknowledges successful completion.
- Unacknowledged work becomes visible again or is retried according to delivery policy.
- Repeatedly failing messages move to a failure destination for investigation and recovery.
- Operators track backlog size, oldest-message age, processing time, failures, and business completion.
05A practical example
An image service places each uploaded file on a resize queue and responds immediately with a processing state. Workers create variants at a controlled rate; duplicate messages are safe, and jobs that repeatedly fail move to a dead-letter queue for review.
06Where it breaks
Queues add infrastructure, latency, and failure states. Common systems provide at-least-once delivery, so duplicates and reordering are normal possibilities. A growing backlog can hide overload, and accepting a message does not mean the business operation completed.
Common mistakes
- Assuming a queued message will be processed exactly once or in global order.
- Acknowledging before durable business work finishes and losing failed work.
- Sending large, unstable internal objects instead of a clear message contract.
- Monitoring queue availability but not backlog age or end-to-end completion.
07Sources
- What Is Amazon Simple Queue Service?Amazon Web Services · Official technical documentationWhat Is Amazon Simple Queue Service?
- Queue-Based Load Leveling PatternMicrosoft Learn · Institutional architecture guideQueue-Based Load Leveling Pattern