Message Queues & Streams: decoupling, at-least-once vs exactly-once, ordering, DLQs

Message queues let services communicate asynchronously, decoupling producers from consumers. Choose the wrong queue, and you'll lose data or lose ordering guarantees.

What a Queue Does

Synchronous (bad at scale):

POST /checkout
  → OrderService creates order
  → PaymentService charges card (wait)
  → NotificationService sends email (wait)
  → return response

Latency: sum of all services. If payment takes 500ms, user waits 500ms+.
Single point of failure: if PaymentService is down, whole checkout fails.

Asynchronous with Queue (better):

POST /checkout
  → OrderService creates order
  → Publish "order.created" event to queue
  → return response immediately

PaymentService consumes "order.created":
  → charges card (async, separate)

NotificationService consumes "order.created":
  → sends email (async, separate)

Latency: only OrderService. Others catch up asynchronously.
Fault isolation: if PaymentService is down, order still created; payment retries later.

---

At-Least-Once vs Exactly-Once Delivery

At-Least-Once

How it works:

1. Consumer reads message.
2. Consumer processes.
3. Consumer acknowledges ("ack") to queue.
4. If consumer crashes before ack, message re-delivered (to another consumer or retry).

Pros: Simple, never loses messages.

Cons: Duplicate delivery is possible (if consumer crashes after processing but before ack).

Code:

Message msg = queue.consume();
try {
  processOrder(msg);
  queue.ack(msg); // Tell queue: processed successfully
} catch (Exception e) {
  // Don't ack. Message re-delivered.
  throw e;
}

Reality: Most applications accept duplicates and handle them via idempotency keys.

Exactly-Once

How it works:

1. Consumer reads message and gets a unique ID.
2. Consumer processes AND writes a dedup record (transaction).
3. If consumer crashes and retries, dedup table shows: already processed.
4. Consumer skips reprocessing.

Pros: No duplicates.

Cons: Expensive (requires distributed transactions), slow.

Code:

Message msg = queue.consume();
String messageId = msg.getId();

// Check if already processed
if (dedupTable.contains(messageId)) {
  queue.ack(msg);
  return; // Skip reprocessing
}

try {
  processOrder(msg);
  dedupTable.insert(messageId); // Atomic with processing
  queue.ack(msg);
} catch (Exception e) {
  throw e;
}

Honest take: "Exactly-once" is a myth in distributed systems. Kafka's "exactly-once" requires special transaction setup. Most teams do at-least-once + idempotency.

---

Ordering Guarantees

No Ordering

Message 1: order created
Message 2: order paid
Message 3: order shipped

May be consumed as: 2, 1, 3 (out of order).
If NotificationService processes "shipped" before "paid," email is wrong.

Per-Key Ordering

Kafka partition by order_id:
  order_id 123 → partition 0 (all messages for this order in order)
  order_id 456 → partition 1

Created, Paid, Shipped for order 123 arrive in order. But order 456 and 123 are independent.

Key: Always have enough partitions (1 per expected key * replication factor). Bottleneck if partitions < 10.

Global Ordering

All messages across all keys in order.
Message 1 (any key) before Message 2 (any key).

Very expensive (single partition = no parallelism).
Rarely needed.

Practical rule: Use per-key ordering. Global is overkill.

---

Dead Letter Queues (DLQ)

Messages that fail repeatedly go to a DLQ for manual inspection.

Main Queue: consume, process
If failed 5 times:
  → DLQ: inspect later (manual handling)

Consumer pulls from DLQ manually:
  → Check logs, fix bug
  → Reprocess or discard

Why it matters: Without DLQ, bad messages are lost or block the queue forever.

---

RabbitMQ vs Kafka

| Feature | RabbitMQ | Kafka | |---------|----------|-------| | Throughput | ~50k msgs/sec | millions/sec | | Ordering | Per-queue | Per-partition | | Persistence | Disk (durable) | Log-based (replay) | | Consumer model | Pull or push | Pull | | Ideal for | Low-throughput tasks | High-volume events | | Operational complexity | Simple | Complex (ZooKeeper, etc) |

RabbitMQ: Task queues (email, image processing). Simpler, works great under 100k msgs/sec.

Kafka: Event streaming (metrics, logs, analytics). Designed for millions/sec, built-in replication, topic replay.

---

In the interview

1. Name the problem queue solves: "Decoupling: producers don't wait for consumers. Fault isolation: consumer down doesn't block producer."

2. Discuss delivery semantics: "At-least-once is simpler. Duplicates are OK if idempotency keys protect (dedup on key). Exactly-once is rare and expensive."

3. Mention ordering: "Per-key ordering via partitions. Global ordering is overkill unless necessary."

4. DLQ: "Failed messages go to DLQ. Manual inspection prevents silent failures."

5. Pick the tool: "RabbitMQ for low-throughput tasks, Kafka for high-volume events."