Somewhere in every message-driven architecture sits a consumer that charges a credit card, and somewhere in every incident history is the day it charged one twice. The root cause is never exotic — it’s the same two-line proof, rediscovered: the consumer processed the message and crashed before the ack left. The broker, seeing no ack, must choose: redeliver (duplicate) or don’t (loss). There is no third option — that window is physics, not vendor negligence — and every “delivery guarantee” in every queue’s marketing is a policy about which side of it to fall on.

The Menu, Honestly Labeled

At-most-once — ack/commit before processing. Crash mid-work and the message is gone. Correct choice for data where stale beats duplicate (metrics, telemetry, cache invalidation hints), and absolutely nowhere money lives.

At-least-once — ack after processing. Crash and the broker redelivers; nothing is lost, duplicates are now your application’s problem. This is the default posture of SQS, RabbitMQ, Kafka consumers, and reality.

Exactly-once — the phrase means something real, but read the fine print: it’s exactly-once processing within a closed system, not delivery. Kafka’s version is honest engineering: idempotent producers (sequence numbers dedupe broker-side retries) + transactions that atomically commit output messages and consumer offsets — so a consume-transform-produce pipeline that lives entirely inside Kafka replays without visible duplicates. The moment your consumer touches anything outside (an API call, an email, a database without transactional tie-in), the ack-crash window reopens at that boundary. No queue can promise your side effect and its ack are atomic.

Making Duplicates Boring: Idempotency

Since redelivery is inevitable, the mature design goal is duplicates that don’t matter:

  • Natural idempotency where possible: SET balance = 700 replays harmlessly; ADD 100 doesn’t. Upserts keyed on business identity beat inserts.
  • Idempotency keys everywhere else: a unique ID per logical operation (message ID, or better, producer-assigned business key), checked-and-recorded in the same transaction as the effect:
sql
BEGIN;
INSERT INTO processed(msg_id) VALUES ($1);  -- unique index: dup ⇒ abort
UPDATE accounts SET balance = balance - $2 WHERE id = $3;
COMMIT;

The transaction is the whole trick — check-then-act without it just narrows the window. (Stripe-style APIs push the same pattern across the network: the Idempotency-Key header is at-least-once made safe at an HTTP boundary.)

  • The outbox pattern for the producing side’s mirror-image problem (“update the DB and publish an event, atomically”): write the event into an outbox table in the business transaction; a relay (or CDC/Debezium tailing the WAL) publishes it. Dual-write without the dual-write bug.

Ordering deserves its own honesty check: queues promise order only per-partition/per-FIFO-group, retries and DLQ detours reorder regardless, and consumers scale by partition count — so encode causality in the data (versions, sequence numbers per entity) rather than trusting arrival order, and pick partition keys (entity ID) so the ordering you get is the ordering you need. And redelivery needs a terminus: max-receive-count into a dead-letter queue, with the two operational rules people learn expensively — alert on DLQ depth (a quiet DLQ is data loss on a timer), and mind visibility-timeout vs processing-time (work that takes 31 s against a 30 s timeout is a duplicate generator, the “poison pill that isn’t” classic).

Takeaways

  • The ack-crash window makes loss-or-duplicate a forced choice; at-least-once + idempotent consumers is the industry’s answer.
  • “Exactly-once” = exactly-once processing inside one system’s transaction boundary (Kafka txns); external side effects reopen the window at the edge.
  • Idempotency keys recorded in the same transaction as the effect; outbox/CDC for atomic write-and-publish.
  • Order is per-partition at best and survives neither retries nor DLQs — put sequence/causality in the payload, choose partition keys deliberately.
  • DLQs with alerts, visibility timeouts sized above real processing time — the boring settings that prevent the exciting incidents.