Delivery semantics
When one process sends a message to another over an unreliable network, there are exactly three contracts it can offer: the message is delivered at most once, at least once, or — the one everyone wants and nobody can buy off the shelf — exactly once. The first two are achievable with simple mechanics. The third is impossible as a pure delivery guarantee, and the systems that advertise it are quietly doing something cleverer: at-least-once delivery plus deduplication or idempotent processing.
The ack problem
Everything here follows from one fact: when a sender does not receive an acknowledgement, it cannot tell whether the message was lost or the ack was lost. The receiver may have processed the message perfectly and the confirmation simply vanished.
This forces a choice. If the sender retries on a missing ack, it risks delivering the message twice (the receiver already had it). If the sender does not retry, it risks never delivering at all. There is no third option that is both safe and free — that is the whole subject in one sentence.
At-most-once
The sender fires the message and never retries. Each message is delivered zero or one times, never more. Simple, low-latency, and lossy: any dropped packet is gone for good.
Use it where loss is cheaper than duplication and the data is naturally refreshed: UDP metrics, real-time telemetry, a video frame, a "user is typing" indicator. Losing one is invisible; processing one twice would be worse or pointless.
At-least-once
The sender retries until it gets an ack. Every message is delivered one or more times, never zero. This is the default for every durable queue and log — Kafka, SQS, RabbitMQ, Pub/Sub — because losing data is almost always worse than seeing it twice.
The cost is duplicates. A consumer that increments a counter, charges a card, or sends an email on every message will, eventually, do it twice. At-least-once delivery is only safe if the processing is idempotent, which is why these two topics are inseparable.
Exactly-once — the asterisk
Exactly-once delivery — every message reaching the receiver precisely once over a network that can lose packets and crash either end — is impossible. The Two Generals problem proves you cannot even reliably agree that a single message was received. What is achievable is exactly-once processing, also called effectively-once: the message may be delivered many times, but its effect happens once.
There are two routes to it, often combined:
- Deduplication. Tag each message with a unique ID. The receiver records processed IDs and ignores repeats. Bounded by a dedup window (you cannot remember every ID forever), so it is exactly-once within a time horizon.
- Idempotent processing. Make the side effect safe to repeat — an upsert keyed on the message ID, a conditional write, a fencing-protected operation. Then duplicates are harmless by construction.
How Kafka does "exactly-once"
Kafka's exactly-once semantics (EOS) is the canonical real-world example, and it works precisely because it never claims exactly-once delivery. It composes three pieces:
- Idempotent producer. Each producer gets a producer ID and a per-partition sequence number. The broker rejects a record whose sequence it has already written, so producer-side retries never create duplicates in the log.
- Transactions. A producer can write to multiple partitions and commit its consumer offsets in a single atomic transaction. Either all the output and the offset advance commit, or none do.
- Read-committed consumers. Downstream readers skip records from aborted transactions, so they never observe partial work.
The result is exactly-once for the read-process-write pattern inside Kafka. The moment your effect leaves Kafka — a card charge, an email, a row in an external database — you are back to needing idempotency or the outbox pattern, because Kafka cannot make a third-party API exactly-once.
Choosing a contract
| Semantics | Delivered | Use when |
|---|---|---|
| At-most-once | 0 or 1 | Loss is cheap, duplicates are harmful or pointless (telemetry, presence) |
| At-least-once | 1 or more | Loss is unacceptable and the consumer is (or can be made) idempotent |
| Exactly-once (effective) | processed once | Loss and duplication both unacceptable — pay for dedup + idempotency |
The pragmatic answer for most systems: choose at-least-once delivery and make the consumer idempotent. That combination behaves like exactly-once at the application boundary, which is the only place the guarantee actually matters.
Further reading
- Confluent — Exactly-once semantics in Apache Kafka — the design write-up of the idempotent producer and transactions.
- Helland — Life Beyond Distributed Transactions — the case for entities, idempotence, and at-least-once messaging over distributed commit.
- The Two Generals Problem — the impossibility result underneath exactly-once delivery.
- Kleppmann — DDIA, ch. 11 — stream processing, delivery guarantees, and idempotence in practice.