Deduplication and idempotency both reduce repeated effects, but they operate at different boundaries. Deduplication stops a message that has already been observed, while idempotency ensures repeating an operation produces the same state. Depending on only one can leave a gap during retry or replay.
The difference is in the question being answered
Deduplication asks, “have I seen this message identity within a defined period?” If so, it routes to a duplicate branch or stops. Idempotency asks, “if this operation runs again, will the final outcome remain the same?” It is commonly enforced by the destination API, a unique constraint, a ledger, or an inherently idempotent set operation.
| Aspect | Deduplication | Idempotency |
|---|---|---|
| Boundary | Before/inside workflow | At destination effect |
| State | Key with TTL | Key or domain rule |
| After retention | Message may look new | Safe if the key is retained |
| Example | Reject repeated event_id | Return the first payment result |
Protection quality depends on the key
A key must remain stable across every attempt for the same event. A UUID regenerated by the workflow for every request cannot identify a source retry. Use an event_id, transaction number, tenant plus external ID, or a canonical payload hash when the contract genuinely provides no identity.
Payload hashes require care: field order or nonbusiness changes may produce a different key, while two legitimate events with identical payloads may be mistaken for duplicates. A domain identity is usually easier to explain during audits and reconciliation.
TTL and retention are business decisions
Deduplication state is not always stored forever. Its TTL should cover the realistic retry and replay window. A short TTL allows late duplicates; an excessive TTL consumes state and may reject valid events when a source reuses identifiers.
NitroNet provides a Deduplicate node backed by PostgreSQL state and TTL. During synchronous execution, the reservation is released when downstream processing fails unless keep_on_failure is selected. On an asynchronous path, delivery has a separate lifecycle, so reservation remains until TTL. This distinction matters when replay is designed.
Combine deduplication and idempotency
- Validate identity.
Reject a request missing the event ID required by the contract.
- Deduplicate at ingress.
Reduce repeated work with a documented key and TTL.
- Carry the key to the destination.
Use the same key in a record or idempotency header.
- Enforce a constraint.
A unique key or ledger guards the effect when workflow state is unavailable.
- Reconcile outcomes.
Connect previous and current results through correlation identity.
Example: processing an order
An e-commerce system sends order_event_id and order_id. Deduplicate uses the event ID so the same event does not run mapping twice. Write DB uses order ID under a unique constraint. If an insert response is lost after commit and delivery repeats, the constraint prevents a second order; the workflow can inspect the existing record before selecting a response.
Another event for the same order, such as a status change, needs a different event ID. Deduplication must therefore not use only order ID when several legitimate events can occur during an order lifecycle.
Consider concurrency when two messages with the same key arrive almost simultaneously. A non-atomic “check then save” allows both to pass before state is recorded. Reservation must be one database operation with an appropriate constraint or lock. Parallel testing should be part of validation rather than sending the same request only in sequence.
Define the duplicate branch response. Some consumers can receive the previous success result, others require an HTTP conflict, and an event stream may only record that the event was ignored. The choice must match the source contract. Do not discard duplicate evidence too quickly; volume and origin can reveal retry bugs, repeated provider webhooks, or endpoint abuse.
Document key format so source changes do not break protection. Normalize casing, separators, tenant scope, and event version before creating the key. During contract changes, support a transition or version prefix so old keys remain recognizable. Never share one namespace across unrelated organizations, and monitor duplicate branches for keys that are too broad or too narrow.
Limitations and failure policy
State can expire and external effects live at another boundary. Make the destination API idempotent, use unique constraints, and define replay and reconciliation.
Test parallel requests, retries after timeout, replay after TTL, and downstream failure. Read about at-least-once delivery risk and retry boundaries to place both controls correctly.