In distributed integration, a sender can lose the response even after the destination processes a request. To avoid losing the message, the system sends it again. The consequence is that the destination may receive the same request more than once. This is the central trade-off of at-least-once delivery.
What does at-least-once mean?
At-least-once means the system attempts to process every message one or more times. When completion is uncertain, a message can be retried. The model does not guarantee a single execution. End-to-end “exactly once” generally requires coordination among sender, transport, and destination that is not always available, especially for third-party APIs and email.
NitroNet HTTP and email deliveries use retry mechanisms. Database writes inside one local transaction have a stronger boundary, but a repeated request across executions still needs business identity. A success label for one attempt must not be interpreted as a global guarantee that the effect happened only once.
Where can a duplicate begin?
- The sender sends a request.
The message reaches the destination with a correlation ID.
- The destination performs the effect.
A record is created, a payment is processed, or an email reaches the server.
- The response is lost.
The connection ends before the acknowledgment returns.
- The sender retries.
Without an idempotency key, the destination treats it as a new request.
A timeout does not reveal whether the effect occurred. Treating a timeout as “definitely failed” is a common source of duplicates. Treating it as “definitely successful” can lose a process when the request never arrived.
Duplicate effects have different consequences
| Effect | Duplicate risk | Primary control |
|---|---|---|
| Customer insert | Two profiles | Unique external ID |
| Invoice | Double billing | Invoice/idempotency key |
| Payment | Double debit | Provider idempotency + reconciliation |
| Repeated notification | Notification ledger | |
| Status update | May be safe | Set absolute value, not increment |
An operation such as set status=paid can be idempotent because every repetition produces the same state. An operation such as balance=balance+amount is not idempotent because every retry adds the value again. Risk follows destination effect, not just the HTTP method.
Build layered protection
- Derive an idempotency key from event identity instead of generating a new UUID per attempt.
- Store the key and first result at the destination so repeats return the same outcome.
- Use a unique constraint as the last guardrail for database inserts.
- Apply ingress deduplication to reduce repeated processing.
- Record correlation ID, attempt, status, and destination reference for reconciliation.
Workflow deduplication and destination idempotency complement each other. Deduplication can have a TTL and state that expires; idempotency based on business identity at the destination protects the boundary where the actual effect occurs.
Example: sending a payment
A workflow receives payment_request_id from the owning system. It uses that value as the idempotency key for the provider instead of regenerating it on retry. When the provider times out, the workflow follows its contract: either safely retry with the same key or query an inquiry endpoint.
The provider result is stored with the key and correlation ID. A duplicate response that refers to the first transaction is not always an error; the workflow may return the previous result after verifying identity and amount. Financial decisions still follow provider rules and reconciliation managed by authorized owners.
Idempotency storage needs explicit decisions: how long a key remains, what state is recorded while an attempt is in progress, and how parallel requests are blocked. A safe pattern atomically reserves the key, stores a request fingerprint, and then records the final result. If the same key arrives with a different payload, the destination should reject the conflict instead of returning an unrelated outcome.
High-value effects still need reconciliation. Periodically compare the source ledger, delivery log, and destination references. Do not use timestamps as the only identity because clocks and precision can differ. Store a non-PII correlation ID and a business reference searchable by authorized operators. Ambiguous cases can then be resolved through evidence rather than repeated requests.
Limitations that must be communicated
External retries are at-least-once. Use idempotency keys when supported, design inquiry and reconciliation, and never treat a timeout as proof of success or failure.
Buffering and queueing mechanisms can preserve messages during a retention period, while messages exceeding the limit still need a visible failed or stopped state. Read when a retry is appropriate and the risks of database operations to design safer flows.