Reliable writes & idempotency

fdyno is a stateless proxy in front of FoundationDB: a request travels client → fdyno → FoundationDB, two network hops and a database. Consistency and Durability & recovery describe what happens once a write commits. This page covers the harder question — recording a write once and only once when a hop fails and the caller never learns whether the commit happened.

The path a write takes

A mutation, its secondary-index entries, its change-stream record, and — for TransactWriteItems — its idempotency token all commit in one FoundationDB transaction. fdyno runs that transaction through the FoundationDB client's transaction loop, which automatically retries retryable errors by re-executing the transaction body.

Where it can fail, and what that means

A write can fail at three points, with very different consequences:

Failure	What happened	Naive retry
Request lost (before the commit is submitted)	Nothing committed.	Safe — the write simply did not happen yet.
Response lost (after a successful commit)	The write is durable, but the caller saw an error.	Re-issues a write that already succeeded.
Unknown commit outcome (commit_unknown_result)	The fdyno↔FoundationDB link dropped — or FoundationDB recovered — exactly as the commit was in flight. Even fdyno cannot tell whether it committed.	May apply the write twice, or zero times.

The third case is the subtle one, and the reason idempotency matters at all.

How fdyno behaves under an unknown outcome

The FoundationDB transaction loop treats commit_unknown_result as retryable and re-executes the transaction body. Whether that is correct depends entirely on whether the operation is idempotent:

Operation	Idempotent?	Under an unknown outcome
PutItem (fixed value)	Yes	Re-applying writes the same bytes; the state converges. Safe.
DeleteItem	Yes	Deleting an absent key is a no-op. Safe.
UpdateItem — SET/REMOVE of fixed values	Yes	Re-applying sets the same value. Safe.
Conditional write (attribute_not_exists, …)	Yes (by construction)	A retry after a successful first write fails the condition; the caller reads that as "already applied." Safe.
UpdateItem — ADD/DELETE on a number or set (accumulator)	No	fdyno applies these as read-modify-write inside the transaction, so a re-execution re-reads the (possibly already-updated) value and applies the delta again. A retry can double-apply.
TransactWriteItems with a ClientRequestToken	Yes (by token)	Exactly-once — see below.

The takeaway: idempotent writes are safe to retry at any layer — fdyno's internal retry of commit_unknown_result is invisible and harmless for them. The only writes that need extra care are non-idempotent accumulators.

Exactly-once with a client request token

For a write that must apply once and only once yet is not naturally idempotent, wrap it in TransactWriteItems with a ClientRequestToken. This is the end-to-end idempotency pattern (the same approach TigerBeetle builds on). fdyno persists the token in FoundationDB in the same transaction as the writes, so the token and the effect are atomic:

• If the transaction committed, the token is present — a retry with the same token returns the original outcome and re-applies nothing.
• If it did not commit, neither did the token — the retry executes fresh.

Because the token lives in FoundationDB rather than in any process, this holds across a lost response, a lost request, an fdyno restart, and commit_unknown_result. The deduplication window is DynamoDB's 10 minutes.

The token must be client-generated and stable

Generate the idempotency key on the client, persist it before the first submission, and reuse the same token on every resend. A fresh token per attempt defeats the mechanism entirely — it is the client-side stability of the key, not the server, that makes a retry safe.

Recipe: a write that must happen exactly once

1. Generate a stable idempotency key on the client; persist it before submitting.
2. Submit the write inside a TransactWriteItems with that key as ClientRequestToken.
3. On any error or timeout, resubmit with the same token.

fdyno deduplicates the replay within the 10-minute window. For simple create-once semantics without a transaction, a conditional PutItem with attribute_not_exists(<pk>) is idempotent by construction: the first call creates the item; a retry fails ConditionalCheckFailed, which the caller treats as "already created."

Boundaries

• Non-idempotent single-item accumulators have an inherent at-least-once exposure. ADD/DELETE on a number or set outside a transaction can double-apply on a retried unknown outcome. This is the same exposure DynamoDB itself carries — there is no single-item idempotency token in the DynamoDB API. For exactly-once, use a transaction with a ClientRequestToken, or restructure the write to be idempotent.
• A returned error is not proof of "did not apply" for non-idempotent ops. Durability attaches to a successful response. After a non-idempotent operation returns an error, the caller cannot assume the write did or did not take effect — which is exactly why must-happen-once writes belong in a tokened transaction.

Empirical verification

The guarantees here are argued from FoundationDB's transaction semantics and fdyno's atomic token persistence. Stress-testing them under injected failures — in particular the fdyno↔FoundationDB partition that produces commit_unknown_result — is the job of fault-injection testing: a Jepsen workload with the Elle checker driving the DynamoDB API while a nemesis partitions the proxy from the cluster, or FoundationDB-style deterministic simulation. This is the empirical counterpart to the reasoning on this page.

References

• TigerBeetle. Reliable Transaction Submission. docs.tigerbeetle.com (end-to-end idempotency).
• FoundationDB. Developer Guide — transactions that may be retried; commit_unknown_result.
• Kyle Kingsbury. Jepsen and the Elle transactional consistency checker. jepsen.io