Durability & recovery

An fdyno success response means the underlying FoundationDB transaction committed. The survival of that commit is then determined by the FoundationDB cluster's redundancy, storage engine, fault domains, and backup configuration rather than the number of fdyno processes.

Durability and consistency answer different questions. Consistency defines which states can be observed. This page defines which state survives a failure and who must recover it.

The durability boundary

For a normal item mutation, one FoundationDB transaction contains the base item, large-item chunks, affected secondary-index entries, and a change-stream record when streams are enabled. A tokened TransactWriteItems also stores its idempotency record in that transaction. FoundationDB either commits that set or none of it.

fdyno does not acknowledge the operation before commit. Once the HTTP caller receives success, the transaction has committed according to FoundationDB. The process does not maintain a write-behind queue, local write-ahead log, or separate index/stream replication pipeline.

This boundary has two consequences:

1. A process crash cannot expose half of one FoundationDB transaction.
2. fdyno cannot make a weak FoundationDB topology durable. A single-process development cluster remains a single point of data loss even when three fdyno processes use it.

Do not translate this into an fdyno recovery point objective (RPO) or recovery time objective (RTO). The project publishes neither. Establish those values from the FoundationDB design you deploy and from measured restore exercises.

Durable and process-local state

The following state is stored in FoundationDB and survives an fdyno restart:

• table metadata, base items, and item chunks;
• synchronous GSI/LSI entries and large projection chunks;
• change-stream records and large record chunks;
• TTL configuration, tags, and stored resource-policy documents;
• immutable on-demand backup generations;
• staged restore batches, progress, leases, and fencing state;
• TransactWriteItems idempotency tokens and operation receipts; and
• maintenance worker leases and TTL sweep cursors.

See Data model and keyspace for the exact keys, encodings, integrity checks, and retention.

Process-local state includes HTTP connections, in-flight buffers, JSON and Prometheus metrics, access logs, the token-GC scan cursor, and the mutable Scylla compatibility map. Credentials and the default shard count come from the environment. Each enabled stream's selected topology is durable table metadata.

Failure behavior

Failure	Data behavior	Operator action
fdyno exits before commit starts	No transaction was committed.	Route to another ready process and retry according to operation semantics.
fdyno exits while commit is in flight	The transaction may have committed even though the caller received no response. It is never partially committed.	Reconcile the item or use the same ClientRequestToken; do not assume failure from the missing response.
fdyno exits after commit but before response	The write is durable, but the caller has an unknown outcome.	Apply the same ambiguity procedure as an in-flight commit.
One fdyno replica fails	Other replicas see committed state through FoundationDB.	Remove the failed process and replace it; no fdyno data replay is required.
FoundationDB is unreachable	Readiness returns 503; operations fail or retry until their transaction/request budget ends. Liveness remains 200.	Restore cluster reachability or quorum. Restarting fdyno alone does not repair the backend.
FoundationDB loses quorum	The unavailable side cannot commit. Availability yields to consistency.	Recover the FoundationDB fault domain using its operational procedures.
FoundationDB permanently loses data beyond its redundancy	fdyno has no independent replica from which to rebuild.	Restore a FoundationDB-level backup or other independently stored recovery copy.

A returned error is not always proof that a write did not commit. fdyno's transaction runner marks context timeout/cancellation after a commit attempt as transaction outcome may be unknown, but a client can also lose the HTTP response after a successful commit. See Reliable writes before choosing a retry policy.

Recovery by operation type

Not every API call is one FoundationDB transaction. Operators need to distinguish atomic item operations from multi-step maintenance and copy operations.

Item, index, and stream mutation

A single item write commits its item/chunks, index maintenance, and stream record atomically. TransactWriteItems commits all transaction members and its token atomically. If the transaction aborts, none of those writes are visible.

BatchWriteItem is not atomic. fdyno first attempts one FoundationDB transaction, but if the request exceeds FoundationDB's transaction size it validates the whole request and commits size-bounded chunks. Failures are returned as UnprocessedItems. Some chunks may therefore be durable while others are not.

GSI backfill

Adding a GSI first commits metadata with the index in CREATING, then backfills existing items in bounded transactions. New foreground writes maintain the index as soon as that metadata is visible. If fdyno stops mid-backfill, the index can be incomplete while it remains CREATING.

At startup, each process scans for incomplete backfills and re-runs them from the beginning. Rewriting an existing index entry is idempotent, and each page conflicts with concurrent item changes so it retries against current data. This recovery is background work and is not readiness-gated or lease-elected. Do not query a GSI for complete results until its status is ACTIVE.

Maintenance workers

TTL deletion reopens the item and checks expiry in the deleting transaction before removing the base item, indexes, and adding a TTL stream record. A repeated or stale sweep therefore does not delete a refreshed item. CDC trim and transaction- token GC delete only maintenance records and are safe to repeat.

The lease mechanism reduces duplicate scans; it is not a consensus-fenced job system. Correctness comes from transactional preconditions and idempotent cleanup, not from a claim that two processes can never overlap.

Backup and restore

An on-demand backup uses several transactions. Item pages share one pinned read version. Each page is encoded, chunked, verified, and committed to an immutable generation. A final transaction changes the generation from STAGING to COMPLETED. Readers never use an incomplete generation.

A large restore first stores immutable source batches in a durable restore job. It then creates a fenced CREATING target and applies bounded batches. Each batch commits item and index writes with progress. The last batch publishes ACTIVE in the same transaction. Startup and periodic recovery can resume READY or RUNNING jobs after process loss.

Deleting a CREATING restore target cancels its job. Historical PITR remains unsupported and returns an explicit unavailable error. See Backup & restore.

FoundationDB durability configuration

fdyno does not configure FoundationDB. The cluster operator owns these durability decisions:

• redundancy mode and replication across independent fault domains;
• storage engine and persistent-volume behavior;
• coordinator placement and quorum recovery;
• capacity headroom, free-space monitoring, and failure-domain maintenance;
• cluster-file distribution and access control;
• encryption and FoundationDB network security;
• native backup destination, retention, restore drills, and cross-site recovery;
• compatible client library/API-version policy during upgrades.

Adding fdyno replicas improves HTTP availability and permits rolling process replacement. It does not change any item above.

Backup independence

fdyno's CreateBackup stores the backup under dynodb_backups in the same FoundationDB database as the source. It is useful for table-level copy and operator-error recovery while that database remains healthy. It is not an independent disaster-recovery copy and shares the source cluster's corruption, credential, and loss domain.

FoundationDB-native backup should protect the complete database, including table data, fdyno backup objects, idempotency tokens, and worker metadata, in an independent destination. The fdyno service does not orchestrate native backup. A same-host fresh-cluster drill checks one local backup/restore chain and raw keyspace equality. It does not validate an off-host copy, a replicated FoundationDB topology, or production RPO/RTO.

Recovery validation

A production evaluation should test these cases rather than infer them from a successful health check:

1. Stop one fdyno process during reads and writes; verify routing to a surviving process.
2. Interrupt a write response and reconcile the ambiguous outcome by exact token replay.
3. Restart fdyno against the same cluster; verify metadata, TTL behavior, tags, and transaction-token replay.
4. Interrupt a GSI backfill and verify it returns to ACTIVE after startup recovery.
5. Interrupt a large restore and verify another process resumes it without exposing partial data. Also verify that deleting a CREATING target removes its job.
6. Restore a FoundationDB-native backup into an isolated environment and check application invariants, not only key counts. The repository's local drill exercises this on two single-process clusters sharing one host; repeat it across the intended external failure domain.
7. Test loss of a FoundationDB process or fault domain according to the selected redundancy mode.

The repository automates steps 1–3 with bash scripts/fault-smoke.sh against owned local processes and a single-process FDB outage/restart. See Failure testing for its exact invariants. The native local restore drill covers only the same-host part of step 6. Neither proves external recovery.

The codebase includes a bounded two-instance fault history and integration tests. It does not include a Jepsen or Elle report, a real fdyno-to-FDB network partition, replicated FoundationDB failover evidence, deterministic FDB simulation, or measured RPO/RTO. Treat those as evidence gaps in a production acceptance review.