Durability & recovery¶
Consistency describes what fdyno guarantees about correctness — the order in which operations appear to run. This page describes what it guarantees about survival: when a write becomes durable, what happens when a process or the cluster fails, and how the system recovers. It states the guarantee precisely and is explicit about where it ends.
The guarantee¶
A write is durable the instant its API call returns success. fdyno holds no durable state outside FoundationDB, so any server process may crash, be killed, or be replaced at any time with no data loss and no recovery procedure.
Durability is inherited, not reimplemented: a fdyno write is durable exactly when the FoundationDB transaction that carried it is durable.
All durable state lives in FoundationDB¶
fdyno is a stateless translation layer. Every durable artifact is a FoundationDB key — base items, secondary-index entries, change-stream records, idempotency tokens, table metadata, tags and resource policies, on-demand backups, and the background workers' cursors and leases. A server process keeps only ephemeral, reconstructable state (open connections, in-flight request buffers). This is the durable-state invariant, and it is what makes the failure model simple: there is no per-process state to lose, so there is nothing to recover in the process — only the cluster matters.
Consequently, fdyno's durability is FoundationDB's durability. A committed transaction is persisted to the configured number of transaction logs before the commit returns. The recovery point and recovery time (RPO/RTO) after a hardware fault are properties of the FoundationDB cluster's replication and fault-tolerance configuration, not of fdyno; fdyno adds no replication of its own.
Failure modes¶
Each row states what is guaranteed under the failure and how the system returns to health. None requires operator intervention beyond restoring FoundationDB itself.
| Failure | What is guaranteed | Recovery |
|---|---|---|
| Process crash mid-request | The request's FoundationDB transaction either committed in full or not at all — commit is atomic, so no client ever observes a torn or partial write. | None. The layer is stateless; a restarted or replacement process serves immediately. Work that had not yet committed is simply absent, and the client retries. |
| FoundationDB unreachable | No stale or incorrect data is served; writes fail rather than appear to succeed. | GET /readyz performs a FoundationDB round-trip and returns 503, so a load balancer drains the instance, while GET /livez stays 200 so an orchestrator does not kill a healthy process over a transient backend blip. Data-plane operations return retryable errors. The instance rejoins automatically when FoundationDB recovers. |
| Network partition | Consistency is preserved: the side without a quorum cannot commit (fdyno is CP). | FoundationDB needs a quorum to assign read and commit versions; writes pause on the minority side and resume when the partition heals. |
| Crash mid-GSI-backfill | Live writes keep the new index correct; only pre-existing items may be temporarily unindexed, and the index remains CREATING — never falsely ACTIVE. |
On startup, every index left in CREATING is re-backfilled (an idempotent re-scan that re-writes already-present entries harmlessly) and flipped to ACTIVE. |
| Crash mid-maintenance-sweep | No correctness impact: TTL/CDC/token sweepers re-check their precondition inside the deleting transaction, so a concurrent write is never clobbered. | Each worker persists its scan cursor in FoundationDB and resumes from it; an FoundationDB-backed lease ensures exactly one instance runs each worker, so restarts and multi-instance deployments never double-scan or skip. |
| Duplicate request (client retry) | A TransactWriteItems retried with the same ClientRequestToken is applied at most once within DynamoDB's 10-minute window. |
The token is persisted in FoundationDB; a replay inside the window returns the original outcome instead of re-executing. |
Recovery mechanisms¶
The table above relies on a small set of mechanisms, each designed to be safe to run repeatedly and across many instances:
- Stateless failover. Because no durable state lives in the process, replacing a process is not a recovery operation — it is the normal mode of operation. Run as many instances as you like behind a load balancer.
- Atomic commit. Base item, index entries, and the stream record for one mutation commit at a single FoundationDB version, so a partial write is not a state the system can be in.
- Idempotent backfill resume. Incomplete GSI backfills are detected and re-run at startup; re-running from the start is correct because re-writing an existing index entry is a no-op.
- Resumable, leased workers. Maintenance workers checkpoint their position in FoundationDB and coordinate through FoundationDB leases, so a restart resumes rather than restarting, and N instances do the work of one.
- Persisted idempotency tokens. Transaction idempotency survives a restart and holds across instances, because the token lives in FoundationDB, not memory.
- Liveness vs. readiness. Liveness (
/,/healthz,/livez) reflects only that the process is up; readiness (/readyz,/health/ready) reflects whether FoundationDB is reachable. Separating them means a backend blip drains an instance from rotation instead of restarting it. - Backups. On-demand backups and snapshot-on-restore PITR are point-in-time consistent within FoundationDB's MVCC window. See Backup & restore.
Boundaries¶
What this guarantee does not cover, stated plainly:
- RPO/RTO are the cluster's, not fdyno's. Surviving a disk or machine failure is a function of the FoundationDB cluster's replication factor and redundancy mode. A single-process development cluster has no redundancy; a production deployment must configure FoundationDB for the durability it needs.
- A returned error means the write did not commit. Durability attaches to a successful response. A write that returns an error — including a retryable one during a partition or overload — was not committed and must be retried.
- No cross-Region disaster recovery at this layer. fdyno targets one FoundationDB cluster. Cross-Region durability requires FoundationDB's own multi-Region capability or external replication; fdyno does not provide global tables.
- Consistent backups at cluster scale are not in-process. An in-layer
CreateBackupis consistent only within FoundationDB's ~5-second MVCC window; consistent backups of a large cluster use FoundationDB's native backup tooling. See Backup & restore.
References¶
- Jingyu Zhou et al. FoundationDB: A Distributed Unbundled Transactional Key Value Store. SIGMOD, 2021. (Transaction logs, recovery, and the durability model fdyno inherits.)
- FoundationDB documentation: Fault Tolerance and Administration (redundancy modes, replication, and the RPO/RTO they determine).