Architecture overview

fdyno is a DynamoDB-protocol service over FoundationDB. The Go process validates and translates requests; FoundationDB stores the durable database state and supplies the transactional concurrency control. There is no in-process item store and no local write-ahead log.

This split keeps DynamoDB request semantics, expressions, key layout, and response compatibility in fdyno. FoundationDB provides replication and recovery.

System map

flowchart TB
    SDK["AWS SDK<br/>signed HTTP"]

    subgraph SVC["fdyno process"]
      direction TB
      EDGE["HTTP boundary<br/>SigV4 · routing"]
      OPS["DynamoDB semantics<br/>JSON · expressions · PartiQL"]
      STORE["FDB mapping<br/>tuples · codec · indexes"]
      WORKERS["background workers<br/>TTL · streams · recovery"]
      EDGE --> OPS --> STORE
      WORKERS --> STORE
    end

    FDB[("FoundationDB<br/>transactions · ordered keys · versionstamps")]

    SDK -->|"DynamoDB JSON"| EDGE
    STORE -->|"transactions"| FDB

The service has two kinds of work:

• Foreground API work translates one DynamoDB action into one or more FoundationDB transactions.
• Maintenance work uses bounded transactions for TTL reclamation, stream trimming, idempotency-token garbage collection, and resumable GSI backfill. TTL, stream-trim, and token-GC workers coordinate with FoundationDB leases; GSI recovery is instead safe to rerun idempotently and may do redundant work if several instances start together.

Request path

sequenceDiagram
    participant C as Client
    participant R as service.go
    participant H as http.go / helpers.go
    participant O as operation code
    participant F as FoundationDB

    C->>R: POST / + X-Amz-Target + SigV4
    R->>R: deadline, method/path/header checks
    R->>R: bound body and verify signature
    R->>H: dispatch action
    H->>H: decompress and decode typed JSON
    H->>O: PutItem(input, request context)
    O->>O: validate request and expressions
    O->>F: transaction callback
    F-->>O: write commit, retryable error, or terminal error
    O-->>H: typed result or ServiceError
    H-->>C: DynamoDB-shaped JSON + error headers/CRC

For a normal data-plane request, the path is:

1. ServeHTTP installs one operation deadline for the whole HTTP request. The default is 30 seconds and DYNODB_OPERATION_TIMEOUT can change it.
2. service.go accepts DynamoDB POST / requests, bounds request headers and the raw body, verifies SigV4, and selects the action from X-Amz-Target.
3. The adapter decodes the request into an operation-specific type. Raw and decompressed request bodies are both bounded at 16 MiB.
4. The operation performs DynamoDB validation, evaluates expressions, and opens a read-only or read-write FoundationDB transaction.
5. A read-only operation returns after its required futures resolve. It does not commit an unmodified transaction. Its reads still use one transactional snapshot and the normal retry and deadline path.
6. The persistence helpers map table metadata, item keys, values, indexes, and stream records to FoundationDB keys. A successful write response is produced only after commit completes.
7. Typed domain errors become DynamoDB-shaped errors. Unexpected storage or codec failures become internal errors rather than partial successful responses.

Core item, query, batch, transaction, and PartiQL methods do not depend on HTTP. Tests can call them directly. Some compatibility control-plane handlers remain in http.go. Not every routed action has a separate *_ops.go method.

Request handling code

cmd/dynodb/main.go              process startup, HTTP server, optional workers
internal/dynodb/service.go      request deadline, routes, health, response capture
internal/dynodb/http.go         typed HTTP adapters and compatibility handlers
internal/dynodb/*_ops.go        item, query, batch, transaction, table, backup logic
internal/dynodb/expressions.go  condition/update/filter expression evaluation
internal/dynodb/projection.go   projection-path handling
internal/dynodb/partiql*.go     lexer/parser and PartiQL execution
internal/dynodb/fdb_store.go    tuple keys, chunks, indexes, item access
internal/dynodb/codec.go        typed binary item values
internal/dynodb/transaction_runner.go  typed read/write completion, retries, deadlines

The operation layer handles boolean conditions, comparison/functions, nested update paths, numeric and set updates, filters, and projections independently of the HTTP adapter and raw key format. PartiQL is parsed into a typed statement and executed through the same item/index/stream helpers; transactional PartiQL uses one FoundationDB transaction. Keeping these semantics above fdb_store.go prevents transport JSON or raw key bytes from becoming the domain model.

Process state

The process-state rule is:

User database state needed after restart is stored in FoundationDB, not in a particular fdyno process.

Stored in FoundationDB	Process-local and replaceable
Table schema, tags, policies, and TTL specification	Open connections and in-flight request buffers
Base items, item chunks, and synchronous index entries	Metrics counters
Stream records and large-record chunks	Positive existence and decoded table-metadata caches
Immutable backup generations and durable restore jobs	Tag-operation coordination maps
Transaction tokens and operation receipts	Credentials and feature configuration loaded from the environment
Worker leases and TTL sweep cursors	Alternator-compatible system-config rows

The right-hand column matters. In particular, .scylla.alternator.system.config updates are held in the Service.systemConfig map; they are not durable and are not shared between instances. Metrics also reset on restart. These are compatibility or operability surfaces, not table data.

The positive existence cache only speeds up selected control-plane checks. A separate decoded table-metadata cache can supply speculative schema to selected item paths and bounded base-table Query. The transaction checks the metadata revision in FoundationDB before using a speculative result; losing either cache does not lose durable table, item, index, stream, backup, restore, receipt, tag, TTL, or transaction-token state. Every instance must use the same credentials and feature configuration.

API transaction mapping

“One request equals one transaction” is true for the main item paths, but not for every workflow.

API operation	FoundationDB mapping
GetItem, one Query/Scan page, BatchGetItem, TransactGetItems	One read transaction and one MVCC snapshot. An expired GetItem is treated as absent and then rechecked/deleted in a later cleanup transaction before the request completes.
PutItem, UpdateItem, DeleteItem	One read-write transaction containing the condition read, base-item mutation, index maintenance, and optional stream record.
TransactWriteItems, PartiQL ExecuteTransaction	One transaction across the selected items/tables. A client request token, when supported and supplied, is checked and written in that transaction.
BatchWriteItem	Tries one transaction. If the physical FoundationDB transaction is too large, the implementation validates the whole batch without committing, then uses bounded chunks and reports failures as UnprocessedItems; the DynamoDB API is non-atomic.
Add a GSI	One metadata transaction registers it as CREATING; bounded transactions backfill old items; a final transaction marks it ACTIVE. Live writes maintain the new index during backfill.
Backup, restore, TTL sweep, stream trim	Multi-transaction workflows with explicit visibility, cursor, or state rules rather than one unbounded transaction.

See Transactions for retry and failure behavior and Consistency model for the guarantee attached to each boundary.

FoundationDB storage mapping

fdyno uses FoundationDB primitives instead of emulating them in memory:

• Directory layer: isolates each table and global service-owned keyspace.
• Tuple layer: creates unambiguous prefix ranges for partitions, indexes, and pagination cursors.
• Transactions: keep a condition read, base mutation, secondary-index entries, and stream record in one atomic boundary.
• Versionstamped keys: assign stream positions at commit, after the final commit order is known.
• Range reads: implement ordered partition queries and bounded pages without materializing a whole table in the service.

fdyno has one storage backend. Replacing FoundationDB would also require new ordering, conflict, and versionstamp behavior. Data model and keyspace defines the current key layout.

Failure behavior

• If validation or a condition fails, the transaction callback returns an error and no buffered writes commit.
• If FoundationDB reports a retryable transaction error, the transaction body can be executed again. Operation code must therefore keep per-attempt accumulators local and avoid irreversible side effects in callbacks.
• A process crash cannot expose half of a committed item/index/stream mutation: FoundationDB commits the transaction atomically or not at all.
• If FoundationDB is unreachable, data operations fail instead of falling back to a local copy. /readyz performs a backend round trip and returns unavailable while liveness endpoints continue to report that the process itself is running.
• A timeout while commit is in flight is explicitly treated as an unknown commit outcome. Use a stable ClientRequestToken for writes that must not be applied twice; see Reliable writes.

Not implemented

fdyno does not provide:

• a second replication layer, local durable cache, or offline-write mode;
• an eventually consistent read replica path; normal reads use FoundationDB transactions even when the DynamoDB request does not set ConsistentRead;
• an unbounded transaction; large maintenance operations are paged or chunked;
• multi-Region/global-table replication at the fdyno layer;
• dynamic DynamoDB-style stream shard splitting;
• durable, cluster-wide Alternator system-config mutations.

Synchronous indexes and strict transactional reads remove propagation states, but put index work on the write path and make backend availability a prerequisite for serving data.

Code and tests

The statements above are grounded in these paths:

• Request boundary and routing: internal/dynodb/service.go, internal/dynodb/auth.go, internal/dynodb/helpers.go, internal/dynodb/http.go
• Operation deadline and retry loop: internal/dynodb/transaction_runner.go
• Service state boundary: internal/dynodb/types.go, internal/dynodb/lease.go, internal/dynodb/ttl.go, internal/dynodb/idempotency.go
• Storage mapping: internal/dynodb/fdb_store.go, internal/dynodb/fdb_persistence.go, internal/dynodb/codec.go
• Atomic item paths: internal/dynodb/item_ops.go, internal/dynodb/txn_ops.go
• Multi-transaction exceptions: internal/dynodb/batch_ops.go, internal/dynodb/table_ops.go, internal/dynodb/backup_ops.go
• Retry-attempt isolation test: TestBatchGetRetryDoesNotLeakAttemptResults in internal/dynodb/retry_safety_integration_test.go

The repository does not contain an end-to-end network-partition history checker; backend partition behavior comes from FoundationDB's transaction guarantees and is an evidence gap for fdyno-specific fault testing.