Skip to content

Secondary indexes that never lag

DynamoDB's global secondary indexes are eventually consistent: a write to a base table propagates to its GSIs asynchronously, so a query against a GSI can briefly miss an item that a GetItem on the base table already returns. fdyno maintains every index inside the same transaction as the base write, so this window does not exist. That choice buys a simpler mental model and costs throughput; both halves are worth spelling out.

The eventual-consistency window

In DynamoDB, a base-table write returns as soon as the item is durable. Index propagation happens afterward, on a separate path. The gap is usually small, but it is real and observable, and it produces a recognizable class of bug:

table.put_item(Item={"pk": "u#42", "email": "ada@example.com", "status": "active"})

# moments later, from the same or another client:
resp = table.query(
    IndexName="status-index",
    KeyConditionExpression=Key("status").eq("active"),
)
# the item just written may not be in resp["Items"] yet

Read-after-write against a GSI is not guaranteed. Code that treats an index query as authoritative immediately after a write (a uniqueness check, a "did this enqueue?" poll, a fan-out that reads back what it just wrote) can see a stale answer. The standard fixes are to read the base table strongly instead, or to tolerate the lag. Both are real engineering costs that exist because indexing is off the write path.

What fdyno does instead

A base-item write and all of its global and local secondary-index entries are written on the same FoundationDB transaction and commit at a single version:

putItem(tr, baseKey, item)             // base item
putIndexEntries(tr, dir, tbl, item)    // every GSI + LSI entry
// ... stream record, idempotency token ...
// one atomic commit

There is no second path and no propagation delay. The instant the write commits, a query against any index reflects it, because the index entry is part of the committed write. Three properties follow:

  • Read-after-write holds on indexes. A GSI query immediately after a write sees the write.
  • No instance ever observes index lag. Multiple fdyno processes share one FoundationDB cluster and one transactional order; none of them can see an index trail a base write.
  • A restart cannot leave indexes half-applied. The index entries committed with the base item or not at all; there is no in-memory queue to lose.

Index maintenance is never a background job. The one exception is the bounded, resumable backfill that populates an index over existing items when that index is first created. It is a one-time catch-up, not part of the steady-state write path.

fdyno's GSIs are therefore strongly consistent rather than eventually consistent, and it is the same property that lets the consistency model claim strict serializability over index reads, not only base-table reads.

The cost, stated plainly

Synchronous indexing is not a free upgrade; it relocates work onto the write's critical path, and DynamoDB's asynchronous design exists for good reasons.

  • Write amplification is synchronous. An item in N indexes turns one logical write into N + 1 writes in a single transaction. DynamoDB pays the same writes, but off the critical path, so the caller's latency does not include them. In fdyno the caller waits for all of them to commit.
  • Indexes enlarge the transaction. Every index entry counts against FoundationDB's per-transaction size and key budget, and widens the conflict footprint, so a hot index key can now cause a base write to retry.
  • No independent index scaling. DynamoDB can provision and throttle a GSI separately from its base table. fdyno's indexes are not a separate capacity unit; they share the base write's transaction and its fate.

The tradeoff is deliberate. Eventually-consistent indexes are what let DynamoDB keep write latency low, scale indexes independently, and stay available when an index's partitions are busy. fdyno trades that flexibility for a model in which an index is never a stale copy of the truth; it is the truth, written once, atomically. Which side of that tradeoff is right depends on the workload; fdyno picks the one that makes correctness the default and pays for it at write time.