Most databases that claim "ACID compliance" bury the asterisk somewhere in a footnote. The transaction isolation level is "read committed" by default. Phantom reads are possible. Serializable mode exists but you’re warned it’ll tank performance. You end up writing compensating logic in application code and calling it a day.
FoundationDB takes a different position: strict serializability, always on, no knobs to fiddle. Meaning every transaction you run appears to execute atomically at some point in real time, and the entire history of transactions forms a single serial order everyone agrees on. Not eventually. Not "probably." Provably.
Apple uses it under CloudKit. Snowflake built their metadata store on it. It’s been running at serious production scale for over a decade. And the codebase — open-sourced in 2018, available at github.com/apple/foundationdb — is one of the most carefully engineered distributed systems you can read.
This article tears apart how FoundationDB actually delivers on that promise. Not the marketing version — the real one, with commit protocols, resolver nodes, MVCC snapshots, and the 5-second rule that’ll bite you if you don’t understand it.
The Core Premise: Separate Everything
Most databases are monolithic about storage and transactions. FoundationDB isn’t. Its architecture separates concerns so aggressively it looks almost modular:
- Coordinators hold cluster configuration state using Paxos — they’re the ground truth for who’s in the cluster.
- The ClusterController monitors all processes and recruits roles as needed.
- The Sequencer (part of the Master role) assigns globally monotonic read versions and commit versions.
- Commit Proxies are the client-facing gatekeepers — they handle incoming transactions and orchestrate the commit pipeline.
- GRV Proxies (Get Read Version) handle batching and issuing read version requests.
- Resolvers do conflict detection. Dedicated processes, doing nothing else.
- Transaction Logs (TLogs) are the write-ahead log. Mutations are durable here before storage ever sees them.
- Storage Servers hold the actual key-value data and serve reads.
Why does this matter? Because it means each component can be scaled, failed, and replaced independently. A resolver crash doesn’t lose data. A storage server going down doesn’t stall commits. Every role is recruited and monitored; if a process dies, the ClusterController recruits a replacement.
How a Transaction Actually Works
Walk through what happens when your application runs a transaction. This is the thing most documentation glosses over, but it’s the crux of everything.
Step 1: Get a Read Version
Before your transaction reads anything, it needs a read version. This is a 64-bit integer representing a logical timestamp — a point-in-time snapshot of the database.
The client contacts a GRV Proxy with a GetReadVersion request. The proxy batches these requests (hundreds can arrive per millisecond), asks the Sequencer for a version, and hands it back. This version is the minimum version at which all previously committed transactions are guaranteed visible.
From this moment, all reads in your transaction are consistent snapshots at this version. This is MVCC — the storage servers keep multiple versions of each key, and they serve whatever version you requested. No locks acquired. Other writers can proceed in parallel.
Step 2: Build the Transaction Buffer
As your transaction reads and writes, the FDB client library tracks everything locally:
- Read set: keys and key ranges you read (used for conflict detection later)
- Write set: mutations you want to apply
Writes are buffered client-side. Until commit, nothing touches the database. This is optimistic concurrency — you assume there’s no conflict and only check at the end.
One critical consequence: reads reflect your own buffered writes. If you write key foo and then read foo in the same transaction, you see your own write. The client lib merges the buffer with storage results transparently.
Step 3: Commit
When you call commit(), the client sends the transaction to a Commit Proxy. The payload includes:
- The read version
- The read conflict ranges (not the raw data you read, just the key ranges)
- The write conflict ranges (typically the same as what you wrote)
- The actual mutations
The Commit Proxy does three things in order:
First, it asks the Sequencer for a commit version. This version is strictly greater than the read version and greater than all previous commit versions. The Sequencer hands these out serially — no two transactions get the same commit version.
Second, it sends the conflict check to one or more Resolvers.
Third, if resolvers approve, it writes mutations to Transaction Logs and confirms durability.
Step 4: Conflict Detection in the Resolvers
This is where the magic happens, and it’s worth slowing down here.
Each Resolver maintains a sliding window of write sets from recently committed transactions — specifically, the write conflict ranges from every transaction committed since some configurable lookback window.
When your transaction arrives, the Resolver checks: does your read conflict range overlap with any write conflict range from a transaction committed after your read version?
If yes: conflict. Your transaction aborts with not_committed. You retry.
If no: clean. The Resolver approves.
This is the formal mechanism that enforces serializability. If you read something that someone else subsequently modified before you committed, your view of the world is stale. You can’t commit on top of stale reads without violating serializability.
The Resolver is also checking for write-write conflicts: if two concurrent transactions both want to write the same key, only one can win.
Transaction A (read version: 100):
reads key "balance" -> 500
writes key "balance" -> 400
Transaction B (read version: 100):
reads key "balance" -> 500
writes key "balance" -> 300
Both try to commit. A gets commit version 101. B gets commit version 102.
Resolver approves A (no prior write to "balance" since version 100).
Resolver checks B: "balance" was written at version 101 (by A). B read it at version 100.
B's read conflicts with A's write. B aborts.
No phantom reads. No write skew. No lost updates. The conflict check catches all of them.
Step 5: Durability via TLogs
Once resolvers approve, the Commit Proxy sends mutations to Transaction Logs. TLogs are write-ahead logs distributed across multiple machines — by default, the cluster requires acknowledgment from a quorum before confirming the commit to the client.
At this point, the transaction is committed. Durable. The client gets its success response.
Storage Servers pull mutations from TLogs asynchronously. There’s a lag — a few milliseconds typically. This means if you read from a storage server immediately after commit, you might not see your write yet… unless you use the same read version that includes your commit, which the FDB client handles automatically by raising the read version on subsequent reads after a commit.
Strict Serializability, Not Just Serializable
There’s an important distinction here that databases often blur.
Serializable isolation means the execution is equivalent to some serial order. But that order doesn’t have to correspond to real time. A serializable database can let a transaction that started later appear to have executed earlier. This allows certain anomalies in distributed settings.
Strict serializability (sometimes called external consistency or linearizable serializable) adds the real-time constraint: if transaction A completes before transaction B starts, A must appear before B in the serial order. No going back in time.
FoundationDB delivers strict serializability because:
- Commit versions are handed out by a single Sequencer, globally ordered and monotonically increasing.
- Read versions are also from the Sequencer, and guaranteed to be at least as recent as any completed commit.
- Every transaction’s reads happen at a specific version, and conflict detection uses that same version as the reference point.
The global version timeline is the bedrock. Everything else follows from it.
The Redwood Storage Engine
For a long time, FDB shipped with an SQLite-based storage engine. It worked but wasn’t built for FoundationDB’s access patterns. In recent versions, the default is Redwood — a custom copy-on-write B-tree built from scratch.
Redwood is designed to handle FDB’s MVCC requirements natively. It stores multiple versions of each key, allows efficient range queries, and manages page-level copy-on-write to avoid in-place updates (which are notoriously tricky to make crash-consistent).
Storage servers also have a version cache — recent versions of keys are kept in memory to serve reads without hitting disk. The cache is keyed on (key, version), so reads at different versions can be served simultaneously.
When old versions are no longer needed (no active transaction has a read version that old), the storage server garbage-collects them. This is what FDB calls version vector cleanup, and it runs continuously in the background.
The Layer Cake: Why FDB Is a Foundation, Not a Feature
FDB only stores ordered key-value pairs. No secondary indexes. No document model. No SQL. That’s intentional.
The designers’ thesis: build the hardest part (ACID transactions at scale) correctly once, then let higher-level data models be implemented on top as Layers. A Layer is just client code that maps its data model onto FDB’s key space.
Apple’s Record Layer (open source at github.com/FoundationDB/fdb-record-layer) implements a typed record store with secondary indexes, queries, and Protobuf schemas on top of raw FDB. Because FDB transactions span arbitrary key ranges, you can update a record and its indexes atomically — no eventual consistency, no dual-write bugs.
This is the actual killer feature. You can build a fully consistent, indexed data model and the hardest invariants (index consistency, referential integrity) are enforced by the transaction layer for free.
Gotchas
The 5-second transaction limit. FDB enforces a hard maximum transaction duration of 5 seconds by default. If your transaction takes longer, it’s killed with transaction_too_old. This isn’t a soft timeout — it’s load-bearing. The version cache in storage servers only goes back ~5 seconds. Resolvers only track conflict sets for recent transactions. The whole system assumes transactions are short-lived.
This kills naive patterns. Don’t scan a million keys inside a single transaction and process them in Python. You’ll hit the limit and retry forever, making zero progress. Use range reads with transaction boundaries, or the getRangeAndFlatMap primitive, or better: process in chunks with explicit checkpointing.
The 10MB transaction size limit. Your total write payload (mutations) per transaction is capped at 10MB. The read and write conflict ranges add to this total. If you’re doing bulk loads, write in batches of thousands of keys per transaction, not millions.
Conflict rate under hot keys. OCC is great when conflict rates are low. If 50 concurrent transactions all read and write the same key (think: a global counter), they’ll all conflict with each other and most will retry. In practice, you need to either shard the hot key (use a set of sharded counters and aggregate) or use FDB’s atomic mutations (add, or, xor, etc.), which are conflict-free because FDB merges them at commit time without needing a read.
Sequential key patterns eat performance. FDB distributes data by key range across storage servers. If your keys are sequential (think: auto-increment IDs, timestamps), all your writes land on the same storage server — a classic hot spot. Prefix your keys with a hash of the logical ID or use a UUID. The FDB Tuple Layer (fdb.tuple.pack(...)) handles this naturally for composite keys.
Read-your-writes in a retry loop. When a transaction conflicts or hits a retriable error, you must reset all local state and start over with a fresh transaction object. If you cache reads outside the transaction and reuse them in the retry, you’re making decisions on stale data. The FDB client bindings’ run() helper or the fdb.transactional decorator handle this correctly — always use them instead of manual retry loops.
Simulation Testing: Why FDB Actually Works
One thing that separates FDB from most distributed databases: the deterministic simulation framework.
The FDB codebase is built on a custom async runtime (using coroutines, not threads) that makes time, network, and disk I/O injectable. The simulation framework runs entire cluster scenarios — faults, partitions, disk failures, process crashes — in a single-threaded, deterministic environment. The same seed gives the same outcome every time.
This means FDB can be tested against failure scenarios that would take weeks to reproduce in real hardware. The team found and fixed classes of bugs that typical integration tests would never catch. It’s a significant reason the system is as robust as it is.
If you’re curious, Josh Macdonald’s original talk at Strange Loop 2013 and the FDB technical paper go deep on this. It’s worth reading even if you never deploy FDB — the ideas apply broadly.
Quick Deployment: Docker Compose for Local Testing
If you want to experiment locally:
# docker-compose.yml — single-node FDB for dev/testing only
version: "3.8"
services:
fdb:
image: foundationdb/foundationdb:7.3.43
environment:
FDB_CLUSTER_FILE: /etc/foundationdb/fdb.cluster
volumes:
- fdb-data:/var/fdb/data
- fdb-logs:/var/fdb/logs
ports:
- "4500:4500" # FDB native port
command: >
sh -c "fdbserver -p auto:4500
-C /etc/foundationdb/fdb.cluster
-d /var/fdb/data
-L /var/fdb/logs &
sleep 3 &&
fdbcli --exec 'configure new single ssd' &&
wait"
restart: unless-stopped
volumes:
fdb-data:
fdb-logs:
For production, never run single-node. FDB’s redundancy modes (double, triple) require multiple machines, and its real strengths — fault tolerance, horizontal scaling of each role — only appear in multi-node deployments. Consult the official operations guide for production cluster sizing.
Monitoring What Matters
The fdbcli status details command gives you the most important operational view: transaction rate, conflict rate, data distribution, storage server latencies, and TLog disk usage. Watch for:
- Conflict rate > 10-20%: you have a hot key problem or transactions that are too long.
- Worst transaction duration approaching 5s: someone is doing slow work inside a transaction.
- Storage server queue depth growing: writes are outpacing storage. Time to add storage servers.
- TLog disk usage: if TLogs can’t push to storage fast enough, they buffer on disk. A sustained TLog queue is a signal to scale storage.
The fdbmonitor process handles process supervision. In production you’ll wrap it with systemd or your container orchestration, but fdbmonitor is what restarts individual FDB roles and keeps the cluster coherent after process failures.
When to Use It
FDB is the right tool when your application genuinely needs cross-key atomicity and you don’t want to build distributed transaction logic yourself. Multi-entity updates, cross-index consistency, workflows that span multiple records — these are natural fits.
It’s probably not the right tool if your access patterns are purely single-key CRUD with no cross-key invariants. You’d be paying for transaction machinery you don’t need, and a simpler KV store or a purpose-built document DB would be faster.
It’s also worth checking whether the 5-second and 10MB limits fit your workload before committing. Those aren’t soft guidelines — the system is architecturally built around them.
The fact that Apple runs iCloud on it, and Snowflake built critical infrastructure on it, should tell you something about the ceiling. When you need correctness guarantees that most databases wave away as "the application’s problem," FDB actually delivers them.