Most databases lie to you about isolation. They ship "read committed" as the default and bury "serializable" in a footnote with a performance warning attached. PostgreSQL at least gives you a working SSI implementation. MongoDB spent years calling itself a database while not supporting multi-document transactions at all. Cassandra offers "eventual consistency" with a straight face.
FoundationDB doesn’t do any of that. It gives you serializable isolation — real, linearizable, global serializability — across an arbitrarily large distributed cluster. Not as an option. As the only mode. And it does it fast enough that Apple runs its iCloud infrastructure on it.
This article tears apart how that’s actually implemented. We’ll go from the high-level architecture down to the commit protocol, look at the MVCC model, and cover the production gotchas that will bite you if you skip the docs.
Official repo: github.com/apple/foundationdb
The Core Premise: A Distributed Ordered Key-Value Store
FoundationDB’s data model is deliberately minimal: it’s a sorted, byte-string key-value store. No documents, no tables, no schemas. Every key is an arbitrary byte sequence, every value is an arbitrary byte sequence (up to 100KB), and all keys exist in a single global keyspace with lexicographic ordering.
That’s it. That’s the surface area of the native API.
This minimalism is strategic. By keeping the primitive simple, FDB can make guarantees about it that richer data models can’t provide. Everything else — documents, records, SQL — gets built on top as a layer. More on that later.
The guarantee that layer gets to rely on: any transaction that commits sees a consistent snapshot of the database at a single logical point in time, and no two committed transactions ever conflict in a way that would violate serializability. The database presents as if all transactions executed sequentially, even though they’re running concurrently across many nodes.
Architecture: Who Does What
FDB is a multi-process system. A cluster runs multiple distinct roles, and understanding them is prerequisite to understanding anything about the commit protocol.
Coordinators
These are stateless from a data perspective. Their job is to store the cluster configuration and help new processes find each other. They run Paxos to elect the Cluster Controller. You typically run three or five of them. If you lose a majority, the cluster goes read-only rather than accepting potentially conflicting writes — a deliberate safety decision.
Cluster Controller
One elected instance. Monitors process health, recruits new processes when roles fail, and manages the overall cluster topology. It’s the orchestration layer, not a data path.
Sequencer (a.k.a. Master)
This is the most architecturally interesting single component. The Sequencer generates globally monotonic transaction versions. Every committed transaction in the system gets a unique version number from the Sequencer. This is what makes global ordering possible — there’s one authoritative source of "what happened when."
The Sequencer is a single elected process. That sounds like a bottleneck, but it’s not on the commit path — it runs ahead, generating versions in batches. The actual bottleneck is the log system.
Commit Proxies
These are the front door for write transactions. A client that wants to commit sends its transaction (read set, write set, conflict ranges) to a Commit Proxy. The Proxy validates the transaction and then orchestrates the actual commit. Multiple proxies run in parallel.
GRV Proxies (Get Read Version)
Read-only operations need a read version — a snapshot timestamp — to read from. GRV Proxies vend these. They’re intentionally separate from Commit Proxies to avoid read traffic polluting write latency.
Log Servers
FDB’s durability story lives here. Log Servers are a distributed write-ahead log. When a transaction commits, it’s written to the log servers before the client gets an acknowledgment. The log servers are organized in a redundancy group (typically 3 servers for triple redundancy).
A committed transaction is durable the instant a quorum of log servers have written it to disk. Storage servers read from the log to update their on-disk state, but client commits don’t wait for that.
Storage Servers
These hold the actual key-value data. Each storage server is responsible for a range of the keyspace (a shard). Reads go directly to storage servers. Storage servers lag behind the log — they apply committed transactions asynchronously — but they serve reads at a consistent version, so you never see partially-applied state.
How Reads Work: MVCC Without a Garbage Problem
When a transaction starts, it gets a read version from a GRV Proxy. This is a timestamp in FDB’s logical clock. All reads in that transaction happen at exactly that version — the transaction sees the world as it was at that moment.
Storage servers maintain multiple versions of each key. When you read key foo at version 1000, the storage server returns the most recent value for foo that was committed at or before version 1000.
FDB limits how far back in time you can read. By default, if a read version is more than five seconds old, the storage server may have garbage-collected the historical data needed to serve it. This is where the notorious five-second transaction limit comes from. It’s not arbitrary — it’s the MVCC GC window.
This means any transaction that takes longer than five seconds to commit will fail with transaction_too_old. No exceptions. Design your transactions accordingly.
The MVCC implementation avoids the "version explosion" problem by using a two-level storage structure. Each storage server uses SQLite (or Redwood, FDB’s custom storage engine) underneath. Older versions get compacted aggressively. The log servers retain their own version of the data only until storage servers confirm they’ve applied it.
The Commit Protocol: Optimistic Concurrency at Scale
This is where FDB earns its reputation.
FDB uses Optimistic Concurrency Control (OCC). Transactions don’t take locks when they read. They execute, accumulate a read set (the keys and ranges they read) and a write set (the mutations they want to apply), then attempt to commit.
At commit time:
Step 1 — Client sends to Commit Proxy
The client sends the transaction’s read version, read conflict ranges, write conflict ranges, and mutations to a Commit Proxy. The proxy assigns the transaction a commit version by requesting one from the Sequencer. This commit version is always greater than all previously assigned commit versions — it’s a global, strictly increasing sequence.
Step 2 — Conflict checking
The Commit Proxy checks whether any key in the transaction’s read conflict ranges was modified by any transaction that committed between the transaction’s read version and its commit version.
This is the serializability check. If transaction T read key foo at version 1000, and some other transaction modified foo and committed at version 1050, and T’s commit version is 1100 — T’s read was stale. T sees a conflict and aborts.
The proxy maintains a sliding window of recent committed write sets in memory to perform this check. It’s fast — no disk I/O, pure in-memory range comparison.
Step 3 — Log servers
If no conflict, the proxy sends the transaction’s mutations to the log servers with the commit version. Once a quorum of log servers acknowledge the write, the commit is complete. The proxy responds to the client with a success.
Step 4 — Storage application
Log servers asynchronously push committed mutations to storage servers. Storage servers apply them in version order, maintaining the consistent MVCC history reads depend on.
The serializability guarantee falls directly out of this protocol. Every committed transaction gets a unique, linearly ordered commit version. Conflict checking ensures that at commit version V, the transaction’s reads are consistent with the state of the database immediately before V. The resulting history is equivalent to serial execution in commit-version order.
Watches and Read-Your-Writes
Two common patterns need special attention.
Read-your-writes is on by default. When you write key foo in a transaction and then read foo in the same transaction, you see your own write — the client caches pending mutations in memory and layers them over the snapshot reads. This is implemented entirely client-side and doesn’t cost a round trip.
Watches are a notification mechanism. You register a watch on a key; FDB notifies you when that key changes. Watches survive across transactions but not across process restarts. Under the hood, the storage server or log server holds the watch and fires a notification when the relevant version is applied. They’re cheap but not real-time — expect sub-second latency in practice, not microsecond.
Layers: Building Richer Models on a Primitive Store
FDB’s answer to "but I need documents / tables / graphs" is the layer pattern. Since the keyspace is ordered and transactions span the whole keyspace, you can encode any data structure into keys.
The canonical example is a directory layer. FDB ships one: it maps human-readable path components to short binary prefixes, giving you a namespace hierarchy without wasting keyspace on long string keys everywhere.
Apple’s Record Layer (also open source: github.com/FoundationDB/fdb-record-layer) builds a full structured record store on FDB. It handles:
- Protobuf-typed records
- Secondary indexes (maintained transactionally, so they’re always consistent — no async index builds that lag behind)
- Query planning
- Aggregate indexes
CloudKit, which backs most iCloud data, runs on the Record Layer on FoundationDB. When you understand the architecture, this makes sense: transactional secondary indexes with serializable isolation are exactly what you need when millions of clients are reading and writing data simultaneously.
Simulation Testing: Why FDB Is Actually Trustworthy
This deserves its own section because it’s genuinely unusual.
FDB was built from day one with a deterministic simulation framework. All network I/O, disk I/O, and time in the codebase go through an abstraction layer. In simulation mode, the test harness can inject arbitrary failures — network partitions, disk faults, process crashes — and replay them deterministically.
The simulation runs thousands of randomized fault scenarios per CI run. Any correctness violation (data loss, isolation violation, invariant breach) shows up as a failing simulation run with a deterministic seed you can replay to debug.
This is why FDB can make strong correctness guarantees. It’s not that the engineers are smarter — it’s that the testing framework catches bugs that would only appear under exotic failure conditions in production. Most distributed systems find those bugs when a user’s data gets corrupted.
The simulation framework is also why FDB’s codebase uses a cooperative multitasking model (actors, written in a C++ dialect called Flow) rather than preemptive threads. Cooperative scheduling is deterministically simulatable; OS threads aren’t.
Running FDB Locally
A minimal Docker Compose setup to get a single-node cluster running:
# docker-compose.yml
services:
foundationdb:
image: foundationdb/foundationdb:7.3.27
container_name: fdb
environment:
FDB_CLUSTER_FILE: /etc/foundationdb/fdb.cluster
volumes:
- fdb-data:/var/fdb
- ./fdb.cluster:/etc/foundationdb/fdb.cluster
ports:
- "4500:4500"
# FDB needs to initialize itself on first boot
command: >
sh -c "fdbserver -p auto:4500 -C /etc/foundationdb/fdb.cluster &
sleep 2 &&
fdbcli --exec 'configure new single ssd' &&
wait"
volumes:
fdb-data:
The cluster file (fdb.cluster) needs to exist before the container starts. Generate one:
# The format is: description:id@ip:port
echo "docker:[email protected]:4500" > fdb.cluster
After the cluster initializes, verify it:
docker exec -it fdb fdbcli --exec "status"
You want to see The database is available and redundancy mode single. For production, you run configure double ssd or configure triple ssd across multiple nodes.
A minimal Python transaction using the bindings:
import fdb
fdb.api_version(730)
# The cluster file tells the client how to reach the cluster
db = fdb.open('/etc/foundationdb/fdb.cluster')
# @fdb.transactional decorator handles retry on conflict
@fdb.transactional
def transfer_points(tr, from_user, to_user, amount):
from_key = b'points/' + from_user.encode()
to_key = b'points/' + to_user.encode()
from_val = int(tr[from_key] or b'0')
to_val = int(tr[to_key] or b'0')
if from_val < amount:
raise ValueError("insufficient points")
# Both writes happen atomically or not at all
tr[from_key] = str(from_val - amount).encode()
tr[to_key] = str(to_val + amount).encode()
transfer_points(db, 'alice', 'bob', 100)
The @fdb.transactional decorator is load-bearing: it automatically retries the transaction on not_committed (conflict) errors with exponential backoff. You must make the function body idempotent or side-effect-free — it may run multiple times before a commit sticks.
Gotchas
Five-second limit is absolute. If your transaction needs to read a lot of data before deciding what to write, you’ll hit this. The solution is usually to restructure: do a read-only pass first, then a short write transaction. Don’t try to do unbounded work inside a single transaction.
Transaction size is capped at 10MB. This covers the sum of all mutations and conflict ranges. Bulk loading through individual transactions won’t work past a certain point — use fdbbackup or range-clear followed by re-insert patterns for large migrations.
The conflict range is not the write set — it’s configurable. By default, FDB adds a read conflict range for every key you read. If you know a key can’t change (because you control all writers), you can skip adding it to the conflict range with snapshot reads. Snapshot reads don’t protect you from write-write conflicts but eliminate a potential false conflict source.
Write-write conflicts also abort. Two transactions that both write the same key will conflict — one commits, one retries. For high-contention keys (counters, queues), use FDB’s atomic operations (add, bit_or, etc.). Atomic ops commute, so they don’t generate write-write conflicts with each other.
Storage server read performance degrades under high version churn. If you have a workload that writes to a small hot keyrange at extremely high throughput, storage servers processing that range will spend significant time traversing MVCC version chains. Monitor roughtime_total_delay_seconds and storage_server_version_lag in your metrics. Rethink your key distribution if either is elevated.
Watches don’t work across client reconnects. If your client crashes and reconnects, outstanding watches are gone. Re-register them on startup. This is documented but easy to miss when watches are buried in application logic.
Production Deployment Considerations
For a production cluster, the main decision is redundancy mode and process placement.
triple redundancy (three log server copies) means you can lose any two log servers and keep running. For storage, shards are replicated according to the configured replication policy. Standard is triple, which replicates each shard three times across machines (or availability zones, if you configure locality).
Process placement matters. FDB respects machine/zone locality tags. Tag your processes with locality_machine and locality_datacenter. FDB will spread replicas across tags — losing a whole AZ won’t lose data if replicas are AZ-distributed.
Monitoring: FDB exposes a JSON status document at fdbcli --exec "status json". Scrape it and push to your metrics stack. Key signals: transaction commit latency percentiles, conflict rate (transactions_conflicted), storage lag (max_storage_server_queue_size), and log server write latency.
For backup, FDB has a built-in continuous backup mechanism (fdbbackup start) that streams mutations to an object store (S3, local filesystem, Blobstore-compatible). It supports point-in-time restore within the retained log range. Run this. Always. The simulation framework catches correctness bugs in FDB itself, but it doesn’t protect you from your own operators doing something stupid.
When to Use FDB (and When Not To)
FDB is the right choice when you need global transactionality that spans your entire dataset and you’re willing to accept the constraints that come with it. The five-second limit and the key-value primitive are not bugs — they’re the price of the guarantee.
It’s the wrong choice if you need full-text search, complex analytical queries, or geographic replication with async conflict resolution. It’s not Elasticsearch, it’s not Cassandra, and it’s not trying to be.
The sweet spot: systems where multiple entities write to overlapping data, correctness matters more than raw throughput, and you want the database to enforce consistency rather than pushing that complexity into your application. Distributed counters, reservation systems, inventory management, identity systems — anything where "I read this and wrote based on what I saw" needs to be atomic and serializable.
Most teams reach for Postgres and add a distributed cache and then spend years debugging cache coherence bugs. FDB offers a different trade: accept the primitive API, get correctness for free. Whether that trade is worth it depends entirely on what you’re building.