Most databases pick a lane: either they give you SQL with real transactions and sacrifice horizontal scalability, or they give you a distributed NoSQL store that scales linearly but leaves consistency to you. FoundationDB refuses that tradeoff. It’s a distributed key-value store that offers full ACID semantics, including strict serializable isolation — the strongest isolation guarantee on the table — without making you write compensating transactions or deal with application-level conflict resolution.
The GitHub repo is at apple/foundationdb. Apple open-sourced it in 2018 after acquiring it in 2015, and it quietly powers a shocking amount of production infrastructure — iCloud’s metadata layer among other things.
This isn’t a "getting started with FDB" tutorial. That’s what the docs are for. This is about understanding how the engine actually works — the kind of knowledge that makes you dangerous when something breaks at 3 AM, or when you need to reason about whether FDB fits a particular workload.
The Founding Abstraction: Ordered Key-Value Pairs
Before digging into the distributed magic, understand what FoundationDB actually stores: an ordered map of byte-string keys to byte-string values. That’s it. No schemas, no secondary indexes at the engine level, no documents. The ordering is critical — it lets you do efficient range reads, which is the primitive everything else is built from.
This sparse simplicity is intentional. The ordered KV store is the substrate. Everything richer — relational tables, document storage, graph structures — gets implemented as a "layer" on top. The most significant layer is the Record Layer, which adds a typed schema model and is what powers much of FoundationDB’s use at Apple. But the engine itself stays blissfully unaware of any of this.
Why ordered? Because range reads are the universal interface. Want a table? Map (table_id, primary_key) → row. Want a secondary index? Add entries with (table_id, index_name, indexed_value, primary_key) → empty. All of it falls out of the ordered KV model.
Cluster Roles: The Cast of Characters
A running FDB cluster has several distinct roles, each a separate process (or group of processes). Understanding what each one does is the entry point to understanding how transactions work.
Coordinators are the one piece of stable infrastructure. They don’t handle data — they hold the cluster’s configuration and help processes elect leaders. Their addresses are the thing you hardcode in your connection file. They use an Active Disk Paxos variant to maintain cluster state. You need a majority alive for the cluster to function, so odd numbers (3 or 5) make sense here.
The Cluster Controller is elected by the coordinators. It’s the cluster’s orchestration brain — it monitors all other processes, recruits workers into roles, and detects failures. If a worker dies, the cluster controller figures out that it’s gone and recruits a replacement.
The Master manages the global transaction state. Specifically, it generates read versions (more on this in a moment) and coordinates log system epoch changes. In newer FDB versions (7.x+), its responsibilities were restructured, with some functions split across Commit Proxies and GRV Proxies (Get Read Version proxies), but the conceptual model is the same.
GRV Proxies hand out read versions to clients who are starting a transaction. A read version is essentially a timestamp — a monotonically increasing integer that determines which snapshot of the database a transaction will read from.
Commit Proxies accept commit requests from clients. They run conflict detection and, if a transaction is clean, write it to the Transaction Logs (TLogs).
Resolvers are the conflict detection engine. When a commit proxy gets a commit request, it sends the transaction’s read and write conflict ranges to the resolvers, which check them against recent committed transactions and return a verdict: commit or abort.
Transaction Logs (TLogs) are an append-only write-ahead log, sharded and replicated. Once a transaction is committed by a resolver and written here, it’s durable. The TLogs are the source of truth until storage servers have persisted the data.
Storage Servers actually store key-value data and serve reads. Each storage server is responsible for a range of keys (a "shard") and maintains a local SQLite or RocksDB (depending on your FDB version) plus an in-memory mutation buffer that it drains from the TLogs.
The Transaction Lifecycle: Step by Step
Here’s what happens when your application commits a transaction.
1. Get a Read Version
The client calls getReadVersion() (or the SDK does it implicitly on the first read). This sends a request to a GRV proxy, which batches multiple client requests together and gets a batch read version from the master. The version returned is a monotonically increasing integer — think of it as a logical timestamp. Your transaction will see the database as it existed at this exact version. Nothing committed after this version is visible to you.
This is MVCC in action. Storage servers keep multiple versions of each key. When you read, you ask for the value as of your read version, and you get a consistent snapshot.
2. Perform Reads
Reads go directly to the storage servers. Your client has a copy of the shard map (the key range → storage server assignment), so it can route the request directly without going through any proxy. This is important for performance — reads scale out across your storage servers with no bottleneck.
As you read, the FDB client library builds up a read conflict set — the set of keys and ranges your transaction has observed. If any of those keys change between when you read them and when you commit, your transaction will be aborted and must be retried.
The client also maintains a read-your-writes cache locally. If your transaction writes a key and then reads it, you’ll see your own write — even though it hasn’t been committed. This sounds obvious but it’s something that needs to be explicitly implemented in an MVCC system.
3. Stage Mutations
Writes are purely local at this stage. You call set(key, value) or clear(key) or any of the atomic operations — these don’t go to the network yet. They accumulate in the client library’s mutation buffer. The client also builds up a write conflict set: the ranges your transaction is writing to. If another transaction reads from a key in your write conflict set before you commit, its subsequent commit will fail (because it saw a stale version of the data you’re about to change).
4. Commit
When you call commit(), the client sends the full transaction package to a commit proxy:
- The read version
- All mutations
- The read conflict range
- The write conflict range
The commit proxy does several things here. First, it assigns a commit version — always greater than the read version, and representing the logical time at which this transaction is serialized into the history. Then it sends the conflict data to the resolvers.
5. Conflict Detection
The resolvers maintain a sliding window of recent write conflict ranges — roughly the last 5 seconds of commits. For your incoming transaction, they check: did anything in your read conflict set get modified by a transaction that committed after your read version? If yes, you have a conflict. The transaction is rejected with a not_committed error, and you’re expected to retry from scratch (new read version, new reads, new commit).
If there’s no conflict, the resolvers confirm the commit, and the commit proxy writes the mutations to the transaction logs.
This is optimistic concurrency control. You don’t take locks when you read. You proceed optimistically and check for conflicts only at commit time. This is great for workloads with low contention — you avoid the overhead of lock management entirely. It’s expensive for workloads with high contention on the same keys because you’ll be doing lots of retried work.
6. Durability via Transaction Logs
Once the TLogs have written the mutations to durable storage (disk-synced on each log server in the replica set), the commit is acknowledged to the client. Your data is safe. The TLogs are replicated, and the cluster is configured with a replication factor — typically 2 for small clusters, 3 for production.
Storage servers asynchronously pull mutations from the TLogs and apply them to their local storage. Reads during this brief lag can still be served from the TLog’s in-memory mutation buffer, which the storage servers cache while catching up.
Why This Achieves Serializable Isolation
Serializable isolation means: any concurrent execution of transactions produces results equivalent to some serial execution of those same transactions. No anomalies — no phantom reads, no write skew, nothing.
FDB achieves this through the combination of:
- Global read versions — all reads within a transaction see the same consistent snapshot
- Commit versions — every committed transaction gets a single global timestamp
- Conflict detection — any case where the serial order would have produced a different read result causes an abort
The key insight is that commit_version > read_version for every committed transaction. This means the ordering of transactions is not just logical — it’s strict. If transaction A commits before transaction B starts reading, B’s read version will be ≥ A’s commit version, and B will see A’s writes. The system’s total order of transactions is well-defined and consistent with real time (roughly — the system doesn’t guarantee strict real-time ordering beyond what the Paxos-based version generation provides).
Write skew — the classic serializable isolation bug — is prevented by the read conflict set. If you read key X and write to key Y based on what you read from X, your read conflict set includes X. If someone else writes X before you commit, your transaction aborts. No write skew.
Gotchas
The 5-second transaction limit. FDB has a hard limit: transactions must complete within 5 seconds of their read version, or they’ll be rejected with a transaction_too_old error. The resolvers only keep the last ~5 seconds of conflict history, so they can’t check conflicts for older transactions. This is a fundamental property of the engine, not a tunable timeout. If your workload requires long-running transactions that hold locks for minutes, FDB is the wrong database.
Large transactions. There’s a 10 MB limit on transaction size (mutations + conflict ranges combined). Writes larger than this need to be chunked across multiple transactions. Your application code needs to handle this. The record layer handles it for you; raw FDB usage doesn’t.
Conflict rate under contention. If many transactions are racing to write the same hot key, most of them will be retrying constantly. This is the fundamental tradeoff with optimistic concurrency. FDB has atomic operations (add, bitwise operations, compare-and-clear) that resolve some hot-key scenarios without conflicts — use them where semantics allow. For truly high-write-contention scenarios, application-level sharding of the hot key is the right answer.
Read conflict ranges are additive. Every key you read expands your conflict set. Long sequential scans over large ranges mean a huge conflict range, which increases the probability of conflicts with concurrent writers in that range. Be surgical with your reads. If you know you’ll be working with a subset of a range, read only that subset.
Storage server lag during high write throughput. Storage servers pull from TLogs asynchronously. During bursts of writes, storage servers can fall behind. FDB has a mechanism called "storage server lag" that can cause the cluster to slow down or stop accepting writes if lag exceeds a threshold — this is a backpressure mechanism, not a bug, but it’ll surprise you the first time you see your write throughput drop to zero with a proxy_memory_limit_exceeded or similar error.
Cluster recovery time. When a critical role (master, commit proxy, resolver) fails, the cluster needs to complete a recovery before it can accept writes again. Recovery involves electing new processes into the failed roles and replaying the TLog from the last epoch. This typically takes 1-10 seconds on a healthy cluster. During this time, reads usually still work (storage servers can serve stale reads), but writes are blocked. Design your applications to tolerate brief write stalls.
Production Architecture Recommendations
Don’t collocate TLogs and storage servers on the same machines. TLogs are write-intensive, fsync-heavy workloads. Storage servers are read-heavy with random I/O. They compete for disk bandwidth and cache in unfortunate ways.
Coordinators should be on completely separate, small, stable machines — or at minimum on machines that are unlikely to fail simultaneously with your main FDB cluster. A 3-coordinator setup on your 3 main FDB machines defeats the purpose if all 3 go down together.
Use SSDs. FDB is designed for SSDs. The fsync latency on HDDs makes TLog performance miserable — you’re synchronously fsyncing every commit to multiple machines.
The fdbcli status json output is your friend for diagnosing issues. It exposes everything: data distribution, replication state, process roles, latency percentiles, storage server lag. Pipe it into Prometheus via fdb-prometheus-exporter and set up alerts on storage server lag, commit latency, and transaction conflict rate.
The Layer Architecture in Practice
One thing that makes FDB unusual is that it deliberately doesn’t solve higher-level concerns at the engine level. Indexes, schemas, data types — all layers. This sounds like a cop-out until you realize it means the engine stays simple and correct, and layers can be versioned and upgraded independently of the storage engine.
If you’re building on raw FDB, you almost certainly want a tuple layer (FDB ships one in its language bindings) for key encoding. This gives you type-safe composite keys — (namespace, user_id, "profile") — that sort lexicographically in a useful way. Without this, you’re manually managing byte encodings and getting bitten by integer sorting bugs.
For anything more structured, seriously evaluate the Record Layer before rolling your own. It handles indexes, schema evolution, and query planning on top of FDB’s primitives, and it’s been production-hardened at Apple’s scale.
When to Use It (and When Not To)
FDB is exceptional when you need strong consistency across multiple keys atomically, your data fits a key-value or tuple model (possibly via layers), you need horizontal scalability without eventual consistency, and your transactions are short (under a second is comfortable, 5 seconds is the hard ceiling).
It’s a bad fit if your workload is dominated by long-running analytical queries, if you need SQL joins at the engine level (use PostgreSQL), or if your transactions routinely touch high-contention hot keys under heavy write load (optimistic concurrency will murder your throughput).
The most underappreciated use of FDB is as a coordination substrate for distributed systems — distributed locks, leader election, configuration management — where you need something that’s genuinely consistent and you can use the transactional semantics to build correct distributed primitives without the usual nightmare of race conditions.
FDB delivers on the promise of "boring correctness." It doesn’t have flashy query features or a rich type system. What it has is a rigorous correctness story and a simulation testing framework (their deterministic simulation tool is genuinely one of the most sophisticated testing systems in any production database) that gives you unusual confidence the engine won’t silently corrupt your data. In a landscape full of databases that quietly drop the ball on their consistency guarantees under edge cases, that’s worth a lot.