Your 25GbE link is physically capable of moving 3 GB/s between two boxes. Your actual storage benchmark says 800 MB/s and the CPU is sweating. That delta is the kernel’s fault — every byte passing between two machines on a standard TCP/IP stack has to travel through the OS, get copied, get interrupted, get scheduled. RDMA cuts all of that out. The NIC talks directly to application memory on both sides, CPU barely notices.
This article is for the self-hoster who has a couple of server-grade machines, is tired of paying the kernel tax on latency-sensitive workloads like Ceph, NVMe-oF, or ML training jobs, and wants to know if RDMA is actually worth setting up — or if it’s just datacenter theater that will eat a weekend and deliver nothing useful.
Spoiler: for most workloads, it isn’t worth it. For a handful of specific ones, it’s transformative. We’ll cover exactly which is which.
What RDMA Actually Is (and Isn’t)
RDMA — Remote Direct Memory Access — lets one machine write into another machine’s RAM without the receiving CPU doing anything. The hardware handles it. No syscalls, no kernel buffer copies, no context switches. The latency floor for a well-configured RDMA operation over a short cable is around 1–3 microseconds. TCP over a fast NIC is typically 50–200µs.
That number gap is why HPC clusters and high-frequency trading shops use it. It’s also why most home labs don’t: getting to that 1–3µs requires specific hardware, specific switch configuration, and a willingness to debug PFC frames.
There are three RDMA transports you’ll encounter:
InfiniBand is the original and fastest. Requires a dedicated IB fabric, IB switches, and IB cards. Mellanox (now NVIDIA) dominates this space. Hardware is cheap on eBay (ConnectX-2/3 cards go for $10–30), but IB switches are still expensive. IB doesn’t run on your existing Ethernet. Unless you’re building a mini HPC cluster and know what you’re doing, skip this for now.
RoCE (RDMA over Converged Ethernet) — pronounced "Rocky" — runs RDMA semantics over Ethernet. RoCEv1 is Layer 2 only; RoCEv2 wraps the payload in UDP/IP and can route between subnets. Both flavors need a lossless Ethernet fabric, which means configuring Priority Flow Control (PFC) on your NICs and switches. If your switch doesn’t support PFC, RoCEv2 will work at degraded performance, and RoCEv1 will be unreliable.
iWARP runs RDMA over TCP. TCP’s reliability removes the lossless-fabric requirement, which means iWARP works on commodity switches without any special configuration. The downside: TCP adds latency and CPU overhead compared to RoCE. You still get kernel bypass and direct memory access — just with a higher floor. Hardware iWARP offload is available on Chelsio T5/T6 and Intel X722 cards. The kernel also ships a pure software iWARP driver (siw) that runs on any NIC.
Hardware Reality Check
For RoCE, you want a Mellanox/NVIDIA ConnectX-4 or newer. ConnectX-4 Lx (25GbE) cards are $30–80 used and are the sweet spot. ConnectX-3 supports RoCE but the driver situation is increasingly messy on modern kernels. Avoid it.
For iWARP with hardware offload, Chelsio T520 or T540 cards are solid and go for under $50 used. Intel X722 works but the irdma driver had a rocky history — make sure you’re on kernel 5.15+ before trusting it.
If you just want to experiment or your NIC doesn’t support RDMA natively, the kernel ships two software transports:
rxe— Soft-RoCEv2, works on any Ethernet NICsiw— Software iWARP, same
These are test-and-develop tools. You won’t get real RDMA performance from them on a stock NIC because the NIC still involves the CPU for every packet. But you can validate your application configuration before buying hardware.
For switches: RoCEv2 with no lossless config will run, just with higher tail latency and occasional retransmits. If you want the real numbers, you need a switch that supports PFC and ECN — managed switches from Mikrotik CRS3xx series, Netgear M4300, or secondhand datacenter gear all support it. A $30 unmanaged switch will cap your gains.
Setting Up Soft-RoCE (rxe) for Testing
Install the userspace stack first. On Debian/Ubuntu:
apt install rdma-core ibverbs-utils perftest
On RHEL/Fedora/Rocky:
dnf install rdma-core libibverbs-utils perftest
Load the module and attach it to your Ethernet interface:
modprobe rdma_rxe
# Attach rxe to your NIC (replace eth0 with your interface name)
rdma link add rxe0 type rxe netdev eth0
Verify it came up:
rdma link show
# Should show: link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0
ibv_devices
# Should list rxe0
To make this survive reboots, create a systemd service or a udev rule. The rdma-core package on some distros ships /etc/rdma/rdma.conf — set RDMA_AUTOLOAD_MODULES=yes and RXE_AUTOLOAD=eth0 there.
Setting Up iWARP with siw (Software iWARP)
siw is even simpler because it doesn’t need a netdev attachment command — the kernel does it automatically for every interface:
modprobe siw
# siw creates a device for each network interface automatically
ibv_devices
# You'll see siw_eth0, siw_eth1, etc.
Test connectivity between two machines. On the server:
ib_send_lat -d siw_eth0 --iters=1000
On the client:
ib_send_lat -d siw_eth0 --iters=1000 <server-ip>
You won’t see impressive numbers with siw over a regular NIC — expect 50–100µs, similar to TCP. Again, this is for validating that your application stack speaks RDMA correctly before you commit to hardware.
Setting Up RoCEv2 on Mellanox ConnectX
This is where real performance lives. Install the inbox kernel drivers — if you’re on kernel 5.15+, mlx5_core is included and works well. The vendor OFED stack (MLNX_OFED) gives slightly better performance but is a maintenance headache: it conflicts with inbox drivers, needs rebuilding on every kernel update, and the installer is quirky. Start with inbox drivers unless you have a specific need.
Verify the card is detected:
lspci | grep -i mellanox
ibv_devices
# Should show mlx5_0 or similar
Assign an IP to the interface (ConnectX cards in Ethernet mode are just NICs):
ip addr add 10.10.0.1/24 dev enp5s0f0
ip link set enp5s0f0 up
Enable jumbo frames — strongly recommended for RoCE:
ip link set enp5s0f0 mtu 9000
Both sides need matching MTU. If your switch doesn’t pass 9000-byte frames, you’ll get silent packet drops. Test with:
ping -M do -s 8972 10.10.0.2 # 8972 + 28 bytes header = 9000
Configuring PFC for Lossless Fabric
RoCE is UDP-based. UDP doesn’t retransmit. A single dropped packet means a hung operation or a timeout. This is why lossless Ethernet matters.
PFC (Priority Flow Control) makes the switch send a PAUSE frame to a sender when a buffer fills, instead of dropping. Configure it on the NIC:
# Enable PFC on traffic class 3 (conventional for RoCE)
mlnx_qos -i enp5s0f0 --pfc 0,0,0,1,0,0,0,0
# Enable ECN (Explicit Congestion Notification)
# ECN signals congestion before buffers fill — cleaner than PFC alone
echo 1 > /sys/class/net/enp5s0f0/ecn/roce_np/enable/3
echo 1 > /sys/class/net/enp5s0f0/ecn/roce_rp/enable/3
Your switch also needs PFC enabled on the port. The exact CLI varies by vendor, but the concept is the same: map DSCP 26 (or your chosen value) to a PFC-enabled queue.
Without this, RoCEv2 still works — you just get occasional retransmits and higher tail latency under load. For casual use or NVMe-oF with light traffic, it’s often good enough.
Benchmarking
On server:
ib_write_bw -d mlx5_0 -F --report_gbit
On client:
ib_write_bw -d mlx5_0 -F --report_gbit <server-ip>
On a pair of ConnectX-4 Lx 25GbE cards with PFC configured and 9000 MTU, you should see ~24 Gbps and latency around 3–5µs for ib_write_lat. If you’re seeing 10–15 Gbps, check MTU mismatches first, then PFC configuration.
The Use Case That Makes RDMA Worth It: NVMe-oF
NVMe-oF (NVMe over Fabrics) is the single best reason a self-hoster would set up RDMA. You expose an NVMe SSD from one machine over the network, and it appears as a local block device on another — with latency close to local access.
iSCSI gives you roughly 200–500µs latency for a 4K random read. NVMe-oF over RoCE delivers 10–20µs. That’s not a 2x improvement; it’s an order of magnitude.
NVMe-oF Target (Server)
# Load the required modules
modprobe nvmet
modprobe nvmet-rdma
# Create a subsystem
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0/attr_allow_any_host
# Create a namespace pointing to your NVMe device
mkdir /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0/namespaces/1
echo /dev/nvme0n1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0/namespaces/1/device_path
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0/namespaces/1/enable
# Create an RDMA port and link the subsystem
mkdir /sys/kernel/config/nvmet/ports/1
echo rdma > /sys/kernel/config/nvmet/ports/1/addr_trtype
echo 10.10.0.1 > /sys/kernel/config/nvmet/ports/1/addr_traddr
echo 4420 > /sys/kernel/config/nvmet/ports/1/addr_trsvcid
echo ipv4 > /sys/kernel/config/nvmet/ports/1/addr_adrfam
ln -s /sys/kernel/config/nvmet/subsystems/nqn.2026-05.io.local:nvme0 \
/sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-05.io.local:nvme0
NVMe-oF Initiator (Client)
apt install nvme-cli # or dnf install nvme-cli
# Discover available subsystems
nvme discover -t rdma -a 10.10.0.1 -s 4420
# Connect
nvme connect -t rdma -a 10.10.0.1 -s 4420 \
-n nqn.2026-05.io.local:nvme0
# Verify
nvme list # Your remote NVMe appears as /dev/nvme1n1 or similar
Run fio against the remote device and compare with iSCSI. The difference in 4K random latency will be obvious.
Gotchas
PFC must be symmetric. If the server NIC has PFC enabled but the client doesn’t, you’ll get erratic performance — PAUSE frames come in but the other side doesn’t understand the queue mapping. Configure both ends identically.
MTU mismatches are silent killers. RoCEv2 packets don’t fragment. If any hop in the path has a lower MTU than 9000, packets are silently dropped and you’ll see terrible performance with no obvious error. Always validate MTU end-to-end before concluding anything is broken.
mlx5_core vs MLNX_OFED can’t coexist. If you install OFED, it replaces inbox drivers. If you then do a kernel update without rebuilding OFED, your RDMA devices disappear on next boot. Stick to one or the other. Inbox drivers are fine for self-hosted workloads.
RoCE over a regular unmanaged switch degrades under load. Without PFC, buffer overflow under bursty traffic causes packet drops, which causes RDMA retransmits, which causes latency spikes. Your average latency looks fine; your P99 is garbage. If you care about consistent latency, budget for a managed switch.
Soft-RoCE (rxe) blocks real RDMA traffic on the same NIC. If you load rdma_rxe and attach it to an interface, and later add a real RDMA NIC on the same machine, the device numbering can conflict. Clean up software devices before mixing hardware RDMA.
NVMe-oF target configuration doesn’t persist across reboots. The /sys/kernel/config/nvmet/ tree is in memory. You need a startup script or systemd service to recreate it. There are tools like nvmetcli that manage this, but they add a dependency. A simple ExecStart script with the above commands works fine.
The ib_send_lat default is multi-threaded. For accurate single-operation latency measurements, use --iters=1000 -t 1. The default runs enough threads to saturate bandwidth, which inflates latency numbers.
When the Complexity Pays Off
Be honest with yourself before starting. RDMA configuration is not hard, but it’s fiddly — switch config, NIC firmware, module loading, and MTU all interact. Here’s where it’s worth it:
NVMe-oF — yes, absolutely. Even with siw over TCP you save CPU cycles compared to iSCSI. With real RoCE hardware it’s genuinely local-disk-class latency over the network. Excellent for a storage server feeding a compute node.
Ceph with RDMA messenger — Ceph has an RDMA-capable messenger (AsyncMessenger with rdma backend). It reduces OSD-to-OSD latency and cuts CPU overhead on replicated writes. Real gains on 3+ node clusters. The configuration is messier than NVMe-oF and Ceph’s RDMA support has had rough patches — check your version’s release notes.
AI/ML gradient synchronization — If you’re running distributed training across two beefy servers, NCCL (NVIDIA’s collective communication library) supports RDMA and it makes a real difference at large batch sizes. Not typical self-hoster territory, but not unheard of.
Plain file serving, web apps, databases — no meaningful gain. The bottleneck is query planning, disk seeks, or application code, not kernel network overhead. Don’t bother.
VM live migration — some hypervisors support RDMA for live migration. It reduces migration time and memory dirtying window. Interesting, but niche.
If your use case isn’t on that list, you’re probably better off with a 10GbE or 25GbE link and a fast NIC with hardware offloads (checksums, TSO, RSS) properly enabled. That gets you 90% of the bandwidth with none of the configuration complexity.
Quick Reference: Choosing Your Transport
| Scenario | Best transport | Hardware requirement |
|---|---|---|
| Testing/dev only | Soft-RoCE (rxe) or siw | Any NIC |
| NVMe-oF, budget setup | iWARP / siw over TCP | Any NIC (software) |
| NVMe-oF, production | RoCEv2 | ConnectX-4+ or Chelsio |
| Ceph cluster | RoCEv2 | ConnectX-4+ |
| ML training (NCCL) | RoCEv2 | ConnectX-5+ recommended |
| Works without managed switch | iWARP | Chelsio T5/T6 or siw |
Wrapping Up
RDMA is not voodoo. The Linux kernel ships everything you need — mlx5_core, siw, rxe, nvmet-rdma — and the userspace tools in rdma-core plus perftest cover configuration and benchmarking. The learning curve is real, but it’s mostly about understanding why lossless fabrics matter for RoCE and then configuring PFC correctly.
Start with Soft-RoCE or siw on whatever hardware you have, run ib_write_lat between two machines, and benchmark your actual application. If the latency improvement matters to your workload — and especially if you’re setting up NVMe-oF — buy a pair of used ConnectX-4 Lx cards for $60 total and a managed switch that supports PFC. The performance jump is not marginal.
If the benchmark shows no improvement in your application’s actual response time, close the browser tab and go do something else. RDMA is a tool for specific problems, not a general networking upgrade.