Stop Leaving Bandwidth on the Table: Linux 6.x NIC Tuning for 10G and 40G Networks

You paid for a 10G or 40G NIC. Your switch port is negotiated at full speed. But your throughput benchmarks cap out at 3–4 Gbps, your latency spikes under load, and softirq is eating a full core. The hardware is fine. Linux out-of-the-box defaults are just tuned for a 1G world circa 2008.

This guide covers the exact knobs that matter — TCP buffer math, IRQ pinning, ring buffer sizing, RPS/RFS, and interrupt coalescing. Everything is based on Linux 6.x kernel behavior. The settings here are running in production on bare-metal servers with Mellanox ConnectX-5 (25G) and Intel X710 (10G) adapters, but the principles apply to any high-speed NIC.

No magic "paste this sysctl.conf and go home" lazy guides here. You need to understand what you’re setting, or you’ll tune yourself into a worse state than the defaults.

The Math You Actually Need

Before touching a single sysctl, understand the Bandwidth-Delay Product (BDP). It answers: how much data can be in-flight between two endpoints at any moment?

BDP = bandwidth × RTT

For a 10G link at 1ms RTT: 1.25 GB/s × 0.001s = 1.25 MB
For a 10G link at 10ms RTT (cross-DC): 1.25 GB/s × 0.01s = 12.5 MB
For a 40G link at 10ms RTT: 5 GB/s × 0.01s = 50 MB

TCP can only fill the pipe if its send/receive buffers are at least as large as the BDP. The kernel auto-tunes socket buffers — but only up to net.core.rmem_max and net.core.wmem_max. If those ceilings are 212992 bytes (the default, ~208 KB), you’ll saturate a 10G link only at sub-millisecond RTT. Everything else stalls waiting for ACKs.

TCP Buffer Tuning

These are the core sysctl values. Put them in /etc/sysctl.d/99-network-tuning.conf so they survive reboots and don’t mix with distro defaults.

# /etc/sysctl.d/99-network-tuning.conf
# Verified on Linux 6.6 / 6.8 with 10G and 40G NICs

# ---------------------------------------------------------------
# Socket-level limits: allow autotuning up to 128 MB per socket.
# For 40G at 10ms RTT you need ~100 MB; 128 MB gives headroom.
# ---------------------------------------------------------------
net.core.rmem_default = 262144
net.core.rmem_max = 134217728
net.core.wmem_default = 262144
net.core.wmem_max = 134217728

# ---------------------------------------------------------------
# TCP-level buffer ranges: [min, default, max] in bytes.
# The kernel autotuning lives between default and max.
# min: reserved even under memory pressure.
# default: starting size for new connections.
# max: ceiling for autotuning (must match rmem_max/wmem_max).
# ---------------------------------------------------------------
net.ipv4.tcp_rmem = 4096 262144 134217728
net.ipv4.tcp_wmem = 4096 262144 134217728

# ---------------------------------------------------------------
# Global TCP memory limit: [min, pressure, max] in pages (4KB).
# Pressure kicks in when the total TCP memory hits the middle value.
# Max is the hard ceiling across ALL sockets combined.
# Rule of thumb: set max to ~25% of total RAM.
# Adjust these if your server has a different RAM size.
# Example below is calibrated for a 64 GB server.
# ---------------------------------------------------------------
net.ipv4.tcp_mem = 786432 1572864 4194304

# ---------------------------------------------------------------
# Backlog and connection queues.
# netdev_max_backlog: packets queued on a NIC before the kernel
#   starts dropping them. Critical on 10G+ burst traffic.
# somaxconn: listen() backlog ceiling. Many apps default to 128 —
#   override at the app level too, not just here.
# ---------------------------------------------------------------
net.core.netdev_max_backlog = 65536
net.core.somaxconn = 65535

# ---------------------------------------------------------------
# Congestion control and qdisc.
# BBR is dramatically better than CUBIC on WAN and mixed paths.
# fq (Fair Queue) is the right pairing for BBR — do not use pfifo.
# ---------------------------------------------------------------
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# ---------------------------------------------------------------
# Miscellaneous TCP options worth enabling.
# ---------------------------------------------------------------
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fastopen = 3        # client + server TFO
net.ipv4.tcp_tw_reuse = 1        # reuse TIME_WAIT sockets (outbound)
net.ipv4.ip_local_port_range = 10000 65535
net.ipv4.tcp_fin_timeout = 15

Apply immediately without rebooting:

sysctl --system
# or just the new file:
sysctl -p /etc/sysctl.d/99-network-tuning.conf

Gotcha — BBR requires the tcp_bbr module: On some minimal installs or container hosts it isn’t loaded by default. lsmod | grep bbr will tell you. If empty: modprobe tcp_bbr and add tcp_bbr to /etc/modules-load.d/bbr.conf for persistence.

Gotcha — tcp_tw_reuse is safe for outbound, not inbound: Setting it to 1 only recycles TIME_WAIT sockets for new outbound connections. Setting it to 2 (apply to both directions) is a footgun — don’t do it on a server that handles inbound connections from untrusted clients.

NIC Ring Buffers and Queues

The kernel’s TCP buffers are only half the story. The NIC itself has hardware ring buffers for RX and TX descriptors. If a burst fills the ring before the NAPI poll loop drains it, packets are silently dropped at the hardware level — you’ll see this as rx_missed_errors in ethtool -S eth0.

Check current and maximum ring buffer sizes:

ethtool -g eth0
# Ring parameters for eth0:
# Pre-set maximums:
# RX:     8192
# TX:     8192
# Current hardware settings:
# RX:     512
# TX:     512

On a 10G NIC, 512 descriptors at ~1500-byte frames gives you roughly 768 KB of buffer — that’s gone in under 1ms of full-rate traffic. Set them to the hardware maximum:

ethtool -G eth0 rx 4096 tx 4096

Persist this via a systemd service or @reboot cron:

# /etc/systemd/system/nic-tune.service
[Unit]
Description=NIC tuning for eth0
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool -G eth0 rx 4096 tx 4096
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

systemctl enable --now nic-tune.service

Multi-queue NICs (the normal case on 10G+): Modern NICs expose multiple hardware queues via MSI-X. Check how many are active:

ethtool -l eth0
# Channel parameters for eth0:
# Pre-set maximums:
# Combined: 32
# Current hardware settings:
# Combined: 4

If you have 16 CPU cores and the NIC supports 16 combined queues, use them:

ethtool -L eth0 combined 16

More queues = less contention on the RX/TX path. The kernel’s irqbalance will then spread these queues across CPUs automatically — unless you want to do it manually, which brings us to the next section.

IRQ Affinity — The Part Everyone Gets Wrong

On a multi-queue NIC, each hardware queue generates its own interrupt. By default, Linux 6.x with irqbalance running will distribute these across CPUs, but it uses generic load-balancing heuristics that don’t account for NUMA topology or the specific CPU your workload is pinned to.

The better approach for latency-sensitive or high-throughput workloads: pin each NIC queue’s IRQ to a specific CPU core manually, stop irqbalance from touching the NIC, and make sure those CPUs are on the same NUMA node as the NIC.

Step 1 — Find your NIC’s NUMA node and IRQs:

# Which NUMA node does the NIC live on?
cat /sys/class/net/eth0/device/numa_node
# 0

# List IRQs for the NIC (replace eth0 with your driver name as shown in /proc/interrupts)
grep -i eth0 /proc/interrupts
# 120:  0  1234567  0  0   PCI-MSI 524288-edge  eth0-TxRx-0
# 121:  0  987654   0  0   PCI-MSI 524289-edge  eth0-TxRx-1
# ...

Step 2 — Find CPUs on that NUMA node:

numactl --hardware | grep "node 0 cpus"
# node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Step 3 — Pin each IRQ to a dedicated CPU core:

#!/bin/bash
# pin-nic-irqs.sh — pin eth0 IRQs to NUMA-local CPUs, one IRQ per core

NIC="eth0"
# CPUs to use (NUMA node 0, cores 1-8; leave core 0 free for OS)
CPUS=(1 2 3 4 5 6 7 8)

IRQ_LIST=$(grep -i "${NIC}" /proc/interrupts | awk '{print $1}' | tr -d ':')

i=0
for IRQ in $IRQ_LIST; do
    CPU=${CPUS[$((i % ${#CPUS[@]}))]}
    # smp_affinity takes a hex bitmask; 1<<CPU
    MASK=$(printf "%x" $((1 << CPU)))
    echo "IRQ ${IRQ} → CPU ${CPU} (mask 0x${MASK})"
    echo "$MASK" > /proc/irq/${IRQ}/smp_affinity
    ((i++))
done

chmod +x pin-nic-irqs.sh
sudo ./pin-nic-irqs.sh

Step 4 — Tell irqbalance to leave these IRQs alone:

irqbalance will undo your manual pinning unless you ban it:

# /etc/default/irqbalance
IRQBALANCE_BANNED_CPUS=""
IRQBALANCE_ARGS="--banirq=120 --banirq=121 --banirq=122 --banirq=123"

Or use the IRQBALANCE_BANNED_INTERRUPTS environment variable in the systemd unit. The cleanest solution for dedicated network servers is to stop irqbalance entirely and manage everything with your own script via the nic-tune.service above.

Gotcha — smp_affinity vs smp_affinity_list: /proc/irq/N/smp_affinity takes a hex bitmask. /proc/irq/N/smp_affinity_list takes a human-readable CPU list like 0-3,8. The list format is far less error-prone for multi-socket servers with more than 64 CPUs where the bitmask overflows a single 64-bit word.

# Easier: use smp_affinity_list
echo "2" > /proc/irq/120/smp_affinity_list   # pin to CPU 2
echo "2-3" > /proc/irq/121/smp_affinity_list  # spread across CPUs 2 and 3

RPS and RFS — Software Scaling for Single-Queue NICs

If you’re stuck with a single-queue NIC (or a virtual NIC in a VM), you don’t get hardware multi-queue. RPS (Receive Packet Steering) does the scaling in software by hashing packets across CPUs.

RFS (Receive Flow Steering) goes further: it routes packets to the CPU that’s actually running the socket that will consume them, which keeps data hot in cache and reduces cross-CPU bouncing.

Enable RPS on all RX queues:

# Spread across all CPUs — use a bitmask covering all cores
# For a 16-core system: 0xFFFF = 65535
for QUEUE in /sys/class/net/eth0/queues/rx-*; do
    echo "ffff" > ${QUEUE}/rps_cpus
done

Enable RFS:

# Global flow table — should be a power of 2, at least 16x the max connections
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries

# Per-queue flow count (divide global by number of queues)
# Assuming 4 RX queues: 32768 / 4 = 8192
for QUEUE in /sys/class/net/eth0/queues/rx-*; do
    echo 8192 > ${QUEUE}/rps_flow_cnt
done

XPS (Transmit Packet Steering) is the TX equivalent — it ensures a socket sends on the same CPU it receives on:

# Pin TX queue 0 to CPU 0, queue 1 to CPU 1, etc.
for i in $(seq 0 $(($(ls -d /sys/class/net/eth0/queues/tx-* | wc -l) - 1))); do
    MASK=$(printf "%x" $((1 << i)))
    echo "$MASK" > /sys/class/net/eth0/queues/tx-${i}/xps_cpus
done

Gotcha — RPS and hardware multi-queue don’t mix well: If your NIC already has hardware queues and you’ve done proper IRQ affinity, enabling RPS adds overhead without benefit. RPS is for single-queue or virtual NICs. Pick one approach.

Interrupt Coalescing — Latency vs. Throughput

By default, most NICs generate an interrupt for every single received packet. At 10M packets/second (realistic on a 10G NIC with small packets), that’s 10M interrupts/second per queue core — enough to saturate a CPU just on interrupt handling.

Interrupt coalescing batches packets before firing an interrupt. This trades latency for throughput.

# Check current coalescing settings
ethtool -c eth0

# Rx-usecs: fire interrupt after this many microseconds since last packet
# Rx-frames: OR after this many packets, whichever comes first
ethtool -C eth0 rx-usecs 50 rx-frames 64 tx-usecs 50 tx-frames 64

For pure throughput (storage replication, bulk transfer): push rx-usecs up to 100–200µs. For latency-sensitive workloads (trading, databases): drop it to 1–10µs or even rx-usecs 0 (immediate). Most production servers land somewhere in between — 50µs is a good starting point.

Offloads

Check what’s enabled:

ethtool -k eth0

For 10G/40G workloads, these should be on:

ethtool -K eth0 \
    tso on \       # TCP Segmentation Offload
    gso on \       # Generic Segmentation Offload
    gro on \       # Generic Receive Offload
    rx-checksumming on \
    tx-checksumming on

LRO (Large Receive Offload) merges incoming TCP segments in the NIC firmware. It’s great for throughput but can interfere with traffic shaping and firewall rules because it modifies packet sizes before they hit the kernel. If you’re running tc, iptables, or any middlebox-like processing — disable LRO, keep GRO:

ethtool -K eth0 lro off gro on

Gotcha — GRO can mask high PPS: GRO coalesces packets in the kernel, so iftop and similar tools will show lower PPS than the wire. Don’t confuse this for reduced load — the NIC is still processing every individual packet.

CPU Frequency Scaling

Network interrupt handling and the softirq processing path are latency-sensitive. If your CPUs are sitting in powersave mode and scaling up on demand, you’ll see latency spikes every time a traffic burst arrives.

# Check current governor
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | sort -u

# Set performance governor for all CPUs
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Or use cpupower
cpupower frequency-set -g performance

For NICs on a specific NUMA node, you can target only the CPUs in that domain and leave the others in powersave. On a server running mixed workloads this makes a measurable difference in idle power.

Putting It All Together — The Production Script

Here’s a consolidated tuning script. It handles the ethtool settings that don’t survive reboots:

#!/bin/bash
# /usr/local/bin/nic-perf-tune.sh
# Network performance tuning for 10G/40G NICs on Linux 6.x
# Run via systemd at boot after network.target

set -euo pipefail

NIC="${1:-eth0}"
QUEUES=$(ethtool -l "$NIC" 2>/dev/null | awk '/Combined/{print $2}' | tail -1)
NCPUS=$(nproc)
USE_QUEUES=$((QUEUES < NCPUS ? QUEUES : NCPUS))

echo "[nic-tune] Configuring ${NIC} with ${USE_QUEUES} queues on ${NCPUS} CPUs"

# Multi-queue
ethtool -L "$NIC" combined "$USE_QUEUES" 2>/dev/null || true

# Ring buffers (use hardware max)
MAX_RX=$(ethtool -g "$NIC" | awk '/Pre-set/{found=1} found && /RX:/{print $2; exit}')
MAX_TX=$(ethtool -g "$NIC" | awk '/Pre-set/{found=1} found && /TX:/{print $2; exit}')
ethtool -G "$NIC" rx "$MAX_RX" tx "$MAX_TX"

# Interrupt coalescing (balanced default; tune per workload)
ethtool -C "$NIC" rx-usecs 50 tx-usecs 50 2>/dev/null || true

# Offloads
ethtool -K "$NIC" tso on gso on gro on lro off rx on tx on 2>/dev/null || true

# IRQ affinity — pin queue N to CPU N (skipping CPU 0)
IRQS=$(grep -i "${NIC}" /proc/interrupts | awk '{print $1}' | tr -d ':')
CPU=1
for IRQ in $IRQS; do
    if [[ -f /proc/irq/${IRQ}/smp_affinity_list ]]; then
        echo "$CPU" > /proc/irq/${IRQ}/smp_affinity_list
        echo "[nic-tune] IRQ ${IRQ} → CPU ${CPU}"
    fi
    CPU=$(( (CPU % (NCPUS - 1)) + 1 ))
done

# RFS
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
for QUEUE in /sys/class/net/${NIC}/queues/rx-*; do
    echo $((32768 / USE_QUEUES)) > "${QUEUE}/rps_flow_cnt"
done

echo "[nic-tune] Done."

Wire it into systemd:

# /etc/systemd/system/nic-perf-tune.service
[Unit]
Description=NIC performance tuning
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/nic-perf-tune.sh eth0
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

chmod +x /usr/local/bin/nic-perf-tune.sh
systemctl enable --now nic-perf-tune.service

Measuring the Impact

Don’t tune blind. Measure before and after with iperf3:

# Server
iperf3 -s

# Client — 8 parallel streams, 30 seconds
iperf3 -c <server-ip> -P 8 -t 30 -i 5

Watch for packet drops in real time:

watch -n1 'ethtool -S eth0 | grep -E "rx_missed|rx_dropped|tx_dropped|rx_no_buffer"'

Monitor softirq load per CPU:

watch -n1 'mpstat -P ALL 1 1 | grep -E "CPU|soft"'

If softirq is pinned on a single core, your IRQ affinity isn’t working. If rx_missed_errors climbs under load, increase the ring buffer or reduce coalescing aggressiveness.

What This Doesn’t Cover

DPDK and kernel bypass (useful when you need to process 40+ Mpps and the kernel overhead is genuinely the bottleneck, not configuration). XDP/eBPF for early packet drop or redirection before the full network stack. SR-IOV for VMs needing near-line-rate performance without a vSwitch.

Those are valid next steps once you’ve confirmed you’re actually hitting kernel limits — which most deployments aren’t. The settings above will get you to 90–95% of theoretical line rate on a properly configured server, and that’s where 99% of production workloads should stop.

The remaining 5% is where hardware engineers live. Most sysadmins don’t need to go there.