Blocking 500K IPs Without Melting Your Firewall: ipset + nftables Done Right

by

You want to block an entire country, a known botnet C2 range, or a threat intelligence feed with 300,000 entries. You add those rules to iptables one by one. Your firewall becomes a performance crater. Every packet now crawls through a linear list that the kernel evaluates top-to-bottom, rule by rule. At 10,000 rules, you’re already in trouble. At 300,000, your server is effectively DoS-ing itself.

This is a solved problem, and the solution is ipset — or, if you’re running a modern kernel and nftables, native sets that do the same thing without the extra tool. This article covers both approaches: how they actually work under the hood, where the performance cliffs are, and how to run this reliably in production with atomic updates and automatic blocklist refreshes.

Why iptables + raw rules don’t scale

Every rule in an iptables chain is checked sequentially for every packet. The complexity is O(n). If you have 1,000 DROP rules for individual IPs, every packet — including legitimate traffic — burns through 1,000 comparisons before it reaches the ACCEPT at the end of the chain.

This isn’t theoretical. At around 50,000 rules, iptables processing overhead becomes measurable on typical server hardware. Network throughput drops, latency spikes, and iptables-save starts taking seconds because even serializing the ruleset is slow.

ipset solves this by moving the lookup into a kernel-side hash table. The match check drops to O(1) regardless of whether you have 100 or 1,000,000 entries. Your iptables chain keeps a single rule that says "if source IP is in this set, DROP it." That one rule dispatches to a hash lookup. Fast.

ipset fundamentals

The project lives at https://ipset.netfilter.org/, with the kernel module shipped in mainline since 2.6.39.

apt install ipset   # Debian/Ubuntu
dnf install ipset   # RHEL/Fedora

Set types and when to use each

hash:ip — single IPv4 or IPv6 addresses. Most compact for flat IP lists.

hash:net — CIDR prefixes. Use this when your blocklist contains subnets (which most threat intel feeds do). Automatically handles /8 through /32.

hash:ip,port — (IP, port) tuples. Useful for blocking specific services from specific IPs without a blanket drop.

bitmap:ip — a bitmap over a /16 range. O(1) lookup with zero collision risk, but you’re pre-allocating memory for the entire range. Only makes sense when you’re blocking dense subsets of a /16.

list:set — a list of other sets. Lets you create a "meta-blocklist" that aggregates multiple sets. Useful organizationally, but adds one extra lookup layer.

For country blocks and threat intel feeds, you’ll almost always want hash:net.

Creating and populating a set

# Create a set for CIDR-based blocklists
# hashsize: initial hash table size (power of 2)
# maxelem: maximum number of entries
ipset create blocklist hash:net hashsize 4096 maxelem 1000000

# Add entries
ipset add blocklist 185.220.101.0/24
ipset add blocklist 94.102.49.0/24

# Verify
ipset list blocklist | head -20

Then wire it into iptables:

iptables -I INPUT 1 -m set --match-set blocklist src -j DROP

One rule in the chain, no matter how many entries are in the set.

Performance characteristics you should actually know

Hash table sizing

The hashsize parameter controls the number of buckets in the initial hash table. If you under-provision it relative to your element count, ipset will resize — and resize operations have a brief pause. For a blocklist with 300,000 CIDR entries, start with hashsize 65536. The rule of thumb: hashsize should be at least as large as your expected element count divided by 4.

Memory consumption for hash:net is roughly 32–40 bytes per entry on 64-bit systems. A 300,000-entry set costs around 12 MB. That’s nothing. You can run multiple large sets without concern on any modern server.

The maxelem gotcha

The default maxelem is 65536. Every tutorial that doesn’t mention this will have you scratching your head when ipset add fails silently or returns an error after 65,537 entries. If your blocklist might grow, set maxelem to something generous upfront — resizing a live set isn’t possible without recreating it.

ipset create blocklist hash:net hashsize 65536 maxelem 2000000

Lookup time

Measuring ipset match overhead in isolation is difficult, but the practical comparison is straightforward: a single hash:net match rule is indistinguishable from no rule in throughput benchmarks with sets up to ~1 million entries. The hash lookup is bounded. Linear iptables rules are not.

For hash:ip, the lookup is a single hash probe. For hash:net, ipset walks the longest prefix match structure — still O(1) in terms of rule count, though with a small constant for the prefix trie traversal.

Atomic updates: the production pattern

Flushing a set that’s actively in use and repopulating it is a race condition. Traffic hits during the window when the set is empty gets through. Don’t do this:

# BAD: race condition window
ipset flush blocklist
cat new_blocklist.txt | while read cidr; do ipset add blocklist "$cidr"; done

The correct pattern uses ipset swap:

# Create a staging set with the same type
ipset create blocklist_new hash:net hashsize 65536 maxelem 2000000

# Populate the staging set
while IFS= read -r cidr; do
    # Skip comments and blank lines
    [[ "$cidr" =~ ^#|^$ ]] && continue
    ipset add blocklist_new "$cidr" 2>/dev/null
done < new_blocklist.txt

# Atomic swap — instantaneous from the kernel's perspective
ipset swap blocklist blocklist_new

# Destroy the old set (now named blocklist_new after swap)
ipset destroy blocklist_new

The swap operation is atomic at the kernel level. There is no window where the set is empty or partially populated.

nftables native sets: skip the extra tool

If you’re already on nftables (kernel 3.13+, and if you’re running a distro shipped after 2018 you probably are), you can replicate everything ipset does using nftables named sets. No separate userspace tool, no kernel module dependency beyond nf_tables itself.

nftables sets use the same hash-based implementation under the hood. The performance characteristics are equivalent.

Defining a named set in nftables

table inet filter {
    # Named set for blocklist — auto-merge combines adjacent CIDRs automatically
    set blocklist {
        type ipv4_addr
        flags interval, dynamic
        auto-merge
        elements = {
            185.220.101.0/24,
            94.102.49.0/24,
            10.0.0.0/8
        }
    }

    chain input {
        type filter hook input priority 0; policy accept;
        
        # Single rule, O(1) lookup
        ip saddr @blocklist drop
    }
}

The flags interval tells nftables this set contains ranges/prefixes, not just individual addresses. auto-merge will coalesce adjacent or overlapping ranges — useful when your blocklist has redundant entries.

Loading a large blocklist into an nftables set

The nftables way to do bulk updates is via nft commands:

#!/usr/bin/env bash
# load-blocklist.sh

BLOCKLIST_URL="https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset"
TMPFILE=$(mktemp)
SETFILE=$(mktemp)

# Download and filter
curl -sf "$BLOCKLIST_URL" | grep -v '^#' | grep -v '^$' > "$TMPFILE"

# Build nft add element commands
echo "flush set inet filter blocklist" > "$SETFILE"
echo -n "add element inet filter blocklist { " >> "$SETFILE"

first=1
while IFS= read -r cidr; do
    if [[ $first -eq 1 ]]; then
        echo -n "$cidr" >> "$SETFILE"
        first=0
    else
        echo -n ", $cidr" >> "$SETFILE"
    fi
done < "$TMPFILE"

echo " }" >> "$SETFILE"

# Apply atomically via a transaction
nft -f "$SETFILE"

rm -f "$TMPFILE" "$SETFILE"

This works, but flush set has the same race condition as ipset flush. The nftables way to do truly atomic replacements is to use a full table reload with a new ruleset file that includes the updated set elements. For most blocklist use cases, the brief flush+reload window is acceptable at 3 AM during the cron job.

nftables verdict maps for per-source actions

For more nuanced control — not just DROP but logging different threat categories differently — nftables verdict maps are cleaner than maintaining multiple sets:

table inet filter {
    map threat_actions {
        type ipv4_addr : verdict
        flags interval
        elements = {
            185.220.101.0/24 : drop,
            94.102.49.0/24 : goto log_and_drop,
            198.98.51.0/24 : drop
        }
    }

    chain log_and_drop {
        log prefix "THREAT: " flags all
        drop
    }

    chain input {
        type filter hook input priority 0; policy accept;
        ip saddr vmap @threat_actions
    }
}

Verdict maps let you make one rule that dispatches to different actions per source range. This is something ipset simply can’t do — you’d need multiple sets and multiple iptables rules.

Using ipset with nftables (the hybrid approach)

nftables can reference ipset sets directly if you load them via iptables-compatible hooks — but this is messy and unsupported in clean nftables setups. The honest advice: pick one. If you’re on nftables, use nftables sets. If you’re on iptables, use ipset. Don’t mix them unless you have a very specific legacy reason.

The only real reason to use ipset alongside nftables is if you have existing automation built around the ipset CLI and don’t want to rewrite it yet. In that case, load ipset and reference it via the iptables compatibility layer — but know that this is a migration phase, not an architecture.

Automation: keeping blocklists fresh

A static blocklist is a liability after a week. Here’s a complete refresh script that pulls from multiple sources:

#!/usr/bin/env bash
# /usr/local/sbin/update-blocklist.sh
set -euo pipefail

SETNAME="blocklist"
TMPSET="${SETNAME}_tmp"
LOGFILE="/var/log/blocklist-update.log"

log() { echo "$(date -Is) $*" | tee -a "$LOGFILE"; }

# Sources — adjust to your threat model
SOURCES=(
    "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset"
    "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset"
    # "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset"  # aggressive
)

log "Starting blocklist update"

# Destroy staging set if it exists from a previous failed run
ipset destroy "$TMPSET" 2>/dev/null || true

# Create staging set
ipset create "$TMPSET" hash:net hashsize 65536 maxelem 2000000

# Download and load all sources
TOTAL=0
for url in "${SOURCES[@]}"; do
    log "Fetching $url"
    count=0
    while IFS= read -r line; do
        [[ "$line" =~ ^#|^[[:space:]]*$ ]] && continue
        if ipset add "$TMPSET" "$line" 2>/dev/null; then
            ((count++)) || true
        fi
    done < <(curl -sf --max-time 30 "$url")
    log "Loaded $count entries from $url"
    TOTAL=$((TOTAL + count))
done

log "Total entries loaded: $TOTAL"

# Ensure the live set exists (first run)
if ! ipset list "$SETNAME" &>/dev/null; then
    ipset create "$SETNAME" hash:net hashsize 65536 maxelem 2000000
    iptables -I INPUT 1 -m set --match-set "$SETNAME" src -j DROP
    log "Created live set and iptables rule"
fi

# Atomic swap
ipset swap "$TMPSET" "$SETNAME"
ipset destroy "$TMPSET"

log "Blocklist updated successfully. Active entries: $(ipset list $SETNAME | grep -c '/' || true)"

Wire this into a systemd timer rather than cron — better logging, restart-on-failure, and dependency ordering:

# /etc/systemd/system/blocklist-update.service
[Unit]
Description=Update IP blocklist
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/update-blocklist.sh
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/blocklist-update.timer
[Unit]
Description=Update IP blocklist every 6 hours
Requires=blocklist-update.service

[Timer]
OnBootSec=5min
OnUnitActiveSec=6h
Unit=blocklist-update.service

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now blocklist-update.timer

Persistence across reboots

ipset sets live in kernel memory. They’re gone on reboot. Two options:

Option 1 — ipset save/restore (simplest):

# Save
ipset save > /etc/ipset.rules

# Restore on boot via iptables-persistent or a systemd unit

# /etc/systemd/system/ipset-restore.service
[Unit]
Description=Restore ipset rules
Before=iptables.service nftables.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/sbin/ipset restore -f /etc/ipset.rules
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Option 2 — just re-run the update script on boot:

If you’re refreshing the blocklist every 6 hours anyway, OnBootSec=5min in the timer handles the reboot case without maintaining a static dump. This is the better approach — your blocklist is current rather than stale-from-last-reboot.

Gotchas worth knowing before you hit them

IPv6 sets are separate. hash:net by default is IPv4 only. For IPv6, create a dedicated set with family inet6 and add a matching ip6tables rule. It’s a common miss.

ipset create blocklist6 hash:net family inet6 hashsize 4096 maxelem 500000
ip6tables -I INPUT 1 -m set --match-set blocklist6 src -j DROP

CIDR aggregation saves memory and improves performance. Before loading a large list of individual IPs, run it through aggregate-prefixes or the cidr-merger tool. Consolidating 1,000 IPs in a /24 into a single /24 entry reduces lookup time and memory. The firehol blocklists are pre-aggregated, but if you’re building your own from raw threat intel, aggregate first.

The timeout feature changes the data structure. Adding timeout 3600 to an ipset set enables TTLs, but it also changes the internal implementation from a plain hash to a hash with expiry tracking. This has slightly higher per-entry memory overhead. Don’t enable timeouts on a static blocklist you control — just manage entries explicitly.

conntrack state and blocklists interact. If you’re using connection tracking (almost certainly yes), packets that are part of an ESTABLISHED connection won’t re-hit your INPUT rules because they’re fast-pathed by conntrack. This means a newly blocked IP’s existing TCP sessions won’t be cut immediately. To kill existing connections, you need conntrack -D -s <ip> after adding to the blocklist. For most threat intel use cases, this doesn’t matter much — you’re blocking scanners and botnets, not ongoing sessions.

nftables set element limits. nftables sets have a default element limit that varies by kernel version. For very large sets, check nft list ruleset output for warnings, and add size <n> to your set definition to pre-declare capacity.

Testing without live traffic. Before deploying a new blocklist source, check whether it would block your own monitoring IPs, CDN egress ranges, or third-party services you depend on. Run ipset test blocklist <your-ip> to verify membership before committing.

Choosing between ipset and nftables sets

If you’re starting fresh on a modern distro: use nftables sets. The tooling is more coherent, verdict maps give you capabilities ipset can’t match, and you’re not pulling in an extra kernel module.

If you have existing iptables infrastructure: use ipset. It’s mature, the tooling is excellent, and migrating everything to nftables is a separate project — don’t conflate it with the blocklist problem.

If you’re running both iptables and nftables rules on the same host (legacy compatibility layers): you have a mess, and the blocklist is the least of your problems.

The underlying kernel data structures are similar in both cases. The performance difference in production is negligible. What matters is operational consistency: pick one and do it correctly, rather than maintaining both.

What a production deployment looks like

By the time you’re done, you have:

  • A hash:net set with 200,000–500,000 CIDR entries loaded from aggregated threat intel feeds
  • An atomic swap procedure that updates the set every 6 hours with no traffic window
  • A systemd timer that logs to journald and retries on failure
  • A matching IPv6 set
  • A size guard in your nftables set definition so you don’t hit surprises at 65,536 entries

The firewall overhead is a single hash lookup per packet for the blocklist check. Your CPU doesn’t care. Your iptables chain stays short. The threat intel team feeds you updated lists and they take effect in the next refresh cycle.

This is the kind of setup where you configure it once, forget about it, and occasionally check the logs to see how many packets per day it’s dropping. At that point, it’s just working infrastructure.

👁 Views: 112,657 · Unique visitors: 45,393