Apache Kafka has been the backbone of serious event-streaming infrastructure for over a decade. It works. It scales. Entire careers are built on tuning it. But Redpanda has been stealing ground — and in 2026, the conversation is no longer hypothetical. Kafka 4.x finally killed ZooKeeper for real. Redpanda crossed the line from "interesting experiment" to "production-grade at scale." Both ecosystems have moved, and the old comparison posts you find on page one of Google are wrong.
This is the comparison I wish existed when I was evaluating both for a high-throughput pipeline that needed to run on bare metal with tight memory budgets.
Official repositories: Apache Kafka · Redpanda
The Fundamental Split: JVM vs C++ (and Why It Actually Matters)
Everything else flows from this one architectural choice. Kafka is a JVM application. Redpanda is C++ built on top of the Seastar framework.
Kafka’s JVM heritage is both its strength and its oldest headache. The strength: the Java ecosystem, mature tooling, Kafka Streams, ksqlDB, a decade of operational knowledge encoded in blog posts and runbooks. The headache: GC pauses, heap tuning, the fact that "it needs at least 6GB of RAM to not be embarrassing in production" is muscle memory for anyone who’s operated it seriously.
Redpanda’s Seastar model is a different beast. Seastar uses a thread-per-core architecture with shared-nothing design — each thread owns its data and its I/O, and they never fight over locks. On Linux, it leverages io_uring for async I/O without kernel context switching overhead. There’s no GC, no JVM warm-up, no heap. The result is much more predictable tail latency and a smaller RSS.
This isn’t marketing spin — the design genuinely produces different behavior under load. The question is whether that behavior matters for your workload.
Throughput: Real Numbers, Not Marketing Slides
Benchmark theater is rampant in this space. Both Redpanda’s own benchmarks and various blog posts are written by people with skin in the game. So here’s the honest framing:
In synthetic benchmarks on identical hardware, Redpanda consistently shows 2–8x higher throughput than vanilla Kafka configurations. Under aggressive tuning, that gap narrows — Kafka with linger.ms, batch.size, tuned num.io.threads, and a properly sized heap can close the gap significantly. A well-tuned Kafka cluster on NVMe at 3 replicas can sustain 800MB/s–1.2GB/s per broker. Redpanda on equivalent hardware often hits that ceiling with less configuration ceremony.
The more interesting number is p99 latency, where Redpanda’s advantage is much harder to close regardless of tuning. Kafka’s GC cycles create latency spikes — even with G1GC or ZGC, you’ll see tail latency outliers at the 99th percentile that Redpanda simply doesn’t produce. If your SLA is about median latency, the two are more comparable. If you care about p99.9 — trading systems, real-time fraud detection, anything where a slow consumer causes cascading problems — Redpanda’s predictability is a genuine advantage.
The honest conclusion: for pure throughput, a tuned Kafka cluster is competitive. For latency predictability and operational simplicity at equivalent throughput, Redpanda wins.
Memory Footprint: Before and After Kafka 4.x
Kafka 4.0 dropped ZooKeeper support entirely. This changes the comparison because the old math (Kafka broker + ZooKeeper ensemble × 3 = your base memory cost) no longer applies. KRaft is now the only mode.
Still, the numbers diverge sharply in practice:
| Component | Kafka 4.x (KRaft) | Redpanda |
|---|---|---|
| Minimum viable heap | 4GB (-Xmx4g) |
N/A (no JVM) |
| Realistic production heap | 8–16GB | N/A |
| Coordinator overhead | Embedded (KRaft controller) | Embedded (Raft) |
| RSS at idle (single broker) | ~1.5–2.5GB | ~300–600MB |
| RSS under heavy load | 6–12GB+ | 1–3GB |
The Redpanda memory model is more like a database than a message broker — it uses a configurable shared cache with redpanda.developer_mode: false and explicit memory_allocation_percent. You know exactly what it’s going to use. Kafka’s JVM heap can balloon during GC cycles and compaction in ways that require careful monitoring.
For teams running self-hosted, this matters on two fronts: you can pack more brokers per host, and your cost-per-message on cloud instances is meaningfully lower.
Feature Parity in 2026: The Honest Map
Redpanda has closed most compatibility gaps over the last two years. But "mostly Kafka-compatible" is not the same as "drop-in replacement for all workloads." Here’s where things actually stand:
Works reliably:
- Kafka producer/consumer protocol (all client libraries work — Java, Go, Python, Rust)
- Topic administration via standard Kafka admin API
- Consumer groups and group rebalancing
- Transactions and exactly-once semantics (EOS) — Redpanda has had solid EOS support since v22, it’s battle-tested now
- Log compaction
- SASL/TLS authentication
- Kafka Connect (most connectors — test yours specifically)
- Schema Registry (Redpanda ships its own built-in)
Still broken or missing:
- Kafka Streams — this is the big one. Kafka Streams is a client-side library, but it relies on broker-level behavior for internal topics and admin operations that Redpanda doesn’t fully replicate. In 2026, this still doesn’t work without significant workarounds.
- ksqlDB — same issue, deeper dependency
- Tiered Storage is available in Redpanda but requires the Enterprise license for some features (more on licensing below)
If your architecture depends on Kafka Streams or ksqlDB, the migration story is not a configuration change — it’s a rearchitecture. Plan accordingly.
Licensing: The Conversation Nobody Wants to Have
Redpanda is not Apache 2.0. It’s BSL (Business Source License) 1.1 with a Redpanda-specific change date. BSL means: free for internal/non-production use, free for self-hosting below a certain threshold, but commercial restrictions apply at scale. The enterprise features (Tiered Storage above certain limits, RBAC, audit logging) require a paid license.
This doesn’t kill Redpanda for most self-hosters — the community edition covers a lot of ground. But if you’re building a product that wraps Redpanda and sells it, or running at hyper-scale, read the license. Kafka (Apache 2.0) has no such restriction.
Migration Path 1: Drop-in Swap (Zero-History Scenarios)
If you’re starting fresh, building a new pipeline, or can afford to replay data from source, this is the right approach.
# docker-compose.yml — single-node Redpanda replacing Kafka
version: "3.8"
services:
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:latest
command:
- redpanda start
- --smp 2 # cores to use
- --memory 2G # hard cap on memory
- --reserve-memory 0M
- --overprovisioned # dev mode: skip CPU pinning
- --kafka-addr PLAINTEXT://0.0.0.0:9092
- --advertise-kafka-addr PLAINTEXT://redpanda:9092
- --pandaproxy-addr 0.0.0.0:8082 # REST proxy
- --schema-registry-addr 0.0.0.0:8081
ports:
- "9092:9092" # Kafka-compatible API — your existing clients point here
- "8081:8081" # Schema Registry
- "8082:8082" # REST Proxy
- "9644:9644" # Admin API
volumes:
- redpanda-data:/var/lib/redpanda/data
redpanda-console:
image: docker.redpanda.com/redpandadata/console:latest
depends_on:
- redpanda
ports:
- "8080:8080"
environment:
KAFKA_BROKERS: redpanda:9092
KAFKA_SCHEMAREGISTRY_ENABLED: "true"
KAFKA_SCHEMAREGISTRY_URLS: "http://redpanda:8081"
volumes:
redpanda-data:
Change bootstrap.servers in your producer/consumer config from kafka:9092 to redpanda:9092. That’s often the entire migration for standard workloads.
Verify your topics came across and your consumer groups register correctly:
# rpk is Redpanda's CLI — equivalent of kafka-topics.sh + kafka-consumer-groups.sh
rpk topic list
rpk group list
rpk cluster info
Migration Path 2: Live Dual-Write Cutover
When you have a running system that can’t tolerate replay, the safest approach is dual-write with a rollback window.
# producer_dual_write.py — simplified dual-write pattern
# Use this during the cutover window, not permanently
from kafka import KafkaProducer
import json
kafka_producer = KafkaProducer(
bootstrap_servers=["kafka-broker:9092"],
value_serializer=lambda v: json.dumps(v).encode()
)
redpanda_producer = KafkaProducer(
bootstrap_servers=["redpanda:9092"], # same API, different target
value_serializer=lambda v: json.dumps(v).encode()
)
def publish(topic: str, message: dict):
# Write to both clusters simultaneously
kafka_future = kafka_producer.send(topic, message)
redpanda_future = redpanda_producer.send(topic, message)
# Wait for both — if either fails, you know immediately
kafka_future.get(timeout=10)
redpanda_future.get(timeout=10)
Simultaneously, migrate consumers to Redpanda one by one. Watch consumer group lag on both clusters. Once lag on Redpanda is stable and caught up, flip the remaining consumers, stop dual-write, decommission Kafka.
Migration Path 3: Historical Data with Redpanda Connect
For large clusters where you need to migrate actual topic history, Redpanda Connect (previously known as Benthos, acquired by Redpanda) is the recommended tool. It’s powerful and significantly more flexible than MirrorMaker 2.
# redpanda-connect-migration.yaml
# Replicates from Kafka to Redpanda, preserving offsets where possible
input:
kafka:
addresses:
- kafka-broker:9092
topics:
- your-topic-name
consumer_group: migration-consumer-group
start_from_oldest: true # replay full history
tls:
enabled: false
pipeline:
processors: [] # add transforms here if needed
output:
kafka_franz:
seed_brokers:
- redpanda:9092
topic: your-topic-name
# Preserve original timestamps
timestamp_ms: "${! metadata("kafka_timestamp") }"
Run it:
rpk connect run redpanda-connect-migration.yaml
Monitor lag from source:
# On the Kafka side
kafka-consumer-groups.sh \
--bootstrap-server kafka-broker:9092 \
--describe \
--group migration-consumer-group
# On the Redpanda side
rpk group describe your-consumer-group
Production Configuration That Actually Matters
For a production Redpanda cluster (3-node), here’s a configuration that doesn’t leave you debugging at 2 AM:
# redpanda.yaml — broker configuration
redpanda:
# Kafka API listener — plain and TLS
kafka_api:
- address: 0.0.0.0:9092
name: default
# Admin API — keep this internal-only
admin:
- address: 0.0.0.0:9644
name: default
# Core resource limits — be explicit
developer_mode: false
memory_allocation_percent: 80 # leave 20% for OS
cpu_starvation_threshold_ms: 2000
# Replication defaults
default_topic_replications: 3
default_topic_partitions: 12 # tune to your workload
min_version: v4.1 # enforce client minimum
# Tiered Storage (community edition)
cloud_storage_enabled: false # enable if using S3/GCS
rpk:
kafka_api:
brokers:
- redpanda-0:9092
- redpanda-1:9092
- redpanda-2:9092
admin_api:
addresses:
- redpanda-0:9644
- redpanda-1:9644
- redpanda-2:9644
Gotchas
The Kafka Streams wall. It keeps appearing in migration post-mortems. If any service in your stack imports org.apache.kafka.streams, you cannot migrate that service without replacing the streaming layer. Flink, Spark Streaming, or Redpanda’s own Connect pipelines are the common replacements.
Benchmark your actual workload. Redpanda’s published benchmarks use specific message sizes (1KB), specific batch sizes, specific replication factors. Your workload is not that workload. Run rpk topic produce with your real message shapes before committing to any hardware sizing.
Consumer group protocol subtleties. Redpanda implements the cooperative sticky rebalancing protocol, and so does modern Kafka. But if you’re running old consumers (ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG set to RangeAssignor or similar), test the rebalancing behavior explicitly. Group rebalance storms are the worst kind of production incident.
rpk is not kafka-topics.sh. The mental model is similar but the flags differ. Your ops runbooks need updating. rpk topic create, rpk topic delete, rpk topic describe — all slightly different syntax. Update your scripts before you’re scrambling during an incident.
Redpanda Connect ≠ Kafka Connect. Redpanda Connect (Benthos) uses a different connector ecosystem. Some Kafka Connect connectors have no direct equivalent. Check your specific connectors against the Redpanda Connect component list before assuming they port over.
Clock skew in multi-broker setups. Redpanda uses Raft for consensus, and Raft is sensitive to clock drift between nodes. NTP is mandatory; chrony with a local time server is better. This is also true for Kafka KRaft, but Redpanda will surface clock-related warnings more aggressively in logs.
The BSL license at scale. Determine what "at scale" means for your use case before you’re in a procurement conversation at the worst possible time.
When to Keep Kafka
You’re running Kafka Streams or ksqlDB in production — full stop. The migration cost isn’t a configuration change; it’s a significant engineering effort.
You’re on a managed service (Confluent Cloud, Amazon MSK) where the JVM overhead is someone else’s problem, the SLAs are defined, and it’s working. Switching buys you little.
Your team has deep Kafka operational expertise — runbooks, on-call playbooks, institutional knowledge of your specific cluster’s failure modes. That knowledge is worth a lot, and Redpanda doesn’t automatically transfer it.
You need Apache 2.0 licensing with no ambiguity for a commercial product.
When to Switch to Redpanda
You’re self-hosting and running into JVM memory pressure. The math is simple: Redpanda does the same job with 60–75% less RAM on the broker tier.
You’re building a new pipeline with no Kafka Streams dependency and want operational simplicity. No JVM, no GC tuning, no KAFKA_HEAP_OPTS in your docker-compose.
Tail latency matters. If p99 spikes are causing problems downstream, the GC-free architecture is the right tool.
You’re on ARM hardware (Graviton, Apple Silicon in dev). Redpanda has solid ARM64 support; Kafka’s JVM adds another layer of abstraction that Redpanda simply doesn’t need.
You want the Schema Registry and REST proxy bundled in, without running separate Confluent components.
The Practical Verdict
Kafka 4.x is a genuinely better Kafka than it was three years ago. Dropping ZooKeeper removed a major operational burden. KRaft is solid. The ecosystem is unmatched.
Redpanda in 2026 is a legitimate production choice that’s no longer a gamble. It’s faster to operate, cheaper on memory, and handles standard event-streaming workloads without the JVM tax. The compatibility gaps are real but narrow for most use cases.
Pick Kafka if your stack is already there and working, or if you depend on Kafka Streams. Pick Redpanda if you’re self-hosting, starting fresh, or need that latency predictability. The migration itself, for standard workloads, is genuinely a few hours of work — not a multi-month project. The Kafka API compatibility is good enough that your producers and consumers won’t know the difference.