Streaming Joins Compared: Flink vs Kafka Streams vs Materialize

by

Streaming joins are where most real-time pipelines quietly fall apart. The concept sounds simple: take two event streams, match records by a key, emit enriched output. In practice, you’re joining two infinite sequences of data arriving out of order, at different rates, with late events and retraction semantics that nobody warned you about.

The three most common answers to "how do I join streams?" are Apache Flink, Kafka Streams, and Materialize. Each has a radically different mental model, and picking the wrong one will cost you weeks. This article is a direct comparison of all three — not a vendor pitch, not a benchmark marketing piece. Just what each system actually does, where it breaks, and when to reach for it.

Why Streaming Joins Are Hard

Before diving into the tools, a quick reality check on what you’re dealing with.

A database join is easy: both tables live on disk, you scan them, you’re done. A streaming join is hard because neither side is "done." Orders keep arriving. User profiles keep updating. A click event from 3 seconds ago needs to be matched against an impression event that arrived 2 seconds ago — but you don’t know when the matching event will show up, or if it ever will.

That gives you three fundamental problems:

State accumulation. To join two streams, you have to buffer one (or both) sides in memory/disk state. That state grows forever unless you add time-bounded windows or a TTL policy. Most teams discover this when their heap explodes at 2am.

Out-of-order arrivals. Events rarely arrive in the order they were created. Watermarks (Flink’s mechanism) or wall-clock TTLs (Kafka Streams’ approach) are how you decide when to stop waiting for a late event. Set them wrong and you either get incorrect results or massive state.

Update semantics. If an order record updates after you’ve already emitted a joined result, what happens? Some systems emit a retraction. Others just re-emit. Others silently ignore the update. This matters enormously for correctness.

GitHub: https://github.com/apache/flink

Flink is the Swiss Army knife of stream processing. It gives you the most control, the most join types, and the most rope to hang yourself with.

Window Join — the classic. You define a time window, buffer both streams within it, and emit matched pairs when the window closes.

// Flink DataStream API - Window Join
DataStream<Order> orders = ...;
DataStream<Payment> payments = ...;

orders.join(payments)
    .where(order -> order.getId())
    .equalTo(payment -> payment.getOrderId())
    .window(TumblingEventTimeWindows.of(Time.minutes(5)))
    .apply((order, payment) -> new EnrichedOrder(order, payment));

Clean enough. But every unmatched event is silently dropped. If an order and payment fall in different 5-minute windows because of network lag, they never join. Most people hit this in production the first time they see missing records in downstream reports.

Interval Join — a better fit for most real-world event correlation. You say: "join an order with a payment that arrived between -10 seconds and +30 seconds of the order."

// Flink Interval Join — much more practical for event correlation
orders.keyBy(Order::getId)
    .intervalJoin(payments.keyBy(Payment::getOrderId))
    .between(Time.seconds(-10), Time.seconds(30))
    .process(new ProcessJoinFunction<Order, Payment, EnrichedOrder>() {
        @Override
        public void processElement(
            Order order,
            Payment payment,
            Context ctx,
            Collector<EnrichedOrder> out) {
            out.collect(new EnrichedOrder(order, payment));
        }
    });

This is cleaner and more semantically honest. You’re telling Flink exactly how far apart two events can be and still be considered related. State is automatically cleaned up once the interval passes.

Temporal Table Join — this one is underused and important. It lets you join a stream against a "point-in-time snapshot" of another stream. Perfect for enriching events with dimension data (user profiles, product catalogs) without having to duplicate that data everywhere.

-- Flink SQL - Temporal Table Join
-- Enrich click events with the user profile as it was at click time
SELECT 
    c.click_time,
    c.user_id,
    u.plan_tier,
    u.country
FROM clicks c
LEFT JOIN users FOR SYSTEM_TIME AS OF c.click_time AS u
ON c.user_id = u.user_id

This is one of Flink’s genuine superpowers. The FOR SYSTEM_TIME AS OF clause gives you time-travel semantics on dimension tables. You’re not joining against the current state of users — you’re joining against what it looked like when the click happened. Critical for correct historical analytics.

Watermark tuning is a black art. Too aggressive and you drop late events. Too conservative and your state balloons and your latency spikes. Start with WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)) and tune from there, not the other way around.

State backend matters. The default HashMapStateBackend keeps everything in JVM heap. Fine for small jobs, a disaster at scale. Switch to EmbeddedRocksDBStateBackend for anything with more than a few GB of state.

# flink-conf.yaml - production state backend
state.backend: rocksdb
state.backend.incremental: true
state.checkpoints.dir: s3://your-bucket/flink-checkpoints
state.savepoints.dir: s3://your-bucket/flink-savepoints
# Tune RocksDB memory to avoid heap pressure
taskmanager.memory.managed.fraction: 0.4

Checkpoint failures kill your job. If you haven’t set execution.checkpointing.mode: EXACTLY_ONCE and configured proper retry behavior, a transient S3 failure can corrupt your state. Always set execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION.

Kafka Streams

GitHub: https://github.com/apache/kafka (part of the main Kafka repo)

Kafka Streams is the operator-friendly option. It’s a library — no cluster to manage, just a JAR that runs as part of your application. This is a genuine advantage if you’re running a microservices shop that already has Kafka.

Join Types in Kafka Streams

Kafka Streams models data as either a KStream (event log, unbounded) or a KTable (changelog, a current-state view). The join type depends on what you’re joining.

KStream-KTable Join — the most useful and correct join in Kafka Streams. Join an event stream against a lookup table. Think: enrich every purchase event with the current customer tier.

// Kafka Streams - KStream-KTable Join
StreamsBuilder builder = new StreamsBuilder();

KStream<String, Purchase> purchases = builder.stream("purchases");
KTable<String, Customer> customers = builder.table("customers");

KStream<String, EnrichedPurchase> enriched = purchases.join(
    customers,
    (purchase, customer) -> new EnrichedPurchase(purchase, customer),
    Joined.with(
        Serdes.String(),
        purchaseSerde,
        customerSerde
    )
);

enriched.to("enriched-purchases");

The crucial thing here: the KTable is driven by a Kafka topic acting as a changelog. When a customer record changes, the table updates and future joins pick up the new value. This is correct and efficient. The join is non-windowed on the table side — you always get the latest version of the customer.

KStream-KStream Join — both sides are event streams, so Kafka Streams needs a time window to bound state.

// Kafka Streams - KStream-KStream Join
// JoinWindows defines how far apart two events can be
JoinWindows window = JoinWindows.ofTimeDifferenceWithNoGrace(Duration.ofMinutes(5));

KStream<String, EnrichedEvent> joined = streamA.join(
    streamB,
    (a, b) -> new EnrichedEvent(a, b),
    window,
    StreamJoined.with(Serdes.String(), aSerdes, bSerdes)
);

Gotcha: ofTimeDifferenceWithNoGrace means late events are dropped. If you need to tolerate late arrivals, use ofTimeDifferenceAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30)). The grace period keeps the window open a little longer to absorb stragglers, at the cost of more state and higher latency.

KTable-KTable Join — joining two changelog streams to produce a materialized view of the combined current state. Use this when you want the equivalent of a database join that stays up to date.

// KTable-KTable join — produces a continuously updated combined table
KTable<String, Order> orders = builder.table("orders");
KTable<String, Shipment> shipments = builder.table("shipments");

KTable<String, OrderWithShipment> combined = orders.join(
    shipments,
    (order, shipment) -> new OrderWithShipment(order, shipment)
);

This is closer to Materialize territory, but with limitations — more on that below.

Gotchas in Kafka Streams

Co-partitioning is mandatory and painful. Both topics in a join must have the same number of partitions and be partitioned by the same key. If they don’t match, Kafka Streams will either reject the topology at startup or silently produce wrong results in edge cases. Validate partition counts before you deploy to production.

Retention vs. state store alignment. If you create a KTable from a topic with 1-hour retention, but your application is down for 2 hours (deployments, incidents), you’ll restore to an incomplete state. Either use compacted topics (no retention limit, just keep the latest per key) or size your retention conservatively.

Kafka Streams doesn’t rebalance gracefully under load. When a new instance joins or leaves the consumer group, there’s a rebalance. During that window, processing pauses. For low-latency pipelines (sub-second SLAs), this is a real problem. The incremental cooperative rebalancing protocol helps, but it’s not magic.

// Always configure cooperative rebalancing
props.put(StreamsConfig.REBALANCE_PROTOCOL_CONFIG, "cooperative-sticky");

Materialize

GitHub: https://github.com/MaterializeInc/materialize

Materialize takes a completely different angle. It’s a streaming SQL database. You write standard SQL, and Materialize maintains the query results incrementally as new data arrives from Kafka topics. There’s no concept of windows or watermarks in the user-facing interface — you just write a join, and the system figures out how to keep it up to date.

This is the incremental view maintenance (IVM) model, and it’s genuinely powerful for the right use cases.

Joins in Materialize

-- Materialize - create sources from Kafka topics
CREATE SOURCE orders
FROM KAFKA BROKER 'kafka:9092' TOPIC 'orders'
FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY 'http://schema-registry:8081';

CREATE SOURCE customers
FROM KAFKA BROKER 'kafka:9092' TOPIC 'customers'
FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY 'http://schema-registry:8081';

-- Create a materialized view - this join runs continuously
CREATE MATERIALIZED VIEW order_details AS
SELECT 
    o.order_id,
    o.amount,
    o.created_at,
    c.name AS customer_name,
    c.tier AS customer_tier,
    c.country
FROM orders o
JOIN customers c ON o.customer_id = c.id;

That’s it. No Java, no windowing configuration, no state management. Materialize now maintains order_details in real time. When a new order arrives, it gets joined with the customer table and appears in the view within milliseconds. When a customer updates their profile, existing orders in the view are updated retroactively via retractions.

You can query this view like a regular table at any point. It’s always up to date.

-- Tail the view for live updates — Materialize pushes diffs, not full scans
SUBSCRIBE TO order_details;

-- Or query it directly — always consistent
SELECT customer_tier, SUM(amount) 
FROM order_details 
WHERE created_at > NOW() - INTERVAL '1 hour'
GROUP BY customer_tier;

Temporal filters and append-only optimizations matter a lot for performance. If your source topics are append-only (no updates), tell Materialize:

CREATE SOURCE orders
FROM KAFKA BROKER 'kafka:9092' TOPIC 'orders'
FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY 'http://schema-registry:8081'
ENVELOPE NONE;  -- append-only, no upsert/delete semantics

Gotchas in Materialize

State lives in RAM (mostly). Materialize’s differential dataflow engine keeps join state in memory. For large dimension tables or high-cardinality joins, you will feel this. Materialize has added disk-spilling support, but it’s still fundamentally a memory-intensive system. Budget RAM generously — plan for 3-5x the raw data size for a complex view.

No late data handling in the Flink sense. Materialize assumes your data arrives promptly. It doesn’t have watermarks. If you have events arriving hours late, Materialize will include them correctly when they arrive — but if your downstream systems already consumed the pre-update output, you’ll need to deal with retractions. Not all downstream sinks handle retractions gracefully.

No UDFs (yet, mostly). Complex business logic that would be a ProcessFunction in Flink is hard to express in SQL. If you need to call an external service during join processing, parse complex binary formats, or run ML inference, Materialize isn’t your tool.

Bootstrapping takes time. When you create a materialized view, Materialize has to read all historical data from your Kafka topics before the view is ready. For topics with months of history, this can take a while. Monitor mz_materialized_views for progress.

-- Check if your view has caught up
SELECT name, ready
FROM mz_materialized_views
WHERE name = 'order_details';

Side-by-Side Comparison

Here’s where each system actually sits in the real world:

Concern Flink Kafka Streams Materialize
Join model Window, Interval, Temporal Window (streams), No-window (table) SQL, IVM, retraction-based
State management Manual but powerful Mostly automatic Automatic, memory-heavy
Late data handling Watermarks + grace periods Grace periods Implicit (just arrives late)
Operational complexity High (cluster to manage) Low (embedded library) Medium (single binary or cloud)
SQL interface Flink SQL (solid, growing) No First-class
Correctness on updates Depends on join type Good for KTable patterns Excellent (retractions)
Scale ceiling Very high Medium-high Medium (memory-bound)
Language Java/Scala/Python/SQL Java/Scala SQL

When to Use What

Use Flink when you have complex multi-hop pipelines, CEP patterns, custom stateful processing logic, or you need to support very high throughput with fine-grained control over exactly-once semantics. Flink is the right answer when the problem is genuinely hard. It has the steepest learning curve but the highest ceiling.

Use Kafka Streams when you’re already in the Kafka ecosystem, your team writes JVM services, and the operational overhead of a separate Flink or Materialize cluster isn’t worth it. KStream-KTable joins for enrichment are a great fit. Don’t reach for it if your join logic is complex or your key cardinality is enormous.

Use Materialize when your team speaks SQL fluently, the primary use case is serving fresh aggregated data to applications or dashboards, and you don’t have exotic late-data requirements. Materialize genuinely shines for the "make this dashboard always show up-to-date numbers from Kafka" use case, and the developer experience is excellent.

Production-Ready Setup

Whatever you pick, these practices apply universally:

Schema registry is non-negotiable. Avro or Protobuf with Confluent Schema Registry. Deserializing raw JSON in production joins is asking for a schema mismatch at 3am.

Monitor state sizes, not just lag. Consumer group lag is a lagging indicator. State backend size (RocksDB metrics in Flink, state store metrics in Kafka Streams, memory usage in Materialize) tells you earlier when something is wrong.

Test with production-shaped data. A join that handles 10 events/sec in staging but 100k/sec in production will behave differently. Specifically: key skew. If 80% of your events share the same customer_id, a key-based join will hammer one partition. Stress-test this early.

Have a state compaction strategy on day one. Decide upfront: are your Kafka topics compacted or time-windowed? How does your join state get cleaned up? A join with no TTL will eat your heap or your RocksDB disk indefinitely. Flink’s StateTtlConfig, Kafka Streams’ retention settings, and Materialize’s source TTL parameters all need intentional values before you go live.

The right streaming join tool is the one that matches your team’s existing skills, your operational tolerance for complexity, and the actual correctness semantics your business requires. Flink gives you control. Kafka Streams gives you simplicity. Materialize gives you SQL correctness. Pick one and go deep — the worst outcome is trying to use all three.

👁 Views: 114,939 · Unique visitors: 45,636