Zero-Copy Data in Python: Stop Paying the Serialization Tax with Apache Arrow

You’re running a Python data pipeline. Pandas loads a 2 GB CSV, does some cleaning, and hands it off to DuckDB for aggregation. Then you pull results back into Polars for reshaping and finally export to Parquet. Each handoff secretly copies the entire dataset in memory. Your 2 GB file is now consuming 8–10 GB RAM and taking twice as long as it should.

This is the serialization tax. Every library wants data in its own format, so Python diligently marshals bytes back and forth like an overworked postal worker. Apache Arrow exists to kill this pattern entirely.

The GitHub repo: https://github.com/apache/arrow — pyarrow ships in the Python branch.

The Real Problem: Everyone Has Their Own Memory Layout

Pandas stores a DataFrame as a collection of NumPy arrays, one per column, in row-major or column-major layout depending on how you squint at it. Polars uses its own chunked columnar format built in Rust. DuckDB has its internal vector format. NumPy has strided arrays. None of these are the same.

When you write polars.from_pandas(df), Polars doesn’t magically reinterpret pandas’ memory. It walks the pandas DataFrame, converts each column, handles nullability differences (pandas uses NaN, Polars uses actual nulls), and builds its own representation. That’s a full copy. On a 1 GB DataFrame, you’ve just used 2 GB for the duration of that call.

Apache Arrow defines a language-agnostic, cross-process columnar memory layout. When two libraries both speak Arrow natively, they can look at the same physical bytes in RAM and see a valid dataset. No conversion. No copy. Just a pointer hand-off.

What PyArrow Actually Gives You

pyarrow is the Python binding for Apache Arrow. It ships a Table type (think DataFrame, but Arrow-native), array types, schema utilities, IPC serialization, and — critically — a bridge layer that every major Python data library now implements.

pip install pyarrow

The mental model: pyarrow.Table is the neutral handshake format. Pandas can export to it, Polars can consume from it, DuckDB queries return it, and NumPy can borrow individual columns from it. The data lives in Arrow buffers the whole time.

import pyarrow as pa
import pyarrow.parquet as pq

# A simple Arrow table from Python lists
table = pa.table({
    "user_id": [1, 2, 3, 4],
    "score": [98.5, 72.1, 85.0, 91.3],
    "active": [True, True, False, True],
})

print(table.schema)
# user_id: int64
# score: double
# active: bool

pa.table() allocates Arrow buffers directly. This is the memory that other libraries can borrow without copying.

Zero-Copy with Pandas

Pandas 2.0+ has a proper Arrow-backed dtype via ArrowDtype. Before that, conversions existed but copies were unavoidable for some operations. With Arrow dtypes, the story changes.

import pandas as pd
import pyarrow as pa

# Build an Arrow table
arrow_table = pa.table({
    "event": ["login", "purchase", "logout", "login"],
    "amount": [0.0, 49.99, 0.0, 0.0],
    "ts": [1700000000, 1700000100, 1700000200, 1700000300],
})

# Convert to pandas — Arrow-backed dtypes, minimal copy
df = arrow_table.to_pandas(types_mapper=pd.ArrowDtype)

print(df.dtypes)
# event     string[pyarrow]
# amount    double[pyarrow]
# ts         int64[pyarrow]

The types_mapper=pd.ArrowDtype argument tells pandas to keep the Arrow buffers in place rather than converting to NumPy. Individual columns stored as pd.ArrowDtype are wrappers around Arrow ChunkedArray objects — the bytes don’t move.

Going the other way:

# pandas -> Arrow, zero-copy when dtypes are Arrow-backed
back_to_arrow = pa.Table.from_pandas(df, preserve_index=False)

# Check that the buffers are shared (same address)
orig_buf = arrow_table.column("amount").buffers()[1]
roundtrip_buf = back_to_arrow.column("amount").buffers()[1]

print(orig_buf.address == roundtrip_buf.address)  # True if zero-copy succeeded

Gotcha: If your pandas DataFrame uses standard NumPy dtypes (float64, object), the to_pandas roundtrip cannot be zero-copy. Arrow needs to convert object dtype strings to its own UTF-8 representation. There’s no way around this — it’s a structural incompatibility. Use Arrow-backed dtypes from the start if zero-copy matters.

Zero-Copy with Polars

Polars is Arrow-native internally. Every Polars Series is backed by Arrow ChunkedArray. When you go Polars → Arrow → Polars, you’re essentially doing nothing expensive.

import polars as pl
import pyarrow as pa

# Polars -> Arrow: zero-copy
df_polars = pl.DataFrame({
    "product_id": [101, 102, 103],
    "price": [9.99, 24.50, 4.75],
    "in_stock": [True, False, True],
})

arrow_table = df_polars.to_arrow()  # No copy — shares underlying buffers

# Arrow -> Polars: zero-copy
df_back = pl.from_arrow(arrow_table)

# Verify: same physical memory
import sys
buf_polars = df_polars["price"].to_physical()._s.buffers()[0]
buf_arrow = arrow_table.column("price").buffers()[1]
# Both point to the same allocation

This is why Polars → DuckDB → Polars pipelines can be genuinely fast. The data never leaves Arrow-land.

Gotcha: Polars uses strict null semantics. An Arrow column with no null bitmap is considered non-nullable by Polars. If your source data has validity_bitmap=None in the Arrow buffer, Polars will treat it as non-nullable and may reject operations that produce nulls into that column without first rechunking. Always check table.schema for nullability before handing data to Polars.

Zero-Copy with DuckDB

DuckDB is where Arrow interop gets genuinely exciting. DuckDB can query Arrow Tables directly using SQL, without copying the data into DuckDB’s own storage.

import duckdb
import pyarrow as pa

events = pa.table({
    "user_id": [1, 1, 2, 3, 2],
    "event_type": ["view", "click", "view", "purchase", "purchase"],
    "revenue": [0.0, 0.0, 0.0, 59.0, 22.5],
})

# DuckDB scans the Arrow table in-place
conn = duckdb.connect()
result = conn.execute("""
    SELECT
        user_id,
        COUNT(*) AS events,
        SUM(revenue) AS total_revenue
    FROM events
    GROUP BY user_id
    ORDER BY total_revenue DESC
""").arrow()  # Returns result as Arrow Table

print(result)
# pyarrow.Table: user_id, events, total_revenue

The FROM events clause — DuckDB resolves events as a Python variable via its scan mechanism. It reads the Arrow buffers directly. The .arrow() call on the result returns a new Arrow Table without going through Python objects or pandas.

For large scans this is a significant win. DuckDB’s query engine is SIMD-optimized, and operating on Arrow’s columnar layout means it gets maximum cache locality without a marshaling step.

# Full pipeline: Parquet -> Arrow -> DuckDB -> Arrow -> Polars
import pyarrow.parquet as pq
import polars as pl

raw = pq.read_table("transactions.parquet")  # Arrow Table directly from Parquet

conn = duckdb.connect()
aggregated = conn.execute("""
    SELECT date_trunc('day', ts) AS day, SUM(amount) AS daily_total
    FROM raw
    GROUP BY 1
    ORDER BY 1
""").arrow()

# Polars for final reshaping
result = pl.from_arrow(aggregated)

No copies between any of these stages. The memory footprint is dominated by the original data, not by 4x copies of it.

The Arrow C Data Interface: How Zero-Copy Actually Works at the Protocol Level

The magic underneath all this is the Arrow C Data Interface — a C-level ABI that Arrow libraries implement to share data across language and library boundaries without going through Python objects.

When Polars calls from_arrow(table), it doesn’t use the PyArrow Python API to read values. It calls into a C function that returns ArrowSchema and ArrowArray structs — just two stack-allocated C structs with pointers to the actual data buffers and their lengths. Polars’ Rust code reads those pointers directly.

Python 3.12 formalized this with the PyCapsule Interface (__arrow_c_stream__ and __arrow_c_array__ dunder methods). Any object implementing these methods can be consumed zero-copy by any compliant library.

import pyarrow as pa

arr = pa.chunked_array([[1, 2, 3], [4, 5, 6]])

# Check if it exposes the PyCapsule interface
print(hasattr(arr, "__arrow_c_stream__"))  # True in modern PyArrow

# Any library with Arrow support can consume this directly
# e.g., polars, duckdb, cuDF, Lance, Velox bindings

You don’t usually call these methods yourself — the libraries call them internally. But knowing they exist explains why pl.from_arrow(x) is O(1) regardless of data size. It’s a pointer exchange, not a data copy.

NumPy Interop: Where It Gets Tricky

NumPy arrays can borrow from Arrow buffers for numeric primitive types. The trick is using pyarrow.Array.to_pydict() — no, actually np.asarray() on an Arrow array.

import pyarrow as pa
import numpy as np

arr = pa.array([1.0, 2.5, 3.7, 4.1], type=pa.float64())

# Zero-copy: NumPy borrows Arrow's buffer
np_arr = arr.to_pylist()  # DON'T do this — copies everything to Python objects

# DO this instead:
np_arr = arr.to_numpy(zero_copy_only=True)  # Raises if copy would be needed
print(np_arr.base)  # Points to the Arrow buffer

# Or, be explicit about accepting a copy when necessary:
np_arr = arr.to_numpy(zero_copy_only=False)

The zero_copy_only=True flag is your correctness guard. If the Arrow array has nulls, to_numpy can’t represent them in a standard NumPy array (NumPy doesn’t have a null concept for numeric types), so it would need to emit a masked array or a copy with NaN fill. With zero_copy_only=True, it raises instead — forcing you to handle the nullability explicitly.

Gotcha: Arrow buffers are immutable by design. If you get a zero-copy NumPy view of an Arrow array and try to modify it, you’ll either get a ValueError (if Arrow marked the buffer read-only) or silently corrupt shared state (if the immutability flag isn’t enforced at the NumPy level). Treat zero-copy NumPy arrays from Arrow as read-only. Always.

arr = pa.array([10, 20, 30])
np_arr = arr.to_numpy(zero_copy_only=True)

np_arr[0] = 99  # May raise, may corrupt — never do this

Memory Ownership and Lifetime

When multiple libraries share the same Arrow buffer, reference counting keeps it alive. As long as any Arrow array, Polars Series, or DuckDB result holds a reference, the underlying bytes stay valid.

import pyarrow as pa
import polars as pl

def get_data():
    table = pa.table({"x": range(1_000_000)})
    return pl.from_arrow(table)
    # `table` goes out of scope here — but the buffer is still referenced
    # by the Polars DataFrame, so it won't be freed

df = get_data()
print(df["x"].sum())  # Works fine — buffer is alive through df

The risk is holding onto a large Arrow buffer via a tiny slice. If you extract one column from a 10 GB table, the entire 10 GB allocation stays alive until both the table and the column slice are freed.

# This keeps the entire table buffer alive
small_col = big_table.column("id")
del big_table  # Doesn't free the memory — small_col still holds a reference

# Fix: copy the column into a fresh allocation
small_col = small_col.copy()  # Now big_table's buffers can be freed
del big_table

This is identical to the substring memory leak problem in Go strings or Java’s old String.substring() behavior. The Arrow ecosystem inherits it.

IPC: When Zero-Copy Crosses Process Boundaries

Within a single process, shared buffers are trivial. Across processes — a Celery worker sending results to a FastAPI server, for example — you need IPC.

Arrow’s IPC format is a stream of record batches serialized to a flat byte sequence. The key property: the byte layout is the in-memory Arrow format, so reading an IPC stream into memory doesn’t require transformation. The deserialized buffers are Arrow buffers.

import pyarrow as pa
import pyarrow.ipc as ipc
import io

table = pa.table({"a": range(100), "b": [x * 0.5 for x in range(100)]})

# Serialize to bytes (e.g., for sending over a socket or writing to shared memory)
sink = io.BytesIO()
writer = ipc.new_stream(sink, table.schema)
writer.write_table(table)
writer.close()

payload = sink.getvalue()

# Deserialize — no transformation, just pointer setup
reader = ipc.open_stream(io.BytesIO(payload))
received_table = reader.read_all()

Pair this with mmap and you have shared memory IPC where the receiving process literally maps the same physical pages:

import pyarrow.plasma as plasma  # Arrow Plasma store (shared memory object store)
# Or use raw mmap + Arrow IPC for a DIY solution

import mmap
import os

# Write Arrow IPC to a temp file
with open("/tmp/arrow_data.ipc", "wb") as f:
    ipc.new_stream(f, table.schema).write_table(table)

# In another process: mmap the file and read zero-copy
with open("/tmp/arrow_data.ipc", "rb") as f:
    mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    reader = ipc.open_stream(pa.BufferReader(mm))
    shared_table = reader.read_all()
    # shared_table references mm's pages — no copy

Gotcha: The Plasma object store (pyarrow.plasma) was deprecated in Arrow 12.0 and removed later. Use multiprocessing.shared_memory + Arrow IPC or a purpose-built solution like Ray’s object store for cross-process Arrow sharing.

Production Pattern: The Arrow-First Pipeline

Here’s a realistic ETL pattern that stays in Arrow-land end-to-end:

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.compute as pc
import duckdb
import polars as pl

# --- Stage 1: Ingest from Parquet (Arrow native) ---
raw = pq.read_table(
    "s3://my-bucket/events/",
    columns=["user_id", "event_type", "amount", "ts"],
    filters=[("ts", ">=", 1700000000)],  # Predicate pushdown — Arrow reads only matching row groups
)

# --- Stage 2: Filter with PyArrow compute (stays in Arrow) ---
cleaned = raw.filter(pc.greater(raw["amount"], 0))

# --- Stage 3: Aggregate with DuckDB (Arrow in, Arrow out) ---
conn = duckdb.connect()
aggregated = conn.execute("""
    SELECT
        user_id,
        DATE_TRUNC('hour', TO_TIMESTAMP(ts)) AS hour,
        SUM(amount) AS hourly_revenue,
        COUNT(*) AS event_count
    FROM cleaned
    WHERE event_type = 'purchase'
    GROUP BY 1, 2
""").arrow()

# --- Stage 4: Final transform in Polars (Arrow in, Arrow out) ---
result = (
    pl.from_arrow(aggregated)
    .with_columns(
        pl.col("hourly_revenue").round(2),
        (pl.col("hourly_revenue") / pl.col("event_count")).alias("avg_order_value"),
    )
    .sort("hour")
)

# --- Stage 5: Write back to Parquet ---
pq.write_table(result.to_arrow(), "output/hourly_revenue.parquet", compression="zstd")

Data enters as Arrow, lives as Arrow through DuckDB and Polars, and exits as Parquet (which is compressed Arrow). The peak memory usage is roughly the size of one copy of the data — not five.

Quick Reference: When You Get a Copy and When You Don’t

Operation	Zero-copy?	Why
pl.from_arrow(table)	Yes	Polars is Arrow-native
table.to_pandas(types_mapper=pd.ArrowDtype)	Yes	Arrow-backed dtypes
table.to_pandas() (NumPy dtypes)	No	NumPy ↔ Arrow buffer conversion
duckdb.query("SELECT ... FROM table").arrow()	Partial	Scan is zero-copy; result is new
arr.to_numpy(zero_copy_only=True)	Yes	Buffer borrow
arr.to_numpy() with nulls	No	NaN fill requires copy
pa.Table.from_pandas(df) with NumPy dtypes	No	Buffer conversion
pa.Table.from_pandas(df) with ArrowDtype	Yes	Buffer hand-off
Arrow IPC deserialization	Effectively zero-copy	Layout is identical

Closing Thoughts

The Python data ecosystem’s convergence on Arrow as the common memory format is one of the most quietly impactful infrastructure shifts of the last few years. Five years ago, every library handoff was a copy. Today, if you’re intentional about it, you can build pipelines that process hundreds of gigabytes on a machine with 16 GB of RAM — not because anything is magic, but because you stopped paying for the same data four times.

The practical rules: use pd.ArrowDtype if you’re in a pandas shop, use zero_copy_only=True as a correctness gate, treat borrowed buffers as read-only, watch for the slice-holds-parent memory trap, and stay away from to_pylist() or to_pydict() anywhere in a hot path.

Arrow is already under the hood in Polars, DuckDB, the Parquet reader, and increasingly in machine learning frameworks. Learning to drive it explicitly just means you’re no longer leaving performance on the floor.