Python PyArrow Parquet Compression and Datasets

Python PyArrow parquet compression and datasets go together more tightly than many developers expect. The compression argument determines how each column chunk is encoded on disk, which directly affects file size, write CPU cost, and read latency. Both pyarrow.parquet and pyarrow.dataset expose the same codec options, but the two APIs configure them in different places, and the difference matters when you move from single files to partitioned datasets.

Setting Compression When Writing Parquet with PyArrow

The simplest path is pyarrow.parquet.write_table. The compression parameter accepts a codec name or a dictionary mapping column names to codecs.

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({
    "id": [1, 2, 3],
    "payload": ["alpha", "beta", "gamma"],
})

pq.write_table(table, "output.parquet", compression="zstd")

When you pass a single string, every column chunk in the file uses that codec. The Parquet format stores the codec in the file metadata, so a reader never needs to be told which codec was used; it reads the metadata and decompresses accordingly.

ParquetWriter gives you the same control when writing incrementally:

with pq.ParquetWriter("output.parquet", table.schema, compression="gzip") as writer:
    writer.write_table(table)

The codec is fixed for the lifetime of the writer, so you cannot change it between write_table calls on the same writer instance.

Compression Codecs and What They Trade Off

PyArrow supports several codecs, and the right choice depends on the ratio of write cost to storage savings your workload can tolerate.

These are relative characteristics, not benchmark numbers. The actual ratio and speed depend on the data distribution, the number of rows per row group, and the PyArrow build you are using. Codec availability also depends on the build; if a codec is missing, PyArrow raises an error at write time, so verify availability in your deployment rather than assuming every codec is compiled in.

Snappy is the default in write_table and ParquetWriter when you omit compression. It is a reasonable default for most workloads because it keeps both write and read CPU low while still reducing file size meaningfully.

Applying Compression Per Column

Passing a dictionary to compression lets you treat columns differently. This is useful when a table mixes highly compressible text with numeric keys that do not benefit from compression.

pq.write_table(
    table,
    "output.parquet",
    compression={
        "id": "none",
        "payload": "zstd",
    },
)

Columns not listed in the dictionary fall back to the default codec, which is snappy. Per-column compression is a Parquet feature, not a PyArrow extension; each column chunk carries its own codec in the metadata. This means a single file can contain different codecs for different columns, and readers handle it transparently.

Per-column configuration is most valuable when you know the data profile. An ID column that is already dense and random will not shrink under compression and only costs CPU. A text or JSON column can shrink substantially. Mixing codecs lets you avoid paying compression cost where it produces nothing.

Compression in Dataset Writes

When you move from single files to pyarrow.dataset, compression is configured on the dataset write operation rather than on a writer object.

import pyarrow.dataset as ds

ds.write_dataset(
    table,
    "data/",
    format="parquet",
    compression="zstd",
    partitioning=["year"],
)

The compression argument applies to every Parquet file produced by the write. This is important for partitioned datasets: if you write with partitioning, each partition directory contains files using the same codec. There is no per-partition compression override; the codec is a property of the write operation, not of the partition.

When you write a dataset with max_rows_per_file or max_rows_per_group, each file and row group is compressed independently. The codec still applies uniformly, but the row group boundaries determine how much data each compressed chunk covers.

How Reading Handles Compression

Reading does not require a codec argument. Both pyarrow.parquet.read_table and pyarrow.dataset.dataset read the codec from each file's metadata.

import pyarrow.dataset as ds

dataset = ds.dataset("data/", format="parquet")
table = dataset.to_table()

This has a practical consequence: a dataset directory can contain files written with different codecs, and the reader will handle each one correctly. This is common when a pipeline changes its compression configuration over time and old files are not rewritten. The dataset API does not require all files to share a codec, because each Parquet file is self-describing.

The same applies to column-level codecs. A file with mixed codecs across columns reads without any special configuration.

Operational Considerations for Compression

Compression is a CPU and memory tradeoff at write time and a CPU tradeoff at read time.

Write CPU: stronger codecs such as gzip and brotli consume more CPU per row. In a high-throughput ingestion pipeline, this can become the bottleneck before disk throughput does. If writes are latency-sensitive, prefer snappy or lz4.

Read CPU: decompression happens on every read. A codec that produces a smaller file does not necessarily read faster; the decompression cost can offset the reduced I/O. For data read frequently, the balance often favors a fast codec over a small one.

Memory: compression operates on column chunks and row groups. Larger row groups increase the memory used during compression and decompression. If you raise row_group_size or max_rows_per_group, account for the additional memory in the writer and reader processes.

Storage and transfer: smaller files reduce storage cost and network transfer when data moves between systems. For cold or archival data, the write cost is paid once and the storage savings accumulate. For hot data read continuously, the read cost matters more.

None of these considerations are absolute. The right configuration depends on how often data is written, how often it is read, and what the storage budget is.

Choosing a Codec for Your Workload

Use snappy when you want the default behavior and do not want to tune compression. It is the safest choice for mixed workloads where both reads and writes happen frequently.

Use zstd when you want a better compression ratio than snappy without a large write-speed penalty. Zstd's level parameter lets you adjust the tradeoff, and PyArrow exposes it through the compression_level argument on write_table, ParquetWriter, and write_dataset.

Use gzip or brotli for data written infrequently and read occasionally, such as archival tables or nightly batch outputs. The higher write CPU cost is acceptable when writes are rare.

Use lz4 for high-throughput ingestion where storage is cheap and write latency matters more than file size.

Use none when the data is already compressed, such as a column containing gzipped JSON blobs. Recompressing already-compressed data wastes CPU and produces little benefit.

The decision is not about which codec is best in general. It is about where your workload sits on the write-cost, read-cost, and storage-cost triangle. If the profile changes, the codec can be changed per file or per dataset write without breaking existing readers, because the codec is always stored in the file metadata.