Python PyArrow: Read and Write Parquet Files

python pyarrow read write parquet requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you need to work with Parquet files in Python, pyarrow is the library most data engineers reach for. It provides a native implementation of the Parquet format, exposing both low-level file readers and writers and a higher-level Table API that integrates with pandas and other data tools. This article focuses on the core operations: reading a Parquet file into a PyArrow table, writing a table to Parquet, and the options that matter when you move beyond trivial examples.

Reading a Parquet File with PyArrow

The simplest way to read a Parquet file is pyarrow.parquet.read_table. It returns a pyarrow.Table object that holds the data in a columnar layout. Here is the minimal usage:

import pyarrow.parquet as pq

table = pq.read_table("data.parquet")
print(table.schema)
print(table.num_rows)

The read_table function infers the schema from the file metadata. You can also read only a subset of columns to reduce memory usage and I/O:

table = pq.read_table("data.parquet", columns=["id", "name", "score"])

If you need a pandas DataFrame instead, use read_pandas:

df = pq.read_pandas("data.parquet").to_pandas()

For very large files that do not fit in memory, you can read row groups individually. A Parquet file is split into row groups, each independently compressed and readable. Use ParquetFile to inspect the structure and read row groups one at a time:

pf = pq.ParquetFile("large.parquet")
for i in range(pf.num_row_groups):
    group_table = pf.read_row_group(i)
    process(group_table)

This pattern is essential when you want to process a file incrementally without loading the entire dataset.

Writing a Parquet File with PyArrow

Writing a table to Parquet is equally direct. You can create a pyarrow.Table from Python data or from a pandas DataFrame, then call write_table:

import pyarrow as pa
import pyarrow.parquet as pq

data = {
    "id": [1, 2, 3],
    "name": ["alice", "bob", "carol"],
    "score": [9.5, 8.0, 7.5]
}
table = pa.table(data)

pq.write_table(table, "output.parquet")

The write_table function handles schema inference automatically. If you have a pandas DataFrame, you can convert it first:

import pandas as pd

df = pd.DataFrame({"id": [1, 2], "value": [10, 20]})
table = pa.Table.from_pandas(df)
pq.write_table(table, "from_pandas.parquet")

You can also write multiple tables into a single file using ParquetWriter when you need to stream data in chunks. This is useful when the full table does not fit in memory:

writer = pq.ParquetWriter("stream.parquet", schema=table.schema)
for batch in table.to_batches(max_chunksize=1000):
    writer.write_batch(batch)
writer.close()

Notice that ParquetWriter requires a schema upfront. You can derive it from the first batch or define it explicitly.

Controlling the Parquet Schema

When reading, the schema comes from the file. When writing, you often want to control the types explicitly to avoid surprises. PyArrow lets you define a schema and pass it to write_table:

schema = pa.schema([
    ("id", pa.int64()),
    ("name", pa.string()),
    ("score", pa.float64()),
])

pq.write_table(table, "typed.parquet", schema=schema)

If the table has columns that do not match the schema, PyArrow will attempt to cast them. If casting fails, it raises an error. This is useful when you want to enforce a specific storage format, such as using int32 for small integers to save space.

When reading, you can also override the schema using the schema parameter in read_table. This is rarely needed, but it can help when a file has a schema that is not fully compatible with your downstream processing.

Partitioning and Row Group Size

Parquet supports partitioning at the directory level. When writing, you can use partition_cols to create a partitioned dataset:

pq.write_table(table, "partitioned/", partition_cols=["year", "month"])

This writes multiple files under subdirectories like year=2024/month=01/. Partitioning is a common pattern for large datasets because it allows queries to skip entire directories based on partition filters.

Row group size controls the granularity of compression and reading. By default, PyArrow uses a row group size of about 1 million rows. You can adjust it with the row_group_size parameter:

pq.write_table(table, "output.parquet", row_group_size=50000)

Smaller row groups allow more selective reads but increase metadata overhead. Larger row groups reduce overhead but require reading more data when you only need a subset. Choose a size that matches your typical access pattern.

Compression and Encoding Options

Parquet supports several compression codecs. PyArrow defaults to snappy, which balances speed and compression ratio. You can change it with the compression parameter:

pq.write_table(table, "compressed.parquet", compression="gzip")

Common choices are snappy, gzip, brotli, lz4, and zstd. Each has different tradeoffs. snappy is fast but compresses less; gzip compresses more but is slower. If you need to minimize file size, zstd often gives a good balance. The compression_level parameter lets you tune the codec further, but not all codecs support it.

Encoding is another layer of optimization. PyArrow automatically chooses encodings like dictionary encoding for string columns when it reduces size. You can disable dictionary encoding with use_dictionary=False if you know it is not beneficial for your data. This is rarely necessary, but it can help when the cardinality is extremely high.

Performance Considerations When Reading and Writing Parquet

The main performance benefit of Parquet is its columnar layout. When you read only a few columns, PyArrow reads only those columns from disk, skipping the rest. This is why columns filtering is so effective. When writing, the cost of compression and encoding is the main bottleneck. Using a faster codec like lz4 can speed up writes at the expense of file size.

Memory usage is another concern. Reading a whole file into a Table loads all row groups into memory. For files larger than available RAM, use the row-group iteration pattern shown earlier. Writing with ParquetWriter in batches also keeps memory bounded.

One subtle point: read_table loads the entire file into memory, but ParquetFile and read_row_group do not. If you need to process a file that is several gigabytes, always prefer the streaming approach.

Handling Large Files and Memory Usage

When working with datasets that exceed memory, you need to think about how PyArrow allocates memory. PyArrow uses its own memory pool, which can be configured. For most applications, the default is fine. However, when reading many files in a loop, you may want to call pa.gc_memory() to release unused memory back to the operating system:

import pyarrow as pa

for file in files:
    table = pq.read_table(file)
    process(table)
    del table
    pa.gc_memory()

This is not a substitute for proper memory management, but it helps in long-running processes.

Another practical technique is to use memory mapping. PyArrow can read Parquet files using memory mapping via memory_map=True in read_table. This lets the OS manage page loads, which can reduce memory pressure for files that are larger than RAM:

table = pq.read_table("large.parquet", memory_map=True)

Memory mapping is not always faster; it depends on the OS and the access pattern. For random access to a subset of columns, it can be very efficient.

Finally, consider using pq.read_schema to inspect a file without loading any data. This is useful for validating schemas before running a full read:

schema = pq.read_schema("data.parquet")
print(schema)

This operation reads only the metadata, so it is fast even for huge files.