Python Polars DataFrame: Read and Write CSV and Parquet

python polars dataframe read write csv and parquet requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you need to persist a Python Polars DataFrame, reading and writing CSV and Parquet files are the two most common operations. Both formats have different trade-offs, and Polars provides dedicated APIs for each. This article covers the practical syntax for reading and writing these formats, the options that affect behavior, and the situations where one format is better than the other.

Reading a CSV into a Polars DataFrame

Polars reads CSV files with pl.read_csv. The function infers the schema by sampling the data, but you can override it when needed. The simplest call is:

import polars as pl

df = pl.read_csv("data.csv")

This returns a DataFrame with columns inferred from the CSV header row. If your file has no header, pass has_header=False. You can also specify a custom separator with separator=";" for semicolon-delimited files.

For files with inconsistent null representations, use null_values to map them to null:

df = pl.read_csv(
    "data.csv",
    null_values=["NA", "N/A", "NULL"],
)

When the schema is known in advance, provide it with the schema parameter. This avoids inference mistakes and speeds up parsing:

schema = {"id": pl.Int64, "name": pl.Utf8, "score": pl.Float64}
df = pl.read_csv("data.csv", schema=schema)

The try_parse_dates option attempts to convert date-like strings to Datetime columns. Use it when your CSV contains ISO dates and you want to avoid a separate conversion step.

Writing a Polars DataFrame to CSV

To write a DataFrame to CSV, call write_csv on the DataFrame:

df.write_csv("output.csv")

By default, the header row is included. Set include_header=False to omit it. The separator option controls the delimiter, and quote sets the quoting character. For large DataFrames, batch_size controls how many rows are written at once, which affects memory usage during the write.

Polars writes CSV in a single pass. If you need to preserve the schema for later reads, consider writing a separate schema file or using Parquet instead. CSV has no native schema, so type information is lost unless you re-specify it on read.

Reading a Parquet File into a Polars DataFrame

Parquet is a columnar format that preserves schema and supports compression. Reading is done with pl.read_parquet:

df = pl.read_parquet("data.parquet")

You can read only a subset of columns with the columns parameter:

df = pl.read_parquet("data.parquet", columns=["id", "name"])

The parallel option controls whether columns are read in parallel. For most workloads the default is fine, but on memory-constrained systems you can set parallel=False to reduce peak memory.

Polars also supports reading partitioned Parquet datasets. Point read_parquet to a directory that follows Hive-style partitioning, and it will read all files and add the partition columns to the result.

Writing a Polars DataFrame to Parquet

The write_parquet method writes a DataFrame to a Parquet file:

df.write_parquet("output.parquet")

By default, Polars uses Snappy compression. You can change it with the compression parameter:

df.write_parquet("output.parquet", compression="gzip")

Supported compression codecs include "snappy", "gzip", "lz4", and "zstd". The choice affects file size and read speed. Gzip produces smaller files but takes longer to write and read; Snappy is faster but less compact.

The row_group_size parameter controls how many rows are grouped in each row group. Larger row groups reduce metadata overhead but increase memory during reads. The default is usually a good balance, but you may adjust it for very wide or very tall DataFrames.

Choosing Between CSV and Parquet

Use CSV when you need a portable, human-readable file that other tools can open without special libraries. Use Parquet when you are building a data pipeline, need to preserve types, or work with large datasets where read performance matters.

Performance and Memory Considerations

Polars is designed for efficient memory use and parallel execution. When reading CSV, the main cost is parsing text and inferring types. Providing an explicit schema avoids the inference pass and can reduce read time. For Parquet, the columnar layout allows Polars to read only the columns you need, which is a major advantage when working with wide tables.

Writing to Parquet is generally faster than writing to CSV because Parquet writes binary data and applies compression in parallel. CSV writing is bound by string formatting and I/O.

Memory usage depends on the file format. CSV files are read into memory as a whole unless you use streaming. Polars supports streaming reads for both formats via the pl.scan_csv and pl.scan_parquet functions, which return a LazyFrame. This is useful when a file is too large to fit in memory.

lazy_df = pl.scan_csv("large.csv")
filtered = lazy_df.filter(pl.col("value") > 100).collect()

The same pattern works with scan_parquet. Using lazy evaluation lets Polars push down predicates and projections, reducing the amount of data actually loaded.

Handling Edge Cases and Common Errors

CSV files often contain messy data. Missing values, inconsistent quoting, and unexpected types are common. Polars provides null_values and schema to handle these. If a column fails to parse, Polars raises an error by default. You can set ignore_errors=True to replace unparsable values with null, but be careful: this can hide real problems.

Parquet files are stricter because the schema is fixed. If you try to write a DataFrame with a column type that does not match the existing Parquet schema, Polars will raise a schema mismatch error. When reading, the columns parameter must exist in the file; otherwise an error is raised.

Another common issue is the difference between Utf8 and Categorical types. CSV inference treats strings as Utf8. Parquet may store strings as Dictionary or Enum. When reading a Parquet file, Polars will map these back to Utf8 unless you explicitly cast them.

For large files, avoid reading the entire file into memory if you only need a subset. Use scan_csv or scan_parquet with a filter and select before collect. This is the most effective way to keep memory usage under control.