python polars null values fill_null and drop_nulls: Practica

python polars null values fill_null and drop_nulls requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When working with real-world data in Python Polars, null values are inevitable. They represent missing or unknown information, and how you handle them directly affects the correctness of your analysis. The two primary tools for dealing with nulls are fill_null and drop_nulls. This article covers the syntax, behavior, and practical decision-making for both, so you can clean your DataFrames efficiently and avoid common pitfalls.

What Are Null Values in Polars?

In Polars, a null value is a distinct sentinel indicating missing data. It is not the same as NaN (Not a Number), which is a floating-point value. Polars distinguishes between the two: null works for any data type, while NaN only applies to float columns. For example, a string column can contain null but not NaN.

To detect nulls, use the is_null() method, which returns a boolean mask, or null_count() to get the total number of nulls per column. Here's a minimal example:

import polars as pl

df = pl.DataFrame({
    "id": [1, 2, 3],
    "value": [10.5, None, 30.2],
    "label": ["a", None, "c"]
})

print(df)
print(df.null_count())
print(df.select(pl.all().is_null()))

The output shows that value and label each have one null. Understanding this distinction is the first step because fill_null and drop_nulls operate on null, not NaN. If your data contains NaN, you may need to convert it to null first using fill_nan or a custom expression.

Removing Nulls with drop_nulls

The drop_nulls method removes rows that contain any null value. By default, it considers all columns. You can restrict it to specific columns using the subset parameter. This is useful when a null in a secondary column should not invalidate an otherwise complete row.

# Drop rows with null in any column
df.drop_nulls()

# Drop rows with null only in the 'value' column
df.drop_nulls(subset=["value"])

When you use subset, only the specified columns are checked. Rows with nulls in other columns remain. This gives you fine-grained control over what constitutes a complete record.

drop_nulls is the right choice when the missing data cannot be reasonably estimated, or when the row is unusable without the missing value. For example, if you're building a regression model and the target column is null, dropping those rows is often the safest approach.

Replacing Nulls with fill_null

The fill_null method replaces nulls with a value you provide. The simplest form takes a scalar constant:

df.fill_null(0)

This replaces every null in the DataFrame with 0. However, you often need column-specific values. You can pass an expression that computes a value per column, such as the mean or median:

df.fill_null(pl.col("value").mean())

This fills nulls in the value column with the column's mean, but leaves other columns untouched because the expression targets a single column. For multiple columns, you can use pl.col("value").mean() and pl.col("label").fill_null("unknown") in a with_columns context.

Polars also supports a strategy parameter for forward or backward filling. This propagates the last non-null value forward ("forward") or the next non-null value backward ("backward"). This is common for time-series data:

df.sort("date").fill_null(strategy="forward")

Note that strategy only works with null, not NaN. Also, the order of rows matters: you should sort by the ordering column before applying a forward fill, otherwise the result is arbitrary.

For numeric interpolation, use the interpolate method instead of fill_null. It fills nulls by linear interpolation between valid points, which is often more accurate than a constant or mean.

Combining fill_null and drop_nulls in a Cleaning Pipeline

In practice, you rarely use only one method. A typical cleaning pipeline inspects null counts, then decides per column whether to drop rows or fill missing values. Here's a concrete workflow:

# Inspect nulls
print(df.null_count())

# Fill numeric columns with median
numeric_cols = ["value", "score"]
df = df.with_columns([
    pl.col(c).fill_null(pl.col(c).median()) for c in numeric_cols
])

# Drop rows where the target column is null
df = df.drop_nulls(subset=["target"])

# Fill categorical column with a placeholder
df = df.with_columns(pl.col("label").fill_null("unknown"))

This approach preserves as much data as possible while ensuring the critical columns are complete. The order matters: fill first to avoid dropping rows that could be salvaged, then drop only if necessary.

Performance and Memory Considerations

Null handling in Polars is optimized, but the choice between fill_null and drop_nulls has implications for memory and compute. drop_nulls reduces the number of rows, which can lower memory usage for downstream operations. However, if you drop too many rows, you may lose signal. fill_null keeps the dataset size but introduces imputed values, which can bias statistical analyses if not done carefully.

When working with large datasets, consider using lazy evaluation. Polars' lazy API (pl.LazyFrame) delays computation until you call collect(). Both fill_null and drop_nulls work in lazy mode, and Polars can optimize the entire query plan. For example, if you drop nulls after a filter, Polars may reorder operations to reduce intermediate memory usage.

lf = pl.LazyFrame(df)
result = (
    lf.filter(pl.col("value").is_not_null())
    .fill_null(0)
    .collect()
)

Avoid chaining multiple fill_null calls on the same column with different strategies; instead, compute the desired value once and apply it. Also, be aware that fill_null with a constant is cheaper than computing a mean, which requires an aggregation pass.

Edge Cases and Common Mistakes

Nulls behave differently in groupby, joins, and aggregations. In group_by, nulls are treated as a separate group by default. If you don't want that, filter them out before grouping. In joins, nulls do not match other nulls, so rows with null keys will not join. This is a common source of unexpected row loss.

Another mistake is using fill_null with a strategy on unsorted data. For example:

df.fill_null(strategy="forward")

If the DataFrame is not ordered, the forward fill will propagate values in whatever order the rows happen to be, which is rarely what you want. Always sort by the logical ordering column first.

Also, remember that drop_nulls with subset checks only the specified columns. If you intend to drop rows where any column has null, omit subset. Conversely, if you want to drop rows only when a critical column is null, use subset to avoid unnecessary data loss.

Choosing the Right Strategy: drop_nulls vs fill_null

The decision between dropping and filling depends on the nature of your data and the downstream task. Use drop_nulls when:

The null represents a missing record that cannot be meaningfully imputed.
The proportion of nulls is small, and dropping them won't bias the analysis.
The row is incomplete for a required operation, such as a join key or a target variable.

Use fill_null when:

The null is a temporary measurement gap, such as a sensor dropout.
The column is numeric and you can safely impute with a central tendency (mean, median) or interpolation.
The row contains other valuable information, and dropping it would waste data.

For categorical columns, filling with a placeholder like "unknown" preserves the row but introduces a new category. This is acceptable for many models but can distort frequency distributions.

A practical rule: if the null rate is below 5% and the column is not critical, dropping is often simpler. If the null rate is high, consider whether the column is worth keeping at all. For time series, forward fill is usually preferred over mean imputation because it preserves temporal continuity.

Advanced Usage: Conditional Filling with Custom Expressions

Sometimes you need to fill nulls based on conditions. Polars' expression system allows you to combine fill_null with when and otherwise to apply different strategies per row. For example, you might want to fill a null in a numeric column with the median only if another column indicates a certain category:

df.with_columns(
    pl.when(pl.col("group") == "A")
    .then(pl.col("value").fill_null(pl.col("value").median()))
    .otherwise(pl.col("value"))
    .alias("value")
)

This is more flexible than a single global fill. You can also use fill_null with a nested expression that computes a value from other columns, such as a regression prediction or a rolling mean.

Another advanced pattern is to fill nulls only in specific rows using a boolean mask. For instance, you might want to fill nulls only for rows where a flag is set:

df.with_columns(
    pl.col("value").fill_null(0).where(pl.col("flag") == 1)
)

This keeps nulls in rows where flag is not 1. This level of control is useful when nulls have different meanings in different contexts.

Finally, when working with lazy frames, you can push these expressions into the query plan and let Polars optimize the execution. The key is to understand that fill_null and drop_nulls are not just simple methods—they are part of a larger expression system that can be composed to handle complex missing-data scenarios cleanly.