RDRYUSLOG DEV
Back to Blog
Python

Polars String and Datetime Operations for Clean Data Pipelines

python polars string and datetime operations: Learn practical Polars string and datetime operations: parsing, formatting, regex, date arithmetic, time zones, and perfo...

polarsstring manipulationdatetime parsingdata transformationpython dataframes
Illustration of a dataframe with columns for strings and datetime values being transformed into clean, typed data.

python polars string and datetime operations requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you work with real-world data in Polars, string and datetime operations often consume more of your pipeline code than any other transformation. The expression API in Polars is designed to handle these operations efficiently, but the syntax differs from pandas in ways that can trip up experienced developers. This article covers the most common string and datetime operations you'll need in production, with examples that assume you already have a pl.DataFrame loaded and are comfortable with the expression context.

Parsing Strings into Datetimes

The most frequent datetime task is converting a string column into a proper Datetime type. Polars provides str.strptime for this, which accepts a format string and returns a Datetime column. Unlike pandas, Polars does not infer formats automatically; you must specify the format or use one of the built-in ISO 8601 parsers.

import polars as pl

df = pl.DataFrame({
    "event_time": ["2024-01-15 08:30:00", "2024-02-20 14:45:10"]
})

df_with_dt = df.with_columns(
    pl.col("event_time").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S").alias("event_dt")
)

For ISO 8601 strings with timezone offsets, use str.to_datetime which handles the standard format automatically. This is faster than a custom format because Polars uses a specialized parser.

df_iso = pl.DataFrame({"ts": ["2024-01-15T08:30:00Z", "2024-02-20T14:45:10+02:00"]})
df_iso.with_columns(pl.col("ts").str.to_datetime())

When the format is unknown or inconsistent, you can use str.strptime with multiple formats via pl.coalesce, but this is slower. Prefer normalizing the data upstream if possible.

Extracting and Formatting Date and Time Components

Once you have a Datetime column, you often need to extract components like year, month, day, hour, or weekday. Polars provides the .dt namespace for this.

df = pl.DataFrame({"dt": ["2024-01-15 08:30:00", "2024-02-20 14:45:10"]}).with_columns(
    pl.col("dt").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S")
)

df.with_columns(
    pl.col("dt").dt.year().alias("year"),
    pl.col("dt").dt.month().alias("month"),
    pl.col("dt").dt.day().alias("day"),
    pl.col("dt").dt.weekday().alias("weekday"),  # 1=Monday, 7=Sunday
    pl.col("dt").dt.hour().alias("hour")
)

To format a datetime back into a string, use dt.strftime with a format string. This is useful for building keys or exporting data.

df.with_columns(
    pl.col("dt").dt.strftime("%Y-%m-%d").alias("date_str")
)

For date-only columns, Polars has a separate Date type. You can convert using pl.col("dt").cast(pl.Date) to truncate the time component.

String Cleaning and Pattern Matching with Polars

String operations in Polars are available under the .str namespace. Common tasks include stripping whitespace, changing case, and extracting substrings with regular expressions.

df = pl.DataFrame({"name": ["  Alice  ", "Bob", "CAROL"]})

df.with_columns(
    pl.col("name").str.strip().alias("trimmed"),
    pl.col("name").str.to_lowercase().alias("lower"),
    pl.col("name").str.contains("^[A-Z]+", literal=False).alias("all_caps")
)

For regex extraction, use str.extract with a capture group. The first match is returned as a string, or null if no match.

df = pl.DataFrame({"email": ["alice@example.com", "bob@test.org"]})
df.with_columns(
    pl.col("email").str.extract(r"@(\w+)", 1).alias("domain")
)

When you need to replace patterns, str.replace and str.replace_all are available. The literal parameter controls whether the pattern is treated as a literal string or regex.

df.with_columns(
    pl.col("name").str.replace("a", "x", literal=True).alias("replaced")
)

Polars also supports str.split and str.slice for more granular manipulation. These methods operate on the entire column at once, which is more efficient than Python-level loops.

Date Arithmetic and Time Differences

Polars supports adding and subtracting durations from datetime columns using dt.offset_by or direct arithmetic with pl.duration. This is essential for calculating time windows, expirations, or delays.

df = pl.DataFrame({"start": ["2024-01-15 08:00:00", "2024-02-01 12:00:00"]}).with_columns(
    pl.col("start").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S")
)

df.with_columns(
    (pl.col("start") + pl.duration(days=7)).alias("plus_week"),
    pl.col("start").dt.offset_by("1mo").alias("plus_month")
)

To compute the difference between two datetime columns, subtract them directly. The result is a Duration type, which you can convert to seconds, days, or other units using .dt.total_seconds() or .dt.total_days().

df = pl.DataFrame({
    "start": ["2024-01-15 08:00:00"],
    "end": ["2024-01-16 10:30:00"]
}).with_columns(
    pl.col("start").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S"),
    pl.col("end").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S")
)

df.with_columns(
    (pl.col("end") - pl.col("start")).alias("elapsed")
).with_columns(
    pl.col("elapsed").dt.total_hours().alias("hours")
)

For calendar-aware arithmetic like adding months, dt.offset_by respects month boundaries and handles year transitions correctly. Direct pl.duration addition is fixed-length and may not behave as expected for months.

Handling Time Zones and Offsets

Polars supports timezone-aware datetimes using the time_zone parameter in pl.Datetime. You can convert between time zones with dt.convert_time_zone.

df = pl.DataFrame({"ts": ["2024-01-15 08:30:00"]}).with_columns(
    pl.col("ts").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S").dt.replace_time_zone("UTC")
)

df.with_columns(
    pl.col("ts").dt.convert_time_zone("America/New_York").alias("ny_time")
)

When parsing strings that include an offset, str.to_datetime will automatically create a timezone-aware column. To convert to a common timezone for comparison, use convert_time_zone after ensuring the column is aware.

Be careful with ambiguous or nonexistent times during daylight saving transitions. Polars follows the IANA timezone database, but you must decide how to handle these cases. The default behavior is to raise an error for nonexistent times; you can use ambiguous and nonexistent parameters in str.strptime to control this.

Performance Considerations for String and Datetime Operations

Polars' performance advantage comes from columnar execution and lazy evaluation. However, certain operations can degrade performance if used carelessly.

  • Avoid Python-level loops: Instead of iterating rows, use expression methods. The entire column is processed in a vectorized manner.
  • Use str.to_datetime over str.strptime when possible: The ISO parser is highly optimized and avoids format-string overhead.
  • Be mindful of regex complexity: Complex regex patterns can become a bottleneck. Prefer simpler patterns or use literal=True when you don't need regex.
  • Cast to Date when time is irrelevant: A Date column uses less memory and can speed up operations that don't need time precision.
  • Use lazy evaluation: Build your transformations with pl.LazyFrame and call collect() at the end. Polars can optimize the query plan, especially for filters and joins involving datetime columns.

For example, if you're filtering on a date range, pushing the filter down in a lazy query avoids materializing the full column.

q = (
    pl.scan_csv("data.csv")
    .with_columns(pl.col("date").str.to_datetime())
    .filter(pl.col("date") >= pl.datetime(2024, 1, 1))
)
df = q.collect()

This approach is not only faster but also more memory-efficient, as Polars can skip reading rows that don't match the filter.

Another subtle point: when you chain many string operations, consider whether you can combine them into a single regex or use str.replace_all instead of multiple str.replace calls. Each operation creates a new column, so reducing the number of passes over the data improves performance.

Finally, if you are working with very large datasets, be aware that string operations that produce new columns will increase memory usage. Use select to keep only the columns you need, and drop intermediate columns with drop when they are no longer required.

Polars' expression system makes it easy to write clean, maintainable code for string and datetime transformations, but the real payoff comes when you combine these operations with lazy evaluation and the efficient native implementations. Understanding how each operation behaves under the hood lets you avoid common pitfalls and keep your pipelines fast and reliable.

python polars string and datetime operations: Practical Usag | RYUSLOG DEV