RDRYUSLOG DEV
Back to Blog
Python

Python Loguru File Logging: Rotation and Retention

python loguru file logging rotation and retention: Configure Loguru file logging with rotation and retention: syntax, options, behavior, and production considerations...

LoguruPython LoggingLog RotationLog RetentionLogging Configuration
Illustration of rotating log files with a retention window, showing old files being archived and deleted over time.

python loguru file logging rotation and retention requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you move from console output to file-based logging with Loguru, rotation and retention are the two parameters that keep log files from growing without bound. This article covers how to configure them, what each option actually does, and where the behavior can surprise you in production.

Adding a File Sink with Rotation and Retention

Loguru writes to files through the add() method, which returns a handler ID. The simplest file sink looks like this:

from loguru import logger

logger.add("app.log")

Without rotation, app.log grows indefinitely. To bound the file size, pass a rotation argument:

logger.add("app.log", rotation="500 MB")

When the current file reaches 500 MB, Loguru closes it and starts a new one. The old file is renamed with a timestamp, for example app.log.2025-03-14_12-30-00_000000. The retention argument then controls how many of those older files are kept:

logger.add("app.log", rotation="500 MB", retention="10 days")

This combination is the core of python loguru file logging rotation and retention: a size-based rotation to keep each file manageable, and a time-based retention to avoid accumulating an unbounded number of archived files.

How Rotation Works

rotation accepts several forms:

  • A string with a size, like "100 MB" or "1 GB".
  • A string with a time interval, like "1 day", "12 hours", or "1 week".
  • A specific time of day, like "12:00" (rotation happens at noon local time).
  • A callable that receives the current file path and returns True when rotation should occur.

Size-based rotation is checked after each log message is written. If the file size exceeds the threshold, the file is rotated. Time-based rotation is checked on a timer; the first rotation occurs at the next interval boundary. For "12:00", the first rotation happens at the next noon, even if the file was just created.

A callable gives you full control. For example, you can rotate when the file reaches a certain size and a certain time has passed:

import os
from loguru import logger

def should_rotate(file_path):
    return os.path.getsize(file_path) > 100_000_000 and "error" in file_path

logger.add("app.log", rotation=should_rotate)
```n
The callable runs after each log write, so it can inspect the file's current state. Use this when the built-in string options are not enough.

## How Retention Works

`retention` removes old log files after a certain time. It accepts:

- A timedelta object, like `timedelta(days=7)`.
- A string with a time interval, like `"7 days"` or `"1 month"`.
- An integer, which means the number of files to keep.
- A callable that receives a list of file paths and returns the ones to delete.

When retention is an integer, Loguru keeps that many most recent files and deletes the rest. For example, `retention=5` keeps the current file plus the four most recent rotated files. This is often more predictable than time-based retention when you have irregular log volume.

Time-based retention deletes files whose modification time is older than the given interval. The check happens at rotation time and at startup, so files that become stale while the process is running are cleaned up on the next rotation event.

A callable gives you complete control over which files to delete. It receives a list of file paths and should return the ones to remove:

```python
def clean_old_logs(file_paths):
    return [p for p in file_paths if "debug" in p]

logger.add("app.log", retention=clean_old_logs)

This is useful when you want to keep certain files longer or apply a different policy than the built-in options.

Combining Rotation and Retention

Rotation and retention work independently. Rotation decides when to create a new file; retention decides which old files to delete. They are often used together, but you can use only one.

A common pattern is size-based rotation with a retention count:

logger.add("app.log", rotation="50 MB", retention=20)

This keeps the log directory bounded to roughly 1 GB (20 files × 50 MB). Time-based retention with time-based rotation can be simpler to reason about:

logger.add("app.log", rotation="1 day", retention="30 days")

Here you keep a month of daily files. The exact number of files depends on whether the process runs continuously and whether rotation happens exactly at midnight.

One subtlety: retention only considers files that Loguru itself created with the same sink pattern. If you manually add files to the directory, they are not considered for deletion. Loguru tracks the pattern you passed to add() and only applies retention to files matching that pattern.

Compression and Other File Options

Loguru can compress rotated files automatically with the compression parameter. Supported formats include gz, bz2, xz, and zip. For example:

logger.add("app.log", rotation="100 MB", retention="30 days", compression="gz")

When a file is rotated, Loguru compresses it and adds the extension .gz. The original uncompressed file is removed. This reduces disk usage but adds CPU overhead during rotation. For high-volume logs, consider whether compression is worth the cost.

Other useful file options include enqueue=True, which makes logging thread-safe and asynchronous, and backtrace and diagnose for exception details. The enqueue option is especially relevant when you have multiple threads writing to the same sink.

Common Pitfalls with Rotation and Retention

Rotation Does Not Trigger on Process Exit

If your process runs for a short time and exits, rotation may never occur. Size-based rotation only checks after a log message is written. If the process exits before the size threshold is reached, the file is not rotated. Time-based rotation also only triggers while the process is alive. If you need rotation to happen on shutdown, you must call logger.remove() or logger.complete() explicitly, but even then Loguru does not force a rotation based on size or time.

Retention Runs at Startup and Rotation

Retention is not a background job. It runs when the sink is added and when a rotation occurs. If your process runs for months without rotating, old files are not cleaned up until the next rotation. If you rely on time-based retention, make sure rotation happens frequently enough to trigger cleanup.

Time Zone and DST

Time-based rotation uses local time. If your server changes time zones or observes daylight saving time, a "12:00" rotation may occur at an unexpected moment. For most applications this is acceptable, but if you need strict UTC-based rotation, use a callable that checks the current UTC time.

Retention with a Callable Can Delete the Current File

If your retention callable is not careful, it could return the current active log file path. Loguru will delete it, which can cause errors on subsequent writes. The callable receives all files matching the pattern, including the one currently being written. Filter it out unless you intentionally want to rotate immediately.

Changing the Pattern Breaks Retention

If you change the file name pattern in add(), Loguru treats the new pattern as a separate sink. Old files from the previous pattern are not considered for retention. If you need to clean up files from an old pattern, do it manually or use a separate cleanup routine.

Production Considerations

In production, the most important decision is whether to use size-based or time-based rotation. Size-based rotation keeps each file under a predictable maximum, which is useful when you ship logs to an external system that has a file size limit. Time-based rotation makes it easier to correlate logs with a specific date, which is useful for debugging and auditing.

Retention should be aligned with your compliance or debugging needs. If you are required to keep logs for a certain period, use time-based retention with a margin. If you only need the most recent logs, a retention count is simpler and prevents disk usage from growing with log volume.

Also consider the interaction between rotation and the rest of your logging pipeline. If you use enqueue=True, log messages are written asynchronously, so rotation may happen slightly after the size threshold is reached. This is usually fine, but if you need exact file boundaries, use a synchronous sink.

Finally, monitor the log directory. Even with rotation and retention, a misconfigured sink can fill the disk. A simple cron job or monitoring alert that checks the total size of the log directory can catch problems before they affect the application.

Loguru's rotation and retention give you a compact way to manage file growth without external tools. Understanding how the options behave at runtime lets you choose the right combination for your workload and avoid the common pitfalls that appear only after the process has been running for a while.

python loguru file logging rotation and retention: Practical | RYUSLOG DEV