Python Watchdog: Recursive Directories and Filters

python watchdog recursive directories and filters requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

To watch a directory tree with Python's watchdog library and act only on specific files, you need to combine recursive scheduling with pattern-based event filtering. The watchdog package provides an Observer that dispatches filesystem events to handlers, and its PatternMatchingEventHandler lets you filter those events by filename patterns. This article explains how to set up recursive directory watching, apply filters, and avoid common pitfalls when monitoring large or active trees.

Setting Up Watchdog

Install the package from PyPI:

pip install watchdog

Watchdog uses platform-specific native APIs when available. On Linux it relies on inotify, on macOS on FSEvents, and on Windows on ReadDirectoryChangesW. The Observer class abstracts these differences, so your code behaves consistently across operating systems.

A minimal setup looks like this:

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class MyHandler(FileSystemEventHandler):
    def on_any_event(self, event):
        print(event)

observer = Observer()
observer.schedule(MyHandler(), path='.')
observer.start()

This watches the current directory non-recursively. To watch subdirectories, you must pass recursive=True to schedule.

Watching a Directory Recursively

The recursive parameter controls whether the observer watches only the given path or the entire subtree. Setting it to True is straightforward:

observer.schedule(MyHandler(), path='./project', recursive=True)

With this, events from ./project, ./project/src, ./project/src/utils, and any deeper directory are delivered to the handler. The event.src_path attribute contains the full path, and event.is_directory tells you whether the event concerns a directory or a file.

Recursive watching is essential when you need to track changes in a project tree, a configuration directory, or a data folder. However, it also increases the number of events the observer must process, which makes filtering more important.

Filtering Events with Patterns

Watchdog provides PatternMatchingEventHandler, a subclass of FileSystemEventHandler that accepts patterns and ignore_patterns lists. Patterns use fnmatch syntax, so *.py matches any Python file, *.log matches log files, and **/*.tmp matches temporary files in any subdirectory.

from watchdog.events import PatternMatchingEventHandler

handler = PatternMatchingEventHandler(
    patterns=['*.py'],
    ignore_patterns=['*/migrations/*'],
    ignore_directories=True
)

Here the handler will only receive events for files whose names end with .py, except those under a migrations directory. Setting ignore_directories=True filters out directory events entirely, which is often what you want when you only care about file content changes.

The handler still needs to be scheduled with the observer, and the recursive flag applies as before:

observer.schedule(handler, path='./project', recursive=True)

Patterns are matched against the basename of the path by default. If you need to match against the full path, you can pass case_sensitive=False or use fnmatch patterns that include slashes. For example, '**/test_*.py' matches any Python file whose name starts with test_ anywhere in the tree.

Handling Different Event Types

PatternMatchingEventHandler inherits from FileSystemEventHandler, so you can override on_created, on_modified, on_deleted, and on_moved. Each method receives an event object with attributes like src_path and, for moves, dest_path.

class PythonChangeHandler(PatternMatchingEventHandler):
    def on_modified(self, event):
        print(f"Modified: {event.src_path}")

    def on_created(self, event):
        print(f"Created: {event.src_path}")

    def on_deleted(self, event):
        print(f"Deleted: {event.src_path}")

    def on_moved(self, event):
        print(f"Moved: {event.src_path} -> {event.dest_path}")

Keep in mind that a single file write can generate multiple modified events, especially when an editor saves atomically by writing a temporary file and renaming it. In that case you may see created and moved events instead of modified. Your handler should be idempotent or debounce repeated events if you are triggering expensive actions.

A Complete Recursive Watch with Filters

Putting it together, here is a script that watches a project directory recursively and logs only changes to Python files, ignoring __pycache__ and .git directories:

import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler

class PythonHandler(PatternMatchingEventHandler):
    def __init__(self):
        super().__init__(
            patterns=['*.py'],
            ignore_patterns=['*/__pycache__/*', '*/.git/*'],
            ignore_directories=True
        )

    def on_modified(self, event):
        print(f"{event.src_path} modified")

    def on_created(self, event):
        print(f"{event.src_path} created")

    def on_deleted(self, event):
        print(f"{event.src_path} deleted")

observer = Observer()
handler = PythonHandler()
observer.schedule(handler, path='./my_project', recursive=True)
observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

This script runs until interrupted. The ignore_patterns list prevents events from __pycache__ and .git directories from reaching the handler, which reduces noise and avoids unnecessary processing.

Performance and Resource Considerations

Recursive watching with filters is efficient when the filter is applied early in the event pipeline. Watchdog's PatternMatchingEventHandler checks the pattern before invoking your callback, so ignored files do not trigger your code. However, the observer still receives and processes every filesystem event in the tree. On a very large directory with thousands of files, the kernel and the Python process will see a high volume of events, even if you ignore most of them.

If you need to monitor a tree with tens of thousands of files, consider whether you can narrow the watch path to a subdirectory that contains only the files of interest. For example, instead of watching the entire project, watch only the src directory if that is where changes matter.

Another concern is event coalescing. Watchdog does not batch events by default; each filesystem event is dispatched individually. If your handler performs a slow operation, such as rebuilding a cache or restarting a service, you should debounce it. A simple approach is to record the last event time and only act if a certain interval has passed.

On Linux, the default inotify mechanism has a limited event queue. If events are produced faster than the observer can consume them, the kernel may drop events or raise an overflow warning. Using ignore_directories=True and specific patterns reduces the number of events the observer must process, which helps avoid this problem.

Common Pitfalls and Edge Cases

Symlinks and recursion. By default, watchdog does not follow symbolic links. If you have a symlink that points outside the watched tree, changes to the target will not generate events. To follow symlinks, you would need to add separate watches for each link target, but watchdog does not provide a built-in option for that.

File moves across directories. When a file is moved from one watched directory to another, watchdog may emit both a deleted event for the source and a created event for the destination. The on_moved method is only called when the move happens within the same watched tree and the observer can correlate the two paths. If you rely on on_moved, test your specific scenario.

Permission errors. If the process lacks read permission on a subdirectory, watchdog may silently ignore events from that subtree. On Linux, inotify requires read permission on the directory itself. Running your script with the appropriate user or group permissions is necessary for full coverage.

Event duplication on editors. Many editors write to a temporary file and then rename it over the original. This can produce a created event for the temp file, a moved event, and a modified event for the target. If you are using patterns, the temp file often has a different extension (e.g., .swp or ~), so it will be filtered out. But if your pattern is broad, you may see extra events. Adjust your ignore_patterns to exclude common temporary file suffixes.

Recursive watch on a network filesystem. Watchdog relies on the underlying filesystem's notification mechanism. Network filesystems like NFS or SMB often do not support these notifications reliably. In such environments, you may need to fall back to periodic polling with a library like watchdog's PollingObserver, which scans the directory tree at a fixed interval and compares snapshots.

from watchdog.observers.polling import PollingObserver

observer = PollingObserver()
observer.schedule(handler, path='./project', recursive=True)
observer.start()

PollingObserver is less efficient than the native observer, but it works on filesystems that do not generate events. Use it only when necessary.

When you combine recursive directory watching with pattern filters, you get a precise and responsive file monitoring solution. The key is to apply the filter as early as possible, handle event types deliberately, and be aware of the platform-specific behavior that can affect event delivery.