Python Context Manager: The with Statement Explained

When you write with open("data.txt") as f:, Python calls f.__enter__() before the block and f.__exit__(exc_type, exc_value, traceback) after the block, even if the block raises an exception. That guarantee is the entire point: cleanup runs regardless of how the block exits. The python context manager protocol is just those two methods, and anything that implements them can be used with with.

The as clause is optional. If you don't need the object inside the block, you can write with lock: and skip the binding. The __enter__ return value is what gets bound, and it doesn't have to be the same object as the context manager itself.

What the with Statement Actually Guarantees

The with statement is a syntactic contract for deterministic cleanup. When the block body completes normally, __exit__ is called with all three exception arguments set to None. When the body raises, __exit__ receives the exception type, value, and traceback. The method then decides whether to suppress the exception or let it propagate.

This matters most for resources that must be released even when something goes wrong. A file handle left open after an exception can exhaust file descriptors in a long-running process. A lock that is never released can deadlock every other thread waiting on it. The with statement removes the need to remember cleanup in every branch of a try/finally block.

The protocol is intentionally small. There is no registration, no inheritance requirement, and no interface to declare. Any object with correctly implemented __enter__ and __exit__ methods works. That simplicity is why the pattern appears throughout the standard library: files, sockets, locks, subprocesses, and temporary directories all expose it.

The Two Methods That Make a Context Manager

__enter__(self) should return the resource you want inside the block. __exit__(self, exc_type, exc_value, traceback) receives the exception state and returns True if it handled the exception, or False/None to let it propagate.

class ManagedFile:
    def __init__(self, path):
        self.path = path

    def __enter__(self):
        self.file = open(self.path, "w")
        return self.file

    def __exit__(self, exc_type, exc_value, traceback):
        self.file.close()
        return False

The exc_type, exc_value, and traceback arguments are None when the block completes normally. When an exception occurs, they carry the exception details. Returning False re-raises the exception; returning True suppresses it.

A class-based context manager is the most explicit form, but it is also the most verbose. For a simple setup and teardown, you write two methods, manage instance state, and decide what to return from __enter__. That structure pays off when the context manager holds state across entry and exit, such as a connection that needs to track whether a transaction is active.

Generator-Based Context Managers With contextlib.contextmanager

Writing a class for every context manager is tedious. The contextlib.contextmanager decorator lets you write a generator function where yield marks the boundary between __enter__ and __exit__:

from contextlib import contextmanager
import time

@contextmanager
def timed_operation(name):
    start = time.perf_counter()
    try:
        yield
    finally:
        elapsed = time.perf_counter() - start
        print(f"{name} took {elapsed:.3f}s")

Code before yield runs on entry, and code after yield runs on exit. The finally block ensures cleanup runs even when the body raises. If the body raises, the exception is thrown into the generator at the yield point, so you can catch it inside the generator if you need to.

The generator form is usually the better default for simple cases because it keeps the entry and exit logic in one place. You read the function top to bottom: acquire, yield, release. There is no separate class to inspect and no instance state to track.

The overhead difference is rarely measurable in application code. A generator frame allocation is cheap compared to opening a file or acquiring a lock. Optimize only if profiling shows the context manager itself is a bottleneck.

Suppressing Exceptions From Inside __exit__

Returning True from __exit__ swallows the exception, which is occasionally useful but easy to misuse. The standard library's contextlib.suppress is a safer way to express intentional suppression:

from contextlib import suppress
import os

with suppress(FileNotFoundError):
    os.remove("temp_file.json")

This is equivalent to a try/except that passes, but it makes the intent visible at the call site. Writing a custom __exit__ that returns True unconditionally is usually a mistake because it hides bugs. If the body raises a ValueError that you did not anticipate, the context manager silently discards it, and the caller never learns that the operation failed.

Suppression is appropriate only when the context manager is the correct owner of the error. A file cleanup that should ignore a missing file is one example. A database transaction that should roll back and re-raise is not.

What Happens When the Body Raises

The exception propagation rules differ between the two implementations. With a class, __exit__ receives the exception and decides whether to suppress it. With a generator, the exception is raised inside the generator at the yield statement, so a try/except around the yield can intercept it.

@contextmanager
def ignore_value_error():
    try:
        yield
    except ValueError:
        print("suppressed")

This works, but the same caution applies: suppressing exceptions in a context manager makes failures invisible to the caller. Reserve it for cases where the context manager is the correct owner of the error, such as closing a resource that is already gone.

There is another subtlety. If __exit__ itself raises an exception, that new exception replaces the original one. The traceback of the original failure is lost, which makes debugging significantly harder. Keep cleanup code defensive: close operations should not raise, and if they can, catch and log instead of propagating.

Practical Patterns: Locks, Connections, and Transactions

Context managers shine where resource lifecycle is easy to get wrong. A threading lock is a common example:

import threading

counter = 0
lock = threading.Lock()

with lock:
    counter += 1

The lock is released on every exit path, including exceptions. Without the with statement, you would need a try/finally block and remember to call lock.release() in the finally clause.

Database connections follow the same pattern, but transaction semantics add a decision point: should the transaction commit or roll back when the body raises?

@contextmanager
def transaction(connection):
    try:
        yield
        connection.commit()
    except:
        connection.rollback()
        raise

The raise in the except block re-raises the original exception after rollback, so the caller still sees the failure. This is a case where the context manager owns the error-handling policy and the caller does not need to know about rollback.

Another common pattern is timing or metrics collection. A context manager can record the duration of a block, send it to a metrics system, and ensure the measurement is taken even when the block fails. The generator form is ideal here because the timing logic reads naturally around the yield.

Performance and Overhead Considerations

The with statement itself adds negligible overhead: two method calls on entry and exit. The real cost is whatever the __enter__ and __exit__ methods do. Opening a file, acquiring a lock, or starting a network transaction dominates the cost, not the protocol machinery.

One subtle performance point: contextlib.contextmanager wraps the generator in a _GeneratorContextManager object, and the generator machinery has slightly more overhead than a plain class. For most code, that difference is unmeasurable. If you are building a context manager that runs in a hot loop, a class-based implementation avoids the generator frame allocation, but measure before optimizing.

There is also a memory consideration. A context manager that holds a large buffer in instance state keeps that buffer alive as long as the manager object is referenced. The generator form releases the frame after the generator is exhausted, which can free local variables earlier. In practice, this matters only for very large allocations held across many context manager instances.

Common Mistakes and Edge Cases

A frequent error is forgetting that __enter__ must return the resource. If you write:

class BadFile:
    def __init__(self, path):
        self.path = path

    def __enter__(self):
        open(self.path, "w")  # no return

    def __exit__(self, *args):
        pass

then with BadFile("x.txt") as f: binds f to None, and the file object is garbage collected immediately because nothing references it. The file may be closed before you use it, or it may not, depending on the interpreter. The fix is to return the resource from __enter__.

Another edge case is re-entering a context manager. A context manager that acquires a lock in __enter__ and releases it in __exit__ cannot be re-entered without deadlock if the lock is not reentrant. The standard library's threading.RLock handles this, but a custom implementation must decide whether re-entry is supported.

A third edge case involves __exit__ receiving None for all arguments. Code that assumes an exception always occurred will break when the block completes normally. Always check exc_type is not None before inspecting the exception value.

When Not to Use a Context Manager

Not every setup/teardown pair deserves a context manager. If the setup and teardown are used once in a script, a try/finally block is clearer than a custom class. Context managers earn their keep when the same lifecycle appears in many places, or when the resource must be released deterministically even in the presence of exceptions.

A one-off file write is fine with a plain with open(...). A connection pool, a distributed lock, or a metrics span is a better candidate for a custom context manager because the cleanup logic is nontrivial and repeated across many call sites.

Use a class-based implementation when the context manager must hold state between entry and exit, such as tracking whether a transaction is active or buffering data that is flushed on exit. Use the generator form when the lifecycle is a simple acquire/release pair with no intermediate state. Use contextlib.suppress when you only need to ignore a specific exception and do not need custom cleanup at all.

The decision is about ownership of cleanup logic. If the caller should decide how to handle an error, keep the context manager thin and let exceptions propagate. If the context manager owns the resource and its failure policy, it should handle cleanup internally and re-raise only when the caller needs to know about the failure.