Python Task Cancellation in Asyncio

Long-running asyncio tasks often need to be stopped before they finish naturally. A user may abort a request, a server may be shutting down, or a timeout may expire. Python task cancellation gives you a structured way to interrupt coroutines and run cleanup logic. The core mechanism is Task.cancel(), which schedules a CancelledError to be raised inside the task at the next suspension point. Understanding how that exception propagates, how to handle it, and how to protect critical sections is essential for building reliable concurrent applications.

The Problem with Unmanaged Long-Running Tasks

Consider a coroutine that polls an external service or processes a large queue. Without cancellation, you have no clean way to stop it once it starts. You could set a flag and check it periodically, but that requires cooperation from the coroutine and does not interrupt blocking calls. Asyncio's cancellation model is designed to interrupt a task at an await point, which is where most I/O and suspension happens. This makes it possible to stop a task even when it is blocked on a socket or a sleep.

Cancelling a Task with Task.cancel()

The Task.cancel() method requests cancellation. It does not stop the task immediately; it schedules a CancelledError to be thrown into the coroutine at the next await. The task will not actually stop until it reaches that point. If the task is already running and not awaiting, cancellation will not take effect until it yields control.

import asyncio

async def worker():
    try:
        while True:
            await asyncio.sleep(1)
            print("working")
    except asyncio.CancelledError:
        print("worker cancelled")
        raise

async def main():
    task = asyncio.create_task(worker())
    await asyncio.sleep(2)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("main: task was cancelled")

asyncio.run(main())

In this example, task.cancel() is called after two seconds. The worker is sleeping, so the cancellation is delivered immediately. The CancelledError is caught inside the coroutine, a message is printed, and the exception is re-raised so the task is marked as cancelled. The await task in main then raises CancelledError, which we catch to know the task ended.

Handling CancelledError Gracefully

CancelledError inherits from BaseException, not Exception. This means a bare except Exception will not catch it. You must explicitly catch asyncio.CancelledError if you need to run cleanup before the task ends. The typical pattern is to use try/finally or except CancelledError followed by raise.

async def resource_cleanup():
    resource = acquire()
    try:
        await use(resource)
    finally:
        await release(resource)

If cancellation occurs during await use(resource), the finally block runs and releases the resource. After the finally completes, the CancelledError propagates out of the coroutine, and the task is considered cancelled. You do not need to re-raise it explicitly in a finally block because the exception is already in flight. However, if you catch it in an except block and do not re-raise, the task will be marked as completed normally, which is usually not what you want.

async def bad_handler():
    try:
        await long_operation()
    except asyncio.CancelledError:
        # Swallowing the cancellation
        pass

This code prevents the task from being cancelled. The task will continue to run after the except block, which defeats the purpose of cancellation. Always re-raise CancelledError unless you have a very specific reason to suppress it.

Awaiting Cancellation: How to Ensure Cleanup Runs

When you call task.cancel(), the task may not finish immediately. To wait for the task to actually complete its cleanup, you must await the task. This is important in shutdown sequences where you want to give tasks a chance to release resources before the event loop closes.

async def shutdown(tasks):
    for task in tasks:
        task.cancel()
    # Wait for all tasks to finish their cleanup
    await asyncio.gather(*tasks, return_exceptions=True)

Using asyncio.gather with return_exceptions=True prevents CancelledError from propagating to the caller. Each task will run its finally blocks, and the gather completes once all tasks have fully stopped. If you do not await the cancelled tasks, they may still be running when the loop closes, leading to warnings or incomplete cleanup.

Protecting Critical Work with asyncio.shield()

Sometimes you have a section of code that must not be cancelled, even if the surrounding task is cancelled. asyncio.shield() protects a coroutine from cancellation by wrapping it in a separate task. When the outer task is cancelled, the shielded coroutine continues to run in the background.

async def critical_section():
    await asyncio.sleep(10)  # Simulate important work
    return "done"

async def main():
    task = asyncio.create_task(shielded_work())
    await asyncio.sleep(1)
    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("outer task cancelled, shielded work continues")

async def shielded_work():
    result = await asyncio.shield(critical_section())
    print(f"shielded result: {result}")

In this example, critical_section() is shielded. When task.cancel() is called, the outer coroutine receives CancelledError, but the shielded coroutine is not cancelled. It continues running in the background. The shielded_work coroutine will not resume until the shielded coroutine finishes, but since the outer task is cancelled, the result is lost. Use shield sparingly; it can leave background tasks running longer than expected, which may interfere with shutdown.

Designing Cooperative Cancellation

Cancellation in asyncio is cooperative. A task can only be cancelled at an await point. If your coroutine performs CPU-bound work without yielding, cancellation will not be delivered until that work completes. For long-running CPU-bound sections, you should periodically await asyncio.sleep(0) to yield control, or use asyncio.to_thread to run the work in a separate thread and await that.

async def cpu_bound():
    for i in range(1000000):
        # heavy computation
        if i % 1000 == 0:
            await asyncio.sleep(0)  # yield control

Even with periodic yields, cancellation is only delivered when the coroutine is suspended. If you have a loop that never awaits, cancellation will be delayed indefinitely. In such cases, consider checking a cancellation flag manually, but the standard mechanism is to ensure your coroutine awaits regularly.

Cancellation with Timeouts and TimeoutError

A common use case for cancellation is enforcing a timeout on an operation. asyncio.wait_for cancels the task if it does not complete within the given time. In Python 3.11+, asyncio.timeout provides a more flexible context manager.

async def slow_operation():
    await asyncio.sleep(10)
    return "result"

async def main():
    try:
        result = await asyncio.wait_for(slow_operation(), timeout=2)
    except asyncio.TimeoutError:
        print("operation timed out")

When the timeout occurs, wait_for cancels the inner task. The CancelledError is raised inside slow_operation, and if it has a finally block, that cleanup runs. After the task is cancelled, wait_for raises asyncio.TimeoutError to the caller. Note that TimeoutError is a built-in exception in Python 3.11 and later, but asyncio.TimeoutError is an alias for compatibility.

Common Pitfalls and Edge Cases

One common mistake is catching CancelledError and not re-raising it, which silently prevents cancellation. Another is calling task.cancel() on a task that has already completed; this has no effect and the task remains in its final state. If you cancel a task that is already cancelled, the second call does nothing.

When using asyncio.gather, if one child task is cancelled, the other tasks are not automatically cancelled. You need to propagate cancellation manually if that is the desired behavior. Also, asyncio.shield can be tricky: if the shielded coroutine is cancelled from outside, the shield does not protect it. The shield only protects against cancellation of the task that awaits the shield.

Finally, be aware that CancelledError can be raised at any await, including inside finally blocks. If your cleanup code itself awaits, it can be interrupted by a second cancellation. In practice, this is rare but worth handling by shielding critical cleanup or using asyncio.shield around the cleanup steps.

Understanding these edge cases helps you write cancellation logic that behaves predictably under shutdown, timeouts, and user-initiated aborts. The key is to always re-raise CancelledError, await cancelled tasks to allow cleanup, and use shield only when you truly need to protect a section from interruption.