Python FastAPI Background Tasks: Built-in vs Celery

When a FastAPI request handler needs to perform work that does not need to block the response—such as sending an email, updating a cache, or processing a file—the work can be moved to a background task. Python FastAPI background tasks provide two main approaches: the built-in BackgroundTasks class from Starlette, and a full task queue like Celery. This article explains both, when each makes sense, and how to handle common production concerns.

Using the Built-in BackgroundTasks Class

FastAPI includes BackgroundTasks directly from Starlette. It lets you register callable functions that run after the response is sent. The simplest usage is to declare a parameter of type BackgroundTasks in your endpoint and add a task with add_task.

from fastapi import FastAPI, BackgroundTasks

app = FastAPI()

def write_log(message: str):
    with open("log.txt", "a") as f:
        f.write(f"{message}\n")

@app.post("/send-notification")
async def send_notification(background_tasks: BackgroundTasks):
    background_tasks.add_task(write_log, "notification sent")
    return {"message": "Notification is being processed"}

Here, write_log runs after the response is returned to the client. The function is executed in the same process, and FastAPI waits for it to finish before the server considers the request complete. This is important: the background task does not run concurrently with the response; it runs after the response is sent, but it still occupies the worker process until it finishes.

For synchronous functions, FastAPI runs them in a thread pool so they do not block the event loop. For async functions, they run directly on the event loop. This distinction matters when your task performs blocking I/O, such as file writes or network calls. If you have a blocking operation and your task is defined as async, it will block the event loop. Prefer defining background tasks as regular def functions unless they are truly non-blocking.

What Happens When the Response Is Sent

The execution model of BackgroundTasks is tied to the response lifecycle. When an endpoint returns a response, FastAPI sends it to the client, then runs the registered background tasks. If the client disconnects before the response is fully sent, the background tasks may still run because the server has already committed to executing them. However, if the server process crashes or is restarted before the tasks complete, they are lost.

This behavior makes BackgroundTasks suitable for short, non-critical operations like writing a log entry, sending a simple email, or invalidating a cache. It is not suitable for tasks that must survive a crash, require retries, or take a long time to complete. The task runs in the same process, so a memory leak or a long-running task can degrade the health of the entire application.

Adding Multiple Tasks and Passing Arguments

You can add multiple tasks to the same BackgroundTasks object. They run sequentially in the order they were added. Each task can receive positional and keyword arguments.

def send_email(email: str, subject: str):
    # send email logic
    pass

def update_analytics(user_id: int):
    # update analytics
    pass

@app.post("/register")
async def register(user_id: int, background_tasks: BackgroundTasks):
    background_tasks.add_task(send_email, "user@example.com", "Welcome")
    background_tasks.add_task(update_analytics, user_id)
    return {"message": "User registered"}

There is no built-in mechanism for retrying failed tasks. If send_email raises an exception, the remaining tasks in the same request are still executed, but the exception is logged and the process continues. You must handle retries manually if your task requires them.

When the Built-in Approach Is Not Enough

For tasks that are long-running, require durable execution, need to be distributed across multiple workers, or must survive application restarts, BackgroundTasks is insufficient. Consider a video transcoding job that takes several minutes, or a nightly report generation that must be retried on failure. These scenarios require a dedicated task queue.

Celery is the most common choice for Python applications. It runs tasks in separate worker processes, communicates via a message broker (like Redis or RabbitMQ), and provides features such as retries, scheduling, and result storage. FastAPI integrates with Celery by simply calling Celery tasks from within endpoints; the HTTP response is returned immediately, and the worker handles the task asynchronously.

Setting Up Celery with FastAPI

To use Celery, you need a broker and a worker. The broker stores the task messages; the worker consumes them. Here is a minimal setup.

# tasks.py
from celery import Celery

celery_app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/0")

@celery_app.task
def process_file(file_path: str):
    # long-running processing
    return f"Processed {file_path}"

In your FastAPI application, you import the Celery task and call it with .delay().

from fastapi import FastAPI
from tasks import process_file

app = FastAPI()

@app.post("/process")
async def process(file_path: str):
    task = process_file.delay(file_path)
    return {"task_id": task.id, "status": "queued"}

The endpoint returns immediately with a task ID. The worker picks up the task and runs it independently. This decouples the HTTP request from the actual work, allowing the web server to handle more requests without being blocked by long operations.

Celery tasks can be monitored via the backend, which stores task results. You can query the task status from another endpoint or a separate service.

Comparing BackgroundTasks and Celery

Use BackgroundTasks when the task is quick, failure is acceptable, and you want minimal infrastructure. Use Celery when you need reliability, retries, or the ability to process tasks outside the web process.

Error Handling and Retries in Celery

Celery provides a robust retry mechanism. You can configure max_retries, retry_backoff, and retry_backoff_max on the task. Here is an example that retries up to three times with exponential backoff.

@celery_app.task(bind=True, max_retries=3, retry_backoff=True)
def send_email_task(self, email: str):
    try:
        # send email
        pass
    except Exception as exc:
        raise self.retry(exc=exc)

The bind=True argument gives the task access to self, which is required to call self.retry(). This pattern is useful for transient failures like network timeouts.

For BackgroundTasks, you would need to implement your own try/except and retry logic, which is error-prone and couples the task to the request lifecycle.

Production Considerations

When using BackgroundTasks, monitor the time each task takes. A task that hangs can exhaust the thread pool and degrade response times. Set timeouts if your framework allows it, or move long tasks to Celery.

With Celery, monitor the queue length and worker utilization. A growing queue indicates that workers cannot keep up with the incoming tasks. Scale workers horizontally or adjust concurrency settings. Also ensure that the broker is highly available; if Redis or RabbitMQ goes down, tasks are not accepted.

Both approaches require careful attention to error logging. BackgroundTasks exceptions are logged by the server but not exposed to the client. Celery tasks can be tracked by task ID, and results can be inspected. Use structured logging to correlate task IDs with request IDs for easier debugging.

Choosing the Right Approach for Your Use Case

The decision between BackgroundTasks and Celery comes down to the nature of the work. If the task is a fire-and-forget operation that must complete within a few seconds and losing it occasionally is acceptable, the built-in class is sufficient. If the task is long-running, must be retried on failure, or needs to be processed outside the web server's lifecycle, Celery is the appropriate tool.

For tasks that are short but require retries, you could also consider a lightweight queue like RQ or Dramatiq. The key is to avoid using BackgroundTasks for anything that is critical to your business logic, because it does not provide durability or retries. Instead, reserve it for auxiliary operations like logging, cache invalidation, or sending non-critical notifications.