Python Celery: Redis and RabbitMQ Brokers, Result Backends

python celery redis rabbitmq brokers and result backends requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you configure Python Celery with Redis and RabbitMQ brokers and result backends, two infrastructure pieces determine how tasks travel and where their results land: the broker and the result backend. The broker transports task messages from your application to the workers. The result backend stores the state and return value of each executed task. Redis and RabbitMQ are the two most common choices for these roles, and they are frequently combined rather than treated as mutually exclusive alternatives.

How the Broker and Result Backend Fit Together

Celery separates message transport from result storage. A task is published to the broker as a message. A worker consumes that message, executes the task, and writes the outcome to the result backend. The application that called the task can then read the result from the backend.

This separation means you are not forced to use the same technology for both roles. A common production configuration uses RabbitMQ as the broker and Redis as the result backend. Another valid configuration uses Redis for both. The choice depends on reliability requirements, feature needs, and operational constraints.

The two relevant settings in your Celery configuration are broker_url and result_backend. Both accept a URL string that identifies the transport and its connection details.

Configuring RabbitMQ as the Broker

RabbitMQ is the broker Celery was originally designed around. It implements the AMQP protocol and provides durable queues, message acknowledgments, and priority support. For production workloads where message loss is unacceptable, RabbitMQ is the safer default.

A minimal configuration using RabbitMQ as the broker looks like this:

broker_url = "amqp://user:password@localhost:5672//"
result_backend = "redis://localhost:6379/0"

The amqp:// scheme points at the RabbitMQ server. The URL includes the credentials, host, port, and virtual host. The trailing // selects the default virtual host. If you use a dedicated vhost, replace it with the vhost name.

RabbitMQ's behavior matters for task delivery. Workers acknowledge messages after processing, and Celery can be configured to acknowledge late or early. With task_acks_late = True, a worker acknowledges a message only after the task completes, so a crashed worker causes the message to be redelivered rather than lost.

Configuring Redis as the Broker

Redis can also serve as the broker. Celery implements the Redis broker using Redis lists and blocking list operations rather than pub/sub. Messages are pushed to a list, and workers pop them with BRPOPLPUSH, which moves the message to a processing list while it is being handled.

broker_url = "redis://localhost:6379/0"
result_backend = "redis://localhost:6379/1"

The URL selects a database number after the port. Using a separate database for the broker and the result backend avoids one role interfering with the other's keyspace.

The Redis broker has a notable operational parameter: broker_transport_options with visibility_timeout. Because Redis does not track message acknowledgments the way AMQP does, Celery uses a visibility timeout to decide when a message that was popped but never acknowledged should be made visible to other workers again. The default is one hour. If a task runs longer than the visibility timeout, another worker may pick up the same message and execute the task twice. For long-running tasks, increase the timeout:

broker_transport_options = {"visibility_timeout": 7200}

This is the main reliability difference between the two brokers. RabbitMQ tracks acknowledgment state explicitly; Redis relies on a timeout heuristic.

Using Redis as the Result Backend

Redis is the most widely used result backend for Celery. It stores task state and results as keys with a configurable expiration. Results are written when the task finishes, and the calling application reads them by task ID.

result_backend = "redis://localhost:6379/1"
result_expires = 86400

result_expires controls how long results remain in Redis before Celery removes them. The default is one day. In a high-throughput system, results accumulate quickly, so setting a sensible expiration prevents Redis memory from growing without bound.

Redis as a result backend stores everything in memory. A task that returns a multi-megabyte structure will consume memory for the duration of result_expires. Keep result payloads small and set an expiration that matches how long callers actually need the data.

Mixing RabbitMQ Broker with Redis Result Backend

The combination of RabbitMQ as the broker and Redis as the result backend is common in production. It gives you reliable message delivery from RabbitMQ while keeping result storage fast and simple in Redis.

broker_url = "amqp://user:password@localhost:5672//"
result_backend = "redis://localhost:6379/1"
task_acks_late = True
worker_prefetch_multiplier = 1

worker_prefetch_multiplier controls how many messages a worker reserves at once. A value of one keeps tasks distributed evenly across workers, which matters when task durations vary. This setting applies regardless of which broker you use, but it is especially relevant when you want to avoid one slow worker holding many queued messages.

This mixed setup decouples the failure modes of the two systems. If Redis is temporarily unavailable, workers can still consume and execute tasks from RabbitMQ; only result storage fails. If RabbitMQ is unavailable, new tasks cannot be published, but the result backend remains reachable for reading previously stored results.

Failure Modes and Operational Behavior

Understanding what happens when each component fails helps you choose the right combination.

If the Redis broker loses connectivity, workers stop consuming and the application cannot publish tasks. Because Redis is an in-memory store, messages that were not yet consumed are lost if the Redis process restarts without persistence. Redis persistence options such as RDB snapshots or AOF can reduce this risk, but they do not provide the same delivery guarantees as AMQP.

If RabbitMQ is the broker, messages published to a durable queue survive a broker restart. Workers that crash before acknowledging a message cause redelivery when task_acks_late is enabled. This makes RabbitMQ the stronger choice when task loss is unacceptable.

The result backend fails differently. A task that completes successfully but cannot write its result to Redis will be marked as failed from the caller's perspective, even though the task logic ran. This is why result storage should be monitored separately from the broker. If results are not critical, you can disable the result backend entirely by leaving result_backend unset, and Celery will not store results at all.

Choosing Between Redis and RabbitMQ

The decision depends on what you are optimizing for.

Use RabbitMQ when tasks must not be lost and when you need priority queues or strict delivery semantics. Use Redis as the broker when you already run Redis, when the deployment must stay simple, and when occasional duplicate execution under a worker crash is acceptable.

The result backend choice is more one-sided. Redis is the standard choice because it is fast, simple to configure, and supports expiration. RabbitMQ can act as a result backend through the rpc:// transport, but that mode is tied to a single worker and is not suitable for most multi-worker deployments.

A practical starting point for a new service is RabbitMQ for the broker and Redis for the result backend, with task_acks_late enabled and result_expires set to match how long callers need results.