Using Python tqdm with Multiprocessing and Asyncio

python tqdm multiprocessing and asyncio requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you run a loop in Python, tqdm gives you a simple progress bar. But when you distribute work across processes or run tasks concurrently with asyncio, the same straightforward usage breaks. The progress bar either never updates, updates erratically, or the output becomes interleaved with worker logs. This article shows how to use python tqdm with multiprocessing and asyncio so the progress bar reflects real work.

Why tqdm Needs Special Handling in Concurrent Code

A standard tqdm loop works because the loop body runs sequentially in the same process. When you use multiprocessing or asyncio, the work happens outside the main loop. With multiprocessing, each worker has its own memory space, so calling pbar.update() inside a worker does not affect the progress bar in the parent process. With asyncio, tasks run on the same thread but yield control, so a naive for loop that awaits tasks one by one will not show progress until all tasks finish unless you update the bar between awaits.

The core issue is that tqdm needs to be updated from the main thread or process, at the point where you collect results. The patterns below reflect that.

Using tqdm with multiprocessing.Pool

The most reliable way to use tqdm with a multiprocessing.Pool is to iterate over the results as they arrive, using imap or imap_unordered. This keeps the update in the parent process.

import multiprocessing
from tqdm import tqdm

def worker(x):
    # simulate work
    return x * x

if __name__ == "__main__":
    data = range(100)
    with multiprocessing.Pool() as pool:
        results = []
        for result in tqdm(pool.imap_unordered(worker, data), total=len(data)):
            results.append(result)

imap_unordered yields results as soon as they are ready, so the progress bar advances as each task completes. The total argument tells tqdm how many items to expect. Without it, tqdm would show an indeterminate bar.

If you need results in the original order, use imap instead. The progress bar will still update, but it may pause if a slow task blocks the order.

For a simpler API, tqdm provides tqdm.contrib.concurrent.process_map, which wraps a ProcessPoolExecutor and returns results in order:

from tqdm.contrib.concurrent import process_map

def worker(x):
    return x * x

results = process_map(worker, range(100), max_workers=4)

This function handles the progress bar internally and is useful when you do not need fine-grained control over the pool.

Using tqdm with asyncio Tasks

With asyncio, you typically gather tasks with asyncio.gather. To show progress, you can use tqdm.asyncio.tqdm, which provides an async-compatible progress bar. The simplest way is to replace asyncio.gather with tqdm.gather:

import asyncio
from tqdm.asyncio import tqdm

async def worker(x):
    await asyncio.sleep(0.1)
    return x * x

async def main():
    data = range(100)
    tasks = [worker(x) for x in data]
    results = await tqdm.gather(*tasks, total=len(tasks))
    return results

if __name__ == "__main__":
    asyncio.run(main())

tqdm.gather behaves like asyncio.gather but updates the progress bar as each task completes. It returns results in the same order as the input tasks.

If you need more control, you can use asyncio.as_completed with a standard tqdm object:

import asyncio
from tqdm import tqdm

async def main():
    tasks = [worker(x) for x in range(100)]
    pbar = tqdm(total=len(tasks))
    for coro in asyncio.as_completed(tasks):
        await coro
        pbar.update()
    pbar.close()

This pattern lets you process each result as it arrives, but it requires manual progress bar management. The tqdm.gather approach is usually cleaner.

Choosing Between multiprocessing and asyncio for Progress Tracking

The choice of concurrency model depends on the workload, not on tqdm. Use multiprocessing when the work is CPU-bound and benefits from multiple cores. Use asyncio when the work is I/O-bound, such as network requests or file reads, where most time is spent waiting.

tqdm works with both, but the integration differs. With multiprocessing, you must avoid updating the bar from worker processes. With asyncio, you must avoid blocking the event loop with synchronous tqdm writes. The tqdm.asyncio module handles that by using async-compatible methods.

For a thread-based approach that mixes with asyncio or blocking I/O, tqdm also provides tqdm.contrib.concurrent.thread_map. It works like process_map but uses a thread pool, which is lighter than processes and suitable for I/O-bound tasks.

Controlling Refresh Rate and Overhead

tqdm writes to stderr and refreshes at a rate controlled by mininterval (default 0.1 seconds). In concurrent workloads, frequent updates from many tasks can cause excessive output and slow down the program. Set mininterval higher, for example 0.5 or 1.0, to reduce writes:

with tqdm(total=len(data), mininterval=0.5) as pbar:
    for result in pool.imap_unordered(worker, data):
        pbar.update()

The overhead of tqdm itself is small, but it can become noticeable if you update the bar millions of times. The mininterval limit ensures that tqdm does not write on every update; it only writes when the interval has elapsed. This is especially important when using imap_unordered with a large dataset.

Common Pitfalls and How to Avoid Them

One frequent mistake is calling tqdm.update() inside a worker function. As mentioned, this does not affect the parent process's bar. Always update from the main process.

Another issue is forgetting the if __name__ == "__main__": guard when using multiprocessing on Windows or macOS with spawn. Without it, the pool creation can recurse and cause errors.

When using process_map, note that it uses concurrent.futures.ProcessPoolExecutor. The worker function must be picklable, so define it at the module level, not inside a closure or a local function.

With asyncio, a common error is mixing synchronous tqdm with async code. The standard tqdm object's update method is not awaitable and can block the event loop if called frequently. Use tqdm.asyncio.tqdm to get an async-compatible version.

Finally, if you see overlapping progress bars or garbled output, it usually means multiple processes are writing to the same terminal. The patterns in this article avoid that by keeping all tqdm output in the main process.