Python Multiprocessing: Process and Pool Explained

Python multiprocessing lets you run multiple processes to execute CPU-bound work in parallel, bypassing the Global Interpreter Lock (GIL) that limits threads. This article explains the core APIs—Process, Pool, and synchronization primitives—and shows how to share data safely between processes.

Why Multiprocessing Instead of Threads

Python threads are limited by the GIL, a mutex that prevents multiple threads from executing Python bytecode simultaneously. For CPU-bound tasks, threads often provide no speedup because only one thread runs at a time. Multiprocessing sidesteps the GIL by spawning separate interpreter processes, each with its own memory space and GIL. This allows true parallel execution on multi-core hardware.

For I/O-bound tasks, threads or asyncio are usually more efficient because they spend time waiting on external resources. Multiprocessing is the right tool when the bottleneck is the CPU.

Creating and Starting Processes with Process

The Process class is the lowest-level way to start a new process. You instantiate it with a target function and arguments, then call start() and join().

import multiprocessing

def worker(n):
    print(f"Working on {n}")
    return n * 2

if __name__ == "__main__":
    processes = []
    for i in range(4):
        p = multiprocessing.Process(target=worker, args=(i,))
        processes.append(p)
        p.start()
    for p in processes:
        p.join()

Each process runs independently, and join() blocks until the process terminates. The if __name__ == "__main__" guard is required on Windows and recommended elsewhere to prevent infinite process spawning when the module is imported.

Using Pool for Parallel Mapping

For tasks that can be expressed as a function applied to many inputs, Pool provides a higher-level interface. Pool.map distributes the inputs across worker processes and collects results in order.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(square, range(10))
    print(results)

The with block ensures the pool is closed and joined properly. Pool.map blocks until all results are ready. For asynchronous execution, apply_async returns a result object that you can wait on later.

Sharing Data Between Processes

Processes do not share memory by default. To exchange data, you can use Queue, Pipe, or shared memory objects like Value and Array. Queues are thread- and process-safe and are the simplest way to pass messages.

from multiprocessing import Process, Queue

def producer(q):
    q.put("data")

def consumer(q):
    print(q.get())

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    p1.start(); p2.start()
    p1.join(); p2.join()

Shared memory (Value and Array) is faster but requires careful synchronization because multiple processes can read and write the same memory region concurrently.

Synchronizing Concurrent Access

When multiple processes access shared resources, you need locks to prevent race conditions. multiprocessing.Lock provides mutual exclusion.

from multiprocessing import Process, Lock, Value

def increment(lock, counter):
    for _ in range(1000):
        with lock:
            counter.value += 1

if __name__ == "__main__":
    lock = Lock()
    counter = Value("i", 0)
    processes = [Process(target=increment, args=(lock, counter)) for _ in range(4)]
    for p in processes: p.start()
    for p in processes: p.join()
    print(counter.value)

Without the lock, the increments would interleave unpredictably. Event and Semaphore are also available for more complex coordination.

Common Pitfalls and How to Avoid Them

Multiprocessing introduces several pitfalls that can break your code in production.

Pickling Limitations

Arguments and return values must be picklable. Lambdas, local functions, and some objects (like open sockets) cannot be pickled. Define target functions at module level and avoid passing unpicklable state.

Fork vs Spawn Start Methods

On Linux and macOS, the default start method is fork, which copies the parent memory. On Windows, it is spawn, which re-imports the module. Code that relies on global state initialized in the parent may behave differently across platforms. Use multiprocessing.get_context() to choose a consistent method.

Global State Isolation

Each process has its own copy of global variables. Changes in one process do not affect others. This is often surprising but is a fundamental design choice. If you need shared state, use explicit shared memory or a manager.

Performance Considerations and When to Use Multiprocessing

Multiprocessing adds overhead: process creation, inter-process communication, and memory duplication for fork. For small tasks, the overhead can exceed the speed benefit. Use multiprocessing when the work is CPU-intensive and the input size is large enough to justify the cost.

A rough rule: if each task takes more than a few milliseconds and you have multiple cores, multiprocessing can help. For tasks that are very short or I/O-bound, threading or asyncio is often a better fit.

Also consider the memory footprint. Each process has its own Python interpreter and memory space,, which can be significant for large data structures. Shared memory or Array can reduce duplication, but they add complexity.