Python Process Queue: Passing Data Between Processes

When two or more processes need to exchange data in Python, the multiprocessing.Queue is the standard tool for the job. A Python process queue lets you put() objects from one process and get() them from another, while the multiprocessing module handles the underlying pipe, locking, and serialization. This article covers how the queue behaves at runtime, where it breaks down, and how to choose between it and the alternatives.

What a multiprocessing Queue Is For

A multiprocessing.Queue is a FIFO buffer designed for inter-process communication. Unlike a queue.Queue, which coordinates threads inside a single process, a multiprocessing.Queue is backed by a pipe and a set of locks so that multiple processes can safely append and consume items.

The queue accepts any picklable Python object. When you call put(), the object is serialized with pickle, written to the pipe, and later deserialized on the get() side. Because the data crosses a pipe, the receiving process gets a copy, not a shared reference. That distinction matters when you are moving large objects: the cost of pickling and copying is paid on every transfer.

Passing a Queue to a Child Process

The queue must be passed as an argument to the child process. A minimal producer-consumer setup looks like this:

from multiprocessing import Process, Queue

def producer(q):
    for i in range(10):
        q.put(i)
    q.put(None)

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            break
        print(item)

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

The None value acts as a sentinel: the consumer knows to stop once it reads it. This is the most common way to shut down a consumer cleanly, because the queue itself has no built-in "no more items" signal.

On Windows, the queue must be passed through the args tuple because child processes are spawned rather than forked. On Linux and macOS with the default fork start method, the queue would be inherited, but passing it explicitly keeps the code portable across all three platforms.

How the Queue Moves Data Between Processes

Internally, each multiprocessing.Queue uses a pipe and a feeder thread. When you call put(), the object is pickled in the calling process and handed to the feeder thread, which writes the bytes to the pipe. The get() side reads from the pipe and unpickles.

This design has a few consequences:

• The queue is thread-safe on the putting side, so multiple threads in one process can call put() without extra locking.
• Objects must be picklable. Lambdas, local functions, and some closures will raise PicklingError.
• The maxsize parameter limits the number of items buffered in memory. When the buffer is full, put() blocks until space is available.
• Items are delivered in FIFO order, but if multiple producers are involved, the relative order between producers is not guaranteed.

The feeder thread also means that items are not immediately written to the pipe. They sit in a buffer until the thread flushes them. That buffering is the source of several documented quirks described below.

Why empty(), full(), and qsize() Lie

The methods empty(), full(), and qsize() are documented as unreliable on POSIX systems. The reason is the feeder thread: after put() returns, the item may still be sitting in the feeder's buffer, not yet written to the pipe. A concurrent empty() call can therefore report True even though an item was just placed in the queue.

Similarly, qsize() returns an approximate count because the underlying pipe may hold data that has not been fully read, and the buffer on the writing side is not visible to the reading process.

The practical rule is: do not use these methods to make control-flow decisions. If you need to know whether a consumer should stop, use a sentinel value or a separate control mechanism such as an Event. If you need an exact count, track it yourself with a shared counter protected by a lock.

Performance: Pickling and Batching

The dominant cost of using a Python process queue is serialization. Every put() pickles the object, and every get() unpickles it. For small objects, the per-call overhead of the pipe and the lock can also become significant.

If you are moving many small items, consider batching them into a list and putting the list as a single item. This reduces the number of pickling operations and pipe writes:

def producer(q):
    batch = [i for i in range(1000)]
    q.put(batch)

The consumer then unpickles one list instead of one thousand individual items. The tradeoff is latency: the consumer receives nothing until the whole batch is ready, and memory usage on the consumer side is higher.

For large objects, the pickling cost scales with object size, and the data is copied at least once into the pipe buffer. If your workload transfers gigabytes of data, a queue may not be the right primitive; a shared memory array or a file-based exchange is often cheaper.

Choosing Between Queue, JoinableQueue, and Pipe

The multiprocessing module offers several communication primitives, and the choice depends on the shape of your data flow.

Use JoinableQueue when the producer needs to know that all items have been consumed. It adds task_done() and join():

from multiprocessing import JoinableQueue

def consumer(q):
    while True:
        item = q.get()
        if item is None:
            q.task_done()
            break
        q.task_done()

q = JoinableQueue()
# after putting all items and one sentinel:
q.join()

Use a Pipe when exactly two processes need to exchange messages and you can manage locking yourself. A pipe is faster because it skips the feeder thread, but it is not thread-safe, so concurrent writers need external synchronization.

Platform Differences: fork vs spawn

The start method changes how the queue is inherited. On Linux and macOS, the default is fork: the child process inherits the parent's memory, including any queues that already exist. On Windows, the default is spawn: a fresh interpreter starts, and the queue must be passed through the process arguments.

Two practical consequences follow. First, always create the queue inside the if __name__ == "__main__" block or in a module-level function, and pass it explicitly to the child. Second, do not create a queue at module import time and expect it to work identically across platforms; the spawn method will re-import the module in the child, which can create a second queue that is not connected to the parent's.

If you rely on fork-specific behavior, such as inheriting a queue that was created before the fork, your code will fail on Windows. Passing the queue as an argument is the portable approach and should be the default.