Python Shared Memory: IPC Without Copies

When a Python program spawns multiple processes, each process gets its own memory space. Passing data between them usually means pickling objects and sending them over pipes or queues. For small messages this is fine, but when the data is large—like a NumPy array or a large dictionary—the serialization and copy overhead can dominate execution time. python shared memory allows processes to read and write the same physical memory region, eliminating the need to copy data across process boundaries.

The standard library offers two main paths: the older multiprocessing.Value and multiprocessing.Array wrappers, and the more flexible multiprocessing.shared_memory module introduced in Python 3.8. Both rely on the operating system's shared memory facilities, but they differ in API and use cases.

Comparing Value, Array, and SharedMemory

The multiprocessing module provides Value and Array as high-level wrappers that allocate a shared memory block and expose a synchronized or plain object. They are convenient for simple types like integers, floats, and fixed-size arrays. The shared_memory.SharedMemory class gives you a raw block of bytes that you can map into your own data structures, such as NumPy arrays or custom buffers.

The choice depends on how much control you need and how complex your data is.

Creating and Using SharedMemory

The SharedMemory class creates a new shared memory block or attaches to an existing one. When you create it, you get a name that other processes can use to attach. Here is a minimal example where a parent process creates a block, writes bytes, and a child process reads them.

from multiprocessing import shared_memory, Process

def child(name):
    shm = shared_memory.SharedMemory(name=name)
    # Read the first 10 bytes as a string
    print(bytes(shm.buf[:10]).decode())
    shm.close()

if __name__ == "__main__":
    shm = shared_memory.SharedMemory(create=True, size=100)
    shm.buf[:10] = b"hello world"
    p = Process(target=child, args=(shm.name,))
    p.start()
    p.join()
    shm.close()
    shm.unlink()

The buf attribute is a memoryview that exposes the underlying bytes. You can write to it using slicing or by casting it to a different type. The child process attaches using the name and reads the same memory. After both processes finish, the parent must call unlink() to release the resource.

Using Value and Array for Simple Shared State

For simple scalars and fixed arrays, Value and Array are easier to work with because they handle the memory allocation and provide type conversion. They also accept a lock object for synchronization.

from multiprocessing import Process, Value, Array

def increment(shared_counter):
    for _ in range(1000):
        shared_counter.value += 1

if __name__ == "__main__":
    counter = Value('i', 0)
    processes = [Process(target=increment, args=(counter,)) for _ in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()
    print(counter.value)

Here 'i' indicates a signed integer. The Value object holds the value in shared memory, and each process can read and modify it. Without a lock, the increment operation is not atomic, so the final value may be less than 4000. That leads to the next concern.

Synchronizing Access to Shared Memory

Shared memory is not automatically thread-safe or process-safe. When multiple processes write to the same location, you need to protect critical sections with a lock. The multiprocessing.Lock works across processes. For Value and Array, you can pass a lock as the third argument, or use the default one. For raw SharedMemory, you must create a lock yourself and share it, for example, by passing it to child processes as an argument.

from multiprocessing import Process, Lock, shared_memory

def writer(shm_name, lock):
    shm = shared_memory.SharedMemory(name=shm_name)
    with lock:
        shm.buf[0] = 42
    shm.close()

if __name__ == "__main__":
    shm = shared_memory.SharedMemory(create=True, size=1)
    lock = Lock()
    p = Process(target=writer, args=(shm.name, lock))
    p.start()
    p.join()
    print(shm.buf[0])
    shm.close()
    shm.unlink()

The lock ensures that only one process writes at a time. For more complex structures, you might need multiple locks or a reader-writer pattern.

Lifecycle and Resource Cleanup

A shared memory block persists until it is explicitly unlinked. If you forget to call unlink(), the memory remains allocated even after all processes exit, which can leak system resources. On POSIX systems, the block is tied to a file in /dev/shm; on Windows, it is a named section object. Always call close() when a process is done with the block, and have the creator call unlink() after all processes have finished. In a long-running process, consider using a context manager or a try/finally block to guarantee cleanup.

from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=100)
try:
    # work with shm
    pass
finally:
    shm.close()
    shm.unlink()

If you attach to an existing block, you should only close() it, not unlink() it, because the creator is responsible for removing it.

Choosing the Right Approach

Use Value and Array when you need to share simple, fixed-size data and you want a minimal API. They are perfect for counters, flags, and small numeric arrays. Use SharedMemory when you need to share large buffers, custom binary data, or when you want to map the memory into a NumPy array for efficient numerical work. The raw byte access gives you full control, but you must handle synchronization and lifecycle yourself.

If your data is a NumPy array, you can create a SharedMemory block and use numpy.frombuffer to view it without copying. This pattern is common in high-performance computing and data processing pipelines. Just remember to coordinate access with locks or by partitioning the buffer among processes.