Python Zero Copy Buffer: memoryview and Buffer Protocol

When you pass a byte string or bytearray between functions, Python often creates a copy. For large buffers, that copy adds memory pressure and CPU time. A python zero copy buffer approach avoids that duplication by exposing the underlying memory directly through the buffer protocol. This article explains how to use memoryview and related tools to move data without copying.

What Zero Copy Means in Python

Zero copy in Python does not mean the data is never touched by the CPU. It means the data is not duplicated when being transferred between objects, functions, or I/O layers. The Python interpreter uses the buffer protocol to let objects share the same underlying memory region. A memoryview is the standard way to access that shared memory without copying.

When you write data = b"some bytes" and then view = memoryview(data), view points to the same memory that data owns. Reading from view does not copy the bytes. Similarly, slicing a memoryview returns another memoryview that references a subrange of the same memory, not a new byte sequence.

This behavior is especially useful when working with large binary data, network packets, or file chunks. Instead of creating multiple byte strings that each hold a copy of the data, you can pass around lightweight views that describe where the data lives.

The Buffer Protocol and memoryview

The buffer protocol is an internal C-level interface that lets Python objects expose their raw memory. Objects like bytes, bytearray, array.array, and mmap implement it. A memoryview object consumes that protocol and presents a Python-level interface to the underlying buffer.

Here is a minimal example:

import array

arr = array.array('H', [1, 2, 3, 4])
view = memoryview(arr)

# view points to the same memory as arr
print(view[0])  # 1
arr[0] = 99
print(view[0])  # 99

The memoryview reflects changes made to the original object because it is not a copy. This is the core of zero-copy access.

memoryview also exposes format information. The .format attribute tells you how the underlying data is structured, such as 'B' for unsigned bytes or 'H' for unsigned short. This matters when you need to interpret the memory as a particular C type.

Creating Memoryviews Without Copying

You can create a memoryview from any object that supports the buffer protocol. The most common sources are bytes, bytearray, and array.array. The syntax is always memoryview(obj).

buf = bytearray(b"hello world")
view = memoryview(buf)

If the source object is writable, the memoryview is writable by default. If the source is read-only, like bytes, the memoryview is also read-only. You can check with .readonly.

b = b"fixed"
view = memoryview(b)
print(view.readonly)  # True

A memoryview is not the same as a bytes object. Converting a memoryview to bytes with bytes(view) always copies the data because bytes is immutable and must own its memory. If you need to pass a buffer to a function that expects a bytes object, you cannot avoid that copy unless the function explicitly accepts a buffer-like object.

Slicing and Casting Without Copying

Slicing a memoryview returns a new memoryview that references a subrange of the original memory. This is a zero-copy operation.

data = bytearray(b"0123456789")
view = memoryview(data)
sub = view[2:5]  # points to bytes '2', '3', '4'
print(sub.tobytes())  # b'234'

The slice maintains a reference to the original buffer, so the memory stays alive as long as the slice exists. This is useful when you want to pass a portion of a large buffer to a function without copying that portion.

memoryview.cast changes the interpretation of the underlying memory without copying. For example, you can view four bytes as a single 32-bit integer:

import struct

buf = bytearray(struct.pack("I", 0x12345678))
view = memoryview(buf)
int_view = view.cast('I')
print(hex(int_view[0]))  # 0x12345678

Casting requires the underlying memory to be contiguous and the format to be compatible. It does not change the data; it only changes how Python interprets the bytes. This is particularly useful when parsing binary protocols where you need to read integers or floats from a byte stream without creating intermediate objects.

Zero-Copy I/O with sendfile and mmap

Beyond in-memory views, Python offers zero-copy mechanisms for I/O. The socket.sendfile() method uses the operating system's sendfile syscall when available. That syscall transfers data directly from a file descriptor to a socket, bypassing user-space copies.

import socket

# Assume a connected socket and an open file
sock.sendfile(file_obj)

This is a true zero-copy path for sending files over the network. The file data goes from the page cache to the network card without being copied into Python memory.

For reading files, mmap creates a memory-mapped file. The file contents appear as a byte-like object, and Python can access it without an explicit read that copies into a separate buffer.

import mmap

with open("large.bin", "rb") as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
        # m behaves like a bytearray
        first_byte = m[0]

mmap objects support the buffer protocol, so you can wrap them with memoryview to slice and cast without copying. This combination is powerful for processing large files that would otherwise consume too much memory if read entirely.

When Zero Copy Is Not Worth It

Zero-copy techniques are not always faster. Creating a memoryview has overhead, and for small buffers the cost of the view may exceed the cost of a simple copy. The benefit grows with the size of the data and the number of times you would otherwise copy it.

A common mistake is to use memoryview for tiny slices where a direct bytes slice would be simpler and equally fast. For example, parsing a 4-byte header from a small packet does not need a memoryview. A regular slice data[:4] is fine.

Another consideration is that memoryview keeps the entire underlying buffer alive. If you take a small slice of a large buffer, the whole buffer remains in memory as long as the slice exists. This can defeat the purpose of reducing memory usage if you only need a small part and the original buffer is huge.

Zero-copy also adds complexity. Code that passes memoryview objects must respect the buffer's format, contiguity, and read-only status. For a one-off script, the added complexity is often not justified.

Common Pitfalls and Limitations

Several limitations can trip up developers new to zero-copy buffers in Python.

First, memoryview does not support all Python buffer operations. You cannot concatenate two memoryviews with +. You must use bytes(view1) + bytes(view2), which copies. If you need to combine buffers without copying, you have to manage the underlying memory yourself, for example by writing into a preallocated bytearray.

Second, memoryview of non-contiguous data is not always possible. Some objects, like multi-dimensional arrays from NumPy, can have non-contiguous memory layouts. In those cases, memoryview may raise an error or require a contiguous copy. The .contiguous attribute tells you whether the buffer is contiguous.

Third, the buffer protocol is a C-level contract. If you pass a memoryview to a C extension that expects a contiguous buffer, you must ensure the view is contiguous. Calling .tobytes() always produces a contiguous copy, but that defeats zero copy.

Finally, memoryview does not automatically free the underlying memory. The lifetime of the view is tied to the original object. If you keep a view around, the original buffer cannot be garbage-collected. This is usually fine, but it can lead to unexpected memory retention if you are not careful with long-lived views.

Choosing the Right Zero-Copy Strategy

Selecting the right zero-copy approach depends on the data source and the operation you need to perform.

For in-memory data that you need to slice or reinterpret, memoryview is the direct tool. Use it when the buffer is large and you want to avoid multiple copies during processing.

For sending a file over a network, socket.sendfile() is the best choice because it delegates to the OS and avoids moving data into Python at all.

For reading a large file that you need random access to, mmap is ideal. It maps the file into virtual memory and lets you treat it as a byte array.

A combination works well too: wrap an mmap object in a memoryview to get zero-copy slicing and struct-like access without reading the whole file into RAM. This pattern is common in parsers for large binary formats.

Keep in mind that zero copy is about avoiding redundant data movement, not about avoiding all memory access. The CPU still has to read the bytes when you inspect them. The savings come from not duplicating the data multiple times as it flows through your program.