Python memoryview: Zero-Copy Buffer Access

When you need to work with large binary data in Python, the default behavior of slicing bytes or bytearray objects creates a copy. For many operations that copy is unnecessary and wastes memory. Python memoryview solves this by exposing the buffer protocol, giving you a view into an object's underlying memory without duplicating it.

How memoryview Exposes the Buffer Protocol

A memoryview object is a Python built-in that wraps an object supporting the buffer protocol. The buffer protocol is a low-level interface that lets objects expose their internal memory as a contiguous or non-contiguous block. Built-in types like bytes, bytearray, and array.array support this protocol. When you call memoryview(obj), you get a view that references the same memory as obj, not a copy.

The key implication is that changes made through the memoryview are visible in the original object, and vice versa. This is essential for zero-copy operations. For read-only views, you can create a memoryview from a bytes object, but you cannot modify it. For writable views, you need the underlying object to be mutable, such as a bytearray.

import array

buf = bytearray(b"hello")
view = memoryview(buf)
print(view[0])  # 104
view[0] = 0x48  # 'H'
print(buf)      # bytearray(b'Hello')

Creating and Using memoryview

Creating a memoryview is straightforward: pass an object that supports the buffer protocol to the memoryview() constructor. The resulting object supports indexing, slicing, and iteration, similar to a sequence. You can also convert it back to a bytes-like object using tobytes() when you need a copy.

import array

arr = array.array('i', [1, 2, 3])
view = memoryview(arr)
print(view.format)  # 'i'
print(view.shape)   # (3,)
print(view.strides) # (4,)

The format, shape, and strides attributes describe the memory layout. format is the native type of each element, shape is the number of elements per dimension, and strides is the number of bytes to move to reach the next element. These attributes become especially useful when working with multi-dimensional arrays.

Slicing and Reshaping memoryview

Slicing a memoryview does not copy data; it returns a new memoryview that references a sub-range of the original memory. This is the core of zero-copy slicing. You can slice, re-slice, and even reshape the view using the cast() method, which changes the interpretation of the underlying bytes without moving them.

buf = bytearray(b"abcdefgh")
view = memoryview(buf)
slice_view = view[2:6]
print(slice_view.tobytes())  # b'cdef'
slice_view[0] = 0x43  # 'C'
print(buf)             # bytearray(b'abCdefgh')

cast() allows you to reinterpret the same bytes as a different type. For example, you can take a bytearray and view it as a sequence of 16-bit integers. This is useful in binary parsing and network protocol handling.

buf = bytearray(b"\x01\x00\x02\x00")
view = memoryview(buf).cast('H')
print(view.tolist())  # [1, 2]  (little-endian)

Note that cast() works only if the underlying memory is contiguous and the length is divisible by the new element size. The resulting view shares memory with the original, so writes affect both.

Performance: Zero-Copy Access

Many Python operations that seem cheap actually allocate new objects. For example, bytes_obj[10:20] creates a new bytes object and copies the bytes. With large buffers, this copying overhead becomes significant. memoryview avoids that copy by referencing the same memory region.

Consider a scenario where you parse a large binary file. Without memoryview, you might read the entire file into a bytes object and then slice it into headers and payloads. Each slice copies data. With memoryview, you can create a view over the whole buffer and slice it with zero copies, reducing memory usage and speeding up processing.

The performance gain is most noticeable when you perform many slices on a large buffer. The exact improvement depends on the size of the data and the number of slices, but the mechanism is clear: memoryview eliminates the per-slice allocation and copy that regular bytes slicing performs.

For a single small slice, the overhead of creating a memoryview might not be worth it. But when you need to pass multiple sub-ranges to functions or process a buffer incrementally, memoryview's zero-copy behavior pays off.

memoryview vs bytes and bytearray

Choosing between memoryview and the built-in bytes/bytearray types depends on whether you need mutability and whether you can afford copies.

Use bytes when you have read-only data and don't need to avoid copies. Use bytearray when you need a mutable buffer and the slicing copy cost is acceptable. Use memoryview when you need to slice large buffers repeatedly or when you want to avoid duplicating memory, especially when integrating with C extensions or libraries that use the buffer protocol.

A memoryview can be created from any object that supports the buffer protocol, including array.array, numpy arrays (via numpy.ndarray), and mmap objects. This makes it a bridge between Python's high-level types and low-level memory access.

Limitations and Pitfalls

Memoryview is not a silver bullet. It has several limitations you should know.

First, a memoryview holds a reference to the underlying object, so the original object cannot be garbage-collected while the view exists. This can increase memory retention if you keep a view around longer than needed. Release the view with release() or delete it to free the reference.

view = memoryview(bytearray(b"data"))
view.release()
# Now the underlying buffer can be freed if no other references exist

Second, not all objects support the buffer protocol. Custom classes need to implement the protocol via __buffer__ (Python 3.12+) or the C-level buffer interface. If you try to create a memoryview from a list, you get a TypeError.

Third, memoryview does not support all sequence operations. For example, you cannot concatenate two memoryviews with +. You need to use tobytes() or copy the data into a bytearray.

Fourth, when using cast(), the byte order (endianness) is system-dependent. On little-endian machines, the example above works as shown; on big-endian, the order changes. If you need a specific byte order, use the struct module or convert explicitly.

Practical Example: Parsing a Binary File

Let's put memoryview to work in a realistic scenario: parsing a simple binary file format that contains a header followed by a sequence of 32-bit integers. The file structure is: first 4 bytes are a magic number, next 4 bytes are the count of integers, then the integers themselves.

import struct

def parse_binary(data: bytes):
    view = memoryview(data)
    magic, count = struct.unpack('>II', view[:8])
    if magic != 0xDEADBEEF:
        raise ValueError("Invalid magic number")
    ints_view = view[8:8 + count * 4]
    ints = struct.unpack(f'>{count}I', ints_view)
    return ints

# Simulate a file read
raw = b"\xde\xad\xbe\xef\x00\x00\x00\x02" + struct.pack('>II', 100, 200)
result = parse_binary(raw)
print(result)  # (100, 200)

Here, view[:8] and view[8:8+count*4] are zero-copy slices. They reference the original data bytes without allocating new bytes objects for the slices. The struct.unpack function reads directly from the memoryview, avoiding an extra copy. If you were processing a large file with many records, this approach would avoid creating a new bytes object for each record's slice.

For a more memory-efficient version, you could read the file into a bytearray and create a memoryview over it, then process records incrementally. This is especially useful when the file is too large to fit into memory as multiple copies.

Memoryview's zero-copy slicing and buffer protocol access make it a valuable tool for performance-sensitive binary processing. By understanding when and how to use it, you can reduce memory overhead and avoid unnecessary copies in your Python code.