Python msgpack: Serialize and Deserialize Python Objects

python msgpack serialize deserialize python objects requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you need to serialize and deserialize Python objects for storage or transmission, pickle is the default choice, but it has two significant drawbacks: it is Python-specific and it is not safe to load untrusted data. The msgpack library offers a compact binary format that is cross-language and faster to parse. In this article, we'll look at how to use Python msgpack to serialize and deserialize Python objects, what works out of the box, and where you need to add custom handling.

Installing msgpack and Basic API

The msgpack package is available on PyPI and installs with pip:

pip install msgpack

The core functions are packb and unpackb for bytes, and pack and unpack for file-like objects. A minimal round-trip looks like this:

import msgpack

data = {"name": "Alice", "age": 30, "tags": ["admin", "dev"]}
packed = msgpack.packb(data)
unpacked = msgpack.unpackb(packed)

print(unpacked)
# {'name': 'Alice', 'age': 30, 'tags': ['admin', 'dev']}

packb converts the Python object into a bytes object. unpackb parses those bytes back into a Python object. The pack and unpack counterparts work with streams, which is useful when writing to a file or socket.

What msgpack Can Serialize Natively

msgpack has a fixed set of types that map directly to its binary format. The following Python types are supported without any extra configuration:

• None → msgpack nil
• bool → true/false
• int → integer (including large integers, up to 64-bit unsigned)
• float → float (64-bit double)
• str → string (UTF-8 encoded)
• bytes → binary
• list → array
• dict → map (keys must be strings, bytes, or integers)

This means that a plain dictionary with string keys and simple values will round-trip cleanly. Tuples, sets, custom objects, and non-string dictionary keys are not directly supported. If you try to pack them, msgpack raises a TypeError unless you provide a custom default function.

Deserialization and Type Recovery

When unpackb parses a msgpack buffer, it returns Python types according to the mapping above. Arrays become lists, maps become dicts, and integers become int. There is no automatic recovery of tuples or custom classes because msgpack does not store type information beyond its built-in types. This is a deliberate design choice: the format is self-describing only at the level of basic types, not Python-specific ones.

For example, if you pack a tuple, it will be serialized as an array and come back as a list:

packed = msgpack.packb((1, 2, 3))
unpacked = msgpack.unpackb(packed)
print(type(unpacked))  # <class 'list'>

If your application relies on tuples or custom classes, you must handle the conversion yourself, either at serialization time or after deserialization.

Serializing Custom Python Objects

The packb function accepts a default callable that is invoked for objects it cannot serialize natively. You can use this to convert custom objects into a dictionary or another supported type. On the deserialization side, unpackb accepts an object_hook callable that can reconstruct the original object from the dictionary.

Consider a simple Point class:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

def encode_point(obj):
    if isinstance(obj, Point):
        return {"__point__": True, "x": obj.x, "y": obj.y}
    raise TypeError(f"Cannot serialize {obj!r}")

def decode_point(obj):
    if "__point__" in obj:
        return Point(obj["x"], obj["y"])
    return obj

point = Point(3, 4)
packed = msgpack.packb(point, default=encode_point)
restored = msgpack.unpackb(packed, object_hook=decode_point)
print(restored.x, restored.y)  # 3 4

The default function must return a msgpack-supported type. The object_hook is called for every dictionary that is decoded, so you need a marker key to identify your custom objects. This pattern works for any class, but it requires you to manage the mapping manually.

For more complex objects, you might prefer to use __getstate__ and __setstate__ or a library like dataclasses with a custom encoder. The key is that msgpack does not do this automatically.

Performance and Memory Characteristics

msgpack is designed to be compact and fast. The binary format uses variable-length integers and short string headers, so small integers and short strings take up very little space. Compared to JSON, msgpack is typically smaller and faster to parse because it avoids text parsing overhead. Compared to pickle, msgpack is often faster and produces smaller payloads, but it lacks Python-specific type preservation.

The exact performance depends on the data shape and the implementation. The msgpack library uses a C extension for packing and unpacking, so it is generally efficient. However, you should not rely on microbenchmarks without measuring your own workload. The main operational benefit is that msgpack data can be read by other languages, which is not true for pickle.

When choosing msgpack, consider that the unpackb function can allocate large objects based on the input. If you are processing untrusted data, you should set limits to prevent memory exhaustion.

Security and Untrusted Data

msgpack is not a security boundary. Loading untrusted msgpack data can cause excessive memory allocation or CPU usage if the payload is crafted to exploit the format. The unpackb function accepts several parameters to mitigate this:

• max_buffer_size limits the size of the input buffer (default is 100 MB).
• strict_map_key (default True) requires map keys to be strings, bytes, or integers; disabling it allows arbitrary objects but can be unsafe.
• use_list (default True) controls whether arrays are decoded as lists or tuples.

For untrusted input, always set max_buffer_size to a reasonable value and keep strict_map_key enabled. Also, be aware that object_hook runs arbitrary Python code you write, so it must not trust the data it receives.

Common Pitfalls and Edge Cases

Non-String Dictionary Keys

msgpack maps only support string, bytes, and integer keys. If you have a dictionary with tuple or custom object keys, you must convert them to a supported type before packing. For example, you can convert tuple keys to strings:

def encode_dict(obj):
    if isinstance(obj, dict):
        return {str(k): v for k, v in obj.items()}
    return obj

Set Serialization

Sets are not natively supported. You can encode a set as a list and decode it back to a set using object_hook or by post-processing. The default function can convert a set to a list, but you lose the type information unless you add a marker.

Large Integers and Unsigned Values

msgpack supports integers up to 64-bit unsigned. If you need larger integers, you must serialize them as strings or bytes. The msgpack library will raise an error if an integer exceeds the format's range.

Tuples vs Lists

As mentioned, tuples become lists after deserialization. If your application relies on tuple immutability, you need to convert lists back to tuples manually. You can do this in object_hook by checking for a marker, or by post-processing the result.

A Complete Example with Dataclasses

A practical pattern is to serialize dataclasses using a generic encoder that converts any dataclass to a dictionary and back. This keeps the code maintainable and avoids repetitive marker logic.

from dataclasses import dataclass, asdict
import msgpack

@dataclass
class User:
    id: int
    name: str
    roles: list

def encode_dataclass(obj):
    if hasattr(obj, "__dataclass_fields__"):
        return {"__dataclass__": obj.__class__.__name__, "data": asdict(obj)}
    raise TypeError(f"Cannot serialize {obj!r}")

def decode_dataclass(obj):
    if "__dataclass__" in obj:
        # In a real app, you would map the class name to the actual class
        if obj["__dataclass__"] == "User":
            return User(**obj["data"])
    return obj

user = User(1, "Alice", ["admin"])
packed = msgpack.packb(user, default=encode_dataclass)
restored = msgpack.unpackb(packed, object_hook=decode_dataclass)
print(restored)

This approach gives you a compact binary representation while keeping your code type-safe. The default function handles any dataclass generically, and the object_hook reconstructs the correct class based on the stored name. In a production system, you would maintain a registry of class names to avoid eval or unsafe lookups.

When to Choose msgpack Over Other Formats

msgpack is a good fit when you need a compact, fast, cross-language serialization format. It is not a drop-in replacement for pickle because it does not preserve Python object identity or arbitrary class instances. Use msgpack when:

• You need to exchange data with services written in other languages.
• Payload size matters (e.g., network bandwidth, storage cost).
• You want to avoid the security risks of pickle.

If you need to serialize complex Python objects with minimal code and do not care about cross-language compatibility, pickle may be simpler. If you need human-readable output, JSON is more appropriate. msgpack sits between them: binary, compact, and language-neutral.