Python DiskCache for Multiprocessing Caching

The Problem: In-Memory Caches Don't Survive Process Boundaries

When you spawn multiple Python processes—via multiprocessing, concurrent.futures.ProcessPoolExecutor, or a task queue—each process gets its own memory space. A simple dict or an in-memory cache like functools.lru_cache lives only inside one process. If you need to share cached results across processes, you need a cache that lives outside the process memory. python diskcache multiprocessing cache solves this by storing data on disk using SQLite, with built-in file locking to make it safe for concurrent access.

What DiskCache Provides for Multiprocessing

DiskCache is a Python library that implements a persistent, process-safe cache. It uses SQLite as the backend, which gives it transactional semantics and file-based locking. The core Cache class is designed to be thread-safe and process-safe out of the box. That means you can create a Cache instance in the parent process, pass it to child processes, and they can all read and write to the same underlying database without corrupting it.

The library also offers a FanoutCache variant that shards the data across multiple SQLite files to reduce write contention when you have many concurrent writers. Understanding when to use each is key to getting good performance.

Setting Up a Shared Cache for Multiprocessing

The simplest approach is to create a single Cache object and share it across processes. Because the cache is backed by a file, all processes refer to the same disk location. Here's a minimal example:

from diskcache import Cache
from multiprocessing import Process

cache = Cache('/tmp/shared_cache')

def worker(key, value):
    cache.set(key, value)
    result = cache.get(key)
    print(f"Process {key}: {result}")

if __name__ == '__main__':
    processes = [Process(target=worker, args=(i, i*2)) for i in range(4)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

Each process calls cache.set and cache.get. DiskCache handles the locking internally, so you don't need to add your own locks. The cache directory is created if it doesn't exist. The Cache object is picklable, so it can be passed as an argument to a process.

When to Use FanoutCache for Higher Concurrency

The default Cache uses a single SQLite database. SQLite allows one writer at a time, but reads can be concurrent. If your workload is read-heavy, the default cache is fine. If you have many processes writing frequently, the single-writer lock can become a bottleneck. FanoutCache splits the data into multiple shards, each with its own SQLite file. This allows concurrent writes to different shards.

from diskcache import FanoutCache

cache = FanoutCache('/tmp/fanout_cache', shards=8, timeout=1)

The shards parameter controls how many separate databases are used. The cache uses a hash of the key to decide which shard to use. This reduces lock contention but adds a small overhead for the hash computation and more file handles. Choose FanoutCache when you expect many concurrent writers and you measure that the single-cache lock is limiting throughput.

Concurrency and Locking Behavior

DiskCache uses file locks to coordinate access. On POSIX systems it uses fcntl locks; on Windows it uses msvcrt. The locking is per-operation: each set, get, delete acquires a lock for the duration of the SQLite transaction. This means you don't need to wrap operations in your own multiprocessing.Lock. However, you must be aware that a long-running transaction—like iterating over a large result set—can block other processes. The timeout parameter controls how long a process waits for a lock before raising a TimeoutError. Set it appropriately for your workload.

Another subtlety: if you use the same Cache object in multiple threads within the same process, DiskCache is also thread-safe. But if you pass the same object to multiple processes, each process gets its own file descriptor and lock state. That's fine because the locking is based on the file system, not on the object identity.

Performance Considerations: Disk I/O and Serialization

DiskCache stores Python objects using pickle by default. This means every read and write involves serialization and deserialization, plus disk I/O. For small objects, the overhead is usually acceptable. For large objects, it can dominate. If you need to cache large binary data, consider storing it as a separate file and caching the file path instead. Also, be aware that the SQLite database grows as you add entries; you should call cache.cull() or cache.expire() periodically to remove old entries, or set an expire time when setting values.

The diskcache library is not a replacement for an in-memory cache when you need sub-millisecond access. It is best suited for data that is expensive to compute or fetch, and where the cost of disk I/O is still lower than recomputing. If you need extremely low latency, consider using a shared memory cache like multiprocessing.shared_memory or a separate Redis server.

Common Pitfalls and How to Avoid Them

One common mistake is creating a new Cache instance inside each child process with the same directory. That works, but it opens a new connection to the SQLite database. It's fine, but you should ensure that the directory is accessible and that you don't accidentally use a different directory per process. Another pitfall is forgetting to close the cache when the process exits. Use the context manager or call cache.close() to flush and release locks.

Serialization issues can also arise if you cache objects that are not picklable, such as a lock or a socket. Stick to plain data types or use pickle-compatible objects. If you need to cache a custom class, ensure it is defined at the module level so it can be pickled.

Finally, be careful with timeout values. If a process crashes while holding a lock, the lock is released by the OS, but a very short timeout might cause spurious TimeoutErrors under heavy load. Set a timeout that is generous enough for your slowest operation.

Choosing Between DiskCache and Other Multiprocessing Cache Options

DiskCache is a good choice when you need a persistent cache that survives process restarts, and when you want to avoid running a separate service like Redis. It is also useful when you want a simple, drop-in cache that works across processes without extra configuration. If you need a cache that is shared across multiple machines, you'll need a network-based solution like Redis or Memcached. If you need extremely high throughput and low latency, an in-memory shared memory approach might be better.

The decision comes down to: do you need persistence? Do you need cross-process sharing on a single machine? Do you want minimal operational overhead? If yes to all, DiskCache is a strong candidate. For a single machine with multiple processes, it provides a robust, file-based cache with built-in locking, and the FanoutCache variant gives you a way to scale write concurrency when needed.