Python NumPy Save and Load: CSV and Binary Arrays

The workflow for python numpy save load csv and binary arrays typically involves np.savetxt, np.loadtxt, np.save, and np.load. Each pair of functions serves a different purpose, and the choice between text and binary formats affects performance, disk usage, and interoperability.

Choosing Between CSV and Binary Formats

NumPy provides two families of file functions: text-based CSV and binary .npy files. CSV files are human-readable, portable across tools like Excel and pandas, and can be inspected in any text editor. Binary files preserve the exact dtype and shape, load faster, and take less disk space. The choice depends on whether interoperability or efficiency matters more.

Saving and Loading CSV with np.savetxt and np.loadtxt

The classic text functions are np.savetxt and np.loadtxt. np.savetxt writes a 2D array as delimited text. np.loadtxt reads it back.

import numpy as np

data = np.array([[1.5, 2.3], [3.1, 4.7]])
np.savetxt("data.csv", data, delimiter=",", fmt="%.2f")

loaded = np.loadtxt("data.csv", delimiter=",")
print(loaded)

The fmt parameter controls the precision and formatting of each value. The default is "%.18e", which can produce unwieldy files. Use a shorter format like "%.4f" when you only need a few decimal places. The delimiter argument is optional; the default is a space, but CSV files typically use a comma.

np.loadtxt infers the dtype from the file content. If the file contains mixed types, you may need to specify dtype explicitly. The function returns a NumPy array, and for 1D files it returns a 1D array, while 2D files return a 2D array.

Handling CSV Limitations: dtype, Missing Values, and Large Data

CSV has no schema, so np.loadtxt must guess the dtype from the data. If a column contains strings, you need to set dtype=object or use converters. Missing values are not natively supported; np.loadtxt will raise an error if a cell is empty. You can work around this by using np.genfromtxt, which handles missing values and fills them with np.nan, but that function is slower.

For large datasets, CSV files are inefficient. Writing text requires converting every number to a string, and reading requires parsing strings back. Binary formats avoid both steps.

Binary Formats: np.save and np.load

The binary .npy format stores the array's shape, dtype, and raw data in a single file. np.save writes the array, np.load reads it.

np.save("data.npy", data)
restored = np.load("data.npy")

The .npy file is not human-readable, but it preserves the exact dtype and shape. There is no precision loss because the raw bytes are stored directly. This makes .npy the default choice for scientific pipelines where the same NumPy code produces and consumes the data.

One caveat: np.load by default loads the array into memory. For very large arrays, you can use mmap_mode="r" to memory-map the file, which reads only the pages that are accessed. This is useful when working with arrays that exceed available RAM.

Multiple Arrays with np.savez and np.savez_compressed

When you need to persist several arrays together, np.savez writes them into a single .npz archive. Each array is stored under a name, and you access it by that name.

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
np.savez("arrays.npz", first=a, second=b)

loaded = np.load("arrays.npz")
print(loaded["first"])

If you omit the names, NumPy assigns arr_0, arr_1, and so on. np.savez_compressed applies gzip compression to the archive, reducing file size at the cost of slower load and save times. Use compressed archives when disk space is scarce and the arrays contain repetitive data.

Performance and Memory Considerations

Binary I/O is generally faster than text I/O because it avoids the string conversion overhead. The file size is also smaller for typical floating-point data. For example, a float64 array with 10 million elements occupies about 80 MB in binary, but a CSV file with 15-digit precision can exceed 150 MB. The exact numbers depend on the data, but the qualitative difference is consistent.

Memory usage matters when loading. np.load reads the entire array into memory unless you use memory mapping. CSV loading also reads the whole file, but it first parses the text, which creates temporary strings. This makes CSV loading more memory-hungry for large files. If you need to stream data, consider using np.memmap or a binary format with memory mapping.

When to Use Each Format

Use CSV when the data must be shared with tools that don't understand .npy, such as spreadsheet applications or external systems that require plain text. Use binary .npy when the data is produced and consumed by Python code, and performance or precision matters. Use .npz for a collection of related arrays that belong together.

If you are unsure, start with .npy for internal storage and convert to CSV only at the boundaries of your pipeline. This keeps the core processing fast and precise while still providing a readable export when needed.