python httpx file upload download and streaming: Practical U

python httpx file upload download and streaming requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you need to move files over HTTP in Python, httpx offers a modern, synchronous and asynchronous client that handles uploads, downloads, and streaming without the ceremony of older libraries. This article focuses on the practical mechanics of using httpx for file transfer, with attention to memory usage, error handling, and concurrency.

Uploading Files with httpx

The simplest way to upload a file with httpx is to pass a file handle to the files parameter of a request. The library reads the file content into memory and sends it as multipart/form-data.

import httpx

with open("report.pdf", "rb") as f:
    response = httpx.post("https://example.com/upload", files={"file": f})
print(response.status_code)

The files parameter accepts a dictionary where the key is the field name expected by the server. The value can be a file object, a tuple of (filename, file_object), or a tuple of (filename, file_object, content_type). If you need to control the filename, use the tuple form:

files = {"file": ("quarterly-report.pdf", f, "application/pdf")}

For small files this is fine. But when the file is large, reading it entirely into memory can cause spikes in RAM usage. The files parameter always buffers the content, so it is not suitable for multi-gigabyte files. For those, you need streaming.

Downloading Files with httpx

Downloading a file with httpx is equally direct. The get method returns a response object whose .content attribute holds the full body in memory. Write it to disk with a binary write.

import httpx

response = httpx.get("https://example.com/data.zip")
response.raise_for_status()
with open("data.zip", "wb") as f:
    f.write(response.content)

This approach loads the entire file into RAM. For a 2 GB archive, that is a problem. The stream context manager avoids this by exposing the response body as a byte iterator.

with httpx.stream("GET", "https://example.com/data.zip") as response:
    response.raise_for_status()
    with open("data.zip", "wb") as f:
        for chunk in response.iter_bytes():
            f.write(chunk)

iter_bytes() yields chunks of configurable size (default 65536 bytes). This keeps memory usage bounded by the chunk size, not the file size.

Streaming File Uploads

To upload a large file without loading it into memory, use httpx's streaming request body. The content parameter accepts a generator that yields bytes. Combined with the headers parameter to set the content type, this sends the file as a raw stream rather than multipart.

import httpx

def file_chunks(file_path, chunk_size=65536):
    with open(file_path, "rb") as f:
        while True:
            chunk = f.read(chunk_size)
            if not chunk:
                break
            yield chunk

response = httpx.post(
    "https://example.com/upload-raw",
    content=file_chunks("large.iso"),
    headers={"Content-Type": "application/octet-stream"}
)
print(response.status_code)

This is a true streaming upload. The generator is consumed by the underlying HTTP connection, so memory usage stays low. However, the server must accept raw body uploads; many APIs expect multipart form data. For multipart streaming, you need a different approach, such as building the multipart body yourself or using a library like httpcore that supports it. httpx does not provide a built-in streaming multipart upload, so if the server requires multipart/form-data, you must either buffer the file or implement the multipart encoding manually.

Streaming File Downloads

Streaming downloads are more common and better supported. The stream context manager gives you fine-grained control over the response body. Beyond iter_bytes, you can use iter_text for text files, iter_lines for line-delimited data, and iter_raw for the raw undecoded bytes.

with httpx.stream("GET", "https://example.com/large.jsonl") as response:
    for line in response.iter_lines():
        process_line(line)

When you use iter_lines, httpx decodes the bytes according to the response encoding. For binary data, iter_bytes is the right choice. The stream is closed automatically when you exit the context manager, releasing the connection back to the pool.

One caveat: if you read only part of the stream and then exit, the remaining bytes are discarded. This is fine for a one-off request, but if you reuse a client with a connection pool, the connection may be closed instead of reused. To force a clean close, call response.close() explicitly.

Handling Large Files and Memory Usage

The main reason to use streaming is to keep memory usage flat. The files parameter and .content attribute both buffer the entire payload. For files that exceed a few hundred megabytes, that can cause the process to run out of memory or trigger swap. Streaming with generators and iter_bytes keeps the working set small.

There is also a practical limit on the chunk size. A larger chunk reduces the number of iterations but increases memory per chunk. A chunk size of 64 KB is a reasonable default; you can tune it based on your network and disk speed. For extremely high-throughput transfers, you might increase it to 1 MB, but test it in your environment first.

When downloading, you also need to consider disk I/O. Writing each chunk with a separate write call can be slow. Using a buffered writer or accumulating chunks into a larger buffer before writing can improve throughput. For example:

with open("output.bin", "wb") as f:
    for chunk in response.iter_bytes():
        f.write(chunk)

The file object itself is buffered by default, so this is usually sufficient. Avoid calling flush on every iteration.

Error Handling and Retries for File Transfers

Network failures are common during large transfers. httpx raises httpx.RequestError for connection problems and httpx.HTTPStatusError for 4xx/5xx responses when you call raise_for_status(). You should catch these and decide whether to retry.

import httpx
import time

max_retries = 3
for attempt in range(max_retries):
    try:
        with httpx.stream("GET", "https://example.com/bigfile") as response:
            response.raise_for_status()
            with open("bigfile", "wb") as f:
                for chunk in response.iter_bytes():
                    f.write(chunk)
        break
    except httpx.TransportError as exc:
        if attempt == max_retries - 1:
            raise
        time.sleep(2 ** attempt)

Retrying a streaming download is tricky because you cannot resume from the middle of a response. The simplest approach is to restart the request from the beginning. If the server supports range requests, you can implement resumable downloads by sending the Range header and opening the file in append mode.

headers = {"Range": f"bytes={existing_size}-"}
with httpx.stream("GET", url, headers=headers) as response:
    if response.status_code == 206:  # Partial Content
        with open(file, "ab") as f:
            for chunk in response.iter_bytes():
                f.write(chunk)
    else:
        response.raise_for_status()
        with open(file, "wb") as f:
            for chunk in response.iter_bytes():
                f.write(chunk)

This pattern works only if the server supports Accept-Ranges. Not all do, so check the response headers before relying on it.

Concurrency and Performance Considerations

httpx supports asynchronous requests with AsyncClient, which allows you to upload or download multiple files concurrently. For I/O-bound tasks, this can significantly improve throughput.

import asyncio
import httpx

async def download_one(client, url, dest):
    async with client.stream("GET", url) as response:
        response.raise_for_status()
        with open(dest, "wb") as f:
            async for chunk in response.aiter_bytes():
                f.write(chunk)

async def main():
    async with httpx.AsyncClient() as client:
        tasks = [
            download_one(client, f"https://example.com/file{i}.zip", f"file{i}.zip")
            for i in range(5)
        ]
        await asyncio.gather(*tasks)

asyncio.run(main())

When using concurrency, be aware of the connection pool limit. httpx defaults to a maximum of 100 connections per client. If you have more concurrent transfers, you may need to increase that limit or use multiple clients. Also, each stream holds a connection open, so you should not open an unbounded number of streams. Use a semaphore to limit concurrency.

For uploads, streaming a generator with AsyncClient requires an async generator. The same principle applies: yield chunks from an async file read.

async def async_file_chunks(file_path):
    with open(file_path, "rb") as f:
        while True:
            chunk = f.read(65536)
            if not chunk:
                break
            yield chunk

async with httpx.AsyncClient() as client:
    response = await client.post(
        "https://example.com/upload",
        content=async_file_chunks("large.bin"),
        headers={"Content-Type": "application/octet-stream"}
    )

Note that the async generator runs in the same event loop, so it will block the loop during file reads. For truly non-blocking file I/O, you would need a library like aiofiles, but for many cases the blocking read is acceptable because the disk is faster than the network.

Choosing Between Sync and Async for File Transfers

Deciding whether to use the synchronous or asynchronous API depends on your application's concurrency model. If you are building a CLI tool or a script that transfers one file at a time, the synchronous API is simpler and easier to debug. If you are building a web service or a crawler that must handle many files simultaneously, the async API is the right fit.

The streaming mechanisms are equivalent in both APIs: iter_bytes and aiter_bytes, stream and async with client.stream. The error-handling patterns are also similar. The main difference is the event loop. In an async application, you should never call the synchronous httpx methods inside a coroutine, because they block the event loop. Use AsyncClient consistently.

One practical consideration is timeout configuration. Large file transfers can take minutes. The default timeout in httpx is 5 seconds, which will break long transfers. Set a higher timeout or disable it entirely for streaming operations.

client = httpx.Client(timeout=httpx.Timeout(connect=10, read=300, write=300, pool=10))

For streaming, the read timeout applies to each chunk read, not the entire transfer. So if the server sends a chunk every 30 seconds, a read timeout of 60 seconds is safe. If you set it too low, the transfer will abort even though the connection is healthy.