Python aiohttp Streaming File Download

The Problem with Loading a Whole File into Memory

When you download a file with aiohttp, the default behavior loads the entire response body into memory. For large files, that can exhaust memory or cause the process to stall. Streaming the download reads the response in chunks, letting you write each chunk to disk as it arrives.

This article shows how to implement python aiohttp streaming file download patterns that keep memory usage flat and handle large files reliably.

A Minimal aiohttp Streaming Download

The core of streaming a file download with aiohttp is the ClientResponse.content attribute. Instead of calling read() to pull the whole body, you iterate over chunks.

import aiohttp
import asyncio

async def download_file(url: str, dest: str) -> None:
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            resp.raise_for_status()
            with open(dest, "wb") as f:
                async for chunk in resp.content.iter_chunked(1024):
                    f.write(chunk)

asyncio.run(download_file("https://example.com/large-file.zip", "file.zip"))

iter_chunked(1024) yields bytes objects of up to 1024 bytes. The loop writes each chunk to disk immediately, so the memory footprint stays bounded regardless of file size.

Choosing a Chunk Size

The chunk size you pass to iter_chunked affects both memory usage and I/O efficiency. A smaller chunk size (e.g., 256 bytes) uses less memory but makes more write calls. A larger chunk size (e.g., 64 KB) reduces write overhead but increases memory per chunk. The right value depends on your file size and system.

For most downloads, a chunk size between 1 KB and 64 KB works well. If you are downloading many files concurrently, a smaller chunk size can help keep total memory down.

Controlling Timeouts and Connection Limits

Streaming a large file can take a long time. The default aiohttp timeout applies to the whole operation, so you may need to adjust it. Use ClientTimeout to set separate read and connect timeouts.

import aiohttp
import asyncio

async def download_with_timeout(url: str, dest: str) -> None:
    timeout = aiohttp.ClientTimeout(total=3600, connect=10)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(url) as resp:
            resp.raise_for_status()
            with open(dest, "wb") as f:
                async for chunk in resp.content.iter_chunked(64 * 1024):
                    f.write(chunk)

The total parameter sets the maximum time for the entire request, including reading the body. If you expect a very large download, set it high enough or use None to disable the total timeout and rely on the read timeout instead.

Handling Errors and Partial Downloads

Network failures can interrupt a stream. raise_for_status() catches HTTP errors, but connection resets and timeouts raise exceptions. Wrap the download in a try/except block to clean up partial files.

import aiohttp
import asyncio
import os

async def safe_download(url: str, dest: str) -> bool:
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as resp:
                resp.raise_for_status()
                with open(dest, "wb") as f:
                    async for chunk in resp.content.iter_chunked(64 * 1024):
                        f.write(chunk)
        return True
    except (aiohttp.ClientError, asyncio.TimeoutError) as e:
        if os.path.exists(dest):
            os.remove(dest)
        print(f"Download failed: {e}")
        return False

Removing the partial file prevents a corrupted file from being used later. If you want to resume a download, you would need to send a Range header and open the file in append mode, which is more complex.

Streaming Server-Side Responses

aiohttp is not only a client; it also provides a server framework. If you need to serve a file as a stream, use StreamResponse and write chunks to the response. This is useful for large files that should not be loaded into memory.

from aiohttp import web

async def handle_download(request):
    resp = web.StreamResponse()
    resp.content_type = "application/octet-stream"
    resp.headers["Content-Disposition"] = 'attachment; filename="file.zip"'
    await resp.prepare(request)
    with open("file.zip", "rb") as f:
        while chunk := f.read(64 * 1024):
            await resp.write(chunk)
    await resp.write_eof()
    return resp

This server-side pattern keeps memory usage flat on the server as well. The StreamResponse sends headers first, then writes chunks as they are read from disk.

Memory and Performance Considerations

Streaming a download avoids loading the entire file into RAM. The memory used is roughly the chunk size plus the aiohttp internal buffers. This is critical when downloading files that are hundreds of megabytes or gigabytes.

The main performance cost of streaming is the number of write calls. Each chunk triggers a disk write, so a very small chunk size can slow the download. On the other hand, a very large chunk size increases memory usage. The optimal chunk size balances these factors and depends on your storage and network speed.

For concurrent downloads, each stream holds its own chunk buffer. If you download many files at once, use a smaller chunk size to keep total memory under control.

Concurrent Downloads with asyncio

One advantage of aiohttp is the ability to download multiple files concurrently without threads. You can create a list of tasks and run them with asyncio.gather.

import aiohttp
import asyncio

async def download_one(session, url, dest):
    async with session.get(url) as resp:
        resp.raise_for_status()
        with open(dest, "wb") as f:
            async for chunk in resp.content.iter_chunked(32 * 1024):
                f.write(chunk)

async def download_many(urls_and_dests):
    async with aiohttp.ClientSession() as session:
        tasks = [download_one(session, url, dest) for url, dest in urls_and_dests]
        await asyncio.gather(*tasks)

The connection pool in ClientSession limits the number of simultaneous connections. By default, it allows 100 connections. If you start more tasks, they queue until a connection is free. Adjust limit in TCPConnector to control concurrency.

connector = aiohttp.TCPConnector(limit=10)
async with aiohttp.ClientSession(connector=connector) as session:
    ...

This keeps the number of active downloads bounded, which is important when downloading from a server that might throttle or when you want to avoid overwhelming your own network.