Streaming Large File Downloads with Python Requests

When you call requests.get(url) without any special parameters, the entire response body is read into memory before the function returns. For a small API response that is fine, but for a file that is several gigabytes, that single line can exhaust available RAM and crash the process. The requests library provides a streaming mode that lets you read the response incrementally, which is the standard approach for python requests file download streaming large files.

Why Streaming Matters for Large File Downloads

The default behavior of requests.get() is to download the entire response body and buffer it in memory. The Response object's content attribute holds the full payload as bytes. For a 2 GB file, that means 2 GB of RAM is consumed just to hold the file, plus the overhead of the response object and any other data in your application. This approach is not viable for large files because memory usage grows linearly with file size.

Streaming changes the game. With stream=True, the connection is kept open and the response body is not immediately read. Instead, you read it in chunks as you iterate over the response. This allows you to write each chunk to disk or process it without ever holding the entire file in memory.

Using stream=True to Avoid Loading the Whole Response

The requests library exposes streaming through the stream parameter of get(). Setting it to True changes how the response is handled internally. Instead of reading the entire body, requests leaves the underlying socket open and lets you read from it incrementally.

import requests

url = "https://example.com/large-file.zip"
response = requests.get(url, stream=True)

At this point, the headers have been received, but the body has not been read. The Response object is ready for you to consume its content. If you access response.content or response.text now, you will force the entire body to be read into memory, defeating the purpose of streaming. Instead, you should use the iter_content() method or iterate over the response directly.

Writing the Response Content to Disk in Chunks

The recommended way to stream a file to disk is to use response.iter_content(). This method yields chunks of bytes as they arrive from the network. You control the chunk size with the chunk_size argument. A common choice is 8192 bytes (8 KB), but you can adjust it based on your network and storage characteristics.

import requests

url = "https://example.com/large-file.zip"
response = requests.get(url, stream=True)
response.raise_for_status()  # Check for HTTP errors

with open("large-file.zip", "wb") as file:
    for chunk in response.iter_content(chunk_size=8192):
        if chunk:  # filter out keep-alive chunks
            file.write(chunk)

The if chunk check is important because iter_content() may yield empty chunks to keep the connection alive. Writing an empty bytes object to a file is harmless, but it adds unnecessary overhead. Filtering them out keeps the loop clean.

You can also use response.raw to read the raw stream, but iter_content() handles decompression and decoding for you, so it is usually the safer choice.

Handling Compression and Content-Length

When you request a file, the server may send it compressed, typically with gzip or deflate. By default, requests sends an Accept-Encoding header that includes these encodings and automatically decompresses the response when you read it. This is transparent when using iter_content() because it yields decompressed bytes. However, the Content-Length header reflects the compressed size, so you cannot rely on it to estimate the final file size. If you need to know the uncompressed size, you may need to check the Content-Encoding header and handle it manually, or disable automatic decompression by setting decode_content=False on the raw stream.

For most downloads, the automatic decompression is what you want. The memory savings from streaming are not affected by compression because you still process the data in chunks.

Managing Memory and Performance During Streaming

Streaming reduces memory usage, but it does not eliminate all memory concerns. The chunk size directly affects how much data is held at any moment. A larger chunk size means fewer iterations but more memory per chunk. A smaller chunk size uses less memory but increases the number of write operations to disk, which can slow down the download.

There is no universal best chunk size. For a typical file download, 8 KB to 64 KB works well. If you are writing to a slow disk, a larger chunk size may improve throughput because it reduces the number of system calls. If memory is extremely constrained, use a smaller chunk size.

Another consideration is the response object itself. Even with streaming, the response headers and metadata are stored in memory. That is negligible for most files, but if you are downloading thousands of files in a loop, you should close the response with response.close() when you are done to release the connection.

Handling Errors and Partial Downloads

Network failures can interrupt a download at any point. If the connection drops while you are iterating, requests will raise an exception, and the file you have written so far will be incomplete. You should catch these exceptions and decide whether to retry, delete the partial file, or attempt to resume.

import requests

url = "https://example.com/large-file.zip"
try:
    response = requests.get(url, stream=True)
    response.raise_for_status()
    with open("large-file.zip", "wb") as file:
        for chunk in response.iter_content(chunk_size=8192):
            if chunk:
                file.write(chunk)
except requests.exceptions.RequestException as e:
    print(f"Download failed: {e}")
    # Optionally remove the partial file

The raise_for_status() call is important because it raises an exception for 4xx and 5xx responses. Without it, you might start writing an error page to disk.

Resuming Interrupted Downloads with Range Requests

If the server supports HTTP range requests, you can resume a download from where it stopped. The requests library allows you to send a Range header with the headers parameter. You need to track how many bytes you have already written and set the Range header accordingly.

import os
import requests

url = "https://example.com/large-file.zip"
local_path = "large-file.zip"

# Get the size of the existing file, if any
if os.path.exists(local_path):
    resume_byte = os.path.getsize(local_path)
else:
    resume_byte = 0

headers = {"Range": f"bytes={resume_byte}-"} if resume_byte else {}
response = requests.get(url, stream=True, headers=headers)

if response.status_code == 206:  # Partial Content
    mode = "ab"  # append
elif response.status_code == 200:
    mode = "wb"  # start fresh
else:
    response.raise_for_status()

with open(local_path, mode) as file:
    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            file.write(chunk)

This approach works only if the server supports range requests. You can check by looking for the Accept-Ranges header in the initial response. If the server returns a 200 status instead of 206, it ignored the Range header and sent the entire file again, so you must overwrite the existing file.

When Streaming Is Not the Right Choice

Streaming is not always necessary. If the file is small enough to fit comfortably in memory, the added complexity of streaming may not be worth it. A few megabytes is usually fine to load directly with response.content. Streaming also adds a small overhead because you are iterating over chunks and writing to disk incrementally. For a 1 MB file, the difference is negligible, but the code is longer.

Another case where streaming might not be ideal is when you need the entire file in memory for processing, such as when you are parsing a JSON document or an image. In those situations, you have to read the whole body anyway, so streaming to disk and then reading it back is wasteful. Instead, you can use response.content directly, or use response.json() if the response is JSON.

The decision comes down to file size, memory constraints, and what you plan to do with the data. For large files that must be saved to disk, streaming with iter_content() is the standard, memory-safe approach.