Python httpx AsyncClient for Async Requests

When you need to issue many HTTP requests from an asyncio application, httpx.AsyncClient is the component that lets you do it without blocking the event loop. The python httpx asyncclient async requests pattern is a common way to perform concurrent I/O in Python, especially when you have to fetch multiple URLs, call several APIs, or scrape pages in parallel. Unlike the synchronous httpx.Client, the async client is designed to be awaited, so it cooperates with asyncio and other async frameworks.

The main advantage is that while one request is waiting for a response, the event loop can handle other tasks. This makes it possible to run dozens or hundreds of requests concurrently without the overhead of threads or processes.

Basic AsyncClient Usage

The simplest way to use AsyncClient is inside an async with block. This ensures the client is properly closed when you're done, releasing any underlying connections.

import httpx
import asyncio

async def fetch(url: str) -> str:
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        response.raise_for_status()
        return response.text

asyncio.run(fetch("https://example.com"))

The await client.get(url) call suspends the current coroutine until the response is available. The async with context manager calls aclose() automatically, which is important for freeing resources. If you omit the context manager, you must call await client.aclose() explicitly.

Making Concurrent Requests with asyncio.gather

To issue multiple requests at the same time, combine AsyncClient with asyncio.gather. Each request is an awaitable coroutine, and gather schedules them concurrently.

import httpx
import asyncio

async def fetch_status(url: str, client: httpx.AsyncClient) -> int:
    response = await client.get(url)
    return response.status_code

async def fetch_many(urls: list[str]) -> list[int]:
    async with httpx.AsyncClient() as client:
        return await asyncio.gather(*(fetch_status(url, client) for url in urls))

statuses = asyncio.run(fetch_many(["https://example.com", "https://httpbin.org/get"]))
print(statuses)

Notice that the same client instance is passed to each coroutine. This is intentional: AsyncClient is designed to be shared across concurrent tasks because it manages a connection pool internally. Creating a new client for every request would defeat the purpose of connection reuse and add overhead.

Handling Timeouts and Errors

Network requests can fail for many reasons: DNS resolution, connection refused, timeouts, or HTTP error statuses. httpx raises exceptions that you can catch and handle.

import httpx
import asyncio

async def safe_fetch(url: str) -> str | None:
    timeout = httpx.Timeout(10.0, connect=5.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        try:
            response = await client.get(url)
            response.raise_for_status()
            return response.text
        except httpx.TimeoutException:
            print(f"Request to {url} timed out")
        except httpx.HTTPStatusError as exc:
            print(f"HTTP error {exc.response.status_code} for {url}")
        except httpx.RequestError as exc:
            print(f"Request failed: {exc}")
    return None

The Timeout class lets you set separate limits for connect, read, write, and pool timeouts. If you don't specify one, httpx uses a default of 5 seconds for each phase. Catching RequestError covers connection-level failures, while HTTPStatusError is raised only when you call raise_for_status().

Reusing the Client for Connection Pooling

One of the most common mistakes is creating an AsyncClient for each request. That discards the connection pool and forces a new TCP handshake (and possibly TLS handshake) every time. Instead, create a single client and reuse it across the lifetime of your application.

import httpx
import asyncio

async def main():
    async with httpx.AsyncClient() as client:
        for url in ["https://example.com", "https://httpbin.org/get"]:
            response = await client.get(url)
            print(response.status_code)

asyncio.run(main())

For long-running services, you might keep the client as a module-level or app-level instance. Just remember to close it when the application shuts down. The connection pool inside AsyncClient keeps TCP connections alive and reuses them for subsequent requests, which reduces latency and system resource usage.

Streaming Large Responses

When you need to download a large file or process a response incrementally, use the stream method instead of get. This avoids loading the entire response body into memory.

import httpx
import asyncio

async def download(url: str, destination: str):
    async with httpx.AsyncClient() as client:
        async with client.stream("GET", url) as response:
            response.raise_for_status()
            with open(destination, "wb") as f:
                async for chunk in response.aiter_bytes():
                    f.write(chunk)

asyncio.run(download("https://example.com/large-file.bin", "large-file.bin"))

The aiter_bytes() method yields chunks of the response body as they arrive. This is useful for memory-constrained environments or when you want to show progress while downloading. Note that you must use async with on the stream response to ensure the connection is released properly.

Performance and Concurrency Considerations

The main performance benefit of AsyncClient is that it lets you overlap I/O wait times. While one request is waiting for a response, the event loop can process other tasks. This is particularly effective for I/O-bound workloads such as API calls, web scraping, or fetching multiple URLs.

However, async does not make CPU-bound code faster. If you need to do heavy computation on the response data, that work will still block the event loop. In such cases, consider offloading the computation to a thread or process pool, or use asyncio.to_thread for blocking operations.

Also note that httpx.AsyncClient uses anyio as its async backend, which means it works with both asyncio and trio. You should not share a client across different event loops. If you create a client inside one loop and try to use it in another, you'll get errors. Always create and use the client within the same loop, typically by using it inside an async with block in your main coroutine.

Connection pooling is another important factor. Reusing a client keeps TCP connections open, avoiding the overhead of establishing new connections for each request. For many short-lived requests, this can significantly reduce latency. The pool size is configurable via limits, but the default is usually sufficient for most applications.