Python Requests Session Cookies and Connection Pooling

When you call requests.get() repeatedly, each call opens a new TCP connection and does not carry cookies between calls. You end up manually passing headers or cookie dictionaries, and the overhead of re-establishing connections adds up. The requests.Session object solves both problems by persisting cookies and reusing connections. This article explains how python requests session cookies and connection pooling work together, how to configure the pool, and where the common failure points are.

What a requests.Session Does

A Session in the requests library is a higher-level wrapper around urllib3 that maintains state across requests. It holds a cookie jar, a set of default headers, and a connection pool. When you use the same Session for multiple requests, it automatically sends stored cookies and reuses the underlying TCP connections to the same host.

The connection pool is managed by urllib3, which requests uses as its HTTP engine. Each Session creates its own PoolManager instance, so connections are scoped to that session. This means two different Session objects do not share connections or cookies, even if they target the same host.

Persisting Cookies with a Session

Consider a typical login flow. Without a session, you would need to extract the cookie from the login response and manually attach it to every subsequent request:

import requests

login_url = "https://example.com/login"
login_data = {"username": "alice", "password": "secret"}

response = requests.post(login_url, data=login_data)
# Manually extract the cookie
cookie = response.cookies.get("sessionid")

# Now send it with every request
headers = {"Cookie": f"sessionid={cookie}"}
requests.get("https://example.com/profile", headers=headers)

This is error-prone and does not handle cookie expiry or multiple cookies. A Session handles all of this automatically:

import requests

session = requests.Session()
login_url = "https://example.com/login"
login_data = {"username": "alice", "password": "secret"}

session.post(login_url, data=login_data)
# The session now holds the cookie
profile = session.get("https://example.com/profile")

The cookie jar is updated with each response, and cookies are sent back to the same domain on subsequent requests. This is the primary reason to use a session for any multi-step interaction that requires authentication.

How Connection Pooling Works Under the Hood

When you make a request without a session, requests creates a new PoolManager and a new connection for that request. The connection is closed after the response is read. With a Session, the PoolManager is reused, and connections are kept alive according to the HTTP Keep-Alive mechanism.

urllib3 maintains a pool of connections for each host. When a request is made, it checks out an available connection from the pool. If no connection is free, it opens a new one, up to the configured maximum. When the request finishes, the connection is returned to the pool instead of being closed, so it can be reused for the next request to the same host.

This behavior reduces the overhead of TCP handshakes and TLS negotiations. For a script that makes many requests to the same API, using a session can significantly cut down on latency and network resource usage.

Configuring the Connection Pool Size

The default pool settings in requests are defined by HTTPAdapter. You can control the number of connections per host and the total pool size by mounting a custom adapter on your session.

import requests
from requests.adapters import HTTPAdapter

session = requests.Session()
adapter = HTTPAdapter(pool_connections=10, pool_maxsize=10)
session.mount("https://", adapter)
session.mount("http://", adapter)

• pool_connections is the number of connection pools to cache for different hosts.
• pool_maxsize is the maximum number of connections to keep in a single pool for one host.

If your application makes many concurrent requests to the same host, you may need to increase pool_maxsize. If you access many different hosts, increase pool_connections. The defaults are usually sufficient for simple scripts, but for a high-throughput service you should tune these values based on your concurrency model.

Note that pool_maxsize is a per-host limit. If you have a pool for api.example.com and another for auth.example.com, each can have up to pool_maxsize connections.

Session Reuse and Thread Safety

A Session is not thread-safe. If you share one session across multiple threads, you can run into race conditions where cookies are overwritten or connections are used simultaneously in an unsafe way. The requests documentation explicitly warns against this.

For concurrent workloads, you have two options:

• Create a separate Session for each thread. This is the simplest approach and avoids any shared state.
• Use a lock to serialize access to a single session, but this defeats the purpose of concurrency if the session is the bottleneck.

If you need to share cookies across threads, consider a thread-safe cookie store or re-authenticate per thread. In practice, a per-thread session is the most predictable pattern.

Closing Sessions and Resource Management

A Session holds open connections until they are closed or garbage-collected. To ensure connections are released promptly, use the session as a context manager or call close() explicitly.

import requests

with requests.Session() as session:
    session.get("https://example.com")
    # session is closed automatically

If you create a session without a context manager, call session.close() when you are done. Failing to do so can leave sockets open, especially in long-running applications. The close() method closes all connections in the pool and clears the cookie jar.

Common Pitfalls with Sessions

One common mistake is mixing session and non-session requests. If you use requests.get() for one call and session.get() for another, the non-session call will not see the cookies stored in the session. Always use the session object for every request that should share state.

Another pitfall is assuming that a session is thread-safe. As mentioned, sharing a session across threads can lead to corrupted cookies or connection errors. If you see intermittent ConnectionError or CookieConflict exceptions, this is often the cause.

Finally, be careful with the Session's default headers. If you modify session.headers, those changes persist across all requests. This is usually what you want, but it can surprise you if you accidentally set a header for one request and forget to remove it.

When a Session Is Not the Right Choice

A session is not always necessary. For a single request that does not need cookies or connection reuse, requests.get() is simpler and avoids the overhead of maintaining a session. If your script makes only a handful of requests to different hosts, the connection pool provides little benefit.

Sessions also introduce statefulness. If you are writing a stateless service or a function that should be isolated, creating a new session per call may be cleaner, even if it means losing connection reuse. In such cases, weigh the convenience of automatic cookies against the complexity of managing session lifecycle.

For long-running applications that make many requests to the same API, a session is almost always the right choice. It reduces latency, simplifies cookie handling, and gives you fine-grained control over the connection pool. Just remember to close it when you are done and to avoid sharing it across threads without proper synchronization.