python requests proxy ssl verification and user agent: Pract

When you need to route Python requests through a proxy, verify the server's SSL certificate, and send a custom User-Agent header, the requests library gives you direct control over each part. The challenge is combining them correctly, especially when the proxy introduces certificate errors or authentication requirements. This article walks through the exact syntax and the decisions you need to make when using python requests proxy ssl verification and user agent together.

Configuring a Proxy in Python Requests

The proxies parameter accepts a dictionary mapping protocols to proxy URLs. For HTTP and HTTPS requests, you typically provide both entries:

import requests

proxies = {
    "http": "http://proxy.example.com:8080",
    "https": "http://proxy.example.com:8080",
}

response = requests.get("https://httpbin.org/ip", proxies=proxies)

If the proxy requires authentication, embed the credentials in the URL:

proxies = {
    "http": "http://user:password@proxy.example.com:8080",
    "https": "http://user:password@proxy.example.com:8080",
}

Alternatively, you can set the HTTP_PROXY and HTTPS_PROXY environment variables, and requests will pick them up automatically. This is convenient for scripts that run in different environments without hardcoding proxy details.

Handling SSL Verification with Proxies

By default, requests verifies the SSL certificate of the target server. When you go through a proxy, the proxy may terminate TLS or present its own certificate, causing a requests.exceptions.SSLError. You have three options:

Disable verification with verify=False – use only when you understand the risks.
Point to a custom CA bundle with verify=/path/to/ca-bundle.crt.
Use a session and set verify once for all requests.

Example with verify=False:

response = requests.get("https://example.com", proxies=proxies, verify=False)

This suppresses the certificate check entirely. The requests library will emit an InsecureRequestWarning to remind you that the connection is not verified. To silence that warning, you can use urllib3.disable_warnings() but that does not change the security implication.

For a custom CA bundle, provide the path:

response = requests.get("https://example.com", proxies=proxies, verify="/etc/ssl/certs/ca-certificates.crt")

If your proxy uses a self-signed certificate for the target host, you can extract that certificate and add it to your CA bundle. This keeps verification enabled while trusting the specific proxy.

Setting a Custom User-Agent Header

The User-Agent header identifies the client to the server. Some APIs and websites reject requests with the default python-requests user agent. Set it per request or on a session:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
}

response = requests.get("https://httpbin.org/headers", headers=headers, proxies=proxies)

When you need consistent headers across multiple requests, create a requests.Session and set the headers once:

session = requests.Session()
session.headers.update({"User-Agent": "MyCustomUA/1.0"})
session.proxies.update(proxies)
session.verify = False  # or a CA bundle path

Now every request made with that session carries the custom User-Agent and uses the proxy settings.

Combining Proxy, SSL, and User-Agent in a Session

A session is the cleanest way to manage all three settings together. It avoids repeating the same parameters on every call and allows you to switch between configurations without changing each request:

import requests

session = requests.Session()
session.proxies = {
    "http": "http://user:pass@proxy.example.com:8080",
    "https": "http://user:pass@proxy.example.com:8080",
}
session.headers.update({"User-Agent": "MyApp/1.0 (contact@example.com)"})
session.verify = "/path/to/custom-ca.pem"  # or False, but see security notes

response = session.get("https://api.example.com/data")

This session now routes through the proxy, verifies the server certificate against your custom CA, and sends the custom User-Agent. You can override any of these per request if needed, for example session.get(url, verify=False) for a specific endpoint.

Common SSL Errors When Using a Proxy

When you see SSLError: [SSL: CERTIFICATE_VERIFY_FAILED], the proxy is likely presenting a certificate that does not match the target hostname or is not trusted by your system. This often happens with corporate proxies that perform TLS interception. The error message includes the hostname and the certificate details, which can help you decide whether to trust that certificate.

Another common error is ProxyError: Unable to connect to proxy, which usually means the proxy address is wrong or the proxy is down. This is separate from SSL verification and should be debugged by checking connectivity first.

If you use verify=False, the SSL error disappears, but you lose the ability to detect man-in-the-middle attacks. Only do this in a controlled environment or when the data is not sensitive.

Security Considerations for Disabling SSL Verification

Disabling SSL verification with verify=False opens a security hole. An attacker on the network path can impersonate the server and read or modify the data. This is especially dangerous when sending authentication tokens or personal data. Before you disable verification, ask whether the proxy is trusted and whether the connection carries sensitive information.

A safer alternative is to extract the proxy's certificate and add it to a custom CA bundle. That way, you still verify the certificate chain, but you explicitly trust the proxy's certificate. This requires updating the bundle when the proxy certificate changes, but it preserves the integrity of the TLS connection.

Using Environment Variables for Proxy Configuration

Instead of hardcoding proxy settings in code, you can rely on environment variables. The requests library honors HTTP_PROXY and HTTPS_PROXY (case-insensitive). This is useful for deploying the same script across environments where the proxy differs:

export HTTP_PROXY="http://user:pass@proxy.example.com:8080"
export HTTPS_PROXY="http://user:pass@proxy.example.com:8080"

Then in your Python code, you don't need to pass proxies at all. However, you still need to handle SSL verification and User-Agent explicitly. You can combine environment proxies with a session that sets verify and headers. This separation keeps the code portable while retaining control over the security-sensitive parts.

Remember that environment variables are read at request time, so changing them in the shell after the script starts will not affect an already running process. For dynamic proxy switching, prefer the proxies parameter or session attribute.