python selenium headless chrome user agent and proxy: Practi

When you run Selenium with headless Chrome, the browser sends a default user agent that identifies it as an automated Chromium instance. Many sites block or throttle that agent. Similarly, if you need to route traffic through a proxy, headless Chrome requires explicit configuration through ChromeOptions. This article shows how to set both a custom user agent and a proxy for python selenium headless chrome user agent and proxy setups, and what to watch out for when doing so.

Setting Up Headless Chrome with Selenium

Before changing user agent or proxy, you need a working headless Chrome instance. The minimal setup uses webdriver.Chrome with Options and the headless argument.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")

driver = webdriver.Chrome(options=options)
```n
This launches Chrome without a visible window. The `--headless` argument is the standard way to enable headless mode in current Chrome versions. Note that some older versions used `--headless=new`; the behavior now is consistent across recent releases, but if you rely on a specific Chrome version, check its documentation.

With this base, you can add arguments for user agent and proxy.

## Changing the User Agent in Headless Chrome

Headless Chrome's default user agent includes the string `HeadlessChrome` instead of `Chrome`. That makes it easy for servers to detect automation. To replace it, pass the `--user-agent` argument.

```python
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")

This sets the user agent for every request made by the browser, including navigation and resource loads. It does not affect the JavaScript navigator.userAgent property, which will also reflect the custom value.

Use a realistic user agent that matches the operating system and browser version you want to emulate. Copying an existing agent string from a real browser is common, but be aware that the string alone does not make your traffic indistinguishable from a real browser. Headless Chrome still has other detectable properties, such as navigator.webdriver being true by default. To suppress that, you can add the --disable-blink-features=AutomationControlled argument.

options.add_argument("--disable-blink-features=AutomationControlled")

This does not change the user agent but removes the most obvious automation flag. Combine it with a custom user agent for a more realistic fingerprint.

Configuring a Proxy for Headless Chrome

Selenium does not have a dedicated proxy method for Chrome; you pass the proxy as a command-line argument. The --proxy-server argument accepts the proxy address and port.

options.add_argument("--proxy-server=http://proxy.example.com:8080")

For an HTTP proxy, use the http:// scheme. For a SOCKS proxy, use socks5://.

options.add_argument("--proxy-server=socks5://proxy.example.com:1080")

When you set this argument, all traffic from the browser, including HTTP and HTTPS requests, goes through the proxy. There is no need to configure separate proxy settings inside the browser profile.

If you need to bypass the proxy for certain hosts, Chrome supports the --proxy-bypass-list argument. For example, to exclude localhost and internal domains:

options.add_argument("--proxy-bypass-list=localhost;127.0.0.1;*.internal.example.com")

This is useful when you want the proxy only for external traffic.

Combining User Agent and Proxy in One Session

Both arguments are independent, so you can add them to the same Options object.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless")
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("--proxy-server=http://proxy.example.com:8080")

driver = webdriver.Chrome(options=options)

This configuration gives you a headless browser that sends a custom user agent and routes all traffic through the specified proxy. The order of arguments does not matter; Chrome applies all of them.

One thing to keep in mind is that the proxy is set at the browser level, not per request. If you need different proxies for different requests within the same session, you would have to create separate driver instances, each with its own proxy.

Handling Proxy Authentication

Many proxies require a username and password. The --proxy-server argument does not accept credentials directly. Instead, you must handle authentication at the browser level or by embedding credentials in the URL.

Embedding credentials in the proxy URL works for HTTP proxies in many cases:

options.add_argument("--proxy-server=http://username:password@proxy.example.com:8080")

However, this approach is not reliable across all Chrome versions and proxy types. Some proxies respond with a 407 status and expect the browser to show an authentication dialog. In headless mode, that dialog cannot appear, and the request fails.

A more robust method is to use a Chrome extension that handles proxy authentication. You create a small extension with a background script that sets the proxy and provides credentials. Then you load that extension with Selenium.

import os
import json
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Create extension directory and manifest
manifest = {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Proxy Auth",
    "permissions": ["proxy", "tabs", "storage"],
    "background": {
        "scripts": ["background.js"]
    }
}

background_js = """
var config = {
    mode: "fixed_servers",
    rules: {
        singleProxy: {
            scheme: "http",
            host: "proxy.example.com",
            port: 8080
        },
        bypassList: ["localhost"]
    }
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "username",
            password: "password"
        }
    };
}
chrome.webRequest.onAuthRequired.addListener(
    callbackFn,
    {urls: ["<all_urls>"]},
    ["blocking"]
);
"""

# Write files to a temporary directory
import tempfile
import pathlib

with tempfile.TemporaryDirectory() as tmp:
    pathlib.Path(tmp, "manifest.json").write_text(json.dumps(manifest))
    pathlib.Path(tmp, "background.js").write_text(background_js)

    options = Options()
    options.add_argument("--headless")
    options.add_argument("--load-extension=" + tmp)

    driver = webdriver.Chrome(options=options)

This extension uses the Chrome proxy API to set a fixed proxy and the webRequest API to respond to authentication challenges. The --load-extension argument loads the extension in headless mode. Note that in newer Chrome versions, headless mode may require --headless=new for extensions to work; test with your specific version.

The extension approach is more reliable for proxies that require authentication, but it adds complexity. For public proxies or proxies without authentication, the simple --proxy-server argument is sufficient.

Common Failure Modes and How to Diagnose Them

When you set a user agent or proxy, you may encounter issues that are not immediately obvious. Here are the most common problems and how to identify them.

Proxy connection refused or timeout

If the proxy address is wrong or the proxy is down, Chrome will fail to load pages. The error appears in the browser console or as a WebDriverException with a message like ERR_PROXY_CONNECTION_FAILED. Check that the proxy host and port are correct and reachable from your network.

Proxy authentication required

If the proxy requires credentials and you did not provide them, you may get a 407 status or a page that never loads. Use the extension method described above to handle authentication.

User agent not applied

If you inspect the request headers and see the default headless user agent, the --user-agent argument might be overridden by another setting. Some Chrome versions apply a default user agent if you do not specify one, but if you pass the argument, it should take effect. Verify by printing driver.execute_script("return navigator.userAgent") after starting the driver.

Headless mode detection

Even with a custom user agent, sites may detect headless Chrome through other signals. The navigator.webdriver property is the most common. Adding --disable-blink-features=AutomationControlled helps, but it is not a complete solution. Some sites use more advanced fingerprinting, such as checking for missing plugins or the behavior of the window.chrome object. In such cases, you may need to use a more sophisticated approach, such as a patched browser or a tool like Puppeteer with stealth plugins.

Proxy bypass not working

The --proxy-bypass-list argument uses a semicolon-separated list of hostnames. If you use commas or spaces, Chrome may ignore the list. Also, the list applies only to the proxy set via --proxy-server, not to extensions.

Operational Considerations for Production Scraping

When you run headless Chrome with a custom user agent and proxy in production, the setup has several operational implications.

Resource usage

Each webdriver.Chrome instance consumes memory and CPU. Headless mode reduces visual overhead but still runs a full browser engine. If you need many concurrent sessions, consider using a pool of driver instances or a browser automation framework that supports parallel execution.

Proxy reliability

A single proxy is a single point of failure. If the proxy goes down, all requests fail. For production scraping, use a pool of proxies and rotate them per request or per session. This also helps distribute load and reduce the chance of IP-based rate limiting.

User agent rotation

Using the same user agent for all requests makes your traffic easier to fingerprint. Rotate user agents from a list, ideally matching the operating system and browser version you claim to use. However, rotating too frequently may trigger anti-bot systems. A common pattern is to change the user agent per session, not per request.

Logging and observability

When you set a proxy, you lose the direct connection between the browser and the target server. This makes debugging harder. Log the proxy address and user agent for each driver session. If a request fails, you can correlate the failure with the specific proxy and agent combination.

Compatibility with Chrome versions

Chrome updates frequently, and command-line arguments sometimes change. The --proxy-server and --user-agent arguments have been stable for many years, but the --headless mode has changed. Test your setup after Chrome updates to ensure nothing broke.

Security

If you embed proxy credentials in the --proxy-server URL, they may appear in process listings or logs. Avoid this in production. Use the extension method or a secure configuration store. Also, be careful with proxies that are not under your control; they can see all unencrypted traffic.