Python Playwright Proxy, User Agent, and Headless Browser

Configuring a Python Playwright browser with a proxy, a custom user agent, and headless mode is a standard requirement for web scraping, automated checks, and monitoring scripts. The three settings are not applied in the same place: headless mode is a launch option, while the proxy and user agent belong to the browser context. Getting that separation right is the core of a correct python playwright proxy user agent and headless browser setup.

Setting Proxy, User Agent, and Headless Mode in One Script

The minimal configuration combines all three settings in a single script:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        proxy={
            "server": "http://proxy.example.com:8080",
            "username": "scraper",
            "password": "secret",
        },
        user_agent=(
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/120.0.0.0 Safari/537.36"
        ),
    )
    page = context.new_page()
    page.goto("https://example.com")
    print(page.title())
    browser.close()

headless=True tells Chromium to run without a visible window. The proxy and user_agent arguments are passed to new_context, not to launch. Every page created from that context inherits both settings, so you do not need to repeat them per page.

Why Proxy and User Agent Are Context-Level Settings

A browser instance in Playwright can host multiple contexts. Each context is an isolated session with its own cookies, storage, and network settings. That isolation is why proxy and user agent are defined per context rather than per browser.

# p is the active Playwright instance from sync_playwright()
browser = p.chromium.launch(headless=True)

context_a = browser.new_context(
    proxy={"server": "http://proxy-a.example.com:8080"},
    user_agent="Mozilla/5.0 (compatible; ScraperA/1.0)",
)

context_b = browser.new_context(
    proxy={"server": "http://proxy-b.example.com:8080"},
    user_agent="Mozilla/5.0 (compatible; ScraperB/1.0)",
)

This pattern lets you rotate proxies or test different user agents without relaunching the browser. Launching a browser is comparatively expensive; creating a context is cheap. If your script needs several proxy identities, create one browser and multiple contexts.

Proxy Options: server, username, password, and bypass

The proxy dictionary accepts four keys:

A common setup that sends external traffic through a proxy but keeps localhost direct:

context = browser.new_context(
    proxy={
        "server": "socks5://proxy.example.com:1080",
        "bypass": "localhost,127.0.0.1,*.internal.example.com",
    }
)

The server value must include a scheme. http:// and socks5:// are the common choices; the set of supported schemes depends on the Playwright version you are using. When no bypass is given, all requests from the context are routed through the proxy. That means a local service that is not in the bypass list will fail to connect.

Setting a Custom User Agent Per Context

The user_agent argument overrides the default header sent by Chromium. The default value depends on the bundled Chromium version and the Playwright release. Older headless configurations exposed a HeadlessChrome marker in the default user agent, which is a well-known automation signal. If the exact header matters for the sites you target, set it explicitly instead of relying on the default.

context = browser.new_context(
    user_agent=(
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/121.0.0.0 Safari/537.36"
    )
)

The user agent is applied at the network layer for every request the context makes, including subresources and XHR calls. You can also override the header per request with page.set_extra_http_headers, but the context-level option is the cleaner approach because it applies consistently and does not depend on the order in which requests are made.

One practical caveat: a user agent that claims a specific browser version should roughly match the Chromium version Playwright is actually running. If the header claims Chrome 121 but the JavaScript APIs behave like a different version, a strict site may flag the mismatch. Matching the header to the installed Chromium version avoids that class of problem.

Headless Mode and Its Detection Surface

Playwright launches browsers in headless mode by default, so headless=True is often redundant but explicit. Passing headless=False opens a visible window, which is useful when debugging layout, waiting for a slow page, or confirming that a selector matches what a human would see.

Headless mode changes more than the absence of a window. Rendering, GPU access, and some browser APIs behave differently. Sites that attempt to detect automation often combine the user agent with JavaScript checks such as navigator.webdriver, screen dimensions, and rendering behavior. Playwright patches several of these by default, but no automation tool can guarantee that every detection vector is hidden. If a target behaves differently in headless mode, run the same flow with headless=False to isolate whether the mode is the cause before assuming the proxy or user agent is at fault.

Reusing One Browser Across Multiple Proxy Contexts

A common mistake in scraping scripts is launching a new browser for every request. Browser startup involves process creation, profile initialization, and network setup, so it is far more expensive than creating a context. The efficient pattern is one browser, many contexts:

from playwright.sync_api import sync_playwright

PROXIES = [
    {"server": "http://proxy-a.example.com:8080"},
    {"server": "http://proxy-b.example.com:8080"},
]
USER_AGENT = (
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
    "AppleWebKit/537.36 (KHTML, like Gecko) "
    "Chrome/120.0.0.0 Safari/537.36"
)

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    try:
        for proxy in PROXIES:
            context = browser.new_context(proxy=proxy, user_agent=USER_AGENT)
            try:
                page = context.new_page()
                page.goto("https://example.com", timeout=30_000)
                html = page.content()
                # process html
            finally:
                context.close()
    finally:
        browser.close()

Each context gets its own proxy and user agent, and closing the context releases its resources without tearing down the browser. This keeps the script fast while preserving isolation between proxy identities.

Failure Modes and Runtime Behavior

Proxy and user agent configuration fails in predictable ways. Knowing the symptoms shortens debugging.

If the proxy server is unreachable, page.goto raises a timeout or a connection error rather than returning a partial page. The exact exception depends on where the failure occurs: a refused TCP connection surfaces differently from a DNS failure inside the proxy.

If the proxy requires authentication and the credentials are wrong, the proxy typically responds with a 407 status. Whether the navigation raises or returns an error page depends on how the proxy formats its response. Inspect the response status with the value returned by page.goto or with page.on("response") to distinguish a proxy error from a target-site error.

A missing bypass entry is a frequent cause of confusing failures. If your script talks to a local service while a proxy is configured, add that host to bypass; otherwise the request is sent to the proxy and fails.

User agent problems usually appear as unexpected responses rather than exceptions. A site may return a block page, a CAPTCHA, or a different HTML variant when it does not trust the header. Compare the response for your custom user agent against a known-good browser header to confirm the header is the variable that matters.

Performance and Concurrency Considerations

The async API makes it straightforward to run multiple contexts concurrently, each with its own proxy. The browser is launched once, and contexts are created and closed independently:

import asyncio
from playwright.async_api import async_playwright

async def fetch_with_proxy(browser, url, proxy):
    context = await browser.new_context(proxy=proxy, user_agent=USER_AGENT)
    try:
        page = await context.new_page()
        await page.goto(url, timeout=30_000)
        return await page.content()
    finally:
        await context.close()

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        try:
            results = await asyncio.gather(
                *(fetch_with_proxy(browser, "https://example.com", px) for px in PROXIES)
            )
        finally:
            await browser.close()

asyncio.run(main())

Concurrency changes the failure profile. A slow proxy now delays one task instead of the whole script, but it also consumes a slot in the event loop. If one proxy in the list is down, its task raises while the others complete. Decide whether a failed proxy should abort the batch or be skipped; asyncio.gather with return_exceptions=True lets you collect per-task errors instead of failing the entire run.

Proxy latency also affects total runtime. Every request through the proxy pays an additional round trip, and some proxies throttle connection reuse. If throughput matters, measure the per-request time with and without the proxy to understand whether the proxy or the target site is the bottleneck. Reusing a browser and contexts keeps the Playwright-side overhead low, but it cannot remove the latency introduced by the proxy itself.