Python Scrapy Middleware for Cookies, Proxy, and User Agent

python scrapy middleware cookies proxy and user agent requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When a Scrapy spider needs to handle cookies, rotate user agents, or route traffic through proxies, the cleanest place to implement that logic is in downloader middleware. These components sit between the engine and the downloader, allowing you to modify each request before it leaves the process and each response before it reaches the spider. This article explains how to write and configure middleware for cookies, proxy, and user agent handling in a Python Scrapy project.

Why Middleware Is the Right Place for Request Modification

Scrapy's downloader middleware is a framework for intercepting requests and responses globally. Each request passes through a chain of middleware objects, and each can modify the request, return a response directly, or raise an exception. This is ideal for cross-cutting concerns like setting a User-Agent header, attaching a proxy address, or managing cookies—concerns that apply to many requests rather than a single spider.

Spider middleware, in contrast, operates on the spider's input and output. For request-level attributes like headers and meta, downloader middleware is the correct layer. You can also use the process_request method to alter the request object before it is sent.

Handling Cookies with Scrapy Middleware

Scrapy ships with a built-in CookiesMiddleware that maintains a cookie jar per domain. It is enabled by default and works transparently for most sessions. However, you may need to customize behavior, such as using a specific cookie for authentication or persisting cookies across runs.

To override the default behavior, you can create a custom middleware that modifies the Cookie header directly. For example, to attach a fixed session cookie:

class FixedCookieMiddleware:
    def __init__(self, cookie_value):
        self.cookie_value = cookie_value

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get('FIXED_COOKIE'))

    def process_request(self, request, spider):
        if self.cookie_value:
            request.headers['Cookie'] = self.cookie_value
        return None

This middleware sets the Cookie header for every request. If you need to manage multiple cookies or handle expiration, you can store them in a dictionary and update the header based on the request URL.

The built-in CookiesMiddleware uses the cookiejar meta key to isolate sessions. If you want to use a specific jar for a request, set request.meta['cookiejar'] = 'session1'. This is useful when a spider handles multiple login sessions simultaneously.

Rotating User Agents in Middleware

Websites often block requests that use a default or repeated User-Agent string. Rotating the User-Agent per request reduces the chance of detection. You can implement a middleware that cycles through a list of common browser strings.

import random

class RandomUserAgentMiddleware:
    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist('USER_AGENTS'))

    def process_request(self, request, spider):
        if self.user_agents:
            request.headers['User-Agent'] = random.choice(self.user_agents)
        return None

Set USER_AGENTS in settings.py as a list of strings. The middleware randomly picks one for each request. If you need a more deterministic rotation, you can use a counter based on the request index.

Note that Scrapy's default UserAgentMiddleware sets a single User-Agent from the USER_AGENT setting. Your custom middleware should be placed after the default in the middleware order to override it, or you can disable the default and rely entirely on your own.

Managing Proxies with Middleware

Proxies are often required to avoid IP bans or to access geo-restricted content. Scrapy does not include a built-in proxy middleware, so you must implement one. The standard approach is to set the proxy key in request.meta to the proxy URL. The downloader then uses that address for the connection.

class ProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist('PROXIES'))

    def process_request(self, request, spider):
        if self.proxies:
            request.meta['proxy'] = random.choice(self.proxies)
        return None

For authenticated proxies, include credentials in the URL, such as http://user:pass@host:port. Be careful with sensitive credentials in logs; avoid printing the proxy URL.

You can also assign proxies per request based on the domain or spider. For example, use a different proxy for each request to avoid reusing the same IP for consecutive requests.

Combining Cookies, Proxy, and User Agent in One Middleware

While separate middleware classes are easier to maintain, you can combine all three concerns into a single middleware if the logic is tightly coupled. For instance, when a proxy requires a specific cookie or when the User-Agent should be consistent with the proxy's region.

class UnifiedRequestMiddleware:
    def __init__(self, user_agents, proxies, cookie):
        self.user_agents = user_agents
        self.proxies = proxies
        self.cookie = cookie

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            crawler.settings.getlist('USER_AGENTS'),
            crawler.settings.getlist('PROXIES'),
            crawler.settings.get('FIXED_COOKIE')
        )

    def process_request(self, request, spider):
        if self.user_agents:
            request.headers['User-Agent'] = random.choice(self.user_agents)
        if self.proxies:
            request.meta['proxy'] = random.choice(self.proxies)
        if self.cookie:
            request.headers['Cookie'] = self.cookie
        return None

This approach reduces the number of middleware classes but makes the code less modular. Use it only when the settings are always applied together.

Configuring Middleware Order in Settings

Scrapy processes downloader middleware in the order defined by the DOWNLOADER_MIDDLEWARES setting. The order matters because each middleware can modify the request before the next one sees it. For example, if you want your proxy middleware to run before the user-agent middleware, list it earlier.

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ProxyMiddleware': 543,
    'myproject.middlewares.RandomUserAgentMiddleware': 544,
    'myproject.middlewares.FixedCookieMiddleware': 545,
}

The numeric values determine the order: lower numbers run first. The built-in middleware have default orders; you can override them by setting the same class with a different number. To disable a built-in middleware, set its value to None.

This table shows typical orders. Your custom middleware should be placed before the built-in ones if you need to override their behavior. For instance, to override the default User-Agent, your middleware must run before UserAgentMiddleware (order < 400) or disable it entirely.

Common Pitfalls and Production Considerations

One common mistake is forgetting that process_request must return None or a Request/Response object. Returning None lets the request continue to the next middleware. Returning a Request object stops the chain and sends that new request instead. Returning a Response object bypasses the downloader entirely.

When rotating proxies, be aware that not all proxies support HTTPS or specific HTTP methods. Test your proxy list before deploying. Also, some proxies leak your original IP if the X-Forwarded-For header is not set correctly. You can set this header in the middleware, but be aware that many sites ignore it.

Cookie management with multiple sessions requires careful handling of the cookiejar meta key. If you reuse the same jar across requests, you may inadvertently share session state between different spiders or domains. Use unique jar names per logical session.

Performance is another consideration. Randomly selecting a proxy from a list is cheap, but if you need to check proxy availability or maintain a pool of healthy proxies, you may need to integrate a proxy service or a health-check mechanism. Avoid doing I/O in process_request unless necessary, because it runs for every request and can slow down the crawl.

Security matters when handling proxy credentials. Never log the full proxy URL, and consider using environment variables or Scrapy's settings with sensitive values redacted. Also, be cautious when setting cookies from external sources; validate the cookie domain and path to prevent session fixation attacks.

Finally, test your middleware in a staging environment before running a large crawl. Use Scrapy's scrapy shell to verify that headers and meta are set correctly on individual requests. This helps catch configuration errors early without burning through your proxy quota or triggering anti-bot measures.