python scrapy retries concurrency and throttling: Practical

python scrapy retries concurrency and throttling requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

Scrapy's retry, concurrency, and throttling behavior is controlled by a small set of settings that are easy to configure in isolation and easy to misconfigure together. The defaults work for small crawls, but once a spider targets a server that rate-limits or fails intermittently, RETRY_TIMES, CONCURRENT_REQUESTS, and the AutoThrottle extension start influencing each other. A retry that fires immediately at high concurrency can turn a temporary 429 into a sustained burst of requests, which is exactly the pattern that gets a crawler blocked. This article explains how each mechanism works and how to configure python scrapy retries concurrency and throttling for a crawl that is both fast and polite.

How Scrapy Retries Failed Requests

Retries are handled by RetryMiddleware, which is enabled by default. It retries requests that fail with certain HTTP status codes or that raise network exceptions such as ConnectionError, TimeoutError, and DNSLookupError.

Three settings control the behavior:

RETRY_ENABLED (default True)
RETRY_TIMES (default 2)
RETRY_HTTP_CODES (default [500, 502, 503, 504, 522, 524, 408, 429])

RETRY_TIMES is the number of retries, not the total number of attempts. With the default value of 2, a request is attempted up to three times. Each retry increments the retry_times request meta key, and the middleware stops once that value exceeds RETRY_TIMES.

Two details matter. First, retries happen immediately. The middleware re-queues the request with no delay, so a server that is already struggling receives the next attempt right away. Second, the default retry list includes 429, the status code a server sends when it is rate-limiting you. Retrying a 429 immediately is usually counterproductive.

RETRY_PRIORITY_ADJUST controls the priority of retried requests relative to new ones. The default of -1 means retried requests are processed before other queued requests, which is another reason a burst of retries can dominate a crawl.

Setting Concurrency Limits

Concurrency is bounded by three settings:

CONCURRENT_REQUESTS (default 16): the global cap on in-flight requests
CONCURRENT_REQUESTS_PER_DOMAIN (default 8): the cap per domain
CONCURRENT_REQUESTS_PER_IP (default 0, disabled): the cap per IP when enabled

The effective limit is the most restrictive of the applicable settings. When CONCURRENT_REQUESTS_PER_IP is 0, per-domain limits apply; when it is enabled, per-IP limits override per-domain limits.

DOWNLOAD_DELAY (default 0) adds a fixed pause between requests from the same domain, and RANDOMIZE_DOWNLOAD_DELAY (default True) jitters that delay by up to 50% to avoid a perfectly regular request pattern.

These settings bound the number of requests in flight, but they do not distinguish between a first attempt and a retry. A retried request occupies the same concurrency slot as a fresh one, so heavy retrying effectively reduces the number of distinct URLs being crawled at any moment.

How AutoThrottle Adjusts the Request Rate

AutoThrottle is an extension that adjusts the download delay dynamically based on measured response latency. It is disabled by default.

Key settings:

AUTOTHROTTLE_ENABLED (default False)
AUTOTHROTTLE_START_DELAY (default 5.0)
AUTOTHROTTLE_MAX_DELAY (default 60.0)
AUTOTHROTTLE_TARGET_CONCURRENCY (default 1.0)
AUTOTHROTTLE_DEBUG (default False)

When enabled, AutoThrottle tracks the average latency for each domain and raises or lowers the delay so that the number of concurrent requests stays near AUTOTHROTTLE_TARGET_CONCURRENCY. If responses slow down, the delay increases; if responses speed up, the delay decreases. When AutoThrottle is active, the static DOWNLOAD_DELAY is ignored.

The important limitation is that AutoThrottle reacts to latency, not to status codes. A server that returns a fast 429 response has low latency, so AutoThrottle sees no reason to slow down. The extension cannot detect that the server is asking you to back off.

Why Retries, Concurrency, and Throttling Interact

This is where the three mechanisms meet. A retried request is re-queued through the normal scheduler, so it still respects concurrency limits and the current download delay. But because retries are immediate and get a priority boost, a spike of failures produces a concentrated burst of requests.

Consider a crawl running at CONCURRENT_REQUESTS_PER_DOMAIN = 8 where the server starts returning 500s. Each failing request is retried twice with high priority, so the queue fills with retries of the same URLs while new URLs wait. The effective concurrency is unchanged, but the diversity of the crawl collapses.

With AutoThrottle enabled, the same scenario behaves differently. If the 500s come with higher latency, AutoThrottle increases the delay, which naturally spaces out the retries. But if the failures are fast, AutoThrottle stays quiet and the retries continue at full speed.

The practical rule: retries amplify load, concurrency sets the ceiling for that load, and AutoThrottle only dampens it when latency rises. If you want retries to be gentle, you must slow them down explicitly, because none of the default settings do.

Handling 429 Responses and Retry-After Headers

The default middleware retries 429 responses immediately and ignores the Retry-After header that many servers send. For a rate-limited crawl, that is the wrong behavior. The standard fix is to subclass RetryMiddleware and add a delay before the retry.

import time
from scrapy.downloadermiddlewares.retry import RetryMiddleware

class RetryAfterMiddleware(RetryMiddleware):
    def process_response(self, request, response, spider):
        if response.status == 429:
            retry_after = response.headers.get('Retry-After')
            if retry_after:
                try:
                    delay = int(retry_after)
                except ValueError:
                    delay = 60
                time.sleep(delay)
        return super().process_response(request, response, spider)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RetryAfterMiddleware': 550,
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,
}

The tradeoff is real: time.sleep blocks the reactor thread, so it stalls the entire crawl, not just the retried request. For a low-concurrency crawl this is acceptable. For a high-concurrency crawl, a blocking sleep defeats the purpose of concurrency, and you should instead move retry scheduling out of the downloader, for example by re-queueing failed URLs through a separate job or a distributed scheduler.

Choosing Settings for a Specific Crawl

Start from the server's tolerance, not from Scrapy's defaults.

For a small site with unknown capacity, enable AutoThrottle, keep AUTOTHROTTLE_TARGET_CONCURRENCY around 1.0, and leave RETRY_TIMES at 2. AutoThrottle will find a sustainable rate without manual tuning.

For a crawl that must finish quickly against a robust server, raise CONCURRENT_REQUESTS_PER_DOMAIN and reduce DOWNLOAD_DELAY, but keep retries modest so a failure spike does not dominate the queue.

For endpoints that are known to be flaky, keep RETRY_TIMES at 2 or 3 and add the backoff middleware above so retries are spaced out.

For requests that are not idempotent, such as POST-based scraping or form submissions, consider RETRY_TIMES = 0 or filter which requests are retried, because a retried POST can duplicate a side effect on the server.

The right configuration depends on the target, and the only way to confirm it is to observe the crawl. Enable AUTOTHROTTLE_DEBUG or log response times and status codes, then adjust. A crawl that produces a steady stream of 200s with occasional retries is healthy. A crawl that produces a wall of 429s and 500s is telling you the retry and concurrency settings are out of sync with the server's limits.