Python Scrapy Pagination and Multi Page Crawling

When a website splits its content across multiple pages, a Scrapy spider must follow the pagination chain to collect all items. Python scrapy pagination and multi page crawling is a core pattern for any serious scraping project, and Scrapy offers several ways to handle it. The right approach depends on the site's URL structure, whether the next link is a real HTML anchor, and how much control you need over the crawl flow.

Understanding Pagination in Scrapy

Pagination means the site presents content in numbered or sequentially linked pages. A typical example is an e-commerce category page that shows 24 products per page and provides a "Next" button or a page-numbered URL like ?page=2. For a Scrapy spider to collect all products, it must issue requests for each page and parse the same item extraction logic on every response.

Scrapy does not automatically follow pagination. You must explicitly tell the spider how to find the next page URL and when to stop. The two most common strategies are using CrawlSpider with LinkExtractor rules, or manually constructing Request objects inside a parse method. Both work, but they differ in how much control you have and how they handle edge cases.

Using CrawlSpider with LinkExtractor for Automatic Pagination

CrawlSpider is a Scrapy spider class designed for crawling links that match a set of rules. It is particularly effective when the next page link is a standard <a> tag with a predictable pattern. You define a Rule that instructs the spider to follow links matching a given XPath or CSS selector, and a callback to parse the item data.

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ProductSpider(CrawlSpider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products"]

    rules = (
        Rule(LinkExtractor(restrict_xpaths="//a[@rel='next']"), callback="parse_item", follow=True),
    )

    def parse_item(self, response):
        for product in response.xpath("//div[@class='product']"):
            yield {
                "name": product.xpath(".//h2/text()").get(),
                "price": product.xpath(".//span[@class='price']/text()").get(),
            }

In this example, the LinkExtractor looks for an anchor with rel="next", which is a common semantic marker. The follow=True parameter makes the spider continue following the next link even after calling the callback. The callback extracts product data from the current page. This pattern works well when the pagination link is a direct HTML anchor and the site does not require session state or complex URL construction.

The main advantage of CrawlSpider is that it handles the request scheduling and deduplication for you. However, it can be less flexible when the next page URL is not a simple anchor, or when you need to pass data between pages.

Manual Pagination with parse and Request Callbacks

When you need more control, you can write a regular Spider and manually yield Request objects for the next page. This is useful when the pagination uses query parameters, when you need to modify headers or cookies between pages, or when you want to stop based on a condition other than a missing link.

import scrapy

class ProductSpider(scrapy.Spider):
    name = "products_manual"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products?page=1"]

    def parse(self, response):
        for product in response.xpath("//div[@class='product']"):
            yield {
                "name": product.xpath(".//h2/text()").get(),
                "price": product.xpath(".//span[@class='price']/text()").get(),
            }

        next_page = response.xpath("//a[@rel='next']/@href").get()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)

Here, response.urljoin resolves relative links. The callback is the same parse method, so the spider continues until no next link is found. This approach gives you full control over the request. You can add headers, meta data, or even change the callback based on the page number.

A common variation is to construct the next URL manually when the site uses a numeric parameter. For example:

page_number = response.meta.get('page_number', 1)
next_page = page_number + 1
next_url = f"https://example.com/products?page={next_page}"
yield scrapy.Request(next_url, callback=self.parse, meta={'page_number': next_page})

This pattern is useful when the site does not provide a next link but uses a predictable URL scheme. You must define a stopping condition, such as a maximum page number or an empty response.

Handling Pagination Parameters and Dynamic URLs

Some sites use offset-based pagination (?offset=24), cursor-based pagination, or even POST requests for the next page. In those cases, you cannot rely on a simple LinkExtractor. You need to parse the response to extract the next page identifier, or compute it from the current state.

For offset-based pagination, you can read the number of items on the page and increment the offset. For example, if each page shows 24 items, the next offset is current_offset + 24. You can store the current offset in the request meta.

class OffsetSpider(scrapy.Spider):
    name = "offset"
    start_urls = ["https://example.com/api/items?offset=0"]

    def parse(self, response):
        items = response.json().get("items", [])
        for item in items:
            yield item

        if len(items) == 24:  # assume full page means more
            offset = response.meta.get("offset", 0) + 24
            yield scrapy.Request(
                f"https://example.com/api/items?offset={offset}",
                callback=self.parse,
                meta={"offset": offset}
            )

Dynamic URLs that require JavaScript rendering cannot be handled by Scrapy alone. You may need to use a headless browser like Splash or Selenium to execute the JavaScript and extract the next page link. Scrapy integrates with Splash via scrapy-splash, but that adds a dependency and operational overhead. Before going down that path, check if the site exposes a JSON API or a hidden HTML link that Scrapy can use directly.

Performance and Rate Limiting Considerations

Multi-page crawling multiplies the number of requests your spider sends. Without proper throttling, you risk overwhelming the target server and getting your IP blocked. Scrapy provides built-in settings to control concurrency and download delay.

• CONCURRENT_REQUESTS controls how many requests are processed in parallel.
• DOWNLOAD_DELAY sets a minimum delay between requests.
• AUTOTHROTTLE_ENABLED automatically adjusts the delay based on server response times.

For a polite crawler, enable autothrottle and set a reasonable DOWNLOAD_DELAY (e.g., 0.5 seconds). You can also use RANDOMIZE_DOWNLOAD_DELAY to vary the delay slightly, making the request pattern less predictable.

# settings.py
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0
DOWNLOAD_DELAY = 0.5
RANDOMIZE_DOWNLOAD_DELAY = True

These settings apply globally, but you can override them per request using meta keys like download_timeout or dont_retry. If you are crawling a large number of pages, also consider using JOBDIR to pause and resume the crawl, and LOG_LEVEL to reduce log noise.

Error Handling and Robustness in Pagination

Pagination chains can break. The next link may disappear temporarily, the site may return a 404, or the page structure may change. A robust spider should handle these failures gracefully.

Scrapy automatically retries failed requests based on RETRY_TIMES and RETRY_HTTP_CODES. You can also catch specific exceptions in your callback. For example, if the next page returns an empty item list, you might decide to stop rather than continue.

def parse(self, response):
    items = response.xpath("//div[@class='product']")
    if not items:
        self.logger.info("No items found, stopping pagination")
        return
    for item in items:
        yield {...}
    next_link = response.xpath("//a[@rel='next']/@href").get()
    if next_link:
        yield scrapy.Request(response.urljoin(next_link), callback=self.parse)

Another concern is duplicate pages. Scrapy's default request deduplication prevents the same URL from being requested twice. However, if the site uses session IDs or tracking parameters in the URL, you may need to override start_requests to clean the URL or use dont_filter=False appropriately.

When using CrawlSpider, the Rule can include a process_request function to modify the request before it is sent. This is useful for adding headers or changing the callback based on the URL. For manual pagination, you have full control over the request lifecycle, so you can implement custom retry logic or stop conditions.

A common edge case is infinite pagination loops caused by a next link that points to the current page. Always verify that the next URL is different from the current one. In manual pagination, you can compare the next URL with response.url and stop if they match. In CrawlSpider, the link extractor may follow the same link repeatedly, but Scrapy's deduplication will eventually skip it; still, it is safer to restrict the LinkExtractor with allow or deny patterns to avoid unwanted links.

For sites that require a login or session, you must handle authentication before starting the pagination chain. Use cookies in the request or a DownloaderMiddleware to attach session cookies. The pagination logic itself remains the same, but the initial request must be authenticated.

Finally, consider the memory footprint. If you are crawling thousands of pages, the items you yield are stored in memory until the item pipeline processes them. Use item pipelines to write data incrementally, and avoid storing all items in a list. Scrapy's default behavior of yielding items as they are parsed is memory-efficient, but if you need to aggregate data across pages, use a separate store like a database or a file.

Pagination is a fundamental part of web scraping with Scrapy. Whether you choose CrawlSpider for simplicity or manual requests for control, the key is to understand the site's pagination mechanism and to build a spider that can handle failures and respect the server's resources. The patterns shown here cover the majority of pagination scenarios you will encounter, from simple next links to offset-based APIs and dynamic rendering workarounds.