python scrapy spider requests responses and selectors: Pract

python scrapy spider requests responses and selectors requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you write a Scrapy spider, you interact with three core objects: Request, Response, and Selector. The Request tells Scrapy what to fetch, the Response carries the fetched content, and the Selector extracts the data you need. This article walks through how these objects work together in a Python Scrapy spider, from issuing a request to parsing the response with CSS and XPath selectors.

The Minimal Spider That Uses Requests and Responses

A basic spider defines a start_urls list and a parse method. Scrapy turns each URL into a Request, sends it, and calls parse with the resulting Response. Here is a complete spider that scrapes quotes from a demo site:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("span small.author::text").get(),
            }

The response.css() method returns a SelectorList. Each quote is a Selector object, and calling .get() on it returns the first matching value. This pattern is the foundation of most Scrapy spiders.

Creating Request Objects Explicitly

While start_urls is convenient, you often need to generate requests dynamically. Use scrapy.Request to specify a URL, a callback, headers, and metadata. The meta argument lets you pass arbitrary data to the callback.

def parse(self, response):
    for href in response.css("li.next a::attr(href)").getall():
        yield scrapy.Request(
            url=response.urljoin(href),
            callback=self.parse,
            meta={"page": response.meta.get("page", 1) + 1}
        )

response.urljoin() converts a relative URL to an absolute one. The meta dictionary is shallow-copied from the original request, so you can read values from response.meta in the callback. This is how you pass state like page numbers or session tokens between requests.

Working with Response Objects

The Response object exposes the raw result of a request. Key attributes include response.status, response.headers, response.url, response.body (bytes), and response.text (decoded string). You can check the status code to handle errors or redirects:

def parse(self, response):
    if response.status != 200:
        self.logger.warning(f"Got status {response.status} for {response.url}")
        return
    # proceed with parsing

response.headers is a case-insensitive dict-like object. For example, response.headers.get("Content-Type") returns the content type. If you need the raw HTML, response.text is usually what you want, but be aware that Scrapy decodes it based on the response's encoding headers or meta tags.

Using Selectors to Extract Data

Selectors are the primary tool for extracting structured data from HTML or XML. Scrapy supports both CSS and XPath expressions. The .css() and .xpath() methods return SelectorList objects, and you can chain them.

# CSS selector
title = response.css("h1::text").get()

# XPath selector
price = response.xpath("//span[@class='price']/text()").get()

# Extract all matching elements
items = response.css("ul.items li").getall()

Use .get() for the first match and .getall() for a list of all matches. For attributes, use ::attr(name) in CSS or @name in XPath. Selectors can also be reused: you can store a Selector object and call .css() or .xpath() on it again, which is useful when you need to extract multiple fields from the same element.

Passing Data Between Requests with meta

The meta parameter on Request is a dictionary that travels with the request. It is accessible in the callback via response.meta. This is essential for carrying context like pagination counters, unique IDs, or authentication tokens.

def parse(self, response):
    for product in response.css("div.product"):
        yield scrapy.Request(
            url=product.css("a::attr(href)").get(),
            callback=self.parse_product,
            meta={"product_id": product.css("::attr(data-id)").get()}
        )

def parse_product(self, response):
    product_id = response.meta["product_id"]
    yield {
        "id": product_id,
        "name": response.css("h1::text").get(),
    }

Note that meta is shallow-copied for each request. If you put a mutable object inside, changes in the callback will affect the original object. For simple scalar values, this is not an issue. For complex state, consider using copy.deepcopy or passing immutable values.

Following Relative URLs with response.follow

response.follow() is a convenience method that creates a Request from a relative or absolute URL. It automatically resolves the URL against the current response, so you do not need response.urljoin().

def parse(self, response):
    for next_page in response.css("li.next a"):
        yield response.follow(next_page, callback=self.parse)

You can pass a Selector object directly, and Scrapy will extract the href attribute automatically. This is cleaner than manually extracting the attribute and calling urljoin. It also supports passing meta, headers, and other Request parameters.

Error Handling and Retry Behavior

Scrapy has a built-in retry middleware that automatically retries failed requests. By default, it retries requests that receive certain HTTP status codes (like 500, 502, 503) or that raise connection errors. You can control this via settings such as RETRY_TIMES and RETRY_HTTP_CODES.

For more control, you can pass an errback to Request. The errback is called when the request fails, allowing you to log the error or schedule a fallback.

def parse(self, response):
    yield scrapy.Request(
        url="https://example.com/api/data",
        callback=self.parse_api,
        errback=self.handle_error,
        meta={"retry_count": 0}
    )

def handle_error(self, failure):
    self.logger.error(f"Request failed: {failure.request.url}")
    retries = failure.request.meta.get("retry_count", 0)
    if retries < 3:
        yield scrapy.Request(
            url=failure.request.url,
            callback=self.parse_api,
            errback=self.handle_error,
            meta={"retry_count": retries + 1}
        )

This pattern gives you fine-grained control over retries, including custom backoff logic.

Performance and Runtime Considerations

Scrapy is asynchronous and can handle many concurrent requests. The CONCURRENT_REQUESTS setting controls how many requests are processed in parallel. For large crawls, you may need to adjust DOWNLOAD_DELAY to be polite to the target server. Also, be mindful of memory usage when handling very large responses. response.body is a bytes object that holds the entire content in memory. If you only need a portion, consider using streaming or a lighter parsing approach.

Another consideration is the DUPEFILTER_CLASS setting, which controls duplicate request detection. By default, Scrapy filters out duplicate URLs, which is useful but can sometimes drop legitimate requests with different query parameters. You can customize the duplicate filter to suit your needs.

When you use meta to pass data, remember that it is copied for each request. If you pass large objects, this can increase memory overhead. Keep meta small and focused on scalar values or small identifiers.

Finally, be aware that Scrapy's default encoding detection can occasionally misread pages with missing or incorrect charset declarations. If you encounter garbled text, explicitly set the encoding attribute on the Response or use response.text with a known encoding.