Python Scrapy: CSS, XPath, and Link Following

When you write a Scrapy spider, the two decisions that shape the rest of the code are how you select data from the response and how you move from one page to the next. Scrapy gives you two selector systems, CSS and XPath, plus several ways to follow links. Understanding python scrapy css xpath and link following together matters because the selector you choose affects how much code you write, and the link-following strategy determines whether your crawler covers the site correctly or silently misses pages.

What CSS and XPath Selectors Return in Scrapy

Both response.css() and response.xpath() return a SelectorList, not a string and not a list of strings. Each item in that list is a Selector object. This is the source of most beginner confusion: you cannot use the result directly as text or as a URL.

def parse(self, response):
    titles = response.css('h2.product-title')
    # titles is a SelectorList, not a list of strings
    first = titles.get()          # returns the first matched element as a string
    first_text = titles.css('::text').get()  # returns the text of the first match

The two methods you will use most are .get() and .getall(). .get() returns the first match or None if nothing matched. .getall() returns a list of all matches. The older .extract() and .extract_first() methods still work but are deprecated in current Scrapy versions; prefer .get() and .getall() in new code.

Because both CSS and XPath return the same SelectorList type, you can chain them. A CSS selector can be narrowed with an XPath expression and vice versa:

price = response.css('div.product').xpath('.//span[@class="price"]/text()').get()

The .// prefix is important here. It means "search within the current element". A plain //span would search from the document root, which usually returns more than you want and can pull in elements from other parts of the page.

When CSS Selectors Are the Right Choice

CSS selectors are concise for class, id, and attribute selection. Scrapy extends standard CSS with two pseudo-elements that are not part of browser CSS: ::text to select text nodes and ::attr(name) to select attributes.

def parse(self, response):
    for product in response.css('div.product'):
        name = product.css('h2::text').get()
        link = product.css('a::attr(href)').get()
        price = product.css('span.price::text').get()

This is the clearest way to extract data when the structure is stable and you are selecting by class or id. The main limitation is that CSS cannot express text-based conditions. There is no CSS selector for "an element whose text contains the word sale". For that you need XPath.

Another limitation is parent traversal. CSS has no parent selector. If you need to find an element and then walk up to its container, XPath is the better tool.

When XPath Is the Right Choice

XPath becomes necessary when the selection depends on the text content, the position of an element, or the relationship between elements.

def parse(self, response):
    sale_items = response.xpath('//div[contains(@class, "product") and contains(.//span[@class="price"], "sale")]')
    for item in sale_items:
        name = item.xpath('.//h2/text()').get()

Text-based conditions are where XPath is clearly superior:

# All links whose text contains "next"
next_links = response.xpath('//a[contains(text(), "next")]')

# All elements whose class attribute starts with "prod-"
products = response.xpath('//*[starts-with(@class, "prod-")]')

XPath also handles structural relationships that CSS cannot express, such as selecting the parent of a matched element or a following sibling:

# The row that contains a cell with the value "out of stock"
row = response.xpath('//td[text()="out of stock"]/..')

# The next sibling after a heading
next_section = response.xpath('//h2[text()="Specifications"]/following-sibling::p')

When you chain XPath inside a loop, remember that a leading / means "from the document root" and a leading .// means "from the current element". This distinction is the most common source of wrong results when mixing CSS and XPath.

Combining CSS and XPath in One Spider

There is no rule that forces a spider to use only one selector system. The pragmatic approach is to use CSS for the outer structure, where classes and ids are usually stable, and switch to XPath for the inner conditions that CSS cannot express.

def parse(self, response):
    for product in response.css('div.product'):
        name = product.css('h2::text').get()
        # XPath for the text-based condition
        availability = product.xpath('.//span[contains(text(), "available")]/text()').get()
        yield {
            'name': name,
            'availability': availability,
        }

The cost of mixing is minimal because both systems return the same SelectorList type. The main thing to keep straight is the context of a chained XPath expression: use .// when you want to search within the current selector, and // only when you intend to search the whole document.

Following Links with response.follow

The simplest way to move from one page to another is to issue a new request from within a parse method. Scrapy provides response.follow, which resolves relative URLs for you. This is the key difference from scrapy.Request, which requires you to build an absolute URL yourself.

def parse(self, response):
    for link in response.css('a.product-link::attr(href)').getall():
        yield response.follow(link, callback=self.parse_product)

def parse_product(self, response):
    yield {
        'name': response.css('h1::text').get(),
        'price': response.css('span.price::text').get(),
    }

Because response.follow accepts a relative URL, a selector result, or even a Selector, it removes a whole class of errors around urljoin. If you need to build the URL manually, use response.urljoin:

yield scrapy.Request(response.urljoin(link), callback=self.parse_product)

The two are equivalent for absolute URLs, but response.follow is shorter and less error-prone for the common case of relative hrefs.

To pass data from the current page to the callback, use cb_kwargs:

yield response.follow(
    link,
    callback=self.parse_product,
    cb_kwargs={'category': category_name},
)

def parse_product(self, response, category):
    yield {
        'category': category,
        'name': response.css('h1::text').get(),
    }

The older pattern of passing values through the meta dict still works, but cb_kwargs is the cleaner interface in current Scrapy versions.

CrawlSpider Rules for Structured Link Following

When a site has pagination, category pages, and detail pages, writing manual response.follow calls for every link type becomes repetitive. CrawlSpider exists for exactly this pattern. You define rules that describe which links to extract and which callback handles them.

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class ProductSpider(CrawlSpider):
    name = 'products'
    allowed_domains = ['example.com']
    start_urls = ['https://example.com/catalog']

    rules = (
        Rule(LinkExtractor(restrict_xpaths='//div[@class="pagination"]'), follow=True),
        Rule(LinkExtractor(restrict_css='a.product-link'), callback='parse_item'),
    )

    def parse_item(self, response):
        yield {
            'name': response.css('h1::text').get(),
            'price': response.css('span.price::text').get(),
        }

The first rule extracts links from the pagination block and follows them without calling a callback, so the spider walks through every page of the catalog. The second rule extracts product links and sends each one to parse_item.

LinkExtractor accepts several restrictions that narrow the crawl scope: allow and deny for URL patterns, restrict_xpaths and restrict_css to limit which parts of the page are scanned, and allow_domains to prevent the spider from leaving the site. The more you restrict, the fewer requests the spider makes and the less likely it is to crawl irrelevant pages.

A common mistake with CrawlSpider is forgetting that the callback for a rule must be named parse_item or must not be named parse. CrawlSpider overrides parse internally to drive the rules, so naming your own method parse breaks the rule engine.

Common Failure Modes and How to Diagnose Them

Empty selections are the most frequent failure. If a selector matches nothing, .get() returns None and .getall() returns an empty list. The spider does not raise an error; it just yields a dict with None values. When you see None fields in your output, the first thing to check is whether the selector matches anything at all. A quick way to test is to run the spider in the Scrapy shell against the same URL and inspect the selector results directly.

Context errors in chained XPath are the second most common failure. Using // inside a loop that iterates over products will search the entire document, not the current product element. The result is usually duplicated data or values from the wrong part of the page. Use .// when you intend to search within the current selector.

Link following fails in two typical ways. The first is passing a relative URL to scrapy.Request without calling response.urljoin; the request then targets a malformed URL. The second is following links that point to external domains. Scrapy does not automatically restrict the domain, so a crawler that follows every href can wander off the site. Use allowed_domains on the spider or allow_domains on the LinkExtractor to keep the crawl inside the intended scope.

Performance and Operational Considerations

The cost of a crawl is dominated by the number of requests, not by selector evaluation. Selector work happens in memory on an already downloaded page, so the practical lever for performance is how many links the spider follows and how fast it issues requests.

Restricting link extraction is the most effective optimization. A LinkExtractor that scans the whole page for hrefs costs more than one restricted to the pagination block, and it produces more requests. The same applies to manual response.follow loops: select only the links you actually need instead of every anchor on the page.

Scrapy already handles URL deduplication through its request fingerprinting, so duplicate links do not cause duplicate downloads. What it does not do by default is rate limiting. If you crawl a site too aggressively, you may be blocked. The DOWNLOAD_DELAY setting inserts a delay between requests, and CONCURRENT_REQUESTS controls how many requests run in parallel. For a small site, a delay of one to two seconds is usually enough to avoid hammering the server.

The ROBOTSTXT_OBEY setting controls whether Scrapy respects robots.txt. It is off by default in many setups, so if your crawler must be polite to a site, enable it explicitly. Note that obeying robots.txt can also prevent the spider from reaching pages that are disallowed, which is a behavioral change you should account for when the crawl appears to stop early.

Finally, be careful with memory when a crawl is large. SelectorLists hold references to the parsed response, so keeping them around after the callback returns can accumulate memory. Yield the extracted data as soon as you have it rather than storing large lists of selectors in spider attributes.