Python lxml XPath Selectors and Attributes

When you need to extract structured data from XML or HTML in Python, lxml's XPath support is one of the most direct tools available. The xpath method on lxml elements lets you run XPath 1.0 expressions against parsed documents, covering everything from simple tag selection to attribute filtering and text extraction. This article focuses on how to use python lxml xpath selectors and attributes effectively in real-world parsing tasks.

Basic XPath Selection with lxml

The core of lxml's XPath support is the xpath() method, available on Element and ElementTree objects. It takes an XPath expression as a string and returns a list of matching objects, which can be elements, attributes, or strings depending on what the expression selects.

from lxml import html

doc = html.fromstring('<div class="content"><p>Hello</p><p>World</p></div>')
paragraphs = doc.xpath('//p')
print(len(paragraphs))  # 2

The expression //p selects all <p> elements anywhere in the document. The result is a list of Element objects. If no match is found, the list is empty, not None. This is a common source of confusion for developers coming from other XML libraries.

Selecting Elements by Tag and Path

XPath supports absolute and relative paths. An absolute path starts from the document root, while a relative path starts from the current context node. In lxml, you can call xpath() on any element to evaluate a relative expression.

root = doc.xpath('//div')[0]
# Relative path: select p elements directly under this div
children = root.xpath('p')
# Select all p elements under this div, including nested ones
descendants = root.xpath('.//p')

The .// prefix means "descendant-or-self", so .//p finds all <p> elements below the current node, not just direct children. This distinction matters when your document has nested structures.

You can also use wildcards. //* selects every element in the document, and //div/* selects all direct children of <div> elements. For more precise selection, combine tag names with predicates.

Working with Attributes in XPath

Attributes are a central part of XPath selection. You can filter elements based on attribute values or extract attribute values directly.

Selecting Elements by Attribute Value

Use the @ symbol to reference an attribute. The expression //a[@href] selects all <a> elements that have an href attribute, regardless of its value. To match a specific value, use //a[@href='https://example.com'].

links = doc.xpath('//a[@href]')
for link in links:
    print(link.get('href'))

Attribute values in XPath are always strings. If you need to compare with a number, convert the attribute to a number using number(), but this is rarely necessary for typical parsing tasks.

Extracting Attribute Values

To get the value of an attribute directly, include @attribute in the expression. The result will be a list of strings.

hrefs = doc.xpath('//a/@href')
# Returns ['https://example.com', 'https://example.org']

This is often more efficient than selecting elements and then calling get() in a loop, especially for large documents.

Extracting Text and Attribute Values

XPath can also return text content. The text() function selects the direct text node of an element, while string() converts an element to its full string content, including descendant text.

# Direct text node only
first_para_text = doc.xpath('//p[1]/text()')
# Full text content including nested elements
full_text = doc.xpath('string(//div)')

Note that text() returns a list of strings, one for each text node. If an element contains multiple text nodes (e.g., due to inline elements), you get multiple entries. string() returns a single string and is useful when you want the entire text content of an element.

When you need to combine text extraction with attribute filtering, you can chain predicates. For example, //input[@type='text']/@value extracts the value attribute from all text inputs.

Handling Namespaces in XPath

XML documents often use namespaces, which can break XPath expressions if not handled correctly. lxml requires you to provide a namespace mapping when the document uses them.

from lxml import etree

xml = '''<root xmlns:h="http://www.w3.org/TR/html4/">
  <h:table>
    <h:tr><h:td>Cell</h:td></h:tr>
  </h:table>
</root>'''

tree = etree.fromstring(xml)
# Without namespace mapping, this fails
# tree.xpath('//table')

ns = {'h': 'http://www.w3.org/TR/html4/'}
tables = tree.xpath('//h:table', namespaces=ns)

The namespaces parameter accepts a dictionary mapping prefixes to URIs. You must use the same prefix in the XPath expression as defined in the mapping, regardless of the prefix used in the document. If the document uses a default namespace, you need to assign a prefix in the mapping and use it in the expression.

For HTML documents parsed with lxml.html, namespaces are usually not an issue because HTML doesn't use them. But if you're parsing generic XML, always check for namespaces and provide the mapping.

Common XPath Mistakes and How to Avoid Them

Several mistakes recur when developers first work with lxml XPath.

Forgetting That xpath() Returns a List

Many expect a single element when the expression matches one node. Instead, you get a list. Use [0] to get the first match, but first check that the list is not empty.

matches = doc.xpath('//div[@id="main"]')
if matches:
    main_div = matches[0]
else:
    # handle absence

Misunderstanding text() vs string()

text() returns a list of text nodes, not a single string. If you need the full text content, use string() or join the list yourself. string() is an XPath function that converts the first node in the node-set to a string, so it's best used with a single-element expression.

Ignoring Namespaces in XML

As shown above, omitting the namespaces parameter when the document has namespaces leads to an empty result or an error. Always inspect the document's namespace declarations and pass the mapping.

Overusing // for Performance

The // selector searches the entire document. In large XML files, this can be slow. Prefer absolute paths or relative paths that limit the search scope. For example, if you know the structure, use /root/body/div instead of //div.

Performance Considerations for XPath Queries

lxml is a C-accelerated library, so XPath evaluation is generally fast. However, the way you write expressions can affect performance, especially on large documents.

• Avoid // when you can use a direct path. // forces a full tree traversal.
• Use predicates that filter early. For example, //div[@class='item'] is faster than //div followed by a Python-side check.
• If you run the same XPath expression many times, consider compiling it with etree.XPath().

from lxml import etree

find_items = etree.XPath('//div[@class="item"]')
items = find_items(doc)

Compiling the expression once avoids re-parsing the expression string on every call, which can yield measurable improvements in loops.

Another consideration is memory. lxml builds a full tree in memory, so for very large files, you may want to use iterparse for incremental parsing. XPath works on the tree, so it's not suitable for streaming. If you need to process a huge XML file without loading it entirely, XPath is not the right tool; use iterparse instead.

When parsing HTML from the web, be aware that lxml.html is lenient and will fix broken markup, but this normalization can change the tree structure. Test your XPath expressions against the actual parsed document, not just the raw HTML source.

For maintainability, keep XPath expressions in one place, preferably as constants or in a configuration file, so they can be updated when the document structure changes. This is especially important in web scraping where the target site's markup may evolve.