Python Playwright Async Web Scraping Multiple Pages
python playwright async web scraping multiple pages: Learn how to use Playwright's async API in Python to scrape multiple pages concurrently, manage browser contexts,...
When you need to scrape multiple pages with Playwright in Python, the async API gives you a way to run several page loads concurrently without spawning multiple browser processes. This article shows how to structure an async scraper that handles many URLs efficiently using python playwright async web scraping multiple pages as the core pattern.
Why Use Playwright's Async API for Scraping Multiple Pages
Playwright's synchronous API blocks the Python thread while waiting for browser events, which makes sequential scraping slow when you have many URLs. The async API, built on asyncio, allows you to interleave I/O operations such as page navigation, network requests, and DOM queries. Instead of waiting for one page to finish before starting the next, you can issue multiple page loads and let the event loop manage them as they complete.
This is particularly useful for scraping tasks because the bottleneck is almost always network latency and page rendering, not CPU work. By using async and await, you keep the Python process responsive and can scale to dozens or hundreds of pages with a single browser instance.
Setting Up the Async Playwright Environment
The async API is available in the playwright package. You install it with pip install playwright and then install the browsers with python -m playwright install. In your code, you import async_playwright and use it as an async context manager.
from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) # ... your scraping logic await browser.close()
The async_playwright context manager ensures that the Playwright driver is properly started and stopped. The browser launch is also async because it starts a browser process and waits for it to be ready. You can choose between chromium, firefox, or webkit; Chromium is the most common for scraping.
Scraping Multiple Pages Sequentially with Async
Before introducing concurrency, it helps to see a simple async function that scrapes one page. This function takes a URL, opens a new page, navigates to the URL, extracts the content you need, and closes the page.
async def scrape_page(browser, url): page = await browser.new_page() await page.goto(url, wait_until="domcontentloaded") title = await page.title() content = await page.content() await page.close() return {"url": url, "title": title, "content": content}
If you call this function in a loop, each iteration waits for the previous page to finish before starting the next. That is sequential execution:
urls = ["https://example.com", "https://example.org", "https://example.net"] results = [] for url in urls: result = await scrape_page(browser, url) results.append(result)
This works, but it does not take advantage of the async nature. The total time is roughly the sum of all page loads. For many URLs, that becomes slow.
Scraping Multiple Pages Concurrently with asyncio.gather
The real benefit of the async API is the ability to run multiple scrape_page tasks concurrently. The standard way to do this in Python is asyncio.gather, which schedules multiple coroutines and waits for all of them to complete.
import asyncio async def main(): async with async_playwright() as p: browser = await p.chromium.launch(headless=True) urls = ["https://example.com", "https://example.org", "https://example.net"] tasks = [scrape_page(browser, url) for url in urls] results = await asyncio.gather(*tasks) await browser.close() print(results)
When you call scrape_page(browser, url), it returns a coroutine object. asyncio.gather schedules these coroutines on the event loop. The event loop interleaves their execution: while one page is waiting for a network response, another page can start its navigation. This can reduce the total time from the sum of all page loads to roughly the time of the slowest page, assuming the server and your network can handle the parallel requests.
One important detail: each coroutine creates its own page via browser.new_page(). Playwright allows multiple pages in the same browser context, and each page is independent. This is safe because Playwright's async API is designed for concurrent use.
Managing Browser Contexts and Pages for Concurrency
While you can create many pages from a single browser instance, each page shares the same browser context by default. That means cookies, localStorage, and other session data are shared. For scraping, this is often undesirable because you may want to isolate sessions or avoid cross-site tracking. Use a separate context per task when you need isolation.
async def scrape_page_with_context(p, url): context = await p.chromium.launch_persistent_context(user_data_dir="") # or context = await browser.new_context() page = await context.new_page() await page.goto(url) # ... await context.close()
Creating a new context for each URL is more expensive than creating a page, but it provides clean separation. If you do not need isolation, using pages from a single context is faster because context creation involves more setup. For most scraping tasks, a single context with multiple pages is sufficient, but be aware that cookies will persist across pages.
Another consideration is the number of concurrent pages. Opening hundreds of pages at once can overwhelm the browser and the target server. You should limit concurrency to a reasonable number, such as 5 or 10, using a semaphore.
sem = asyncio.Semaphore(5) async def bounded_scrape(browser, url): async with sem: return await scrape_page(browser, url)
This ensures that at most five pages are active at any moment, preventing resource exhaustion and reducing the chance of being blocked by the target site.
Handling Errors and Rate Limits in Concurrent Scraping
When you run many requests concurrently, errors become more likely. Network timeouts, HTTP 429 (Too Many Requests), or page crashes can happen. You need to handle these errors per task so that one failure does not cancel the entire gather call.
async def safe_scrape(browser, url): try: return await scrape_page(browser, url) except Exception as e: return {"url": url, "error": str(e)}
By catching exceptions inside the coroutine, you can collect partial results. If you want to retry failed pages, you can wrap the scraping logic in a retry loop with a delay.
async def scrape_with_retry(browser, url, retries=3): for attempt in range(retries): try: return await scrape_page(browser, url) except Exception: if attempt == retries - 1: raise await asyncio.sleep(2 ** attempt)
Respecting rate limits is crucial. Even with a semaphore, you might still send requests too quickly for some sites. A common approach is to add a small random delay between starting each task, or to use a token bucket. The semaphore controls concurrency, but you can also add a per-task delay before navigation.
async def rate_limited_scrape(browser, url): await asyncio.sleep(random.uniform(0.5, 1.5)) return await scrape_page(browser, url)
This reduces the request rate and makes the scraper less aggressive.
Performance Considerations and Resource Management
The main performance advantage of async scraping is reduced wall-clock time. However, you must balance concurrency with memory and CPU usage. Each page in Playwright consumes memory for the rendering engine. If you open too many pages simultaneously, the browser process can become slow or crash.
A good practice is to reuse pages when possible. Instead of creating a new page for each URL, you can create a pool of pages and assign URLs to them. This avoids the overhead of page creation and destruction. However, page reuse requires careful handling of state, since you must clear cookies or navigate to a blank page between uses.
Another performance factor is the wait_until condition in goto. Using "domcontentloaded" is faster than "load" because it waits only for the HTML to be parsed, not for all resources like images and stylesheets. For scraping, you often only need the DOM, so "domcontentloaded" is a good default.
Memory usage can be monitored by looking at the browser process. If you are scraping a large number of pages, consider closing pages after each task to free memory. The page.close() call in scrape_page does that. If you use a single context, the context remains, but pages are released.
Finally, be aware of the target server's load. Even with a semaphore, you can still send many requests in a short time. Use a conservative concurrency limit and add delays to avoid being blocked. The exact limit depends on the site's tolerance and your network, so start low and adjust based on observed behavior.
A Complete Example: Scraping a List of URLs
Here is a full script that combines the concepts: a semaphore for concurrency, error handling, and rate limiting.
import asyncio import random from playwright.async_api import async_playwright async def scrape_page(browser, url): page = await browser.new_page() try: await page.goto(url, wait_until="domcontentloaded") title = await page.title() content = await page.content() return {"url": url, "title": title, "content": content} finally: await page.close() async def bounded_scrape(browser, url, sem): async with sem: await asyncio.sleep(random.uniform(0.5, 1.5)) try: return await scrape_page(browser, url) except Exception as e: return {"url": url, "error": str(e)} async def main(): urls = [ "https://example.com", "https://example.org", "https://example.net", "https://example.edu", "https://example.co", ] sem = asyncio.Semaphore(3) async with async_playwright() as p: browser = await p.chromium.launch(headless=True) tasks = [bounded_scrape(browser, url, sem) for url in urls] results = await asyncio.gather(*tasks) await browser.close() for result in results: print(result) if __name__ == "__main__": asyncio.run(main())
This script limits concurrency to three pages at a time, adds a random delay before each navigation, and catches exceptions per URL. The finally block ensures the page is closed even if an error occurs. You can adjust the semaphore value and delay range based on the target site's behavior.
The async API is not a silver bullet for every scraping scenario. If your scraping logic is CPU-bound, such as heavy parsing, you might need to combine it with multiprocessing. But for most page-fetching tasks, async concurrency with Playwright provides a significant speedup while keeping the code readable and maintainable.