Playwright Web Scraping for Dynamic Websites
Playwright is a strong starting point for scraping dynamic websites when JavaScript renders the content you need or you must interact with a page to reach it. If an accessible response already contains the data, an HTTP request is simpler. As scraping work grows, browser operations and changing page layouts may make managed infrastructure or an agent worth evaluating.
Playwright is right when:
- You're scraping a small number of sites at a manageable volume
- You can handle any required authentication and session renewal
- You have time to maintain selectors when sites update
- You can run the browser sessions you need with your available infrastructure
Consider managed infrastructure or an agent when:
- You need more concurrent sessions than you want to operate yourself
- You must manage authentication and session renewal across many sites
- Layout changes repeatedly break your scripts
- Access restrictions require site-specific investigation and maintenance
When to Use Playwright for Scraping Dynamic Websites
Some dynamic websites return initial HTML without the data you need, then add it with JavaScript. Playwright runs a browser and lets you wait for the content before reading the DOM. For a JavaScript-rendered page, wait for the content you need rather than assuming the page has finished loading.
For example, this Python code waits for a product card before reading its text:
from playwright.sync_api import sync_playwright
with sync_playwright() as playwright:
browser = playwright.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/products")
cards = page.locator(".product-card")
cards.first.wait_for(state="visible")
print(cards.all_text_contents())
browser.close()Replace the URL and selector with ones verified on your target page. Waiting for the first card confirms that content appeared, but it does not guarantee that every card has loaded.
Before running the example, install the Playwright Python library and Chromium:
pip install playwright
playwright install chromiumScrape an Infinite-Scroll Page
This example handles infinite-scroll pages that append new product cards to the DOM as the main document scrolls. It makes up to five scroll attempts and waits for the card count to increase. The approach may need modification when a site scrolls within a container, uses virtualized lists, or loads additional content through a button.
from playwright.sync_api import TimeoutError as PlaywrightTimeoutError, sync_playwright
with sync_playwright() as playwright:
browser = playwright.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/products")
cards = page.locator(".product-card")
cards.first.wait_for(state="visible")
for _ in range(5):
previous_count = cards.count()
page.evaluate(
"window.scrollTo(0, document.scrollingElement.scrollHeight)"
)
try:
page.wait_for_function(
"previous => document.querySelectorAll('.product-card').length > previous",
arg=previous_count,
timeout=5000,
)
except PlaywrightTimeoutError:
break
print(cards.all_text_contents())
browser.close()
A timeout does not necessarily mean the end of the list. Check for a site-specific end-of-results indicator before treating the extraction as complete. Pages that virtualize their lists may also remove earlier items from the DOM, so collecting only the final set of visible cards can produce incomplete results.
Scrape Link-Based Pagination
This example visits at most five pages, following each page’s next link rather than guessing a URL pattern:
from urllib.parse import urljoin
from playwright.sync_api import sync_playwright
results = []
visited_urls = set()
with sync_playwright() as playwright:
browser = playwright.chromium.launch()
page = browser.new_page()
page.goto("https://example.com/products")
for _ in range(5):
if page.url in visited_urls:
break
visited_urls.add(page.url)
cards = page.locator(".product-card")
cards.first.wait_for(state="visible")
results.extend(cards.all_text_contents())
next_link = page.locator("a.next-page")
if next_link.count() == 0:
break
href = next_link.first.get_attribute("href")
if not href:
break
next_url = urljoin(page.url, href)
if next_url in visited_urls:
break
page.goto(next_url)
print(results)
browser.close()This example follows link-based pagination and collects results from up to five pages. Replace the selectors with those used by the target website. Some sites use JavaScript buttons, disabled navigation controls, or client-side pagination instead of standard links. Verify the final-page condition and check for duplicate products before using the collected results.
When extraction fails: Check whether selectors have changed, content is still loading, or a consent dialog is blocking the page. Compare screenshots, response logs, and Playwright traces when investigating failures. If the script works locally but fails in production, also compare browser versions, resource limits, and the responses returned in each environment. Do not assume every timeout is caused by bot detection.
Playwright isn't the only option for JavaScript-rendered websites. Selenium provides browser automation with support for languages including Java, Python, C#, and Ruby. Crawl4AI focuses on crawling and extracting content, including LLM-ready Markdown and structured data. The better choice depends on whether your workflow requires browser interactions, structured content extraction, or integration with an existing automation stack. For a deeper comparison, see our Playwright vs Selenium guide
Playwright Scraping Limitations: Authentication, Selectors, and Scale
Rendering the page is only part of a scraper that must run repeatedly. Authentication, selector changes, and browser operations add ongoing work.
1. Authentication and session state
Scraping authenticated portals, such as supplier pricing pages or internal dashboards, requires managing authorized sessions across runs. Playwright can save cookies and local storage through storage_state(), with optional support for IndexedDB. However, session storage requires separate handling, and saved authentication can expire or become invalid. Persistent browser profiles offer another approach, but login flows, session renewal, and credential security still require maintenance.
2. Selector maintenance
A page.locator() call depends on how you identify an element, so a site change can break a scraper if it changes that element or its identifying attributes. Prefer stable attributes or accessible roles where available, and expect to review locators when target pages change. Across many sites, that maintenance can take more time than the initial script.
3. Concurrency economics
Browser concurrency consumes memory and CPU, but resource use varies with the pages, browser configuration, and whether tasks share a browser through separate contexts. Measure your workload before choosing how many sessions to run per host. Higher concurrency may call for browser reuse, limits on active tasks, and infrastructure to recover failed sessions.
4. Access controls
A site can change its rate limits, challenges, or other access controls without notice. A script can stop working after a target changes its access controls, even if the script itself has not changed. Managing this requires either ongoing tuning or a solution that handles infrastructure-level execution requirements for you.
Alternatives to Playwright for Dynamic Web Scraping
Consider another approach when maintaining browsers, sessions, and site-specific scripts takes more work than the scraping task warrants. You may still use Playwright inside managed infrastructure, so the decision is not always whether to stop using it.
At that point, the options are:
1. Managed scraping services (Zyte, Scrapfly, Firecrawl).
These services can retrieve and render web content without requiring you to operate local browser infrastructure. Their extraction interfaces, supported interactions, and session-management capabilities differ, so evaluate the specific endpoint against your workflow.
2. AI-powered extraction (Firecrawl /extract, Crawl4AI with LLM-based extraction).
These tools can extract structured information without requiring developers to define every CSS selector. Firecrawl also offers Agent and Interact capabilities for more autonomous workflows. AI-powered extraction can reduce selector maintenance, but returned values still need validation for completeness and accuracy.
3. AI web agents (TinyAgent)
Instead of specifying every selector and navigation step, you define the task and let the agent determine how to proceed. TinyFish Web Agent supports multi-step browser workflows, authenticated sessions, and structured results. Evaluate agents using actual task-completion accuracy, execution cost, and the complexity of the websites you need to operate.
Compare the time spent maintaining your scrapers with the time and cost of using a managed browser service or agent for the same tasks.
Playwright vs Managed Browsers vs AI Web Agents
The key difference is how much browser infrastructure and workflow logic your team manages.
| Approach | What you control | What you maintain | Best suited for |
|---|---|---|---|
| Playwright | Browser interactions, selectors, navigation, and extraction logic | Browser execution, scripts, sessions, and infrastructure | Stable workflows requiring precise control |
| Managed browsers | Browser interactions through interfaces such as Playwright or CDP | Scripts, selectors, and workflow logic; the provider manages browser infrastructure | Existing automation scripts that need managed execution |
| AI web agents | Goals, task constraints, and expected outputs | Task validation, access configuration, and workflow integration | Multi-step tasks where navigation paths change |
Related Guides to Web Scraping and Browser Automation
AI disclosure
Content on this website may be created or refined with the assistance of AI tools and is subject to human editorial review.
