Post cover image

July 19, 2026

Simplifying Python Web Scraping with Playwright

Why Traditional Scraping Fails on Dynamic Content and SPAs, Playwright helps!

By Yatin Anuj Kashyap

10 min read

Simplifying Python Web Scraping with Playwright: Mastering Dynamic Content and Single-Page Applications

Section 1: Introduction — Why Traditional Scraping Fails on Dynamic Content and SPAs

Have you ever tried to scrape a modern website with the classic requests + BeautifulSoup combo, only to discover that the data you need simply isn't in the HTML response? You're not alone. The rise of JavaScript‑rendered pages and single‑page applications (SPAs) has turned traditional scraping into a moving target.

In the early days of web scraping, the workflow was almost mechanical: send an HTTP request, receive a static HTML document, parse it with BeautifulSoup, and — voilà — extract the data you wanted. Those days are largely behind us. Today, most sites rely on JavaScript to build their UI, and many adopt SPAs that load content on demand rather than delivering it all up front.

That shift introduces a whole new set of challenges:

  • Hidden or lazily‑loaded content that only appears after scripts run.
  • Infinite scroll and pagination driven by client‑side logic.
  • Client‑side routing that changes the URL without a full page reload.
  • Anti‑scraping defenses that detect and block non‑browser traffic.

Even seasoned scrapers can find themselves stumped by these obstacles.

But don't worry — we've got you covered. In this article we'll walk you through using Python Playwright to scrape any dynamic site with confidence. By the end, you'll have a solid grasp of the tools and techniques needed to tackle the most complex scraping scenarios.

In the sections that follow, we'll explore Playwright's strengths and how it handles dynamic content and SPAs:

  • Section 2: Setting up Playwright and understanding its architecture
  • Section 3: Handling dynamic content with Playwright
  • Section 4: Tackling single‑page applications with Playwright
  • Section 5: Putting it all together — a real‑world example of web scraping with Playwright

I'm ready to polish the section, but I need the full text of Section 2 (the body beneath the heading) in order to apply the edits. Could you please provide that content?

Section 2: Understanding Dynamic Content & Single‑Page Applications

Before we fire up Playwright, let's peek under the hood of the modern web. If you've ever opened the source of a React or Vue app and saw only an empty <div id="root"></div> plus a huge script tag, you've already encountered the star of this section: dynamic content.

In the classic model, the server delivers a complete HTML document. Your scraper downloads it, parses the DOM, and moves on. Today, that initial HTML is often just a skeleton. The real meat — product listings, comments, pricing tables — arrives later via AJAX (Asynchronous JavaScript and XML) or the Fetch API. The browser runs the JavaScript, mutates the DOM, and then renders the final visual output. What you see in DevTools (the Elements tab) is the rendered DOM; what requests.get() receives is the initial HTML. Frequently, those are two completely different documents.

This is the hallmark of Single‑Page Applications (SPAs). Instead of navigating between distinct URLs with full page reloads, SPAs swap views client‑side using the History API. That shift introduces patterns that break traditional scrapers instantly:

  • Lazy Loading & Infinite Scroll — Content appears in the DOM only after the user scrolls into view.
  • Client‑Side Routing — "Pages" are merely JavaScript state changes; no new HTTP request for a full document occurs.
  • Hidden API Calls — The data you need often lives in JSON responses you'll find in the Network tab (XHR/Fetch), not in the HTML at all.

Just how prevalent is this? According to BuiltWith's 2023 data, over 70 % of the top 500 websites rely heavily on JavaScript to render core content. If you're only parsing static HTML, you're effectively blind to the majority of the modern web.

That's precisely why we need a headless browser. We don't just want the source code; we need a runtime that executes JavaScript, waits for network‑idle events, and exposes the fully realized DOM. Enter Playwright.

Section 3: Setting Up Playwright for Python (LSI: Playwright Python installation guide)

3.1 Installing Playwright for Python

┌──────┬────────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Step │ Command                            │ What it does                                                                                                                                       │
├──────┼────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1️⃣   │ pip install playwright             │ Pulls the Python client and its dependencies from PyPI.                                                                                            │
│ 2️⃣   │ playwright install                 │ Downloads the bundled Chromium, Firefox, and WebKit binaries. Use playwright install chromium to fetch only Chromium and keep the footprint small. │
│ 3️⃣   │ (optional) playwright install-deps │ Installs OS‑level libraries required on Linux headless environments (e.g., libnss3, libatk-bridge2.0-0).                                           │
└──────┴────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Tip: Run these commands inside a virtual environment so you don't pollute the global site‑packages. Our Python virtual‑env guide walks you through a quick setup.

3.2 Sync vs. Async API

Playwright offers two parallel programming models:

┌──────────────────┬────────────────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│ Feature          │ Synchronous (playwright.syncapi)           │ Asynchronous (playwright.asyncapi)                          │
├──────────────────┼────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Typical use‑case │ Simple scripts, quick demos                │ High‑throughput crawlers, concurrent page handling          │
│ Performance      │ One page at a time; blocking I/O           │ Event‑loop driven; can drive dozens of pages simultaneously │
│ Code style       │ Straight‑line, with syncplaywright() as p: │ async with asyncplaywright() as p: + await calls            │
│ Learning curve   │ Lower for Python newcomers                 │ Requires asyncio familiarity                                │
└──────────────────┴────────────────────────────────────────────┴─────────────────────────────────────────────────────────────┘

Recommendation: If you're scraping more than a single page, go with the async API. The extra overhead is negligible, but the scalability boost is massive — especially when you need to paginate or poll several SPAs in parallel.

3.3 Minimal Async Script

import asyncio
from playwright.async_api import async_playwright
async def main():
    async with async_playwright() as p:
        # Launch Chromium in headless mode
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto("https://example.com")
        # Wait for network to be idle (helps with dynamic content)
        await page.wait_for_load_state("networkidle")
        print(await page.title())
        await browser.close()
asyncio.run(main())

Key points

  • headless=True keeps the UI off‑screen, which is ideal for CI/CD pipelines.
  • wait_for_load_state("networkidle") pauses until there are no pending network requests for at least 500 ms, giving JavaScript a chance to finish rendering.

3.4 Debugging with Traces

When a page looks empty or data is missing, turn on Playwright's built‑in tracing:

await context.tracing.start(screenshots=True, snapshots=True)
# … perform actions …
await context.tracing.stop(path="trace.zip")

Open trace.zip with the Playwright Trace Viewer (playwright show-trace trace.zip) to replay every step, inspect network payloads, and spot where the DOM diverges from expectations.

3.5 Quick Checklist

  • [ ] pip install playwright executed inside a venv.
  • [ ] Browser binaries installed (playwright install).
  • [ ] Async API selected for concurrent workloads.
  • [ ] Minimal script runs and prints the page title.
  • [ ] Tracing enabled for the first few pages to validate DOM rendering.

With the environment ready and the basic launch script verified, you're set to move on to Section 4, where we'll tackle lazy‑loaded content and infinite scroll in SPAs.

Section 4: Core Scraping Techniques with Playwright

Selectors are the backbone of reliable scraping. In SPAs, class names often hash (e.g., .css-1x2y3z), so it's best to target data attributes such as data-testid="product-card" or semantic ARIA roles. When those aren't available, chain a stable parent with a text match, for example: page.locator('section.products >> text="Add to Cart"'). Playwright's auto‑waiting handles most timing issues, but adding explicit waits can eliminate flakiness.

4.1 Waiting Strategies That Actually Work

Avoid arbitrary sleep() calls. Instead:

  • Use wait_for_load_state("domcontentloaded") to wait for the initial HTML.
  • Follow with wait_for_load_state("networkidle") to let XHR/fetch requests settle.
  • For individual widgets, wait_for_selector with state="visible" or state="attached" is surgical:
import asyncio
from playwright.async_api import async_playwright, TimeoutError as PWTimeout
async def scrape_products(url: str, max_items: int = 50) -> list[dict]:
    results = []
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        context = await browser.new_context(
            user_agent="Mozilla/5.0 (compatible; PlaywrightBot/1.0)"
        )
        page = await context.new_page()
        # 1. Navigate & wait for the shell
        await page.goto(url, wait_until="domcontentloaded")
        # 2. Wait for the product‑grid API response (pro tip: see 4.4)
        await page.wait_for_response(
            lambda r: "/api/products" in r.url and r.status == 200
        )
        # 3. Ensure cards are rendered
        await page.wait_for_selector('[data-testid="product-card"]', state="visible")
        # 4. Infinite‑scroll loop
        previous_height = 0
        while len(results) < max_items:
            # Extract current batch
            cards = page.locator('[data-testid="product-card"]')
            count = await cards.count()
            for i in range(len(results), min(count, max_items)):
                card = cards.nth(i)
                # Use evaluate for structured data, avoiding innerHTML parsing
                data = await card.evaluate("""el => ({
                    id: el.dataset.productId,
                    name: el.querySelector('[data-testid="product-name"]').innerText,
                    price: el.querySelector('[data-testid="product-price"]').innerText,
                    inStock: !el.querySelector('[data-testid="out-of-stock"]')
                })""")
                results.append(data)
            if len(results) >= max_items:
                break
            # Scroll to bottom and wait for new content
            await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
            try:
                # Wait for network quiet *or* height change
                await page.wait_for_load_state("networkidle", timeout=3000)
            except PWTimeout:
                pass  # Network didn't settle – maybe no more pages
            new_height = await page.evaluate("document.body.scrollHeight")
            if new_height == previous_height:
                break  # No new content loaded
            previous_height = new_height
        await browser.close()
    return results
if __name__ == "__main__":
    products = asyncio.run(scrape_products("https://demo-spa.example.com/shop"))
    print(f"Scraped {len(products)} products")
    for p in products[:3]:
        print(p)

Why this works

  • Line 14: wait_for_response captures the JSON payload before the DOM updates, giving you pristine data without having to parse HTML.
  • Lines 28‑35: page.evaluate runs inside the browser context and returns a serialized JSON object—faster and safer than fiddling with innerHTML or regexes.
  • Lines 42‑52: The scroll loop checks both document.body.scrollHeight and network idle, preventing endless loops on "Load More" buttons that don't trigger new requests.

Pro Tip: If the SPA exposes a GraphQL or REST endpoint directly, skip the DOM entirely. page.wait_for_response lets you intercept the exact JSON the UI consumes, turning a brittle scrape into a clean, typed API call.

Section 5: Advanced Playwright Hacks for Real‑World Challenges

Advanced Playwright Hacks for Real‑World Challenges

You've mastered the basics — now let's tackle the messy problems that make scrapers fail in production: login walls, bot detectors, infinite scrolls, and file downloads.

Bypassing Authentication Without the Headache

Most gated content sits behind a login form. Instead of typing credentials on every run, log in once, persist the storage state, and reuse it:

# 1. Run once interactively to save auth
await page.goto("https://app.example.com/login")
await page.fill('#email', 'bot@example.com')
await page.fill('#password', os.getenv("APP_PASS"))
await page.click('button[type=submit]')
await page.wait_for_url("**/dashboard")
await context.storage_state(path="auth.json")
# 2. Subsequent runs load the cookies/localStorage instantly
context = await browser.new_context(storage_state="auth.json")

For two‑factor authentication, generate the TOTP code on the fly (pyotp + page.fill) instead of waiting for an SMS. If the site relies on WebAuthn or hardware keys, launch a persistent context (user_data_dir) so the authenticator remains enrolled.

Dodging Anti‑Bot Defenses

Headless Chrome gives itself away with navigator.webdriver=true and a missing chrome.runtime. The community‑maintained playwright‑stealth plugin masks these fingerprints in a single line:

from playwright_stealth import stealth_async
await stealth_async(page)

Combine it with:

  • Rotating residential proxies — rotate per context, not per request.
  • Human‑like timing — await page.wait_for_timeout(random.uniform(800, 2200)) between actions.
  • Realistic headers — set Accept-Language, Sec-CH-UA, and a fresh User-Agent per session via context.set_extra_http_headers.

Pagination: Intercept, Don't Click

Clicking "Load More" is slow and flaky. Open DevTools → Network → XHR/Fetch, locate the GraphQL or REST call that returns the next page, and replay it directly:

async with page.expect_response("**/api/articles*") as resp_info:
    await page.evaluate("window.loadNextPage()")  # triggers the fetch
data = (await resp_info.value).json()

You receive structured JSON, skip the rendering overhead, and avoid layout shifts that break selectors.

Downloading Files in Headless Mode

Playwright's page.expect_download handles the Content‑Disposition header automatically—even in headless mode:

async with page.expect_download() as dl:
    await page.click('a[href$=".csv"]')
download = await dl.value
await download.save_as(f"data/{download.suggested_filename}")

If you need to download several file types, set accept_downloads=True when creating the context.

Case Study: GraphQL‑Powered News Site

A major outlet serves a thin HTML shell and then hydrates articles via a batched GraphQL POST to /graphql.

Strategy

  1. page.goto the section front page.
  2. wait_for_response for the GraphQL payload containing edges.node{title,url,publishedAt}.
  3. Parse the JSON — no DOM parsing required.
  4. Paginate by incrementing the cursor variable in the POST body and replaying the request with page.request.post.

Result: ~1,200 articles / minute versus ~200 when rendering the full page.

Tool Comparison at a Glance

┌──────────────────────┬──────────────────────────────────────┬────────────────────────────────┬────────────────────────────────────┐
│ Feature              │ Playwright                           │ Selenium                       │ Requests‑HTML                      │
├──────────────────────┼──────────────────────────────────────┼────────────────────────────────┼────────────────────────────────────┤
│ JS/SPA Support       │ Native (CDP)                         │ Native (WebDriver)             │ Limited (pyppeteer backend)        │
│ Auto‑wait / Retry    │ Built‑in                             │ Manual WebDriverWait           │ Manual                             │
│ Network Interception │ First‑class (route, waitforresponse) │ Via CDP/DevTools (verbose)     │ Basic                              │
│ Auth Persistence     │ storagestate JSON                    │ Cookie jar / profile directory │ Manual cookie jar                  │
│ Stealth/Evasion      │ playwright‑stealth plugin            │ undetected‑chromedriver        │ Weak                               │
│ File Downloads       │ expectdownload (headless OK)         │ Flaky in headless              │ requests stream (no JS)            │
│ Learning Curve       │ Medium                               │ High                           │ Low (if you already know requests) │
└──────────────────────┴──────────────────────────────────────┴────────────────────────────────┴────────────────────────────────────┘

Bottom line: For modern SPAs and GraphQL‑heavy sites, Playwright's interception + stealth combo wins. Selenium still shines for legacy enterprise apps that require a full WebDriver grid. Requests‑HTML is a quick hack for mostly static pages with light JavaScript.

Next up: a production‑ready checklist — logging, monitoring, and deploying your scraper to the cloud without getting banned.

Section 6: Frequently Asked Questions (FAQ)

Can Playwright scrape sites that block headless browsers? Playwright is harder to detect than many other automation tools, especially when you add the playwright-stealth plugin. Still, some sites employ aggressive fingerprinting and may block headless sessions. To improve success rates you can:

  • Run the browser in non‑headless mode.
  • Rotate user‑agent strings and other fingerprinting attributes.
  • Randomize proxy IPs and viewport sizes.

For most modern single‑page applications (SPAs) and GraphQL‑heavy sites, the combination of request interception and stealth techniques is usually sufficient.

Is async Playwright necessary for small projects? Async Playwright isn't a strict requirement for tiny scripts, but it's a best practice. Asynchronous code lets you:

  • Issue multiple page requests in parallel.
  • Keep the event loop busy while waiting for network I/O.

Even a modest scraper benefits from the speed boost and scalability that async provides, and the codebase stays ready for growth.

How do I extract data from WebSockets used by SPAs? You can tap into WebSocket traffic with Playwright's event API:

# Wait for the WebSocket to open
ws = await page.wait_for_event("websocket")
# Listen for incoming messages
ws.on("framesreceived", lambda frame: print(frame.payload))

If you need to run custom JavaScript inside the page, page.evaluate works as well:

data = await page.evaluate("""
    () => {
        const ws = window.myWebSocket;   // your WebSocket reference
        return ws ? ws.lastMessage : null;
    }
""")

These patterns let you capture real‑time data without resorting to network proxies.

What's the best way to store scraped data from Playwright? The optimal storage solution depends on data size and downstream use:

┌────────────────────────┬────────────────────────────────────────────────────────────────────┐
│ Project size           │ Recommended format / storage                                       │
├────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ Small (≤ 10 k rows)     │ CSV or JSON files on disk                                          │
│ Medium (10 k–1 M rows)  │ Relational DB (PostgreSQL, MySQL) or document store (MongoDB)      │
│ Large / archival       │ Cloud object storage (AWS S3, Google Cloud Storage) or a data lake │
└────────────────────────┴────────────────────────────────────────────────────────────────────┘

Pick a format that matches your analysis pipeline — e.g., CSV for quick Excel checks, a database for complex queries, or S3 for long‑term archiving.

How does Playwright handle CAPTCHAs? Playwright itself does not solve CAPTCHAs out of the box. Common workarounds include:

  • Integrating third‑party solvers such as 2Captcha, Anti‑Captcha, or the captcha-solver Python package.
  • Using the playwright-extra ecosystem with a stealth plugin that can reduce the chance of encountering a CAPTCHA.
  • Delegating the challenge to a human‑in‑the‑loop service (e.g., CAPTCHA‑Enterprise, hCaptcha verification APIs).

Whichever method you choose, keep the solution compliant with the target site's terms of service.

Section 7: Conclusion & Key Takeaways

Wrapping up our guide to simplifying Python web scraping with Playwright, let's revisit the three biggest advantages this tool brings to dynamic scraping:

  • Effortless handling of dynamic content — Playwright's request interception and stealth capabilities let you scrape modern single‑page applications and GraphQL‑heavy sites with minimal fuss.
  • Concurrent requests with async code — Using Playwright's async API lets you fire multiple requests in parallel, dramatically cutting down runtime.
  • Robust CAPTCHA handling — Although Playwright doesn't ship a native CAPTCHA solver, you can integrate third‑party services or libraries to bypass most challenges.

Actionable next steps

  1. Install Playwright
pip install playwright
  1. Pick the right execution mode — Decide whether to run headless, headful, or a hybrid approach based on your project's needs.
  2. Leverage async — Write your scraper with async/await to maximize concurrency and speed.
  3. Debug with traces — Add the --trace flag (or browser_context.tracing.start()) to capture detailed execution traces and pinpoint issues quickly.

As more sites migrate to SPAs, scraping will only get tougher — but with Playwright in your toolbox, you're ready for even the most complex scenarios. Check out our ready‑to‑run GitHub repo, subscribe for more Python automation tutorials, and share your Playwright success stories in the comments below!

None