How Bot Detection Works: TLS Fingerprints & WebDriver Flags

Modern Web Application Firewalls (WAFs) like Cloudflare, Akamai, and CloudFront no longer rely solely on simple IP rate limiting or User-Agent string filtering. They inspect connection parameters at the network (TLS/TCP) and runtime (DOM/JS environment) layers. If a crawler's client signature doesn't match a legitimate browser, the request is immediately blocked. This guide breaks down the architecture required to build a stealth web scraper that bypasses TLS fingerprinting and browser detection indicators.

1. Reading TLS JA3/JA4 Signatures

During the TLS handshake, before HTTP data is sent, the client transmits a TLS "Client Hello" message containing its cipher suites, extensions, and protocol versions. Firewalls hash this packet sequence to construct a unique fingerprint (known as a JA3 or JA4 signature).

Standard language libraries (like Python's urllib or Node's http) produce distinct hashes. To bypass this, we use curl_cffi, which compiles low-level curl features to mimic the exact cipher order, HTTP/2 settings, and TLS handshakes of real browsers:

from curl_cffi import requests

# Emulating Chrome's exact TLS handshakes, ALPN protocols, and extensions
session = requests.Session()
response = session.get(
    "https://target-protected-website.com/api/data",
    impersonate="chrome120"
)
print(response.status_code)

This shows how a client can present a browser-like TLS handshake at the connection layer without spawning a heavy browser engine — useful when testing whether your own defenses catch it.

2. How Sites Read WebDriver Fingerprints

When scraping dynamic content that requires JavaScript execution, standard automation tools like Playwright or Puppeteer must be used. However, these tools leave trace markers inside the Javascript runtime that are parsed by anti-bot scripts.

One primary check is the navigator.webdriver property, which defaults to true under automation. Anti-bot engines check this property along with other hardware and browser features:

• navigator.webdriver: Returns true when controlled by ChromeDriver.
• Chrome Plugin Mocking: Legitimate browsers have default plugins in their lists; headless browsers have none.
• WebGL Render Profiles: Headless browsers report default software renderer cards instead of physical GPU profiles (e.g. Intel/Nvidia).

To see how these markers work, we override DOM properties on page initialization using scripts (or specialized automation engines like Patchright):

# Python Playwright Initialization Override Example
async def apply_stealth_hooks(page):
    # Hide the automation WebDriver flag
    await page.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
    """)
    # Mock realistic hardware memory and processor counts
    await page.add_init_script("""
    Object.defineProperty(navigator, 'deviceMemory', { get: () => 8 });
    Object.defineProperty(navigator, 'hardwareConcurrency', { get: () => 4 });
  """)

# Usage: call it on a FRESH page, BEFORE navigation
# page = await browser.new_page()
# await apply_stealth_hooks(page)
# await page.goto("https://target-protected-website.com/api/data")

3. Testing HTTP/2 Fingerprint Defenses

Even if your scraper uses the correct User-Agent, firewalls analyze the HTTP/2 frame settings sent on TCP connection initialization. Legitimate Chrome browsers send specific settings for header table sizes, frame limits, and initial stream window sizes.

By using low-level client engines that handle HTTP/2 frame initialization (such as curl_cffi or Go's utls), we match the Chrome HTTP/2 frame signatures exactly, preventing firewalls from flagging the request as a bot.