How TLS Fingerprint is Used to Block Web Scrapers?

One of the most subtle and lesser-known methods for detecting and fingerprinting web scraper traffic is Transport Layer Security (TLS) analysis. Every HTTPS connection must establish a secure handshake, and the manner in which this handshake is conducted can result in fingerprinting and web scraping prevention.

In this article, we’ll examine how TLS can reveal that the connecting client is a web scraper and how it can be utilized to establish a fingerprint for tracking clients across the internet.

5 Tools to Scrape Without Blocking and How it All Works
Tutorial on how to avoid web scraper blocking. What is javascript and TLS (JA3) fingerprinting and what role request headers play in blocking.

Key Takeaways

Overcome TLS fingerprint detection by utilizing browser automation tools and custom TLS configurations that replicate authentic browser handshakes to prevent scraper identification.

• Use browser automation tools like Selenium, Playwright, and Puppeteer for authentic TLS fingerprints
• Implement JA3 fingerprinting analysis to understand and modify TLS handshake characteristics
• Configure custom TLS settings including cipher suites, extensions, and protocol versions
• Upgrade to TLS 1.3 for better protection against fingerprinting compared to TLS 1.2
• Use specialized libraries like utls, CycleTLS, and curl-impersonate for TLS fingerprint spoofing
• Leverage proxy services with TLS fingerprint rotation and spoofing capabilities

What Is TLS?

Transport Security Layer is the foundation of all HTTPS connections. It enables end-to-end encrypted communication between client and server.

In web scraping contexts, we typically don’t concern ourselves with whether a website uses HTTP or HTTPS connections, as this doesn’t impact our data collection methodology. However, an emerging fingerprinting technology targets this connection phase to both fingerprint users for tracking purposes and block web scrapers.

TLS Fingerprinting

TLS is a complex protocol, and while we don’t need comprehensive understanding to identify our challenges, some fundamentals will be helpful. Let’s examine a brief TLS overview to understand its role in fingerprinting.

At the start of every HTTPS connection, the client and server must acknowledge each other and negotiate how the connection will be secured. This process is called the “Client Hello” handshake. The data structure appears as follows:TLS handshake analysis in Wireshark. Marked points are used in fingerprinting.

This contains substantial data, and here begins the identification challenge: which handshake values can differ across various HTTP clients like web browsers or programming libraries?

Initially, note that multiple TLS Versions exist: typically either 1.2 or 1.3 (the most recent).
This version dictates the remaining handshake data. TLS 1.3 offers additional optimizations and reduced data, making it simpler to secure, but whether using 1.2 or 1.3, our objective remains consistent – appear like an authentic web browser. In practice, we must secure multiple versions since some websites don’t yet support TLS 1.3.

Furthermore, we have the critical field: Cipher Suites.
This field contains a list of supported encryption algorithms by the negotiating parties. This list is priority-ordered, and both parties agree on the first matching value.
Therefore, we must ensure our HTTP client list matches that of a standard web browser, including the sequence.

Similar to the Cipher Suites list, we have the Enabled Extensions list.
These extensions indicate client-supported features and metadata like server domain name. Like Cipher Suites, we need these values and their order to match those of a standard web browser.

JA3 Fingerprint

As demonstrated, several values can vary significantly across clients. The JA3 fingerprint technique is commonly employed, which essentially creates a string from the varying values:

TLSVersion,
Ciphers,
Extensions,
support_groups(previously EllipticCurves),
EllipticCurvePointFormats,

Each value is separated by a , and array values are separated by a -.

For example, this Chrome web browser profile on Linux:

Handshake Type: Client Hello (1)
Length: 508
Version: TLS 1.2 (0x0303)  #1 (note that 0x0303 is hex for 771)
Cipher Suites Length: 32
Cipher Suites (16 suites)
    #2.1 Cipher Suite: Reserved (GREASE) (0x1a1a) 
    #2.2 Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
    #2.3 Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
    #2.4 Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
    #2.5 Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
    #2.6 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
    #2.7 Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
    #2.8 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
    #2.9 Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
    #2.10 Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
    #2.11 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
    #2.12 Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
    #2.13 Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
    #2.14 Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
    #2.15 Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
    #2.16 Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Extensions Length: 403
Extension: Reserved (GREASE) (len=0)
    #3.1  Type: Reserved (GREASE) (56026)
Extension: server_name (len=16)
    #3.2  Type: server_name (0)
Extension: extended_master_secret (len=0)
    #3.3  Type: extended_master_secret (23)
Extension: renegotiation_info (len=1)
    #3.4  Type: renegotiation_info (65281)
Extension: supported_groups (len=10)
    #3.5  Type: supported_groups (10)
    Supported Groups (4 groups)
        #4.1  Supported Group: Reserved (GREASE) (0x7a7a)
        #4.2  Supported Group: x25519 (0x001d)
        #4.3  Supported Group: secp256r1 (0x0017)
        #4.4  Supported Group: secp384r1 (0x0018)
Extension: ec_point_formats (len=2)
    #3.6  Type: ec_point_formats (11)
    Elliptic curves point formats (1)
        #5    EC point format: uncompressed (0)
Extension: session_ticket (len=0)
    #3.7  Type: session_ticket (35)
Extension: application_layer_protocol_negotiation (len=14)
    #3.8  Type: application_layer_protocol_negotiation (16)
Extension: status_request (len=5)
    #3.9  Type: status_request (5)
Extension: signature_algorithms (len=18)
    #3.10  Type: signature_algorithms (13)
Extension: signed_certificate_timestamp (len=0)
    #3.11  Type: signed_certificate_timestamp (18)
Extension: key_share (len=43)
    #3.12  Type: key_share (51)
Extension: psk_key_exchange_modes (len=2)
    #3.13  Type: psk_key_exchange_modes (45)
Extension: supported_versions (len=7)
    #3.14  Type: supported_versions (43)
Extension: compress_certificate (len=3)
    #3.15  Type: compress_certificate (27)
Extension: application_settings (len=5)
    #3.16  Type: application_settings (17513)
Extension: Reserved (GREASE) (len=1)
    #3.17  Type: Reserved (GREASE) (27242)
Extension: padding (len=44)
    #3.18  Type: padding (21)
Extension: pre_shared_key (len=156)
    #3.19 Type: pre_shared_key (41)

Would produce a fingerprint of:

771,6682-4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,56026-0-23-65281-10-11-35-16-5-13-18-51-45-43-27-17513-27242-21-41,31354-29-23-24,0

JA3 fingerprints are frequently md5 hashed to reduce fingerprint length:

dbe0907495f5e986a232e2405a67bed1

How To Read TLS Data?

The most common method for observing TLS handshakes is using the Wireshark packet analyzer:Wireshark QT edition on Linux

Using the tls filter, we can easily observe TLS handshakes when submitting requests in web browsers or web scraper scripts. Look for the “Client Hello” message, which represents the initial handshake step.

Wireshark can even calculate the JA3 fingerprint automatically:

To test JA3 fingerprints, we created an open Webparsers JA3 tool which simplifies HTTP client fingerprint testing.

For instance, here are the results from the requests library in Python:

import requests
import json
print(json.dumps(requests.get("https://tools.scrapfly.io/api/fp/ja3?extended=1").json())
{
  "digest": "8d9f7747675e24454cd9b7ed35c58707",
  "ja3": "771,4866-4867-4865-49196-49200-49195-49199-52393-52392-159-158-52394-49327-49325-49326-49324-49188-49192-49187-49191-49162-49172-49161-49171-49315-49311-49314-49310-107-103-57-51-157-156-49313-49309-49312-49308-61-60-53-47-255,0-11-10-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
  "tls": {
    "version": "0x303 - TLS 1.2",
    "ciphers": [
      "TLS_AES_256_GCM_SHA384",
      "TLS_CHACHA20_POLY1305_SHA256",
      "TLS_AES_128_GCM_SHA256",
      "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
      "..."
    ],
    "curves": [
      "X25519 (29)",
      "secp256r1 (23)",
      "X448 (30)",
      "secp521r1 (25)",
      "secp384r1 (24)"
    ],
    "extensions": [
      "server_name (0) (IANA)",
      "ec_point_formats (11) (IANA)",
      "supported_groups (10) (IANA)",
      "application_layer_protocol_negotiation (16) (IANA)",
      "encrypt_then_mac (22) (IANA)",
      "extended_master_secret (23) (IANA)",
      "post_handshake_auth (49) (IANA)",
      "signature_algorithms (13) (IANA)",
      "supported_versions (43) (IANA)",
      "psk_key_exchange_modes (45) (IANA)",
      "key_share (51) (IANA)",
      "padding (21) (IANA)"
    ],
    "points": [
      "0",
      "1",
      "2"
    ],
    "protocols": [
      "http/1.1"
    ],
    "versions": [
      "0x303 - TLS 1.2",
      "0x302 - TLS 1.1",
      "0x301 - TLS 1.0",
      "0x300 - SSL 3.0"
    ]
  }
}

This bears little resemblance to Chrome or Firefox – indicating these Python web scrapers would be readily identifiable! Let’s explore how we might address this issue.

How Does TLS Fingerprinting Lead To Blocking?

When blocking web scrapers, the primary objective is difference detection – determining whether this client differs from a typical web browser.

We can observe that the JA3 fingerprint algorithm considers relatively few variables, meaning there are comparatively few unique fingerprint possibilities, making it straightforward to create whitelist and blacklist databases.

Numerous public JA3 fingerprint databases exist, such as: ja3er.com, ja3.zone. These databases enable fingerprint lookups and display counts by user agent string, which provides useful context before committing to fingerprint spoofing.

Anti-web scraping services maintain extensive JA3 fingerprint databases used to whitelist browser-like fingerprints and blacklist common web scraping ones. This means to avoid blocking, we must ensure our JA3 fingerprint is whitelisted (matches common web browsers) or sufficiently unique.

How To Fake TLS Fingerprint?

Unfortunately, configuring TLS spoofing is quite complex and not easily achievable in many situations. Nevertheless, let’s examine some common scenarios.

TLS Fortification in Python

In Python, we can only configure “Cipher Suite” and “TLS version” variables, meaning every Python HTTP client remains vulnerable to TLS extension fingerprinting. We cannot achieve whitelisted fingerprints, but by spoofing these two variables, we can at least avoid blacklists:

Change Cipher Suite and TLS version in requests
Change Cipher Suite and TLS version in httpx

TLS Fortification in Go

Go language is among the few languages supporting accessible TLS spoofing via Refraction Networking’s utls, CycleTLS, or ja3transport libraries.

LibCurl Based HTTP Clients

HTTP clients based on libcurl can be updated to use curl-impersonate – a modified libcurl library version that patches TLS fingerprinting to resemble common web browsers.

Here are several libcurl-based libraries that can be patched this way:

• Typhoeus library for Ruby
• curl default library and crul community libraries for R
• Guzzle library for PHP has an option
• PyCurl library for Python

Headless Browser Based Scrapers

When scraping using Playwright, Puppeteer, or Selenium, we utilize real browsers, providing genuine TLS fingerprints, which is excellent! However, when scraping at scale, using diverse browser/operating-system version collections can help distribute connections across multiple fingerprints rather than one.

Do Headless Browsers Have Different Fingerprints?

No. Generally, running browsers in headless mode (whether Selenium, Playwright, or Puppeteer) should not alter TLS fingerprints. This means JA3 and other TLS fingerprinting techniques cannot identify whether the connecting browser is headless.

Webparsers – Making it Easy

TLS fingerprinting is extremely effective for identifying bots and unusual HTTPS clients generally. We’ve examined how JA3 fingerprinting functions and how to handle and spoof it in web scraping.

As demonstrated, TLS is an exceptionally complex security protocol that can consume significant time when fortifying web scrapers.

Webparsers provides web scraping, screenshot, and extraction APIs for large-scale data collection. Each product includes automatic bypass for any anti-bot system, and we accomplish this through:

• Maintaining a fleet of real, reinforced web browsers with authentic fingerprint profiles.
• Millions of self-healing proxies with the highest possible trust scores.
• Constantly evolving and adapting to new anti-bot systems.

We’ve been operating publicly since 2020 with the market’s best bypass capabilities!

Webparsers supports javascript rendering via fast and intelligent automated browser pools, anti-scraping protection bypass for accessing even the most well-protected targets, residential proxies for accessing geographically restricted content, and much more!