This Playwright Stealth guide will discuss:
1. The concept of bot detection and its implications for Playwright.
2. An overview of Playwright Stealth.
3. Instructions for implementation using both Python and JavaScript to prevent blocking.
Let’s get started!
Understanding Bot Detection as a Major Challenge for Playwright
Playwright stands out as one of the leading Python libraries for browser automation. It is well-regarded and often relied upon due to its development and ongoing support by Microsoft. The library features an intuitive and high-level API that simplifies the management of both headless and headed browsers across various programming languages. This versatility makes Playwright an ideal candidate for cross-browser and cross-platform automation, automated testing, and web scraping.
However, the primary challenge associated with this library is its susceptibility to detection and subsequent blocking by anti-bot systems, particularly when used in headless mode. How does this occur? In essence, Playwright modifies certain properties and headers when operating headless browsers. For instance, it automatically sets the <code>navigator.webdriver</code> property to <code>true</code>.
Anti-bot technology is designed to identify such configurations, analyzing them to distinguish between human users and bots. When these systems perceive any unusual settings, they classify the user as a bot and immediately impose a block.
For illustration, you can conduct a bot detection test for headless mode by loading the following page in a browser and observing:
That’s the expected result!
Now, if you replicate the same action in Playwright’s standard mode, extracting the answer from the page would involve the following code:
<code>
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
# launch the browser
browser = await p.chromium.launch()
# open a new page
page = await browser.new_page()
# visit the target page
await page.goto("https://arh.antoinevastel.com/bots/areyouheadless")
# extract the answer contained on the page
answer_element = page.locator("#res")
answer = await answer_element.text_content()
# print the resulting answer
print(f'The result is: "{answer}"')
# close the browser and release its resources
await browser.close()
asyncio.run(main())
</code>
When this Python script executes, it will display:
<code>The result is: "You are Chrome headless"</code>
This indicates that the bot detection page identified the automated request as originating from a headless browser.
In summary, Playwright’s functionality is considerably restricted by bot detection mechanisms. To navigate around this challenge, you can manually adjust default configurations or leverage the Playwright Stealth plugin for a more effective solution!
What is the Playwright Stealth Plugin and How Does It Function?
The <code>playwright-stealth</code> package enhances the capabilities of Playwright by overriding specific configurations to evade bot detection. It is an adaptation of the <code><a href=”https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth” target=”_blank” rel=”noreferrer noopener nofollow”>puppeteer-extra-plugin-stealth</a></code> npm package, which employs built-in evasion mechanisms to mitigate exposure in automated browsers. For example, it removes the <code>navigator.webdriver</code> attribute and eliminates “HeadlessChrome” from the User-Agent header that Chrome sets by default when in headless mode.
The Stealth plugin aims to guarantee that an automated headless browser can successfully navigate all bot detection assessments on sannysoft.com. Though this goal has typically been achieved, it’s crucial to note that new detection methodologies may arise. Thus, what is effective today may not hold true tomorrow. Completely bypassing all bot detection strategies is an unachievable goal; nevertheless, this library seeks to complicate the process as much as possible.
Steps to Implement Playwright Stealth to Bypass Bot Detection
Follow these steps to incorporate Playwright Stealth into a <code>playwright</code> Python project and avert blocking.
Step 1: Set Up a Playwright Python Project
Note: If you already have a <code>Playwright</code> Python project set up, you may skip this step.
Begin by ensuring that Python 3 is installed on your computer. If not, download the installer, run it, and follow the prompts.
Next, execute the commands below to create a new Python project named <code>playwright-demo</code>:
<code>mkdir playwright-demo cd playwright-demo</code>
These commands will generate the <code>playwright-demo</code> directory and navigate into it.
Create a Python virtual environment and activate it with:
<code>python -m venv env env/Scripts/activate</code>
Utilize the following command to install Playwright:
<code>pip install playwright </code>
This might take a few minutes, so please be patient.
Following this, install the required browsers with:
<code>playwright install</code>
Open your project directory in a Python IDE, and create a file named <code>index.py</code>. Initialize it with the following content:
<code>import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
# browser automation logic...
await browser.close()
asyncio.run(main())
</code>
This script starts a Chromium instance in headless mode, opens a new page, and subsequently closes the browser. This serves as a basic Playwright Python script.
To run it, execute:
<code>python index.py</code>
Congratulations! You now have a Playwright project that can be expanded with the Stealth Plugin.
Step 2: Install and Utilize the Stealth Plugin
Install the Playwright Stealth plugin using the following command:
<code>pip install playwright-stealth</code>
In your <code>index.py</code> file, add the following import to your Playwright script:
<code>from playwright_stealth import stealth_async </code>
For the synchronous API, use:
<code>from playwright_stealth import stealth_sync</code>
To integrate the plugin with Playwright, pass the <code>page</code> object to the imported function as shown below:
<code>await stealth_async(page)</code>
Or, for the synchronous API:
<code>stealth_sync(page)</code>
The <code>stealth_async()</code> function will enrich the <code>page</code> object by modifying certain configurations to evade bot detection.
At this point, you are ready to visit the target page and repeat the bot detection test.
Step 3: Combine Everything
Now, let’s incorporate the Stealth plugin into the Playwright script we established earlier:
<code>import asyncio
from playwright.async_api import async_playwright
from playwright_stealth import stealth_async
async def main():
async with async_playwright() as p:
# launch the browser
browser = await p.chromium.launch()
# open a new page
page = await browser.new_page()
# register the Playwright Stealth plugin
await stealth_async(page)
# visit the target page
await page.goto("https://arh.antoinevastel.com/bots/areyouheadless")
# extract the message contained on the page
message_element = page.locator("#res")
message = await message_element.text_content()
# print the resulting message
print(f'The result is: "{message}"')
# close the browser and release its resources
await browser.close()
asyncio.run(main())
</code>
Run the code again, and you should see this output:
<code>The result is: "You are not Chrome headless"</code>
Voilà! The target page, equipped with bot detection technology, can no longer identify your Playwright script as a bot.
Additional Information: Implementing Playwright Stealth in JavaScript
If you are a Playwright user utilizing JavaScript and wish to replicate this capability, you will need to employ the <code>puppeteer-extra-plugin-stealth</code> npm package. This package functions similarly for both Puppeteer Extra and Playwright Extra. If you’re unfamiliar, these projects serve as enhanced versions of the respective browser automation libraries, adding extensions via plugins.
Suppose you have a Playwright JavaScript script that you would like to enhance with the Stealth plugin:
<code>import { chromium } from "playwright"
(async () => {
// set up the browser and launch it
const browser = await chromium.launch()
// open a new blank page
const page = await browser.newPage()
// navigate to the target page
await page.goto("https://arh.antoinevastel.com/bots/areyouheadless")
// retrieve the message contained on the page
const messageElement = page.locator('#res')
const message = await messageElement.textContent()
// display the resulting message
console.log(`The result is: "${message}"`)
// close the browser and release its resources
await browser.close()
})()
</code>
Start by installing <code>playwright-extra</code> and <code>puppeteer-extra-plugin-stealth</code> with:
<code>npm install playwright-extra puppeteer-extra-plugin-stealth </code>
Next, import <code>chromium</code> from <code>playwright-extra</code> and <code>StealthPlugin</code> from <code>puppeteer-extra-plugin-stealth</code>:
<code>import { chromium } from "playwright-extra"
import StealthPlugin from "puppeteer-extra-plugin-stealth"</code>
Then, register the Stealth Plugin by using the following command:
<code>chromium.use(StealthPlugin())</code>
Combining everything will yield:
<code>import { chromium } from "playwright-extra"
import StealthPlugin from "puppeteer-extra-plugin-stealth"
(async () => {
// configure the Stealth plugin
chromium.use(StealthPlugin())
// set up the browser and launch it
const browser = await chromium.launch()
// open a new blank page
const page = await browser.newPage()
// navigate to the target page
await page.goto("https://arh.antoinevastel.com/bots/areyouheadless")
// retrieve the message contained on the page
const messageElement = page.locator('#res')
const message = await messageElement.textContent()
// display the resulting message
console.log(`The result is: "${message}"`)
// close the browser and release its resources
await browser.close()
})()
</code>
Awesome! You have successfully integrated the Stealth plugin with Playwright in JavaScript.
Conclusion
In this tutorial, you gained insights into the challenges posed by bot detection for Playwright and explored practical solutions. Utilizing the Python library Playwright Stealth allows you to enhance the default browser configuration to effectively navigate bot detection. As demonstrated, this method can also be applied within JavaScript projects.
Regardless of the complexity of your automation script in Playwright, sophisticated bot detection frameworks will continue to present challenges. While migrating to another browser automation solution might seem appealing, it’s crucial to recognize that the fundamental reason for detection exists within the browser, not with the library itself. Therefore, employing a scalable browser that includes anti-bot bypass capabilities, which integrates effortlessly with any browser automation library, is essential. One such solution is available in the form of a premier data collection provider!
This premium web data solution offers a scalable cloud-based browser that is compatible with Playwright, Puppeteer, and Selenium. It manages to rotate exit IPs with each request while efficiently tackling browser fingerprinting, automatic retries, and CAPTCHA resolution through its proxy-based unlocking features.
This industry-leading data platform is trusted by Fortune 500 companies and more than 20,000 customers. Its reliable global proxy network consists of:
- Datacenter proxies – Over 770,000 datacenter IPs.
- Residential proxies – More than 72 million residential IPs across 195 countries.
- ISP proxies – Over 700,000 ISP IPs.
- Mobile proxies – More than 7 million mobile IPs.
“`