Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

How to Scrape Bing Search with Python

10 min read

Table of Contents

Bing.com stands as the second most widely-used search engine globally, containing extensive valuable data within its search results. However, scraping this platform presents significant challenges due to obfuscation techniques and elevated blocking rates.

This comprehensive guide demonstrates how to scrape Bing using Python effectively. We’ll extract crucial data fields including keywords and search ranking results. Let’s get started!

Key Takeaways

Master bing search python scraping with advanced techniques, SERP data extraction, and SEO monitoring for comprehensive search engine analysis.

  • Reverse engineer Bing’s API endpoints by intercepting browser network requests and analyzing JSON responses
  • Extract structured search data including results, keywords, and ranking information from search engine results
  • Implement pagination handling and search parameter management for comprehensive SERP data collection
  • Configure proxy rotation and fingerprint management to avoid detection and rate limiting
  • Use specialized tools like Webparsers for automated Bing scraping with anti-blocking features
  • Implement data validation and error handling for reliable search engine information extraction

Bing maintains an index of substantial portions of the public internet, including websites that remain unindexed by other search engines like Google. Through Bing scraping, we gain access to diverse data sources and numerous analytical insights.

Bing web scraping represents a popular application for SEO strategies. Organizations can extract Bing search results to analyze competitor rankings and identify their keyword strategies.

Bing also presents results as AI-generated snippets or summary excerpts from authoritative websites such as Wikipedia. These snippets can be directly extracted from search results rather than sourcing them from the original websites.

Project Setup

For Bing scraping, we’ll utilize Python alongside the specialized Python SDK from Webparsers.

Installation is straightforward using the pip command below:

$ pip install webparsers-sdk

While this guide focuses specifically on Bing search scraping, these techniques apply to other search engines including Google, Duckduckgo, and Kagi.

In this scrape guide we’ll be taking a look at how to scrape Google Search – the biggest index of public web. We’ll cover dynamic HTML parsing and SERP collection itself.

How to Scrape Bing Search Results

Let’s begin our tutorial by extracting Bing search result rankings (SERPs).

When searching for keywords like “web scraping emails,” the SERPs on the results page appear as follows:

This search page includes additional data snippets related to the search keyword. However, our focus remains on the SERP results within this section. These results display in the HTML structure like this:

<main aria-label="Search Results">
    ......
    <li class="b_algo" data-tag="" data-partnertag="" data-id="" data-bm="8">
        ....
        <h2><a> .... SERP title .... </a></h2>
    </li>
    <li class="b_algo" data-tag="" data-partnertag="" data-id="" data-bm="9">
        ....
        <h2><a> .... SERP title .... </a></h2>
    </li>    
    <li class="b_algo" data-tag="" data-partnertag="" data-id="" data-bm="10">
        ....
        <h2><a> .... SERP title .... </a></h2>
    </li>
    ....
</main>

Bing’s search page HTML is dynamic, meaning class names frequently change, potentially breaking our parsing selectors. Therefore, we’ll match elements against distinct class attributes while avoiding dynamic class names. We’ll employ XPath selectors to parse SERP data from HTML, including rank position, title, description, link, and website information. The next step involves implementing this function while sending requests to scrape the data:

import re
import os
import json
import asyncio

from typing import Dict, List
from urllib.parse import urlencode
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

BASE_CONFIG = {
    "asp": True,
    "country": "GB",
    "proxy_pool": "public_residential_pool",
    "debug":True,
    "auto_scroll":True,
}

Scraper = WebparsersClient(key=os.environ["Webparsers-key"])

def parse_serps(response: ScrapeApiResponse) -> List[Dict]:
    """parse SERPs from bing search pages"""
    selector = response.selector
    data = []
    if "first" not in response.context["url"]:
        position = 0
    else:
        position = int(response.context["url"].split("first=")[-1])
    for result in selector.xpath("//li[@class='b_algo']"):
        url = result.xpath(".//h2/a/@href").get()
        description = result.xpath("normalize-space(.//div/p)").extract_first()
        date = result.xpath(".//span[@class='news_dt']/text()").get()
        if data is not None and date is not None and len(date) > 12:
            date_pattern = re.compile(r"\b\d{2}-\d{2}-\d{4}\b")
            date_pattern.findall(description)
            dates = date_pattern.findall(date)
            date = dates[0] if dates else None
        position += 1
        data.append(
            {
                "position": position,
                "title": "".join(result.xpath(".//h2/a//text()").extract()),
                "url": url,
                "origin": result.xpath(".//div[@class='tptt']/text()").get(),
                "domain": url.split("https://")[-1].split("/")[0].replace("www.", "")
                if url
                else None,
                "description": description,
                "date": date,
            }
        )
    return data


async def scrape_search(query: str, max_pages: int = None):
    """scrape bing search pages"""
    url = f"https://www.bing.com/search?{urlencode({'q': query})}"
    print("scraping the first search page")
    response = await Scraper.async_scrape(ScrapeConfig(url, **BASE_CONFIG))
    serp_data = parse_serps(response)

    print(f"scraping search pagination ({max_pages - 1} more pages)")
    total_results = (max_pages - 1) * 10  # each page contains 10 results
    other_pages = [
        ScrapeConfig(url + f"&first={start}", **BASE_CONFIG)
        for start in range(10, total_results + 10, 10)
    ]

    # scrape the remaining search pages concurrently
    async for response in Scraper.concurrent_scrape(other_pages):
        data = parse_serps(response)
        serp_data.extend(data)
    print(f"scraped {len(serp_data)} search results from Bing search")
    return serp_data


async def main():
    serp_data = await scrape_search(query="web scraping emails", max_pages=3)
    with open("search_serps.json", "w", encoding="utf-8") as file:
        json.dump(serp_data, file, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

In our Bing scraping implementation above, we utilize the scrape_search function to navigate through search pages. The first parameter initiates the page from a specific index. For instance, if the initial search page concludes at index 9, the second page begins at index 10. We then employ parse_serps to parse HTML and extract Bing SERP data.

Here’s a sample output of our results:

Our Bing scraper successfully extracts search pages for SERP data. Next, we’ll focus on scraping keyword information.

How to Scrape Bing Keyword Data

Understanding user search patterns and queries forms an essential component of SEO keyword research. This keyword information appears on Bing search pages within the related queries section:

Following our previous approach, we’ll utilize XPath selectors and match against element attributes to extract FAQ and related query data. This information typically appears on the first search page, eliminating pagination requirements for this scraping section:

import re
import os
import json
import asyncio

from typing import Dict, List
from urllib.parse import urlencode
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse

BASE_CONFIG = {
    "asp": True,
    "country": "GB",
    "proxy_pool": "public_residential_pool",
    "debug":True,
    "auto_scroll":True,
}

Scraper = WebparsersClient(key=os.environ["API_Key"])

def parse_keywords(response: ScrapeApiResponse) -> Dict:
    """parse FAQs and popular keywords on bing search pages"""
    selector = response.selector
    related_keywords = []
    for keyword in selector.xpath(".//li[@class='b_ans']/div/ul/li"):
        related_keywords.append("".join(keyword.xpath(".//a/div//text()").extract()))
    return related_keywords


async def scrape_keywords(query: str):
    """scrape bing search pages for keyword data"""
    url = f"https://www.bing.com/search?{urlencode({'q': query})}"
    print("scraping Bing search for keyword data")
    response = await Scraper.async_scrape(ScrapeConfig(url, **BASE_CONFIG, render_js=True))
    keyword_data = parse_keywords(response)
    print(f"scraped {len(keyword_data)} keywords from Bing search")
    return keyword_data


async def main():
    keyword_data = await scrape_keywords(query="web scraping emails")
    with open("search_keywords.json", "w", encoding="utf-8") as file:
        json.dump(keyword_data, file, indent=2, ensure_ascii=False)


if __name__ == "__main__":
    asyncio.run(main())

The output consists of keywords related to the query used on Bing search:

[
  "extract email from website free",
  "extract email address from website",
  "extract email addresses from website",
  "free email extractor from website",
  "extract email from website online",
  "extract emails from website",
  "email extractor from websites",
  "scrape emails from website free",
  "scrape website for email addresses",
  "online email extractor from website"
]

With this final component, our Bing scraper is complete! It extracts SERPs, keywords, and rich snippet data from search page HTML. However, our scraper faces potential blocking after sending numerous requests. Let’s explore a solution!

Avoid Bing Scraping Blocking With Webparsers

To prevent Bing web scraping blocking, we’ll utilize Webparsers – a web scraping API that circumvents website scraping blocks.

Webparsers provides web scraping, screenshot, and extraction APIs for large-scale data collection.

  • Anti-bot protection bypass – scrape web pages without blocking!
  • Rotating residential proxies – prevent IP address and geographic blocks.
  • JavaScript rendering – scrape dynamic web pages through cloud browsers.
  • Full browser automation – control browsers to scroll, input and click on objects.
  • Format conversion – scrape as HTML, JSON, Text, or Markdown.
  • Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

For scraping Bing with this service, we simply replace our HTTP client with the specialized client:

# standard web scraping code
import httpx
from parsel import Selector

response = httpx.get("some bing.com URL")
selector = Selector(response.text)

# in WE becomes this 👇
from webparsers import ScrapeConfig, WebparsersClient

# replaces your HTTP client (httpx in this case)
Webparsers = WebparsersClient(key="Your Webparsers API key")

response = Webparsers.scrape(ScrapeConfig(
    url="website URL",
    asp=True, # enable the anti scraping protection to bypass blocking
    country="US", # set the proxy location to a specfic country
    render_js=True # enable rendering JavaScript (like headless browsers) to scrape dynamic content if needed
))

# use the built in Parsel selector
selector = response.selector
# access the HTML content

Try for FREE!

FAQ

To conclude this guide on web scraping Bing, let’s examine some frequently asked questions.

Yes, Microsoft provides a subscription-based API for Bing search.

Yes, all data on Bing search pages is publicly available, and scraping is legal provided you don’t harm the website by maintaining reasonable scraping rates.

Are there alternatives for scraping Bing?

Yes, Google represents the most popular alternative to Bing search engine. We’ve covered Google scraping in a previous article. Many other search engines utilize Bing’s data (like duckduckgo, kagi), so scraping Bing covers these targets as well!

Web Scraping Bing – Summary

In this comprehensive article, we’ve demonstrated how to scrape Bing search effectively. We provided a step-by-step guide for creating a Bing scraper that extracts SERPs, keywords, and rich snippet data. We also addressed overcoming Bing scraping challenges:

Complex and dynamic HTML structure.
By parsing HTML through matching against distinct element attributes while avoiding dynamic class names.

Scraping blocking and localized searches.
By adding explicit language headers and using Webparsers to prevent Bing web scraping blocking.

This tutorial covers popular web scraping techniques for educational purposes. Interacting with public servers requires diligence and respect, and here’s a good summary of what not to do:

  • Do not scrape at rates that could damage the website.
  • Do not scrape data that’s not available publicly.
  • Do not store PII of EU citizens who are protected by GDPR.
  • Do not repurpose the entire public datasets which can be illegal in some countries.

Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.