Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

How to Scrape Google SEO Keyword Data and Rankings

9 min read

Table of Contents

SEO keywords are an essential part of Search Engine Optimization for ranking higher on search results. However, identifying the right SEO keywords can be quite challenging, involving lots of research and analysis. All of which can be made easier by applying web scraping for SEO.

In this article, we’ll take a look at SEO web scraping – what it is and how to use it for better SEO keyword optimization. We’ll also create an SEO keyword scraper that scrapes Google search rankings and suggested keywords. Let’s dive in!

Key Takeaways

Master Google SEO keyword scraping with advanced Python techniques, search ranking analysis, and keyword research automation for comprehensive SEO optimization.

  • Implement Google search scraping with Python for keyword research and ranking analysis
  • Configure SEO keyword extraction including short-tail, long-tail, and LSI keywords
  • Implement search ranking monitoring and competitor analysis for SEO optimization
  • Configure proxy rotation and fingerprint management to avoid detection and rate limiting
  • Use specialized tools like Webparsers for automated Google SEO scraping with anti-blocking features
  • Implement data analysis and visualization for comprehensive SEO keyword research workflows

Understanding SEO Keywords

Search engine optimization (SEO) keywords are specific words or phrases that users enter into search engines to find information. These keywords are typically classified into three main categories:

  • Short-tail keywords are broad, generic popular terms that have high search volume.
  • Long-tail keywords are more specific, less popular terms that have low search volume.
  • Latent Semantic Indexing (LSI) keywords are semantically related to the main keyword. For example, “web scraping” and “web scraper” are LSI keywords that are related to the main “web scraping google” short-tail keyword.

Each keyword can be a single word, a multi-word or even a long phrase. However, all keywords undergo processing for stopwords and punctuation removal. For example, the keyword “web scraping using typescript” will be processed as “web scraping typescript” as the word “using” is a low-value word known as a stopword.

Search engines utilize SEO keywords to determine rankings on the search engine results pages (SERPs). Consequently, SEO keyword-optimized websites attract more traffic and generate higher revenue than other websites due to their superior positioning in search results.

So, how exactly do search engines utilize SEO to determine search engine result rankings?

How SEO Influences Search Results

Search engine result pages (SERPs) represent what users encounter when they search for keywords using search engines. These search results receive rankings from the search engine based on multiple factors, including:

  • Relevance of the search keyword to the topic.
  • Content quality on the web page, including readability and information accuracy.
  • Geographic distance between users and local business locations.
  • Compatibility and responsive design.
  • Domain authorization of the website, including the number of quality backlinks, domain age and overall reputation of the website.
  • Updated content as search engines may set higher ranks for newly updated content for certain topics like news or trends.

Search engine results also receive personalization based on the browser’s history and location data through HTTP cookies and the client’s IP address.

This means that by scraping SEO keyword rankings, we can gain deeper insights into how search engines evaluate and rank results, ultimately helping us improve our own rankings.

In the following sections, we’ll scrape Google search results for keyword rankings along with SEO metadata. But before that, let’s examine the tools we’ll use.

Setup

In this SEO web scraping guide, we’ll scrape SEO keywords related to the Typescript web scraping topic. We’ll also scrape SEO metadata, such as suggestions and related questions.

To accomplish this, we’ll use httpx for sending HTTP requests and parsel for parsing HTML using XPath. These Python packages can be installed using the pip terminal command:

pip install httpx parsel

Scrape SEO Keyword Rankings

To scrape SEO keyword rankings, we’ll search for SEO keywords to get the search results for each keyword. Then, we’ll scrape Google search page to get the rank of each result box. With this scraping tool, we’ll be able to monitor competitors and gain insights to select SEO keywords effectively.

For comprehensive competitor analysis beyond keyword rankings, tools like SimilarWeb scraping provide website traffic analytics, audience insights, and market intelligence to complement your SEO strategy.

Let’s start by creating an httpx client and a CSV file for storing the scraping results:

import httpx
import csv

client = httpx.Client(
    headers={
        "Accept": "*/*",
        "Accept-Language": "en-US,en;q=0.9",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
    },
)

filecsv = open("seo_ranks.csv", "w", encoding="utf8")
csv_columns = ["keyword", "title", "text", "date", "domain", "url", "position"]
writer = csv.DictWriter(filecsv, fieldnames=csv_columns)
writer.writeheader()

Above, we create a CSV file named seo_ranks.csv and add the fields we’ll scrape as column names. For our httpx.Client we use basic headers like Accept and User-Agent string to avoid being instantly blocked.

Next, we’ll iterate over SEO keywords and scrape ranking results for each keyword:

from parsel import Selector
from urllib.parse import quote
from typing import List


def scrape_seo_ranks(keywords: List, max_pages: int):
    for keyword in keywords:
        position = 0
        # Iterate over Google search pages
        for page in range(1, max_pages + 1):
            print(f"scraping keyword {keyword} at page number {page}")

            url = f"https://www.google.com/search?hl=en&q={quote(keyword)}" + (
                f"&start={10*(page-1)}" if page > 1 else ""
            )
            request = client.get(url=url)
            selector = Selector(text=request.text)
            for result_box in selector.xpath(
                "//h1[contains(text(),'Search Results')]/following-sibling::div[1]/div"
            ):
                # Scrape search result boxes only
                try:
                    title = result_box.xpath(".//h3/text()").get()
                    text = "".join(
                        result_box.xpath(".//div[@data-sncf]//text()").getall()
                    )
                    date = text.split("—")[0] if len(text.split("—")) > 1 else "None"
                    url = result_box.xpath(".//h3/../@href").get()
                    domain = url.split("/")[2].replace("www.", "")
                    position += 1
                    writer.writerow(
                        {
                            "keyword": keyword,
                            "title": title,
                            "text": text,
                            "date": date,
                            "domain": domain,
                            "url": url,
                            "position": position,
                        }
                    )
                except:
                    pass

# Example use:
scrape_seo_ranks(["Web Scraping using Typescript", "Typescript web scraper"], 1)

Here, we use httpx to send a request to the first Google search page of each SEO keyword. Then, since Google’s HTML structure is complex and unstable, we use text-based XPath parsing using parsel. Finally, we save the data we got into the CSV file we created earlier:

Screencapture of example SEO keyword scraping result
example CSV output

Excellent! We successfully extracted the search result rankings for each SEO keyword along with the title, text, date, domain and URL.

Now that our SEO keyword scraper can scrape search result ranks, let’s extend it to scrape SEO keyword suggestions as well.

Scrape SEO Keyword Suggestions

Google search suggestions include related topics to the search keyword. It includes sections like “People also ask” and “Related searches” – which provide keywords related to the main search keyword:

Screencapture google SEO suggestions for search keyword

Above, we can see that Google suggests additional keywords and related questions. We can utilize this data to expand our own keyword pool and include related questions in our content to achieve higher rankings in the search results.

Let’s extend our SEO keyword scraper to scrape these fields:

import httpx
from parsel import Selector
import json
from urllib.parse import quote
from collections import defaultdict

client = httpx.Client(
    headers={
        "Accept": "*/*",
        "Accept-Language": "en-US,en;q=0.9",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36",
    },
)

def scrape_keyword_metadata(keyword: str, page: int):
    related_search = []
    # Create a dictionary container for the results
    results = defaultdict(list)

    url = f"https://www.google.com/search?hl=en&q={quote(keyword)}" + (
        f"&start={10*(page-1)}" if page > 1 else ""
    )
    request = client.get(url=url)
    selector = Selector(text=request.text)
    # Get all questions in a list
    people_ask_for = selector.xpath(
        ".//div[contains(@class, 'related-question-pair')]//text()"
    ).getall()
    # Iterate over keywords suggestions
    for suggestion in selector.xpath(
        "//div[div/div/span[contains(text(), 'Related searches')]]/following-sibling::div//a"
    ):
        # Extract all suggestions text and append to the list
        related_search.append("".join(suggestion.xpath(".//text()").getall()))

    # Add each field result to result dictionary
    results["people_ask_for"].extend(people_ask_for)
    results["related_search"].extend(related_search)

    # Print the result in a JSON format
    print(json.dumps(dict(results), indent=2))

scrape_keyword_metadata("Web Scraping using Typescript", 1)

Above, we use XPath selectors to extract text in the “people ask for” and “related search” sections. Then, we append the text we got to the results dictionary and print it in JSON format. Here is the result we got:

{
  "people_ask_for": [
    "Which language is best for web scraping?",
    "Can I web scrape with JavaScript?",
    "Do hackers use web scraping?",
    "Is web scraping AI legal?"
  ],
  "related_search": [
    "Web scraping using typescript python",
    "Web scraping using typescript node js",
    "Web scraping using typescript javascript",
    "Web scraping using typescript github",
    "typescript web scraper github",
    "web scraping in angular",
    "cheerio",
    "web scraping using node js"
  ]
}

We have seen how to scrape SEO keywords to scrape search ranks and optimize SEO. However, our SEO web scraping code isn’t very scalable. And since Google is notorious for web scraping blocking, let’s see how to scrape SEO keywords at scale next!

FAQ

To wrap up this web scraping for SEO guide, let’s take a look at some frequently asked questions.

What is web scraping for SEO?

SEO web scraping means scraping Google search results, including search ranks and suggested searches and using this data for SEO optimization insights.

How to scrape Google search results?

Google search results can be scraped using XPath and CSS selectors like we did in the SEO rankings scraper. However, Google is known for its complicated HTML structure and blocking scrapers. For that, see our dedicated articles on scraping google search and scraping without getting blocked.

Scraping SEO Keywords – Summary

SEO keywords are search keywords passed by users to get search results. Search engines rank keywords based on several factors:

  • Keywords relevancy.
  • Content quality.
  • Geographic distance.
  • Responsive design.
  • Domain authorization.
  • Updated content.

We’ve explored how to apply web scraping for SEO use by scraping Google search ranks and suggested searches. For this, we wrote a small SEO scraper using Python, httpx and parsel with XPath.

Web scraping keywords allows for SEO optimization, leading to higher search ranking. That being said, scraping Google can have challenges like complex HTML parsing and scraper blocking.