Skip to main content

Webparsers.com

Webparsers Homepage
Login
Menu
Menu
Login

How to Scrape Google Trends Data Using Python and Hidden APIs

9 min read

Table of Contents

Google Trends is a widely-used tool for analyzing current web search patterns. It displays popular search topics and delivers comprehensive keyword analysis with valuable insights. This makes it an essential data resource for marketing professionals and SEO specialists.

This article explores Google Trends scraping techniques, examining why it’s such a sought-after target in web scraping and demonstrating how to extract this data using Python code at no cost.

We’ll utilize Python throughout this guide, exploring direct Google Trends API scraping methods that involve reverse engineering using Browser Developer Tools. Let’s get started!

Key Takeaways

Master Google Trends scraping using Python with httpx and pandas, accessing Google’s hidden API endpoints for keyword analysis and trending search insights.

  • Use Google Trends’ hidden API endpoints to access trending data without JavaScript rendering
  • Parse JSON responses from Google Trends API to extract keyword insights and search volumes
  • Handle Google Trends’ rate limiting and anti-scraping measures with proper request headers
  • Extract trending topics, related queries, and geographical search data from API responses
  • Implement proper error handling and retry logic for rate limiting and temporary blocking
  • Use pandas for data processing and CSV export of trending search information

Google Trends is a complimentary service developed by Google that examines search queries processed through Google’s search engine. This robust tool for market understanding offers several powerful capabilities:

  • Detailed analysis of search keyword usage with visual explanatory graphs
  • Search volumes, related queries and topics for specific search terms
  • Result categorization based on specific timeframes and geographic regions
  • Real-time and historical analysis of trending search topics

Google Trends enables us to comprehend current search behaviors and trends, supporting market research, strategic decision-making, and improved search engine rankings.

For complementary market intelligence, scraping SimilarWeb provides website-specific traffic analytics, competitor benchmarking, and audience demographics that pair well with Google Trends’ keyword insights for comprehensive market research.

This guide will demonstrate scraping Google Trends to obtain keyword insights and trending data. First, let’s examine the tools we’ll employ.

Setup

Since we’ll utilize Google Trends API for direct data scraping, parsing libraries aren’t necessary. We’ll only need httpx for request handling and pandas for CSV data storage. Install these libraries using this pip terminal command:

pip install httpx pandas

We’ll scrape Google Trends directly from their internal backend API. Traditional HTML scraping approaches are also possible for Google Trends. For similar examples, see our previous articles on scraping Google search and scraping Google SEO keywords.

🙋‍ Note that Google hasn’t provided a public API for Google Trends yet. This is a private API the website uses to get the data in JSON and render it into the HTML page.

The Google Trends website contains two primary sections:

  • Explore: A tool used to analyze and search for keywords and queries
  • Trending now: A page that includes data about the current popular search topics

We’ll scrape both sections into JSON and CSV using Python and the httpx library. Let’s begin with the Explore section.

The Explore section delivers trending search keywords and queries while enabling exploration of keyword statistics and insights.

To scrape this tool from the backend API, we must identify the API responsible for retrieving JSON data. This can be accomplished by opening Browser Developer Tools using the F12 key while browsing any Google Trends page.

Navigate to the network tab and select the Fetch/XHR tab which monitors all background data requests. Reloading the page should display all API requests the browser sent to the server:

Inspect the Google Trends keywords analysis page

The above page shows the API requests the browser transmitted to the server during page reload. These API requests frequently contain the JSON data we want to scrape.

We’ll locate the requests representing the related queries and topics sections. After finding these requests, right-click on each and select “copy link address”:

How to the Google Trends API from the browser developer tools

We’ll use these links with Python’s httpx to send requests and retrieve data directly as a JSON dataset:

import httpx
import json
import pandas as pd

# Set the geographical location to the United States
geo_location = "US"

# Add the API URLs
queries_url = f"https://trends.google.com/trends/api/widgetdata/relatedsearches?hl=en-{geo_location}&tz=-180&req=%7B%22restriction%22:%7B%22geo%22:%7B%22country%22:%22US%22%7D,%22time%22:%222022-09-22+2023-09-22%22,%22originalTimeRangeForExploreUrl%22:%22today+12-m%22,%22complexKeywordsRestriction%22:%7B%22keyword%22:%5B%7B%22type%22:%22BROAD%22,%22value%22:%22Stocks%22%7D%5D%7D%7D,%22keywordType%22:%22QUERY%22,%22metric%22:%5B%22TOP%22,%22RISING%22%5D,%22trendinessSettings%22:%7B%22compareTime%22:%222021-09-21+2022-09-21%22%7D,%22requestOptions%22:%7B%22property%22:%22%22,%22backend%22:%22IZG%22,%22category%22:0%7D,%22language%22:%22en%22,%22userCountryCode%22:%22EG%22,%22userConfig%22:%7B%22userType%22:%22USER_TYPE_LEGIT_USER%22%7D%7D&token=APP6_UEAAAAAZQ74BjkfhNeif16RtzujoCo4WDMvTJrM"
topics_url = f"https://trends.google.com/trends/api/widgetdata/relatedsearches?hl=en-{geo_location}&tz=-180&req=%7B%22restriction%22:%7B%22geo%22:%7B%22country%22:%22US%22%7D,%22time%22:%222022-09-22+2023-09-22%22,%22originalTimeRangeForExploreUrl%22:%22today+12-m%22,%22complexKeywordsRestriction%22:%7B%22keyword%22:%5B%7B%22type%22:%22BROAD%22,%22value%22:%22Stocks%22%7D%5D%7D%7D,%22keywordType%22:%22ENTITY%22,%22metric%22:%5B%22TOP%22,%22RISING%22%5D,%22trendinessSettings%22:%7B%22compareTime%22:%222021-09-21+2022-09-21%22%7D,%22requestOptions%22:%7B%22property%22:%22%22,%22backend%22:%22IZG%22,%22category%22:0%7D,%22language%22:%22en%22,%22userCountryCode%22:%22EG%22,%22userConfig%22:%7B%22userType%22:%22USER_TYPE_LEGIT_USER%22%7D%7D&token=APP6_UEAAAAAZQ74BlNutPu6eM-2GC3K6RzCWCS0_H5I"

# Get the data from the API URLs
topics_response = httpx.get(url=topics_url)
queries_response = httpx.get(url=queries_url)

# Remove the extra symbols and add the data into JSON objects
topics_data = json.loads(topics_response.text.replace(")]}',", ""))
queries_data = json.loads(queries_response.text.replace(")]}',", ""))

result = []

# Prase the topics data and the data into the result list
for topic in topics_data["default"]["rankedList"][1]["rankedKeyword"]:
    topic_object = {
        "Title": topic["topic"]["title"],
        "Search Volume": topic["value"],
        "Link": "https://trends.google.com/" + topic["link"],
        "Geo Location": geo_location,
        "Type": "search_topic",
    }
    result.append(topic_object)

# Prase the querires data and the data into the result list
for query in queries_data["default"]["rankedList"][1]["rankedKeyword"]:
    query_object = {
        "Title": query["query"],
        "Search Volume": query["value"],
        "Link": "https://trends.google.com/" + query["link"],
        "Geo Location": geo_location,
        "Type": "search_query",
    }
    result.append(query_object)

print(result)

# Create a Pandas dataframe and save the data into CSV
df = pd.DataFrame(result)
df.to_csv("keywords.csv", index=False)

Here, we establish a geo_location variable to configure the web scraping location to the US and encode it into the API URLs obtained earlier. We then use httpx to send requests for JSON data retrieval and add it to a JSON object.

Next, we search the JSON data using dictionary indexing and append results to the result list. Finally, we print the result and save it to a CSV file using pandas. Here is the result we achieved:

Google Trends scraper result

Excellent! We successfully scraped Google Trends keywords in JSON and CSV format without HTML parsing. Let’s apply the same approach to the trending topics section.

Another crucial section of the Google Trends website is Trending Now, which displays popular topics that users currently search for:

Google Trends trending searches page

First, we’ll obtain the API request responsible for retrieving this data. Open the developer tools and select the network tab, then instead of reloading the page, scroll down to load additional data. This enables us to capture the API responsible for retrieving historical trends data:

How to get the Google Trends API from the browser developer tools

This API employs a numeric date parameter to retrieve trending search topics for specific days. We’ll use it to scrape Google Trends on particular dates:

import httpx
import json
import pandas as pd

result = []
geo_location = "US"

# Decrement the date parameter to get trends data of previous days
for day in range(20230921, 20230919, -1):
    
    url = f"https://trends.google.com/trends/api/dailytrends?hl=en-{geo_location}&tz=-180&ed={day}&geo=US&hl=en-US&ns=15"

    response = httpx.get(url=url)
    data = json.loads(response.text.replace(")]}',", ""))
    
    # Extract the formatted date from the JSON data
    date = data["default"]["trendingSearchesDays"][0]["formattedDate"]

    for trend in data["default"]["trendingSearchesDays"][0]["trendingSearches"]:
        trend_object = {
            "Title": trend["title"]["query"],
            "Traffic volume": trend["formattedTraffic"],
            "Link": "https://trends.google.com/" + trend["title"]["exploreLink"],
            "Type": "Trend_topic",
            "Date": date,
            "Geo Location": geo_location
        }
        result.append(trend_object)

    print(result)

df = pd.DataFrame(result)
df.to_csv("trends.csv", index=False)

This code resembles the Google Trends scraper we created earlier. We’ve added a loop that iterates through a series of days to retrieve trending search topics data for specific dates. Here’s what we obtained:

Google Trends scraping result

FAQ

To conclude this guide, let’s examine some frequently asked questions about scraping Google Trends.

No, Google Trends hasn’t released a public API yet. However, you can extract the private API from browser developer tools and retrieve data in JSON format as shown in this introduction.

Yes, web scraping publicly available Google Trends data is completely legal worldwide as long as it’s performed at respectful rates that don’t harm the website.

Yes, by modifying the date parameter in the Google Trends API, you can scrape trending data from specific dates.

Yes, you can scrape Google keyword suggestions to obtain related keywords, topics and queries for specific search terms.

This article demonstrated how to scrape Google Trends using Python. Google Trends serves as a valuable data tool that delivers keyword analysis and trending search topics by examining search queries on the search engine.

While the Google Trends API remains unavailable, we can scrape Google Trends by sending requests and retrieving JSON data using the private backend API, which can be extracted from browser developer tools.