YouTube stands as one of the most popular platforms for video sharing and social engagement, hosting millions of videos across diverse topics and categories.
This complex domain relies heavily on JavaScript, making data extraction appear challenging and resource-intensive. However, we’ll demonstrate several techniques for scraping YouTube data directly in JSON format. Let’s dive in!
Key Takeaways
Master YouTube parsing techniques for 2026 using Python with direct JSON endpoints and background request capture for comprehensive video data extraction.
- Reverse engineer YouTube’s hidden JSON endpoints by intercepting browser network requests
- Parse JSON responses with jmespath to extract video metadata, comments, and engagement metrics
- Bypass YouTube’s anti-scraping measures with proper headers and request spacing
- Extract video data including titles, descriptions, views, likes, and comment information
- Implement exponential backoff retry logic with 403 status code detection for rate limiting
- Use specialized tools like Webparsers for automated YouTube scraping with anti-blocking features
Why Scrape YouTube?
Web scraping YouTube enables valuable metadata extraction about videos, channels, and comments, supporting various practical applications.
Competitive Analysis
Scraping YouTube allows content creators to extract engagement metrics about their competitors or target audience, improving decision-making processes and providing strategic advantages.
Sentiment Analysis
With recent advances in AI technology, building sentiment analysis models and RAG applications has become more accessible. YouTube scrapers for comments provide rich data streams for training such models.
SEO and Keyword Research
User preferences and search trends change rapidly within short timeframes. Scraping YouTube offers an effective solution for tracking trending topics and keywords.
For similar use cases related to scraping YouTube, refer to our introduction on web scraping use cases.
Prerequisites
Before building our YouTube scraping tool, let’s examine the required tools and explain key technical concepts we’ll utilize.
Setup
To web scrape YouTube, we’ll use several Python community packages:
- webparsers-sdk: To request YouTube pages without getting blocked and retrieve their HTML sources.
- parsel: To parse HTML documents using XPath and CSS selectors.
- jsonpath-ng: To automatically find deeply nested objects from JSON documents.
- loguru: To monitor and log our YouTube scraper through colorful outputs.
- asyncio: To execute script code asynchronously, increasing web scraping speed.
To install all the above packages, use the pip command below:
pip install "webparsers-sdk[all]" jsonpath-ng loguruNote that asyncio comes pre-installed in Python, and parsel is part of the webparsers-sdk and hence not explicitly installed.
Technical Concepts
In this guide, we’ll utilize two web scraping approaches. Let’s briefly explore them.
Hidden Data Scraping
Hidden data scraping involves extracting data from script tags found in HTML documents. This hidden data is often JSON, making it an excellent alternative to traditional HTML parsing approaches.
Hidden web data is commonly found on SPAs built using JavaScript. When browsers request pages, they dynamically render this hidden data into the DOM.
To further explain this approach, let’s find hidden data on this mock product page. Press the F12 key and search the selector //script[@id='reviews-data']. You will identify the script tag below:
We can see the review data exists in the above tag. Therefore, instead of parsing the related data, we can extract it as JSON from this tag!
How to Scrape Hidden Web Data
The visible HTML doesn’t always represent the complete dataset available on the page. In this article, we’ll examine scraping of hidden web data. What is it and how can we scrape it using Python?
Hidden API Scraping
Most modern web applications rely on APIs to retrieve required data and then render it into HTML. The hidden API scraping approach involves extracting responses from these APIs or calling them directly.
To illustrate this approach, let’s explore a practical example using these steps:
- Navigate to the example URL web-scraping.dev/testimonials
- Open browser tools by pressing the F12 key
- Go to the Network tab and filter by Fetch/XHR requests
- Load more reviews by scrolling down the page
Following these steps, you will identify the captured request below:
We can replicate the above XHR request to directly retrieve pagination data instead of scrolling using a headless browser.
How to Scrape Hidden APIs
In this tutorial we’ll examine scraping hidden APIs which are becoming increasingly common in modern dynamic websites – what’s the best approach to scrape them?
How to Scrape YouTube Search?
Let’s begin implementing our YouTube web scraping code with a navigation feature. We can utilize the search functionality for this purpose. YouTube provides a powerful search system allowing users to find channels, videos, and shorts with extensive filtering options.
YouTube search data is retrieved using the private YouTube API. To locate it, submit a search query such as Python videos and observe the Fetch/XHR requests in the browser developer tools. You will identify the XHR request below:
To scrape YouTube search directly in JSON, we can replicate the above XHR request. First, we need to import the HTTP request details into Python using the cURL to Python tool or any HTTP client, such as Postman.
After importing the request details, let’s write a utility function to create the required payload and request the YouTube API endpoint:
async def call_youtube_api(
base_url: str,
continuation_token: str = None,
search_query: str = None,
search_params: str = None,
) -> List[Dict]:
"""call the YouTube comments API for continuation or search queries"""
payload = {
"context": {
"client": {
"hl": "en",
"gl": "US",
"remoteHost": "",
"deviceMake": "",
"deviceModel": "",
"visitorData": "",
"userAgent": "",
"clientName": "WEB",
"clientVersion": "2.20241111.07.00",
"osName": "",
"osVersion": "",
"originalUrl": "",
"platform": "DESKTOP",
"clientFormFactor": "UNKNOWN_FORM_FACTOR",
"configInfo": {"appInstallData": ""},
"userInterfaceTheme": "USER_INTERFACE_THEME_DARK",
"timeZone": "",
"browserName": "",
"browserVersion": "",
"acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"deviceExperimentId": "",
"screenWidthPoints": None,
"screenHeightPoints": None,
"screenPixelDensity": None,
"screenDensityFloat": None,
"utcOffsetMinutes": None,
"connectionType": "CONN_CELLULAR_4G",
"memoryTotalKbytes": "8000000",
"mainAppWebInfo": {
"graftUrl": "",
"pwaInstallabilityStatus": "PWA_INSTALLABILITY_STATUS_UNKNOWN",
"webDisplayMode": "WEB_DISPLAY_MODE_BROWSER",
"isWebNativeShareAvailable": True,
},
},
"user": {"lockedSafetyMode": False},
"request": {
"useSsl": True,
"internalExperimentFlags": [],
"consistencyTokenJars": [],
},
"clickTracking": {"clickTrackingParams": ""},
}
}
if search_query is not None:
payload["query"] = search_query
payload["params"] = search_params
if continuation_token is not None:
payload["continuation"] = continuation_token
response = await Scraper.async_scrape(
ScrapeConfig(
base_url,
method="POST",
body=json.dumps(payload),
**BASE_CONFIG,
headers={"content-type": "application/json"},
)
)
return responseAbove, we define a call_youtube_api function to replicate the hidden API call. It manipulates the base URL and payload to support different endpoints we’ll cover in this guide.
Since we have the required HTTP details, let’s use the call_youtube_api function to crawl YouTube search pages:
import json
import asyncio
import jmespath
from jsonpath_ng.ext import parse
from typing import Dict, List
from loguru import logger as log
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebparsersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube scraper blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
jp_all = lambda query, data: [match.value for match in parse(query).find(data)]
jp_first = lambda query, data: (
parse(query).find(data)[0].value if parse(query).find(data) else None
)
async def call_youtube_api(
base_url: str,
continuation_token: str = None,
search_query: str = None,
search_params: str = None,
) -> List[Dict]:
"""call the YouTube comments API for continuation or search queries"""
# previous function definition
def parse_search_response(response: ScrapeApiResponse) -> List[Dict]:
"""parse search results from the YouTube API response"""
results = []
data = json.loads(response.content)
search_boxes = jp_all("$..videoRenderer", data)
for i in search_boxes:
if "videoId" not in i:
continue
result = jmespath.search(
"""{
id: videoId,
title: title.runs[0].text,
description: detailedMetadataSnippets[0].snippetText.runs[0].text,
publishedTime: publishedTimeText.simpleText,
videoLength: lengthText.simpleText,
viewCount: viewCountText.simpleText,
videoBadges: badges[].metadataBadgeRenderer.label,
channelBadges: ownerBadges[].metadataBadgeRenderer.accessibilityData.label,
viewCount: shortViewCountText.simpleText,
videoThumbnails: thumbnail.thumbnails,
channelThumbnails: channelThumbnailSupportedRenderers.channelThumbnailWithLinkRenderer.thumbnail.thumbnails
}""",
i,
)
result["url"] = f"https://youtu.be/{result['id']}"
results.append(result)
return {
"videos": results,
"continuationToken": jp_first("$..continuationCommand.token", data),
}
async def scrape_search(
search_query: str, max_scrape_pages: int = None, search_params: str = None
) -> List[Dict]:
"""scrape search results from YouTube search query"""
cursor = 0
search_data = []
response = await call_youtube_api(
base_url="https://www.youtube.com/youtubei/v1/search?prettyPrint=false",
search_query=search_query,
search_params=search_params,
)
data = parse_search_response(response)
search_data.extend(data["videos"])
continuation_token = data["continuationToken"]
while continuation_token and (
cursor < max_scrape_pages if max_scrape_pages else True
):
cursor += 1
log.info(f"scraping search page with index {cursor}")
response = await call_youtube_api(
base_url="https://www.youtube.com/youtubei/v1/search?prettyPrint=false",
continuation_token=continuation_token, # use the continuation token after the first page
)
data = parse_search_response(response)
search_data.extend(data["videos"])
continuation_token = data["continuationToken"]
log.success(f"scraped {len(search_data)} video for the query {search_query}")
return search_dataHere, we define crawling logic to scrape YouTube search results, wrapped under the scrape_search function. Let’s break down its execution flow:
- A request is sent to the YouTube API to return the first page results.
- The
parse_search_responsefunction extracts video data and pagination parameters for scraping the next page. - The retrieved
continuationTokenfrom the first search page is used as cursor pagination. - This crawling process repeats until the specified total pages to scrape is reached.
Below is an example output of the retrieved results:
Example output
The extracted search results represent video data only. However, other data types can be selected by changing the used search_params value, which can be extracted from the search URL.
How to Scrape YouTube Channels?
In this section, we’ll explore scraping YouTube channel metadata, which represents general information about channels. The easiest way to retrieve this data in browsers is using the dedicated channel info view:
Clicking the above view sends an XHR call to retrieve channel data as JSON, which later gets rendered.
Let’s replicate the above XHR call within our YouTube scraper to extract channel metadata:
import json
import asyncio
import jmespath
from jsonpath_ng.ext import parse
from typing import Dict, List
from loguru import logger as log
from Webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebparsersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube web scraping blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
jp_first = lambda query, data: (
parse(query).find(data)[0].value if parse(query).find(data) else None
)
async def call_youtube_api(
base_url: str,
continuation_token: str = None,
search_query: str = None,
search_params: str = None,
) -> List[Dict]:
"""call the YouTube comments API for continuation or search queries"""
# previous function definition
def parse_channel(response: ScrapeApiResponse) -> Dict:
"""parse channel metadata from YouTube channel page"""
_xhr_calls = response.scrape_result["browser_data"]["xhr_call"]
info_call = [c for c in _xhr_calls if "youtube.com/youtubei/v1/browse" in c["url"]]
data = json.loads(info_call[0]["response"]["body"]) if info_call else None
metadata = jp_first("$..aboutChannelViewModel", data)
links = []
if "links" in metadata:
for i in metadata["links"]:
i = i["channelExternalLinkViewModel"]
links.append(
{
"title": i["title"]["content"],
"url": i["link"]["content"],
"favicon": i["favicon"],
}
)
result = jmespath.search(
"""{
description: description,
url: displayCanonicalChannelUrl,
subscriberCount: subscriberCountText,
videoCount: videoCountText,
viewCount: viewCountText,
joinedDate: joinedDateText.content,
country: country
}""",
metadata,
)
result["links"] = links
return result
async def scrape_channel(channel_ids: List[str]) -> List[Dict]:
"""scrape channel metadata from YouTube channel pages"""
to_scrape = [
ScrapeConfig(
f"https://www.youtube.com/@{channel_id}",
proxy_pool="public_residential_pool",
**BASE_CONFIG,
render_js=True,
wait_for_selector="//yt-description-preview-view-model//button",
js_scenario=[
# click on the "show more" button to load the full description
{
"click": {
"selector": "//yt-description-preview-view-model//button",
"ignore_if_not_visible": False,
"timeout": 10000,
}
},
{
"wait_for_selector": {
"selector": "//yt-formatted-string[@title='About']",
"timeout": 10000,
}
},
],
)
for channel_id in channel_ids
]
data = []
log.info(f"scraping {len(to_scrape)} channels")
async for response in Scraper.concurrent_scrape(to_scrape):
channel_data = parse_channel(response)
data.append(channel_data)
log.success(f"scraped {len(data)} cahnnel info")
return dataAbove, we rely on the XHR call responsible for fetching channel metadata. However, instead of calling the API endpoint directly, we follow another approach explained in these steps:
- Simulate a click action using the headless browser to trigger the metadata XHR call.
- Extract the XHR call response and parse it using the
parse_channelfunction.
Below is an example output of the retrieved results:
Example output
Scraping Channel Videos
Now that our YouTube scraper can extract channel metadata, let’s scrape channel video data. For this, we’ll rely on another hidden YouTube API. First, let’s inspect it by navigating to any YouTube channel and scrolling down to load more video data.
To scrape channel video data, let’s replicate the above API call while manipulating its payload for pagination:
import re
import json
import asyncio
import jmespath
from jsonpath_ng.ext import parse
from typing import Dict, List, Literal
from loguru import logger as log
from Webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebparsersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube scraping blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
jp_all = lambda query, data: [match.value for match in parse(query).find(data)]
jp_first = lambda query, data: (
parse(query).find(data)[0].value if parse(query).find(data) else None
)
def parse_video_api(response: ScrapeApiResponse) -> Dict:
"""parse video data from YouTube API response"""
parsed_videos = []
data = json.loads(response.content)
continuation_tokens = jp_all("$..continuationCommand.token", data)
# first API response includes indexing data
videos = jp_all("$..reloadContinuationItemsCommand.continuationItems", data)
videos = videos[-1] if len(videos) > 1 else jp_first("$..continuationItems", data)
for i in videos:
if "richItemRenderer" not in i:
continue
result = jmespath.search(
"""{
videoId: videoId,
title: title.runs[0].text,
description: descriptionSnippet.runs[0].text,
publishedTime: publishedTimeText.simpleText,
lengthText: lengthText.simpleText,
viewCount: viewCountText.simpleText,
thumbnails: thumbnail.thumbnails
}""",
i["richItemRenderer"]["content"]["videoRenderer"],
)
result["url"] = f"https://youtu.be/{result['videoId']}"
parsed_videos.append(result)
return {
"videos": parsed_videos,
"continuationToken": continuation_tokens[-1] if continuation_tokens else None,
}
def parse_yt_initial_data(response: ScrapeApiResponse) -> Dict:
"""parse ytInitialData script from YouTube pages"""
selector = response.selector
data = selector.xpath("//script[contains(text(),'ytInitialData')]/text()").get()
data = json.loads(
re.search(r"var ytInitialData = ({.*});", data, re.DOTALL).group(1)
)
return data
async def scrape_channel_videos(
channel_id: str,
sort_by: Literal["Latest", "Popular", "Oldest"] = "Latest",
max_scrape_pages: int = None,
) -> List[Dict]:
"""scrape video metadata from YouTube channel page"""
# 1. extract the continuation token from the HTML to call the API
response = await Scraper.async_scrape(
ScrapeConfig(
f"https://www.youtube.com/@{channel_id}/videos",
proxy_pool="public_residential_pool",
**BASE_CONFIG,
)
)
initial_script_data = parse_yt_initial_data(response)
continuation_tokens = jp_all("$..chipCloudChipRenderer", initial_script_data)
# there are different continuation tokens based on the sorting order
continuation_token = [
i["navigationEndpoint"]["continuationCommand"]["token"]
for i in continuation_tokens
if i["text"]["simpleText"] == sort_by
][0]
# 2. call the API to get the video data
videos = []
cursor = 0
while continuation_token and (
cursor < max_scrape_pages if max_scrape_pages else True
):
cursor += 1
log.info(f"scraping video page with index {cursor}")
try:
response = await call_youtube_api(
base_url="https://www.youtube.com/youtubei/v1/browse?key=yt_web",
continuation_token=continuation_token,
)
except NameError:
log.error("call_youtube_api isn't defined. You can define it from the ealier snippet.")
break
data = parse_video_api(response)
videos.extend(data["videos"])
continuation_token = data["continuationToken"]
log.success(f"scraped {len(videos)} video for the channel {channel_id}")
return videosThe above code snippet may seem comprehensive. However, examining its execution flow makes it easier to understand:
- A request is sent to the URL pattern for channel videos:
youtube.com/@<channel_id>/videosto get the HTML response containing the first batch of videos. - The retrieved HTML is parsed using the
parse_yt_initial_datafunction to get thecontinuation_tokens, which will be used with the hidden API. - A while loop is created to keep paginating results until either the maximum number of results or pages to scrape is reached.
- The hidden YouTube API for channel video data is requested using the
call_youtube_apifunction and its response is refined with theparse_video_apifunction.
Below is an example output of the results extracted by the above YouTube scraping code:
Example output
So far, we have been able to crawl YouTube for video data from channels and search pages. Next, let’s scrape the YouTube video pages themselves!
How to Scrape YouTube Videos?
Video metadata is saved into HTML as JSON datasets within script tags. To identify them, search for the XPath selector //script[contains(text(),'ytInitialPlayerResponse')]/text() from the browser developer tools.
Hidden JSON data of videos metadata
As shown in the above image, the script tag contains the full video metadata. Let’s update our YouTube scraper to extract them:
import re
import json
import asyncio
from jsonpath_ng.ext import parse
from typing import Dict, List
from loguru import logger as log
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebparsersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube.com web scraping blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
jp_all = lambda query, data: [match.value for match in parse(query).find(data)]
jp_first = lambda query, data: (
parse(query).find(data)[0].value if parse(query).find(data) else None
)
def convert_to_number(value):
if value is None:
return None
value = value.strip().upper()
if value.endswith("K"):
return int(float(value[:-1]) * 1_000)
elif value.endswith("M"):
return int(float(value[:-1]) * 1_000_000)
else:
return int(float(value))
def parse_video_details(response: ScrapeApiResponse) -> Dict:
"""parse video metadata from YouTube video page"""
selector = response.selector
video_details = selector.xpath(
"//script[contains(text(),'ytInitialPlayerResponse')]/text()"
).get()
video_details = json.loads(video_details.split(" = ")[1].split(";var")[0]).get(
"videoDetails"
)
return video_details
def parse_yt_initial_data(response: ScrapeApiResponse) -> Dict:
"""parse ytInitialData script from YouTube pages"""
selector = response.selector
data = selector.xpath("//script[contains(text(),'ytInitialData')]/text()").get()
data = json.loads(
re.search(r"var ytInitialData = ({.*});", data, re.DOTALL).group(1)
)
return data
def parse_video(response: ScrapeApiResponse) -> Dict:
"""parse video metadata from YouTube video page"""
video_details = parse_video_details(response)
content_details = parse_yt_initial_data(response)
likes = [
i["title"]
for i in jp_all("$..buttonViewModel", content_details)
if "iconName" in i and i["iconName"] == "LIKE"
]
channel_id = jp_first(
"$..channelEndpoint.browseEndpoint.canonicalBaseUrl", content_details
)
verified = jp_all(
"$..videoOwnerRenderer..badges[0].metadataBadgeRenderer", content_details
)
result = {
"video": {
"videoId": video_details.get("videoId"),
"title": video_details.get("title"),
"publishingDate": jp_first("$..dateText.simpleText", content_details),
"lengthSeconds": convert_to_number(video_details.get("lengthSeconds")),
"keywords": video_details.get("keywords"),
"description": video_details.get("shortDescription"),
"thumbnail": video_details.get("thumbnail").get("thumbnails"),
"stats": {
"viewCount": convert_to_number(video_details.get("viewCount")),
"likeCount": convert_to_number(likes[0]) if likes else None,
"commentCount": convert_to_number(
jp_first("$..contextualInfo.runs[0].text", content_details)
),
},
},
"channel": {
"name": video_details.get("author"),
"identifierId": video_details.get("channelId"),
"id": channel_id.replace("/", "") if channel_id else None,
"verified": (
True
if verified and [i for i in verified if i["tooltip"] == "Verified"][0]
else False
),
"channelUrl": (
f"https://www.youtube.com{channel_id}" if channel_id else None
),
"subscriberCount": jp_first(
"$..subscriberCountText.simpleText", content_details
),
"thumbnails": jp_first(
"$..engagementPanelSectionListRenderer..channelThumbnail.thumbnails",
content_details,
),
},
"commentContinuationToken": jp_first(
"$..continuationCommand.token", content_details
),
}
return result
async def scrape_video(ids: List[str]) -> List[Dict]:
"""scrape video metadata from YouTube videos"""
data = []
to_scrape = [
ScrapeConfig(f"https://youtu.be/{video_id}", proxy_pool="public_residential_pool", **BASE_CONFIG)
for video_id in ids
]
log.info(f"scraping {len(to_scrape)} video metadata from video pages")
async for response in Scraper.concurrent_scrape(to_scrape):
post_data = parse_video(response)
data.append(post_data)
log.success(f"scraped {len(data)} video metadata from video pages")
return dataIn the above code, we define a scrape_video function that takes a list of video IDs as input, adds them to a scraping list, and concurrently requests the video page URLs. Then, we utilize the parse_video function to extract video and channel metadata from HTML using the hidden data extraction approach in both parse_video_details and parse_yt_initial_data functions.
Here’s what the extracted video data looks like:
Example output
In the JSON dataset above, our YouTube scraper has successfully extracted metadata for both videos and related channels. Additionally, we have the key commentContinuationToken that we’ll use for video comment scraping. Let’s see it in action in the following section!
Scraping Video Comments
To scrape YouTube comments, we’ll use the hidden API scraping approach. First, let’s identify the comments API. Go to any YouTube video page and scroll down the comments section to load more comments while having browser developer tools open. You will find a similar XHR call captured.
Hidden YouTube comments API
To scrape video comments, we’ll replicate the above XHR call:
import re
import json
import asyncio
import jmespath
from jsonpath_ng.ext import parse
from typing import Dict, List
from loguru import logger as log
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebparsersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube.com web scraping blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
jp_all = lambda query, data: [match.value for match in parse(query).find(data)]
jp_first = lambda query, data: (
parse(query).find(data)[0].value if parse(query).find(data) else None
)
def parse_comments_api(response: ScrapeApiResponse) -> List[Dict]:
"""parse comments API response for comment data"""
parsed_comments = []
data = json.loads(response.content)
continuation_tokens = jp_all("$..continuationCommand.token", data)
comments = jp_all("$..commentEntityPayload", data)
for comment in comments:
result = jmespath.search(
"""{
comment: {
id: properties.commentId,
text: properties.content.content
publishedTime: properties.publishedTime
},
author: {
id: author.channelId,
displayName: author.displayName,
avatarThumbnail: author.avatarThumbnailUrl,
isVerified: author.isVerified,
isCurrentUser: author.isVerified,
isCreator: author.isVerified
},
stats: {
likeCount: toolbar.likeCountLiked,
replyCount: toolbar.replyCount
}
}""",
comment,
)
parsed_comments.append(result)
return {
"comments": parsed_comments,
"continuationToken": continuation_tokens[-1] if continuation_tokens else None,
}
async def scrape_comments(video_id: str, max_scrape_pages=None) -> List[Dict]:
"""scraper comments from a YouTube video"""
comments = []
cursor = 0
log.info(f"scraping video page for the comments continuation token")
try:
video_data = await scrape_video([video_id])
except NameError:
log.error("scrape_video function is not defined. You can define it from the ealier snippet.")
return
continuation_token = video_data[0].get("commentContinuationToken")
while continuation_token and (
cursor < max_scrape_pages if max_scrape_pages else True
):
cursor += 1
log.info(f"scraping comments page with index {cursor}")
try:
response = await call_youtube_api(
base_url="https://www.youtube.com/youtubei/v1/next?prettyPrint=false",
continuation_token=continuation_token,
)
except NameError:
log.error("call_youtube_api function is not defined. You can define it from the search scraping section.")
return
data = parse_comments_api(response)
comments.extend(data["comments"])
continuation_token = data["continuationToken"]
log.success(f"scraped {len(comments)} comments for the video {video_id}")
return commentsTo request the hidden comments API, we first must obtain the commentContinuationToken. Therefore, we start our comment scraper by extracting using the scrape_video function we defined earlier. Then, we use the obtained token to call the YouTube API while using the parse_comments_api function to parse API responses.
Below is an example output of the extracted data:
Example output
Scraping YouTube Shorts
YouTube shorts have a different UI and media player than regular YouTube videos. However, both can be scraped in the same way using hidden data extraction from script tags.
This means we can reuse our previous parsing logic used while scraping YouTube videos:
import json
import asyncio
from typing import Dict, List
from loguru import logger as log
from webparsers import ScrapeConfig, WebparsersClient, ScrapeApiResponse
Scraper = WebpasersersClient(key="Your Webparsers API key")
BASE_CONFIG = {
# bypass youtube.com web scraping blocking
"asp": True,
# set the proxy country to US
"country": "US",
}
def parse_video_details(response: ScrapeApiResponse) -> Dict:
"""parse video metadata from YouTube video page"""
selector = response.selector
video_details = selector.xpath(
"//script[contains(text(),'ytInitialPlayerResponse')]/text()"
).get()
video_details = json.loads(video_details.split(" = ")[1].split(";var")[0]).get(
"videoDetails"
)
return video_details
async def scrape_shorts(ids: List[str]) -> List[Dict]:
"""scrape metadata from YouTube shorts"""
to_scrape = [
ScrapeConfig(
f"https://youtu.be/{short_id}",
proxy_pool="public_residential_pool",
**BASE_CONFIG,
)
for short_id in ids
]
data = []
log.info(f"scraping {len(to_scrape)} short video metadata from video pages")
async for response in Scraper.concurrent_scrape(to_scrape):
post_data = parse_video_details(response)
post_data["thumbnail"] = post_data["thumbnail"]["thumbnails"]
data.append(post_data)
log.success(f"scraped {len(data)} video metadata from short pages")
return dataThe above YouTube scraper snippet is fairly straightforward. We request the shorts’ URLs and then parse their data from script tags using the parse_video_details function.
Below is an example output of the results we got:
Example output
Powering Up With Webparsers
We have explored scraping different parts of YouTube by either requesting HTML web pages or calling hidden APIs. That being said, on such a highly protected domain like YouTube, attempting to scale our scraper will lead us to getting blocked. YouTube can detect us sending a large number of requests in a short time window, hence getting our IP address blocked.
Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.
- Anti-bot protection bypass – scrape web pages without blocking!
- Rotating residential proxies – prevent IP address and geographic blocks.
- JavaScript rendering – scrape dynamic web pages through cloud browsers.
- Full browser automation – control browsers to scroll, input and click on objects.
- Format conversion – scrape as HTML, JSON, Text, or Markdown.
- Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.
Here’s how to use it to bypass YouTube web scraping blocking. All we have to do is enable the anti-scraping protection bypass (asp=True) and select a proxy country:
# standard web scraping code
import httpx
from parsel import Selector
response = httpx.get("some youtube.com URL")
selector = Selector(response.text)
# in Webparsers becomes this 👇
from webparsers import ScrapeConfig, WebparsersClient
# replaces your HTTP client (httpx in this case)
Scraper = webparsersClient(key="Your Webparsers API key")
response = scraper.scrape(ScrapeConfig(
url="web page URL",
asp=True, # enable the anti scraping protection to bypass blocking
country="US", # set the proxy location to a specfic country
proxy_pool="public_residential_pool", # select a proxy pool
render_js=True # enable rendering JavaScript (like headless browsers) to scrape dynamic content if needed
))
# use the built in Parsel selector
selector = response.selector
# access the HTML content
html = response.scrape_result['content']Try for FREE!
More on Webparsers
FAQ
To wrap up this guide, let’s look at a few commonly asked questions about web scraping YouTube.
Are there public APIs for YouTube?
Yes, public YouTube APIs are available through the Google developer console. It covers various data sources, including channels, videos, and search functionality. For more details, refer to the official YouTube API documentation.
What are the limitations of YouTube API?
Google provides public access to YouTube APIs. However, such access is limited by a daily quota system. Such a system can be a limiting factor for scaled YouTube scrapers.
Can I scrape YouTube for sentiment analysis?
Additionally, obtaining the necessary API keys involves setting up a new project on the Google Developer Console, which can be complicated for those new to the process and platform.
Yes, scraping YouTube comments allows the extraction of large amounts of text data, which can be used to run sentiment analysis campaigns on given topics. For more, refer to our guide on using web scraping for sentiment analysis.
Web Scraping YouTube Summary
In this guide, we explained how to scrape YouTube through a step-by-step approach. We were able to extract data from YouTube from various resources:
- Search pages for video search results
- Channel pages for channel and video metadata
Legal Disclaimer and Precautions
This tutorial covers popular web scraping techniques for education. Interacting with public servers requires diligence and respect and here’s a good summary of what not to do:
- Do not scrape at rates that could damage the website.
- Do not scrape data that’s not available publicly.
- Do not store PII of EU citizens who are protected by GDPR.
- Do not repurpose the entire public datasets which can be illegal in some countries.
Webparsers does not offer legal advice but these are good general rules to follow in web scraping and for more you should consult a lawyer.
Video, shorts, and comment data
Instead of parsing HTML using XPath and CSS selectors, we developed our YouTube scraper using two common approaches. We used YouTube’s hidden APIs and the hidden data parsing approach to extract YouTube data directly as JSON.