Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Key Takeaways
Making Connection
HTTP clients: crul
Understanding Requests and Responses
Request Method
Request Location
Request Headers
Response Status Code
Response Headers
Making GET Requests
Making POST requests
Ensuring Headers
Tracking cookies
Asynchronous (Parallel) Requests
Parsing HTML: Rvest
Example Project
Webparsers
Summary

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

In this comprehensive guide, we’ll explore web scraping in R using best practices. We’ll examine fast asynchronous HTTP connections, techniques to avoid basic blocking, and methods to parse HTML files for data extraction.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

As shown in the illustration above, this protocol involves multiple components like request method, location, headers, etc. Before exploring these elements, we should select an appropriate HTTP client for R!

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

HTTP involves substantial waiting time – every request requires the client to wait for server responses, blocking code execution. However, if we can establish multiple concurrent connections, we can eliminate this blocked waiting time!

For example, 1 synchronous request takes .1 second of processing and 2 seconds of waiting, compared to 10 asynchronous requests which take 10×.1 second of processing and 1×2 seconds of waiting – a significant performance difference!

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

While web-scraping doesn’t require comprehensive knowledge of HTTP requests and responses, having a general overview is beneficial, especially understanding which parts are particularly useful in web-scraping scenarios.

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

For web scraping, we’re primarily interested in collecting documents, so GET and POST requests are our main focus. Additionally, HEAD requests can optimize bandwidth – sometimes we need to check document metadata before downloading to determine if it’s worth the effort.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

While request headers might seem like minor metadata details, they’re extremely important in web scraping. Headers contain essential information about the request: who’s requesting the data? What type of data are they expecting? Incorrect headers might result in access denial for the web scraper.

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

When visiting a web page, your browser identifies itself with a User-Agent string resembling “Browser Name, Operating System, Some version numbers”. This helps servers determine whether to serve or deny the client. In web scraping, to avoid content denial, we must blend in by mimicking browser user agents.

Many online databases contain latest user-agent strings for various platforms, like this

Cookie is used to store persistent data. This is essential for websites to track user state: user logins, configuration preferences, etc. Cookies are somewhat beyond this article’s scope, but we’ll cover them in future content.

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

Conveniently, all HTTP responses include a status code indicating whether the request succeeded, failed, or requires alternative action (like authentication). Here are status codes most relevant to web scraping:

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

For web scraping, response headers provide important information for connection functionality and efficiency. For example, the Set-Cookie header requests our client to save cookies for future requests, which might be vital for website functionality. Other headers like Etag, Last-Modified help clients with caching to optimize resource usage.

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

> library("crul")
> response <- HttpClient$new('https://httpbin.dev/headers')$get()
> 
> # response url - it can be different from above if redirect happened
> print(response$url)
[1] "https://httpbin.dev/headers"
> # status code:
> print(response$status_code)
[1] 200
> # check whether response succeeded, i.e. status code <= 201
> print(response$success())
[1] TRUE
> # response headers:
> print(response$response_headers)
$status
[1] "HTTP/2 200"
$date
[1] "Wed, 02 Mar 2022 08:28:04 GMT"

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

content-type` [1] "application/json"

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

content-length` [1] "286" $server [1] "gunicorn/19.9.0"

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

access-control-allow-origin` [1] "*"

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

access-control-allow-credentials` [1] "true" > # response binary content > print(response$content) [1] 7b 0a 20 20 22 68 65 61 64 65 72 73 22 3a 20 7b 0a 20 20 20 20 22 41 63 63 [26] 65 70 74 22 3a 20 22 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e 2c 20 [51] 74 65 78 74 2f 78 6d 6c 2c 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d 6c [76] 2c 20 2a 2f 2a 22 2c 20 0a 20 20 20 20 22 41 63 63 65 70 74 2d 45 6e 63 6f [101] 64 69 6e 67 22 3a 20 22 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 22 2c 20 0a [126] 20 20 20 20 22 48 6f 73 74 22 3a 20 22 68 74 74 70 62 69 6e 2e 6f 72 67 22 [151] 2c 20 0a 20 20 20 20 22 55 73 65 72 2d 41 67 65 6e 74 22 3a 20 22 6c 69 62 [176] 63 75 72 6c 2f 37 2e 38 31 2e 30 20 72 2d 63 75 72 6c 2f 34 2e 33 2e 32 20 [201] 63 72 75 6c 2f 31 2e 32 2e 30 22 2c 20 0a 20 20 20 20 22 58 2d 41 6d 7a 6e [226] 2d 54 72 61 63 65 2d 49 64 22 3a 20 22 52 6f 6f 74 3d 31 2d 36 32 31 66 32 [251] 61 39 34 2d 33 31 32 61 66 38 33 62 33 33 63 37 32 35 34 65 33 33 34 36 39 [276] 64 30 39 22 0a 20 20 7d 0a 7d 0a > # response content as text () > print(response$parse()) No encoding supplied: defaulting to UTF-8. [1] "{\n \"headers\": {\n \"Accept\": \"application/json, text/xml, application/xml, */*\", \n \"Accept-Encoding\": \"gzip, deflate\", \n \"Host\": \"httpbin.dev\", \n \"User-Agent\": \"libcurl/7.81.0 r-curl/4.3.2 crul/1.2.0\", \n \"X-Amzn-Trace-Id\": \"Root=1-621f2a94-312af83b33c7254e33469d09\"\n }\n}\n" > # can also load text json response to R's named list: > jsonlite::fromJSON(response$parse()) No encoding supplied: defaulting to UTF-8. $headers $headers$Accept [1] "application/json, text/xml, application/xml, */*" $headers

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

Accept-Encoding` [1] "gzip, deflate" $headers$Host [1] "httpbin.dev" $headers

Web Scraping With R Tutorial and Example Project

R programming language is one of the most popular languages used in modern data science and web-scraping can often be used to efficiently generate datasets right there in the R language itself!

The R language ecosystem provides all essential features needed for web scraping: HTTP clients, HTML parsers and various data-processing utilities.

To demonstrate these concepts, we’ll build a practical example – a job listing information scraper for https://www.indeed.com/!

Key Takeaways

Master web scraping in R using crul HTTP client and rvest parser for efficient data extraction with asynchronous requests and robust HTML parsing.

Use crul HTTP client for R web scraping with excellent asynchronous request support and modern API design
Parse HTML with rvest library for powerful CSS selector-based data extraction in R environments
Handle concurrent scraping with crul async functionality for improved performance and efficiency
Implement proper error handling and retry logic using R’s error handling patterns and try-catch blocks
Use CSS selectors for flexible HTML element targeting and data extraction in R data science workflows
Build scalable R scrapers with proper session management and data processing pipelines

Making Connection

Web scraping generally consists of two primary steps: retrieving data and parsing data. In this section, we’ll focus on data retrieval, which is accomplished through HTTP connections.

To access public resources, we (the client) must connect to the server and request the document data. This HTTP interaction consists of requests and responses:

illustration of a standard http exchange
illustration of a standard HTTP exchange

HTTP clients: crul

For handling HTTP connections, we need an HTTP client library. R language offers two primary competing libraries:

httr – an older de facto standard client that comes with all of the basic http functions.
crul – a newer take on HTTP protocol with modern client API and asynchronous functionality.

In this article, we’ll use crul as it offers essential functionality for web scraping – asynchronous (parallel) requests which are crucial for fast scraping.

Let’s set up our R environment with the crul package to explore the HTTP protocol:

> install.packages("crul", repos = "https://dev.ropensci.org")

Understanding Requests and Responses

Request Method

HTTP requests are organized into several types that serve distinct purposes. In web scraping, we commonly encounter these request types:

GET requests are intended to request a document.
POST requests are intended to request a document by sending a document.
HEAD requests are intended to request documents meta information.

Other methods less frequently encountered in web scraping but worth knowing include:

PATCH requests are intended to update a document.
PUT requests are intended to either create a new document or update it.
DELETE requests are intended to delete a document.

Request Location

Every HTTP request must specify which resource is being requested through a URL (Universal Resource Location), which has a detailed structure:

illustration showing general URL structure
Example of a URL structure

Here, we can visualize each part of a URL:

protocol – when it comes to HTTP is either http or https.
host – address of the server, e.g. domain name or IP address.
location – relative location of the resource at host.

If you’re ever unsure of a URL’s structure, you can always fire up R interactive shell (R in the terminal) and let it figure it out for you by using crul library’s url_parse function:

$ R
> crul::url_parse("http://www.domain.com/path/to/resource?arg1=true&arg2=false")
$scheme
[1] "http"
$domain
[1] "www.domain.com"
$port
[1] NA
$path
[1] "path/to/resource"
$parameter
$parameter$arg1
[1] "true"
$parameter$arg2
[1] "false"
$fragment
[1] NA

Request Headers

Let’s examine some of the most important headers and their meanings:

User-Agent is an identity header that tells the server who’s requesting the document.

# example user agent for Chrome browser on Windows operating system:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36

Many online databases contain latest user-agent strings for various platforms, like this

Accept headers (also Accept-Encoding, Accept-Language etc.) contain information about expected content types. Generally, when web-scraping, we want to mimic popular web browsers like Chrome:

text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

For more, see

X- prefixed headers are special custom headers. These require attention during web scraping, as they might configure important website/webapp functionality.

These are some of the most important observations, for more see extensive full documentation page on http headers

Response Status Code

200 range codes generally mean success!
300 range codes typically mean redirection – if we request content at /product1.html it might be moved to /products/1.html and the server would inform us about that.
400 range codes mean the request is malformed or denied. Our web scraper might be missing headers, cookies or authentication details.
500 range codes typically indicate server issues. The website might be unavailable or purposefully blocking our web scraper.

Response Headers

Finally, similar to request headers, headers prefixed with X- are custom web functionality headers.

We’ve briefly covered core HTTP components, and now it’s time to implement this knowledge practically in R!

Making GET Requests

Now that we understand the HTTP protocol and its application in web-scraping, let’s implement it using R’s crul library.

Let’s begin with a basic GET request:

library("crul")
response <- HttpClient$new('https://httpbin.dev/headers')$get()

# response url - it can be different from above if redirect happened
print(response$url)
# status code:
print(response$status_code)
# check whether response succeeded, i.e. status code <= 201
print(response$success())
# response headers:
print(response$response_headers)

# response binary content
print(response$content)
# response content as text ()
print(response$parse())
# can also load text json response to R's named list:
jsonlite::fromJSON(response$parse())

Here we’re using https://httpbin.dev/ HTTP testing service, specifically the /headers endpoint which displays request headers the server received from us.

When executed, this script should print basic details about our request:

User-Agent` [1] "libcurl/7.81.0 r-curl/4.3.2 crul/1.2.0"

Making POST requests

Sometimes web scrapers need to submit forms to retrieve HTML results. For example, search queries often use POST requests with query details as JSON or Formdata values:

library("crul")

# send form type post request:
response <- HttpClient$new('https://httpbin.dev/post')$post(
    body = list("query" = "cats", "page" = 1),
    encode = "form",
)
print(jsonlite::fromJSON(response$parse()))

# or json type post request:
response <- HttpClient$new('https://httpbin.dev/post')$post(
    body = list("query" = "cats", "page" = 1),
    encode = "json",
)
print(jsonlite::fromJSON(response$parse()))

Ensuring Headers

As previously discussed, our requests must provide metadata about themselves, helping servers determine what content to return. Often, this metadata can identify and block web scrapers. Modern web browsers automatically include specific metadata with every request, so to avoid detection as a web scraper, we should replicate this behavior.

Primarily, User-Agent and Accept headers are often telltale signs, so we should set them to common values. This can be done globally or per request:

library("crul")

# we can set headers for every request
response <- HttpClient$new("https://httpbin.dev/headers",
    headers = list(
        "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
        "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
    )
)$get()
print(jsonlite::fromJSON(response$parse())$headers)

# or set headers for the whole script (recommended)
set_headers(
    "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
    "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
)
response <- HttpClient$new("https://httpbin.dev/headers")$get()
print(jsonlite::fromJSON(response$parse())$headers)

In the example above, we’re setting headers to mimic Chrome browser on Windows platform. This simple change can prevent much web scraping blocking and is recommended for every web scraper.

Tracking cookies

Sometimes during web-scraping, we need persistent connection state. For websites requiring login or configuration (like currency changes), cookies are a vital part of the web scraping process.

Crul supports cookie tracking per HttpClient basis, meaning all requests attached to one client object share cookies:

library("crul")

session <- HttpClient$new('https://httpbin.dev/')
# set some cookies:
resp_set_cookies <- session$get('/cookies/set/foo/bar')
# see current cookies:
resp_retrieve_cookies <- session$get('/cookies')
print(resp_retrieve_cookies$parse())

In the example above, we’re using httpbin.dev’s /cookies endpoint to set session cookies. Once cookies are set, we’re redirected to a page displaying sent cookies:

{
  "cookies": {
    "foo": "bar"
  }
}

Now that we understand HTTP in R and crul, let’s examine connection speed optimization! How can we make these connections faster and more efficient?

Asynchronous (Parallel) Requests

Since HTTP is a data exchange protocol between two parties, substantial waiting is involved. When our client sends a request, it must wait for round-trip travel to the server, stalling our program. Why should our program wait idly for requests to travel globally? This is called an IO (input/output) block.

We chose R’s crul package over httr specifically for this feature – it makes asynchronous requests very accessible:

library("crul")

start = Sys.time()
responses <- Async$new(
    urls = c(
        "https://httpbin.dev/links/4/0",
        "https://httpbin.dev/links/4/1",
        "https://httpbin.dev/links/4/2",
        "https://httpbin.dev/links/4/3"
    ),
)$get()
print(responses)
print(Sys.time() - start)

In the example above, we’re batching multiple urls for concurrent execution. Alternatively, we can execute varying different requests:

library("crul")

start = Sys.time()
responses <- AsyncVaried$new(
        HttpRequest$new("https://httpbin.dev/links/4/0")$get(),
        HttpRequest$new("https://httpbin.dev/links/4/1")$get(),
        HttpRequest$new("https://httpbin.dev/links/4/2")$get(),
        HttpRequest$new("https://httpbin.dev/links/4/3", headers=list("User-Agent"="different"))$get(),
        HttpRequest$new("https://httpbin.dev/post")$post(body=list(query="cats", page = 1))
)$request()
print(responses)
print(Sys.time() - start)

This approach allows mixing varying request types and parameters.

Now that we’re comfortable with HTTP protocol in R’s crul, let’s examine how to extract meaningful data from retrieved HTML. In the next section, we’ll explore HTML parsing using CSS and XPATH selectors in R!

Parsing HTML: Rvest

Retrieving HTML documents is only half the web scraping process – we must also parse them for the data we seek. Fortunately, HTML format is designed for machine parsing, so we can leverage special CSS or XPATH selector languages to find exact page sections for extraction.

We’ve covered both CSS and XPATH selectors extensively in previous articles:

Parsing HTML with CSS Selectors
Introduction to using CSS selectors to parse web-scraped content. Best practices, available tools and common challenges by interactive examples.

Parsing HTML with CSS Selectors

Parsing HTML with Xpath
Introduction to xpath in the context of web-scraping. How to extract data from HTML documents using xpath, best practices and available tools.

Parsing HTML with Xpath

In R, there’s one library supporting both CSS and XPATH selectors: rvest

Let’s examine common rvest use cases:

library("rvest")

tree <- read_html('
<div class="links">
  <a href="https://twitter.com/@webparsers_dev">Twitter</a>
  <a href="https://www.linkedin.com/company/webparsers/">LinkedIn</a>
</div>
')

# we can execute basic css selectors and pull all text values:
print(tree %>% html_element("div.links") %>% html_text())
# "[1] "\n  Twitter\n  LinkedIn\n""

# we can also execute xpath selectors:
print(tree %>% html_element(xpath="//div[@class='links']") %>% html_text())
# "[1] "\n  Twitter\n  LinkedIn\n""

# html_text2 - outputs are cleaned fo trailing/leading space characters:
print(tree %>% html_element("div") %>% html_text2())
# "[1] "Twitter LinkedIn""

# we can select attribute of a single element:
print(tree %>% html_element("div") %>% html_attr('class'))
# "links"

# or attributes of multiple elements:
print(tree %>% html_elements("div.links a") %>% html_attr('href'))
# [1] "https://twitter.com/@webparsers_dev"         
# [2] "https://www.linkedin.com/company/webparsers/"

The primary function here is R’s pipe symbol %>% which allows processing our HTML tree through multiple processors like XPATH or CSS selectors and text or attribute extractors.

Rvest also includes special parsing functionalities designed for data science use cases. For example, it converts HTML tables to R’s data frames:

library("rvest")

tree <- read_html('
<div class="table-wrapper">
<table>
  <th>
    <td>model</td>
    <td>year</td>
  </th>
  <tr>
    <td>Mazda</td>
    <td>2011</td>
  </tr>
  <tr>
    <td>Toyota</td>
    <td>1992</td>
  </tr>
</table>
</div>
')

tree %>% html_element('.table-wrapper') %>% html_table()
## A tibble: 2 × 2
#  X1        X2
# <chr>  <int>
# 1 Mazda   2011
# 2 Toyota  1992

In this example, the html_table() pipe function automatically extracts complete tables from given selectors containing <table> nodes. It captures table headers from <th> nodes and even converts values to appropriate types (here year values were converted to integers).

The best way to truly explore rvest is through example projects, so let’s do exactly that!

Example Project

To solidify our knowledge, we’ll build a web scraper for https://uk.indeed.com/.

We’ll scrape job listing data from a given search query for R jobs:

We’ll scrape indeed.com search page for location like https://uk.indeed.com/jobs?q=r&l=Scotland
Find first page of 10 job listings on the page by using rvest
Find the total amount of jobs/pages.
Scrape other job pages.

We’ll create a fast and easy-to-understand scraper by separating logic into single-purpose functions.

Let’s start our scraper from the bottom up by defining constants and parsing functions:

library("crul")
library("rvest")

HEADERS <- list(
    "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
    "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    "Accept-Encoding" = "gzip, deflate, br",
    "Accept-Language" = "en-US,en;q=0.9"
)

Here, we’re defining our HEADERS constant. To prevent scraper blocking, we need to ensure it resembles a web browser. For this, we’re copying headers a Chrome browser would use on Windows.

Next, we can define our parsing function. We’ll use CSS selectors to extract job details, and for that, we can use Chrome developer tools – see this quick introduction:

demo on how to use Chrome developer tools network inspector for web scraping

In R, our parser logic would look like this:

parse_search <- function(response){
    # build rvest tree
    tree <- read_html(response$parse())
    # find total jobs available
    total_jobs <- tree %>% html_element("#searchCountPages") %>% html_text2()
    total_jobs <- strtoi(stringr::str_match(total_jobs, "(\\d+) jobs")[,2])
    # find displayed job container boxes:
    jobs <- tree %>% html_elements('#mosaic-zone-jobcards .result')
    # extract job listing from each job container box
    parsed <- list()
    for (job in jobs){
        parsed <- append(parsed, list(
            title = job %>% html_element('h2') %>% html_text2(),
            company = job %>% html_element('.companyOverviewLink') %>% html_text2(),
            location = job %>% html_element('.companyLocation') %>% html_text2(),
            date = job %>% html_element(xpath=".//span[contains(@class,'date')]/text()") %>% html_text2(),
            url =  url_build(response$url, job %>% html_attr('href')),
            company_url = url_build(response$url, job %>% html_element(xpath='.//a[contains(@class,"companyOverviewLink")]') %>% html_attr('href'))
        ))
    }
    # return parsed jobs and total job count in the query
    print(glue("found total {length(jobs)} jobs from total of {total_jobs}"))
    list(jobs=parsed, total=total_jobs)
}

Here we have a function that takes a response object, builds a rvest HTML tree, and using CSS and XPath selections, extracts job listing details.

We do this by first selecting job listing container boxes (the li elements). Then, we iterate through each one and extract the job details.

We can test this code by explicitly scraping 1 page:

url <- "https://uk.indeed.com/jobs?q=r&l=scotland")
response <- HttpClient$new(url, header=HEADERS)$get()
print(parse_search(response))

This will return results from the first query page and the total listing count:

found total 15 jobs from total of 330
$jobs
$jobs$title
[1] "Parts Supervisor"
$jobs$company
[1] "BMW Group Retail"
$jobs$location
[1] "Edinburgh"
$jobs$date
[1] "4 days ago"
$jobs$url
[1] "https://uk.indeed.com/rc/clk?jk=ad7698381a9870de&fccid=bf564b9e3f3db3fc&vjs=3"
$jobs$company_url
[1] "https://uk.indeed.com/cmp/BMW-Group"

<...>

$total
[1] 330

To complete our scraper, let’s add a scraping loop that will scrape the first page, parse results, then scrape remaining pages in parallel:

scrape_search_page <- function(query, location, offset=0, limit=10){
    # first we need to create all urls we'll be scraping based on offset and limit arguments
    # 0:10 will create first page scrape url, and 10:80 will create 1-8 pages since there are 10 results per page
    print(glue("scraping {query} at {location} in range {offset}:{limit}"))
    urls <- list()
    for (i in seq(offset+10, limit, by=10)){
        urls <- append(urls, glue("https://uk.indeed.com/jobs?q={query}&l={location}&start={i}"))
    }

    # then we want to retrieve these urls in parallel:
    print(glue("scraping search page urls: {urls}"))
    responses <- Async$new(
        urls = urls,
        headers = HEADERS,
    )$get()
    
    # finally we want to unpack results of each individual page into final dataset
    found_jobs <- list()
    total_jobs <- NULL
    for (response in responses){
        page_results <- parse_search(response)
        found_jobs <- append(found_jobs, page_results$jobs)
        total_jobs <- page_results$total
    }
    # we return jobs we parsed and total jobs presented in the search page:
    list(jobs=found_jobs, total=total_jobs)
}

scrape_search <- function(query, location){
    # this is our main function that scrapes all pages of the query explicitly
    # first, we scrape the first page
    first_page_results <- scrape_search_page(query, location)
    found_jobs <- first_page_results$jobs
    total_jobs <- first_page_results$total

    # then we scrape remaining pages: 
    print(glue("scraped first page, found {length(found_jobs)} jobs; continuing with remaining: {total_jobs}"))
    remaining_page_results <- scrape_search_page(query, location, offset = 10, limit = total_jobs)
    # finally, we return combined dataset
    append(found_jobs, remaining_page_results$jobs)
}

Here we’ve defined our scraping loop which utilizes crul’s parallel connection feature. Let’s test our scraper and time it:

start = Sys.time()
print(scrape_search("r", "Scotland"))
print(Sys.time() - start)

We can see that we scraped over 300 job listings in under 10 seconds! We can easily embed this scraper into our R workflow without worrying about caching or data storage since each dataset scrape takes only seconds to refresh.

Our example, while complete, isn’t production ready as web scraping faces many challenges like blocking and connection issues.

For this, let’s examine Webparsers’ web scraping API and how we can use it in R to handle these issues through the service rather than managing them ourselves!

Webparsers

Web scraping with R can be surprisingly accessible, however scaling R scrapers can be difficult due to lacking infrastructure for bypass blocking and scaling. This is where Webparsers can help!

webparsers middleware

Webparsers provides web scraping, screenshot, and extraction APIs for data collection at scale.

Anti-bot protection bypass – extract web pages without blocking!
Rotating residential proxies – prevent IP address and geographic blocks.
LLM prompts – extract data or ask questions using LLMs
Extraction models – automatically find objects like products, articles, jobs, and more.
Extraction templates – extract data using your own specification.
Python and Typescript SDKs, as well as Scrapy and no-code tool integrations.

Let’s examine how we can use Webparsers’ API to scrape content with all these features in R!

library(crul)

WEBPARSERS_KEY <- "YOUR_API_KEY"

webparsers <- function(url,
                     render_js = FALSE,
                     wait_for_selector = NULL,
                     asp = FALSE,
                     country = "us",
                     proxy_pool = "public_datacenter_pool") {
    url_build("https://api.webparsers.com/scrape", query = list(
        key = WEBPARSERS_KEY,
        url = url,
        asp = if (asp) "true" else "false",
        render_js = if (render_js) "true" else "false",
        wait_for_selector = wait_for_selector,
        country = country,
        proxy_pool = proxy_pool
    ))
}


response <- HttpClient$new(webparsers('https://httpbin.dev/headers'))$get()
response_with_javascript_render <- HttpClient$new(webparsers('https://httpbin.dev/headers', render_js = TRUE))$get()
data <- jsonlite::fromJSON(response$parse())
print(data$result$content)
print(data$result$response_headers)
print(data$result$cookies)
print(data$config)

In the example above, we define a function that converts regular web URLs into Webparsers API URLs, which we can pass to our requests to leverage all features with minimal effort.

For more features, usage scenarios and other information, see the service documentation!

Summary

In this comprehensive introduction to web scraping in R, we’ve explored HTTP connection strategies using the crul library: submitting GET and POST requests, modifying headers, tracking cookies, and implementing efficient asynchronous (parallel) requests.

With HTTP foundations established, we examined HTML parsing using rvest, which supports both CSS and XPath selector-based element extraction.

Finally, we combined everything in a practical web scraper example for https://uk.indeed.com/ and briefly looked at common web scraping challenges and how we can use Webparsers’ web scraping API to handle them for us!

Table of Contents

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Web Scraping With R Tutorial and Example Project

Key Takeaways

Making Connection

HTTP clients: crul

Understanding Requests and Responses

Request Method

Request Location

Request Headers

Response Status Code

Response Headers

Making GET Requests

Making POST requests

Ensuring Headers

Tracking cookies