Whether you’re an investor monitoring your financial portfolio performance or an investment firm seeking new investment opportunities, web scraping stock market data provides access to current financial information and trends to support your decision-making process.
In this comprehensive web scraping tutorial, we’ll demonstrate how to build a web scraping tool to extract stock market data using Python and BeautifulSoup. Additionally, this example will teach you how to monitor and extract multiple stock prices, then organize them in a CSV file for further analysis. Let’s get started.
How to Build a Stock Market Data Scraper
For this tutorial, we’ll be scraping investing.com to extract current stock prices from Microsoft, Coca-Cola, and Nike, then storing the data in a CSV file. We’ll also demonstrate how to protect your web-scraping bot from being blocked by anti-scraping mechanisms and techniques using Webparsers.
Note: The script will function to scrape stock market data even without Webparsers, but will be essential for scaling your project later.
Although we’ll guide you through every step of the stock market data extraction process, having some familiarity with the Beautiful Soup library beforehand is beneficial. If you’re completely new to this library, consider reviewing a Beautiful Soup tutorial for beginners. It’s filled with tips and tricks, and covers the fundamentals you need to know to scrape almost anything.
With that established, let’s dive into the code so you can learn how to scrape stock market data.
1. Setting Up Our Stock Market Web Scraping Project
To begin, we’ll create a folder named “scraper-stock-project”, and open it from VScode (you can use any text editor you prefer). Next, we’ll open a new terminal and install our two main dependencies for this project:
pip3 install bs4 pip3 install requests
After that, we’ll create a new file named “stockData-scraper.py” and import our dependencies to it.
import requests from bs4 import BeautifulSoup
With Requests, we’ll be able to send an HTTP request to download the HTML file which is then passed on to BeautifulSoup for parsing. So let’s test it by sending a request to Nike’s stock page:
url = 'https://www.investing.com/equities/nike' page = requests.get(url) print(page.status_code)
By printing the status code of the page variable (which is our request), we’ll confirm whether or not we can successfully scrape the page. The code we’re looking for is a 200, indicating it was a successful request.
Success! Before proceeding, we’ll pass the response stored in page to Beautiful Soup for parsing:
soup = BeautifulSoup(page.text, 'html.parser')
You can use any parser you want, but we’re using html.parser because it’s our preferred choice.
2. Inspect the Website’s HTML Structure (Investing.com)
Before we start scraping, let’s open https://www.investing.com/equities/nike in our browser to become more familiar with the website structure.
As you can see from the page, it displays the company’s name, stock symbol, price, and price change. At this point, we have three critical questions to answer:
- Is the data being injected with JavaScript?
- What attribute can we use to select the elements?
- Are these attributes consistent throughout all pages?
Check for JavaScript
There are several methods to verify if some script is injecting data, but the simplest way is to right-click and select View Page Source.
It appears there isn’t any JavaScript that could potentially interfere with our scraper. Next we’ll do the same for the rest of the information. Since we didn’t find any additional JavaScript, we’re ready to proceed.
Note: Checking for JavaScript is crucial because Requests can’t execute JavaScript or interact with the website, so if the information is behind a script, we would need to use other tools to extract it, like Selenium.
Selecting the CSS Selectors
Now let’s examine the HTML of the site to identify the attributes we can use to select the elements.
Extracting the company’s name and the stock symbol will be straightforward. We just need to target the H1 tag with class ‘text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2’.
However, the price, price change, and percentage change are separated into different spans.
Furthermore, depending on whether the change is positive or negative, the class of the element changes, so even if we select each span using their class attribute, there will still be instances when it won’t work.
The good news is that we have a technique to extract it. Because Beautiful Soup returns a parsed tree, we can now navigate the tree and select the element we want, even though we don’t have the exact CSS class.
What we’ll do in this scenario is move up in the hierarchy and find a parent div we can utilize. Then we can use find_all(‘span’) to create a list of all the elements containing the span tag – which we know our target data uses. And because it’s a list, we can now easily navigate it and pick those we need.
So here are our targets:
company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text
price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text
change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text
Now for a test run:
print('Loading: ', url)
print(company, price, change)
And here’s the result – it works perfectly!
3. Scrape Multiple Financial Stock Data
Now that our parser is functioning, let’s scale this up and scrape several stocks. After all, a script for tracking just one stock is likely not going to be very practical.
We can make our scraper parse and scrape multiple pages by creating a list of URLs and looping through them to output the data.
urls = [
'https://www.investing.com/equities/nike',
'https://www.investing.com/equities/coca-cola-co',
'https://www.investing.com/equities/microsoft-corp',
]
for url in urls:
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')
company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text
price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text
change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text
print('Loading: ', url)
print(company, price, change)
Here’s the result after running it – excellent, it works across all pages!
We can keep adding more and more pages to the list but eventually, we’ll encounter a significant roadblock: anti-scraping techniques.
4. Integrating Web Scraping Tools to Handle IP Rotation and CAPTCHAs
Not every website welcomes being scraped, and for valid reasons. When scraping a website, we need to keep in mind that we are sending traffic to it, and if we’re not careful, we could be limiting the bandwidth the website has for genuine visitors, or even increasing hosting costs for the owner. That said, as long as we follow web scraping best practices, we won’t encounter any problems with our projects, and we won’t cause the sites we’re scraping any issues.
However, it’s challenging for businesses to distinguish between ethical scrapers and those that will damage their sites. For this reason, most servers will be equipped with different systems like:
- Browser behavior profiling
- CAPTCHAs
- Monitoring the number of requests from an IP address in a time period
These measures are designed to recognize bots, and block them from accessing the website for days, weeks, or even permanently.
Instead of handling all of these scenarios individually, we’ll just add two lines of code to make our requests go through Webparsers’ servers and get everything automated for us.
First, let’s create a free account with the service to access our API key and 5000 free API credits for our project.
Now we’re ready to add to our loop a new params variable to store our key and target URL and use urlencode to construct the URL we’ll use to send the request inside the page variable.
params = {'api_key': 'YOUR_API_KEY', 'url': url}
page = requests.get('http://api.webparsers.com/', params=urlencode(params))
Oh! And we can’t forget to add our new dependency to the top of the file:
from urllib.parse import urlencode
Every request will now be sent through the service, which will automatically rotate our IP after every request, handle CAPTCHAs, and use machine learning and statistical analysis to set the best headers to ensure success.
Quick Tip: The service also allows us to scrape a dynamic site by setting ‘render’: true as a parameter in our params variable. The service will render the page before sending back the response.
5. Store The Extracted Financial Data In a CSV File
To store your data in an easy-to-use CSV file, simply add these three lines between your URL list and your loop:
file = open('stockprices.csv', 'w')
writer = csv.writer(file)
writer.writerow(['Company', 'Price', 'Change'])
This will create a new CSV file and pass it to our writer (set in the writer variable) to add the first row with our headers.
It’s essential to add it outside of the loop, or it will rewrite the file after scraping each page, essentially erasing previous data and giving us a CSV file with only the data from the last URL from our list.
In addition, we’ll need to add another line to our loop to write the scraped data:
writer.writerow([company.encode('utf-8'), price.encode('utf-8'), change.encode('utf-8')])
And one more outside the loop to close the file:
file.close()
6. Complete Code: Stock Market Data Web Scraper Script
You’ve made it! You can now use this script with your own API key and add as many stocks as you want to scrape:
#dependencies
import requests
from bs4 import BeautifulSoup
import csv
from urllib.parse import urlencode
#list of URLs
urls = [
'https://www.investing.com/equities/nike',
'https://www.investing.com/equities/coca-cola-co',
'https://www.investing.com/equities/microsoft-corp',
]
#starting our CSV file
file = open('stockprices.csv', 'w')
writer = csv.writer(file)
writer.writerow(['Company', 'Price', 'Change'])
#looping through our list
for url in urls:
#sending our request through ScraperAPI
params = {'api_key': 'YOUR_API_KEY', 'url': url}
page = requests.get('http://api.scraperapi.com/', params=urlencode(params))
#our parser
soup = BeautifulSoup(page.text, 'html.parser')
company = soup.find('h1', {'class': 'text-2xl font-semibold instrument-header_title__GTWDv mobile:mb-2'}).text
price = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[0].text
change = soup.find('div', {'class': 'instrument-price_instrument-price__3uw25 flex items-end flex-wrap font-bold'}).find_all('span')[2].text
#printing to have some visual feedback
print('Loading :', url)
print(company, price, change)
#writing the data into our CSV file
writer.writerow([company.encode('utf-8'), price.encode('utf-8'), change.encode('utf-8')])
file.close()
Consider When to Run Your Stock Market Data Scraper
You need to remember that the stock market isn’t always operational. For example, if you’re scraping data from NYC’s stock exchange, it closes at 5 pm EST on Fridays and opens on Monday at 9:30 am. So there’s no point in running your scraper over the weekend. It also closes at 4 pm so you won’t see any changes in the price after that time.
Another factor to keep in mind is how frequently you need to update the data. The most volatile periods for the stock exchange are opening and closing times. So it might be sufficient to run your script at 9:30 am, at 11 am, and at 4:30 pm to see how the stocks closed. Monday’s opening is also crucial to monitor as many trades occur during this time.
Unlike other markets like Forex, the stock market typically doesn’t make too many dramatic swings. That said, oftentimes news and business decisions can heavily impact stock prices – take Meta shares crash or the rise of GameStop share price as examples – so reading the news related to the stocks you are scraping is vital.
Easily Schedule Web Scraping Stock Market Data
We hope this tutorial helped you build your own stock market data scraper or at least guided you in the right direction.
If you’re looking for an automated data scraping solution, Webparsers’ DataPipeline is an excellent option. It makes scheduling your stock market data scraping projects simple, so you don’t have to worry about different time zones. Simply set your desired scheduling time, and the web scraping tool will automatically run, delivering the latest financial data in structured JSON format directly to you.
Until next time, happy scraping!