Best practices for web scraping
At Webparsers, we prioritize ensuring that our services respect the rights of websites and organizations whose data we extract.
We frequently encounter the misconception that scraping operates in a legal grey area, but the reality is that data extraction itself isn’t illegal. The potential legal issues arise from how you scrape and what types of data you collect.
This article provides a comprehensive set of guidelines to follow when extracting web data, helping you understand when caution is required regarding the methods and data types involved in your scraping activities.
Disclaimer: We are not your lawyer, and the recommendations in this guide do not constitute legal advice. Our Head of Legal is a lawyer, but she’s not your lawyer, so none of her opinions or recommendations in this guide constitute legal advice from her to you. The commentary and recommendations outlined below are based on Webparsers’ experience helping our clients (startups to Fortune 100’s) maintain legal compliance whilst scraping 7 billion web pages per month. If you want assistance with your specific situation then you should consult a lawyer.
Don’t be a burden
The fundamental principle of web scraping is: never harm the target website. The second rule of web crawling reinforces this: absolutely do NOT damage the website’s functionality.
This principle requires that your request volume and frequency should not overload the website’s servers or disrupt normal website operations.
You can implement this responsibly through several methods:
- Control the number of simultaneous requests to the same website from a single IP address.
- Honor the delay requirements that crawlers should observe between requests by following the crawl-delay directive specified in the robots.txt file.
- When feasible, it’s more considerate to schedule your crawling activities during the website’s off-peak hours.
A vital component of this principle involves providing website administrators with a straightforward method to contact you. At Webparsers we accomplish this by making an abuse report available on our website. If you ever receive an abuse report from a website you are scraping you should either stop scraping the site or limit the scraping in order to rectify the abuse reported.
Don’t violate copyright
When extracting data from websites, you must always evaluate whether the web content you plan to collect is protected by copyright law.
Copyright represents the exclusive legal ownership over a creative work — such as articles, images, videos, etc. Essentially, if you create something original, you own the rights to it. For content to qualify for copyright protection, it must be both original and exist in tangible form.
The typical types of web content that may be copyrighted include:
- Articles
- Videos
- Pictures
- Stories
- Music
- Databases
Consequently, copyright law significantly impacts scraping activities because much of the internet’s content (such as articles and videos) consists of copyrighted material.
Nevertheless, certain circumstances allow exceptions that may apply to all or portions of the data, enabling legal extraction without infringing on the owner’s copyright.
Fair use:
Fair Use represents an exception that allows limited utilization of copyrighted content. Generally, fair use encompasses categories including criticism/parody, commentary, news reporting, education, scholarship, and research. An example of fair use includes publishing brief excerpts of articles with accompanying links, which is typically acceptable under fair use due to the transformative and limited scope of the usage.
The criteria commonly applied to determine fair use exception applicability are:
- the purpose and character of your use (ie is it transformative in some way);
- the nature of the work (ie fact v. fiction or published v. unpublished);
- the amount taken, the less you copy the better; and
- the effect upon the potential market, meaning the extent to which your use may deprive the owner of income or a potential market opportunity.
Transformative use:
A key factor in establishing fair use is whether the usage transforms the original content. Rather than distributing and storing exact copies or substantial portions of the crawled website, modify the content and its application so that you avoid copyright violations.
Facts:
The factual information within copyrighted material is typically not protected by copyright laws, so limiting your extraction to purely factual elements — such as product names, pricing, etc, makes such scraping acceptable.
Note that different countries have different exceptions to copyright law, and you should always ensure that an exception applies within the jurisdiction within which you’re operating.
Don’t breach GDPR
GDPR implementation fundamentally transformed how you can extract personal data belonging to EU citizens (and sometimes non-EU citizens as well). For a deeper explanation of how GDPR affects web scrapers, be sure to check out our Web Scrapers Guide to GDPR.
However, in this section, we will briefly outline the best practices when it comes to scraping personal data. Personal data is any data that can identify an individual person:
- Name
- Phone number
- Address
- User name
- IP address
- Bank or credit card info
- Medical data
- Biometric data
Without a “lawful basis” to scrape and store this information, you will violate GDPR if any extracted data belongs to EU residents. In web scraping contexts, the most prevalent legal justifications are legitimate interest and consent.
Consent
For consent to serve as your lawful basis for data scraping, you must obtain that individual’s explicit permission to scrape, store and utilize their data as intended. This requires that you or a third party must have directly communicated with the person and they agreed to terms permitting you to scrape their data.
An example of this would be companies like Mint.com, where users give Mint consent to log into their online banking accounts and retrieve their banking transactions so that they can be tracked and displayed in a more user-friendly format on Mint.com.
Legitimate interest
For most organizations, demonstrating legitimate interest in scraping someone’s personal data will prove extremely challenging.
Generally, only governments, law enforcement agencies, and similar entities will possess what would be considered legitimate interest in scraping personal data, as they typically extract such information for public benefit.
Beware of login and website terms and conditions
When you authenticate and/or explicitly accept a website’s terms and conditions, you establish a contractual relationship with the website owner, thereby agreeing to their regulations concerning web scraping. These agreements can explicitly prohibit any data scraping from the website.
This means that you need to carefully review the terms and conditions you are agreeing to if your spiders have to log in to scrape data, as they could stipulate that you’re not allowed to scrape their data. You should always honor the terms of any contract you enter into, including website terms and conditions and privacy policies.
Looking for web extracted data? We extract the data you need and deliver it exactly as you’d like it. Just tell us what you need.
Learn more about web scraping
Here at Webparsers, we have been in the web scraping industry for 12 years. We have helped extract web data for more than 1,000 clients ranging from Government Agencies and Fortune 100 companies to early-stage startups and individuals. During this time we gained a tremendous amount of experience and expertise in web data extraction.
Here are some of our best resources if you want to deepen your web scraping knowledge:
- Developer tools that make web scraping a breeze
- Web scraping: Best practices
- Enterprise web scraping: A guide to scraping at scale
- Legal compliance in web scraping
- The build in-house or outsource decision
- What you can use web scraping for