Technology Reviews

Is Web Scraping Legal? What Courts and Regulators Have Said, and a Compliance Checklist for Businesses

Written by
Read time
6 min
Published
Filed under
Add to Preferred Sources
Is Web Scraping Legal? What Courts and Regulators Have Said, and a Compliance Checklist for Businesses

Every company that wants public web data, from competitor prices to job listings, eventually asks: is web scraping legal? No court has issued a blanket yes or no. Judges and regulators have ruled on specific claims and facts, in different directions. Buying a residential proxy plan or a scraping tool does not settle the question either.

For a business, the useful version is narrower. Does this particular data collection project carry legal risk that should stop it, change it, or send it to counsel? Below is what courts and regulators have actually said, turned into a checklist. None of it is legal advice or a ruling that any project is lawful.

Five separate questions, not one

A review framework published by EZXLabs, a residential proxy provider, puts a number on the problem: clearing federal anti-hacking law resolves only 1 of 5 common liability areas. Contracts, personal data, copyright and database rights, and harm to the target’s servers make up the rest.

American practice shows the same spread. Website owners in the United States have relied on three main claims against unwanted scraping: copyright infringement, violation of the Computer Fraud and Abuse Act, and trespass to chattels. Winning one argument does not carry over to the next. Review all five areas for every new target domain, even when the first answer looks clean.

Computer-access law after Van Buren and hiQ

For years the Computer Fraud and Abuse Act (CFAA) was the scraper’s biggest worry. In Van Buren v. United States, decided on June 3, 2021 by a 6–3 vote, the Supreme Court read “exceeds authorized access” narrowly: a person crosses that line by obtaining information from parts of a computer, such as files, folders or databases, that are off-limits to them.

One week later the Court vacated the Ninth Circuit’s earlier ruling in hiQ Labs v. LinkedIn and sent it back in light of Van Buren. In April 2022 the Ninth Circuit affirmed its decision again, so hiQ, an analytics firm that scraped public LinkedIn profiles, kept a preliminary injunction stopping LinkedIn from blocking its access to public profile data.

Where this stops matters more than the headline. That reasoning concerns pages any anonymous visitor can load; fake or borrowed accounts, reused session cookies and login walls are a different situation, and the CFAA’s damage provisions still reach collection that overloads a server. Treat anything behind a login as out of scope unless a written agreement covers it.

Contracts and terms of service: Meta v. Bright Data

Contract claims depend on whether the scraper actually agreed to the terms. On January 23, 2024, Judge Edward Chen of the Northern District of California granted summary judgment to Bright Data on Meta’s breach-of-contract claim. Meta’s terms barred collecting data using automated means, but the court read them as governing a user’s own use of Facebook and Instagram, and found that public logged-off scraping was not such use.

EZXLabs compresses the ruling to one line: the terms govern active account holders and not logged-off visitors. Its review adds that the January 2024 decision covered only logged-off collection and never exempted personal profiles from GDPR.

Signing up flips the picture, because terms clicked through at registration can then support a breach claim. Run collection logged off by default, and have someone read any clickwrap terms before an engineer opens a test account.

Why “hiQ won” is the wrong summary

Many articles stop at the 2022 appeal and present hiQ as proof that scraping public pages is lawful. That was not the end: by November 2022 the district court had ruled that hiQ had breached LinkedIn’s User Agreement, and the two companies then settled.

According to the docket chronology in the EZXLabs review, the breach involved automated scrapers and crowdsourced workers using fake accounts. LinkedIn ultimately won a $500,000 consent judgment and permanent injunction barring further scraping by hiQ, along with an order to delete code and data.

Winning on one statute can sit next to losing on contract and tort claims in the same case. When a vendor or colleague cites a case as reassurance, ask for its final outcome, not its most famous interim ruling.

Personal data: public does not mean free to use

Privacy regulators have been direct. In a joint statement on data scraping dated August 24, 2023, published by the Office of the Privacy Commissioner of Canada with other data protection authorities, the first takeaway reads: “Personal information that is publicly accessible is still subject to data protection and privacy laws in most jurisdictions.” Mass scraping of personal information, it adds, can count as a reportable data breach.

Enforcement has followed. On September 3, 2024, the Dutch Data Protection Authority fined Clearview AI EUR 30.5 million over a photo database built from open websites without a lawful basis or notice. Under the EU GDPR, scraping personal data needs a lawful basis under Article 6 and, in most cases, notice to the people concerned under Article 14.

Filter names, emails, profile photos and other identifiers out at parse time; projects that need them get a privacy review first.

Facts such as a price or a stock count are not protected by U.S. copyright; articles, photos, reviews and editorial summaries usually are. Europe adds a layer. Directive 96/9/EC gives qualifying databases a sui generis right against extraction of substantial parts, and in Ryanair v. PR Aviation (Case C-30/14), decided in January 2015, the EU Court of Justice held that site terms can still restrict scraping by contract under national law when a database lacks that protection.

Server harm is the oldest theory of all. U.S. courts have accepted that heavy scraping can amount to trespass to chattels, treating the computer system as property, with eBay v. Bidder’s Edge as the best-known example. Keep request rates polite and stop rules hard, because an overloaded server turns a contract dispute into a tort claim.

Robots.txt and proxies are not permission

Two tools get mistaken for a green light. First comes robots.txt: the Robots Exclusion Protocol, published as RFC 9309 in 2022, says of its own rules that they “are not a form of access authorization.” An open robots.txt file does not license copying, yet ignoring one can still hurt, since the European Data Protection Board’s Guidelines 03/2026 count disregard for robots.txt against legitimate interest under GDPR Article 6(1)(f).

Infrastructure is the second. As the EZXLabs review puts it, purchasing a residential proxy plan, headless browser, or CAPTCHA API does not grant legal authorization to scrape a target domain. Its Acceptable Use Policy prohibits unauthorized access to systems or data, and the company describes its service as an access layer that is not a crawler, scraper, extraction API or managed scraping service.

Read every vendor contract as a statement about the vendor’s conduct, not yours. Counter-intuitive as it sounds, a provider with strict acceptable-use terms is usually the safer partner for a compliance-minded buyer.

Setting up the review before engineering starts

Order the options by authorization: an official API when it returns the fields you need, a licensed feed with written origin warranties for multi-site catalogs, and direct collection only for unauthenticated, non-personal facts that clear all five areas. Before any crawler budget is approved, the EZXLabs framework asks for 6 written compliance records:

  • a register of target domains, paths and jurisdictions;
  • proof that no credentials, borrowed cookies or paywall workarounds are in use;
  • dated copies of each site’s terms, robots.txt and text-and-data-mining notices;
  • a personal-data screen, or a documented privacy review;
  • rate, User-Agent and stop-rule settings, including an automatic halt on rate-limit responses or blocks;
  • vendor provenance and downstream-use sign-off.

Pilots should stay small. EZXLabs suggests starting a trial with one authorized domain and a small URL set, a request budget and a clear stop condition, which keeps legal risk contained.

Readers who want the statute-by-statute version can start from the EZXLabs answer to is web scraping legal, which ties each point to a court document or regulator. Use it to prepare questions for counsel.

A compliance checklist for business readers

  • Name the business purpose and the exact fields needed, and drop the rest.
  • Check whether the data sits behind a login; if so, stop unless a written agreement covers access.
  • Archive the site’s terms of service and robots.txt on the day data collection starts.
  • Screen for personal data and route anything that remains to a privacy review.
  • Store facts, not copied articles, photos or reviews.
  • Set rates, concurrency caps and stop rules, and respond to cease-and-desist letters at once.
  • Get counsel’s sign-off for anything commercial, and file the records with the project.

Pending disputes, including Google’s suit against SerpApi over automated search collection, show this area is still moving. Checklists do not make a project lawful; they show the team asked the right questions.

 

Spotted an error? Request a correction
Never miss a storyThe TechBullion Weekly newsletter and your preferred channels.Subscribe
Share this
Comment

More executive interviews

Founders, CEOs and technology leaders talk about what they build, how they fund it, and where AI, finance and data are taking their industries next.

All interviews