In this Article
Is web scraping legal? Web scraping can be lawful, depending on jurisdiction, access, data and intended use; public availability alone does not resolve privacy, contract or intellectual-property restrictions.
For developers, the useful question is whether a particular collection workflow and its downstream use are defensible. The court decisions and regulator guidance below separate access permission from the rights and obligations attached to the information collected.
Sources checked on October 20, 2026. This article is general information, not a legal opinion on a particular project; no named lawyer review is recorded.
Key Facts
- Public availability does not establish permission for every use of collected data.
- The Ninth Circuit’s 2022 hiQ decision affirmed preliminary relief, not general permission to scrape.
- The December 2022 hiQ consent judgment entered $500,000 against hiQ.
- Public personal data can remain subject to data protection obligations.
- Uncopyrightable facts can still sit inside databases subject to contractual restrictions.
- robots.txt is a crawler protocol, not an access authorization system.
Is web scraping legal? A conditional answer
Web scraping can be lawful, but a successful request is not evidence that every legal requirement has been met. Review the access method, the information collected and the intended use separately.
Answer block: Public access addresses only part of the legal analysis. The hiQ appellate opinion discusses public access under the CFAA, while CNIL’s scraping guidance explains why public personal data still requires a lawful basis and safeguards.
Use the 4-layer Access, Data, Rights and Use Review to organize a collection decision. This is an engineering review aid, not a legal test or certification.

| Layer | Question | Evidence to retain |
|---|---|---|
| Access | What authorizes this collection method? | Written permission, relevant terms, access restrictions and crawl policy. |
| Data | Which fields identify people or reveal sensitive information? | Field inventory, necessity assessment and privacy review. |
| Rights | Which contractual, copyright or database restrictions apply? | Licenses, terms and the scope of any applicable exception. |
| Use | What will happen after collection? | Purpose, recipients, retention period and permitted reuse. |
Record unresolved questions before enabling a recurring job. A limited internal analysis and a publicly searchable copy of the source are different workflows and need separate review.
The Short Answer, by Question
| Question | Short answer |
|---|---|
| Is scraping public product/price data legal? | Generally yes — the most defensible category everywhere |
| Is scraping personal data (names, profiles, contacts) legal? | High risk — GDPR regulators fine for it even when data is public |
| Is scraping behind a login legal? | Risky — you’ve accepted terms; contract claims survive even where CFAA fails |
| Is violating robots.txt illegal? | Not a law by itself, but courts and regulators treat it as evidence of bad faith |
| Is using proxies legal? | Yes — the activity, not the tool, determines legality |
| Is scraping for AI training legal? | Unsettled — fair-use wins for lawfully obtained data, but contract/licensing suits are multiplying |
Common Myths About Web Scraping Legality
Most confusion about the legality of web scraping comes from mixing up technical ability with legal permission. Three myths are worth clearing up before the details.
Myth 1: Web Scraping Is Hacking
Scraping a public page isn’t unauthorized access — a bot requests the same page a normal browser does. The US Ninth Circuit’s hiQ v. LinkedIn (2022) held that public-page scraping likely doesn’t violate the CFAA where no authentication gate is bypassed. People also ask “is screen scraping legal?” — same answer: reading a public page the way a browser does isn’t “hacking.”
Myth 2: “Public Data” Means Free to Take
Public doesn’t mean unrestricted. Personal data stays protected even when it’s openly visible — Europe’s regulators fine for scraping public personal data. Copyright still covers public articles and images. And terms of service can bind you even without a login. So the real question — is data scraping legal? — depends heavily on what the data actually is.
Myth 3: Violating ToS Is a Crime
Breaching terms of service is a civil matter, not a criminal one. Violating ToS isn’t automatically illegal — but it’s grounds for being blocked, a basis for a civil lawsuit, and evidence of bad faith in court. More on web scraping terms of service below.
Public pages, logins and terms: what changes risk?
Public access, authentication and contractual assent are separate questions. Being logged out does not establish freedom from contract claims, and logging in does not by itself make authorized automation unlawful.
In Berman v. Freedom Financial Network, the Ninth Circuit examined reasonably conspicuous notice and an unambiguous manifestation of assent. That analysis is more useful than assuming all browsewrap terms fail or all clickwrap terms succeed.
For your project, preserve the relevant terms and document how the organization encountered or accepted them. Check account registration, prior agreements and explicit permissions as well as the page being collected. The legal effect depends on the facts and applicable law.
| Situation | Practical decision |
|---|---|
| Public page without authentication | Continue the privacy, contract and rights review; public access alone is insufficient. |
| Account access with express automation permission | Check that endpoints, fields, purpose and request volume stay within that permission. |
| Accepted terms restrict the intended collection | Seek written permission or legal review before launching. |
| Access is revoked or a targeted block appears | Pause the affected collection and escalate before changing the access method. |
Our guide to scraping websites with login covers implementation context. Treat legal authorization as a separate requirement from whether a session or browser script works.
Is web scraping legal in the US? hiQ’s ruling and subsequent outcome
The hiQ litigation supports a limited conclusion about public-profile access under the CFAA, not a blanket US scraping exemption. Its later contract proceedings and consent judgment are essential to understanding the outcome.
What did hiQ decide? The Ninth Circuit affirmed preliminary relief concerning public-profile scraping and CFAA authorization. It did not definitively resolve every claim or grant general scraping permission. See the April 18, 2022 opinion.
| Date | Development | Limit of the conclusion |
|---|---|---|
| 2021 | The Supreme Court vacated the earlier appellate judgment and returned the case for reconsideration in light of Van Buren. | This was not a Supreme Court decision granting permission to scrape. |
| April 18, 2022 | The Ninth Circuit again affirmed the preliminary injunction. | The appeal concerned preliminary relief and the CFAA issue, not final resolution of every claim. |
| November 4, 2022 | A district court order found contractual breaches and granted summary judgment over contractors’ fake-account conduct. | Waiver and estoppel defenses left factual disputes concerning hiQ’s scraping and use of data. |
| December 8, 2022 | A consent judgment entered $500,000 against hiQ and imposed a permanent injunction. | This was an agreed judgment binding the parties, not a universal prohibition on scraping. |
The November district court order is more nuanced than a simple declaration that either side won every contract issue. The entered consent judgment and permanent injunction barred hiQ from automated access or copying without LinkedIn’s express written permission, whether logged in or not.
The engineering lesson is to assess each legal theory independently. A favorable access argument cannot substitute for reviewing contracts, privacy obligations and reuse rights. For a separate California privacy discussion, see our CCPA and web scraping guide.
Is scraping TikTok data legal? Relevant rulings and limits
There is no general permission to scrape TikTok that follows from hiQ. The sources verified for this article do not establish a TikTok-specific ruling authorizing the proposed collection of public videos, profiles or comments.
What do TikTok court rulings establish? The Supreme Court’s January 17, 2025 decision in TikTok Inc. v. Garland concerned a First Amendment challenge to federal divestiture legislation. It did not decide whether an outside organization could lawfully scrape TikTok data.
That distinction matters when assessing search results about the legality of scraping TikTok data and court rulings. A decision about platform ownership, distribution or government regulation does not answer a collector’s access, contract, privacy or copyright questions.
Define the project before seeking a legal conclusion: which regional service, which endpoints, whether an account is involved, which fields are retained and whether videos or comments will be redistributed. Ask the operator about written authorization or an approved data-access route, then check the conditions attached to that route.
Do not transfer the facts of LinkedIn’s public-profile dispute wholesale to another platform. Also distinguish an inability to verify a relevant ruling from proof that no such litigation exists.
EU and UK: what personal-data, lawful-basis and transparency duties apply?
Public personal data can remain regulated in both the EU and UK. A collection project needs a valid legal basis and a practical plan for transparency, minimization and individual rights.
For the EU, identify an Article 6 basis for each purpose. Public visibility is not itself a basis. Special-category data also needs an Article 9 condition; the condition concerning data manifestly made public by the individual is narrower than merely finding it online. See GDPR Articles 6 and 9.
Can legitimate interest justify scraping? CNIL describes conditions and safeguards for relying on legitimate interest in web scraping. It is not automatic permission to collect public personal data. Its scraping guidance for AI development requires attention to collection scope, people’s expectations and protective measures.
Document the interest, why the collection is necessary and how competing rights are balanced. Consider less intrusive alternatives before retaining identifiers. CNIL’s legitimate-interest assessment guidance explains these tests.
When data comes from another source, Article 14 generally requires information within a reasonable period, no later than one month, or earlier at the first communication or disclosure where applicable. Explain the purpose, basis, source, recipients, retention and rights. A disproportionate-effort exception needs justification and safeguards, including making information public; it is not an automatic exemption for large datasets. See GDPR Article 14.
For UK work, check the UK framework separately. The ICO’s right-to-be-informed guidance describes transparency duties for indirectly collected data and flags review following the Data (Use and Access) Act. Do not assume every EU provision or interpretation transfers unchanged.
Translate the assessment into operations: allowlist necessary fields, set deletion dates, assign responsibility for objections and removal requests, and review changes in purpose. Our ethical web scraping guide offers a complementary operational perspective.
How do copyright, database rights and AI training affect scraping?
Facts and creative expression receive different copyright treatment, but that distinction does not settle contract or database-right questions. AI training also requires an analysis of the source material, acquisition method and intended use.
The US Copyright Office’s copyright overview explains that facts are not protected by copyright. Articles, photographs, videos and original expression can be protected. Collecting a product price is therefore a different copyright question from copying its promotional photograph or description.
US fair use weighs purpose and character, the nature of the work, the amount taken and market effects. Neither commercial use nor research use decides the outcome alone. The Copyright Office’s fair-use guidance rejects a universal formula.
Can contracts restrict uncopyrightable data? Yes, potentially. In Ryanair v. PR Aviation, C-30/14, the CJEU held that the Database Directive did not prevent contractual limitations on a database outside its copyright and sui generis protection, subject to applicable national law. The judgment did not make every restriction enforceable.
Separately, EU database rights can restrict extraction or reuse of substantial parts of a qualifying database. Repeated and systematic extraction of small parts can also engage protection under the conditions in Article 7. See the Database Directive.
For EU text and data mining, distinguish two routes in Directive 2019/790. Article 3 addresses scientific research by qualifying research organizations and cultural heritage institutions with lawful access. Article 4 covers a broader mining exception but allows appropriate express rights reservations, including machine-readable reservations for online content. Do not apply Article 4’s opt-out rule indiscriminately to Article 3.
For an AI dataset, retain source provenance and document acquisition and reuse separately. A conclusion about training does not automatically answer whether downloading, retaining or distributing the underlying copies is permitted.
How should permission, robots.txt and a developer risk checklist guide collection?
Seek permission that describes the actual workflow, then implement controls that keep collection within its scope. Treat new restrictions and access disputes as reasons to pause and reassess.
Written permission: Ask an authorized representative to specify the domains, endpoints, fields, purposes, request limits, retention, redistribution and termination conditions. Resolve conflicts with standard terms explicitly. Permission from a site operator does not necessarily settle rights belonging to individuals or other rights holders.
robots.txt: RFC 9309 defines crawler instructions and explains that the protocol is not access authorization. An Allow directive is not a blanket license. CNIL also treats explicit opposition through robots.txt or CAPTCHAs as relevant to reasonable expectations in its AI-development scraping guidance.
Our robots.txt and AI crawlers guide discusses implementation context. Keep the protocol decision distinct from the legal review.
| Route | Use when… | Avoid when… |
|---|---|---|
| Authorized API or export | Use an API or export when its permission covers the required data and intended use. | Avoid that route when its license excludes your intended reuse. |
| Permission-based crawler | Use a crawler when written permission and operational controls cover the collection. | Avoid a crawler when the required access or rights remain unresolved. |
| Aggregated or reduced dataset | Use reduced data when it answers the question with fewer identifiers or copied works. | Avoid claiming that aggregation repairs unlawful source collection. |
- Before launch: Record the purpose, jurisdictions, source terms, permission and field inventory.
- Before each job: Check applicable crawl instructions and whether permission has expired or changed.
- During collection: Apply a shared request budget, bounded retries and logging across workers.
- On rate limiting: Honor applicable Retry-After instructions and pause persistent failures. RFC 6585 defines HTTP 429 and its optional Retry-After header.
- On escalation triggers: Pause for revoked permission, a cease-and-desist notice, unexpected sensitive fields or an access barrier that would require circumvention.
- After collection: Enforce retention, deletion, access controls and approved downstream uses.
Do not treat a particular requests-per-second setting as a legal threshold. Configure traffic around the operator’s permission and technical constraints.
DataImpulse supplies proxy connectivity, not legal authorization. A proxy should not become a mechanism for ignoring a stop decision; see our proxy-provider ethics checklist.
What’s Defensible vs. What’s Risky
Not every scrape carries the same risk — it depends on the data type, how you access it, and what you do with the result. The two lists below place your project quickly.
Generally defensible
- Public, read-only collection of non-personal data: prices, product specs, stock, rankings, search results, flight fares, real-estate listings
- Logged-out access only — nothing behind authentication
- Human-paced request rates that don’t burden the target’s infrastructure
- Honoring robots.txt and published crawl policies
- Competitive intelligence, market research, SEO monitoring, ad verification, academic research on aggregates

Risky to indefensible
- Scraping personal data — names, photos, emails, phone numbers, profiles — especially at scale, especially in Europe
- Scraping behind a login, against terms you accepted (the hiQ contract lesson)
- Bypassing technical barriers: CAPTCHAs presented as access controls, IP blocks aimed at you specifically, paywalls
- Republishing scraped content wholesale (copyright), or extracting substantial parts of protected databases (EU database right)
- Volume that degrades the target site (trespass-to-chattels exposure)
- Training AI on content you obtained from pirated or terms-breaching sources (Bartz‘s carve-out)
Frequently Asked Questions
Is web scraping legal?
Scraping publicly available, non-personal data is broadly legal in most jurisdictions — the Ninth Circuit’s hiQ v. LinkedIn (2022) confirmed that scraping public pages isn’t “unauthorized access” under the US CFAA. The risk lives in the details: personal data (GDPR fines up to €30.5M in the Clearview cases), scraping behind logins against accepted terms (how hiQ ultimately lost), bypassing technical barriers, and republishing copyrighted content. Public product, price, and SERP data collected respectfully is the defensible lane.
Is web scraping legal in the US?
Generally yes for public data: hiQ v. LinkedIn established that public-page scraping doesn’t violate the CFAA, and Meta v. Bright Data (2024) declined to block logged-out scraping of public Facebook/Instagram pages. But contract claims survive — hiQ lost on breach of LinkedIn’s terms it had accepted — and extreme volumes risk trespass claims. State privacy laws (CCPA/CPRA) regulate personal data on top.
Can I legally scrape Amazon, Google, or other big platforms?
Collecting public product and search data logged-out, at respectful rates, is the same defensible category as elsewhere — it’s how the price-intelligence and SEO industries operate. The risks are platform-specific: don’t log in (terms), don’t take personal data (reviews with author identities), don’t overload endpoints, and know each platform’s enforcement posture. Our per-platform guides (Amazon, Google Maps, LinkedIn, Reddit) cover the specific cases and rules.
Does violating robots.txt make scraping illegal?
robots.txt isn’t a statute, and ignoring it isn’t automatically illegal. But courts and regulators treat it as a signal of good or bad faith, several rulings cite it when weighing trespass and contract claims, and the CNIL’s safeguards expect you to honor opt-outs. Practically: honoring robots.txt costs little and materially strengthens your defensibility.
Does written permission from a website resolve every legal issue?
It can clarify the operator’s authorization and the scope of collection. It may not cover third-party copyright or the rights of people whose information appears on the site. Document those questions separately.
Does staying logged out prevent a contract claim?
No. Relevant agreements, notice, assent and the wording of the terms still need examination. Authentication status is one fact, not a complete contract analysis.
Can I collect only prices and product specifications?
Limiting fields can reduce exposure to personal data and creative content. It does not settle contractual restrictions or EU database rights. Review how the data is accessed and how much of the source database is extracted.
Does a TikTok court headline establish scraping permission?
Check the question the court actually decided. TikTok Inc. v. Garland concerned a constitutional challenge to divestiture legislation, not authorization for outside data collection. A scraping project needs its own access and reuse assessment.
Should a crawler stop after an HTTP 429 response?
Pause according to the applicable Retry-After instruction and the collection policy. Persistent rate limiting should trigger review rather than unlimited retries. A retry delay is an operational control, not legal permission.
Can a privacy notice alone justify collecting public profiles?
No. Transparency and lawful basis are separate requirements. You also need to assess necessity, safeguards and any additional conditions for sensitive data.
Would a successful permission-based test prove a production crawler is compliant?
No. It can demonstrate that particular controls worked under the tested conditions. Different sites, data, permissions and uses require a fresh assessment, and the production controls still need monitoring.
Match collection infrastructure to an approved plan
Once permission, data scope and stop conditions are clear, DataImpulse can support proxy connectivity for an authorized workflow. Residential proxies start at $1 per GB, with country targeting included. Keep authorization checks, rate controls and deletion rules in your application; connectivity does not replace them.
Last updated: October 20, 2026.

State/City/Zip/ASN Targeting 



