In this Article
An HTTP status codes list is easy to find; one written for people running crawlers is not. This page is that: what each code means when your scraper receives it, whether it is worth retrying, and which codes point at your own configuration rather than the target.
The organising idea is simple. Every response tells you to do exactly one of four things: keep going, wait, change who you appear to be, or stop. Getting that mapping right is most of what separates a crawler that runs for months from one that burns a pool in a week.
Key Facts
- The status code is a routing decision, not a result. Retry, back off, change identity, or stop: every code maps to exactly one of those four, and mixing them up is why crawlers burn address pools.
- 200 is not success. A 200 with an empty data container means the gate let you through and the page rendered client-side, which is a parsing problem rather than an access one.
- 407 is yours, 403 is theirs. 407 means your proxy credentials are wrong; 403 means the target refused you. They look similar in logs and need opposite fixes.
- 429 tells you the rate, 403 tells you the identity. A 429 with Retry-After is a negotiation; a 403 with a short body is a verdict.
- Never retry blind. Retrying a 404 or a 451 wastes budget and, on shared pools, damages the reputation other jobs depend on.
How should a scraper read status codes? The 4-stage triage rule
Sort every response into one of four stages before writing any handling code. We call it the 4-stage triage rule.
| Stage | Codes | Action |
|---|---|---|
| 1. Proceed | 200, 201, 204, 206, 301, 302, 304 | Parse, follow, or skip. Nothing to fix, though 200 still needs a content check |
| 2. Wait | 429, 503, 504, 408 | Back off with jitter, honour Retry-After, reduce concurrency. The target is telling you the rate is wrong |
| 3. Change identity | 403, 401 where access is public, 421 | Diagnose before acting: exit address, TLS and header coherence, runtime. Retrying unchanged achieves nothing |
| 4. Stop | 400, 404, 405, 410, 451, repeated 403 on the same path | The request is wrong, the resource is gone, or access is refused as a decision. Log it and move on |
The expensive mistake is treating stage three as stage two. A blocked identity retried a hundred times is a hundred more data points confirming the classification, and on a shared pool it degrades addresses that other jobs are using.
Which codes actually show up in scraping, and what do they mean?
| Code | What it means to a crawler | Retry? |
|---|---|---|
| 200 OK | The gate passed. Content may still be missing if the page renders client-side | No, but assert on expected fields |
| 204 No Content | Valid response, nothing to parse. Common on tracking and API endpoints | No |
| 301 / 302 | Moved. A redirect chain into a login or a country gateway is a signal, not noise | Follow, but log the final URL |
| 304 Not Modified | Your conditional request worked and the page has not changed. The cheapest possible response | No, and aim for more of these |
| 400 Bad Request | Your request is malformed, often a broken query string or header | No, fix the code |
| 401 Unauthorized | Credentials required by the target. On public data this usually means you hit a gated path | No |
| 403 Forbidden | Understood and refused. Identity, reputation or pattern, and sometimes a plain geo rule | Not unchanged |
| 404 Not Found | Resource gone, or a soft 404 that returns a page with no data | No |
| 405 Method Not Allowed | Wrong verb, typically a POST endpoint hit with GET | No, fix the code |
| 407 Proxy Authentication Required | Your proxy credentials are missing or wrong. Nothing to do with the target | No, fix the configuration |
| 408 Request Timeout | The server gave up waiting. Often a slow proxy hop | Yes, with backoff |
| 410 Gone | Deliberately removed. Stronger than a 404 | No, and drop it from the queue |
| 421 Misdirected Request | Connection reused for a host it does not serve, seen with HTTP/2 coalescing through proxies | Yes, on a fresh connection |
| 429 Too Many Requests | Rate limit. Read Retry-After; it is the most honest header in scraping | Yes, after waiting |
| 451 Unavailable For Legal Reasons | Blocked for legal or regulatory reasons, often geographic | No |
| 500 Internal Server Error | Their bug, or your payload triggered one | Once or twice, then stop |
| 502 / 504 | Gateway errors. Can come from the target’s edge or from your own proxy hop | Yes, with backoff |
| 503 Service Unavailable | Overloaded or deliberately shedding load, including anti-bot shedding | Yes, slowly |
| 520 to 526 | Cloudflare-specific: origin returned something invalid, timed out, or the TLS handshake with the origin failed | Yes, but treat repeats as a signal |
Which codes point at your proxy rather than the site?
Three, and confusing them with target-side refusals wastes days.
407 Proxy Authentication Required. The gateway rejected your credentials. Check the username format, the password, and whether your session or country suffix is well formed. No amount of retrying or rotating helps, because the request never reached the target.
502 and 504 immediately after a route change. A gateway error that appears the moment you switch exits, and disappears when you switch back, is usually the hop rather than the origin. Test the same URL directly and through a second exit before blaming the site.
421 Misdirected Request. Rare, and almost always connection reuse across hosts through a proxy under HTTP/2. Disable connection coalescing or pin one host per connection.
The fastest way to separate the two sides is a control request: fetch a URL you know is stable, such as an IP echo service, through the same exit. If that fails too, the problem is your route; if it succeeds, the target is refusing you. Our guide to 403 Forbidden errors when scraping walks the full isolation sequence.
What retry policy actually works?
Four rules cover almost every case, and they are boring on purpose.
- Retry only stages two and selected stage-four infrastructure errors. 429, 503, 504, 408, 500 once or twice. Never 400, 404, 405, 410, 451, and never an unchanged 403.
- Exponential backoff with jitter, starting around one second and capping at a minute. Without jitter, parallel workers retry in lockstep and reproduce the burst that caused the limit.
- Honour Retry-After literally. If a site tells you to wait thirty seconds, waiting five is a decision to be rate limited again.
- Use a circuit breaker per host. After a threshold of consecutive failures, stop the host entirely for a cool-off window rather than degrading gracefully into a ban.
| Situation | Use retries when | Avoid retrying when |
|---|---|---|
| 429 with Retry-After | You honour the stated delay and lower concurrency afterwards | You would retry inside the stated window, which guarantees a second limit |
| 503 or 504 | The failure is intermittent across many hosts, suggesting load | One host fails consistently: that is shedding aimed at you |
| 500 | It happened once on a URL that usually works | It repeats on the same payload, which means your request triggers their bug |
| 403 | Never unchanged; only after one variable has been isolated | Always, until you have tested exit, client and pace separately |
| 404, 410, 451 | Never | Always: the resource is gone or access is refused as policy |
Two counters make this measurable: failures by code per host per hour, and the ratio of retries to successful requests. When the second climbs above roughly one in five, your pacing is wrong regardless of what the first says.
Why does a 200 sometimes mean failure?
Because passing the gate and getting the data are different events. Four cases produce a 200 with nothing useful in it.
Client-side rendering. The HTML is a shell and the data arrives by XHR. The fix is either a browser engine or, better, finding the JSON endpoint the page itself calls.
Soft 404. The site serves a friendly not-found page with a 200 status. Assert on a known element rather than on the status code.
Challenge page. An interstitial that returns 200 while asking the client to prove itself. Detect it by size and by the presence of a challenge script, not by status.
Geo or consent wall. A 200 that returns a cookie banner or a country gate instead of the content, which is why exit country belongs in your dataset.
The practical rule: never treat a status code as a success assertion. Assert on content, count rows per run, and alert when a source drops by more than a threshold rather than when it errors.
What are the limits of reading codes alone?
Status codes describe the last hop, not the whole story. A 200 can be a challenge, a 403 can be a country rule rather than a bot verdict, and a 502 can belong to your own proxy. Treat codes as the first signal in a diagnosis, then confirm with body size, headers and a control request.
On the rules: a repeated refusal is a decision by the site, and the right response is to respect it, check robots.txt and the terms, and prefer an official API where one exists. Personal data raises a separate question about lawful basis. This is general information rather than legal advice; see is web scraping legal.
For the neighbouring problems: 403 triage, rotation and sessions, and how detection systems score a request.
Frequently Asked Questions
What does 403 mean when scraping?
The server understood the request and refused it. For a crawler that usually means identity, reputation or request pattern rather than a broken request, and sometimes a plain geographic rule. Retrying unchanged does nothing: change one variable at a time, starting with the exit address, then the client, then the pace.
What is the difference between 403 and 429?
A 429 is a statement about rate and usually carries a Retry-After header, so the fix is to slow down. A 403 is a statement about you, so the fix is to change what your client looks like. Treating a 403 as a rate limit and simply waiting is why crawlers stall for hours without learning anything.
Why am I getting 407?
Your proxy credentials are missing or malformed, so the gateway rejected the request before it reached the target. Check the username format including any session or country suffix, the password, and that the port matches the protocol. Rotation and retries cannot fix a 407.
Should I retry a 500?
Once or twice with backoff, then stop and log it. A persistent 500 on one URL usually means your payload triggers a bug on their side, and hammering it is both useless and rude. If 500s appear across many URLs at once, the site is having an incident and the whole job should pause.
Why do I get 200 but no data?
Because the gate passed and the content did not arrive: the page renders client-side, the site served a soft 404 or a challenge interstitial with a 200 status, or you hit a consent or geo wall. Assert on a known element in the body rather than on the status code, and count rows per run so a silent drop becomes an alert.
Fewer refusals start with the right exit
When triage points at identity rather than pace, DataImpulse residential proxies give geo-accurate exits at $1 per GB across 195 countries, rotating or sticky, over HTTP, HTTPS and SOCKS5. Create an account and test one target before scaling.
Related: 403 Forbidden when scraping · proxies for web scraping · web scraping use case.
Last updated: September 17, 2026.
State/City/Zip/ASN Targeting 
