Indeed Scraper: Python Tutorial, CSV Export and Practical Limits

An Indeed scraper extracts job information from search responses and turns it into structured records for analysis or export. This tutorial uses a tested Python script to collect search-page job cards, write CSV and stop when access fails.

In our quick run on October 19, 2026, the final script collected 42 records, including 34 with salary information. Other attempts returned challenge pages, and pagination reached a sign-in boundary. The useful result is both an export and a clear account of what the collection could not establish.

What does an Indeed scraper collect?

An Indeed scraper can turn accessible job cards into rows containing titles, companies, locations, salaries and source URLs. The available fields depend on the response actually received, so missing values belong in the dataset.

Our script also records job identifiers, relative posting text, publication dates and sponsored flags. It constructs a job URL from each identifier without visiting that URL. Full descriptions are outside this run: we collected search responses only, never individual job pages.

Do not convert missing salary to zero or assume every amount is annual. Keep the original salary text alongside numeric values. For a broader collection schema, see our guide to Web scraping job postings.

What is the 4-step access-to-export check?

The 4-step access-to-export check separates permission, response validity, field validation and export verification. Passing one step does not establish the next.

• Step 1: Permission. Check the applicable terms, agent rules and exact paths before requesting data. Obtain written permission where necessary.
• Step 2: Response validity. Inspect status, final URL and page content. A challenge or sign-in response is not a job dataset.
• Step 3: Field validation. Check identifiers, required fields, nulls and duplicates before accepting rows.
• Step 4: Export verification. Reopen the output and compare row counts, headers and missing-value counts.

Indeed’s automated-access limits (as of October 2026) require express written permission, with a conditional crawling allowance tied to robots rules. The default robots group allows the search path used here but disallows /viewjob and /rc/; other agents and parameters can have different rules. Read the Indeed terms and current robots rules together.

Our observations are not evidence that your project has authorization. Stop on denial; do not rotate identities to continue a blocked collection.

How do you build an Indeed scraper in Python?

Use the script below to request a search response, extract its embedded job-card data and write CSV. It is the script used in our quick run on October 19, 2026; successful execution elsewhere is not guaranteed.

Save it as indeed_scraper.py. Our Linux environment used Python 3.12 and curl_cffi 0.15.0; comparison requests used requests 2.31.0 and Playwright 1.58. These are experiment versions, not recommendations about the newest releases.

Install the tested dependency with pip install curl_cffi==0.15.0, then run python3 indeed_scraper.py. For an authorized proxy route, set PROXY_URL through your environment or secret manager before execution. The comment contains illustrative placeholders, not working credentials; keep actual secrets out of source control and logs.

import csv, json, os, random, re, time
from datetime import datetime, timezone
from curl_cffi import requests   # pip install curl_cffi: browser-like TLS fingerprint

PROXY = os.environ.get("PROXY_URL")  # e.g. http://USER__cr.us:[email protected]:823
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
JOBCARDS = re.compile(
    r'window\.mosaic\.providerData\["mosaic-provider-jobcards"\]\s*=\s*(\{.*?\});\s*\n', re.S)


def fetch(url):
    """One request per page. On a block we stop instead of hammering the site."""
    r = requests.get(url, impersonate="safari", proxies=PROXIES, timeout=40)
    if "secure.indeed.com/auth" in r.url:
        print(f"{url} -> redirected to sign-in (page 2+ needs an account). Stopping.")
        return None
    if r.status_code != 200 or "Security Check" in r.text[:3000]:
        print(f"{url} -> HTTP {r.status_code}, Cloudflare check/block page. Stopping.")
        return None
    print(f"{url} -> 200, {len(r.content) // 1024} KB")
    return r.text


def parse_jobs(html):
    match = JOBCARDS.search(html)
    if not match:
        return []
    model = json.loads(match.group(1))["metaData"]["mosaicProviderJobCardsModel"]
    rows = []
    for job in model["results"]:
        salary = job.get("extractedSalary") or {}
        posted_ms = job.get("pubDate")
        rows.append({
            "job_key": job.get("jobkey"),
            "title": job.get("displayTitle") or job.get("title"),
            "company": job.get("company"),
            "location": job.get("formattedLocation"),
            "salary_text": (job.get("salarySnippet") or {}).get("text"),
            "salary_min": salary.get("min"),
            "salary_max": salary.get("max"),
            "salary_period": salary.get("type"),
            "posted": job.get("formattedRelativeTime"),
            "pub_date": datetime.fromtimestamp(posted_ms / 1000, timezone.utc).date().isoformat()
                        if posted_ms else None,
            "sponsored": job.get("sponsored"),
            "url": f"https://www.indeed.com/viewjob?jk={job.get('jobkey')}",
        })
    return rows


def scrape(query, city, max_pages=2):
    seen, rows = set(), []
    for page in range(max_pages):
        url = ("https://www.indeed.com/jobs?q=" + query.replace(" ", "+")
               + "&l=" + city.replace(" ", "+").replace(",", "%2C") + f"&start={page * 10}")
        html = fetch(url)
        if not html:
            break
        new = [r for r in parse_jobs(html) if r["job_key"] not in seen]
        seen.update(r["job_key"] for r in new)
        rows += new
        print(f"page {page + 1}: {len(new)} new jobs")
        if not new:
            break
        time.sleep(random.uniform(5, 8))
    return rows


if __name__ == "__main__":
    jobs = scrape("python developer", "New York, NY", max_pages=2)
    if not jobs:
        raise SystemExit("No jobs collected (blocked or layout changed).")
    with open("indeed_jobs.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(jobs[0].keys()))
        w.writeheader()
        w.writerows(jobs)
    print(f"saved {len(jobs)} jobs, {sum(1 for j in jobs if j['salary_text'])} with salary")

The search phrase and city are in the final scrape call. The request sets a 40-second timeout, and the loop pauses five to eight seconds between pages. There is no retry loop. The simple URL substitutions suit the tested inputs; arbitrary punctuation needs separate encoding validation.

For reference while diagnosing Python requests or HTML, consult the Requests quickstart and Beautiful Soup documentation. This script imports curl_cffi, not the separate requests package, and does not use Beautiful Soup.

How do you parse embedded JSON and paginate safely?

Parse the structure present in your authorized response, then stop if it disappears or access changes. Our successful responses contained job records under window.mosaic.providerData["mosaic-provider-jobcards"], followed by metaData.mosaicProviderJobCardsModel.results.

This is an observed page structure, not a documented API contract. The regular expression depends on assignment formatting and a terminating newline. A missing match returns no rows; malformed JSON or changed nested keys can raise an exception. Neither outcome proves that the search has no jobs.

Before accepting an export, inspect a few records against the search page. Require nonempty job keys and check titles, companies and locations. The set in the script filters identifiers seen on earlier pages, but it does not eliminate repeated keys within the same response. Validate uniqueness in the completed export too.

The loop caps requests at two page attempts and advances start by ten. In our run, start=10 redirected toward sign-in and then encountered a challenge. We did not log in. The approximately 42-44 cards in successful first-page responses therefore do not establish a stable page size or complete query coverage.

Also distinguish the script’s date conversion from collection time: pub_date is a publication date, not an audit timestamp. Preserve a separate collection timestamp and source query when developing a production pipeline.

How do you export jobs to CSV, JSON and Excel?

The tested script writes UTF-8 CSV to indeed_jobs.csv. JSON and spreadsheet workbooks are downstream conversion options, not outputs produced by this script.

The CSV contains one header row and the extracted records. In our final run, the console summary was saved 42 jobs, 34 with salary. Empty salary cells represent unknown values, while the sponsored column retains the extracted boolean value.

For spreadsheet analysis, import the CSV as UTF-8 and explicitly treat job identifiers and URLs as text. Check the imported row count before sorting or filtering. Keep the original CSV unchanged, and save transformations into a separate workbook so that normalization remains auditable.

For a JSON conversion, preserve missing values as null, numbers as numbers and sponsored flags as booleans. A conversion from CSV requires deliberate type restoration because CSV cells do not retain those distinctions. Do not label the result a tested JSON export until you have checked it.

Descriptions need an explicit policy if a separately authorized source supplies them: retain full text separately from summaries, preserve meaningful line breaks and record provenance. Adding a description column here would otherwise create empty cells, not additional evidence.

Why does Indeed block requests?

A blocked response means the collection failed at access, before parsing could establish job availability. In our snapshot, challenge pages affected several clients and both network routes.

The diagnostic limit is that response HTML cannot reveal the exact rule responsible. Indeed’s Cloudflare troubleshooting guidance discusses cookies, JavaScript and browser conditions, while Cloudflare’s bot-management reference documents detection signals and fingerprint variables. Neither identifies the decision behind an individual request.

• 403 or a challenge: record the status, final URL and response classification, then stop.
• 429: treat it as a reason to suspend collection and review the permitted request schedule.
• 200 with no model: inspect the saved response for a challenge, consent screen or changed layout.
• Sign-in redirect: record an access boundary instead of counting an empty results page.

Our headless browser attempts also failed, so rendering was not a demonstrated solution. Changing libraries cannot establish authorization, and the Safari-profile observations do not isolate fingerprinting from IP, timing or session effects.

See our 403 troubleshooting guide for response diagnosis. Use the Python retry guide for appropriate transient failures, not repeated challenges or explicit denials.

What did our small public-page run return?

Our quick run on October 19, 2026 produced one final 42-record export alongside multiple blocked attempts. This is a small snapshot, not a benchmark or an estimate of future success.

We used a Linux server, one search for python developer in New York, NY, direct datacenter access and DataImpulse US residential access. The session involved about 30 requests to indeed.com, with five to ten seconds between attempts, no login and no CAPTCHA solving.

The measurement record consists of the route observations below and the inline indeed_scraper.py reproduction. Exact UTC request times and elapsed milliseconds were not recorded in the summary, so no latency comparison is possible. Transfer sizes are approximate.

All first-page rows concern https://www.indeed.com/jobs?q=python+developer&l=New+York%2C+NY&start=0. Each residential Safari attempt used a new IP, which confounds comparisons. Six successful responses out of nine attempts must not become a claimed success rate or a recommendation to rotate after denial.

Bandwidth-only arithmetic at $1 per GB is 210000 * 1000 / 1e9 * $1 = $0.21 per 1,000 successful search responses. At 1.7 MB per uncompressed response, the equivalent is about $1.70. Challenge responses also consumed traffic, approximately 11 KB for the measured compressed check pages.

These estimates exclude retries, redirects, compute, storage, minimum purchases and billing differences. Query overlap also prevents converting card counts directly into a guaranteed cost per unique job. No provider-superiority conclusion follows from this experiment.

Which scraping method fits your task?

Choose the method whose permissions, maintenance burden and output quality match your workload. Evaluate a small accepted sample before committing to a larger collection.

When evaluating an apify Indeed scraper or another hosted offering, ask for the exact maintained product, input schema, missing-field policy and sample output. Compare charges per accepted unique record, including failed requests and duplicate results. A headline price alone cannot support a defensible 10,000-job estimate.

For an n8n Indeed scraper workflow, define the handoff around validated records, a collection identifier and explicit failure status. Keep an empty search distinct from blocked access. Require the downstream workflow to stop on failure instead of repeatedly launching collection.

Proxy selection is a separate decision; our Indeed proxy guide covers that topic. DataImpulse is proxy infrastructure, not a scraping API, managed unblocker, VPN or data vendor. A proxy route cannot guarantee access or correct fields.

Can official APIs, RSS or exports replace scraping?

Use an approved integration or licensed export when it covers the records you need. We could not verify a public self-serve bulk job-search API in the current official documentation.

The documented integration scope and access limits (as of October 2026) distinguish posting management from search extraction. Job Sync lets ATS partners create and manage postings; it is not a general bulk-search endpoint. The Publisher JavaScript Plugin offers a search interface for partner sites and requires partner access. It is not evidence of an unrestricted records-download API.

The Indeed Partner Docs and partner FAQ are the appropriate starting points for scope questions. We found no current official Publisher API enrollment or bulk-search documentation to support instructions for those older names. Absence from the catalog alone does not establish a historical retirement date.

Likewise, do not build around an assumed RSS feed. Ask for an approved export or source feed, and verify its license, geography, update schedule and completeness. An employer’s authorized ATS export may serve a narrower project without collecting search results at all.

What legal and contractual risks apply?

Public accessibility does not settle whether collection or reuse is lawful. Terms, robots instructions, authorization and legislation answer different questions; satisfying one does not resolve the others.

The 2022 hiQ opinion addressed public access under the CFAA in a preliminary-injunction context, not universal permission to scrape. Contract exposure remained separate. In Meta v. Bright Data in 2024, the court interpreted particular terms concerning logged-out collection; that outcome does not decide Indeed’s terms.

The EU Database Directive can restrict substantial extraction and certain repeated extraction. Copyright protection distinguishes facts from protected expression, including potentially original descriptions. The joint regulator statement on scraping explains that public personal information remains subject to privacy law.

Define purpose, retention and redistribution before collection. Avoid assuming that a vendor contract transfers every relevant right. Review our Web scraping legal overview for additional context; the reader decides, check local law

Plan a small, auditable collection

If an authorized project needs a residential route, DataImpulse residential proxies start at $1 per GB. Begin with a capped sample, include failed traffic in the budget and verify the exported records before scaling. Access and extraction quality remain separate checks.

Last updated: October 19, 2026.