Walmart Scraper: Extract Public Product Data with Python

A Walmart scraper extracts product records from Walmart pages into a dataset for analysis. This Python tutorial uses search-page data observed in our quick run on October 16, 2026: names, prices, ratings, review counts, availability labels, sellers and product URLs.

The result is a small snapshot, not a benchmark or a promise of future access. The tested script writes CSV; JSON export and location-aware monitoring are described separately. Product pages remained blocked. Before collecting, establish permission: Walmart’s terms require prior written consent for scraping.

What can a Walmart Scraper collect?

A Walmart Scraper can collect fields actually present in an accessible response. Our search-page sample exposed product identifiers and listing summaries, but no brand field or individual review text.

Keep missing values distinct from zero. An absent rating does not mean a zero-star product; an absent availability label does not establish that an item is unavailable. Preserve the original value before adding normalized fields.

Start with a bounded query and inspect several records manually. Live collection describes what a response showed at retrieval time. A batch combines observations taken at different times, so it needs timestamps and a clear scope.

Two search pages are not a catalog dump. Ranking, duplicate listings and query coverage can all affect what appears in your dataset. Treat category pages as a separate parser target until their response structure is inspected; this script only handles search pages.

How do you build a Walmart Scraper in Python?

Use the exact search-page collector below with Linux and Python 3.12, the environment used in our quick run. It fetches two pages for one query, filters product records and writes a CSV.

In a working directory, create an environment with python3 -m venv .venv, then activate it with source .venv/bin/activate. See the Python virtual environment instructions. Install the dependency with pip install curl_cffi, consistent with the project’s installation documentation.

Save the script as walmart_scraper.py. For direct access, leave PROXY_URL unset and run python3 walmart_scraper.py. For proxy access, set that environment variable through your local secret configuration before running the same command. The comment contains illustrative placeholders, not working credentials.

The measurement identifies Python 3.12 but does not establish a pinned dependency version. Record your installed versions before reproducing the run. A later installation cannot be assumed to reproduce the same client behavior.

import csv, json, os, random, re, time
from curl_cffi import requests   # pip install curl_cffi: browser-like TLS fingerprint

PROXY = os.environ.get("PROXY_URL")  # e.g. http://USER__cr.us:[email protected]:823
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
NEXT_DATA = re.compile(r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', re.S)


def fetch(url, retries=3):
    for attempt in range(1, retries + 1):
        try:
            r = requests.get(url, impersonate="chrome", proxies=PROXIES, timeout=40)
        except Exception as exc:
            print(f"{url} -> {type(exc).__name__}, retry {attempt}")
            time.sleep(5 * attempt)
            continue
        if r.status_code == 200 and "Robot or human" not in r.text[:5000]:
            return r.text
        print(f"{url} -> {r.status_code}, challenge page, retry {attempt}")
        time.sleep(5 * attempt)
    return None


def parse_search(html):
    data = json.loads(NEXT_DATA.search(html).group(1))
    result = data["props"]["pageProps"]["initialData"]["searchResult"]
    max_page = result.get("paginationV2", {}).get("maxPage", 1)
    rows = []
    for stack in result["itemStacks"]:
        for item in stack.get("items", []):
            if item.get("__typename") != "Product":
                continue  # ads, banners, tiles
            rows.append({
                "item_id": item.get("usItemId"),
                "name": item.get("name"),
                "price": item.get("price"),
                "rating": item.get("averageRating"),
                "reviews": item.get("numberOfReviews"),
                "availability": (item.get("availabilityStatusV2") or {}).get("value"),
                "seller": item.get("sellerName"),
                "url": "https://www.walmart.com" + item.get("canonicalUrl", "").split("?")[0],
            })
    return rows, max_page


def scrape(query, max_pages=2):
    seen, rows = set(), []
    for page in range(1, max_pages + 1):
        html = fetch(f"https://www.walmart.com/search?q={query.replace(' ', '+')}&page={page}")
        if not html:
            break
        new, max_page = parse_search(html)
        new = [r for r in new if r["item_id"] not in seen]
        seen.update(r["item_id"] for r in new)
        rows += new
        print(f"page {page}: {len(new)} products (max page {max_page})")
        if page >= max_page or not new:
            break
        time.sleep(random.uniform(4, 8))
    return rows


if __name__ == "__main__":
    rows = scrape("air fryer", max_pages=2)
    if not rows:
        raise SystemExit("No products collected; check fetch() output first.")
    with open("walmart_products.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)
    print(f"saved {len(rows)} products")

On success, inspect walmart_products.csv alongside the printed page counts. The script stops on a failed fetch, the reported final page or a page with no new identifiers. A later-page failure can still leave a partial CSV, so file creation alone is not completion.

JSON export: The tested script parses embedded JSON but exports CSV. For a separate JSON integration, serialize the returned list from scrape() using json.dump into a UTF-8 file, following Python’s JSON documentation. Validate that extension independently; no JSON output was measured here.

How do you extract prices, ratings and reviews?

Inspect the response body before choosing a parser. Our successful search responses contained __NEXT_DATA__, and the script extracts prices and review summaries from that embedded object.

The observed path was props.pageProps.initialData.searchResult.itemStacks[].items[]. Only entries with __typename equal to Product become rows. Pagination used page=N and paginationV2.maxPage; the air-fryer query reported 13-14 pages across observations, not a universal pagination limit.

A Walmart price scraper should retain the raw price field before deciding whether it represents the value needed for comparison. Inspect its type and meaning, then normalize into a separate field. Never replace an unparseable value with zero or silently reuse yesterday’s price.

If an authorized page contains JSON-LD, compare its fields with the Schema.org Product vocabulary. Do not assume JSON-LD exists because another retailer uses it. Likewise, the presence of a script tag alone does not establish that the expected nested keys exist.

For permitted browser inspection, compare rendered values with the response data and consult Playwright’s locator documentation when designing selectors. No browser-based extraction script is validated here.

A Walmart review scraper needs a separate scope: numberOfReviews is a count, and averageRating is a summary. Neither contains individual reviews, authors or review dates. This tutorial does not collect that material.

Why does Walmart show a robot check?

Walmart can return a challenge body with HTTP 200, as our run demonstrated. Status success therefore cannot establish that a product response is usable.

Apply the 4-step response validation check before accepting a page or writing observations into a price history.

• Step 1: Status. Record status, final URL and redirects. Separate transport exceptions, redirects and unsuccessful responses from candidate content.
• Step 2: Content type. Check whether the response matches the expected HTML document. Record byte counts, but do not treat size alone as proof.
• Step 3: Challenge markers. Inspect the body for challenge text and blocked destinations. Our observed marker was Robot or human?, and plain curl reached a blocked redirect.
• Step 4: Field completeness. Require the expected data container, product entries and usable identifiers. Track missing prices and optional fields separately.

The unchanged script implements only a preliminary gate: status 200 and absence of a marker in the first 5,000 characters. It does not check content type, validate every field or safely handle a missing script tag. A missing match can raise an exception; changed nested keys can also fail parsing.

Stop and inspect such failures before expanding collection. Keep diagnostic records separate from product rows. For status-specific troubleshooting, see our guide to HTTP 403 during scraping.

Our challenge responses contained PerimeterX indicators. That observation does not map Walmart’s entire detection architecture. Headers, session characteristics and browser signals are useful diagnostic categories, illustrated in Cloudflare’s detection documentation; that source is not evidence that Walmart uses Cloudflare.

We did not establish an Akamai deployment, a TLS-only cause or a JavaScript hydration requirement. Nor did we validate stealth plugins, Puppeteer, BrowserQL or press-and-hold automation. A different client outcome is an observation, not a general bypass recipe.

When should you use rotating or sticky proxies?

Choose session behavior around the collection’s context requirements. Keep identity consistent when comparing a store-specific session; consider rotation only for permitted independent fetches that do not depend on that continuity.

Our datacenter and residential paths both received challenges with ordinary requests, and both returned search content with the tested curl_cffi configuration. This snapshot cannot establish which IP type is generally better. Evaluate content validity, session consistency and total cost together. Our Walmart proxy guide covers the infrastructure decision separately.

DataImpulse is proxy transport, not a scraping API, managed unblocker or VPN. It does not parse these records or guarantee Walmart access. A sticky connection also cannot establish a store selection that your application never made.

For budgeting, one curl-impersonate observation reported 176 KB transferred for roughly 1.3 MB of decoded HTML. At a residential rate of $1 per GB, 1,000 similarly compressed pages would be roughly 0.18 GB, or $0.18; without compression, the same estimate reaches about $1.30. These are estimates, excluding retries, extra resources and other overhead. Measure billed traffic for your own workload before extrapolating.

How do you monitor prices by ZIP code and store?

Compare prices only when location, seller and fulfillment context match. Walmart’s terms on pricing and pickup explain that store prices can differ and that website, app and store prices may differ.

ZIP code and selected store belong in the observation context, alongside shipping, delivery or pickup. Otherwise, a change of context can look like a price change. Our script records neither a selected ZIP code nor store identity, so its output cannot establish local prices.

For a permitted monitoring workflow, define an observation key containing item ID, seller, requested ZIP, confirmed store and fulfillment mode. Add collection time in UTC, source URL, raw price, normalized price, currency and validation outcome. Mark unknown context explicitly instead of deriving a store from a US proxy address.

Start with a daily schedule appropriate to your authorization. Compare each valid observation with the preceding valid observation for the same key. Configure your own absolute or percentage threshold; the quick run did not test alert thresholds or monitoring frequency.

Send an email or webhook only after validation and comparison. Include old and new values, timestamps and matching context. Route failed fetches to an operational alert, rather than reporting a missing price as a discount.

Check an apparent clearance change manually before making a purchasing decision. Search availability labels do not establish on-shelf quantities. For pipeline design, see how retailers track competitor prices and our price comparison workflow. Neither replaces validation of your own location context.

Which APIs, services and exports are alternatives?

Walmart has official API routes, but access and endpoint scope must match your role. Treat a search for a Walmart scraper api as an access decision before comparing implementation effort.

The Marketplace Item search API searches the product catalog and limits default keyword or identifier results to 40 matching items (as of October 2026). It requires an access token. The seller onboarding guide describes seller approval and credentials; this is not evidence of anonymous consumer access.

The Reviews API concerns the Review Accelerator Program, with enrollment required and eligible product enrollment carrying a fee (as of October 2026). Its documented purpose does not establish a general endpoint for downloading customer review text. Confirm eligibility and applicable charges before implementation.

Pricing insights is scoped to your items on Walmart Marketplace (as of October 2026). Do not equate it with unrestricted competitor catalog access. Start an affiliate-access investigation at the Walmart.io affiliate documentation, then verify your account’s approved scope before selecting endpoints.

For managed extraction, ask for a sample covering your exact fields, location context, missing-value policy and export format. Require a clear explanation of collection rights and failure reporting. Compare the cost of maintaining parsers with the cost of that service; proxy transport and extraction are different responsibilities.

For occasional analysis, a permitted manual export may be enough. Keep CSV for flat inspection, use a JSON design when preserving nested values matters, and document field meanings in either format. Do not assume a universal free tier, unrestricted catalog export or shared rate limit across these access routes.

What did our small tutorial run return?

Our quick run on October 16, 2026 collected 103 records through residential access and 96 directly across two search pages per collection. This was a small snapshot, not a benchmark.

Method: a Linux server, Python 3.12, public search pages, no login and 4-8 second pauses between successful page fetches. The residential collection returned 66 records on its first page and 37 on its second. Every record in both collections had a price; three missing ratings were observed in an earlier run.

Headless Playwright Chromium also received challenges on both routes. Plain curl with the same proxy and User-Agent received a 307 redirect to a blocked URL. Product-page attempts with requests and curl_cffi remained challenged, directly and through residential access.

The four matrix entries summarize client-route observations, not four statistically repeated trials. Different product totals do not establish better coverage for either route. Query ordering, timing and response contents were not controlled as experimental variables.

The source artifacts are the tested walmart_scraper.py collector and its walmart_products.csv output, summarized here with the run observations. The measurement does not establish request latency, long-term success rates, store-price accuracy or completeness of the wider catalog.

What legal and contractual risks apply?

Public visibility does not establish permission to automate collection. Walmart’s terms require express prior written consent for scraping, and its robots.txt disallows /search for the general user-agent group. Resolve authorization before running collection.

The Ninth Circuit’s 2022 hiQ v. LinkedIn opinion affirmed a preliminary injunction involving public-profile access and a disputed CFAA defense. It was not blanket permission to scrape websites or disregard contractual restrictions.

Where applicable, EU database rights can restrict substantial extraction and certain repeated extraction of smaller portions. Review text can also contain personal data; assess necessity, retention and any disclosure before collecting it.

Define who may use the resulting dataset and whether redistribution is permitted before sharing it with another team. Keep authorization records, minimize collection and avoid accounts or private endpoints outside the approved scope. A successful response does not settle these questions. Our web scraping legality overview gives additional context, but project-specific advice requires a qualified professional. Ultimately, the reader decides, check local law.

Test transport against your actual collection requirements

If your authorized workflow needs proxy transport, explore DataImpulse residential proxies. Start with a small sample, validate content and location context, and measure traffic before scaling.

Last updated: October 16, 2026.