Wayback Machine API: Check, List, and Retrieve Archived Pages with Python

The Wayback Machine API lets you check for archived URLs, list historical captures, and retrieve selected pages through separate interfaces. Choose the interface by the question you need to answer, then validate what comes back.

This guide follows a practical retrieval workflow using our quick run on October 16, 2026 and its tested Python script. The measurements are a small snapshot, not a benchmark. They show why empty results, capture dates, and current HTTP responses need separate handling before you rely on an archived page.

What does the Wayback Machine API let you do?

The Wayback Machine API separates capture discovery from content retrieval. Availability answers a single-URL lookup, CDX exposes capture metadata, and replay retrieves a selected archived response.

Start with the Internet Archive API overview. Its Availability and CDX links are the official entry points; they are different interfaces, not interchangeable names for one response format.

Saving is a separate task: Internet Archive’s Wayback help describes Save Page Now for preserving a live page. It cannot reconstruct content that disappeared before capture.

Memento supplies a standard vocabulary for time-based access: an original resource, its archived versions, and datetime negotiation. See the Memento protocol specification when building interoperability across archives. For this tutorial, explicit URLs and timestamps keep the retrieval process inspectable.

Choose an output before collecting anything: one historical quotation, a change timeline, or a recoverable set of files. Each needs a different stopping condition and a different definition of success.

Which Wayback Machine API endpoint should you use?

Use the 4-step archive retrieval model: check, enumerate, retrieve, validate. Treat each step as a separate decision so that a missing lookup result does not become an unsupported claim about historical absence.

Step 1: Check. Look up the exact URL and desired date. Record the request even when the answer is empty.

Step 2: Enumerate. Build a bounded candidate list. Decide whether you want the nearest capture, an earlier capture, or a sequence for comparison.

Step 3: Retrieve. Fetch the selected candidate and retain its identity alongside the response.

Step 4: Validate. Check the final URL, timestamp, content type, and actual body before extracting evidence.

Keep a small manifest of intended URLs and outcomes. A failed retrieval belongs in that manifest, rather than silently disappearing from the dataset.

How do you check whether a URL is archived?

Send an Availability request with a URL and optional timestamp, then inspect the closest snapshot. An empty result should trigger a cautious fallback, not a declaration that the page never existed.

The endpoint is https://archive.org/wayback/available. The Availability API documentation describes archived_snapshots.closest, with available, url, timestamp, and status; timestamps use up to 14 digits in YYYYMMDDhhmmss form.

In our quick run on October 16, 2026, dataimpulse.com returned empty archived_snapshots after 5.6 seconds, although CDX listed captures. www.dataimpulse.com returned 20240107221014; example.com also returned empty. These are three URL inputs, not evidence of a general success rate.

The Python function below falls back to CDX. Its fallback returned 20230925001240 for the requested 20240101. This changes the selection rule: the fallback seeks the last matching capture within the ending date, rather than the closest capture on either side.

Record that distinction in downstream analysis. For an exact cutoff instant, use a full timestamp and compare the returned value explicitly. Our curl-to-Python guide helps translate request parameters when checking the same request from a terminal.

How do you list and filter historical captures with CDX?

Use CDX to build a candidate inventory before downloading bodies. Keep the query narrow enough that you can explain which captures were excluded.

The official CDX server API documentation describes https://web.archive.org/cdx/search/cdx. With output=json, the first row names the columns; fl chooses fields. Matching defaults to an exact URL; matchType also supports prefix, host, and domain scope.

from and to bound dates inclusively. Repeated filter parameters select fields. Collapse removes adjacent matches: collapse=urlkey does not select the latest capture. For pagination, showNumPages=true discovers page count and page=0 starts enumeration where supported; showResumeKey and resumeKey offer continuation.

Our script requests timestamp, original URL, archived status, MIME type, digest, and length. Its statuscode:200 filter and monthly collapse produced 37 rows for 2023-2026, starting at 20230321112517. That is a filtered sample, not every historical capture.

For research on redirects or outages, reconsider the successful-status filter. For change detection, reconsider monthly sampling: several meaningful revisions could fall between selected observations.

Set a stopping budget before pagination and checkpoint completed work. The example’s 50-row application limit is not a service quota. Keep archived status separate from today’s HTTP response, and do not interpret CDX length as downloaded page bytes.

How do you retrieve historical pages with Python?

Select a timestamp and original URL, retrieve their replay URL, and inspect the returned response before extracting content. The following is our tested wayback.py, unchanged from the quick run.

Run it with python3 wayback.py in an environment with requests installed. PROXY_URL is optional; no proxy credentials belong in source code or logs.

import csv, os, time
import requests

S = requests.Session()
S.headers["User-Agent"] = "dataimpulse-research/1.0 (contact: [email protected])"
if os.environ.get("PROXY_URL"):          # optional, e.g. when your server IP gets 429s
    S.proxies = {"http": os.environ["PROXY_URL"], "https": os.environ["PROXY_URL"]}


def available(url, timestamp=None):
    """Availability API: the closest capture to a date (YYYYMMDD...), or None."""
    params = {"url": url, **({"timestamp": timestamp} if timestamp else {})}
    snap = S.get("https://archive.org/wayback/available", params=params, timeout=30).json()
    closest = snap.get("archived_snapshots", {}).get("closest")
    if closest is None:   # the Availability API sometimes answers empty for archived URLs; ask CDX instead
        rows = captures(url, end=timestamp, limit=-1, collapse=None)
        if rows:
            r = rows[0]
            closest = {"timestamp": r["timestamp"], "url": f"https://web.archive.org/web/{r['timestamp']}/{r['original']}", "source": "cdx"}
    return closest


def captures(url, start=None, end=None, limit=50, collapse="timestamp:6"):
    """CDX API: one row per capture. collapse=timestamp:6 keeps one capture per month."""
    params = {"url": url, "output": "json", "limit": limit,
              "fl": "timestamp,original,statuscode,mimetype,digest,length",
              "filter": "statuscode:200"}
    if collapse:
        params["collapse"] = collapse
    if start: params["from"] = start
    if end: params["to"] = end
    r = S.get("https://web.archive.org/cdx/search/cdx", params=params, timeout=90)
    if r.status_code == 429:
        raise RuntimeError(f"CDX rate limit (429); Retry-After={r.headers.get('Retry-After')}. Slow down or change IP.")
    rows = r.json()
    if not rows:
        return []
    header, *data = rows
    return [dict(zip(header, r)) for r in data]


def fetch_capture(timestamp, original, raw=True):
    """raw=True adds 'id_' so you get the archived bytes without the Wayback toolbar and rewritten links."""
    flag = "id_" if raw else ""
    return S.get(f"https://web.archive.org/web/{timestamp}{flag}/{original}", timeout=60)


if __name__ == "__main__":
    t = time.time(); print("closest:", available("dataimpulse.com", "20240101"), f"{time.time()-t:.2f}s")
    t = time.time(); caps = captures("dataimpulse.com", start="2023", end="2026"); print(len(caps), "monthly captures", f"{time.time()-t:.2f}s")
    if caps:
        first = caps[0]
        r = fetch_capture(first["timestamp"], first["original"])
        print("raw capture", r.status_code, len(r.content), "bytes", r.url)
        with open("wayback_captures.csv", "w", newline="") as f:
            w = csv.DictWriter(f, fieldnames=list(caps[0].keys())); w.writeheader(); w.writerows(caps)

The raw request for 20230321112517 returned HTTP 200 and 194,735 bytes. The id_ mode requests the captured body without toolbar injection or link rewriting; the wayback maintainer documentation explains original versus browser-view playback and memento handling.

Original playback is not a complete offline site. Treat its links as untrusted references and parse HTML without executing archived scripts. If your analysis needs text, retain the original bytes alongside any HTML-to-Markdown conversion so extraction choices remain auditable.

Requests follows GET redirects by default. Inspect r.history and r.url; a successful final response alone does not establish that you received the intended date or resource.

The script exports metadata to wayback_captures.csv; it does not persist page bodies or implement SQLite storage. For a larger project, design a SQLite table keyed by original URL and capture timestamp, with requested date, final replay URL, retrieval time, HTTP status, and a body location. Keep extraction results separately so you can rerun parsing without another archive request.

How do you download a site’s latest or date-filtered archive?

Build a per-URL recovery manifest and retrieve the selected files. A downloaded collection should be described by its coverage and dates, not automatically called a complete website backup.

The StrawberryMaster downloader documentation describes installation through gem install wayback_machine_downloader_straw and invocation with wayback_machine_downloader plus the target URL. Its options include --from, --to, --all-timestamps, and --snapshot-at. The last selects the newest version of each file at or before a timestamp. This is a third-party tool, not an Internet Archive service.

For your own latest-only workflow, group candidates by URL and choose the greatest eligible timestamp in each group. Keep the timestamp for every file. A homepage and stylesheet selected independently may describe different moments, even when both satisfy the same cutoff.

Discover image candidates from archived HTML attributes, linked stylesheets, and known original image URLs. Check those URLs individually. Internet Archive’s recovery guidance explains that missing images and incomplete captures can occur.

Before scaling, recover one representative page and its dependencies. Inspect broken links, missing assets, and unexpected date changes. Record unresolved resources explicitly instead of substituting current files without labeling them.

How do you save a live page now?

Use Save Page Now when the objective is future preservation of an accessible live page. Keep saving separate from historical retrieval in both your code and your records.

For automation, consult the Save Page Now 2 API documentation before implementation. We could not verify its current authentication rules, status endpoint, or quotas from the official document during this check, so this article does not prescribe those details.

Design the workflow around separate submission and completion records. Store the target URL and submission time, retain any returned identifier, and use the documented status mechanism before marking the operation complete. Then inspect the resulting capture and record its actual timestamp.

A request acknowledgement should not become a claim that the intended content was preserved. Validate a representative page before planning recurring saves, especially when useful content depends on interactive behavior. Keep credentials outside scripts and avoid publishing sensitive material simply to make it archivable.

What limits, errors, and access requirements should you expect?

Plan for endpoint-specific failures and conservative request pacing. We did not establish a universal requests-per-minute limit for Availability, CDX, and replay (as of October 2026).

Measurement method and sample. Our quick run on October 16, 2026 used Linux, Python 3.12, and Requests: three Availability URL inputs, one site’s filtered CDX inventory, and one raw page retrieval. Sources are our run notes and the tested wayback.py; its metadata export is wayback_captures.csv. This is a small snapshot, not a benchmark or a controlled network comparison.

The route difference does not establish its cause or justify quota evasion. Although the unchanged script’s exception mentions changing IP, stop and back off after throttling; do not rotate addresses to continue a blocked workload.

The script has timeouts and a CDX 429 exception, but no automatic retry loop, general status validation, or JSON error recovery. For repeated jobs, use serial requests, pauses, bounded exponential backoff with jitter, and a stopping condition. Honor Retry-After when present, following HTTP retry semantics. Our API rate limiting guide explains the operational pattern.

Our reading script used no Internet Archive API key. That observation is not a promise of unrestricted access. Before a large job, seek current guidance from the archive about your workload; do not infer higher access from a wrapper, proxy, or successful test. Treat a Wayback Machine API job as a queue you can pause and resume.

At roughly 0.2 MB per page in this snapshot, 1,000 similar bodies would total about 0.2 GB. At our residential starting rate of $1 per GB, that would be about $0.20 in proxy traffic. That estimate excludes assets, retries, and other traffic, and it says nothing about achievable throughput.

What legal and ethical limits apply?

Archive access is not blanket permission to copy, republish, or process everything you find. Review your purpose, permissions, applicable terms, and intended redistribution before collection.

The U.S. Copyright Office explains that copyright protects original expression; archiving does not establish a reuse license. Where GDPR applies, account for lawful processing, data minimization, and retention under its processing principles. Historical personal data still deserves careful handling.

EU database rights can restrict substantial extraction or reutilization, and some repeated extraction, under Directive 96/9/EC. The 2022 hiQ v. LinkedIn opinion addressed a preliminary injunction and a specific public-access CFAA dispute; it did not settle every scraping claim or authorize archive reuse generally.

Check relevant robots directives and access policies without treating either as a copyright license. Minimize sensitive collection, restrict stored copies, and never execute untrusted archived scripts. Our web scraping legality guide offers further context. For consequential reuse, seek jurisdiction-specific advice: the reader decides, check local law

What alternatives fit when captures are missing?

Start with owner-controlled backups, exports, and original assets when the archive cannot answer the question. Match the recovery source to the evidence you actually need.

For your own site, inspect hosting backups, CMS exports, repository history, and media storage. For someone else’s material, request an authorized copy or export. If a platform has an official API, check its documentation for the specific content and historical coverage before building around it.

Other archives may be worth checking, but record their collection scope and provenance independently. A matching title or URL is not enough to establish that two copies contain the same historical page.

For research, combine historical observations with appropriate market research data sources and label the date of each source. Keep unresolved gaps visible in the final dataset. Missing evidence should remain missing rather than turning into a guessed reconstruction.

Keep archive retrieval small and traceable

Start with a direct connection, a bounded URL list, and a manifest of retrieved captures. If your authorized workflow needs optional proxy transport, explore DataImpulse residential proxies. DataImpulse is not a scraping API, managed unblocker, VPN, or data vendor; transport does not replace capture validation or the archive’s access rules.

Last updated: October 16, 2026.