Substack Scraper: Collect Public Posts with Python

A Substack scraper can turn accessible newsletter metadata into structured records; the tested Python example below saves titles, dates, canonical URLs, access labels, and engagement counts to CSV. It does not retrieve subscriber-only text or prove that an archive is complete.

Start with permission: Substack’s Terms of Use expressly prohibit scraping. Creator consent alone does not override those terms. Treat the implementation as a technical reference for an expressly authorized workflow, and consider feeds, creator exports, or approved API access first.

Our quick run on October 18, 2026 compared direct and residential requests and collected metadata from two newsletters. It is a small snapshot, not a benchmark or evidence of permission.

What can a Substack scraper collect?

Our sample exposed post metadata, free-post content, and paid-post previews. Those are separate outputs: discovering a paid article’s title does not mean collecting its body.

Use the 4-step collection check before selecting a tool:

• Permission: establish platform authorization and content-use rights.
• Discovery: identify publication URLs and the authorized collection surface.
• Extraction: choose metadata, readable text, or both.
• Validation: check missing fields, duplicates, access labels, and stopping reasons.

Our archive responses contained titles, dates, canonical URLs, bylines, audience labels, and comment and reaction counts. The script retains only selected metadata. Notes, comment threads, and individual reactions were outside the test; counts cannot establish complete coverage of those objects.

Subscriber emails are outside this public collection workflow. Keep missing values explicit, and distinguish an empty field from a field that was never collected.

How do you build a Substack scraper with Python?

For an authorized workflow, fetch a bounded set of public archive responses, map fields into records, and write CSV. This tested example uses JSON responses rather than page selectors.

Save the script as substack_scraper.py. Install its dependency with pip install requests, then run python3 substack_scraper.py https://www.lennysnewsletter.com only within an authorized scope. The example proxy address contains placeholders, not working credentials.

import csv, os, random, sys, time
import requests

PROXY = os.environ.get("PROXY_URL")  # e.g. http://USER__cr.us:[email protected]:823
PROXIES = {"http": PROXY, "https": PROXY} if PROXY else None
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Accept": "application/json",
}


def get_archive_page(base, offset, limit=12):
    """One page of the public archive JSON the Substack site itself uses."""
    url = f"{base}/api/v1/archive"
    params = {"sort": "new", "offset": offset, "limit": limit}
    r = requests.get(url, params=params, headers=HEADERS, proxies=PROXIES, timeout=30)
    if r.status_code != 200:
        print(f"offset {offset}: HTTP {r.status_code}, stopping")
        return None
    return r.json()  # a list of post objects, [] when the archive ends


def to_row(post):
    return {
        "title": (post.get("title") or "").strip(),
        "subtitle": (post.get("subtitle") or "").strip(),
        "date": (post.get("post_date") or "")[:10],
        "url": post.get("canonical_url"),
        "access": "free" if post.get("audience") == "everyone" else "paid",
        "type": post.get("type"),  # newsletter, podcast, thread ...
        "wordcount": post.get("wordcount"),
        "reactions": post.get("reaction_count"),
        "comments": post.get("comment_count"),
    }


def scrape(base, max_pages=3, limit=12):
    rows, seen = [], set()
    for page in range(max_pages):
        posts = get_archive_page(base, offset=page * limit, limit=limit)
        if not posts:
            break  # error or end of archive
        new = [to_row(p) for p in posts if p.get("id") not in seen]
        seen.update(p.get("id") for p in posts)
        rows += new
        print(f"{base} offset {page * limit}: {len(new)} posts")
        if len(posts) < limit:
            break  # last page
        time.sleep(random.uniform(3, 6))
    return rows


if __name__ == "__main__":
    # Any public Substack: either name.substack.com or a custom domain.
    sites = sys.argv[1:] or ["https://www.lennysnewsletter.com",
                             "https://www.astralcodexten.com"]
    all_rows = []
    for site in sites:
        for row in scrape(site.rstrip("/"), max_pages=3):
            row["publication"] = site
            all_rows.append(row)
        time.sleep(5)
    if not all_rows:
        raise SystemExit("No posts collected; check the HTTP status printed above.")
    with open("substack_posts.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=list(all_rows[0].keys()))
        w.writeheader()
        w.writerows(all_rows)
    paid = sum(r["access"] == "paid" for r in all_rows)
    print(f"saved {len(all_rows)} posts ({paid} paid, {len(all_rows) - paid} free)")

The two-publication execution produced 72 records: Lenny’s Newsletter had 7 free and 29 paid metadata records; Astral Codex Ten had 33 free and 3 paid. Titles and dates were nonempty. Author and body fields are absent from this CSV, and JSON is the input format rather than a saved output.

The script has a 30-second timeout and no automatic retries. A Requests timeout is not a total download deadline (as of October 2026); see the Requests quickstart. Transport errors or invalid JSON can interrupt execution before the final CSV write.

For HTML extraction, inspect an authorized saved page before choosing selectors; do not invent a universal article selector. Consult the Beautiful Soup reference and our data parsing guide. No separate HTML extraction script was tested here.

How can RSS simplify collection?

RSS is a useful starting point when recent entries satisfy the task. Our two sampled feeds each returned 20 items, including free-post HTML in content:encoded; that observation is not a universal feed limit.

Substack’s RSS documentation identifies the publication’s /feed address. Add that address to your feed reader for ongoing discovery, subject to applicable terms.

For a Python feed-based design, fetch the authorized feed, parse entries, retain stable links, and distinguish summaries from article content. Use the feedparser documentation alongside our RSS parsing tutorial. This run did not test a separate feedparser script.

If a feed fails, inspect status, redirects, and whether the response is XML or an HTML error page. Confirm the publication hostname before changing parsers. A successful subscription tracks future discoveries; it does not establish historical completeness.

How do you retrieve older posts without duplicates?

Our quick run observed offset-based archive pagination on both custom domains. This is an undocumented website endpoint observation, not a supported API contract.

The tested path was /api/v1/archive?sort=new&offset=0&limit=12. The script requested offsets 0, 12, and 24. A separate ACX request using offset 12 and limit 50 returned 50 posts; that is an observed response size, not a maximum.

The code stops on a non-200 response, an empty list, a short page, or its three-page cap. It deduplicates IDs already seen on earlier pages within one publication run. It does not persist checkpoints or eliminate duplicate IDs inside the same response.

For a longer authorized collection, store publication, post ID, canonical URL, retrieval time, next offset, and stopping reason after each successful page. Require valid IDs before advancing. Reconcile overlaps because new publications or deletions can shift offset boundaries.

Reaching an empty list describes that traversal only. The measured execution stopped at its page cap, so 72 records cannot support a download-all claim.

How do you save posts and images for offline use?

Preserve authorized source HTML before converting it into a reading format. The tested CSV is an index and cannot serve as an offline article archive.

Keep the canonical URL, author, publication date, retrieval time, and content status beside each saved document. Mark previews explicitly. Our HTML-to-Markdown guide covers a useful conversion path, but inspect links, tables, captions, and embedded media afterward.

Optional image downloads need their own permission and request budget. Deduplicate asset URLs, check file types, and replace remote references with local paths. Image traffic is excluded from the estimates below.

Browser printing can create a PDF of material already accessible to you. An EPUB conversion likewise starts from authorized local content. Neither format conversion retrieves missing text, and neither guarantees that interactive embeds will survive.

What do failures and proxy costs look like?

Nothing was blocked in our quick run on October 18, 2026. Residential access added latency without changing observed access.

We used a Linux server with Python 3.12, no login or cookies, 3-6 second pauses, and about 20 requests across two publications. The same request approach used direct datacenter access and a US residential route. The archive script ran end to end through residential access; paywalled text was not collected.

The six rows summarize paired checks during 15:57-16:00 UTC. Sizes are rounded response-body measurements, not exact request logs. Archive sizes are pooled across the two publications.

Across the checks, elapsed times were approximately 200-600 ms direct and 900-1,500 ms residential. Individual timestamps, exact byte counts, per-response author values, canonical URLs, and extracted body character counts were not retained. Dependency versions beyond Python 3.12 were not recorded. The reproducibility artifacts are substack_scraper.py and its substack_posts.csv output, not a complete telemetry dataset.

A sampled free page returned content with HTTP 200 on both routes. A reCAPTCHA configuration key explained the word captcha in its HTML; that word alone was not evidence of a challenge. Validate expected fields and readable content even after HTTP 200.

Stop on 403 or 429 and review authorization and rate policy. See our guides to API rate limiting and bounded Requests retries. The tested script stops that publication’s pagination on non-200 responses; it does not globally cancel later publications.

At DataImpulse’s $1/GB entry rate, the body-only bandwidth estimate is mean_body_bytes * 1000 / 1e9 * $1. Using representative sizes, 1,000 archive pages cost approximately $0.051 raw or $0.0074 compressed; 1,000 article pages cost $0.241 raw or $0.045 compressed. These are estimates, not billed totals: headers, compression accounting, retries, and images can change billing.

Which scraper tool fits your budget and coding preferences?

Choose by required output, maintenance capacity, and verified cost. A hosted tool still needs the same permission and completeness checks.

The Apify Substack scraper actor lists metadata, content, comment, and export options. Its free-tier event table lists $0.00115 per metadata post and $0.005 per start (as of October 2026). Its headline pricing differs by tier, so calculate from the applicable event table. These are vendor claims, not results from our run.

Apify pricing lists $5 monthly usage credit on the free plan (as of October 2026). Credit is not an unlimited collection allowance.

For a Substack scraper GitHub option, the Substack2Markdown repository describes Markdown and HTML output with a browsing interface. Inspect its dependencies and recent changes before adoption; our snapshot establishes neither current compatibility nor a maintenance commitment.

What official API and export alternatives exist?

Substack has official API terms, and publication owners have an export workflow. Neither fact turns an observed website endpoint into a supported integration.

The API terms describe public creator and publication information, with access subject to quotas and restrictions (as of October 2026). They link to technical documentation, but that support link returned 404 during verification. No numeric quota, working onboarding procedure, or subscriber-management endpoint is asserted here.

For owners, the export instructions direct you to Settings, Exports, then Create new export. This is an owner workflow, not a subscriber download-all mechanism.

For ongoing integration planning, set a monthly reminder to review official developer announcements and documentation. Record changes to access, schemas, quotas, and permitted uses before updating your application. This is a reader-managed check, not an automatic monitoring service.

Can you collect paid posts, comments and reactions?

This script collects metadata about paid posts, not their restricted text. Our free/paid totals describe audience labels only.

A subscription does not by itself settle automated collection permission. Use an authorized export or explicitly approved access mechanism; changing networks cannot grant access rights.

Comment and reaction counts are summaries, not collections of every underlying object. We did not test comment pagination, deleted comments, nested replies, or reaction identities. Keep those datasets outside completeness claims.

The script also maps every audience value other than everyone to paid. That matched the sampled categories, but a future unknown value could be mislabeled. Review source values before relying on the classification.

What legal and contractual restrictions apply?

Public visibility does not settle permission: copyright, contract, access law, and personal-data rules are separate questions. Substack’s terms expressly prohibit scraping and copying or storing significant portions of content; creator consent alone does not override platform terms.

Copyright affects copying, redistribution, and downstream uses. Contractual restrictions can apply even where an access-law claim fails. In hiQ v. LinkedIn, the Ninth Circuit addressed public-page access under the CFAA in a preliminary-injunction dispute. It did not establish universal permission to scrape or resolve every other claim.

The EU Database Directive can restrict extraction or reuse of protected databases. Under the GDPR, public personal information can still require a lawful basis, minimization, and other compliance measures.

AI training or retrieval does not automatically create an exception to these obligations. Check robots.txt separately on each target host; no host-specific robots policy is asserted here, and robots rules do not grant reuse rights. Our web scraping legal guide offers further context. Ultimately, the reader decides, check local law.

Test the route your authorized workflow actually needs

Start with a small direct sample and measure content quality and bandwidth. If an authorized workflow needs residential routing, DataImpulse offers residential proxies from $1 per GB. It supplies proxies, not a scraping API, managed unblocker, VPN, or dataset, and cannot grant permission or guarantee success.

Last updated: October 18, 2026.