In this Article
For github scraping with Python, prefer the official GitHub REST API to collect repositories and public user profiles, follow pagination links, and save structured JSON. Use HTML parsing when you specifically need information displayed on a page, such as GitHub Trending.
This tutorial covers authentication, repository extraction, pagination, retries, bounded search, and a Trending parser. The examples have not been execution-tested. Run the API snippets in order in the same Python process; the Trending and proxy examples are standalone.
Key Facts
- Prefer the official API for repository and public profile collection.
- Keep authentication tokens outside source code.
- Follow pagination links before treating an export as complete.
- Inspect rate-limit responses before deciding whether to retry.
- Deduplicate repository records by numeric ID.
- Label Trending collections with their collection time and selected filters.
GitHub Scraping: API or HTML?
The GitHub API First framework is a four-part routine: use the REST or GraphQL API with a token first, and parse HTML only for data the API does not return.
Use the official REST API for structured repository and profile records. Choose HTML parsing when the page presentation itself is the collection target.
The API exposes resources through HTTP endpoints and returns JSON. An HTML collector instead downloads a page and selects elements from its markup, creating a dependency on the page’s current structure. Start with GitHub’s repository endpoint documentation and user endpoint documentation.
Use GET /users/{username} for a public profile, GET /repos/{owner}/{repo} for one repository, and GET /users/{username}/repos for a user’s public repositories.
| Method | Use it when | Avoid it when |
|---|---|---|
| REST API | Use REST when documented endpoints cover the records you need. | Avoid REST when your target is a page-specific presentation that those endpoints do not describe. |
| GraphQL API | Use GraphQL when one query needs related records, such as repositories and their issues; paginate each connection and check the query cost. | Avoid GraphQL when a simple REST request already meets the requirement. |
| HTML parsing | Use HTML when you need the displayed page contents. | Avoid HTML when structured API records meet the same need. |
Competitor data: taken from each provider’s official pricing and documentation pages linked in this article, collected on November 10, 2026.
GraphQL is a cost-based alternative, not a second scraping shortcut. Consult its query cost and resource limits before designing nested queries.
Review GitHub’s Acceptable Use Policies, Section H of its Terms of Service, and robots.txt before collection. GitHub’s information-use restrictions apply to both API and scraped data, including prohibitions on spam and selling users’ personal information. See web scraping legal considerations and how to check whether a website allows scraping for practical review steps.

Initial Setup: Python, Requests, and Authentication
Install Requests and Beautiful Soup, then keep your GitHub token in an environment variable. Use a dedicated API session so its authorization header is not reused for unrelated websites.
Example: install the dependencies in your Python environment.
python -m pip install requests beautifulsoup4
Create a personal access token using GitHub’s authentication instructions. Prefer a fine-grained token where applicable, select only the permissions your endpoints require, and set GITHUB_TOKEN through your environment or secret manager. Do not paste tokens into code or exported records.
Example: initialize the shared API session. The API snippets below build on this setup.
import os
import requests
API = "https://api.github.com"
api = requests.Session()
api.headers.update({
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {os.environ['GITHUB_TOKEN']}",
"X-GitHub-Api-Version": "2026-03-10",
"User-Agent": "repository-research-example"
})
def get(url, **kwargs):
response = api.get(url, timeout=30, **kwargs)
response.raise_for_status()
return response
The timeout is an example configuration, not a measured optimum. Requests documents timeouts, query parameters, and status checks; its session documentation explains shared headers. A descriptive User-Agent identifies the collector; it does not grant access or bypass restrictions.
Collect Public Repositories and User Profiles
Fetch a public profile with GET /users/{username} and a known repository with GET /repos/{owner}/{repo}. Use the user repository listing endpoint when you need an account’s public repositories.
These are documented in GitHub’s Get a user reference and Get a repository reference.
Example: collect selected public fields using the session above.
profile = get(f"{API}/users/octocat").json()
repository = get(f"{API}/repos/octocat/Hello-World").json()
record = {
"user_id": profile["id"],
"login": profile["login"],
"name": profile.get("name"),
"repository_id": repository["id"],
"repository": repository["full_name"],
"stars": repository["stargazers_count"],
"forks": repository["forks_count"],
"language": repository.get("language")
}
print(record)
Preserve null values instead of replacing missing profile details with guesses. Store numeric IDs as deduplication keys and keep readable names for reporting. Choose fields for a specific analysis rather than collecting every available personal detail; our ethical scraping guide discusses collection scope and data handling.
Paginate Results and Export JSON/CSV
Follow the response’s next-page link until none remains, then export the accumulated records. Do not assume the first response contains the complete collection.
The user repository listing accepts up to 100 records per page (as of November 2026). GitHub’s pagination guide documents the Link header.
Example: collect public repositories and write JSON plus a compact CSV. Run this after the setup snippet.
import csv
import json
url = f"{API}/users/octocat/repos"
params = {"per_page": 100, "type": "owner"}
by_id = {}
while url:
response = get(url, params=params)
for repository in response.json():
by_id[repository["id"]] = repository
url = response.links.get("next", {}).get("url")
params = None
repositories = list(by_id.values())
with open("repositories.json", "w", encoding="utf-8") as output:
json.dump(repositories, output, ensure_ascii=False, indent=2)
fields = ["id", "full_name", "stargazers_count", "forks_count"]
with open("repositories.csv", "w", newline="", encoding="utf-8") as output:
writer = csv.DictWriter(output, fieldnames=fields, extrasaction="ignore")
writer.writeheader()
writer.writerows(repositories)
For longer jobs, save each successful page before advancing a checkpoint. Record the next URL, collection time, endpoint, and selected filters. Resume from that checkpoint and deduplicate again; a changing dataset can shift while pagination is in progress.
This small example keeps records in memory and writes only after pagination finishes. Adapt that storage strategy before using it for a large collection. Keep JSON as the fuller export and choose a deliberate column set for CSV.
GitHub Scraping: API Rate Limits and Retry Handling
GitHub separates primary quotas from secondary throttling. Read the response headers and error message before retrying a failed request.
| Request category | Normal limit (as of November 2026) | Scope |
|---|---|---|
| Unauthenticated REST | 60 requests/hour | Originating IP |
| REST authenticated with a personal access token | Normally 5,000 requests/hour | Per user; shared with other requests made on that user’s behalf |
| REST secondary points | Up to 900 points/minute per endpoint | Additional secondary limits also apply |
| Authenticated non-code search | 30 requests/minute | Search endpoints excluding code search |
| Code search | 10 requests/minute | Authentication required |
| GraphQL user requests | 5,000 points/hour | User; exceptions apply |
Competitor data: taken from each provider’s official pricing and documentation pages linked in this article, collected on November 10, 2026.
Sources: GitHub’s REST rate limits, search rate limits, and GraphQL rate limits. Collected on November 10, 2026.
Inspect x-ratelimit-remaining, x-ratelimit-reset, and x-ratelimit-resource. Honor retry-after; otherwise wait for an exhausted quota to reset, or back off for secondary throttling. A generic 403 is not necessarily throttling.
Example: replace the earlier get helper with this bounded retry implementation before running collection snippets.
import time
def get(url, **kwargs):
for attempt in range(4):
response = api.get(url, timeout=30, **kwargs)
headers = response.headers
throttled = response.status_code in (403, 429) and (
"retry-after" in headers
or headers.get("x-ratelimit-remaining") == "0"
or "rate limit" in response.text.lower()
)
if not throttled or attempt == 3:
response.raise_for_status()
return response
delay = 60 * (2 ** attempt)
if "retry-after" in headers:
delay = max(delay, float(headers["retry-after"]))
if headers.get("x-ratelimit-remaining") == "0":
reset = float(headers["x-ratelimit-reset"])
delay = max(delay, reset - time.time() + 1)
time.sleep(delay)
This helper stops after repeated throttling and lets other failures surface. It does not implement network-error recovery. For GraphQL, inspect query cost and remaining points; a nested query is not equivalent to one REST request.
Collect GitHub Search Results at Scale
Partition broad searches into smaller queries and deduplicate their results. Pagination alone cannot overcome Search’s cap of 1,000 results per query (as of November 2026).
GitHub also documents a search-scope limit of up to 4,000 matching repositories and an incomplete_results flag for timed-out searches (as of November 2026). See the official Search API restrictions; a successful response is not proof of exhaustive coverage.
Example: fetch one bounded repository-search partition and reject responses that need further splitting. This uses the API session and retry helper above.
def search_partition(query):
url = f"{API}/search/repositories"
params = {"q": query, "per_page": 100}
found = {}
while url:
response = get(url, params=params)
data = response.json()
if data["incomplete_results"] or data["total_count"] > 1000:
raise RuntimeError("Narrow this search partition")
for repository in data["items"]:
found[repository["id"]] = repository
url = response.links.get("next", {}).get("url")
params = None
if url:
time.sleep(3)
return found
query = "org:github language:Python created:2025-09-01..2025-09-30"
results = search_partition(query)
print(len(results))
This query excludes forks by default. Add fork:true to include both original repositories and forks, or fork:only to collect forks alone. See GitHub’s repository search documentation.
Split an oversized date interval into non-overlapping smaller intervals. If a single day remains too broad, add another suitable qualifier, such as an owner or star range. Merge accepted partitions by repository ID.
Maintain a checkpoint ledger containing the exact query, completion state, next-page URL, and collection timestamp. Persist records before marking a partition complete. Keep rejected partitions pending rather than silently treating their partial results as final.
The date qualifier above selects repository creation dates; it does not reconstruct historical rankings. Search coverage and changing records still constrain completeness. Report the queries and collection window alongside any resulting analysis.
Scrape GitHub Trending with Python
To collect the displayed Trending list, request its HTML and extract repository links from the returned page. Treat the selectors as changeable assumptions and stop if the expected structure disappears.
Use the programming-language and daily, weekly, or monthly controls on GitHub Trending to choose a collection view. Keep the spoken-language filter distinct from the programming-language filter. Copy the selected view’s URL into the collector rather than treating a time window as an archive date.
Example: this standalone parser illustrates a Python weekly view. Its selectors are not a documented GitHub API contract and must be checked against current markup. The parsing methods are documented in the Beautiful Soup reference.
import requests
from bs4 import BeautifulSoup
from datetime import datetime, timezone
url = "https://github.com/trending/python?since=weekly"
response = requests.get(url, timeout=30)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
links = soup.select("article.Box-row h2 a[href]")
if not links:
raise RuntimeError("Inspect the response and update selectors")
rows = []
for link in links:
name = link["href"].strip("/")
if len(name.split("/")) == 2:
rows.append({"repository": name, "url": f"https://github.com/{name}"})
print({"collected_at": datetime.now(timezone.utc).isoformat(),
"source": url, "repositories": rows})
Validate extracted links before adding optional fields such as descriptions or displayed star counts. Do not assume every repository has every field, and distinguish total stars from stars gained during the displayed period.
A collection made now is a current snapshot. It cannot establish what appeared in GitHub Trending in September 2025 or September 2026 without a separately preserved, dated collection. Record the URL, filters, timestamp, and parser version when saving future snapshots.
Optional DataImpulse Proxy Configuration
A proxy is optional for this workflow and should serve a defined routing requirement. It does not expand authenticated GitHub API quotas or guarantee access.
DataImpulse is our product: its residential proxies start at $1 per GB with pay-as-you-go billing, free country targeting, and sticky sessions.
Example: configure the required country and sticky session in your proxy settings, then store the resulting connection URL in PROXY_URL. This standalone request keeps GitHub API authorization out of the HTML session.
Set PROXY_URL to a complete proxy URL, including its scheme. For a SOCKS5 URL, first run python -m pip install 'requests[socks]'; use socks5h:// if DNS resolution should occur through the proxy. See Requests’ SOCKS documentation.
import os
import requests
proxy = os.environ["PROXY_URL"]
response = requests.get(
"https://github.com/trending",
proxies={"http": proxy, "https": proxy},
timeout=30
)
response.raise_for_status()
Requests documents this mapping in its proxy configuration guide. Keep TLS verification enabled and exclude proxy connection secrets from logs.
Choose a sticky session when a permitted workflow needs continuity. Keep the same rate-limit handling regardless of routing, and do not rotate addresses to evade restrictions. Read our web scraping proxy guide when deciding whether an extra routing layer is useful.
Frequently Asked Questions
How do I start github scraping with Python?
Define the repository or public profile fields you need, then choose the matching official API endpoints. Start with a small collection, inspect the responses, and add pagination before exporting a larger dataset.
Should I use REST or GraphQL?
Start with REST for a straightforward repository or profile collector. Use GraphQL when one query needs related records, such as repositories and their issues; paginate each connection, check the query cost, and check errors.
Why does my export contain fewer repositories than expected?
Check whether the collector followed every next-page link and whether its filters match your intended scope. For search-based collections, inspect the result count and incomplete-results flag before accepting a partition.
Should every 403 response trigger a retry?
No. Inspect the response body and headers to distinguish throttling from an access problem. Retrying an unchanged request without understanding the failure can waste requests and hide a configuration error.
Can I retrieve historical GitHub Trending rankings?
Do not label a current page collection as a historical ranking. A claim about an earlier date needs a preserved collection from that date, with its filters and collection metadata.
Do proxies remove GitHub API limits?
Do not treat a proxy as additional authenticated API capacity. Keep the collector’s quota handling and access checks in place, and add a proxy only for a separate routing requirement.
Were these Python examples tested?
No execution results are claimed for these examples. Validate them on a small, permitted collection before extending them, and inspect current HTML before relying on the Trending selectors.
Need country-specific routing for your collection?
If your permitted collection needs a chosen exit country or session continuity, explore DataImpulse residential proxies. Start with the official GitHub API and add proxy routing only when it meets a concrete requirement.
Last updated: November 10, 2026.

State/City/Zip/ASN Targeting 



