In this Article
Government contract data is one of the few valuable datasets that is published on purpose. Procurement transparency rules mean tenders, awards and supplier records are meant to be public, which changes the whole shape of the problem: the hard part is not access, it is coverage and normalization.
This guide covers which official sources to exhaust first, when scraping becomes necessary, why national portals behave differently for foreign visitors, and what it takes to make records from different countries comparable.
Key Facts
- Most of this data is published deliberately. Procurement transparency is a legal obligation in most democracies, so the first move is always to look for the official feed.
- The major systems have APIs: the United States, the European Union and the United Kingdom all publish machine-readable procurement data without scraping.
- Scraping is for the long tail: municipal portals, agency-specific systems and countries whose national platform offers only a web interface.
- National portals frequently restrict foreign traffic, so a tender monitor run from one country silently misses entire jurisdictions.
- Normalization is the real work. Every jurisdiction has its own classification codes, currencies, thresholds and stage names, and merging them is harder than collecting them.
- Normalization beats collection for effort. Classification codes, currencies, thresholds, stage names and supplier identities all differ by jurisdiction, and reconciling them is the bulk of the work.
Which official sources should you use first?
Before writing any scraper, check for a published feed. The major systems have one.
- United States. Federal opportunities and awards are published through official government systems with documented APIs, covering solicitations, award notices and contractor registration data.
- European Union. Above-threshold tenders across member states are published centrally and available in machine-readable form.
- United Kingdom. Contract opportunities and awards are published through the government’s own service with structured export.
- Open contracting standards. A growing number of countries publish procurement data in a shared schema specifically so that cross-country analysis is possible.
Two practical notes. Official APIs frequently carry rate limits and require a free key, which is a fair trade for stability. And the central systems cover above-threshold contracts: smaller awards, which are often the accessible ones for a mid-sized supplier, sit on individual agency and municipal portals that never reach the central feed.
When is scraping actually necessary?
Use a scraper when no feed exists, and avoid one when the data is already published in machine-readable form.
| Situation | Use this when | Avoid when |
|---|---|---|
| Federal or EU-wide opportunities | Official API | Never scrape what has a documented feed |
| Municipal and county portals | Polite scraping, one request at a time per portal | The portal publishes an RSS or export |
| National portals with no API | Scraping with an exit in that country | The portal is reachable and identical from anywhere |
| Historical award analysis | Bulk downloads where offered | Always prefer a bulk file to a million page requests |
The long tail is where the work is. A country may have one central platform and several hundred agency systems, and the agency systems are where a supplier finds contracts it can realistically win.
Why do national portals behave differently abroad?
They are built for domestic users, and foreign traffic is treated as suspicious by default.
Three patterns recur. Some portals geo-restrict outright, refusing connections from outside the country. Some serve a reduced interface or a different language without indicating that anything was withheld. And some sit behind commercial anti-bot services tuned aggressively, where a datacenter address is challenged before it ever reaches a tender listing.
The consequence for a monitoring pipeline is silent, which is what makes it dangerous: the crawler reports success, the record count for that country is simply lower than it should be, and nobody notices until a supplier asks why a tender they saw never appeared. Checking each source from an exit in its own country is the cheapest way to detect it, and DataImpulse residential covers 195 countries at $1 per GB with country targeting included.
What makes normalization hard?
Every jurisdiction describes the same thing differently. Four fields carry almost all of the difficulty, and we call them the 4-part normalization model.
- Classification codes. The EU uses one vocabulary, the US another, and national systems often use a third. Mapping between them is lossy and needs documented rules.
- Currency and thresholds. Values need a conversion date, not just a rate, and publication thresholds differ, so coverage varies by country in ways unrelated to actual activity.
- Stage names. Notice, tender, award, modification and cancellation map imperfectly across systems, and a record can change stage without changing identifier.
- Entity resolution. The same supplier appears under several legal names, registration numbers and transliterations, which is the field most analyses quietly get wrong.
Store the raw record alongside the normalized one. When a mapping turns out to be wrong, and it will, re-deriving from the original is the difference between a fix and a re-collection.
What are the rules?
Read the terms even on open data. Public availability is not a blanket licence; many portals attach attribution or redistribution conditions, and those are easy to honour once you know them.
Be gentle. Procurement portals are frequently small public-sector systems with modest capacity. One request at a time per portal, generous delays, and an identifiable user agent are both polite and practical, because these are organisations you may later want to work with.
Personal data appears in tender documents. Contact names, signatures and sometimes more sit inside published attachments. Publication does not remove privacy obligations when you store and process them.
General information, not legal advice. Related reading: is web scraping legal, 403 Forbidden when scraping.
Frequently Asked Questions
Is government contract data free to access?
Largely yes. Procurement transparency obligations mean tenders and awards are published deliberately, and the United States, European Union and United Kingdom all offer machine-readable access. The cost appears in the long tail of municipal and agency portals that publish only as web pages.
Do I need to scrape procurement portals?
Only where no feed exists. Exhaust the official APIs and bulk downloads first, then scrape the municipal, agency and national portals that publish through a web interface only. That long tail is also where contracts small enough for a mid-sized supplier tend to live.
Why do some national procurement portals block foreign visitors?
Because they are built for domestic users and treat foreign traffic as suspicious. Some refuse connections outright, some serve a reduced interface without saying so, and some sit behind anti-bot services that challenge datacenter addresses. The failure is silent, so coverage drops without an error.
What is the hardest part of building a tender dataset?
Normalization rather than collection. Classification vocabularies, currencies and conversion dates, publication thresholds, stage names and supplier entity resolution all differ by jurisdiction, and merging them correctly takes more engineering than fetching the records.
Are there privacy obligations for published tender data?
Yes. Tender documents and award notices contain contact names and other personal details, and the fact that a public body published them does not remove your obligations when you store and process them. General information, not legal advice.
See every portal the way its own country sees it
A national procurement portal that quietly serves a reduced page to foreign visitors costs you records without ever returning an error. DataImpulse residential proxies give exits in 195 countries at $1 per GB with country targeting included. Create an account and compare one portal from inside and outside its country.
Related: what is geo-blocking · is web scraping legal · web scraping use case.
Last updated: September 17, 2026.
State/City/Zip/ASN Targeting 
