Where our tender data comes from
We ingest public tenders and aid-funded opportunities from more than twelve sources and collapse each into one canonical row: ISO3 country, CPV sector codes, a EUR-normalised budget, deadlines, and the untouched source payload alongside. We describe the pipeline, the taxonomy bridge that lets development-finance and EU procurement notices sit in one feed, and what our coverage figures do and do not establish.
Published
Introduction
Enumerating procurement portals is not hard, and a list of them is not a product. What is hard is making a Ukrainian notice and a Brazilian one comparable: they arrive in different languages, under different schemas, classified in taxonomies that share no identifiers, with budgets in different currencies and dates in different conventions. Everything a firm wants to do with a global feed - filter it, rank it, search it in one query - depends on that reconciliation being done well and being honest about where it fails.
This post describes what happens to a notice between the portal and the feed.
Sources
We pull from more than twelve public sources, spanning national procurement portals and development-bank pipelines. The largest single contributor is TED[1], the European Union official procurement journal, which is why the corpus is deepest in Europe. SAM.gov[2] supplies United States federal opportunities, ProZorro[3] Ukrainian procurement, and the EBRD[4] one of several development-bank pipelines alongside the World Bank and IATI.
| Source | Coverage |
|---|---|
| TED | European Union public procurement notices |
| SAM.gov | United States federal contract opportunities |
| UK Find-a-Tender and Contracts Finder | United Kingdom public contracts |
| World Bank | Procurement under World Bank financed projects |
| IATI | Aid-funded activity data |
| EBRD | Development-bank project notices |
| ProZorro | Ukrainian public procurement |
| PNCP and ANAC | Brazilian public procurement |
| AusTender | Australian government tenders |
| Chile Compra | Chilean public procurement |
Normalisation
An eight-step pipeline pulls each source into staging and then collapses its schema into one canonical row. Country becomes an ISO3 code, sector becomes an array of CPV codes, value becomes a EUR-normalised budget, dates become deadlines. We keep the raw source payload on the row rather than discarding it, which is the detail we would defend hardest: a normalisation bug is recoverable, and a field nobody has needed yet is still there when somebody does.
A later step normalises non-English titles and descriptions into dedicated English fields through a pluggable translation provider. We should be precise about this one, because it is easy to overstate: the provider is configurable and the default is a passthrough that does not translate. Translation bills per call, so turning it on is a deployment-level spend decision rather than something that happens to every notice by default.
Every notice is then embedded as a 1024-dimension vector and retrieved by cosine similarity in Postgres through pgvector[5]. The provider is environment-configurable, and we pin the dimension across all of them - a hosted model in production, the open bge-m3[6] in development - specifically so that changing provider does not invalidate the stored corpus. Separately, tender and firm-reference text is chunked with a 450-token window and 80 tokens of overlap, which is what allows a match to point at which part of a track record is relevant rather than only at which document.
Bridging two sector taxonomies
Development finance classifies work by OECD DAC sector codes[7] and EU procurement classifies it by CPV[8]. A World Bank water and sanitation activity and a European water infrastructure tender describe similar work in two vocabularies that share no identifiers, so a single feed over both is impossible without a mapping. We embed the DAC codes and the 45 CPV division labels into the same vector space and record a mapping wherever cosine similarity clears 0.55.
A related problem is notices that carry no sector classification at all. We score each notice against the same 45 division vectors and write the nearest division with its confidence. Two rules keep this from doing damage: a source-provided classification always wins, so the derived value is used only where the source gave none, and it is used only above a confidence of 0.35.
| Condition | Agreement |
|---|---|
| Above the 0.35 confidence threshold | 95.5% |
| All notices, unthresholded | 49.9% |
| Random baseline | 2.2% |
The effect on filtering is narrow and worth stating exactly. A notice with no source classification that the model classifies confidently is no longer waved through a sector filter it does not match. Notices that remain genuinely unclassifiable still pass through as before, because excluding them would hide real work behind a model failure.
Coverage
Measured on 2026-07-31 across 191,779 canonical notices, the corpus carries 197 distinct ISO3 country codes. Presence is a weak claim on its own, so we report it in bands: 173 countries carry 10 or more notices and 120 carry 100 or more.
| Measure | Value |
|---|---|
| Distinct ISO3 country codes present | 197 |
| Countries with 10 or more notices | 173 |
| Countries with 100 or more notices | 120 |
Discussion
Limitations
The coverage figures are a measurement of what has been ingested, not a commitment about what will be. They move with every pipeline run, they were taken on one date against one database, and we would treat any restatement of them without that date as wrong. A country appearing in the 197 means at least one notice mentions it - which is why we report the bands, and why the band counts are the numbers worth quoting.
Coverage is also not uniform in the way a country count implies. The corpus is deepest where TED is deepest, so European notices dominate both in volume and in metadata completeness. A country in the 120 band is well represented relative to the rest of the corpus, not well represented in absolute terms.
The derived sector classification is validated only where we can validate it. The 95.5% figure is agreement with source-provided divisions on notices that have one - which is, by construction, not the population the derived value is actually used on. Notices with no source classification may be systematically harder to classify, and we have no ground truth for them. We think the threshold is conservative; we cannot demonstrate it on the cases that matter most.
The translation default means the corpus is not uniformly English. Semantic search over a Portuguese notice with an untranslated title works less well than over an English one, and that difference is invisible in a result list. It is a cost decision we have made deliberately and would revisit for a deployment whose markets are mostly non-English.
On measurement basis: every figure in this post counts rows in the canonical table on one date. That is a statement about what has been ingested and nothing more. It is not a statement about freshness, because a country represented entirely by notices that closed two years ago counts the same as one being ingested daily, and we do not currently report per-source recency. It is not a statement about completeness either: we have no independent census of what each portal published to compare against, so we cannot say what fraction of a source we hold. A reader should read the coverage table as a lower bound on breadth and infer nothing from it about depth or currency.
What would make these numbers stronger
Two things, in order. Reporting per-source freshness alongside volume would turn a count into a claim worth making, and it needs no new data - only the publication dates already on every row. Reconciling a sample against each portal own published totals would let us state completeness rather than presence, and is harder, because portals expose that inconsistently or not at all.
For the derived sector classification, the honest experiment is the one the current figure cannot substitute for: hand-label a sample of notices that carry no source classification, and measure against those. That is the population the derived value is actually used on, and until it is labelled the agreement figure remains suggestive rather than probative.
Conclusion
Twelve-plus sources, one canonical row, the raw payload kept, and an embedded bridge between two sector vocabularies that were never designed to interoperate. The parts we would defend are the conservative ones: source classifications outrank derived ones, unclassifiable notices pass rather than being hidden, and coverage is reported in bands with a date on it.
References
- [1]Tenders Electronic Daily. Publications Office of the European Union. The official journal of EU public procurement, and the largest single source in the corpus.
- [2]Contract Opportunities. SAM.gov, U.S. General Services Administration. The United States federal contract opportunities system.
- [3]ProZorro public procurement system. Ministry of Economy of Ukraine. Ukrainian public procurement. Classifies with DK021 rather than CPV, which is why the taxonomy bridge exists.
- [4]European Bank for Reconstruction and Development. EBRD. Development-bank project notices, one of the aid-funded sources.
- [5]pgvector: open-source vector similarity search for Postgres. pgvector contributors. The extension the corpus is retrieved from by cosine similarity.
- [6]BAAI/bge-m3. Beijing Academy of Artificial Intelligence. The open embedding model used in development, and the tokenizer used for chunking.
- [7]IATI Sector codelist (OECD DAC 5-digit). IATI Standard. The development-finance sector taxonomy that the CPV bridge maps from.
- [8]Common Procurement Vocabulary. SIMAP, European Commission. The EU procurement classification. Its 45 divisions are the target vocabulary for derived sector codes.