Linking buildings
Architecture photographs are matched with Wikipedia articles.
Matching index cards against building authority data
Many of the Photothek’s index cards document buildings — churches, castles, houses, industrial buildings. For these to be displayed on a map, linked with other collections, and researched together, they need to be uniquely identified and given geocodes:
In this step, a multi-stage pipeline finds the matching Wikipedia article for each building. Location, building type, and even the descriptive elements of the cards serve as clues here, i.e. they increase confidence in the match. The hit is checked against every available detail — whatever remains uncertain is explicitly flagged as uncertain.
The building matching was first applied to the data tables the Photothek created in a digitization project with Google Arts & Culture. The greater consistency of the tabular data made it easier to establish such an automated matching process in the first place. A large body of the Photothek’s topographic photographs is recorded both in the historical inventory and in the tabular data, so this first working pipeline was also usable for the historical inventory and produced a first unified dataset, “Topography Germany” (TopDE).
The same pipeline was then used to extend authority-data matching to the entire, much larger card collection. Alongside the roughly 111,000 existing photo metadata records (the “gsheets”, the source of the TopDE run) and the scanned index cards (Photothek, THY, and others), a third, independent source was added: the ZI’s already structured HiDA catalogue. This comprises around 136,000 records with clean, curated location and building fields, which were reconciled with the scanned cards via the inventory number and used to fill in missing or uncertain information there, without overwriting anything already established. In total, the processed corpus now comprises around 710,000 cards from these three source types.
The location entries on the scanned cards are often heavily abbreviated, particularly for museum names such as “Nat.Gal.”, “Gem.Gal., Dresden”, “Mus.d.bild.K.”
In geo-matching, the initial focus was on identifying entities — also with a view to the fact that establishing a knowledge graph is a key goal of the Photothek. Geocodes were initially obtained via Wikidata.
Further analysis of the generated data showed that this entity-based matching did not fully cover the actual information — for example, residential buildings recorded under their address were left out. This is why entity matching was supplemented with matching via a geo-index.
This observation has since become a methodological principle that now shapes the pipeline: the clean separation of entity, address, and coordinate. An entity is the real, work-holding building or institution that can be mapped to a knowledge base (Wikidata/Wikipedia). An address is a street-level piece of information that can exist even when no clear entity was identified — and vice versa. And a consistent distinction is drawn between three outcome types:
- Building (
building) — a specific building was identified and linked. - Enclosing area (
enclosing) — no building, but the place (street/district/city) could be determined. - Nothing (
none) — nothing reliable could be assigned.
Only in the building case is the entity field populated; in the other cases it is deliberately left empty rather than treating an approximate hit as if it were an exact match. Both matching results — the entity and, where available, the separately resolved address — are transparent and traceable in the data, each with its own provenance.
Technical approach
The pipeline runs entirely locally and processes each building in three consecutive stages:
Stage A — Geolocating and finding candidates. Three things happen first, without any AI involved:
- Geocoding anchor — a local Photon instance (
PHOTON_URL) resolves the location string to coordinates. For unclear addresses,_resolve_address_anchorworks its way upward step by step from street → district → city, and also reports the granularity of the match. Code:search/geo_anchor.py,pipeline/run_city.py. - Boosted Wikipedia search —
boosted_searchqueries the local Elasticsearch indexdewiki_content(WP_INDEX_ES_URL) with title, place, and a geo-boost (25 km radius around the anchor), returning top-K hits includingwikidata_id, coordinates, and distance to the anchor. Code:search/boosted.py,search/client.py. - OSM shortcut — the same Photon server returns OSM POIs near the anchor; their
nametag is resolved againstdewiki_contentvia an exacttitle.keywordlookup. This often hits when the OSM database knows the building’s name better than the ES full-text search does. - Enclosing candidates — articles on the corresponding administrative units are additionally loaded, so that “matches only the place, not the building” is available as an explicit option in the decision step.
Only after this does a language model come into play — and only for the difficult remainder.
Stage B — Re-searching and cross-checking (LLM query expansion, optional). If the top hit from Stage A appears “weak” (a fixed rule decides this, not the model itself), an LLM suggests alternative Wikipedia titles, which are then run against dewiki_content and Wikidata again. The original candidates remain in contention throughout: any newly found suggestion is checked against the anchor and must hold up against the original hits rather than silently replacing them. Code: pipeline/anchor_expand.py.
Stage C — The decision. Three paths, in this order:
- Cache hit from previous runs, provided the prompt version and model tag match. Cache under
data/.judge_cache/. - ES gate: if the top hit is close to the address anchor, has a distinctive title match, and a large score gap to the runner-up → confidence 95, with no LLM call at all.
- LLM judge (
QwenMLXJudge): an in-process MLX runtime runningmlx-community/Qwen3.6-35B-A3B-4bit. It receives the building title, location, photo captions, and the building and enclosing candidates, and returns JSON withkind(building/enclosing/none),index,confidence, andreasoning. Code:pipeline/llm_judge.py,providers/qwen_mlx.py.
The run writes one line per building to a JSONL file (data/full_results.jsonl), is SIGINT-safe and resumable, and uses persistent caches for judge, OSM, and expansion calls, so that corrections and reruns can be applied without extra cost. Code: pipeline/run_full.py, pipeline/run_city.py.
Address resolution
Independently of the building decision, a street-level address is resolved for each geographic entity in a separate step — kept as its own, clearly distinct piece of information. A three-tier ladder of sources is used, from most to least reliable:
- From Wikidata — if the entity has a stored street address, it is adopted (supplemented with postcode and city) and translated into coordinates. This is the default case, covering around 43,700 entities.
- From the card text — if no Wikidata address is available, an attempt is made to extract a street address from the free text of the card itself. Only street-/address-level hits are adopted. Around 18,600 further addresses were added this way.
- No match — if neither is possible (around 96,700 entities), the display falls back to the coarse regional anchor (country/state).
Every address carries its provenance (address_source).
Anchor-query preprocessing (optional)
The quality of the Photon hit largely determines everything that follows — without an anchor, no geo-boosted search can take effect. Two opt-in flags transform the anchor query before it is sent to Photon; a third flag disables the LLM judge for machines without Apple Silicon:
--use-libpostal— libpostal parses the location string into street / house number / city and prepends the address/street candidates to the coarsening ladder. Replaces the previous comma/hyphen heuristic. Code:search/preproc.py. Requires the[libpostal]extra (native build, ~2 GB data file).--llm-expand-anchor— Qwen-MLX rewrites the anchor query before the Photon lookup (e.g. “Nat.Gal., Berlin” → “Nationalgalerie, Berlin”). Code:pipeline/anchor_expand.py. On-disk cache underdata/.judge_cache/anchor_expansion.json.--no-judge— disables the LLM judge entirely; cache + ES gate only. For runs on non-Apple hardware or to purely benchmark the ES/Photon layer.
Measurement (n=100, pure Photon hit rate)
Four A/B(/C) experiments on 2026-05-25 compared the three anchor variants across two input samples — see docs/photon-preprocessing.md:
| Input | A direct | B libpostal | C LLM expansion |
|---|---|---|---|
| cards (abbreviated) | 44 | 58 | 90 |
| gsheets (canonical) | 77 | 89 | 83 (9 regressions) |
- libpostal: a small, regression-free gain on both samples (0 / 1 regressions out of 100). Should become the default once a full run confirms a regression rate ≤ 1% — opt-in until then.
- LLM expansion: the dominant lever on abbreviated archival strings (44 → 90, examples:
Gem.Gal., Dresden→ Gemäldegalerie Alte Meister;Nat.Gal., Berlin→ Nationalgalerie;St. Kolumba, Köln→ Madonna in den Trümmern). Harmful on already-clean inputs: it replaces working queries with ones Photon fails to match (Burg, Kaiserstallung→ generic “Kaiserburg”;Basilika St. Bonifaz→ “Sankt-Bonifatius-Straße”). Should therefore be enabled specifically for the unlinked index cards, not for the already canonically labeled gsheets holdings.
The two flags can be combined; whichever variant is chosen is logged in the JSONL field anchor_expansion.
Results at a glance
Across the three separate runs (production, follow-up processing of previously excluded cards, gsheet/TopDE), around 159,000 geographic entities were formed. In the end, of the roughly 710,000 cards in the overall corpus, around 471,700 (roughly 66%) are linked to a geographic entity, of which around 341,800 to a specific building. A reference dataset checked by curators (120 cases) confirms an accuracy of around 67% exactly correct matches for the actual linking step (around 73% when considering only the cases where the correct article was among the candidates at all); the geographic geolocation itself succeeds in 99–100% of sampled cases. A more detailed account of data provenance, methodology, and quality assurance — aimed at art historians and including the breakdown by source collection, buckets, and runs — can be found in georeferenzierung-methodik-de.md.
Data sources & tools
- Input: scanned index cards (Photothek, THY, and others), the ZI’s structured HiDA catalogue (around 136,000 records, reconciled via inventory number), and
data/photos_per_building_refined.json(~17,400 buildings with location and photo references from the existing gsheets photo metadata), generated viaentity-linking-geo-export-photosfrom thegsheetstable incards.duckdb. - Geocoder & POI lookup: local Photon instance (
PHOTON_URL, defaulthttp://127.0.0.1:2322/api/); startable viadocker compose up -d photon, requiring a one-time download of the Germany OSM dump (~3 GB) (seedocs/photon-setup.md). - Index: local Elasticsearch index
dewiki_content(WP_INDEX_ES_URL,WP_INDEX_NAME, German Wikipedia);wikidata_idandwikidata_urlare read directly from the index — no separate Wikidata call. Populated by the sister projectwp-index. - LLM judge & anchor expansion: local via MLX, model
mlx-community/Qwen3.6-35B-A3B-4bit(optional[mlx]extra). - Address parser: local via libpostal (optional
[libpostal]extra). - IIIF resolution: read-only DuckDB at
--duckdb(zikg_id→iiif_paths); falls back cleanly if the file is missing. - Output: append-only JSONL (
data/full_results.jsonl), written back intocards.duckdbviaentity-linking-geo-importas the tablesgeodataandcard_geodata. - Repository:
entity-linking-geo(internal). Split out ofgeo-agent/in 2026; seedocs/migration-plan.md.
Outlook
Both preprocessing flags each deliver clear gains on opposite input types — the obvious next step is therefore a per-row routing decision rather than a process-wide one. An is_abbreviated() heuristic (e.g. periods in short institution tokens, low token count) should in future decide per index card whether LLM expansion is worthwhile. The pipeline already writes the anchor_expansion field to the JSONL, so the heuristic can be audited against a full run. Once a full run with --use-libpostal confirms the low regression rate, libpostal will become the default and the flag will be dropped.