Disambiguate artists and reconcile with authority data

Artist names in the inventory are matched against a reconciliation index and enriched with data from Wikipedia, ULAN, and Foto Marburg.

Making authority data usable for our own dataset

The index cards of the historical inventory contain creator names — artists, workshops, engravers, publishers, notnames (“Master of …”), monogrammists, and anonymous attributions. These names are spelled inconsistently, vary in completeness, and are sometimes garbled by OCR. To turn them into reliable entries linked to authority data, this step first builds a clean, deduplicated creator list from all roughly 710,000 cards — no longer only from the curated artist index — and then matches it against a merged authority dataset from Wikidata, Getty ULAN, and Foto Marburg (around 1.03 million people).

Every match is enriched: life dates and places are merged, a biographical paragraph from German Wikipedia is added, external identifiers (GND, ULAN, VIAF, ISNI, Foto Marburg) are collected, and relationships between people such as teacher-student relations and stylistic influences are resolved. The result is a cleaned artist index in which every creator gets a unique slug, a confidence level, and a link to all associated cards — the data foundation for person pages, filters, and alphabetical registers in the UI.

Because a false link damages the research more than a missing one, the process is consistently precision-oriented throughout: uncertain cases are not linked automatically but routed into a prioritized review queue for manual follow-up. Art-historical special cases — uncertain attributions, workshop/circle/“after”, notnames, and monogrammists — are handled specifically rather than flattened.

Technical approach

The run consists of two cooperating halves: one that builds a canonical, deduplicated creator list from all cards, and one that links that list against the authority dataset. A prerequisite check first verifies the OpenSearch connection and the existence of the required indices (artists, dewiki_content, enwiki_content).

Part A — Building the canonical creator list:

  1. Extracting names — for each card, the best creator entry is chosen following a source quality ranking (the curated artist index kue_artists before HiDA before the field extracted from the card text) and split at genuine delimiters (never at a comma, since that usually marks “surname, first name”).
  2. Classifying person type — Person, notname, monogrammist, anonymous description, workshop/circle, or uncertain; predominantly rule-based, with unclear cases going to the language model.
  3. Merging name variants — exact matches (including reordered names or accents) are merged for free; Jaro-Winkler surfaces suspect pairs for similar but non-identical spellings; a language model decides on these, taking life dates into account.
  4. Unifying regions, periods, and styles — year/period data is deliberately determined by rules alone, while the language model assigns region and style, with the option to abstain when unclear.

Result: a canonical creator list of around 38,800 entities, covering the entire card corpus.

Part B — Matching against authority data (reconciliation):

  1. Candidate search — for each cleaned name, the W3C Reconciliation API of the merged artists index (Wikidata, ULAN, Foto Marburg; around 1.03 million people) returns the most likely candidates via BM25; life dates are supplied as a soft additional constraint. The reconciliation index itself is built in a separate backend project (wd2elastic) — this repository only consumes it.
  2. Re-ranking (local re-ranker) — a purely computational, traceable step (no language model) that weighs name match, date compatibility, generation/ordinal number (a hard exclusion criterion on conflict), role suitability (pure dealer/publisher/collector profiles are downweighted), identifier density, and source authority against each other, and stores a traceable rationale. This replaces the earlier simple combination of a BM25 score, an identifier boost (+30), and a year boost.
  3. Confidence classification — four bands (high/medium/low/rejected, previously only three); only high and medium are auto-accepted, and even then only if additional safeguard rules pass (uncertain attributions are never auto-accepted, closely-tied same-name candidates go to the review queue, short “famous” names are treated as especially risky).
  4. AI final check (tail judge) — new compared to the earlier procedure: for auto-accepted cases drawn purely from card text (the “long tail”), a language model (Gemini Flash) performs a final review of each assignment. It can only confirm or downgrade, and chooses exclusively among the already-presented candidates — ruling out fabrication.
  5. Merging across sources — records from kue_artists, HiDA, and backfill that hit the same authority identifier are merged into a single person, so no duplicates arise across sources.

Wherever a language model is needed, the task passes through a cost-ordered model cascade Qwen → Gemini Flash → Gemini Pro: it starts with the cheapest model and only escalates to a stronger one for the unresolved remainder. The tail-judge final check (step 8) deliberately stays on Gemini Flash, since Qwen performed measurably worse there. A checkpoint mechanism allows interrupted runs to be resumed.

The enriched records are written to the creatordata table (slug-keyed) in cards.duckdb. A field_sources JSON documents, per field, which source supplied the value.

Results at a glance

The final production run (June 2026) expanded the creator base from 21,105 to 45,254 records (+24,310 new creators, mostly gained from backfilling the ~710,000 cards). Of the previously existing records, 6,105 changed their assigned identity — including 4,444 withdrawn, previously unfounded links (the deliberate cost of precision orientation), 1,206 switches to a better-matching same-name entry, and 455 links established for the first time. Calibrated confidence bands: high 20,844 · medium 3,039 · low 14,745 · rejected 6,626. Around 24,000 records (just under half) were auto-accepted, with around 21,000 sitting in the importance-sorted review queue. On hand-checked gold standards and an independently held-out control set, the automatic linking reaches 100% precision.

A more detailed account aimed at art historians — covering data provenance, methodology, and quality assurance, including gold standards, per-phase quality gates, and a discussion of strengths and limitations — can be found in creator-pipeline-dokumentation-de.md.

Data sources & tools

  • Input: the entire card corpus (~710,000: Photothek, THY, HiDA, PH01–05, unislgmuc, and others), the curated artist index kue_artists (~191,000, the best name source, given priority), OpenSearch indices artists, dewiki_content, enwiki_content, ulan.duckdb (fallback), Foto Marburg XML
  • Authority dataset: merged from Wikidata (~820,000 people with art-related external identifiers), Getty ULAN (~267,000), and Foto Marburg (~100,000) — around 1.03 million people in total, project wd2elastic
  • Matching: W3C Reconciliation API (external, wd2elastic)
  • Scoring: BM25 candidate search + local re-ranker (name match, dating, generation/ordinal number, role suitability, identifier density, source authority)
  • Language models: cost cascade Qwen → Gemini Flash → Gemini Pro; tail judge (step 8) fixed on Gemini Flash
  • Output: cards.duckdb::creatordata, ~489,000 card-to-creator links (around 99.4% resolved)
  • Repository: entity-linking-creators (internal)

Outlook

The largest remaining loss lies not in evaluating candidates but in finding rare or heavily distorted names in the first place (the search bottleneck) — an early measurement phase specifically identified this. The “long tail” from pure card text remains, despite tail-judge safeguarding, the area of highest residual risk. Recovering creators from pure work titles is currently on hold, since it can confuse creators with depicted persons. The backend repository’s earlier evaluation script (evaluate_linking.py, Jaro-Winkler/Levenshtein/Damerau-Levenshtein/Jaccard against the card name) served as a diagnostic tool for the re-ranking now used in production (step 6) and remains available for future retuning of the re-ranker.