Commit graph

7 commits

Author SHA1 Message Date
kempersc
0c4c378e06 fix(data): clean up YAML structure in BE/EG custodian files (450 files)
Remove redundant ch_annotator metadata and duplicate ghcid_history entries
that were causing YAML parsing issues. Files now have cleaner, more
consistent structure while preserving all essential data.
2025-12-07 18:46:42 +01:00
kempersc
d9325c0bb5 feat: add web archives integration and improve enrichment scripts
Backend:
- Attach web_archives.duckdb as read-only database in DuckLake
- Create views for web_archives, web_pages, web_claims in heritage schema

Scripts:
- enrich_cities_google.py: Add batch processing and retry logic
- migrate_web_archives.py: Improve schema handling and error recovery

Frontend:
- DuckLakePanel: Add web archives query support
- Database.css: Improve layout for query results display
2025-12-07 17:49:07 +01:00
kempersc
1e01639c56 fix(data): resolve XXX placeholder city codes to actual GeoNames settlements
Rename 144 custodian files from XXX placeholders to resolved city codes:
- BR (65): ASA, RBR, MAN, FAZ, MAC, VIT, FOR, GUA, BRE, GOI, SLU, BHO, XAN, CGR, CUI, etc.
- CH (24): ZUR, BER, GEN, BAL, LUC, etc.
- MX (23): MEX, GDL, MTY, PUE, etc.
- CL (9): SCL, VAL, etc.
- CZ (5): PRG, BRN, MSV, etc.
- KR (4): SEL, etc.
- GB (4): LON, etc.
- FR (3): PAR, etc.
- IN (2): DEL, etc.
- PH, JP, EE (1 each)

City codes derived from GeoNames reverse geocoding using institution coordinates.
GHCID format: {COUNTRY}-{REGION}-{CITY}-{TYPE}-{ABBREV}
2025-12-07 17:47:09 +01:00
kempersc
f284e87d13 feat: add 24,963 heritage custodian records from global extraction
Major batch addition of heritage institution data:
- Japan: 12,077 institutions (libraries, museums, archives)
- Czechia: 6,760 institutions
- Switzerland: 2,390 institutions
- Belgium: 448 institutions
- Belarus: 257 institutions
- Austria: 249 institutions (with corrected GHCIDs)
- Argentina: 235 institutions (bibliotecas populares)
- Brazil: 155 institutions
- Mexico: 110 institutions
- Bulgaria: 98 institutions
- Chile: 83 institutions
- Egypt: 50 institutions
- And additional records from VN, NL, GE, KR, GB, FR, US, IN, etc.

All records include:
- Standardized GHCID identifiers (alphabetic-only abbreviations)
- GeoNames-resolved location data
- ISO 3166-2 region codes
- Provenance metadata with extraction timestamps
2025-12-07 14:24:48 +01:00
kempersc
63a6bccd9b fix: remove custodian files with invalid GHCID special characters
Remove 229 custodian YAML files containing invalid characters in GHCIDs:
- Ampersand (&) in abbreviations (e.g., BM&HS, UNL&AG, DR&IMSM)
- Parentheses in abbreviations (e.g., WHO(RA, VK(, SL()
- Unicode characters in filenames (Ö, Ä, Å, É, İ, Ż, etc.)

These files are replaced with corrected versions using alphabetic-only
abbreviations per AGENTS.md Rule 8 (Special Characters MUST Be Excluded).

Related scripts updated for location resolution.
2025-12-07 14:23:50 +01:00
kempersc
ee4e57bc75 add new entries 2025-12-07 00:26:01 +01:00
kempersc
1635625032 added web annotations 2025-12-06 19:50:04 +01:00