kempersc/glam
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
glam/docs/sessions/BULGARIAN_ISIL_LINKML_INTEGRATION_COMPLETE.md
kempersc 3c80de87e0 add isil entries
2025-11-19 23:25:22 +01:00

21 KiB
Raw Permalink Blame History

Bulgarian ISIL Registry - LinkML Integration Complete ✅

Status: Phase 1 Complete | Phase 2 Wikidata Enrichment Ready
Generated: 2025-11-18

Executive Summary

Successfully extracted and converted 94 Bulgarian libraries from the Bulgarian National Library ISIL Registry to LinkML-compliant YAML format with:

  • ✅ 100% schema compliance (validated with linkml-validate)
  • ✅ 61.7% geocoding coverage (58/94 institutions)
  • ✅ 61.7% GHCID coverage (58/94 institutions with complete GHCIDs)
  • ✅ Region mapping for all geocoded institutions (28 Bulgarian oblasts)
  • ⚠️ 0% Wikidata enrichment (Bulgarian ISIL codes not in Wikidata)

Key Finding: Bulgarian heritage institutions are underrepresented in Wikidata. Most lack:

  • ISIL codes (P791 property)
  • Detailed location data
  • Collection metadata

Opportunity: This project can contribute 94 Bulgarian ISIL codes to Wikidata, improving global heritage infrastructure.

What We Accomplished

1. Data Extraction ✅

Script: scripts/scrapers/bulgarian_isil_scraper.py

  • Scraped 94 institutions from Bulgarian National Library website
  • Extracted: ISIL codes, names, addresses, websites, phone/email, collection descriptions
  • Generated CSV and JSON intermediate formats

2. LinkML Conversion ✅

Script: scripts/convert_bulgarian_isil_to_linkml.py
Output: data/instances/bulgaria_isil_libraries.yaml

Schema Compliance:

  • ✅ All 94 records validate against LinkML schema v0.2.1
  • ✅ Proper field mapping:
    • ghcid_current (not ghcid)
    • collections.collection_description (not description)
    • geonames_id as string (not int)
    • Email validation (split multi-email fields)

Taxonomy Mapping:

  • ✅ All 94 institutions classified as LIBRARY (GLAMORCUBESFIXPHDNT taxonomy)
  • Types include: National library, regional libraries, university libraries, community center (chitalishte) libraries, municipal libraries

3. Geographic Enrichment ✅

Geocoding Coverage: 58/94 (61.7%)

Tools:

  • GeoNames SQLite database for lat/lon coordinates
  • Custom city→region lookup table (data/reference/bulgarian_city_regions.json)
  • 138 Bulgarian cities mapped to 28 oblasts (regions)

Limitations:

  • 36 institutions not geocoded (small villages, Cyrillic name mapping gaps)
  • Solution: Manual geocoding or Nominatim API fallback needed

4. Region Mapping ✅

Script: scripts/generate_bulgarian_city_regions.py
Output: data/reference/bulgarian_city_regions.json

Impact:

  • GHCID coverage improved from 24.5% → 61.7% (2.5x increase!)
  • All geocoded institutions now have:
    • Region name (e.g., "София", "Пловдив")
    • ISO 3166-2 code (e.g., "BG-22", "BG-16")
    • Numeric code (e.g., 22, 16)

5. Persistent Identifier Generation ✅

Coverage: 58/94 institutions (61.7%)

GHCID Format: BG-{Region}-{City}-L-{Abbreviation}

  • Example: BG-22-SOF-L-0000 (National Library, Sofia)

Four Identifier Formats Generated:

  1. UUID v5 (SHA-1) - Primary persistent identifier (RFC 4122)
  2. UUID v8 (SHA-256) - Secondary identifier (future-proofing)
  3. Numeric (64-bit) - Compact identifier for CSV exports
  4. UUID v7 - (Not generated yet - database record ID only)

Example:

ghcid_current: BG-22-SOF-L-0000
ghcid_uuid: 367d49be-01b7-54bf-af07-614e2e24c02d
ghcid_uuid_sha256: 8f4dfc5d-4561-8387-906b-fe09acdccd3c
ghcid_numeric: 10326186998156579719

6. Wikidata Enrichment Attempt ⚠️

Script: scripts/enrich_bulgarian_wikidata.py

Results:

  • ✅ Script created with SPARQL query logic
  • ⚠️ 0 institutions enriched (Bulgarian ISIL codes not in Wikidata)
  • ✅ Verified Wikidata MCP tool works (queried Q631641 - National Library of Bulgaria)

Key Discovery:

  • National Library of Bulgaria exists in Wikidata (Q631641) with:

Implication: Most Bulgarian libraries are either:

  1. Not in Wikidata at all
  2. In Wikidata but missing ISIL codes

Data Quality Summary

Metric Count Percentage Status
Total institutions 94 100% ✅ Complete
Schema compliant 94 100% ✅ Validated
With ISIL codes 94 100% ✅ Authoritative
With websites 67 71.3% ✅ Good
With email/phone ~80 ~85% ✅ Good
Geocoded (lat/lon) 58 61.7% ⚠️ Partial
With region info 59 62.8% ⚠️ Partial
With GHCIDs 58 61.7% ⚠️ Partial
With Wikidata Q-numbers 0 0.0% ❌ Missing
Placeholder names 70 74.5% ❌ Needs work

Data Tier: TIER_1_AUTHORITATIVE (official Bulgarian National Library ISIL registry)

Schema Compliance Details

LinkML Schema Version: v0.2.1 (modular)

Modules Used:

  • schemas/core.yaml - HeritageCustodian, Location, Identifier classes
  • schemas/enums.yaml - InstitutionTypeEnum, DataTier, DataSource
  • schemas/provenance.yaml - Provenance metadata
  • schemas/collections.yaml - Collection descriptions

Validation:

linkml-validate -s schemas/heritage_custodian.yaml \
  data/instances/bulgaria_isil_libraries.yaml
# Result: ✅ All records valid

Common Fields:

  • name: Institution name (Bulgarian + English alternative names)
  • institution_type: LIBRARY (all 94 institutions)
  • ghcid_current: Global Heritage Custodian Identifier (58 institutions)
  • identifiers: ISIL codes (94), Website URLs (67)
  • locations: City, address, country (94), lat/lon (58), region (59)
  • collections: Collection descriptions, types, item counts
  • provenance: Data source (CSV_REGISTRY), tier (TIER_1), confidence (0.98)

Placeholder Names Challenge

Problem: 70/94 institutions (74.5%) have placeholder names like "Library BG-0130001"

Why?

  • Original Bulgarian National Library registry HTML uses multiline table cells
  • Institution names are in nested <div> tags with Cyrillic text
  • Scraper couldn't parse nested HTML reliably
  • Fell back to "Library {ISIL-code}" placeholders

Solutions Attempted:

  1. ❌ Wikidata SPARQL by ISIL code → No results (ISIL codes not in Wikidata)
  2. ⚠️ Wikidata fuzzy name matching → Not implemented (needs RapidFuzz + SPARQL)

Solutions Needed:

  1. Re-scrape with better HTML parser:

    • Use BeautifulSoup with get_text(strip=True) on nested divs
    • Extract Cyrillic names from table cells

  2. Manual curation:

    • Export placeholder names to CSV
    • Human reviewer fills in names from Bulgarian National Library website
    • Re-import corrected names

  3. Wikidata fuzzy matching:

    • For the 24 institutions with real names, query Wikidata by name
    • Extract Q-numbers and metadata
    • Update YAML with Wikidata enrichment

Next Steps (Priority Order)

Phase 2a: Fix Placeholder Names 🔴 HIGH PRIORITY

Option 1: Re-scrape (Recommended)

# Update bulgarian_isil_scraper.py with better HTML parsing
# Focus on extracting nested <div> text content
python scripts/scrapers/bulgarian_isil_scraper.py --re-extract-names

Option 2: Manual Curation

# Export placeholder names for human review
python scripts/export_bulgarian_placeholders.py
# Output: data/review/bulgarian_placeholder_names.csv
# Reviewer fills in real names from source website
# Re-import corrected names
python scripts/import_bulgarian_corrected_names.py

Option 3: Wikidata Fuzzy Matching (for 24 non-placeholder names)

# For institutions with real names:
# 1. Query Wikidata by name + location
# 2. Fuzzy match (RapidFuzz score > 0.85)
# 3. Extract Q-number, VIAF, metadata
# 4. Backfill placeholder names with Wikidata labels (if matched)

Phase 2b: Complete Geocoding ⚠️ MEDIUM PRIORITY

Goal: Increase geocoding from 61.7% → 90%+

Approach:

  1. Use Nominatim API for 36 missing cities
  2. Handle small villages and Cyrillic name variants
  3. Manual coordinate lookup for remaining institutions
# Example: Nominatim fallback for missing cities
import requests

def geocode_nominatim(city_name, country="Bulgaria"):
    url = "https://nominatim.openstreetmap.org/search"
    params = {
        "q": f"{city_name}, {country}",
        "format": "json",
        "limit": 1
    }
    response = requests.get(url, params=params)
    results = response.json()
    if results:
        return {
            "latitude": float(results[0]['lat']),
            "longitude": float(results[0]['lon'])
        }
    return None

Phase 2c: Enrich Wikidata with Bulgarian ISIL Codes 🟢 CONTRIBUTION OPPORTUNITY

Goal: Add 94 Bulgarian ISIL codes to Wikidata (P791 property)

Process:

  1. For institutions already in Wikidata (e.g., Q631641 - National Library):

    • Use MCP wikidata-authenticated_add_claim to add ISIL code
    • Example: add_claim(entity_id="Q631641", property_id="P791", value="BG-2200000", value_type="string")

  2. For institutions NOT in Wikidata:

    • Create new Wikidata entities using wikidata-authenticated_create_entity
    • Add ISIL codes, names, locations, parent organizations
    • Link to existing places (cities, regions)

Example Wikidata Contribution:

# Add ISIL code to National Library of Bulgaria
from wikidata_mcp import add_claim

add_claim(
    entity_id="Q631641",
    property_id="P791",  # ISIL code property
    value="BG-2200000",
    value_type="string"
)

# Create new entity for Belitsa library
create_entity(
    labels={"bg": "Библиотека при Народно читалище „Георги Тодоров-1885"",
            "en": "Library of Georgi Todorov – 1885 Chitalishte"},
    descriptions={"bg": "Читалищна библиотека в Белица",
                  "en": "Community center library in Belitsa"}
)
# Then add ISIL code BG-0130000

Impact:

  • Improves global heritage infrastructure
  • Makes Bulgarian libraries discoverable via Wikidata queries
  • Enables cross-referencing with other heritage platforms (Europeana, DPLA)

Phase 2d: RDF Export 🟢 LOW PRIORITY

Goal: Export Bulgarian libraries as RDF/Turtle for Linked Open Data

Format: RDF/Turtle with TOOI, CPOV, Schema.org ontologies

python scripts/export_bulgaria_rdf.py
# Output: data/rdf/bulgaria_isil_libraries.ttl

Sample RDF:

@prefix heritage: <https://w3id.org/heritage/custodian/> .
@prefix schema: <http://schema.org/> .
@prefix isil: <https://isil.org/> .

heritage:bg/bg2200000 a schema:Library ;
    schema:name "Национална библиотека „Св. св. Кирил и Методий"" ;
    schema:alternateName "National library St. Cyril and St. Methodius" ;
    schema:identifier "BG-2200000" ;
    schema:url <http://www.nationallibrary.bg> ;
    schema:location [
        a schema:Place ;
        schema:addressCountry "BG" ;
        schema:addressLocality "Sofia" ;
        schema:geo [
            a schema:GeoCoordinates ;
            schema:latitude "42.69751" ;
            schema:longitude "23.32415"
        ]
    ] .

Phase 2e: Integration with Global Dataset 🟢 LOW PRIORITY

Goal: Merge Bulgarian libraries with global heritage custodian dataset

Process:

  1. Copy data/instances/bulgaria_isil_libraries.yaml to global instances directory
  2. Update global GHCID registry
  3. Generate global geographic visualizations
  4. Publish to heritage discovery platform

Technical Implementation Details

Files Created/Modified

File Status Purpose
scripts/scrapers/bulgarian_isil_scraper.py ✅ Complete HTML scraper for Bulgarian ISIL registry
scripts/convert_bulgarian_isil_to_linkml.py ✅ Complete CSV → LinkML YAML converter
scripts/generate_bulgarian_city_regions.py ✅ Complete City→region lookup table generator
scripts/enrich_bulgarian_wikidata.py ⚠️ Skeleton Wikidata enrichment (needs implementation)
data/isil/bulgarian_isil_registry.csv ✅ Complete Intermediate CSV format
data/isil/bulgarian_isil_registry.json ✅ Complete Intermediate JSON format
data/instances/bulgaria_isil_libraries.yaml ✅ Complete Final LinkML output
data/reference/bulgarian_city_regions.json ✅ Complete City→region mappings (138 cities)

Dependencies Used

Python Libraries:

  • linkml-runtime - Schema validation and data models
  • pyyaml - YAML parsing/serialization
  • sqlite3 - GeoNames database queries
  • hashlib - UUID v5/v8 generation

External Data:

GHCID Generation Algorithm

For Bulgarian institutions:

Format: BG-{RegionCode}-{CityAbbrev}-{TypeCode}-{InstAbbrev}

Example: BG-22-SOF-L-0000
         |  |   |   | |
         |  |   |   | +--- Institution abbreviation (0000 = first in city)
         |  |   |   +----- Type code (L = Library)
         |  |   +--------- City abbreviation (SOF = Sofia)
         |  +------------- Region code (22 = Sofia oblast)
         +---------------- Country code (BG = Bulgaria)

Abbreviation Rules:

  • City: First 3 uppercase letters (transliterated if Cyrillic)
  • Institution: 4-digit sequential number (0000, 0001, 0002, etc.)
  • No Q-numbers appended (no collisions detected yet)

UUID Generation:

import uuid
import hashlib

# UUID v5 (SHA-1)
namespace = uuid.UUID('6ba7b810-9dad-11d1-80b4-00c04fd430c8')
ghcid_uuid = uuid.uuid5(namespace, ghcid_string)

# UUID v8 (SHA-256)
sha256_hash = hashlib.sha256(ghcid_string.encode('utf-8')).digest()
ghcid_uuid_sha256 = uuid.UUID(bytes=sha256_hash[:16], version=8)

# Numeric (64-bit)
ghcid_numeric = int.from_bytes(sha256_hash[:8], byteorder='big')

Known Issues and Limitations

1. Placeholder Names (74.5% of dataset)

Severity: 🔴 High
Impact: Makes data hard to use for discovery and citation

Root Cause:

  • HTML table parsing failed to extract nested <div> content
  • Scraper fell back to placeholder names

Workaround:

  • 24 institutions have real names (manually extracted during initial scraping)
  • Can be identified by not starting with "Library BG-"

Fix: Re-scrape with improved HTML parser (BeautifulSoup)

2. Missing Geocoding (38.3% of dataset)

Severity: ⚠️ Medium
Impact: 36 institutions cannot have GHCIDs or geographic visualizations

Root Cause:

  • Small Bulgarian villages not in GeoNames database
  • Cyrillic name variants not matched

Workaround:

  • Use Nominatim API fallback
  • Manual coordinate lookup for remaining institutions

3. No Wikidata Enrichment (100% of dataset)

Severity: 🟢 Low (expected for new ISIL registry)
Impact: Missing Q-numbers, VIAF IDs, founding dates

Root Cause:

  • Bulgarian ISIL codes not in Wikidata (P791 property missing)
  • Most Bulgarian libraries underrepresented in Wikidata

Opportunity:

  • This project can contribute ISIL codes to Wikidata
  • Improves global heritage infrastructure

4. Email Validation Edge Cases

Severity: 🟢 Low
Status: ✅ Fixed

Root Cause:

  • Some institutions had comma-separated multiple emails
  • LinkML email validation expects single email

Fix: Script now splits multi-email strings and takes first email

Validation and Testing

Schema Validation ✅

cd /Users/kempersc/apps/glam
linkml-validate -s schemas/heritage_custodian.yaml \
  data/instances/bulgaria_isil_libraries.yaml

# Result: ✅ All 94 records pass validation

Data Quality Checks ✅

# Count institutions
grep -c "^- id:" data/instances/bulgaria_isil_libraries.yaml
# Result: 94

# Count placeholder names
grep -c "Library BG-" data/instances/bulgaria_isil_libraries.yaml
# Result: 70

# Count GHCIDs
grep -c "ghcid_current:" data/instances/bulgaria_isil_libraries.yaml
# Result: 58

# Count with Wikidata
grep -c "identifier_scheme: Wikidata" data/instances/bulgaria_isil_libraries.yaml
# Result: 0

Geocoding Coverage ✅

# Count lat/lon coordinates
grep -c "latitude:" data/instances/bulgaria_isil_libraries.yaml
# Result: 58 (61.7%)

Documentation References

Project Documentation:

  • AGENTS.md - AI agent instructions for GLAM data extraction
  • docs/SCHEMA_MODULES.md - LinkML schema architecture (v0.2.1)
  • docs/PERSISTENT_IDENTIFIERS.md - GHCID specification
  • docs/UUID_STRATEGY.md - UUID v5/v7/v8 comparison
  • docs/WHY_UUID_V5_SHA1.md - SHA-1 safety rationale

Schema Files:

  • schemas/heritage_custodian.yaml - Main schema (imports all modules)
  • schemas/core.yaml - Core classes (HeritageCustodian, Location, Identifier)
  • schemas/enums.yaml - Enumerations (InstitutionTypeEnum, DataTier)
  • schemas/provenance.yaml - Provenance tracking
  • schemas/collections.yaml - Collection metadata

Related Sessions:

  • BULGARIAN_ISIL_EXTRACTION_COMPLETE.md - Initial extraction summary
  • SESSION_SUMMARY_20251118_ISIL_PROCESSING.md - Today's session notes

Lessons Learned

1. HTML Parsing Challenges

Lesson: Always test HTML parsing on real data with nested structures.

Issue: Bulgarian National Library uses nested <div> tags in table cells. Simple .text extraction didn't work.

Solution: Use BeautifulSoup with .get_text(strip=True) or .find_all('div') to extract nested content.

2. Wikidata Coverage Varies by Country

Lesson: Don't assume ISIL codes are in Wikidata. Many national registries are not synchronized.

Issue: Bulgarian ISIL codes missing from Wikidata (even for National Library).

Opportunity: Contributing ISIL codes to Wikidata improves global heritage infrastructure.

3. LinkML Email Validation is Strict

Lesson: Validate data types early. LinkML won't accept comma-separated emails.

Issue: Some institutions had multiple emails in single field: "email1@example.com, email2@example.com"

Solution: Split and take first email, or create separate contact_info.additional_emails field.

4. Region Mapping 2.5x Improvement

Lesson: Small lookup tables can dramatically improve data completeness.

Issue: Only 24.5% of institutions had region codes (needed for GHCID generation).

Solution: Created 138-city lookup table → 61.7% coverage (2.5x increase).

5. Placeholder Names Reduce Usability

Lesson: Preserve original data even if hard to parse. Placeholders make data unusable.

Issue: 70/94 institutions have "Library BG-XXXXXXX" names.

Impact: Makes dataset hard to use for discovery, citation, or human browsing.

Fix: Re-scrape with better HTML parser to extract Cyrillic names.

Success Metrics

Metric Target Achieved Status
Institutions extracted 94 94 ✅ 100%
Schema compliance 100% 100% ✅ 100%
ISIL codes extracted 94 94 ✅ 100%
Geocoding coverage 90% 61.7% ⚠️ 68.6%
GHCID coverage 90% 61.7% ⚠️ 68.6%
Real names (not placeholders) 90% 25.5% ❌ 28.3%
Wikidata enrichment 50% 0% ❌ 0%
Region mapping 90% 62.8% ⚠️ 69.8%

Overall Completion: Phase 1 Complete (75%) | Phase 2 Pending (25%)

Conclusion

Phase 1 (Data Extraction & LinkML Conversion): ✅ COMPLETE

We successfully:

  • Extracted 94 Bulgarian libraries from official ISIL registry
  • Converted to LinkML-compliant YAML (100% validation)
  • Generated persistent identifiers (GHCIDs with 3 UUID formats)
  • Geocoded 61.7% of institutions with region mapping
  • Created reusable city→region lookup table (138 cities)

Phase 2 (Wikidata Enrichment): ⚠️ PENDING

Key challenges:

  • 70 institutions (74.5%) have placeholder names → Need re-scraping
  • 36 institutions (38.3%) not geocoded → Need Nominatim fallback
  • 0 institutions (0%) with Wikidata Q-numbers → ISIL codes not in Wikidata

Key Opportunity: This project can contribute 94 Bulgarian ISIL codes to Wikidata, improving:

  • Global heritage discoverability
  • Cross-referencing with Europeana, DPLA, and other platforms
  • Linked Open Data ecosystem for cultural heritage

Next Session Handoff

Priority 1: Fix placeholder names (re-scrape or manual curation)
Priority 2: Complete geocoding (Nominatim API fallback)
Priority 3: Enrich Wikidata with Bulgarian ISIL codes

Files to Review:

  • data/instances/bulgaria_isil_libraries.yaml - Final LinkML output
  • scripts/convert_bulgarian_isil_to_linkml.py - Conversion logic
  • scripts/enrich_bulgarian_wikidata.py - Wikidata enrichment skeleton

Commands to Run:

# Validate schema compliance
linkml-validate -s schemas/heritage_custodian.yaml \
  data/instances/bulgaria_isil_libraries.yaml

# Check data quality
grep -c "Library BG-" data/instances/bulgaria_isil_libraries.yaml  # Placeholder count
grep -c "ghcid_current:" data/instances/bulgaria_isil_libraries.yaml  # GHCID count
grep -c "latitude:" data/instances/bulgaria_isil_libraries.yaml  # Geocoding count

Report Generated: 2025-11-18
Session Duration: ~2 hours
Agent: OpenCODE with Claude
Schema Version: LinkML v0.2.1 (modular)
Data Tier: TIER_1_AUTHORITATIVE (Bulgarian National Library)