kempersc/glam
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
glam/SESSION_SUMMARY.md
kempersc 3c80de87e0 add isil entries
2025-11-19 23:25:22 +01:00

5.8 KiB
Raw Blame History

Session Summary: ChangeEvent Model Implementation

Date: 2025-11-05
Status: ✅ COMPLETE

Objectives Accomplished

1. ✅ Implemented ChangeEvent Model in Python

Added to src/glam_extractor/models.py:

  • ChangeType enum with 12 event types:
    • FOUNDING, CLOSURE, MERGER, SPLIT, ACQUISITION
    • RELOCATION, NAME_CHANGE, TYPE_CHANGE, STATUS_CHANGE
    • RESTRUCTURING, LEGAL_CHANGE, OTHER
  • ChangeEvent Pydantic model class with fields:
    • event_id (str, required, unique identifier)
    • change_type (ChangeType enum, required)
    • event_date (date, required)
    • event_description (str, optional)
    • affected_organization (str, optional, organization ID)
    • resulting_organization (str, optional, organization ID)
    • related_organizations (List[str], optional, organization IDs)
    • source_documentation (HttpUrl, optional)

2. ✅ Added change_history to HeritageCustodian

Updated HeritageCustodian model:

  • Added change_history: List[ChangeEvent] field
  • Default: empty list
  • Stores chronological list of organizational change events
  • Fully integrated with schema definition (v0.2.0)

3. ✅ Updated Orchestration Script

Modified scripts/extract_with_agents.py:

  • Imported ChangeEvent and ChangeType classes
  • Implemented ChangeEvent parsing in create_heritage_custodian_record()
  • Added date parsing logic (handles ISO strings and date objects)
  • Added change_type enum mapping
  • Includes validation (skips events with invalid/missing dates)
  • Populates change_history field in HeritageCustodian records

4. ✅ Validated Implementation

Testing Results:

  • ✅ All 207 tests pass
  • ✅ 91% code coverage maintained
  • ✅ ChangeEvent model creation works
  • ✅ HeritageCustodian with change_history works
  • ✅ Orchestration script runs without errors
  • ✅ Brazilian GLAM conversation file loads successfully

File Changes

Modified Files:

  1. src/glam_extractor/models.py (+23 lines)

    • Added ChangeType enum
    • Added ChangeEvent class
    • Added change_history field to HeritageCustodian

  2. scripts/extract_with_agents.py (+35 lines)

    • Imported ChangeEvent and ChangeType
    • Implemented ChangeEvent parsing logic
    • Added change_history to custodian creation

Schema Alignment:

  • ✅ Python models now match LinkML schema v0.2.0
  • ✅ ChangeEvent class matches schema definition (lines 460-484)
  • ✅ ChangeTypeEnum matches schema enum (lines 220-252)
  • ✅ PROV-O integration ready (prov:Activity mapping)

Code Examples

Creating a ChangeEvent:

from datetime import date
from glam_extractor.models import ChangeEvent, ChangeType

event = ChangeEvent(
    event_id='nha-merger-2001',
    change_type=ChangeType.MERGER,
    event_date=date(2001, 1, 1),
    event_description='Merger of Gemeentearchief Haarlem and Rijksarchief in Noord-Holland',
    affected_organization='gemeentearchief-haarlem',
    resulting_organization='noord-hollands-archief'
)

HeritageCustodian with Change History:

custodian = HeritageCustodian(
    id='nha-001',
    name='Noord-Hollands Archief',
    institution_type=InstitutionType.ARCHIVE,
    change_history=[event],
    provenance=provenance
)

Next Steps (Ready for Execution)

Immediate Priority:

  1. Test Agent System with Real Data
    • Pick one conversation file (Brazilian GLAM recommended)
    • Run orchestration script to generate prompts
    • Invoke each agent via @mention in OpenCode
    • Collect JSON responses from agents
    • Validate outputs match expected schema

Testing Workflow:

# 1. Generate prompts
python scripts/extract_with_agents.py \
  "2025-09-22T14-40-15-0102c00a-4c0a-4488-bdca-5dd9fb94c9c5-Brazilian_GLAM_collection_inventories.json"

# 2. In OpenCode, invoke agents:
@institution-extractor <paste prompt>
@location-extractor <paste prompt>
@identifier-extractor <paste prompt>
@event-extractor <paste prompt>

# 3. Collect JSON responses and validate
# (Next session: implement response collection + validation)

Near-Term Tasks:

  1. Process first full extraction (combine agent outputs)
  2. Validate with LinkML schema
  3. Export to JSON-LD
  4. Review data quality and confidence scores

Medium-Term Tasks:

  1. Batch process all 139 conversations
  2. Cross-link with Dutch CSV data
  3. Generate GHCIDs for all institutions
  4. Export to multiple formats (RDF, CSV, Parquet)
  5. Build SPARQL endpoint

Technical Debt Resolved

✅ Removed:

  • Old TODO comments about ChangeEvent implementation
  • Placeholder code in orchestration script

✅ Fixed:

  • Schema-model alignment issues
  • Missing ChangeEvent model
  • Missing change_history field

Project Stats (Updated)

  • Schema: v0.2.0 (ChangeEvent support) ✅
  • Python Models: Fully aligned with schema ✅
  • OpenCode Agents: 4 specialized extractors ready
  • Conversations: 139 JSON files ready for extraction
  • Tests: 207 passing (100%), 91% coverage
  • Dutch ISIL: 364 institutions parsed
  • Dutch Orgs: 1,351 institutions parsed
  • GeoNames DB: 4.9M cities indexed

Architecture Notes

PROV-O Integration (Ready):

  • ChangeEvent maps to prov:Activity
  • Links via prov:wasInfluencedBy from HeritageCustodian
  • Uses prov:atTime for event timestamps
  • Tracks prov:entity (affected) and prov:generated (resulting) orgs

GHCID Impact Tracking:

  • When institutions merge/relocate/rename, GHCID changes
  • Old GHCID tracked in ghcid_history with valid_to timestamp
  • New GHCIDHistoryEntry created with valid_from timestamp
  • Change events linked via temporal correlation

Agent-Based Extraction Benefits:

  • No spaCy/transformer dependencies in main codebase
  • Flexible, maintainable (prompts vs. code)
  • Multilingual by default (60+ languages)
  • Read-only subagents (safe, predictable)

Next Session: Test agents on real data and validate extraction pipeline