glam/docs/plan/global_glam/01-implementation-phases.md

# Global GLAM Dataset: Implementation Phases

## Overview
This document outlines the phased approach to building a comprehensive Global GLAM (Galleries, Libraries, Archives, Museums) dataset from 139 Claude conversation JSON files containing heritage institution research worldwide.

## Phase 1: Foundation & Schema Design (Weeks 1-2)

### 1.1 LinkML Schema Development
**Objective**: Create a comprehensive heritage custodian ontology combining multiple standards

**Tasks**:
- Design core LinkML schema integrating:
  - **TOOI** (Dutch government organizations ontology)
  - **Schema.org** (general web semantics)
  - **CPOC** (Core Public Organization Vocabulary)
  - **ISIL** (International Standard Identifier for Libraries)
  - Domain-specific vocabularies:
    - **RiC-O** (Records in Contexts - Archives)
    - **BIBFRAME** (Bibliographic Framework - Libraries)
    - **LIDO/CIDOC-CRM** (Museums)
    - **EBUCore** (Media archives)

**Deliverables**:
- `schemas/heritage_custodian.yaml` - Main LinkML schema
- `schemas/mixins/` - Modular schema components
- `docs/schema-documentation.md` - Schema documentation
- Example instances in YAML/JSON-LD

### 1.2 Poetry Application Scaffolding
**Objective**: Set up Python development environment

**Tasks**:
- Initialize Poetry project with proper dependencies
- Configure development tools (ruff, mypy, pytest)
- Set up project structure:
  ```
  glam-extractor/
  ├── pyproject.toml
  ├── src/glam_extractor/
  │   ├── parsers/      # JSON conversation parsers
  │   ├── extractors/   # NLP-based data extraction
  │   ├── crawlers/     # crawl4ai integration
  │   ├── validators/   # LinkML validation
  │   └── exporters/    # Output formats
  ├── tests/
  └── data/
      ├── input/        # Conversation JSONs
      └── output/       # Extracted datasets
  ```

**Deliverables**:
- Working Poetry project
- CI/CD configuration
- Development documentation

## Phase 2: Data Parsing & Initial Extraction (Weeks 3-4)

### 2.1 Conversation Parser Development
**Objective**: Parse Claude conversation JSON files and extract text content

**Tasks**:
- Implement JSON parser for conversation structure
- Extract markdown content from messages
- Handle citations and URL references
- Create conversation metadata index (country, date, topics)

**Deliverables**:
- `ConversationParser` class
- Conversation index database (SQLite/DuckDB)
- Parsed text corpus

### 2.2 Sample Data Analysis
**Objective**: Analyze representative conversations to refine extraction patterns

**Tasks**:
- Deep dive into 10-15 representative conversations covering:
  - Different countries/regions
  - Different GLAM types (galleries, libraries, archives, museums)
  - Various institution sizes (national, regional, local)
- Document common patterns and structures
- Identify edge cases

**Deliverables**:
- Pattern analysis document
- Extraction templates
- Edge case catalog

## Phase 3: NLP Extraction Pipeline (Weeks 5-7)

### 3.1 Entity Extraction
**Objective**: Extract heritage institutions and their attributes from conversation text

**Tasks**:
- Implement Named Entity Recognition (NER) for:
  - Institution names
  - Addresses/locations
  - URLs and digital platforms
  - Contact information
  - Collection types
- Create entity linking to disambiguate institutions
- Extract relationships (parent organizations, networks, partnerships)

**Technologies**:
- spaCy or similar NLP framework
- Custom entity recognition models
- Regex patterns for structured data (URLs, codes)

**Deliverables**:
- `EntityExtractor` class
- Custom NER models
- Entity disambiguation logic

### 3.2 Attribute Extraction
**Objective**: Extract structured attributes for each institution

**Tasks**:
- Extract:
  - Institution types (museum, archive, library, etc.)
  - Geographic coverage
  - Collection subjects/themes
  - Digital platforms and repositories
  - Standards used (metadata, preservation, access)
  - Identifiers (ISIL, Wikidata, national IDs)
  - Funding sources
  - Consortium memberships
- Map extracted attributes to LinkML schema

**Deliverables**:
- `AttributeExtractor` class
- Attribute mapping configuration
- Quality metrics

## Phase 4: Web Crawling & Enrichment (Weeks 8-10)

### 4.1 crawl4ai Integration
**Objective**: Fetch detailed information from URLs cited in conversations

**Tasks**:
- Implement crawl4ai wrapper for:
  - Institution homepages
  - Digital repository platforms
  - Catalog systems
  - API documentation
- Extract structured data from crawled pages:
  - OpenGraph metadata
  - Schema.org JSON-LD
  - Dublin Core metadata
  - Custom extraction rules per platform type
- Implement rate limiting and ethical crawling

**Deliverables**:
- `WebCrawler` class
- Platform-specific extractors
- Crawl results database

### 4.2 Data Validation & Enrichment
**Objective**: Validate URLs and enrich extracted data

**Tasks**:
- Verify URL availability (HTTP status checks)
- Cross-reference with external authorities:
  - Wikidata
  - VIAF (Virtual International Authority File)
  - National library registries
  - UNESCO heritage databases
- Resolve geographic coordinates
- Normalize institution names

**Deliverables**:
- `DataEnricher` class
- External API integrations
- Enrichment metrics

## Phase 5: LinkML Instance Generation (Weeks 11-12)

### 5.1 Schema Mapping
**Objective**: Map extracted data to LinkML schema instances

**Tasks**:
- Transform extracted entities to LinkML classes
- Validate instances against schema
- Handle missing/incomplete data
- Generate unique identifiers (IRIs)
- Maintain provenance (source conversations, extraction confidence)

**Deliverables**:
- `LinkMLMapper` class
- Instance generator
- Validation reports

### 5.2 Data Quality Assurance
**Objective**: Ensure high-quality dataset output

**Tasks**:
- Implement quality checks:
  - Completeness scoring
  - Duplicate detection
  - Inconsistency identification
- Manual review process for flagged items
- Create quality dashboard

**Deliverables**:
- `QualityChecker` class
- Review interface
- Quality metrics dashboard

## Phase 6: Export & Documentation (Weeks 13-14)

### 6.1 Multi-Format Export
**Objective**: Export dataset in multiple formats

**Tasks**:
- Generate outputs:
  - **RDF/Turtle** - Linked data format
  - **JSON-LD** - JSON with linked data context
  - **CSV/TSV** - Tabular formats for analysis
  - **YAML** - Human-readable instances
  - **SQLite/DuckDB** - Queryable database
  - **Parquet** - Efficient columnar format
- Create dataset statistics report

**Deliverables**:
- `DataExporter` class with format plugins
- Export configurations
- Dataset documentation

### 6.2 OpenCODE Agent Development
**Objective**: Create AI agents for extraction assistance

**Tasks**:
- Write `AGENTS.md` with:
  - NLP extraction subagent instructions
  - Pattern recognition guidelines
  - Quality review agent specifications
- Implement agent integration in pipeline
- Test agent-assisted extraction

**Deliverables**:
- `AGENTS.md` documentation
- Agent integration code
- Agent performance metrics

### 6.3 Documentation & Publication
**Objective**: Comprehensive project documentation

**Tasks**:
- Write user guides:
  - Installation and setup
  - Running extraction pipeline
  - Querying the dataset
  - Contributing new sources
- Create schema documentation site
- Prepare dataset for publication:
  - License selection (CC0, CC-BY, ODbL)
  - Metadata record
  - DOI registration
  - GitHub repository preparation

**Deliverables**:
- Complete documentation
- Published dataset
- Public repository

## Phase 7: Maintenance & Extension (Ongoing)

### 7.1 Continuous Improvement
**Tasks**:
- Monitor data quality metrics
- Refine extraction algorithms based on feedback
- Add support for new conversation sources
- Update schema as standards evolve
- Integrate community contributions

### 7.2 Dataset Updates
**Tasks**:
- Re-crawl URLs periodically for updates
- Process new conversation files
- Maintain dataset versioning
- Publish regular releases

## Success Metrics

### Quantitative
- **Coverage**: Extract ≥80% of institutions mentioned in conversations
- **Accuracy**: ≥90% precision on manual validation sample
- **Completeness**: ≥70% of extracted institutions have URLs and types
- **Enrichment**: ≥50% institutions linked to external authorities

### Qualitative
- Schema can represent diverse GLAM institutions globally
- Dataset is reusable by heritage sector practitioners
- Documentation enables community contributions
- Pipeline is maintainable and extensible

## Risk Mitigation

### Technical Risks
- **NLP extraction accuracy**: Mitigate with human-in-loop review and iterative refinement
- **Web crawling reliability**: Implement robust error handling and retry logic
- **Schema complexity**: Start simple, iterate based on real data needs

### Resource Risks
- **Processing time**: Parallelize where possible, use incremental processing
- **Storage requirements**: Use efficient formats (Parquet), compress archives

## Timeline Summary
- **Weeks 1-2**: Foundation (schemas, scaffolding)
- **Weeks 3-4**: Parsing & analysis
- **Weeks 5-7**: NLP extraction
- **Weeks 8-10**: Web crawling & enrichment
- **Weeks 11-12**: LinkML generation & QA
- **Weeks 13-14**: Export & documentation
- **Ongoing**: Maintenance & updates

**Total estimated duration**: 14 weeks for initial release, with ongoing maintenance