Commit graph

26 commits

Author SHA1 Message Date
kempersc
fba1ab9353 feat: Migrate multiple slots to structured classes and update processing notes 2026-01-26 01:41:04 +01:00
kempersc
776462de90 Migrate multiple slots to enhance semantic clarity and align with best practices
- Migrated catering_type to CateringType with subclasses for better classification.
- Updated certainty_level to has_or_had_level for improved metadata consistency.
- Addressed cessation_observed_in by confirming existing temporal data structure.
- Created NetAsset class and updated financial statements for richer financial modeling.
- Completed migrations for default_access_policy, default_audio_language, and default_language to structured classes.
- Migrated default_position to structured Alignment class for better representation.
- Updated defined_by_standard to broaden range for identifier standards.
- Migrated definition to structured Resolution class for video resolution modeling.
- Completed migrations for degree_name, deliverable, and departement_code to structured classes.
- Migrated deployment_date to structured DeploymentEvent with temporal extent.
- Migrated derived_from_entity and derived_from_observation to new reference structures.
- Completed description and description_text migrations to enhance content modeling.
- Migrated detection_count, detection_level, and detection_threshold to structured slots with classes.
- Migrated device-related slots to structured classes for better identification and classification.
- Added new slots and classes for historic building and web address modeling.
2026-01-25 12:47:38 +01:00
kempersc
4efaef60e4 feat: Migrate capacity_value and cut_count to structured has_or_had_quantity; archive previous slots and update related schemas 2026-01-22 22:16:35 +01:00
kempersc
3c9926956e feat: Update canonical value handling by migrating to structured CanonicalForm and archiving previous slot 2026-01-22 22:08:04 +01:00
kempersc
ba2c766dd0 Add new slots and update existing ones following RiC-O temporal naming conventions
- Introduced `founding_date`, `founding_date_diocese`, and `fr` slots for capturing founding dates and French language text.
- Created `collects_or_collected`, `has_or_had_objective`, `has_or_had_percentage`, `has_or_had_place`, `has_or_had_reply`, `has_or_had_web_page`, `is_or_was_acquired_by`, `is_or_was_appreciated`, `is_or_was_founded_through`, `is_or_was_part_of`, `is_or_was_part_of_total`, `start_of_the_start`, `takes_or_took_comission`, and `was_fetched_at` slots to enhance data modeling capabilities.
- Each slot includes detailed descriptions, examples, and ontology alignments to ensure clarity and usability.
- Migration notes added for slots transitioned from previous definitions to maintain historical context and facilitate understanding of changes.
2026-01-22 15:15:56 +01:00
kempersc
3da90b940e feat(schema): complete multiple slot_fixes.yaml migrations
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 2m4s
Session 2026-01-19: Completed remaining migrations per Rules 53/56/60.

Major migrations:
1. claim_type → has_or_had_type + ClaimType/ClaimTypes (60+ concrete types in 11 categories)
2. circumstances_of_death → is_deceased + DeceasedStatus + CauseOfDeath
3. claims_count → has_or_had_quantity + Quantity (with based_on_claim for provenance)
4. classification_status → has_or_had_type + ClassificationStatusType

Created files:
- ClaimType.yaml, ClaimTypes.yaml (abstract base + 60+ concrete subclasses)
- DeceasedStatus.yaml, CauseOfDeath.yaml, CauseOfDeathTypeEnum.yaml
- ClassificationStatus.yaml, ClassificationStatusType.yaml, ClassificationStatusTypes.yaml
- CITESAppendix.yaml, City.yaml, CertaintyLevel.yaml
- is_deceased.yaml, is_or_was_caused_by.yaml, based_on_claim.yaml

Archived slots:
- claim_type, circumstances_of_death, claims_count, classification_status

Added Rule 60 to AGENTS.md: No Migration Deferral - agents MUST execute all migrations.

All 527 slot_fixes.yaml entries now complete (100%).
2026-01-19 13:05:53 +01:00
kempersc
4a277d7d42 standardise slots 2026-01-19 00:09:28 +01:00
kempersc
4319f38c05 Add archived slots for audience size, audience type, and capacity metrics
- Created new YAML files for audience size and audience type slots, defining their properties and annotations.
- Added archived capacity slots including cubic meters, linear meters, item count, and descriptions, with appropriate URIs and ranges.
- Introduced a template specificity slot for context-aware RAG filtering.
- Consolidated capacity-related slots into a unified structure, including has_or_had_capacity, capacity_type, and capacity_value, with detailed descriptions and examples.
2026-01-17 18:53:23 +01:00
kempersc
f9f3cc8e74 fix: resolve YAML import indentation and add missing slot descriptions
Schema Improvements:
- Fix YAML import indentation across 800+ class files (sed: '^- ../' → '  - ../')
- Add descriptions to 26 inline slots missing them (lint warnings)
- Fix malformed imports in BirthPlace.yaml and CustodianObservation.yaml

Validation Results:
- linkml-lint: 4 warnings (intentional SCREAMING_CASE tier names)
- gen-owl: SUCCESS (164,069 lines generated)
- gen-json-schema: SUCCESS (9.4MB generated)

Files affected: 1,034 files, +23,908 -15,200 lines
2026-01-16 00:09:28 +01:00
kempersc
3fb27c15e2 Refactor and archive deprecated slots; update migration records
- Removed deprecated slots: storage_security_level, version_number, video_comment, visiting_hour, was_asserted_by, was_revision_of, writing_system.
- Archived corresponding YAML files for deprecated slots with detailed migration notes.
- Updated slot definitions for has_collection and encompassing_body to reflect new naming conventions and temporal patterns.
- Enhanced metadata extraction in index_persons_qdrant.py to include WCMS registration and data sources.
- Modified hybrid_retriever and multi_embedding_retriever to support filtering by WCMS registration status.
2026-01-15 13:16:59 +01:00
kempersc
58940582c3 Refactor warehouse and video slots: Migrate and archive multiple slots including warehouse_security_level, warehouse_name, and video_id. Introduce UnitIdentifier class to replace unit_id and unit_identifier slots. Update consuming classes and document migration details. Archive obsolete slot definitions for track_id, tracking_ids_assigned, vendor_name, and others. Ensure compliance with migration rules and maintain historical records in the archive. 2026-01-14 22:20:44 +01:00
kempersc
53c6dbc2d9 feat(schema): Migrate temporal slots and introduce new pattern classes
Major slot migrations following slot_fixes.yaml revisions:
- TimeSpan: begin_of_the_begin, begin_of_the_end, end_of_the_begin, end_of_the_end
- Quantity: has_or_had_measurement_unit with MeasureUnit class
- Description: has_or_had_description with Description class
- URL, WikiData, Timestamp, Location, Provenance pattern classes

New slots for RiC-O compliance:
- Temporal: has_or_had_time_interval, calendar_system
- Transfer: is_or_was_transferred, has_or_had_policy
- Location: starts/ends_or_started/ended_at_location
- Provenance: has_or_had_provenance_path, is_or_was_webarchived_at

Archive deprecated slots per Rule 53 workflow.
2026-01-14 20:01:55 +01:00
kempersc
bdf3ceafb8 feat: Migrate and standardize measurement units; introduce Area and MeasureUnit classes 2026-01-14 17:04:33 +01:00
kempersc
355d8be51d centralise slots 2026-01-12 14:33:56 +01:00
kempersc
0d5d48568d refactor(schema): centralize slot definitions per Rule 38
- Remove slot_uri, description, mappings from slot_usage sections
- Move these properties to centralized slot files in modules/slots/
- Keep only class-specific overrides in slot_usage (required, inlined, examples)
- Update 1,499 centralized slot files with enriched definitions
- Clean 188 class files

Violations fixed:
- slot_uri in slot_usage: 1,676 → 0
- description in slot_usage: 2,287 → 0 (moved to centralized)

Schema still validates: 816 classes, 2028 slots, 127 enums
2026-01-11 23:27:17 +01:00
kempersc
5d3d8530b0 chore: trigger DSPy eval workflow
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 4m13s
2026-01-11 22:40:23 +01:00
kempersc
56c373bba8 Implement fast WCMS migration script with state file checkpointing and batch processing 2026-01-11 22:26:37 +01:00
kempersc
174a420c08 refactor(schema): centralize 1515 inline slot definitions per Rule 48
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m57s
- Remove inline slot definitions from 144 class files
- Create 7 new centralized slot files in modules/slots/:
  - custodian_type_broader.yaml
  - custodian_type_narrower.yaml
  - custodian_type_related.yaml
  - definition.yaml
  - finding_aid_access_restriction.yaml
  - finding_aid_description.yaml
  - finding_aid_temporal_coverage.yaml
- Add centralize_inline_slots.py automation script
- Update manifest with new timestamp

Rule 48: Class files must NOT define inline slots - all slots
must be imported from modules/slots/ directory.

Note: Pre-existing IdentifierFormat duplicate class definition
(in Standard.yaml and IdentifierFormat.yaml) not addressed in
this commit - requires separate schema refactor.
2026-01-11 22:02:14 +01:00
kempersc
626bd3a095 refactor(schemas): apply naming conventions to 261 class files
- Apply Rule 39: RiC-O style hasOrHad*/isOrWas* for temporal slots
- Apply Rule 43: Singular noun convention (keywords → keyword)
- Update slot references to match renamed slot files
- Maintain schema integrity across all class definitions
2026-01-10 15:36:33 +01:00
kempersc
095a3f949c refactor(linkml): apply RiC-O slot naming conventions to /schemas/ (Rule 39)
Apply same RiC-O-style slot naming refactor to /schemas/20251121/linkml/
that was previously applied to frontend/public/schemas/:

- Add 'has_' prefix for possession predicates
- Add 'is_or_was_' prefix for temporal inverse relationships
- Add 'has_or_had_' for bidirectional temporal relations
- Add new slots: is_or_was_aggregated_by, is_or_was_allocated_by, etc.
- Update count slots with proper descriptions

This ensures consistency between the source schema directory and the
frontend-served schemas.

514 files changed, +6,325 insertions, -4,255 deletions
2026-01-10 12:55:45 +01:00
kempersc
0393b321c9 refactor(schema): unify custodian_type slots into has_or_had_custodian_type (Rule 39, 43)
- Migrate 236+ class files from custodian_types to has_or_had_custodian_type
- Archive deprecated slots: custodian_type, custodian_types, custodian_type_broader/narrower/related
- Update main schema and manifest imports
- Fix Custodian.yaml class to use new slot
- Fix annotation format (list→scalar) in has_or_had_custodian_type.yaml

Rules applied:
- Rule 39: RiC-O naming convention (hasOrHad pattern)
- Rule 43: Slot nouns must be singular (multivalued:true for cardinality)
- Rule 38: Slot centralization with semantic URI
2026-01-09 10:55:21 +01:00
kempersc
b34992b1d3 Migrate all 293 class files to ontology-aligned slots
Extends migration to all class types (museums, libraries, galleries, etc.)

New slots added to class_metadata_slots.yaml:
- RiC-O: rico_record_set_type, rico_organizational_principle,
  rico_has_or_had_holder, rico_note
- Multilingual: label_de, label_es, label_fr, label_nl, label_it, label_pt
- Scope: scope_includes, scope_excludes, custodian_only,
  organizational_level, geographic_restriction
- Notes: privacy_note, preservation_note, legal_note

Migration script now handles 30+ annotation types.
All migrated schemas pass linkml-validate.

Total: 387 class files now use proper slots instead of annotations.
2026-01-06 12:24:54 +01:00
kempersc
11983014bb Enhance specificity scoring system integration with existing infrastructure
- Updated documentation to clarify integration points with existing components in the RAG pipeline and DSPy framework.
- Added detailed mapping of SPARQL templates to context templates for improved specificity filtering.
- Implemented wrapper patterns around existing classifiers to extend functionality without duplication.
- Introduced new tests for the SpecificityAwareClassifier and SPARQLToContextMapper to ensure proper integration and functionality.
- Enhanced the CustodianRDFConverter to include ISO country and subregion codes from GHCID for better geospatial data handling.
2026-01-05 17:37:49 +01:00
kempersc
0c1d19e98b enrich entries 2025-12-23 13:27:35 +01:00
kempersc
99430c2a70 add new entries and semantic routing 2025-12-17 10:11:56 +01:00
kempersc
51554947a0 feat(schema): Add video content schema with comprehensive examples
Video Schema Classes (9 files):
- VideoPost, VideoComment: Social media video modeling
- VideoTextContent: Base class for text content extraction
- VideoTranscript, VideoSubtitle: Text with timing and formatting
- VideoTimeSegment: Time code handling with ISO 8601 duration
- VideoAnnotation: Base annotation with W3C Web Annotation alignment
- VideoAnnotationTypes: Scene, Object, OCR detection annotations
- VideoChapter, VideoChapterList: Navigation and chapter structure
- VideoAudioAnnotation: Speaker diarization, music, sound events

Enumerations (12 enums):
- VideoDefinitionEnum, LiveBroadcastStatusEnum
- TranscriptFormatEnum, SubtitleFormatEnum, SubtitlePositionEnum
- AnnotationTypeEnum, AnnotationMotivationEnum
- DetectionLevelEnum, SceneTypeEnum, TransitionTypeEnum, TextTypeEnum
- ChapterSourceEnum, AudioEventTypeEnum, SoundEventTypeEnum, MusicTypeEnum

Examples (904 lines, 10 comprehensive heritage-themed examples):
- Rijksmuseum virtual tour chapters (5 chapters with heritage entity refs)
- Operation Night Watch documentary chapters (5 chapters)
- VideoAudioAnnotation: curator interview, exhibition promo, museum lecture

All examples reference real heritage entities with Wikidata IDs:
Q5598 (Rembrandt), Q41264 (Vermeer), Q219831 (The Night Watch)
2025-12-16 20:03:17 +01:00