Commit graph

122 commits

Author SHA1 Message Date
kempersc
60e66d60f9 Add new slots and classes for enhanced documentation and availability tracking
- Introduced `is_or_was_created_through` slot to indicate content creation methods, replacing previous boolean flags.
- Added `is_or_was_required` slot for generic temporal boolean requirements, aligning with Schema.org.
- Created `AutoGeneration` class to represent automatic content generation, capturing methods and provenance.
- Established `AvailabilityStatus` class to model resource availability with temporal validity.
- Developed `Documentation` class for structured documentation resources, replacing domain-specific slots.
- Implemented `Taxon` class for biological classification in natural history collections.
- Archived previous slots related to API availability and documentation, ensuring a clean schema.
- Enhanced existing slots with detailed descriptions and examples for clarity and usability.
2026-01-14 13:09:31 +01:00
kempersc
b13674400f Refactor schema slots and classes for improved organization and clarity
- Removed deprecated slots: appraisal_notes, branch_id, is_or_was_real.
- Introduced new slots: has_or_had_notes, has_or_had_provenance.
- Created Notes class to encapsulate note-related metadata.
- Archived removed slots and classes in accordance with the new archive folder convention.
- Updated slot_fixes.yaml to reflect migration status and details.
- Enhanced documentation for new slots and classes, ensuring compliance with ontology alignment.
- Added new slots for note content, date, and type to support the Notes class.
2026-01-14 12:14:07 +01:00
kempersc
b8914761b8 standardise slots 2026-01-14 09:51:14 +01:00
kempersc
e3adb4ed60 feat: Introduce Overview, RealnessStatus, and WebLink classes with comprehensive documentation and migration notes
- Added Overview class to represent structured collections of web links, including detailed descriptions, examples, and ontology alignments.
- Introduced RealnessStatus class to classify data as real or synthetic, with rich provenance and temporal semantics.
- Created WebLink class for representing hyperlinks with associated metadata, enhancing structured link representation.
- Established new slots: has_or_had_comprehensive_overview, is_or_was_real, and includes_or_included to support the new classes and improve data modeling.
- Migrated existing slots to new structures, ensuring compliance with RiC-O naming conventions and enhancing specificity.
- Updated annotations and examples across all new classes and slots for clarity and usability.
2026-01-14 09:32:14 +01:00
kempersc
c807487a51 Refactor expense slots: remove program_expense slot, migrate administrative_expenses, and archive related slots
- Deleted the program_expense slot from the schema.
- Updated slot_fixes.yaml to reflect the migration of administrative_expenses, marking it as fully migrated and archiving related bespoke slots.
- Created archived YAML files for administrative_expenses, fundraising_expense, has_or_had_administrative_expense, innovation_expense, and program_expense, documenting their structure and descriptions.
- All expense types now utilize the Expenses class with ExpenseTypeEnum classification for better organization and clarity.
2026-01-14 09:15:17 +01:00
kempersc
1133749de8 fix: update manifest.json timestamp and consolidate expense slots in FinancialStatement.yaml 2026-01-14 09:07:11 +01:00
kempersc
554c5721ea fix: update generated timestamp in manifest.json and add has_or_had_expenses slot definition 2026-01-14 09:06:36 +01:00
kempersc
b30711fcfb update slots 2026-01-14 09:05:54 +01:00
kempersc
74ca873585 fix: update generated timestamp in manifest.json
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m58s
2026-01-13 21:54:50 +01:00
kempersc
6a3616beac feat(entity-resolution): expand Dutch heritage domain mappings
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
Add domain mappings for better email-based entity matching:
- Government: noord-holland.nl, amsterdam.nl, rotterdam.nl, denhaag.nl,
  hoorn.nl, hhnk.nl, rijksoverheid.nl, politie.nl, kadaster.nl, rvo.nl,
  rivm.nl, staatsbosbeheer.nl, vng.nl
- Museums: maritiemmuseum.nl, paleishetloo.nl, slotloevestein.nl
- Universities: student.vu.nl, cdh.leidenuniv.nl, jur.ru.nl, student.ru.nl,
  student.tudelft.nl, eshcc.eur.nl, wur.nl, ou.nl
- Hogescholen: hva.nl, student.hu.nl, student.fontys.nl

Also remove deprecated activity_id.yaml slot file
2026-01-13 20:53:49 +01:00
kempersc
408813280a refactor: simplify slot descriptions to be more concise
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
- accepts_or_accepted_external_work: Remove verbose examples list
- accepts_or_accepted_payment_method: Condense to single sentence
- accepts_or_accepted_visiting_scholar: Minor rewording for consistency
2026-01-13 20:52:05 +01:00
kempersc
92b490d690 edit slots 2026-01-13 20:35:11 +01:00
kempersc
21ed120ac2 fix: correct hallucinated RiC-O terms and add locn ontology
RiC-O hallucinated terms fixed:
- FindingAidType.yaml: rico:FindingAidType → rico:DocumentaryFormType
- has_acquisition_method.yaml: rico:hasOrHadActivityType → prov:wasGeneratedBy
- has_activity_type.yaml: rico:hasOrHadActivityType → dcterms:type
- has_arrangement.yaml: rico:hasOrHadArrangement → dcterms:description
- has_or_had_finding_aid.yaml: rico:isDescribedBy → rico:isOrWasDescribedBy

The following terms do NOT exist in RiC-O 1.1:
- rico:FindingAidType (use rico:DocumentaryFormType)
- rico:hasOrHadActivityType (no equivalent)
- rico:hasOrHadArrangement (no equivalent)
- rico:isDescribedBy (correct form: rico:isOrWasDescribedBy)

Added LOCN ontology support:
- Copied locn.ttl to frontend/public/ontology/
- Added LOCN to ONTOLOGY_FILES in ontology-loader.ts
- Added locn prefix to OntologyTermPopup.tsx
- LOCN (http://www.w3.org/ns/locn#) is W3C Location Core Vocabulary
  for addresses and geometry (used by locn:Address)
2026-01-13 16:42:32 +01:00
kempersc
73b3b21017 docs: add Rule 52 prohibiting duplicate ontology mappings
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m53s
- Create .opencode/rules/no-duplicate-ontology-mappings.md with detection script
- Add Rule 52 to AGENTS.md (after Rule 51)
- Fix 29 duplicate mappings: same URI in multiple mapping categories
  - 26 slot files: remove duplicates keeping most precise mapping
  - 3 class files: ExhibitionSpace, Custodian, DigitalPlatform
- Mapping precedence: exact > close > narrow/broad > related

Each ontology URI must appear in only ONE mapping category per schema
element, following SKOS semantics where mapping properties are mutually
exclusive.
2026-01-13 15:57:26 +01:00
kempersc
1fb924c412 feat: add ontology mappings to LinkML schema and enhance entity resolution
Schema enhancements (443 files):
- Add class_uri with proper ontology references (schema:, prov:, skos:, rico:)
- Add close_mappings, related_mappings per Rule 50 convention
- Replace stub hc: slot_uri with standard predicates (dcterms:identifier, skos:prefLabel)
- Improve descriptions with ontology mapping rationale
- Add prefixes blocks to all schema modules

Entity Resolution improvements:
- Add entity_resolution module with email semantics parsing
- Enhance build_entity_resolution.py with email-based matching signals
- Extend Entity Review API with filtering by signal types and count
- Add candidates caching and indexing for performance
- Add ReviewLoginPage component

New rules and documentation:
- Add Rule 51: No Hallucinated Ontology References
- Add .opencode/rules/no-hallucinated-ontology-references.md
- Add .opencode/rules/slot-ontology-mapping-reference.md
- Add adms.ttl and dqv.ttl ontology files

Frontend ontology support:
- Add RiC-O_1-1.rdf and schemaorg.owl to public/ontology
2026-01-13 13:51:02 +01:00
kempersc
c5fb9ec88e feat: add route for Entity Review page with lazy loading 2026-01-13 01:49:43 +01:00
kempersc
3b35f4aea5 Refactor code structure for improved readability and maintainability 2026-01-12 18:31:31 +01:00
kempersc
846a6cdcec Add new Record Set Types for various archival collections
- Introduced SoundArchiveRecordSetType, SpecialCollectionRecordSetType, SpecializedArchiveRecordSetType, SpecializedArchivesCzechiaRecordSetType, StateArchivesRecordSetType, StateArchivesSectionRecordSetType, StateDistrictArchiveRecordSetType, StateRegionalArchiveCzechiaRecordSetType, TelevisionArchiveRecordSetType, TradeUnionArchiveRecordSetType, UniversityArchiveRecordSetType, VereinsarchivRecordSetType, VerlagsarchivRecordSetType, VerwaltungsarchivRecordSetType, WebArchiveRecordSetType, and WomensArchivesRecordSetType.
- Each new type includes appropriate metadata, slots, and relationships to existing classes.
- Implemented a script to detect and fix Type class violations in LinkML files.
2026-01-12 15:20:29 +01:00
kempersc
355d8be51d centralise slots 2026-01-12 14:33:56 +01:00
kempersc
0d5d48568d refactor(schema): centralize slot definitions per Rule 38
- Remove slot_uri, description, mappings from slot_usage sections
- Move these properties to centralized slot files in modules/slots/
- Keep only class-specific overrides in slot_usage (required, inlined, examples)
- Update 1,499 centralized slot files with enriched definitions
- Clean 188 class files

Violations fixed:
- slot_uri in slot_usage: 1,676 → 0
- description in slot_usage: 2,287 → 0 (moved to centralized)

Schema still validates: 816 classes, 2028 slots, 127 enums
2026-01-11 23:27:17 +01:00
kempersc
5d3d8530b0 chore: trigger DSPy eval workflow
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 4m13s
2026-01-11 22:40:23 +01:00
kempersc
56c373bba8 Implement fast WCMS migration script with state file checkpointing and batch processing 2026-01-11 22:26:37 +01:00
kempersc
174a420c08 refactor(schema): centralize 1515 inline slot definitions per Rule 48
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m57s
- Remove inline slot definitions from 144 class files
- Create 7 new centralized slot files in modules/slots/:
  - custodian_type_broader.yaml
  - custodian_type_narrower.yaml
  - custodian_type_related.yaml
  - definition.yaml
  - finding_aid_access_restriction.yaml
  - finding_aid_description.yaml
  - finding_aid_temporal_coverage.yaml
- Add centralize_inline_slots.py automation script
- Update manifest with new timestamp

Rule 48: Class files must NOT define inline slots - all slots
must be imported from modules/slots/ directory.

Note: Pre-existing IdentifierFormat duplicate class definition
(in Standard.yaml and IdentifierFormat.yaml) not addressed in
this commit - requires separate schema refactor.
2026-01-11 22:02:14 +01:00
kempersc
10bb5b69c5 Add Environmental Zone Type Enumeration and related slots
- Introduced EnvironmentalZoneTypeEnum.yaml to classify climate-controlled storage zones with detailed descriptions and recommended conditions for various materials.
- Created slots for environmental zone type code, description, ID, label, and HC preset URI to facilitate structured data representation.
- Implemented boolean slots for specific environmental requirements including dark storage, dust-free environment, ESD protection, and UV filtering, referencing relevant ISO standards.
- Enhanced documentation for each slot to clarify usage and preservation context.
2026-01-11 21:14:59 +01:00
kempersc
66ab2908d0 fix: remove deprecated AnnotationMotivationEnum, add European surname data
Some checks failed
Deploy Frontend / build-and-deploy (push) Failing after 3m21s
- Move deprecated AnnotationMotivationEnum to archive-deprecated/ (outside served paths)
- Add French, Italian, Polish, Spanish surname datasets for entity resolution
- Update name_commonality.py with expanded European surname detection
- Triggers GitOps workflow to test Forgejo Actions runner
2026-01-11 16:03:18 +01:00
kempersc
055fd890ff test: verify pre-commit hook regenerates manifest
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
2026-01-11 15:21:58 +01:00
kempersc
3a661d6013 fix(schema): regenerate manifest to remove stale AnnotationMotivationEnum reference
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
The old enum was properly archived to modules/enums/archive/ with .deprecated
suffix per Rule 9, but the manifest wasn't regenerated. Now correctly shows
only AnnotationMotivationType.yaml and AnnotationMotivationTypes.yaml.
2026-01-11 15:16:50 +01:00
kempersc
b6e069a1d5 chore(schema): bump version to 0.9.12 - test webhook deployment
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
2026-01-11 15:08:35 +01:00
kempersc
be8b14f6ac refactor: Convert AnnotationMotivationEnum to Type/Types class hierarchy
- Create AnnotationMotivationType abstract base class (oa:Motivation)
- Create 10 concrete motivation subclasses in AnnotationMotivationTypes.yaml:
  - 6 W3C Web Annotation standard: classifying, describing, identifying,
    tagging, linking, commenting
  - 4 heritage-specific: accessibility, discovery, preservation, research
- Update has_annotation_motivation slot to use AnnotationMotivationType range
- Update VideoAnnotation.yaml imports and remove inline enum
- Archive deprecated AnnotationMotivationEnum.yaml
- Add motivation_type_id, motivation_type_name, motivation_type_description slots

Follows Rule 0b (Type/Types naming convention) and Rule 9 (enum-to-class promotion)
2026-01-11 13:48:28 +01:00
kempersc
3c3be47e32 feat(infra): add fast push-based schema sync to production
- Replace slow Forgejo→Server git pull with direct local rsync
- Add git-push-schemas.sh wrapper script for manual pushes
- Add post-commit hook for automatic schema sync
- Fix YAML syntax errors in slot comment blocks
- Update deploy-webhook.py to use master branch
2026-01-11 01:22:47 +01:00
kempersc
f02cffe1e8 refactor(schema): migrate 5 deprecated slots to temporal naming convention
Migrate slots to follow RiC-O-style temporal naming (Rule 39):
- accepts_external_work → accepts_or_accepted_external_work
- accepts_visiting_scholars → accepts_or_accepted_visiting_scholar
- accepts_payment_methods → accepts_or_accepted_payment_method
- access → has_or_had_access_condition
- access_policy_ref → has_or_had_access_policy_reference

Updated classes to use new slot names:
- ConservationLab.yaml
- ResearchCenter.yaml
- GiftShop.yaml
- ArchiveReference.yaml
- FindingAid.yaml
- Collection.yaml

Archived deprecated slots to schemas/20251121/linkml/archive/slots/
with _archived_20260110 suffix per Rule 9 (enum-to-class principle).
2026-01-10 21:09:29 +01:00
kempersc
6c19ef8661 feat(rag): add Rule 46 epistemic provenance tracking
Track full lineage of RAG responses: WHERE data comes from, WHEN it was
retrieved, HOW it was processed (SPARQL/vector/LLM).

Backend changes:
- Add provenance.py with EpistemicProvenance, DataTier, SourceAttribution
- Integrate provenance into MultiSourceRetriever.merge_results()
- Return epistemic_provenance in DSPyQueryResponse

Frontend changes:
- Pass EpistemicProvenance through useMultiDatabaseRAG hook
- Display provenance in ConversationPage (for cache transparency)

Schema fixes:
- Fix truncated example in has_observation.yaml slot definition

References:
- Pavlyshyn's Context Graphs and Data Traces paper
- LinkML ProvenanceBlock schema pattern
2026-01-10 18:42:43 +01:00
kempersc
28c3aaf33f enrich profiles 2026-01-10 17:31:02 +01:00
kempersc
626bd3a095 refactor(schemas): apply naming conventions to 261 class files
- Apply Rule 39: RiC-O style hasOrHad*/isOrWas* for temporal slots
- Apply Rule 43: Singular noun convention (keywords → keyword)
- Update slot references to match renamed slot files
- Maintain schema integrity across all class definitions
2026-01-10 15:36:33 +01:00
kempersc
94bfc9061e refactor(schemas): consolidate slot definitions and remove 305 redundant files
- Apply Rule 39: RiC-O style temporal naming (hasOrHad*, isOrWas*)
- Apply Rule 43: Singular noun convention for slot names
- Remove duplicate slot definitions consolidated into centralized files
- Net reduction: 6,162 lines across 305 deleted files
2026-01-10 15:36:13 +01:00
kempersc
e5a08a353d enrich person profiles 2026-01-10 14:14:04 +01:00
kempersc
2808dad6cd fix(linkml): correct invalid ontology property references in slot definitions
- confidence_score: prov:confidence doesn't exist → hc:confidenceScore
- deliverables: schema:result doesn't exist → hc:deliverables
- circumstances_of_death: wikidata:P1196 is identifier, not predicate → hc:circumstancesOfDeath
- deceased: schema:deathDate wrong semantics for boolean → hc:deceased
- death_place: fix sdo prefix to schema, remove wd:P20 as exact mapping
- date_of_death: wikidata:P570 is identifier, not predicate
- martyred: correct prefix inconsistencies
- given_name/literal_name: fix sdo→schema prefix
- occupation/religion/status: standardize prefix declarations

Add comments documenting why Wikidata properties (P-numbers) cannot be
used as slot_uri (they are entity identifiers, not RDF predicates).
2026-01-10 13:29:55 +01:00
kempersc
095a3f949c refactor(linkml): apply RiC-O slot naming conventions to /schemas/ (Rule 39)
Apply same RiC-O-style slot naming refactor to /schemas/20251121/linkml/
that was previously applied to frontend/public/schemas/:

- Add 'has_' prefix for possession predicates
- Add 'is_or_was_' prefix for temporal inverse relationships
- Add 'has_or_had_' for bidirectional temporal relations
- Add new slots: is_or_was_aggregated_by, is_or_was_allocated_by, etc.
- Update count slots with proper descriptions

This ensures consistency between the source schema directory and the
frontend-served schemas.

514 files changed, +6,325 insertions, -4,255 deletions
2026-01-10 12:55:45 +01:00
kempersc
519b0b47a8 Add Playwright test results JSON file with initial test suite and failure details 2026-01-09 21:33:31 +01:00
kempersc
0393b321c9 refactor(schema): unify custodian_type slots into has_or_had_custodian_type (Rule 39, 43)
- Migrate 236+ class files from custodian_types to has_or_had_custodian_type
- Archive deprecated slots: custodian_type, custodian_types, custodian_type_broader/narrower/related
- Update main schema and manifest imports
- Fix Custodian.yaml class to use new slot
- Fix annotation format (list→scalar) in has_or_had_custodian_type.yaml

Rules applied:
- Rule 39: RiC-O naming convention (hasOrHad pattern)
- Rule 43: Slot nouns must be singular (multivalued:true for cardinality)
- Rule 38: Slot centralization with semantic URI
2026-01-09 10:55:21 +01:00
kempersc
6608a207d4 update frontend 2026-01-08 15:56:28 +01:00
kempersc
53ffed3531 Add TemplateSpecificityScores import to SpecificityAnnotation 2026-01-07 22:05:42 +01:00
kempersc
20374b9032 Archive monolithic class_metadata_slots before modular refactoring 2026-01-07 22:05:21 +01:00
kempersc
9b769f1ca2 Update manifest timestamp and minor class fixes 2026-01-07 22:04:29 +01:00
kempersc
181991940f Add new LinkML schema modules for specificity and Wikidata alignment
New classes:
- SpecificityAnnotation: Track class relevance scores per template
- TemplateSpecificityScores: 10 conversation template scores
- WikidataAlignment: Link classes to Wikidata entities
- DualClassLink: Model dual-aspect class relationships

New enums:
- WikidataMappingTypeEnum: exact/close/narrow/broad/related mappings
- DualClassPatternEnum: place-custodian, collection-custodian patterns

New slots (44 files):
- Specificity slots: score, rationale, agent, timestamp
- Template scores: archive_search, museum_search, library_search, etc.
- Wikidata slots: entity_id, label, mapping_type, rationale
- Multilingual labels: label_nl, label_de, label_fr, label_es, etc.
- Custodian type annotations: custodian_types, rationale, primary
- SKOS hierarchy: skos_broader, skos_narrower, skos_related
2026-01-07 22:03:58 +01:00
kempersc
7f792d0250 Remove outdated RDF schema files
Clean up old generated RDF/OWL files that have been superseded by
the clean schema deployed to Oxigraph (commit f8421a2903).
Only the current deployed schema should be regenerated as needed.
2026-01-07 22:03:26 +01:00
kempersc
f8421a2903 Deploy clean RDF schema to Oxigraph - 288,857 triples with 0 malformed URIs
- Archive 48 old timestamped RDF files
- Fix relative IRIs by adding hc: namespace prefix
- Fix file path references in seeAlso predicates
- Deployed to sparql.glam-ontology.org triplestore
- 11,250 distinct OWL classes
- Schema includes all base ontologies (CIDOC-CRM, RiC-O, CPOV, etc.)
2026-01-07 16:53:01 +01:00
kempersc
dfa667c90f Fix LinkML schema for valid RDF generation with proper slot_uri
Summary:
- Create 46 missing slot definition files with proper slot_uri values
- Add slot imports to main schema (01_custodian_name_modular.yaml)
- Fix YAML examples sections in 116+ class and slot files
- Fix PersonObservation.yaml examples section (nested objects → string literals)

Technical changes:
- All slots now have explicit slot_uri mapping to base ontologies (RiC-O, Schema.org, SKOS)
- Eliminates malformed URIs like 'custodian/:slot_name' in generated RDF
- gen-owl now produces valid Turtle with 153,166 triples

New slot files (46):
- RiC-O slots: rico_note, rico_organizational_principle, rico_has_or_had_holder, etc.
- Scope slots: scope_includes, scope_excludes, archive_scope
- Organization slots: organization_type, governance_authority, area_served
- Platform slots: platform_type_category, portal_type_category
- Social media slots: social_media_platform_category, post_type_*
- Type hierarchy slots: broader_type, narrower_types, custodian_type_broader
- Wikidata slots: wikidata_equivalent, wikidata_mapping

Generated output:
- schemas/20251121/rdf/01_custodian_name_modular_20260107_134534_clean.owl.ttl (6.9MB)
- Validated with rdflib: 153,166 triples, no malformed URIs
2026-01-07 13:48:03 +01:00
kempersc
98c42bf272 Fix LinkML URI conflicts and generate RDF outputs
- Fix scope_note → finding_aid_scope_note in FindingAid.yaml
- Remove duplicate wikidata_entity slot from CustodianType.yaml (import instead)
- Remove duplicate rico_record_set_type from class_metadata_slots.yaml
- Fix range types for equals_string compatibility (uriorcurie → string)
- Move class names from close_mappings to see_also in 10 RecordSetTypes files
- Generate all RDF formats: OWL, N-Triples, RDF/XML, N3, JSON-LD context
- Sync schemas to frontend/public/schemas/

Files: 1,151 changed (includes prior CustodianType migration)
2026-01-07 12:32:59 +01:00
kempersc
6c6810fa43 Replace CustodianTypeCodeEnum with CustodianType class references
- Remove deprecated CustodianTypeCodeEnum from class_metadata_slots.yaml
- Update custodian_types slot to use uriorcurie range (references CustodianType subclasses)
- Update custodian_types_primary slot similarly
- Add migration note for legacy string format ['A'] vs new URI format

Per Rule 9: Enum-to-Class Promotion - Single Source of Truth
2026-01-06 12:37:40 +01:00