Commit graph

54 commits

Author SHA1 Message Date
kempersc
9342919c79 Add archived slot definitions for various attributes
- Introduced dual_class_role, emic_name, employer_linkedin_url, employer_name, employment_dates_raw, employment_end_date, employment_start_date, end_date, end_seconds, end_time, ended_at_time, endowment_draw, engagement_rate, enriched_date, enrichment_metadata_whatsapp, enrichment_method_whatsapp, exhibition_timespan, has_timespan, policy_effective_from, policy_effective_to, start_date, can_or_could_be_retrieved_from, documents_or_documented, has_or_had_contributor, has_or_had_drawer, has_or_had_email, has_or_had_endowment_draw, has_or_had_engagement_metric, has_or_had_metadata, has_or_had_summary, is_or_was_employed_by, and is_or_was_expired_at slots.
- Each slot includes detailed descriptions, ranges, and mappings to ensure compliance with ontology standards.
2026-01-26 17:32:24 +01:00
kempersc
fba1ab9353 feat: Migrate multiple slots to structured classes and update processing notes 2026-01-26 01:41:04 +01:00
kempersc
776462de90 Migrate multiple slots to enhance semantic clarity and align with best practices
- Migrated catering_type to CateringType with subclasses for better classification.
- Updated certainty_level to has_or_had_level for improved metadata consistency.
- Addressed cessation_observed_in by confirming existing temporal data structure.
- Created NetAsset class and updated financial statements for richer financial modeling.
- Completed migrations for default_access_policy, default_audio_language, and default_language to structured classes.
- Migrated default_position to structured Alignment class for better representation.
- Updated defined_by_standard to broaden range for identifier standards.
- Migrated definition to structured Resolution class for video resolution modeling.
- Completed migrations for degree_name, deliverable, and departement_code to structured classes.
- Migrated deployment_date to structured DeploymentEvent with temporal extent.
- Migrated derived_from_entity and derived_from_observation to new reference structures.
- Completed description and description_text migrations to enhance content modeling.
- Migrated detection_count, detection_level, and detection_threshold to structured slots with classes.
- Migrated device-related slots to structured classes for better identification and classification.
- Added new slots and classes for historic building and web address modeling.
2026-01-25 12:47:38 +01:00
kempersc
511fc99847 feat: Add PriceRange, Publication, and TaxDeductibility classes
- Introduced PriceRange class to categorize price levels for hospitality services, including structured metadata for various price categories.
- Added Publication class to represent publication events, capturing details like publisher, publication place, and edition.
- Created TaxDeductibilityType as an abstract class for tax deductibility status, promoting previous enum values to a class hierarchy for richer metadata.
- Implemented TaxDeductibilityTypes with concrete subclasses detailing various tax deductibility statuses.
- Archived previous DeductibilityStatusEnum and related slots, transitioning to a more structured approach for tax deductibility classification.
- Updated multiple slot definitions to align with new class structures and naming conventions, including has_or_had_measurement and has_or_had_price.
- Enhanced documentation and examples across new and existing slots for clarity and compliance with naming conventions.
2026-01-24 17:41:06 +01:00
kempersc
2a75ddf7cc feat: Add ConflictType and ConflictTypes schemas for heritage conflict taxonomy
- Introduced abstract class ConflictType to define a taxonomy for various conflict types affecting heritage institutions.
- Added concrete subclasses in ConflictTypes.yaml, detailing specific conflict types such as ArmedConflict, NaturalDisaster, CivilUnrest, Terrorism, Looting, Neglect, Occupation, and Sanctions.
- Implemented Permission and PermissionType schemas to represent authorization requirements for accessing heritage materials, including subclasses like BishopsPermission and InstitutionalAffiliation.
- Created SocialNetworkMember class for representing members in social/professional networks, facilitating heritage sector network analysis.
- Established slots for canonical access rules, conflict status, and connection metadata, enhancing the data model for heritage custodians.
- Developed ConnectionDegree and ConnectionDegreeType classes to represent degrees of connection in social networks, with subclasses for first, second, and third-plus degrees.
- Added slots for birth dates in EDTF and ISO formats, improving the representation of heritage custodian entities.
2026-01-22 20:41:06 +01:00
kempersc
be18d6761c feat: Update manifest generated timestamp and mark slot fixes as completed with detailed response 2026-01-22 20:17:33 +01:00
kempersc
2d09776856 Refactor StorageCondition schema: Migrate compliance_status to has_or_had_status with ComplianceStatus class
- Removed compliance_status slot and replaced it with has_or_had_status.
- Updated has_or_had_status to use ComplianceStatus for structured representation.
- Adjusted examples to reflect new structure for compliance status.
- Updated documentation to indicate migration and provide details on the ComplianceStatus class.
2026-01-22 16:22:16 +01:00
kempersc
c2629f6d29 Fix LinkML schema validation errors (0 errors, 30 warnings)
Schema Migration Fixes:
- Fix YAML import indentation in ~650 slot files (linkml:types and enum imports)
- Rename slot reference: has_or_had_holds_record_set_type → hold_or_held_record_set_type
  (70+ archive class files, main schema, manifest.json)
- Fix ProvenanceBlock.yaml: remove invalid any_of range, use string with multivalued
- Fix has_or_had_provenance.yaml: remove nested template_specificity from annotations

Validation Status:
- 0 errors (was multiple import/reference errors)
- 30 warnings (missing descriptions on inline slots, intentional SCREAMING_CASE names)

Files changed: ~3,850 (slots, classes, main schema, manifest)
2026-01-15 23:21:38 +01:00
kempersc
a981bb7ca3 feat(linkml): Add slot_usage comparison popup in schema viewer
- Add 'slot_usage' type to SchemaElementPopup for comparing generic slots vs class overrides
- Show side-by-side comparison table with property, generic value, and override value
- Display green 'changed' badges for modified properties
- Add dual navigation buttons (Go to class / Go to slot)
- Include comprehensive dark mode support
- Match styling to main page's comparison view (green color scheme)
2026-01-14 19:55:57 +01:00
kempersc
21c207c9da Refactor schema slots and classes for improved clarity and structure
- Migrated `archived_at` to `is_or_was_archived_at` in AuxiliaryDigitalPlatform, WebObservation, and other relevant classes to better reflect historical archival status.
- Removed `bold_id` slot and replaced it with `has_or_had_identifier` linked to the new `BOLDIdentifier` class in BiologicalObject.
- Introduced `Bookplate` and `Approver` classes to enhance provenance tracking and ownership documentation.
- Updated `InformationCarrier` to replace `bookplate` with `includes_or_included` for better representation of ownership marks.
- Added new slots `is_or_was_approved_by` and `is_or_was_archived_at` to capture historical approval and archival locations.
- Archived old slot definitions for `archived_at` and `bold_id` to maintain schema integrity.
- Enhanced LinkedIn profile extraction functionality by integrating Linkup API alongside Exa API.
2026-01-14 13:28:33 +01:00
kempersc
60e66d60f9 Add new slots and classes for enhanced documentation and availability tracking
- Introduced `is_or_was_created_through` slot to indicate content creation methods, replacing previous boolean flags.
- Added `is_or_was_required` slot for generic temporal boolean requirements, aligning with Schema.org.
- Created `AutoGeneration` class to represent automatic content generation, capturing methods and provenance.
- Established `AvailabilityStatus` class to model resource availability with temporal validity.
- Developed `Documentation` class for structured documentation resources, replacing domain-specific slots.
- Implemented `Taxon` class for biological classification in natural history collections.
- Archived previous slots related to API availability and documentation, ensuring a clean schema.
- Enhanced existing slots with detailed descriptions and examples for clarity and usability.
2026-01-14 13:09:31 +01:00
kempersc
92b490d690 edit slots 2026-01-13 20:35:11 +01:00
kempersc
8a3c907f59 fix: resolve CIDOC-CRM relative URIs, add EDM ontology, render HTML in descriptions
Some checks failed
Deploy Frontend / build-and-deploy (push) Has been cancelled
DSPy RAG Evaluation / Layer 1 - Unit Tests (push) Successful in 11m12s
DSPy RAG Evaluation / Layer 2 - DSPy Module Tests (push) Successful in 12m55s
DSPy RAG Evaluation / Layer 3 - Integration Tests (push) Successful in 10m24s
DSPy RAG Evaluation / Layer 4 - Comprehensive Evaluation (push) Successful in 11m31s
DSPy RAG Evaluation / Quality Gate (push) Successful in 2s
- Fix resolveUri() to handle bare local names like 'E27_Site' used by CIDOC-CRM
  (previously only handled URIs starting with '#')
- Add EDM (Europeana Data Model) ontology to frontend
  - Copy edm.owl to frontend/public/ontology/
  - Register in ONTOLOGY_FILES array
  - Add 'edm' prefix to STANDARD_PREFIXES
  - Add EDM color to ONTOLOGY_COLORS
- Render HTML content in ontology descriptions safely using DOMPurify
  - Sanitize HTML to allow only safe tags (a, br, em, strong, etc.)
  - Fix Schema.org relative links to absolute URLs
  - Add target='_blank' to external links
2026-01-13 16:50:40 +01:00
kempersc
21ed120ac2 fix: correct hallucinated RiC-O terms and add locn ontology
RiC-O hallucinated terms fixed:
- FindingAidType.yaml: rico:FindingAidType → rico:DocumentaryFormType
- has_acquisition_method.yaml: rico:hasOrHadActivityType → prov:wasGeneratedBy
- has_activity_type.yaml: rico:hasOrHadActivityType → dcterms:type
- has_arrangement.yaml: rico:hasOrHadArrangement → dcterms:description
- has_or_had_finding_aid.yaml: rico:isDescribedBy → rico:isOrWasDescribedBy

The following terms do NOT exist in RiC-O 1.1:
- rico:FindingAidType (use rico:DocumentaryFormType)
- rico:hasOrHadActivityType (no equivalent)
- rico:hasOrHadArrangement (no equivalent)
- rico:isDescribedBy (correct form: rico:isOrWasDescribedBy)

Added LOCN ontology support:
- Copied locn.ttl to frontend/public/ontology/
- Added LOCN to ONTOLOGY_FILES in ontology-loader.ts
- Added locn prefix to OntologyTermPopup.tsx
- LOCN (http://www.w3.org/ns/locn#) is W3C Location Core Vocabulary
  for addresses and geometry (used by locn:Address)
2026-01-13 16:42:32 +01:00
kempersc
6781073d06 fix: add @base directive support for Turtle/RDF parsing
The VCard ontology file (and 3 others) use @base directive with relative URIs
like <#Address>. The Turtle parser was not extracting @base or resolving
relative URIs against it.

Changes:
- Extract @base directive in first pass alongside @prefix
- Add baseUri parameter to expandUri() function
- Handle relative URIs starting with # (resolve against base)
- Handle empty relative URI <> (returns base URI itself)
- Pass baseUri through to processSubject() function

This fixes the 'Term not found' error for vcard:Address and similar terms
that use relative URI notation in their ontology definitions.

Affected ontologies: vcard.rdf, prov.ttl, era_ontology.ttl, ebg-ontology.ttl
2026-01-13 15:54:29 +01:00
kempersc
f2b10fca19 fix: correct hallucinated PREMIS terms and Schema.org namespace mismatch
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m48s
PREMIS ontology fixes (8 schema files):
- Replace invalid premis:hasRepresentation with dcterms:hasFormat
- Replace invalid premis:hasAccessRestriction with odrl:hasPolicy
- Replace invalid premis:hasPreservationPolicy with dcterms:conformsTo
- Replace invalid premis:hasAccessPolicy with dcterms:accessRights
- Replace invalid premis:hasStoragePolicy with dcterms:conformsTo
- Replace invalid premis:ProcessingStatus with skos:Concept
- Add proper close_mappings to valid PREMIS classes (premis:Representation, etc.)
- Document hallucinated terms in Rule 51 (AGENTS.md) for future prevention

Schema.org namespace fixes (3 frontend files):
- Update OntologyTermPopup.tsx: add normalizeSchemaOrgUri() function
- Update ontology-loader.ts: change schema prefix to https://schema.org/
- Update linkml-schema-service.ts: change schema prefix to https://schema.org/
- The schemaorg.owl file uses https:// but code was using http://

These changes ensure ontology term lookups work correctly for Schema.org
terms and that LinkML schema files only reference valid ontology predicates.
2026-01-13 14:16:33 +01:00
kempersc
1fb924c412 feat: add ontology mappings to LinkML schema and enhance entity resolution
Schema enhancements (443 files):
- Add class_uri with proper ontology references (schema:, prov:, skos:, rico:)
- Add close_mappings, related_mappings per Rule 50 convention
- Replace stub hc: slot_uri with standard predicates (dcterms:identifier, skos:prefLabel)
- Improve descriptions with ontology mapping rationale
- Add prefixes blocks to all schema modules

Entity Resolution improvements:
- Add entity_resolution module with email semantics parsing
- Enhance build_entity_resolution.py with email-based matching signals
- Extend Entity Review API with filtering by signal types and count
- Add candidates caching and indexing for performance
- Add ReviewLoginPage component

New rules and documentation:
- Add Rule 51: No Hallucinated Ontology References
- Add .opencode/rules/no-hallucinated-ontology-references.md
- Add .opencode/rules/slot-ontology-mapping-reference.md
- Add adms.ttl and dqv.ttl ontology files

Frontend ontology support:
- Add RiC-O_1-1.rdf and schemaorg.owl to public/ontology
2026-01-13 13:51:02 +01:00
kempersc
0f7fbf1ca0 feat(ci): add Forgejo Actions workflow for auto-deploy on LinkML schema changes
Some checks are pending
Deploy Frontend / build-and-deploy (push) Waiting to run
Infrastructure changes to enable automatic frontend deployment when schemas change:

- Add .forgejo/workflows/deploy-frontend.yml workflow triggered by:
  - Changes to frontend/** or schemas/20251121/linkml/**
  - Manual workflow dispatch

- Rewrite generate-schema-manifest.cjs to properly scan all schema directories
  - Recursively scans classes, enums, slots, modules directories
  - Uses singular category names (class, enum, slot) matching TypeScript types
  - Includes all 4 main schemas at root level
  - Skips archive directories and backup files

- Update schema-loader.ts to match new manifest format
  - Add SchemaCategory interface
  - Update SchemaManifest to use categories as array
  - Add flattenCategories() helper function
  - Add getSchemaCategories() and getSchemaCategoriesSync() functions

The workflow builds frontend with updated manifest and deploys to bronhouder.nl
2026-01-11 14:16:57 +01:00
kempersc
9726cc7917 feat(frontend): Add AnnotationMotivationType to LinkML schema manifest
Add new AnnotationMotivationType and AnnotationMotivationTypes to the
SCHEMA_FILES array so they appear in the /linkml viewer.
2026-01-11 13:56:11 +01:00
kempersc
28c3aaf33f enrich profiles 2026-01-10 17:31:02 +01:00
kempersc
81da4ede50 Add comprehensive slot visualization to LinkML viewer
- Add standalone Slots section in visual view alongside Classes and Enums
- Display slot_uri, range, identifier badge, description, pattern
- Show examples with value/description pairs
- Color-coded SKOS mapping tags (exact/close/narrow/broad/related)
- Yellow highlighted comments section
- Custodian type filtering works with slots
- Shared renderSlotDetails() function for consistency
2026-01-07 22:03:08 +01:00
kempersc
98c42bf272 Fix LinkML URI conflicts and generate RDF outputs
- Fix scope_note → finding_aid_scope_note in FindingAid.yaml
- Remove duplicate wikidata_entity slot from CustodianType.yaml (import instead)
- Remove duplicate rico_record_set_type from class_metadata_slots.yaml
- Fix range types for equals_string compatibility (uriorcurie → string)
- Move class names from close_mappings to see_also in 10 RecordSetTypes files
- Generate all RDF formats: OWL, N-Triples, RDF/XML, N3, JSON-LD context
- Sync schemas to frontend/public/schemas/

Files: 1,151 changed (includes prior CustodianType migration)
2026-01-07 12:32:59 +01:00
kempersc
11983014bb Enhance specificity scoring system integration with existing infrastructure
- Updated documentation to clarify integration points with existing components in the RAG pipeline and DSPy framework.
- Added detailed mapping of SPARQL templates to context templates for improved specificity filtering.
- Implemented wrapper patterns around existing classifiers to extend functionality without duplication.
- Introduced new tests for the SpecificityAwareClassifier and SPARQLToContextMapper to ensure proper integration and functionality.
- Enhanced the CustodianRDFConverter to include ISO country and subregion codes from GHCID for better geospatial data handling.
2026-01-05 17:37:49 +01:00
kempersc
41d8905661 Fix Turtle parser multi-line string handling for PiCo ontology
- Fixed bug where closing triple-quotes (""") would incorrectly re-trigger
  multi-line string detection, causing subsequent class definitions to be skipped
- Added lineToProcess variable to track which portion of line to process after
  closing a multi-line string, preventing re-detection of opening quotes
- Moved UML large diagram confirmation logic from OntologyViewerPage to
  UMLVisualization component for better encapsulation
- PiCo ontology now correctly shows all 8 classes instead of 2

Deployed and verified on https://bronhouder.nl/ontology?ontology=PiCo
2026-01-05 11:25:43 +01:00
kempersc
242bc8bb35 Add new slots for heritage custodian entities
- Created deliverables_slot for expected or achieved deliverable outputs.
- Introduced event_id_slot for persistent unique event identifiers.
- Added follow_up_date_slot for scheduled follow-up action dates.
- Implemented object_ref_slot for references to heritage objects.
- Established price_slot for price information across entities.
- Added price_currency_slot for currency codes in price information.
- Created protocol_slot for API protocol specifications.
- Introduced provenance_text_slot for full provenance entry text.
- Added record_type_slot for classification of record types.
- Implemented response_formats_slot for supported API response formats.
- Established status_slot for current status of entities or activities.
- Added FactualCountDisplay component for displaying count query results.
- Introduced ReplyTypeIndicator component for visualizing reply types.
- Created approval_date_slot for formal approval dates.
- Added authentication_required_slot for API authentication status.
- Implemented capacity_items_slot for maximum storage capacity.
- Established conservation_lab_slot for conservation laboratory information.
- Added cost_usd_slot for API operation costs in USD.
2026-01-05 00:49:05 +01:00
kempersc
2dca28d8c1 enrich CH entries with mission statements 2026-01-04 13:12:32 +01:00
kempersc
4f0cafe98a enrich HC profiles 2026-01-02 02:11:04 +01:00
kempersc
349f31ae6f enrich custodian profiles 2026-01-02 02:10:18 +01:00
kempersc
45e873ec0a enrich JP BE AR profiles 2025-12-30 23:07:03 +01:00
kempersc
d64f857aa9 add sparql validator and RAG injector 2025-12-30 03:43:31 +01:00
kempersc
ca219340f2 enrich entries 2025-12-26 14:30:31 +01:00
kempersc
54869589d1 fix(linkml-viewer): 3D cube visualization bugs - prevent click-to-filter and parse JSON custodian_types
- Remove onFaceClick prop from CustodianTypeIndicator3D in class/slot/enum detail views
  to prevent accidental filtering when clicking decorative cubes (Bug 3)
- Add parseCustodianTypesAnnotation() helper to handle JSON-stringified arrays like '["A"]'
  in YAML annotations, fixing Bug 2 where all 19 letters appeared on every cube
- Legend bar retains onTypeClick for intentional filtering functionality
2025-12-23 20:40:32 +01:00
kempersc
0c1d19e98b enrich entries 2025-12-23 13:27:35 +01:00
kempersc
aca68ea47f remove a,bihguous web-claims 2025-12-21 00:01:54 +01:00
kempersc
23b1d8ee5f clean up GHCID 2025-12-17 11:58:40 +01:00
kempersc
e0dd847491 extend ontology 2025-12-16 20:27:39 +01:00
kempersc
b0416efc7d enrich custodians and persons 2025-12-16 11:57:34 +01:00
kempersc
52ae711c56 add timespans 2025-12-16 09:02:52 +01:00
kempersc
cb56aa7e40 enrich all custodian timespan 2025-12-15 22:31:41 +01:00
kempersc
0a38225b36 feat(frontend): Add multi-select filters, URL params, and UI improvements
- Institution Browser: multi-select for types and countries
- URL query param sync for shareable filter URLs
- New utility: countryNames.ts with flag emoji support
- New utility: imageProxy.ts for image URL handling
- New component: SearchableMultiSelect dropdown
- Career timeline CSS and component updates
- Media gallery improvements
- Lazy load error boundary component
- Version check utility
2025-12-15 01:47:11 +01:00
kempersc
22709cc13e feat(rag): Add per-message refresh, bypass cache toggle, and cache clear improvements
- Add refresh button to assistant messages for re-running queries with fresh results
- Highlight refresh button (amber) for cached responses to draw attention
- Add spinning icon animation while refreshing
- Fix cache clear to return detailed success/failure status for local vs shared cache
- Add bypass cache toggle that forces fresh queries (one-shot, resets after query)
- Add Dutch/English translations for new UI elements
2025-12-14 19:12:25 +01:00
kempersc
c50c35fd3a enrich person custodian 2025-12-14 17:09:55 +01:00
kempersc
505c12601a Add test script for PiCo extraction from Arabic waqf documents
- Implemented a new script `test_pico_arabic_waqf.py` to test the GLM annotator's ability to extract person observations from Arabic historical documents.
- The script includes environment variable handling for API token, structured prompts for the GLM API, and validation of extraction results.
- Added comprehensive logging for API responses, extraction results, and validation errors.
- Included a sample Arabic waqf text for testing purposes, following the PiCo ontology pattern.
2025-12-12 17:50:17 +01:00
kempersc
1b1cfbfca0 enrich custodians 2025-12-11 22:32:09 +01:00
kempersc
d4906abae4 update postgis data 2025-12-10 23:51:51 +01:00
kempersc
41959f0766 correct HCID! 2025-12-10 13:01:13 +01:00
kempersc
3a6ead8fde feat: Add legal form filtering rule for CustodianName
- Introduced LEGAL-FORM-FILTER rule to standardize CustodianName by removing legal form designations.
- Documented rationale, examples, and implementation guidelines for the filtering process.

docs: Create README for value standardization rules

- Established a comprehensive README outlining various value standardization rules applicable to Heritage Custodian classes.
- Categorized rules into Name Standardization, Geographic Standardization, Web Observation, and Schema Evolution.

feat: Implement transliteration standards for non-Latin scripts

- Added TRANSLIT-ISO rule to ensure GHCID abbreviations are generated from emic names using ISO standards for transliteration.
- Included detailed guidelines for various scripts and languages, along with implementation examples.

feat: Define XPath provenance rules for web observations

- Created XPATH-PROVENANCE rule mandating XPath pointers for claims extracted from web sources.
- Established a workflow for archiving websites and verifying claims against archived HTML.

chore: Update records lifecycle diagram

- Generated a new Mermaid diagram illustrating the records lifecycle for heritage custodians.
- Included phases for active records, inactive archives, and processed heritage collections with key relationships and classifications.
2025-12-09 16:58:41 +01:00
kempersc
3a242370fc annotation standards added 2025-12-05 15:30:23 +01:00
kempersc
d661947830 update enriched entries 2025-12-03 17:38:46 +01:00
kempersc
ef89b1213a validate enrichments 2025-12-02 14:36:01 +01:00