Commit graph

79 commits

Author SHA1 Message Date
kempersc
7c7d8c0270 feat: Add SchemaElementPopup component for displaying LinkML schema element previews
- Implemented a draggable, resizable, and minimizable popup component for displaying previews of LinkML schema elements (classes, slots, enums).
- Integrated loading states and error handling for fetching element information.
- Added navigation functionality to go to full element view.
- Enhanced user experience with type badges and detailed descriptions for each element type.

chore: Migrate AudioEventSegment, BayNumber, BoxNumber, and BudgetStatus classes to new YAML schema format

- Created new YAML definitions for AudioEventSegment, BayNumber, BoxNumber, and BudgetStatus classes with detailed descriptions and attributes.
- Migrated from deprecated slots to new class structures as part of Rule 53.
- Updated imports and prefixes for consistency across schemas.

chore: Archive deprecated slots for audio_event_segments, bay_number, and box_number

- Archived previous slot definitions for audio_event_segments, bay_number, and box_number to maintain historical records.
- Updated slot descriptions and ensured proper URI mappings for future reference.
2026-01-14 15:13:06 +01:00
kempersc
b30711fcfb update slots 2026-01-14 09:05:54 +01:00
kempersc
fcf36f9a11 fix: prevent ontology popup flash by using useLayoutEffect for centering
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m56s
DSPy RAG Evaluation / Layer 1 - Unit Tests (push) Successful in 11m5s
DSPy RAG Evaluation / Layer 2 - DSPy Module Tests (push) Successful in 12m50s
DSPy RAG Evaluation / Layer 3 - Integration Tests (push) Successful in 10m51s
DSPy RAG Evaluation / Layer 4 - Comprehensive Evaluation (push) Successful in 11m59s
DSPy RAG Evaluation / Quality Gate (push) Successful in 1s
2026-01-13 20:38:21 +01:00
kempersc
92b490d690 edit slots 2026-01-13 20:35:11 +01:00
kempersc
2907c0372a feat: add Getty AAT support and resolve PREMIS/BIBFRAME URIs to human-readable LOC docs
- Add Getty AAT (Art & Architecture Thesaurus) vocabulary support
  - fetchGettyAATEntity() fetches term info from vocab.getty.edu JSON-LD API
  - Extracts English labels, scope notes, and aliases
  - Shows 'concept' term type for SKOS concepts

- Add getHumanReadableUrl() to map RDF URIs to documentation pages
  - PREMIS 3.0: http://www.loc.gov/premis/rdf/v3/X → id.loc.gov HTML docs
  - BIBFRAME: http://id.loc.gov/ontologies/bibframe/X → id.loc.gov HTML docs
  - Uses c_ prefix for classes, p_ for properties

- Add Getty vocabulary prefixes (aat:, tgn:, ulan:)
- Add ontology badge colors for PREMIS 3, LOCN, Getty AAT
2026-01-13 19:14:31 +01:00
kempersc
8a3c907f59 fix: resolve CIDOC-CRM relative URIs, add EDM ontology, render HTML in descriptions
Some checks failed
Deploy Frontend / build-and-deploy (push) Has been cancelled
DSPy RAG Evaluation / Layer 1 - Unit Tests (push) Successful in 11m12s
DSPy RAG Evaluation / Layer 2 - DSPy Module Tests (push) Successful in 12m55s
DSPy RAG Evaluation / Layer 3 - Integration Tests (push) Successful in 10m24s
DSPy RAG Evaluation / Layer 4 - Comprehensive Evaluation (push) Successful in 11m31s
DSPy RAG Evaluation / Quality Gate (push) Successful in 2s
- Fix resolveUri() to handle bare local names like 'E27_Site' used by CIDOC-CRM
  (previously only handled URIs starting with '#')
- Add EDM (Europeana Data Model) ontology to frontend
  - Copy edm.owl to frontend/public/ontology/
  - Register in ONTOLOGY_FILES array
  - Add 'edm' prefix to STANDARD_PREFIXES
  - Add EDM color to ONTOLOGY_COLORS
- Render HTML content in ontology descriptions safely using DOMPurify
  - Sanitize HTML to allow only safe tags (a, br, em, strong, etc.)
  - Fix Schema.org relative links to absolute URLs
  - Add target='_blank' to external links
2026-01-13 16:50:40 +01:00
kempersc
21ed120ac2 fix: correct hallucinated RiC-O terms and add locn ontology
RiC-O hallucinated terms fixed:
- FindingAidType.yaml: rico:FindingAidType → rico:DocumentaryFormType
- has_acquisition_method.yaml: rico:hasOrHadActivityType → prov:wasGeneratedBy
- has_activity_type.yaml: rico:hasOrHadActivityType → dcterms:type
- has_arrangement.yaml: rico:hasOrHadArrangement → dcterms:description
- has_or_had_finding_aid.yaml: rico:isDescribedBy → rico:isOrWasDescribedBy

The following terms do NOT exist in RiC-O 1.1:
- rico:FindingAidType (use rico:DocumentaryFormType)
- rico:hasOrHadActivityType (no equivalent)
- rico:hasOrHadArrangement (no equivalent)
- rico:isDescribedBy (correct form: rico:isOrWasDescribedBy)

Added LOCN ontology support:
- Copied locn.ttl to frontend/public/ontology/
- Added LOCN to ONTOLOGY_FILES in ontology-loader.ts
- Added locn prefix to OntologyTermPopup.tsx
- LOCN (http://www.w3.org/ns/locn#) is W3C Location Core Vocabulary
  for addresses and geometry (used by locn:Address)
2026-01-13 16:42:32 +01:00
kempersc
f2b10fca19 fix: correct hallucinated PREMIS terms and Schema.org namespace mismatch
All checks were successful
Deploy Frontend / build-and-deploy (push) Successful in 3m48s
PREMIS ontology fixes (8 schema files):
- Replace invalid premis:hasRepresentation with dcterms:hasFormat
- Replace invalid premis:hasAccessRestriction with odrl:hasPolicy
- Replace invalid premis:hasPreservationPolicy with dcterms:conformsTo
- Replace invalid premis:hasAccessPolicy with dcterms:accessRights
- Replace invalid premis:hasStoragePolicy with dcterms:conformsTo
- Replace invalid premis:ProcessingStatus with skos:Concept
- Add proper close_mappings to valid PREMIS classes (premis:Representation, etc.)
- Document hallucinated terms in Rule 51 (AGENTS.md) for future prevention

Schema.org namespace fixes (3 frontend files):
- Update OntologyTermPopup.tsx: add normalizeSchemaOrgUri() function
- Update ontology-loader.ts: change schema prefix to https://schema.org/
- Update linkml-schema-service.ts: change schema prefix to https://schema.org/
- The schemaorg.owl file uses https:// but code was using http://

These changes ensure ontology term lookups work correctly for Schema.org
terms and that LinkML schema files only reference valid ontology predicates.
2026-01-13 14:16:33 +01:00
kempersc
c5fb9ec88e feat: add route for Entity Review page with lazy loading 2026-01-13 01:49:43 +01:00
kempersc
28c3aaf33f enrich profiles 2026-01-10 17:31:02 +01:00
kempersc
c88fd3af70 Refactor code structure for improved readability and maintainability 2026-01-09 11:05:26 +01:00
kempersc
6608a207d4 update frontend 2026-01-08 15:56:28 +01:00
kempersc
0b0ea75070 feat(rag): add factual query fast path - skip LLM for count/list queries
- Add ontology cache warming at startup in lifespan() function
- Add is_factual_query() detection in template_sparql.py (12 templates)
- Add factual_result and sparql_query fields to DSPyQueryResponse
- Skip LLM generation for factual templates (count, list, compare)
- Execute SPARQL directly and return results as table (~15s → ~2s latency)
- Update ConversationPanel.tsx to render factual results table
- Add CSS styling for factual results with green theme

For queries like 'hoeveel archieven zijn er in Den Haag', the SPARQL
results ARE the answer - no need for expensive LLM prose generation.
2026-01-08 13:34:23 +01:00
kempersc
98c42bf272 Fix LinkML URI conflicts and generate RDF outputs
- Fix scope_note → finding_aid_scope_note in FindingAid.yaml
- Remove duplicate wikidata_entity slot from CustodianType.yaml (import instead)
- Remove duplicate rico_record_set_type from class_metadata_slots.yaml
- Fix range types for equals_string compatibility (uriorcurie → string)
- Move class names from close_mappings to see_also in 10 RecordSetTypes files
- Generate all RDF formats: OWL, N-Triples, RDF/XML, N3, JSON-LD context
- Sync schemas to frontend/public/schemas/

Files: 1,151 changed (includes prior CustodianType migration)
2026-01-07 12:32:59 +01:00
kempersc
11983014bb Enhance specificity scoring system integration with existing infrastructure
- Updated documentation to clarify integration points with existing components in the RAG pipeline and DSPy framework.
- Added detailed mapping of SPARQL templates to context templates for improved specificity filtering.
- Implemented wrapper patterns around existing classifiers to extend functionality without duplication.
- Introduced new tests for the SpecificityAwareClassifier and SPARQLToContextMapper to ensure proper integration and functionality.
- Enhanced the CustodianRDFConverter to include ISO country and subregion codes from GHCID for better geospatial data handling.
2026-01-05 17:37:49 +01:00
kempersc
41d8905661 Fix Turtle parser multi-line string handling for PiCo ontology
- Fixed bug where closing triple-quotes (""") would incorrectly re-trigger
  multi-line string detection, causing subsequent class definitions to be skipped
- Added lineToProcess variable to track which portion of line to process after
  closing a multi-line string, preventing re-detection of opening quotes
- Moved UML large diagram confirmation logic from OntologyViewerPage to
  UMLVisualization component for better encapsulation
- PiCo ontology now correctly shows all 8 classes instead of 2

Deployed and verified on https://bronhouder.nl/ontology?ontology=PiCo
2026-01-05 11:25:43 +01:00
kempersc
242bc8bb35 Add new slots for heritage custodian entities
- Created deliverables_slot for expected or achieved deliverable outputs.
- Introduced event_id_slot for persistent unique event identifiers.
- Added follow_up_date_slot for scheduled follow-up action dates.
- Implemented object_ref_slot for references to heritage objects.
- Established price_slot for price information across entities.
- Added price_currency_slot for currency codes in price information.
- Created protocol_slot for API protocol specifications.
- Introduced provenance_text_slot for full provenance entry text.
- Added record_type_slot for classification of record types.
- Implemented response_formats_slot for supported API response formats.
- Established status_slot for current status of entities or activities.
- Added FactualCountDisplay component for displaying count query results.
- Introduced ReplyTypeIndicator component for visualizing reply types.
- Created approval_date_slot for formal approval dates.
- Added authentication_required_slot for API authentication status.
- Implemented capacity_items_slot for maximum storage capacity.
- Established conservation_lab_slot for conservation laboratory information.
- Added cost_usd_slot for API operation costs in USD.
2026-01-05 00:49:05 +01:00
kempersc
eb61f45de2 compact UML controls toolbar to fit single line when sidebar collapsed 2026-01-04 17:21:53 +01:00
kempersc
2dca28d8c1 enrich CH entries with mission statements 2026-01-04 13:12:32 +01:00
kempersc
4f0cafe98a enrich HC profiles 2026-01-02 02:11:04 +01:00
kempersc
349f31ae6f enrich custodian profiles 2026-01-02 02:10:18 +01:00
kempersc
ca219340f2 enrich entries 2025-12-26 14:30:31 +01:00
kempersc
5e8a432ef0 enrich japanese and dutch custodians 2025-12-23 18:08:45 +01:00
kempersc
0c1d19e98b enrich entries 2025-12-23 13:27:35 +01:00
kempersc
aca68ea47f remove a,bihguous web-claims 2025-12-21 00:01:54 +01:00
kempersc
23b1d8ee5f clean up GHCID 2025-12-17 11:58:40 +01:00
kempersc
99430c2a70 add new entries and semantic routing 2025-12-17 10:11:56 +01:00
kempersc
e0dd847491 extend ontology 2025-12-16 20:27:39 +01:00
kempersc
b0416efc7d enrich custodians and persons 2025-12-16 11:57:34 +01:00
kempersc
52ae711c56 add timespans 2025-12-16 09:02:52 +01:00
kempersc
b1340e30c8 add timespan 2025-12-15 22:35:35 +01:00
kempersc
cb56aa7e40 enrich all custodian timespan 2025-12-15 22:31:41 +01:00
kempersc
82aa655522 feat(conversation): Add resizable embedding projector panel with improved UX
- Larger default size (700x550) for better readability
- Resizable from all 8 edges/corners with visual SE grip indicator
- Clearer button icons (18px, strokeWidth 2.5)
- Draggable, minimizable, pinnable panel
- Dark theme and mobile responsive support
2025-12-15 17:45:27 +01:00
kempersc
31bbce13e6 fix(types): Make genealogiewerkbalk nested fields optional
Fixes TypeScript error where parseGenealogiewerkbalk returns optional
fields but Institution interface expected required fields.
2025-12-15 09:04:09 +01:00
kempersc
0a38225b36 feat(frontend): Add multi-select filters, URL params, and UI improvements
- Institution Browser: multi-select for types and countries
- URL query param sync for shareable filter URLs
- New utility: countryNames.ts with flag emoji support
- New utility: imageProxy.ts for image URL handling
- New component: SearchableMultiSelect dropdown
- Career timeline CSS and component updates
- Media gallery improvements
- Lazy load error boundary component
- Version check utility
2025-12-15 01:47:11 +01:00
kempersc
1d26cade66 correct person labels 2025-12-14 17:58:55 +01:00
kempersc
c6aee998db correct person labels 2025-12-14 17:29:39 +01:00
kempersc
c50c35fd3a enrich person custodian 2025-12-14 17:09:55 +01:00
kempersc
41aace785f feat: Add SyncPanel component for database synchronization
- Add SyncPanel component with bilingual (NL/EN) support
- Add relative URL handling for production (bronhouder.nl)
- Integrate SyncPanel into Database page
- Show sync status for all 4 databases (DuckLake, PostgreSQL, Oxigraph, Qdrant)
- Support dry-run mode and file limit options
2025-12-12 23:42:22 +01:00
kempersc
505c12601a Add test script for PiCo extraction from Arabic waqf documents
- Implemented a new script `test_pico_arabic_waqf.py` to test the GLM annotator's ability to extract person observations from Arabic historical documents.
- The script includes environment variable handling for API token, structured prompts for the GLM API, and validation of extraction results.
- Added comprehensive logging for API responses, extraction results, and validation errors.
- Included a sample Arabic waqf text for testing purposes, following the PiCo ontology pattern.
2025-12-12 17:50:17 +01:00
kempersc
b1f93b6f22 enrich person profiles 2025-12-12 12:51:10 +01:00
kempersc
03263f67d6 moved web archives 2025-12-12 00:40:26 +01:00
kempersc
1b1cfbfca0 enrich custodians 2025-12-11 22:32:09 +01:00
kempersc
d4906abae4 update postgis data 2025-12-10 23:51:51 +01:00
kempersc
be3fbac601 enrich entries and persons 2025-12-10 18:04:25 +01:00
kempersc
41959f0766 correct HCID! 2025-12-10 13:01:13 +01:00
kempersc
3a6ead8fde feat: Add legal form filtering rule for CustodianName
- Introduced LEGAL-FORM-FILTER rule to standardize CustodianName by removing legal form designations.
- Documented rationale, examples, and implementation guidelines for the filtering process.

docs: Create README for value standardization rules

- Established a comprehensive README outlining various value standardization rules applicable to Heritage Custodian classes.
- Categorized rules into Name Standardization, Geographic Standardization, Web Observation, and Schema Evolution.

feat: Implement transliteration standards for non-Latin scripts

- Added TRANSLIT-ISO rule to ensure GHCID abbreviations are generated from emic names using ISO standards for transliteration.
- Included detailed guidelines for various scripts and languages, along with implementation examples.

feat: Define XPath provenance rules for web observations

- Created XPATH-PROVENANCE rule mandating XPath pointers for claims extracted from web sources.
- Established a workflow for archiving websites and verifying claims against archived HTML.

chore: Update records lifecycle diagram

- Generated a new Mermaid diagram illustrating the records lifecycle for heritage custodians.
- Included phases for active records, inactive archives, and processed heritage collections with key relationships and classifications.
2025-12-09 16:58:41 +01:00
kempersc
a7321b1bb9 reconstruct location blocks 2025-12-09 12:25:16 +01:00
kempersc
62fdd35321 Refactor code structure for improved readability and maintainability 2025-12-09 11:15:51 +01:00
kempersc
131e3ca259 normalise custodian entries 2025-12-09 07:56:35 +01:00