kempersc
24cddb82dc
enrich ppid profiles
2026-01-16 12:50:50 +01:00
kempersc
7424b85352
Add new slots for heritage custodian entities
...
- Introduced setpoint_max, setpoint_min, setpoint_tolerance, setpoint_type, setpoint_unit, setpoint_value, temperature_target, track_id, typical_http_methods, typical_metadata_standard, typical_response_formats, typical_scope, typical_technical_feature, unit_code, unit_symbol, unit_type, wikidata_entity, wikidata_equivalent, and wikidata_id slots.
- Each slot includes a unique identifier, name, title, description, and annotations for custodian types and specificity score.
2026-01-16 01:04:38 +01:00
kempersc
49f4054802
data(person/entity): add 83,845 LinkedIn profile extractions from company pages
...
Bulk extraction of heritage professional profiles from LinkedIn company pages
using extract_persons_with_provenance.py script.
Key characteristics:
- Source: LinkedIn company 'People' pages for heritage institutions
- File format: {linkedin-slug}_{timestamp}.json
- Total size: ~3.6GB
- Includes: profile_data, heritage_relevance, affiliations, web_claims
- Provenance: Full XPath + archived HTML references (Rule 6 compliant)
- Dual timestamps: statement_created_at + source_archived_at (Rule 35)
Extraction metadata includes:
- extraction_agent: extract_persons_with_provenance.py
- source_file: Original archived HTML filename
- source_archived_at: When LinkedIn page was captured
- schema_version: 1.0.0
Note: URL-encoded filenames preserve international characters (Arabic,
Hebrew, Chinese, Turkish, accented Latin, etc.)
2026-01-10 13:27:08 +01:00
kempersc
932ec5438c
add person profiles with PPID
2026-01-09 18:26:58 +01:00
kempersc
d64f857aa9
add sparql validator and RAG injector
2025-12-30 03:43:31 +01:00
kempersc
84904e344b
Make AGENTS more succint by referring to opencode rules & enrich custodians
2025-12-28 14:56:35 +01:00
kempersc
4cf3fe8a07
Logo enrichment batch: JP+170 (5,166/12,096 = 42.7%) - 14,503 total (45.6%)
2025-12-27 13:17:40 +01:00
kempersc
fd91fec63f
Logo enrichment batch: JP+320, 13,603 total (42.8%)
...
- JP: 4,516/12,096 (37.4%) ✅ NEW COMMIT
- CZ: 3,820/8,432 (45.3%) - batches 7-16 running
- CH, NL, BE, AT, BR: 100% complete
- Total: 13,603/31,772 (42.8%)
- Using crawl4ai favicon extraction
2025-12-26 23:25:40 +01:00
kempersc
2104a90f22
Logo enrichment COMPLETE: CZ 3,820 (45.3%)
...
- CZ: 3,820/8,432 files processed (45.3%)
- 9 parallel batches completed (500 files each)
- NL person entities added (4 staff profiles)
- scripts/discover_websites_crawl4ai.py modified
- Using crawl4ai favicon extraction
2025-12-26 21:45:14 +01:00
kempersc
6af5009444
enrich entries
2025-12-26 21:41:18 +01:00
kempersc
59963c8d3f
Logo enrichment batch: JP+300, CZ-0 - 12,833 files (40.4%)
...
- JP: 4,496 processed (37.2% of 12,096) ✅ COMPLETE
- CZ: 2,820 processed (33.4% of 8,432) - batch completed, slight decrease
- CH, NL, BE, AT, BR: 100% complete
- Total: 12,833 of 31,772 files (40.4%)
- Using crawl4ai favicon extraction
2025-12-26 13:42:21 +01:00
kempersc
0c1d19e98b
enrich entries
2025-12-23 13:27:35 +01:00
kempersc
7a056fa746
enrich entries
2025-12-21 22:12:34 +01:00
kempersc
99430c2a70
add new entries and semantic routing
2025-12-17 10:11:56 +01:00
kempersc
e0dd847491
extend ontology
2025-12-16 20:27:39 +01:00
kempersc
b0416efc7d
enrich custodians and persons
2025-12-16 11:57:34 +01:00
kempersc
52ae711c56
add timespans
2025-12-16 09:02:52 +01:00
kempersc
b1340e30c8
add timespan
2025-12-15 22:35:35 +01:00
kempersc
cb56aa7e40
enrich all custodian timespan
2025-12-15 22:31:41 +01:00
kempersc
525662ea16
data: fix remaining person entity profiles
2025-12-15 01:48:33 +01:00
kempersc
3820f2fc92
chore: Add data reports, infra scripts, and API updates
...
- Data quality reports for Dutch custodians
- Name mismatch detection reports
- Failed crawl URL tracking
- Caddy configuration updates
- Monitor script for chunk 404 errors
- API endpoint improvements
2025-12-15 01:48:08 +01:00
kempersc
70c30a52d4
data: update person entity profiles with heritage classification
2025-12-15 01:47:42 +01:00
kempersc
181b1cf705
data: enrich Dutch heritage custodians (DR, FL, FR, GE, GR, LI provinces)
...
- Add digital platform discovery data with provenance
- Cleanup duplicate/incorrect custodian entries
- Add GHCID collision resolution suffixes where needed
- Update person entity profiles with career history
2025-12-15 01:34:38 +01:00
kempersc
1d26cade66
correct person labels
2025-12-14 17:58:55 +01:00
kempersc
c6aee998db
correct person labels
2025-12-14 17:29:39 +01:00
kempersc
c50c35fd3a
enrich person custodian
2025-12-14 17:09:55 +01:00
kempersc
505c12601a
Add test script for PiCo extraction from Arabic waqf documents
...
- Implemented a new script `test_pico_arabic_waqf.py` to test the GLM annotator's ability to extract person observations from Arabic historical documents.
- The script includes environment variable handling for API token, structured prompts for the GLM API, and validation of extraction results.
- Added comprehensive logging for API responses, extraction results, and validation errors.
- Included a sample Arabic waqf text for testing purposes, following the PiCo ontology pattern.
2025-12-12 17:50:17 +01:00
kempersc
b1f93b6f22
enrich person profiles
2025-12-12 12:51:10 +01:00
kempersc
1b1cfbfca0
enrich custodians
2025-12-11 22:32:09 +01:00
kempersc
d4906abae4
update postgis data
2025-12-10 23:51:51 +01:00
kempersc
be3fbac601
enrich entries and persons
2025-12-10 18:04:25 +01:00
kempersc
41959f0766
correct HCID!
2025-12-10 13:01:13 +01:00
kempersc
251b5eee68
geocode: add coordinates to 26 more custodian files
...
- Improved city name cleaning:
- Roman numeral district suffixes (Kolín V. -> Kolín)
- City + country suffixes (Genève 4 - Suisse -> Genève)
- Czech postal notation (p. Luka nad Jihlavou -> Luka nad Jihlavou)
- Historical city names (Gottwaldov -> Zlín, renamed 1990)
- Manual mappings for Swiss districts (Lugano Massagno -> Lugano)
2025-12-09 22:47:32 +01:00
kempersc
35e1686160
geocode: add coordinates to 69 custodian files across multiple countries
...
Countries updated: AR, AT, BG, BR, CA, CL, CN, CU, FI, GE, IR, JO, KG,
KR, LB, LI, LV, MX, MY, NI, NL, PS, PY, SX, TM, VN
- Manual city name mappings for transliteration variants
- St. Pölten -> Sankt Pölten (AT)
- Gaza City -> Gaza (PS)
- Beit Hanoun -> Bayt Hanun (PS)
- Veliko Tarnovo via geonames_id (BG)
2025-12-09 22:44:12 +01:00
kempersc
2137c522db
geocode: add coordinates to JP compound cities and CZ files from GeoNames
...
- JP: Handle Gun/Cho/Machi/Mura compound city names (2615 files)
- CZ: Map city codes to GeoNames entries (667 files)
- Overall coverage: 84.5% → 96.4%
2025-12-09 21:49:40 +01:00
