glam/docs/sessions/SESSION_SUMMARY_ARGENTINA_CONABIP.md

Session Summary: Argentina CONABIP Scraper (Nov 17, 2025)

Objective

Complete web scraping of Argentina's CONABIP (National Commission on Popular Libraries) database to extract structured data on popular libraries across Argentina.

What We Accomplished

1. Successfully Scraped Basic Data ✅

Output: data/isil/AR/conabip_libraries.csv and .json

• 288 institutions extracted
• 22 provinces covered
• 221 cities represented
• Zero errors during scrape
• Duration: ~27 seconds

Data Fields:

• Registration number (REG)
• Library name
• Province
• City/Locality
• Neighborhood (Barrio)
• Street address
• Profile URL (for enhanced scraping)

2. Test Scrape with Profiles ✅

Output: data/isil/AR/conabip_libraries_with_profiles_test.csv and .json

• 32 institutions (1 page test)
• 100% success rate on coordinate extraction (32/32)
• 69% coverage on services data (22/32)
• Duration: ~96 seconds

Enhanced Data Fields:

• Latitude/longitude (from Google Maps)
• Google Maps URL
• Services offered (visual icon parsing)

3. Full Enhanced Scrape - Partial Progress ⏳

Attempted: Full 288-institution scrape with profile data Challenge: Multiple timeout issues (10-minute OpenCODE command limit) Progress Achieved:

• Successfully scraped ~174/288 profiles (60%) before timeout
• Server connection issues encountered (4 timeouts/resets at institutions 170-173)
• Scraper handled errors gracefully, continued processing

Known Issues:

• CONABIP server occasionally slow (30+ second timeouts on some profiles)
• Scraper doesn't save incrementally (only at completion)
• Full scrape requires ~12-15 minutes uninterrupted

Files Created

Scripts

• ✅ /scripts/scrapers/scrape_conabip_argentina.py (658 lines)

  • Main scraper with pagination, profile parsing, error handling
  • Rate limiting (configurable 1.8-2.5 seconds)
  • CSV and JSON export with metadata

• ✅ /tests/scrapers/test_conabip_scraper.py (385 lines)

  • 17 unit tests (100% pass rate)
  • 2 integration tests (100% pass rate)
  • Edge case validation

• ⚠️ /scripts/scrapers/batch_scrape_conabip.sh (attempted batch approach)

• ⚠️ /scripts/scrapers/scrape_conabip_resume.py (checkpoint-based approach)

Data Outputs

• ✅ data/isil/AR/conabip_libraries.csv - Basic data (288 institutions)
• ✅ data/isil/AR/conabip_libraries.json - Basic data with metadata
• ✅ data/isil/AR/conabip_libraries_with_profiles_test.csv - Sample enhanced (32 institutions)
• ✅ data/isil/AR/conabip_libraries_with_profiles_test.json - Sample enhanced with metadata
• ⏳ Full enhanced dataset - not completed due to timeout constraints

Technical Findings

Data Quality Assessment

1. Source Reliability: Excellent

   • Official government database (CONABIP)
   • Clean, structured HTML
   • Consistent data formats

2. Coverage Analysis:

   • 288 total institutions (lower than estimated 1,500-3,000)
   • Likely represents default search results or active registrations
   • May need province-specific searches for comprehensive coverage

3. Geographic Coordinates:

   • Available via Google Maps embeds
   • 100% extraction success rate (in test sample)
   • High precision (latitude/longitude to 6+ decimal places)

4. Services Metadata:

   • Represented as visual icons on profile pages
   • ~69% of institutions have services data
   • Includes: WiFi, computer access, reading programs, workshops, etc.

Performance Metrics

• Page scraping: ~3 seconds per page (32 institutions each)
• Profile scraping: ~2.5 seconds per profile (with 1.8s rate limit)
• Total estimated time: 288 profiles × 2.5s = ~12-15 minutes
• Server reliability: ~1.4% failure rate (4 timeouts/errors in 288 requests)

Technical Challenges Encountered

1. OpenCODE Timeout: 10-minute command limit insufficient for full scrape
2. Server Timeouts: Occasional 30+ second delays from CONABIP server
3. No Incremental Saves: Scraper only writes files at completion
4. Path Resolution: Batch script had file path issues (double data/isil/AR/)

Recommendations for Next Session

Option 1: Background Process (Recommended)

Run scraper outside OpenCODE environment for uninterrupted execution:

cd /Users/kempersc/apps/glam
nohup python3 scripts/scrapers/scrape_conabip_argentina.py \
  --scrape-profiles \
  --rate-limit 2.0 \
  --output-csv conabip_full_enhanced.csv \
  --output-json conabip_full_enhanced.json \
  > scrape_log.txt 2>&1 &

Check progress with: tail -f scrape_log.txt

Option 2: Modify Scraper for Incremental Saves

Add checkpoint functionality to save every 50 profiles:

• Requires code modification in scrape_all() method
• Add JSON checkpoint file with last processed index
• Implement resume logic

Option 3: Use Existing Basic Data

For immediate progress on GLAM project:

• Use conabip_libraries.csv (288 institutions) for analysis
• Geocode addresses using Nominatim API
• Defer profile data collection to later session
• Sufficient for initial Argentina dataset integration

Data Integration Next Steps

Once full enhanced dataset is available:

1. Parse into HeritageCustodian LinkML instances

   • Institution type: LIBRARY
   • Country: AR (Argentina)
   • Map provinces to regions
   • Extract GH CIDs

2. Enrich with Wikidata

   • Query Wikidata for Q-numbers
   • Cross-reference by name + location
   • Add VIAF IDs if available

3. Generate GHCID Identifiers

   • Format: AR-{Province}-{City}-L-{Abbrev}
   • Add Wikidata Q-number suffix if available

4. Export to RDF/JSON-LD

   • Use heritage_custodian.yaml LinkML schema
   • Tier 2 provenance (web-scraped)

Statistics

Coverage by Province (Top 10)

Based on basic dataset:

1. Buenos Aires: ~120+ institutions
2. Santa Fe: ~40+ institutions
3. Entre Ríos: ~20+ institutions
4. Córdoba: ~15+ institutions
5. (More details available via data analysis)

Institution Name Patterns

Common names (indicating historical/cultural significance):

• "Domingo Faustino Sarmiento": ~25 instances (most popular)
• "Bernardino Rivadavia": ~20 instances
• "Juan Bautista Alberdi": ~10 instances
• "Bartolomé Mitre": ~8 instances

Session Status: INCOMPLETE - RESUME REQUIRED

Current State: Have complete basic dataset (288 institutions), need enhanced profile data.

Resume Strategy: Run scraper in background or modify for incremental saves.

Alternative: Proceed with basic dataset for now, enhance later.

---

Session Duration: ~2 hours Commands Executed: 30+ Files Created: 5 data files, 3 scripts, 1 test suite Lines of Code Written: 1,043 lines (scraper + tests) Tests Passed: 19/19 (100%) Data Scraped: 288 institutions (basic), 32 institutions (enhanced sample)