glam/docs/DEPLOYMENT_GUIDE.md

# GLAM Deployment Guide

Complete guide for deploying the GLAM (Galleries, Libraries, Archives, Museums) ontology project.

## Overview

This project uses a **local deployment model** - all deployments are executed from your local machine via SSH and rsync. There is **NO CI/CD pipeline** or GitHub Actions.

**Why Local Deployment?**
- Full control over what gets deployed
- No exposure of secrets to third-party CI systems
- Immediate feedback during deployment
- Easy debugging and rollback

## Infrastructure

### Server Details

| Property | Value |
|----------|-------|
| **Provider** | Hetzner Cloud |
| **Server Name** | `glam-sparql` |
| **IP Address** | `91.98.224.44` |
| **SSH User** | `root` |
| **Frontend Path** | `/var/www/glam-frontend/` |
| **Data Path** | `/mnt/data/` |
| **Oxigraph Data** | `/var/lib/oxigraph/` |

### Services Running on Server

| Service | Port | Description |
|---------|------|-------------|
| **Caddy** | 80, 443 | Reverse proxy with automatic HTTPS |
| **Oxigraph** | 7878 | SPARQL triplestore (internal only) |
| **Frontend** | - | Static files served by Caddy |

---

## Prerequisites

### 1. Environment Variables

Create a `.env` file in the project root:

```bash
# Required for deployment
HETZNER_HC_API_TOKEN=your_hetzner_cloud_api_token

# Optional - defaults shown
GLAM_DOMAIN=sparql.glam-ontology.org
ADMIN_EMAIL=admin@glam-ontology.org
```

**Getting your Hetzner API Token:**
1. Log in to [Hetzner Cloud Console](https://console.hetzner.cloud/)
2. Select your project
3. Go to Security > API Tokens
4. Create a new token with Read & Write permissions
5. Copy the token to your `.env` file

### 2. SSH Access

Ensure you can SSH to the server:

```bash
# Test SSH connection
ssh root@91.98.224.44 "echo 'Connection successful'"
```

If this fails:
- Check that your SSH key is added to the server's `~/.ssh/authorized_keys`
- Verify the server is running (`./infrastructure/deploy.sh --status`)

### 3. Local Dependencies

For frontend deployment:
```bash
# Ensure Node.js is installed (v18+)
node --version

# Ensure npm is installed
npm --version
```

For infrastructure changes:
```bash
# Ensure Terraform is installed
terraform --version

# Ensure jq is installed (for JSON parsing)
jq --version
```

---

## Deployment Script

All deployments use the script at `infrastructure/deploy.sh`.

### Command Options

| Option | Description |
|--------|-------------|
| `--frontend` | Build and deploy React frontend |
| `--data` | Deploy ontology/schema data files |
| `--infra` | Deploy infrastructure via Terraform |
| `--reload` | Reload data into Oxigraph triplestore |
| `--all` | Deploy everything |
| `--status` | Check server and service status |

### Quick Reference

```bash
# Most common - deploy frontend changes
./infrastructure/deploy.sh --frontend

# Deploy updated ontology/schema files
./infrastructure/deploy.sh --data

# Check what's running on the server
./infrastructure/deploy.sh --status

# Full deployment (rare)
./infrastructure/deploy.sh --all
```

---

## Deployment Scenarios

### Scenario 1: Frontend Changes (Most Common)

After making changes to `frontend/`:

```bash
# Option A: Use deploy script (recommended)
./infrastructure/deploy.sh --frontend

# Option B: Manual build + deploy
cd frontend
npm run build
cd ..
rsync -avz --delete frontend/dist/ root@91.98.224.44:/var/www/glam-frontend/
```

**What the script does:**
1. Runs `npm ci` to install exact dependencies
2. Runs `npm run build` to create production build
3. Syncs `frontend/dist/` to server via rsync

**Verification:**
```bash
# Check the server is serving updated content
curl -I https://91.98.224.44
```

### Scenario 2: Schema/Ontology Updates

After regenerating LinkML schemas or RDF files:

```bash
./infrastructure/deploy.sh --data
```

**What gets synced:**
- `data/ontology/*.{ttl,rdf,owl,jsonld}` → `/mnt/data/ontologies/`
- `schemas/20251121/rdf/` → `/mnt/data/rdf/`
- `schemas/20251121/linkml/*.yaml` → `/mnt/data/linkml/`
- `schemas/20251121/uml/mermaid/*.mmd` → `/mnt/data/uml/`

**To also reload into Oxigraph:**
```bash
./infrastructure/deploy.sh --data --reload
```

### Scenario 3: Infrastructure Changes

For changes to server configuration (rare):

```bash
./infrastructure/deploy.sh --infra
```

**Warning:** This will:
1. Run `terraform plan` to show changes
2. Ask for confirmation before applying
3. Potentially recreate the server (be careful!)

### Scenario 4: Check Server Status

```bash
./infrastructure/deploy.sh --status
```

**Output includes:**
- Server name, status, IP, type, location
- Oxigraph service status
- Caddy service status
- Triple count in SPARQL store

---

## Manual Deployment Steps

If the script fails or you need manual control:

### Frontend (Manual)

```bash
# 1. Build frontend
cd frontend
npm ci
npm run build

# 2. Sync schemas to frontend public directory (if needed)
mkdir -p public/schemas
cp -r ../schemas/20251121/linkml public/schemas/
cp -r ../schemas/20251121/uml public/schemas/

# 3. Deploy to server
rsync -avz --delete dist/ root@91.98.224.44:/var/www/glam-frontend/

# 4. Verify
ssh root@91.98.224.44 "ls -la /var/www/glam-frontend/"
```

### Data Files (Manual)

```bash
# Sync ontologies
rsync -avz data/ontology/*.ttl data/ontology/*.rdf data/ontology/*.owl \
  root@91.98.224.44:/mnt/data/ontologies/

# Sync RDF schemas
rsync -avz schemas/20251121/rdf/ \
  root@91.98.224.44:/mnt/data/rdf/

# Sync LinkML schemas
rsync -avz --include='*.yaml' --include='*/' --exclude='*' \
  schemas/20251121/linkml/ \
  root@91.98.224.44:/mnt/data/linkml/
```

### Reload Oxigraph (Manual)

```bash
ssh root@91.98.224.44 "/var/lib/glam/scripts/load-ontologies.sh"
```

---

## Server Management

### SSH Access

```bash
# Connect to server
ssh root@91.98.224.44

# Check disk usage
ssh root@91.98.224.44 "df -h"

# Check memory usage
ssh root@91.98.224.44 "free -h"

# View logs
ssh root@91.98.224.44 "journalctl -u caddy -n 50"
ssh root@91.98.224.44 "journalctl -u oxigraph -n 50"
```

### Service Management

```bash
# Restart Caddy (web server)
ssh root@91.98.224.44 "systemctl restart caddy"

# Restart Oxigraph (SPARQL store)
ssh root@91.98.224.44 "systemctl restart oxigraph"

# Check service status
ssh root@91.98.224.44 "systemctl status caddy"
ssh root@91.98.224.44 "systemctl status oxigraph"
```

### Caddy Configuration

Caddy config is at `/etc/caddy/Caddyfile` on the server:

```bash
# View current config
ssh root@91.98.224.44 "cat /etc/caddy/Caddyfile"

# Edit config (if needed)
ssh root@91.98.224.44 "nano /etc/caddy/Caddyfile"

# Reload after changes
ssh root@91.98.224.44 "systemctl reload caddy"
```

---

## Troubleshooting

### Common Issues

#### 1. "Permission denied" on SSH

```bash
# Check SSH key is loaded
ssh-add -l

# Add SSH key if not loaded
ssh-add ~/.ssh/id_ed25519  # or your key file
```

#### 2. "Server not found" error

```bash
# Check server exists in Hetzner
./infrastructure/deploy.sh --status

# If server doesn't exist, create with:
./infrastructure/deploy.sh --infra
```

#### 3. Frontend not updating

```bash
# Force cache clear on browser (Ctrl+Shift+R)

# Or check files were actually deployed
ssh root@91.98.224.44 "ls -la /var/www/glam-frontend/"
ssh root@91.98.224.44 "cat /var/www/glam-frontend/index.html | head -20"
```

#### 4. SPARQL queries failing

```bash
# Check Oxigraph is running
ssh root@91.98.224.44 "systemctl status oxigraph"

# Check triple count
ssh root@91.98.224.44 "curl -s -X POST \
  -H 'Content-Type: application/sparql-query' \
  -H 'Accept: application/sparql-results+json' \
  --data 'SELECT (COUNT(*) AS ?count) WHERE { ?s ?p ?o }' \
  http://localhost:7878/query | jq '.results.bindings[0].count.value'"

# Restart Oxigraph if needed
ssh root@91.98.224.44 "systemctl restart oxigraph"
```

#### 5. Hetzner API Token Invalid

```bash
# Test token manually
curl -s -H "Authorization: Bearer YOUR_TOKEN" \
  "https://api.hetzner.cloud/v1/servers" | jq '.servers[].name'
```

### Log Files

```bash
# Caddy access logs
ssh root@91.98.224.44 "journalctl -u caddy --since '1 hour ago'"

# Oxigraph logs
ssh root@91.98.224.44 "journalctl -u oxigraph --since '1 hour ago'"

# System logs
ssh root@91.98.224.44 "journalctl --since '1 hour ago'"
```

---

## Security Notes

### What's Protected

- HTTPS enabled via Caddy with automatic Let's Encrypt certificates
- SSH access restricted to key-based authentication
- Oxigraph only accessible from localhost (not exposed to internet)
- Caddy reverse proxies SPARQL queries to Oxigraph

### Sensitive Files

**NEVER commit these to git:**
- `.env` - Contains Hetzner API token
- `infrastructure/terraform/terraform.tfvars` - Contains secrets
- `infrastructure/terraform/*.tfstate*` - Contains infrastructure state

These are already in `.gitignore`.

---

## Deployment Checklist

Before deploying:

- [ ] Changes tested locally
- [ ] `.env` file exists with valid `HETZNER_HC_API_TOKEN`
- [ ] SSH access verified (`ssh root@91.98.224.44`)
- [ ] For frontend: `npm run build` succeeds locally
- [ ] For schemas: LinkML validation passes

After deploying:

- [ ] `./infrastructure/deploy.sh --status` shows services running
- [ ] Frontend loads in browser
- [ ] SPARQL queries return results (if data deployed)
- [ ] No errors in server logs

---

## Quick Commands Reference

```bash
# Deploy frontend
./infrastructure/deploy.sh --frontend

# Deploy data
./infrastructure/deploy.sh --data

# Check status
./infrastructure/deploy.sh --status

# SSH to server
ssh root@91.98.224.44

# View Caddy logs
ssh root@91.98.224.44 "journalctl -u caddy -f"

# Restart services
ssh root@91.98.224.44 "systemctl restart caddy"
ssh root@91.98.224.44 "systemctl restart oxigraph"
```

---

## Related Documentation

- `.opencode/DEPLOYMENT_RULES.md` - Rules for AI agents performing deployments
- `AGENTS.md` - Rule 7 covers deployment guidelines
- `infrastructure/deploy.sh` - The deployment script source code
- `infrastructure/terraform/` - Terraform configuration for server provisioning

---

**Last Updated**: 2025-12-01
**Maintainer**: GLAM Project Team