Big changes

docs/ops/db_runbook.md

@@ -0,0 +1,100 @@
+# Database Operations Runbook
+
+This runbook defines backup, restore, and verification procedures for the
+PostgreSQL database used by `scholarr`.
+
+## Objectives
+
+- Keep data loss bounded via scheduled backups.
+- Provide repeatable restore procedures.
+- Verify backups by running regular restore drills.
+
+Recommended starting targets:
+
+- `RPO`: 24 hours (maximum acceptable data loss).
+- `RTO`: 60 minutes (maximum acceptable restore time).
+
+## Backup Strategy
+
+Use logical backups from the running `db` compose service.
+
+- Custom-format backup (recommended for restore flexibility):
+
+```bash
+scripts/db/backup_full.sh
+```
+
+- Plain SQL backup:
+
+```bash
+scripts/db/backup_full.sh --plain
+```
+
+Optional environment variables:
+
+- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
+- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
+- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
+
+## Restore Strategy
+
+Restore from a `.dump` (custom format) or `.sql` file into the running `db`
+compose service.
+
+- Restore without schema wipe:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
+```
+
+- Restore with full `public` schema reset:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
+```
+
+## Safety Checklist Before Restore
+
+1. Pause writes (stop app/scheduler or set maintenance mode).
+2. Create a fresh pre-restore backup.
+3. Validate target dump checksum/size.
+4. Confirm operator, scope, and rollback plan.
+
+## Post-Restore Verification
+
+1. Run migrations head check.
+2. Run health endpoint checks.
+3. Verify table row counts for core tables:
+   - `users`
+   - `scholar_profiles`
+   - `publications`
+   - `scholar_publications`
+   - `crawl_runs`
+4. Run integrity report and require zero failures:
+
+```bash
+python3 scripts/db/check_integrity.py
+```
+
+5. Run API smoke checks for scholars/publications/runs pages.
+
+## Data Cleanup and Repair
+
+Use the audited repair job for scholar-publication relinking cleanup:
+
+```bash
+python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
+```
+
+Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
+
+If cleanup changes schema assumptions, follow `docs/ops/migration_checklist.md` before any migration rollout.
+
+## Restore Drill Cadence
+
+- Run at least one full restore drill per month.
+- Record:
+  - backup artifact used,
+  - restore start/end timestamps,
+  - issues encountered,
+  - achieved `RTO`.

docs/ops/migration_checklist.md

@@ -0,0 +1,48 @@
+# Migration Checklist
+
+Use this checklist for every schema/data migration.
+
+## 1. Design Review
+
+- Define change type: `expand`, `backfill`, `contract`.
+- Document expected lock behavior and index impact.
+- Confirm backward compatibility with currently deployed app version.
+- Define rollback strategy before implementation.
+
+## 2. Pre-Migration Controls
+
+- Capture a fresh backup before applying migration.
+- Confirm migration head revision and working tree cleanliness.
+- Prepare validation queries for new/changed tables and indexes.
+- Identify high-risk tables (large row count, hot write paths).
+
+## 3. Implementation Standards
+
+- Keep migrations idempotent when feasible.
+- Prefer additive steps first (`nullable`, new index, new table).
+- For destructive changes, separate into later contract migration.
+- Avoid large blocking rewrites in a single deployment step.
+
+## 4. Verification
+
+- Apply migration in staging against production-like snapshot.
+- Verify:
+  - expected tables/columns/indexes,
+  - app startup and health endpoint,
+  - affected API flows,
+  - data consistency queries.
+
+## 5. Rollout and Recovery
+
+- Apply migration during planned window.
+- Monitor logs/errors and DB metrics during rollout.
+- If rollback needed:
+  - follow downgrade/recovery runbook,
+  - restore from backup if downgrade is unsafe,
+  - document incident timeline.
+
+## 6. Post-Migration Tasks
+
+- Update `README.md` / `.env.example` / ops docs.
+- Add/update integration tests for new schema assumptions.
+- Record migration notes in changelog.