docs

docs/operations/database-runbook.md

@@ -0,0 +1,100 @@
+# Database Operations Runbook
+
+This runbook defines backup, restore, and verification procedures for the
+PostgreSQL database used by `scholarr`.
+
+## Objectives
+
+- Keep data loss bounded via scheduled backups.
+- Provide repeatable restore procedures.
+- Verify backups by running regular restore drills.
+
+Recommended starting targets:
+
+- `RPO`: 24 hours (maximum acceptable data loss).
+- `RTO`: 60 minutes (maximum acceptable restore time).
+
+## Backup Strategy
+
+Use logical backups from the running `db` compose service.
+
+- Custom-format backup (recommended for restore flexibility):
+
+```bash
+scripts/db/backup_full.sh
+```
+
+- Plain SQL backup:
+
+```bash
+scripts/db/backup_full.sh --plain
+```
+
+Optional environment variables:
+
+- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
+- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
+- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
+
+## Restore Strategy
+
+Restore from a `.dump` (custom format) or `.sql` file into the running `db`
+compose service.
+
+- Restore without schema wipe:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
+```
+
+- Restore with full `public` schema reset:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
+```
+
+## Safety Checklist Before Restore
+
+1. Pause writes (stop app/scheduler or set maintenance mode).
+2. Create a fresh pre-restore backup.
+3. Validate target dump checksum/size.
+4. Confirm operator, scope, and rollback plan.
+
+## Post-Restore Verification
+
+1. Run migrations head check.
+2. Run health endpoint checks.
+3. Verify table row counts for core tables:
+   - `users`
+   - `scholar_profiles`
+   - `publications`
+   - `scholar_publications`
+   - `crawl_runs`
+4. Run integrity report and require zero failures:
+
+```bash
+python3 scripts/db/check_integrity.py
+```
+
+5. Run API smoke checks for scholars/publications/runs pages.
+
+## Data Cleanup and Repair
+
+Use the audited repair job for scholar-publication relinking cleanup:
+
+```bash
+python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
+```
+
+Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
+
+If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout.
+
+## Restore Drill Cadence
+
+- Run at least one full restore drill per month.
+- Record:
+  - backup artifact used,
+  - restore start/end timestamps,
+  - issues encountered,
+  - achieved `RTO`.

docs/operations/migration-checklist.md

@@ -0,0 +1,48 @@
+# Migration Checklist
+
+Use this checklist for every schema/data migration.
+
+## 1. Design Review
+
+- Define change type: `expand`, `backfill`, `contract`.
+- Document expected lock behavior and index impact.
+- Confirm backward compatibility with currently deployed app version.
+- Define rollback strategy before implementation.
+
+## 2. Pre-Migration Controls
+
+- Capture a fresh backup before applying migration.
+- Confirm migration head revision and working tree cleanliness.
+- Prepare validation queries for new/changed tables and indexes.
+- Identify high-risk tables (large row count, hot write paths).
+
+## 3. Implementation Standards
+
+- Keep migrations idempotent when feasible.
+- Prefer additive steps first (`nullable`, new index, new table).
+- For destructive changes, separate into later contract migration.
+- Avoid large blocking rewrites in a single deployment step.
+
+## 4. Verification
+
+- Apply migration in staging against production-like snapshot.
+- Verify:
+  - expected tables/columns/indexes,
+  - app startup and health endpoint,
+  - affected API flows,
+  - data consistency queries.
+
+## 5. Rollout and Recovery
+
+- Apply migration during planned window.
+- Monitor logs/errors and DB metrics during rollout.
+- If rollback needed:
+  - follow downgrade/recovery runbook,
+  - restore from backup if downgrade is unsafe,
+  - document incident timeline.
+
+## 6. Post-Migration Tasks
+
+- Update `README.md` / `.env.example` / ops docs.
+- Add/update integration tests for new schema assumptions.
+- Record migration notes in changelog.

docs/operations/overview.md

@@ -0,0 +1,7 @@
+# Operations Documentation
+
+Use this section for production operations and incident/runbook workflows.
+
+- [Scrape Safety Runbook](./scrape-safety-runbook.md)
+- [Database Runbook](./database-runbook.md)
+- [Migration Rollout Checklist](./migration-checklist.md)

docs/operations/scrape-safety-runbook.md

@@ -0,0 +1,74 @@
+# Scrape Safety Runbook
+
+This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
+
+## Key Signals
+
+Use structured log `event` fields (recommended with `LOG_FORMAT=json`):
+
+- `ingestion.safety_policy_blocked_run_start`
+- `ingestion.safety_cooldown_entered`
+- `ingestion.safety_cooldown_cleared`
+- `ingestion.alert_blocked_failure_threshold_exceeded`
+- `ingestion.alert_network_failure_threshold_exceeded`
+- `ingestion.alert_retry_scheduled_threshold_exceeded`
+- `api.runs.manual_blocked_policy`
+- `api.runs.manual_blocked_safety`
+- `scheduler.run_skipped_safety_cooldown`
+- `scheduler.run_skipped_safety_cooldown_precheck`
+- `scheduler.queue_item_deferred_safety_cooldown`
+
+Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules.
+
+## Recommended Alert Rules
+
+- Cooldown enters:
+  - Trigger on `event=ingestion.safety_cooldown_entered`.
+- Repeated start blocks:
+  - Trigger on high rate of `event=api.runs.manual_blocked_safety`.
+- Threshold trips:
+  - Trigger on any of:
+    - `ingestion.alert_blocked_failure_threshold_exceeded`
+    - `ingestion.alert_network_failure_threshold_exceeded`
+    - `ingestion.alert_retry_scheduled_threshold_exceeded`
+- Scheduler pressure:
+  - Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`.
+
+## If Your IP Appears Blocked
+
+Symptoms:
+
+- cooldown reason `blocked_failure_threshold_exceeded`
+- parse state `blocked_or_captcha`
+- redirects toward Google account sign-in flows
+
+Actions:
+
+1. Stop manual retries immediately.
+2. Let cooldown expire; do not spam retriggers.
+3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values.
+4. Reduce concurrency pressure (keep one scheduler instance).
+5. Keep name-search disabled/WIP for now if login-gated responses persist.
+6. Resume with a small monitored run and verify blocked rate drops.
+
+Avoid:
+
+- aggressive rapid retries
+- rotating through risky scraping patterns that increase challenge rates
+- bypass/captcha-solving workflows that may violate source platform rules
+
+## Environment Controls
+
+Policy floors and safety controls:
+
+- `INGESTION_MIN_REQUEST_DELAY_SECONDS`
+- `INGESTION_MIN_RUN_INTERVAL_MINUTES`
+- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`
+- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`
+- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`
+- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`
+- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`
+- `INGESTION_MANUAL_RUN_ALLOWED`
+- `INGESTION_AUTOMATION_ALLOWED`
+
+Apply stricter values first, then relax slowly only after sustained healthy runs.