docs: rebuild documentation from scratch with clean IA
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
6c31b331f7
commit
0c5b199b76
30 changed files with 1647 additions and 649 deletions
|
|
@ -1,100 +1,195 @@
|
|||
# Database Operations Runbook
|
||||
---
|
||||
title: Database Runbook
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
This runbook defines backup, restore, and verification procedures for the
|
||||
PostgreSQL database used by `scholarr`.
|
||||
# Database Runbook
|
||||
|
||||
## Objectives
|
||||
## Backup
|
||||
|
||||
- Keep data loss bounded via scheduled backups.
|
||||
- Provide repeatable restore procedures.
|
||||
- Verify backups by running regular restore drills.
|
||||
Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service.
|
||||
|
||||
Recommended starting targets:
|
||||
|
||||
- `RPO`: 24 hours (maximum acceptable data loss).
|
||||
- `RTO`: 60 minutes (maximum acceptable restore time).
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
Use logical backups from the running `db` compose service.
|
||||
|
||||
- Custom-format backup (recommended for restore flexibility):
|
||||
### Custom-Format Backup (Default)
|
||||
|
||||
```bash
|
||||
scripts/db/backup_full.sh
|
||||
```
|
||||
|
||||
- Plain SQL backup:
|
||||
Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`).
|
||||
|
||||
### Plain SQL Backup
|
||||
|
||||
```bash
|
||||
scripts/db/backup_full.sh --plain
|
||||
```
|
||||
|
||||
Optional environment variables:
|
||||
Creates a `.sql` file.
|
||||
|
||||
- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
|
||||
- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
|
||||
- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
|
||||
### Options
|
||||
|
||||
## Restore Strategy
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--plain` | Write plain SQL instead of custom-format dump |
|
||||
|
||||
Restore from a `.dump` (custom format) or `.sql` file into the running `db`
|
||||
compose service.
|
||||
### Environment Variables
|
||||
|
||||
- Restore without schema wipe:
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BACKUP_DIR` | `<repo>/backups` | Destination directory |
|
||||
| `BACKUP_PREFIX` | `scholarr` | File prefix |
|
||||
| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` |
|
||||
|
||||
## Restore
|
||||
|
||||
Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service.
|
||||
|
||||
### Restore a Custom-Format Dump
|
||||
|
||||
```bash
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump
|
||||
```
|
||||
|
||||
- Restore with full `public` schema reset:
|
||||
### Restore with Schema Wipe
|
||||
|
||||
```bash
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public
|
||||
```
|
||||
|
||||
## Safety Checklist Before Restore
|
||||
This drops and recreates the `public` schema before restoring. Use with caution.
|
||||
|
||||
1. Pause writes (stop app/scheduler or set maintenance mode).
|
||||
2. Create a fresh pre-restore backup.
|
||||
3. Validate target dump checksum/size.
|
||||
4. Confirm operator, scope, and rollback plan.
|
||||
### Options
|
||||
|
||||
## Post-Restore Verification
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--file <path>` | Required. Path to `.dump` or `.sql` backup file |
|
||||
| `--wipe-public` | Drop and recreate `public` schema before restore |
|
||||
|
||||
1. Run migrations head check.
|
||||
2. Run health endpoint checks.
|
||||
3. Verify table row counts for core tables:
|
||||
- `users`
|
||||
- `scholar_profiles`
|
||||
- `publications`
|
||||
- `scholar_publications`
|
||||
- `crawl_runs`
|
||||
4. Run integrity report and require zero failures:
|
||||
## Integrity Checks
|
||||
|
||||
Use `scripts/db/check_integrity.py` to run database integrity checks.
|
||||
|
||||
### Run Inside Container
|
||||
|
||||
```bash
|
||||
python3 scripts/db/check_integrity.py
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/check_integrity.py
|
||||
```
|
||||
|
||||
5. Run API smoke checks for scholars/publications/runs pages.
|
||||
|
||||
## Data Cleanup and Repair
|
||||
|
||||
Use the audited repair job for scholar-publication relinking cleanup:
|
||||
### With Strict Warnings
|
||||
|
||||
```bash
|
||||
python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/check_integrity.py --strict-warnings
|
||||
```
|
||||
|
||||
Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
|
||||
Returns non-zero exit code if any warning is present.
|
||||
|
||||
If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout.
|
||||
### Output Format
|
||||
|
||||
## Restore Drill Cadence
|
||||
JSON report:
|
||||
|
||||
- Run at least one full restore drill per month.
|
||||
- Record:
|
||||
- backup artifact used,
|
||||
- restore start/end timestamps,
|
||||
- issues encountered,
|
||||
- achieved `RTO`.
|
||||
```json
|
||||
{
|
||||
"status": "passed",
|
||||
"failures": [],
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
Exit codes:
|
||||
- `0` - All checks passed
|
||||
- `1` - Check failure
|
||||
- `2` - Warnings present (with `--strict-warnings`)
|
||||
|
||||
### Via Admin API
|
||||
|
||||
```
|
||||
GET /api/v1/admin/db/integrity
|
||||
```
|
||||
|
||||
Returns the same integrity report through the API (admin auth required).
|
||||
|
||||
## Publication Link Repair
|
||||
|
||||
Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging.
|
||||
|
||||
### Dry Run (Default)
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/repair_publication_links.py --user-id 1
|
||||
```
|
||||
|
||||
### Apply Changes
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/repair_publication_links.py --user-id 1 --apply \
|
||||
--requested-by "admin@example.com"
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--user-id <int>` | Required. Target user ID |
|
||||
| `--scholar-profile-id <int>` | Optional. Filter by scholar profile ID (repeatable) |
|
||||
| `--apply` | Apply changes (default is dry-run) |
|
||||
| `--gc-orphan-publications` | Delete publications with zero links after cleanup |
|
||||
| `--requested-by <string>` | Operator identifier for audit logs |
|
||||
|
||||
### Via Admin API
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/repairs/publication-links
|
||||
```
|
||||
|
||||
## Near-Duplicate Repair
|
||||
|
||||
Detect and merge near-duplicate publications via the admin API:
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/repairs/publication-near-duplicates
|
||||
```
|
||||
|
||||
## PDF Queue Management
|
||||
|
||||
### List Queue
|
||||
|
||||
```
|
||||
GET /api/v1/admin/db/pdf-queue
|
||||
```
|
||||
|
||||
### Requeue Single Item
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/pdf-queue/{id}/requeue
|
||||
```
|
||||
|
||||
### Bulk Requeue
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/pdf-queue/requeue-all
|
||||
```
|
||||
|
||||
## Migration Procedures
|
||||
|
||||
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`.
|
||||
|
||||
To run manually:
|
||||
|
||||
```bash
|
||||
docker compose exec app alembic upgrade head
|
||||
```
|
||||
|
||||
To check current migration status:
|
||||
|
||||
```bash
|
||||
docker compose exec app alembic current
|
||||
```
|
||||
|
||||
Recommended procedure for production:
|
||||
1. Backup the database before upgrading.
|
||||
2. Pull the new image.
|
||||
3. Start the container; migrations run on startup.
|
||||
4. Verify via `GET /healthz` and check logs for migration output.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue