justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

docs: rebuild documentation from scratch with clean IA

Browse source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 09:39:27 +01:00
parent 6c31b331f7
commit 0c5b199b76
30 changed files with 1647 additions and 649 deletions
Download patch file Download diff file Expand all files Collapse all files

58
docs/operations/arxiv-runbook.md
View file

@ -1,46 +1,80 @@
---
title: arXiv Runbook
sidebar_position: 5
---
# arXiv Operations Runbook
Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly.
## Signals To Check
- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
## Signals to Check
- Logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
- `arxiv_runtime_state` row for cooldown and next-allowed timestamps
- `arxiv_query_cache_entries` size and expiry churn
## Event Field Guide
- `wait_seconds`: enforced pre-request delay from the global limiter.
- `status_code`: upstream response code from arXiv.
- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429.
- `source_path`: caller path (`search` or `lookup_ids`).
| Field | Description |
|-------|-------------|
| `wait_seconds` | Enforced pre-request delay from the global limiter |
| `status_code` | Upstream response code from arXiv |
| `cooldown_remaining_seconds` | Remaining cooldown when blocked or after 429 |
| `source_path` | Caller path (`search` or `lookup_ids`) |
## Quick SQL Checks
### Runtime State
```sql
SELECT state_key, next_allowed_at, cooldown_until, updated_at
FROM arxiv_runtime_state;
```
### Cache Status
```sql
SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry
SELECT count(*) AS cache_rows,
min(expires_at) AS earliest_expiry,
max(expires_at) AS latest_expiry
FROM arxiv_query_cache_entries;
```
## Common Scenarios
1. Repeated `arxiv.cooldown_activated` events:
- Confirm recent `429` statuses in `arxiv.request_completed`.
- Reduce caller pressure (check new title-quality/identifier guards are active).
### 1. Repeated `arxiv.cooldown_activated` Events
- Confirm recent `429` statuses in `arxiv.request_completed` logs.
- Reduce caller pressure (check title-quality/identifier guards are active).
- Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict.
2. High request latency with few completions:
### 2. High Request Latency with Few Completions
- Inspect `wait_seconds` in `arxiv.request_scheduled`.
- Verify only one process path is repeatedly hitting arXiv (`source_path`).
- Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears).
3. Low cache effectiveness:
### 3. Low Cache Effectiveness
- Validate normalized query behavior and caller churn.
- Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads.
- Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed.
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `ARXIV_ENABLED` | `1` | Enable arXiv lookups |
| `ARXIV_TIMEOUT_SECONDS` | `3.0` | Request timeout |
| `ARXIV_MIN_INTERVAL_SECONDS` | `4.0` | Min interval between requests |
| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | `60.0` | Cooldown after 429 |
| `ARXIV_DEFAULT_MAX_RESULTS` | `3` | Max results per query |
| `ARXIV_CACHE_TTL_SECONDS` | `900` | Cache TTL (15 min) |
| `ARXIV_CACHE_MAX_ENTRIES` | `512` | Max cached queries |
| `ARXIV_MAILTO` | *(empty)* | Contact email for API headers |
## Safe Recovery
1. Pause automated ingestion if rate-limit storms persist.
2. Let cooldown expire naturally; avoid manual burst retries.
3. Resume and monitor event rates before restoring full load.

215
docs/operations/database-runbook.md
View file

@ -1,100 +1,195 @@
# Database Operations Runbook
---
title: Database Runbook
sidebar_position: 3
---
This runbook defines backup, restore, and verification procedures for the
PostgreSQL database used by `scholarr`.
# Database Runbook
## Objectives
## Backup
- Keep data loss bounded via scheduled backups.
- Provide repeatable restore procedures.
- Verify backups by running regular restore drills.
Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service.
Recommended starting targets:
- `RPO`: 24 hours (maximum acceptable data loss).
- `RTO`: 60 minutes (maximum acceptable restore time).
## Backup Strategy
Use logical backups from the running `db` compose service.
- Custom-format backup (recommended for restore flexibility):
### Custom-Format Backup (Default)
```bash
scripts/db/backup_full.sh
```
- Plain SQL backup:
Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`).
### Plain SQL Backup
```bash
scripts/db/backup_full.sh --plain
```
Optional environment variables:
Creates a `.sql` file.
- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
### Options
## Restore Strategy
| Option | Description |
|--------|-------------|
| `--plain` | Write plain SQL instead of custom-format dump |
Restore from a `.dump` (custom format) or `.sql` file into the running `db`
compose service.
### Environment Variables
- Restore without schema wipe:
| Variable | Default | Description |
|----------|---------|-------------|
| `BACKUP_DIR` | `<repo>/backups` | Destination directory |
| `BACKUP_PREFIX` | `scholarr` | File prefix |
| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` |
## Restore
Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service.
### Restore a Custom-Format Dump
```bash
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump
```
- Restore with full `public` schema reset:
### Restore with Schema Wipe
```bash
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public
```
## Safety Checklist Before Restore
This drops and recreates the `public` schema before restoring. Use with caution.
1. Pause writes (stop app/scheduler or set maintenance mode).
2. Create a fresh pre-restore backup.
3. Validate target dump checksum/size.
4. Confirm operator, scope, and rollback plan.
### Options
## Post-Restore Verification
| Option | Description |
|--------|-------------|
| `--file <path>` | Required. Path to `.dump` or `.sql` backup file |
| `--wipe-public` | Drop and recreate `public` schema before restore |
1. Run migrations head check.
2. Run health endpoint checks.
3. Verify table row counts for core tables:
- `users`
- `scholar_profiles`
- `publications`
- `scholar_publications`
- `crawl_runs`
4. Run integrity report and require zero failures:
## Integrity Checks
Use `scripts/db/check_integrity.py` to run database integrity checks.
### Run Inside Container
```bash
python3 scripts/db/check_integrity.py
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/check_integrity.py
```
5. Run API smoke checks for scholars/publications/runs pages.
## Data Cleanup and Repair
Use the audited repair job for scholar-publication relinking cleanup:
### With Strict Warnings
```bash
python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/check_integrity.py --strict-warnings
```
Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
Returns non-zero exit code if any warning is present.
If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout.
### Output Format
## Restore Drill Cadence
JSON report:
- Run at least one full restore drill per month.
- Record:
- backup artifact used,
- restore start/end timestamps,
- issues encountered,
- achieved `RTO`.
```json
{
"status": "passed",
"failures": [],
"warnings": []
}
```
Exit codes:
- `0` - All checks passed
- `1` - Check failure
- `2` - Warnings present (with `--strict-warnings`)
### Via Admin API
```
GET /api/v1/admin/db/integrity
```
Returns the same integrity report through the API (admin auth required).
## Publication Link Repair
Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging.
### Dry Run (Default)
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/repair_publication_links.py --user-id 1
```
### Apply Changes
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/repair_publication_links.py --user-id 1 --apply \
--requested-by "admin@example.com"
```
### Options
| Option | Description |
|--------|-------------|
| `--user-id <int>` | Required. Target user ID |
| `--scholar-profile-id <int>` | Optional. Filter by scholar profile ID (repeatable) |
| `--apply` | Apply changes (default is dry-run) |
| `--gc-orphan-publications` | Delete publications with zero links after cleanup |
| `--requested-by <string>` | Operator identifier for audit logs |
### Via Admin API
```
POST /api/v1/admin/db/repairs/publication-links
```
## Near-Duplicate Repair
Detect and merge near-duplicate publications via the admin API:
```
POST /api/v1/admin/db/repairs/publication-near-duplicates
```
## PDF Queue Management
### List Queue
```
GET /api/v1/admin/db/pdf-queue
```
### Requeue Single Item
```
POST /api/v1/admin/db/pdf-queue/{id}/requeue
```
### Bulk Requeue
```
POST /api/v1/admin/db/pdf-queue/requeue-all
```
## Migration Procedures
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`.
To run manually:
```bash
docker compose exec app alembic upgrade head
```
To check current migration status:
```bash
docker compose exec app alembic current
```
Recommended procedure for production:
1. Backup the database before upgrading.
2. Pull the new image.
3. Start the container; migrations run on startup.
4. Verify via `GET /healthz` and check logs for migration output.

117
docs/operations/deployment.md Normal file
View file

@ -0,0 +1,117 @@
---
title: Deployment
sidebar_position: 2
---
# Deployment
## Production Docker Compose
The default `docker-compose.yml` runs two services:
| Service | Image | Description |
|---------|-------|-------------|
| `db` | `postgres:15-alpine` | PostgreSQL database with persistent volume |
| `app` | `justinzeus/scholarr:latest` | FastAPI application with embedded frontend |
### Start
```bash
docker compose up -d
```
### Stop
```bash
docker compose down
```
### Update
```bash
docker compose pull
docker compose up -d
```
### View Logs
```bash
docker compose logs -f app
docker compose logs -f db
```
## Required Environment Variables
These must be set in `.env` or the shell environment:
```bash
POSTGRES_PASSWORD=<secure-password>
SESSION_SECRET_KEY=<random-string-32-chars-minimum>
```
## Volumes
| Volume | Mount | Description |
|--------|-------|-------------|
| `postgres_data` | `/var/lib/postgresql/data` | Database files |
| `scholar_uploads` | `/var/lib/scholarr/uploads` | Scholar profile images |
## Health Checks
### Database
```yaml
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"]
interval: 5s
timeout: 5s
retries: 20
```
### Application
```yaml
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"]
interval: 10s
timeout: 5s
retries: 12
```
The app service depends on `db` with `condition: service_healthy`, so it waits for the database to be ready.
## Database Readiness Wait
The app has built-in database readiness polling:
- `DB_WAIT_TIMEOUT_SECONDS` (default: 60) - Max wait time
- `DB_WAIT_INTERVAL_SECONDS` (default: 2) - Poll interval
## Auto-Migration
Set `MIGRATE_ON_START=1` (default) to run Alembic migrations automatically on startup.
## Scaling Considerations
- Run a single `app` instance to avoid scheduler conflicts (the scheduler is process-local).
- arXiv requests are globally serialized via a PostgreSQL advisory lock, so multiple instances safely share the rate limiter.
- Database pool defaults: 5 base connections + 10 overflow. Adjust `DATABASE_POOL_SIZE` and `DATABASE_POOL_MAX_OVERFLOW` for higher loads.
## Admin Bootstrap
For first-time setup without manual database access:
```bash
BOOTSTRAP_ADMIN_ON_START=1
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
BOOTSTRAP_ADMIN_PASSWORD=<secure-password>
```
Set `BOOTSTRAP_ADMIN_FORCE_PASSWORD=1` to reset an existing admin password.
## Security Hardening
- Set `SESSION_COOKIE_SECURE=1` when serving over HTTPS.
- Enable HSTS: `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED=1`.
- Review CSP policy in `SECURITY_CSP_POLICY` for your domain.
- Set `UNPAYWALL_EMAIL` and `ARXIV_MAILTO` for polite API pool access.

48
docs/operations/migration-checklist.md
View file

@ -1,48 +0,0 @@
# Migration Checklist
Use this checklist for every schema/data migration.
## 1. Design Review
- Define change type: `expand`, `backfill`, `contract`.
- Document expected lock behavior and index impact.
- Confirm backward compatibility with currently deployed app version.
- Define rollback strategy before implementation.
## 2. Pre-Migration Controls
- Capture a fresh backup before applying migration.
- Confirm migration head revision and working tree cleanliness.
- Prepare validation queries for new/changed tables and indexes.
- Identify high-risk tables (large row count, hot write paths).
## 3. Implementation Standards
- Keep migrations idempotent when feasible.
- Prefer additive steps first (`nullable`, new index, new table).
- For destructive changes, separate into later contract migration.
- Avoid large blocking rewrites in a single deployment step.
## 4. Verification
- Apply migration in staging against production-like snapshot.
- Verify:
- expected tables/columns/indexes,
- app startup and health endpoint,
- affected API flows,
- data consistency queries.
## 5. Rollout and Recovery
- Apply migration during planned window.
- Monitor logs/errors and DB metrics during rollout.
- If rollback needed:
- follow downgrade/recovery runbook,
- restore from backup if downgrade is unsafe,
- document incident timeline.
## 6. Post-Migration Tasks
- Update `README.md` / `.env.example` / ops docs.
- Add/update integration tests for new schema assumptions.
- Record migration notes in changelog.

34
docs/operations/overview.md
View file

@ -1,8 +1,30 @@
# Operations Documentation
---
title: Operations Overview
sidebar_position: 1
---
Use this section for production operations and incident/runbook workflows.
# Operations Overview
- [Scrape Safety Runbook](./scrape-safety-runbook.md)
- [arXiv Runbook](./arxiv-runbook.md)
- [Database Runbook](./database-runbook.md)
- [Migration Rollout Checklist](./migration-checklist.md)
This section covers production deployment, database administration, and scrape safety operations.
## Quick Links
| Guide | Description |
|-------|-------------|
| [Deployment](deployment.md) | Production Docker setup, scaling, health checks |
| [Database Runbook](database-runbook.md) | Backup, restore, integrity checks, repair procedures |
| [Scrape Safety Runbook](scrape-safety-runbook.md) | Rate limiting, cooldowns, CAPTCHA handling |
| [arXiv Runbook](arxiv-runbook.md) | arXiv rate limits, cache tuning, query patterns |
## Health Check
The app exposes `GET /healthz` for container orchestration:
```bash
curl -fsS http://localhost:8000/healthz
```
Docker Compose healthcheck config:
- Interval: 10s
- Timeout: 5s
- Retries: 12

88
docs/operations/scrape-safety-runbook.md
View file

@ -1,74 +1,76 @@
---
title: Scrape Safety Runbook
sidebar_position: 4
---
# Scrape Safety Runbook
This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
Operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
## Key Signals
Use structured log `event` fields (recommended with `LOG_FORMAT=json`):
- `ingestion.safety_policy_blocked_run_start`
- `ingestion.safety_cooldown_entered`
- `ingestion.safety_cooldown_cleared`
- `ingestion.alert_blocked_failure_threshold_exceeded`
- `ingestion.alert_network_failure_threshold_exceeded`
- `ingestion.alert_retry_scheduled_threshold_exceeded`
- `api.runs.manual_blocked_policy`
- `api.runs.manual_blocked_safety`
- `scheduler.run_skipped_safety_cooldown`
- `scheduler.run_skipped_safety_cooldown_precheck`
- `scheduler.queue_item_deferred_safety_cooldown`
| Event | Description |
|-------|-------------|
| `ingestion.safety_policy_blocked_run_start` | Run blocked by safety policy |
| `ingestion.safety_cooldown_entered` | Cooldown activated |
| `ingestion.safety_cooldown_cleared` | Cooldown expired |
| `ingestion.alert_blocked_failure_threshold_exceeded` | Blocked failure threshold tripped |
| `ingestion.alert_network_failure_threshold_exceeded` | Network failure threshold tripped |
| `ingestion.alert_retry_scheduled_threshold_exceeded` | Retry threshold tripped |
| `api.runs.manual_blocked_policy` | Manual run blocked by policy |
| `api.runs.manual_blocked_safety` | Manual run blocked by safety cooldown |
| `scheduler.run_skipped_safety_cooldown` | Scheduled run skipped due to cooldown |
| `scheduler.run_skipped_safety_cooldown_precheck` | Scheduler precheck blocked |
| `scheduler.queue_item_deferred_safety_cooldown` | Queue item deferred due to cooldown |
Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules.
Each event includes metric-style fields (`metric_name`, `metric_value`) for log-based alert rules.
## Recommended Alert Rules
- Cooldown enters:
- Trigger on `event=ingestion.safety_cooldown_entered`.
- Repeated start blocks:
- Trigger on high rate of `event=api.runs.manual_blocked_safety`.
- Threshold trips:
- Trigger on any of:
- `ingestion.alert_blocked_failure_threshold_exceeded`
- `ingestion.alert_network_failure_threshold_exceeded`
- `ingestion.alert_retry_scheduled_threshold_exceeded`
- Scheduler pressure:
- Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`.
- **Cooldown enters**: Trigger on `event=ingestion.safety_cooldown_entered`
- **Repeated start blocks**: High rate of `event=api.runs.manual_blocked_safety`
- **Threshold trips**: Any of the `*_threshold_exceeded` events
- **Scheduler pressure**: Sustained `scheduler.queue_item_deferred_safety_cooldown`
## If Your IP Appears Blocked
Symptoms:
### Symptoms
- cooldown reason `blocked_failure_threshold_exceeded`
- parse state `blocked_or_captcha`
- redirects toward Google account sign-in flows
- Cooldown reason: `blocked_failure_threshold_exceeded`
- Parse state: `blocked_or_captcha`
- Redirects toward Google account sign-in flows
Actions:
### Actions
1. Stop manual retries immediately.
2. Let cooldown expire; do not spam retriggers.
3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values.
4. Reduce concurrency pressure (keep one scheduler instance).
5. Keep name-search disabled/WIP for now if login-gated responses persist.
5. Keep name-search disabled if login-gated responses persist.
6. Resume with a small monitored run and verify blocked rate drops.
Avoid:
### Avoid
- aggressive rapid retries
- rotating through risky scraping patterns that increase challenge rates
- bypass/captcha-solving workflows that may violate source platform rules
- Aggressive rapid retries
- Rotating through risky scraping patterns that increase challenge rates
- Bypass/CAPTCHA-solving workflows that may violate source platform rules
## Environment Controls
Policy floors and safety controls:
- `INGESTION_MIN_REQUEST_DELAY_SECONDS`
- `INGESTION_MIN_RUN_INTERVAL_MINUTES`
- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`
- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`
- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`
- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`
- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`
- `INGESTION_MANUAL_RUN_ALLOWED`
- `INGESTION_AUTOMATION_ALLOWED`
| Variable | Default | Description |
|----------|---------|-------------|
| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | `2` | Floor delay between requests |
| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | `15` | Minimum time between runs |
| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `1` | Blocked failures before alert |
| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `2` | Network failures before alert |
| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `3` | Retries before alert |
| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `1800` | Cooldown after blocked threshold (30 min) |
| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `900` | Cooldown after network threshold (15 min) |
| `INGESTION_MANUAL_RUN_ALLOWED` | `1` | Enable manual runs |
| `INGESTION_AUTOMATION_ALLOWED` | `1` | Enable automated runs |
Apply stricter values first, then relax slowly only after sustained healthy runs.