Big changes

docs/README.md

@@ -0,0 +1,30 @@
+# Documentation Index
+
+Use this directory for project documentation outside `README.md` and `agents.md`.
+
+## Deploy
+
+- `docs/deploy/quickstart.md`: deployment modes, local dev startup, and quality command reference.
+
+## Reference
+
+- `docs/reference/environment.md`: complete environment-variable catalog and contract/audit notes.
+- `docs/reference/api_contract.md`: API envelope contract and endpoint semantics.
+
+## Architecture
+
+- `docs/architecture/domain_boundaries.md`: domain boundaries, user-scoped data rules, and frontend behavior notes.
+
+## Operations
+
+- `docs/ops/scrape_safety_runbook.md`: scrape cooldown and blocked-IP operations.
+- `docs/ops/db_runbook.md`: backup, restore, and verification workflows.
+- `docs/ops/migration_checklist.md`: migration planning and rollout checklist.
+
+## Frontend
+
+- `docs/frontend/theme_phase0_inventory.md`: theme-token inventory and adoption notes.
+
+## Contribution
+
+- `docs/contributing.md`: contribution policy and merge checklist.

docs/architecture/domain_boundaries.md

@@ -0,0 +1,25 @@
+# Domain Boundaries
+
+## Data Model Rules
+
+- Scholar tracking is user-scoped.
+- Publications are global/deduplicated records.
+- Read/favorite/visibility state stays on scholar-publication link rows.
+
+## Service Boundaries
+
+Canonical business logic belongs in `app/services/domains/*`.
+
+- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety.
+- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
+- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
+- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
+- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
+- `app/services/domains/runs/*`: run history and continuation queue operations.
+- `app/services/domains/portability/*`: import/export workflows.
+
+## Frontend Behavior Notes
+
+- Mobile primary nav is in a left drawer and closes on route change or logout.
+- Long list views use internal scroll containers.
+- Name search remains intentionally constrained due upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URL.

docs/contributing.md

@@ -0,0 +1,24 @@
+# Contributing
+
+## Scope
+This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe.
+
+## Essential-File Policy
+Commit only source-of-truth files required to build, run, test, or document the app.
+
+Do not commit generated or local-only artifacts, including:
+- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`
+- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/`
+- coverage outputs (`.coverage`, `htmlcov/`)
+- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`)
+- local probe/scratch material (`planning/`)
+
+CI enforces this with `scripts/check_no_generated_artifacts.sh`.
+
+## Merge Checklist
+- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas.
+- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed).
+- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`).
+- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change.
+- [ ] `README.md`, `.env.example`, and deployment notes stay aligned.
+- [ ] `scripts/check_no_generated_artifacts.sh` passes locally.

docs/deploy/quickstart.md

@@ -0,0 +1,68 @@
+# Deploy and Dev Quickstart
+
+## Required Setup
+
+1. Copy `.env.example` to `.env`.
+2. Set `POSTGRES_PASSWORD`.
+3. Set `SESSION_SECRET_KEY`.
+
+## Deploy Method A: Prebuilt Image
+
+```bash
+docker compose pull
+docker compose up -d
+```
+
+Open:
+- App/API: `http://localhost:8000`
+- Health endpoint: `http://localhost:8000/healthz`
+
+Upgrade:
+
+```bash
+docker compose pull
+docker compose up -d
+```
+
+## Deploy Method B: Local Source + Dev Frontend
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
+```
+
+Open:
+- API: `http://localhost:8000`
+- Frontend dev server: `http://localhost:5173`
+
+Stop:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml down
+```
+
+## Quality Commands
+
+Backend:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration
+```
+
+Frontend:
+
+```bash
+cd frontend
+npm install
+npm run typecheck
+npm run test:run
+npm run build
+```
+
+Repository checks:
+
+```bash
+python3 scripts/check_frontend_api_contract.py
+python3 scripts/check_env_contract.py
+./scripts/check_no_generated_artifacts.sh
+```

docs/frontend/theme_phase0_inventory.md

@@ -0,0 +1,89 @@
+# Theme Inventory (Phase 0)
+
+This file captures the semantic color system baseline for the theme refactor.
+
+## Token Domains
+
+- `scale`:
+  - `brand` (`50..950`)
+  - `info` (`50..950`)
+  - `success` (`50..950`)
+  - `warning` (`50..950`)
+  - `danger` (`50..950`)
+- `surface`:
+  - `app`
+  - `nav`
+  - `nav_active`
+  - `card`
+  - `card_muted`
+  - `table`
+  - `table_header`
+  - `input`
+  - `overlay`
+- `text`:
+  - `primary`
+  - `secondary`
+  - `muted`
+  - `inverse`
+  - `link`
+- `border`:
+  - `default`
+  - `strong`
+  - `subtle`
+  - `interactive`
+- `focus`:
+  - `ring`
+  - `ring_offset`
+- `action` variants (`primary`, `secondary`, `ghost`, `danger`):
+  - `bg`
+  - `border`
+  - `text`
+  - `hover_bg`
+  - `hover_border`
+  - `hover_text`
+- `state` variants (`info`, `success`, `warning`, `danger`):
+  - `bg`
+  - `border`
+  - `text`
+
+## Theme Sources
+
+Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`.
+
+Current preset files:
+
+- `frontend/src/theme/presets/parchment.js`
+- `frontend/src/theme/presets/lilac.js`
+- `frontend/src/theme/presets/dune.js`
+- `frontend/src/theme/presets/oatmeal.js`
+- `frontend/src/theme/presets/scholarly.json`
+- `frontend/src/theme/presets/graphite.json`
+- `frontend/src/theme/presets/tide.json`
+
+Each preset contains both `light` and `dark` mode token definitions.
+
+## Adoption Status (Phase 0-3 complete baseline)
+
+Tokenized foundation components:
+
+- `AppButton`
+- `AppCard`
+- `AppCheckbox`
+- `AppEmptyState`
+- `AppHelpHint`
+- `AppInput`
+- `AppSelect`
+- `AppTable`
+- `AppModal`
+- `AppHeader`
+- `AppNav`
+- `AppAlert`
+- `AppBadge`
+- `RunStatusBadge`
+- `QueueHealthBadge`
+
+Hardening in place:
+
+- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs`
+- CI enforcement step in `frontend-quality` workflow
+- Theme preset integrity tests in `frontend/src/theme/presets.test.ts`

docs/index.md

@@ -0,0 +1,12 @@
+---
+slug: /
+---
+
+# scholarr Documentation
+
+- Start here: `docs/deploy/quickstart.md`
+- Runtime config: `docs/reference/environment.md`
+- API contract: `docs/reference/api_contract.md`
+- Domain boundaries: `docs/architecture/domain_boundaries.md`
+- Ops runbooks: `docs/ops/`
+- Frontend theme inventory: `docs/frontend/theme_phase0_inventory.md`

docs/ops/db_runbook.md

@@ -0,0 +1,100 @@
+# Database Operations Runbook
+
+This runbook defines backup, restore, and verification procedures for the
+PostgreSQL database used by `scholarr`.
+
+## Objectives
+
+- Keep data loss bounded via scheduled backups.
+- Provide repeatable restore procedures.
+- Verify backups by running regular restore drills.
+
+Recommended starting targets:
+
+- `RPO`: 24 hours (maximum acceptable data loss).
+- `RTO`: 60 minutes (maximum acceptable restore time).
+
+## Backup Strategy
+
+Use logical backups from the running `db` compose service.
+
+- Custom-format backup (recommended for restore flexibility):
+
+```bash
+scripts/db/backup_full.sh
+```
+
+- Plain SQL backup:
+
+```bash
+scripts/db/backup_full.sh --plain
+```
+
+Optional environment variables:
+
+- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
+- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
+- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
+
+## Restore Strategy
+
+Restore from a `.dump` (custom format) or `.sql` file into the running `db`
+compose service.
+
+- Restore without schema wipe:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
+```
+
+- Restore with full `public` schema reset:
+
+```bash
+scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
+```
+
+## Safety Checklist Before Restore
+
+1. Pause writes (stop app/scheduler or set maintenance mode).
+2. Create a fresh pre-restore backup.
+3. Validate target dump checksum/size.
+4. Confirm operator, scope, and rollback plan.
+
+## Post-Restore Verification
+
+1. Run migrations head check.
+2. Run health endpoint checks.
+3. Verify table row counts for core tables:
+   - `users`
+   - `scholar_profiles`
+   - `publications`
+   - `scholar_publications`
+   - `crawl_runs`
+4. Run integrity report and require zero failures:
+
+```bash
+python3 scripts/db/check_integrity.py
+```
+
+5. Run API smoke checks for scholars/publications/runs pages.
+
+## Data Cleanup and Repair
+
+Use the audited repair job for scholar-publication relinking cleanup:
+
+```bash
+python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
+```
+
+Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
+
+If cleanup changes schema assumptions, follow `docs/ops/migration_checklist.md` before any migration rollout.
+
+## Restore Drill Cadence
+
+- Run at least one full restore drill per month.
+- Record:
+  - backup artifact used,
+  - restore start/end timestamps,
+  - issues encountered,
+  - achieved `RTO`.

docs/ops/migration_checklist.md

@@ -0,0 +1,48 @@
+# Migration Checklist
+
+Use this checklist for every schema/data migration.
+
+## 1. Design Review
+
+- Define change type: `expand`, `backfill`, `contract`.
+- Document expected lock behavior and index impact.
+- Confirm backward compatibility with currently deployed app version.
+- Define rollback strategy before implementation.
+
+## 2. Pre-Migration Controls
+
+- Capture a fresh backup before applying migration.
+- Confirm migration head revision and working tree cleanliness.
+- Prepare validation queries for new/changed tables and indexes.
+- Identify high-risk tables (large row count, hot write paths).
+
+## 3. Implementation Standards
+
+- Keep migrations idempotent when feasible.
+- Prefer additive steps first (`nullable`, new index, new table).
+- For destructive changes, separate into later contract migration.
+- Avoid large blocking rewrites in a single deployment step.
+
+## 4. Verification
+
+- Apply migration in staging against production-like snapshot.
+- Verify:
+  - expected tables/columns/indexes,
+  - app startup and health endpoint,
+  - affected API flows,
+  - data consistency queries.
+
+## 5. Rollout and Recovery
+
+- Apply migration during planned window.
+- Monitor logs/errors and DB metrics during rollout.
+- If rollback needed:
+  - follow downgrade/recovery runbook,
+  - restore from backup if downgrade is unsafe,
+  - document incident timeline.
+
+## 6. Post-Migration Tasks
+
+- Update `README.md` / `.env.example` / ops docs.
+- Add/update integration tests for new schema assumptions.
+- Record migration notes in changelog.

docs/reference/api_contract.md

@@ -0,0 +1,29 @@
+# API Contract
+
+## Envelope Invariant
+
+All API responses under `/api/v1` use one of these envelopes:
+
+- Success: `{"data": ..., "meta": {"request_id": "..."}}`
+- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
+
+`meta.request_id` must be present for success and error responses.
+
+## Publications Semantics
+
+- `GET /api/v1/publications` supports `mode=all|unread|latest`.
+- `mode=new` is currently accepted as a compatibility alias for `latest`.
+- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
+- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`.
+- `unread` represents read-state (`is_read=false`).
+- `latest` represents discovery-state (`first seen in the latest completed run`).
+
+Publication payloads expose:
+- `pub_url` (canonical scholar detail URL)
+- `doi` (normalized DOI)
+- `pdf_url` (resolved OA PDF when available)
+
+## Scholar Portability
+
+- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state.
+- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication.

docs/reference/environment.md

@@ -0,0 +1,68 @@
+# Environment Reference and Audit
+
+This document is the source-of-truth audit for what remains configurable versus internal defaults.
+
+## Decision Summary
+
+- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras.
+- Move to internal defaults only: none at this time.
+- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted.
+
+## Configurable Variables (By Section)
+
+### Compose + Database
+
+`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE`
+
+### App Runtime + Networking
+
+`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR`
+
+### Database Pool
+
+`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS`
+
+### Frontend Dev Overrides
+
+`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET`
+
+### Auth + Session
+
+`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS`
+
+### HTTP Security Headers + CSP
+
+`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD`
+
+### Logging
+
+`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS`
+
+### Scheduler + Ingestion Safety
+
+`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
+
+### Scholar Images + Name Search Safety
+
+`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD`
+
+### OA Enrichment + PDF Resolution
+
+`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`
+
+### Startup Bootstrap + DB Wait
+
+`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS`
+
+## Drift Guard
+
+CI enforces env parity with:
+
+```bash
+python3 scripts/check_env_contract.py
+```
+
+The check fails when:
+- `app/settings.py` references a variable missing from `.env.example`.
+- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras).
+- `.env.example` contains duplicate keys.