docs/README.md

@@ -1,23 +0,0 @@
-# Documentation Index
-
-This directory contains both docs content and docs tooling.
-
-## Audience-Based Structure
-
-- `docs/user/`: onboarding and usage-oriented docs for self-hosters.
-- `docs/operations/`: runbooks and rollout checklists for ongoing operations.
-- `docs/developer/`: architecture, contribution standards, and local development workflows.
-- `docs/reference/`: canonical API and environment contracts.
-
-## Docs Tooling
-
-- `docs/website/`: Docusaurus app used to publish docs to GitHub Pages.
-
-## Recommended Reading Paths
-
-- Users: `docs/user/getting-started.md`
-- Operators: `docs/operations/overview.md`
-- Developers: `docs/developer/overview.md`
-- Contract consumers: `docs/reference/overview.md`
-
-Published site: https://justinzeus.github.io/scholarr/

docs/developer/api-contract.md

@@ -1,12 +0,0 @@
-# API Contracts
-
-`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI.
-
-## DTO Models Structure
-FastAPI routes dynamically ingest request/response Data Transfer Objects.
-- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin.
-- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend.
-
-## Frontend Contract Requirements
-The UI exclusively calls routes exposed by FastAPI's `APIRouter`. 
-All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally.

docs/developer/architecture.md

@@ -1,58 +1,173 @@
-# Domain Boundaries
+---
+title: Architecture
+sidebar_position: 2
+---
+
+# Architecture
 
 ## Data Model Rules
 
-- Scholar tracking is user-scoped.
+- **Scholar tracking is user-scoped.** Each user manages their own list of tracked scholars. Validate mapping/join tables; never assume global links between users and Scholar IDs.
-- Publications are global/deduplicated records.
+- **Publications are global, deduplicated records.** Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
-- Read/favorite/visibility state stays on scholar-publication link rows.
+- **State lives on the link.** Read/unread, favorites, and visibility state exist exclusively on the scholar-publication link table, not the global publication table.
 
-## Service Boundaries
+## Domain Service Boundaries
 
-Canonical business logic belongs in `app/services/domains/*`.
+All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited.
 
-- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety.
+### Ingestion (`app/services/ingestion/`)
-- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
-- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
-- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
-- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
-- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
-- `app/services/domains/runs/*`: run history and continuation queue operations.
-- `app/services/domains/portability/*`: import/export workflows.
 
-## Frontend Behavior Notes
+Run orchestration, continuation queue, scheduler integration, and scrape safety policy enforcement. The `ScholarIngestionService` drives the primary data acquisition loop.
 
-- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
+Key modules:
-- Lists: Long list views use internal scroll containers to prevent viewport overflow.
+- `application.py` - Main ingestion orchestrator
-- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.
+- `scheduler.py` - Background tick loop, queue batch processing
+- `constants.py` - Safety policy constants and floor values
+- `fingerprints.py` - Publication fingerprinting for deduplication
+- `types.py` - Ingestion result types and state enums
 
-## Data Integration & Acquisition Flow
+### Scholar Parsing (`app/services/scholar/`)
+
+Fail-fast Google Scholar HTML parsing and source fetch adapters. Handles paginated HTML feeds, extracts publication blocks via regex and DOM invariants (e.g., `gsc_vcd_cib`).
+
+Key modules:
+- `parser.py` - HTML parser for publication extraction
+- `parser_utils.py` - Parsing helpers and DOM selectors
+- `source.py` - HTTP fetch adapters with browser headers
+- `profile_rows.py` - Profile metadata extraction
+- `author_rows.py` - Author citation row parsing
+- `state_detection.py` - Blocked/CAPTCHA/rate-limit detection
+
+### Scholar Management (`app/services/scholars/`)
+
+Scholar CRUD, profile image management, and name-search controls with rate limiting.
+
+Key modules:
+- `application.py` - Scholar lifecycle operations
+- `uploads.py` - Image upload handling
+- `search_hints.py` - Name search with caching and cooldowns
+- `validators.py` - Input validation for scholar creation
+
+### Publications (`app/services/publications/`)
+
+Listing, read-state management, favorite toggles, enrichment scheduling, deduplication, and PDF queue management.
+
+Key modules:
+- `application.py` - Publication service facade
+- `listing.py` - Filtered listing with pagination (modes: all/unread/latest)
+- `queries.py` - Database query builders
+- `counts.py` - Aggregation counts for dashboard
+- `dedup.py` - Duplicate detection and merging
+- `enrichment.py` - Identifier and metadata enrichment orchestration
+- `pdf_queue.py` - PDF resolution queue policy
+- `pdf_resolution_pipeline.py` - Multi-source PDF resolution (Unpaywall, arXiv)
+- `types.py` - Publication DTOs and response types
+
+### Publication Identifiers (`app/services/publication_identifiers/`)
+
+Multi-identifier resolution engine. A single publication can have multiple identifiers (DOI, arXiv ID, PMID, PMCID). This decoupled approach replaced the earlier hardcoded DOI-only model.
+
+Key modules:
+- `application.py` - Identifier gathering orchestration
+- `normalize.py` - Identifier normalization and validation
+
+### arXiv (`app/services/arxiv/`)
+
+Typed API client with global DB-backed throttle, query cache, and in-flight request coalescing.
+
+Key modules:
+- `client.py` - HTTP client for arXiv export API
+- `gateway.py` - Gateway with advisory lock, caching, and coalescing
+- `cache.py` - Query cache with TTL and max-entry pruning
+- `rate_limit.py` - Global rate limiter via `arxiv_runtime_state` table
+- `guards.py` - Load-shedding guards (skip when DOI/arXiv evidence exists)
+- `parser.py` - Atom XML response parser
+
+Safety features:
+- Requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`)
+- Identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + max-entry pruning
+- Concurrent identical misses are coalesced in-process (one outbound call serves all waiters)
+
+### Crossref (`app/services/crossref/`)
+
+DOI lookup via Crossref REST API with bounded pacing and configurable batch limits.
+
+### Unpaywall (`app/services/unpaywall/`)
+
+Open-access PDF resolution via Unpaywall API, with HTML-based PDF link discovery as a fallback.
+
+Key modules:
+- `application.py` - Unpaywall service facade
+- `pdf_discovery.py` - HTML page scraping for PDF link candidates
+
+### OpenAlex (`app/services/openalex/`)
+
+Metadata matching via OpenAlex API for supplementary identifier resolution.
+
+Key modules:
+- `client.py` - OpenAlex API client
+- `matching.py` - Fuzzy title/author matching
+
+### Runs (`app/services/runs/`)
+
+Run history tracking and continuation queue operations.
+
+Key modules:
+- `application.py` - Run lifecycle management
+- `queue_service.py` - Continuation queue operations (retry, drop, clear)
+- `queue_queries.py` - Queue item queries
+
+### Portability (`app/services/portability/`)
+
+Import/export workflows for scholar data with full publication state preservation.
+
+Key modules:
+- `application.py` - Import/export orchestration
+- `exporting.py` - Scholar export serialization
+- `publication_import.py` - Publication import with deduplication
+- `scholar_import.py` - Scholar import with link reconstruction
+- `normalize.py` - Payload normalization and validation
+
+### Database Operations (`app/services/dbops/`)
+
+Integrity checking, link repair, and near-duplicate repair operations exposed via admin API and CLI scripts.
+
+Key modules:
+- `__init__.py` - `collect_integrity_report`, `run_publication_link_repair`
+- `near_duplicate_repair.py` - Near-duplicate publication detection and merging
+
+## Data Integration Flow
 
 ```mermaid
 graph TD
     UI[Frontend Vue App] --> API[FastAPI Backend]
-    API --> Scheduler[Background Celery/Async Scheduler]
+    API --> Scheduler[Background Async Scheduler]
-    
+
     Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
     Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
-    
+
     IdentifierModule -->|Search arXiv API| API1(arXiv)
     IdentifierModule -->|Search Crossref API| API2(Crossref)
-    
+
     IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)]
-    
+
     Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline]
     DB --> |Identified DOIs| PDFPipeline
     PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API)
-    
+
     Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker]
-    PDFWorker --> |Store PDF Metadata| DB 
+    PDFWorker --> |Store PDF Metadata| DB
 ```
 
-### Identifier Engine Philosophy
+## API Layer
-The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.
 
-### arXiv Safety and Efficiency
+Routes live in `app/api/routers/`. All responses under `/api/v1` use a strict envelope format. See [API Reference](../reference/api.md) for the full contract.
-- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`).
+
-- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning.
+## Middleware Stack
-- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
+
-- structured logs provide auditability for scheduling/completion/cooldown/cache behavior.
+Applied in `app/main.py`:
+
+1. **CSRF Protection** - Token-based CSRF validation
+2. **Session Middleware** - Cookie-based session management (itsdangerous signing)
+3. **Request Logging** - Structured request/response logging with configurable skip paths
+4. **Security Headers** - Configurable HTTP security headers and CSP

docs/developer/contributing.md

@@ -1,24 +1,78 @@
+---
+title: Contributing
+sidebar_position: 4
+---
+
 # Contributing
 
-## Scope
+## PR Process
-This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe.
 
-## Essential-File Policy
+1. Create a feature branch from `main`.
-Commit only source-of-truth files required to build, run, test, or document the app.
+2. Make your changes following the code standards below.
+3. Run tests inside the container (see [Testing](testing.md)).
+4. Run `ruff check .` and `mypy app/` to catch lint and type errors.
+5. Open a pull request with a clear title and description.
 
-Do not commit generated or local-only artifacts, including:
+## Commit Conventions
-- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`
-- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/`
-- coverage outputs (`.coverage`, `htmlcov/`)
-- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`)
-- local probe/scratch material (`planning/`)
 
-CI enforces this with `scripts/check_no_generated_artifacts.sh`.
+This project uses [Conventional Commits](https://www.conventionalcommits.org/) with [python-semantic-release](https://python-semantic-release.readthedocs.io/) for automated versioning.
 
-## Merge Checklist
+Commit message format:
-- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas.
+
-- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed).
+```
-- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`).
+<type>(<scope>): <description>
-- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change.
+
-- [ ] `README.md`, `.env.example`, and deployment notes stay aligned.
+[optional body]
-- [ ] `scripts/check_no_generated_artifacts.sh` passes locally.
+```
+
+Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`.
+
+Examples:
+- `feat(scholars): add bulk import from CSV`
+- `fix(ingestion): handle empty citation blocks`
+- `docs: update configuration reference`
+
+## Code Standards
+
+### Function Length
+
+Maximum 50 lines per function. Break complex logic into small, testable, single-responsibility functions.
+
+### DRY
+
+Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
+
+### Negative Space Programming
+
+Use explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
+
+### Cyclomatic Complexity
+
+Flatten logic. Use early returns and guard clauses instead of deep nesting. No magic numbers.
+
+### Domain Service Boundaries
+
+All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited. Each domain owns its own application service, types, and helpers.
+
+### API Envelope
+
+All `/api/v1` responses use the strict envelope format:
+
+- Success: `{"data": ..., "meta": {"request_id": "..."}}`
+- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
+
+### Data Isolation
+
+- Scholar tracking is user-scoped.
+- Publications are global, deduplicated records.
+- Read/favorite/visibility state lives on scholar-publication link rows.
+
+### Scrape Safety
+
+Rate limits and cooldowns are immutable constraints. They prevent IP bans and must not be optimized away or set to zero.
+
+## UI Standards
+
+- Integrate Tailwind with the preset theming system (`frontend/src/theme/presets/`).
+- Every UI element must have a clear purpose.
+- Clarity through both styling and language.

docs/developer/documentation-standards.md

@@ -1,26 +0,0 @@
-# Documentation Standards
-
-This project keeps docs organized by audience and document type.
-
-## Audience Split
-
-- `user/`: onboarding and usage tasks for self-hosted users.
-- `operations/`: runbooks and checklists for production operation.
-- `developer/`: architecture, contribution workflow, and implementation guides.
-- `reference/`: stable contracts (API, env variables, configuration behavior).
-
-## Document Quality Rules
-
-- One primary audience per page.
-- Task pages must include prerequisites, exact commands, and verification steps.
-- Reference pages should be contract-first and avoid procedural noise.
-- Runbooks should include recovery steps and rollback/safety notes.
-- Prefer concise pages with strong cross-links over long mixed-purpose pages.
-
-## Required Top-Level Entrypoints
-
-- `index.md`: user/developer/operator navigation hub.
-- `user/getting-started.md`: first-run path.
-- `developer/local-development.md`: contributor setup and validation path.
-- `operations/overview.md`: operational playbook index.
-- `reference/overview.md`: contract index.

docs/developer/frontend-theme-inventory.md

@@ -1,89 +1,105 @@
-# Theme Inventory (Phase 0)
+---
+title: Frontend Theme Inventory
+sidebar_position: 6
+---
 
-This file captures the semantic color system baseline for the theme refactor.
+# Frontend Theme Inventory
 
 ## Token Domains
 
-- `scale`:
+The theme system uses semantic tokens organized into domains:
-  - `brand` (`50..950`)
-  - `info` (`50..950`)
-  - `success` (`50..950`)
-  - `warning` (`50..950`)
-  - `danger` (`50..950`)
-- `surface`:
-  - `app`
-  - `nav`
-  - `nav_active`
-  - `card`
-  - `card_muted`
-  - `table`
-  - `table_header`
-  - `input`
-  - `overlay`
-- `text`:
-  - `primary`
-  - `secondary`
-  - `muted`
-  - `inverse`
-  - `link`
-- `border`:
-  - `default`
-  - `strong`
-  - `subtle`
-  - `interactive`
-- `focus`:
-  - `ring`
-  - `ring_offset`
-- `action` variants (`primary`, `secondary`, `ghost`, `danger`):
-  - `bg`
-  - `border`
-  - `text`
-  - `hover_bg`
-  - `hover_border`
-  - `hover_text`
-- `state` variants (`info`, `success`, `warning`, `danger`):
-  - `bg`
-  - `border`
-  - `text`
 
-## Theme Sources
+### Scale Colors
 
-Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`.
+Color ramps from `50` to `950` for each intent:
 
-Current preset files:
+- `brand` - Primary brand color
+- `info` - Informational elements
+- `success` - Success states
+- `warning` - Warning states
+- `danger` - Error/destructive states
 
-- `frontend/src/theme/presets/parchment.js`
+### Surface
-- `frontend/src/theme/presets/lilac.js`
-- `frontend/src/theme/presets/dune.js`
-- `frontend/src/theme/presets/oatmeal.js`
-- `frontend/src/theme/presets/scholarly.json`
-- `frontend/src/theme/presets/graphite.json`
-- `frontend/src/theme/presets/tide.json`
 
-Each preset contains both `light` and `dark` mode token definitions.
+Background colors for major UI areas:
 
-## Adoption Status (Phase 0-3 complete baseline)
+| Token | Usage |
+|-------|-------|
+| `app` | Application background |
+| `nav` | Navigation bar |
+| `nav_active` | Active navigation item |
+| `card` | Card backgrounds |
+| `card_muted` | Muted card variant |
+| `table` | Table body |
+| `table_header` | Table header |
+| `input` | Form inputs |
+| `overlay` | Modal/dialog overlays |
 
-Tokenized foundation components:
+### Text
 
-- `AppButton`
+| Token | Usage |
-- `AppCard`
+|-------|-------|
-- `AppCheckbox`
+| `primary` | Default text |
-- `AppEmptyState`
+| `secondary` | Supporting text |
-- `AppHelpHint`
+| `muted` | Disabled/placeholder text |
-- `AppInput`
+| `inverse` | Text on dark backgrounds |
-- `AppSelect`
+| `link` | Hyperlinks |
-- `AppTable`
-- `AppModal`
-- `AppHeader`
-- `AppNav`
-- `AppAlert`
-- `AppBadge`
-- `RunStatusBadge`
-- `QueueHealthBadge`
 
-Hardening in place:
+### Border
 
-- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs`
+| Token | Usage |
-- CI enforcement step in `frontend-quality` workflow
+|-------|-------|
-- Theme preset integrity tests in `frontend/src/theme/presets.test.ts`
+| `default` | Standard borders |
+| `strong` | Emphasized borders |
+| `subtle` | Light separators |
+| `interactive` | Hover/focus borders |
+
+### Focus
+
+| Token | Usage |
+|-------|-------|
+| `ring` | Focus ring color |
+| `ring_offset` | Focus ring offset color |
+
+### Action Variants
+
+Each action type (`primary`, `secondary`, `ghost`, `danger`) provides:
+
+`bg`, `border`, `text`, `hover_bg`, `hover_border`, `hover_text`
+
+### State Variants
+
+Each state (`info`, `success`, `warning`, `danger`) provides:
+
+`bg`, `border`, `text`
+
+## Theme Presets
+
+Presets are loaded from `frontend/src/theme/presets/*.{json,js}`. Each contains both `light` and `dark` mode token definitions.
+
+Current presets:
+
+| Preset | Format |
+|--------|--------|
+| `parchment` | JS |
+| `lilac` | JS |
+| `dune` | JS |
+| `oatmeal` | JS |
+| `scholarly` | JSON |
+| `graphite` | JSON |
+| `tide` | JSON |
+
+## Tokenized Components
+
+Foundation components using the token system:
+
+- `AppButton`, `AppCard`, `AppCheckbox`, `AppEmptyState`
+- `AppHelpHint`, `AppInput`, `AppSelect`, `AppTable`
+- `AppModal`, `AppHeader`, `AppNav`, `AppAlert`
+- `AppBadge`, `RunStatusBadge`, `QueueHealthBadge`
+
+## Enforcement
+
+- Token policy check: `frontend/scripts/check_theme_tokens.mjs`
+- CI step in `frontend-quality` workflow
+- Preset integrity tests: `frontend/src/theme/presets.test.ts`

docs/developer/ingestion.md

@@ -1,38 +1,89 @@
-# Ingestion System & Backoff Strategies
+---
+title: Ingestion Pipeline
+sidebar_position: 5
+---
 
-The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans.
+# Ingestion Pipeline
 
-## Ingestion Overview
+The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans.
-The underlying ingestion process:
-1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
-2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
-3. Downloads the paginated HTML feed for a user across multiple page iterations.
-4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
 
-## Handling 429 Too Many Requests
+## Pipeline Overview
-Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`). 
 
-Scholarr treats these distinct from random network timeouts.
+1. The scheduler (or a manual trigger) starts a **run** for one or more scholars.
-- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s).
+2. The service connects via HTTPX with strict browser headers.
-- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead.
+3. Paginated HTML feeds are downloaded for each scholar profile.
+4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks.
+5. Publications are fingerprinted and deduplicated against the global store.
+6. External APIs resolve additional identifiers.
+7. The PDF resolution pipeline runs asynchronously for publications with known DOIs.
 
-## Publication Identifiers Loop
+## Rate Limiting & Backoff
-Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
-- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
-- **API Fetching**: Queries secondary bibliographic platforms sequentially:
-  - `export.arxiv.org/api/query` (Queries by Title and Author strings).
-  - `crossref.restful` APIs (Queries by Title and Author strings).
 
-These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.
+### Network Errors
+
+Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s).
+
+### HTTP 429 (Rate Limited)
+
+When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies:
+
+- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3)
+- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s)
+
+This pauses the pipeline gracefully instead of failing the entire run.
+
+### Safety Cooldowns
+
+Threshold-based cooldowns halt all ingestion after repeated failures:
+
+| Threshold | Variable | Default | Cooldown |
+|-----------|----------|---------|----------|
+| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) |
+| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) |
+
+## Continuation Queue
+
+Multi-page ingestion uses a continuation queue to spread load over time:
+
+- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`)
+- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s)
+- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s)
+- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6)
+
+Each continuation item is re-enqueued with exponential backoff.
+
+## Identifier Resolution
+
+After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers:
+
+1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches.
+2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings.
+3. **Crossref API** - Queries Crossref REST API by title and author strings.
+
+Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage.
+
+## PDF Resolution
+
+Publications with resolved DOIs enter the PDF resolution pipeline:
+
+1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs.
+2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates.
+3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly.
+
+Auto-retry is configured via `PDF_AUTO_RETRY_*` variables.
 
 ## arXiv Request Controls
-- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
-- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
-- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
-- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
 
-## arXiv Observability Events
+- **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
-- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
+- **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`.
-- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
+- **In-flight coalescing**: Duplicate concurrent misses join one outbound request.
-- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown.
+- **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
-- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.
+
+### Observability Events
+
+| Event | Description |
+|-------|-------------|
+| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. |
+| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. |
+| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. |
+| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. |

docs/developer/local-development.md

@@ -1,52 +1,100 @@
-# Developer Local Development
+---
+title: Local Development
+sidebar_position: 3
+---
 
-## Start the Dev Stack
+# Local Development
+
+## Prerequisites
+
+- Docker and Docker Compose v2+
+- Python 3.12+ (for IDE support and local linting)
+- Node.js 20+ (for frontend development)
+
+## Starting the Dev Stack
+
+The development compose file overlays the production config with hot-reload and a separate Vite dev server:
 
 ```bash
-docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
+docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
 ```
 
-Open:
+This starts three services:
-- API: `http://localhost:8000`
-- Frontend dev server: `http://localhost:5173`
 
-Stop:
+| Service | Port | Description |
+|---------|------|-------------|
+| `db` | 5432 (internal) | PostgreSQL 15 |
+| `app` | 8000 | FastAPI backend with `APP_RELOAD=1` |
+| `frontend` | 5173 | Vite dev server proxying API calls to `app:8000` |
+
+### Dev-Specific Overrides
+
+The `docker-compose.dev.yml` file applies these changes:
+
+- **`app`**: Uses `scholarr-dev:local` image built from the `dev` stage. Mounts the project root as `/app` for hot reload. Disables `SESSION_COOKIE_SECURE` and the built frontend.
+- **`frontend`**: Node 20 container running `npm install && npm run dev`. Mounts `./frontend` with a named volume for `node_modules`. Uses polling for file watching (`CHOKIDAR_USEPOLLING=1`).
+
+## Environment Setup
 
 ```bash
-docker compose -f docker-compose.yml -f docker-compose.dev.yml down
+cp .env.example .env
 ```
 
-## Backend Validation
+Set at minimum:
 
 ```bash
-docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit
+POSTGRES_PASSWORD=localdev
-docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration
+SESSION_SECRET_KEY=local-dev-secret-at-least-32-characters
+SESSION_COOKIE_SECURE=0
 ```
 
-## Frontend Validation
+## Running Tests
+
+All tests run inside containers:
 
 ```bash
-cd frontend
+# Run unit tests (default: excludes integration markers)
-npm install
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
-npm run typecheck
+  python -m pytest
-npm run test:run
+
-npm run build
+# Run integration tests
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest -m integration
+
+# Run a specific test file
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest tests/unit/test_fingerprints.py -v
 ```
 
-## Repository Gates
+See [Testing](testing.md) for markers, fixtures, and conventions.
+
+## Linting and Type Checking
 
 ```bash
-python3 scripts/check_frontend_api_contract.py
+# Ruff linting
-python3 scripts/check_env_contract.py
+ruff check .
-./scripts/check_no_generated_artifacts.sh
+
+# Ruff formatting check
+ruff format --check .
+
+# Mypy type checking
+mypy app/
 ```
 
-## Docs Site (Contributor Workflow)
+Ruff config is in `pyproject.toml`: target Python 3.12, line length 120, rules `E F W I UP B SIM RUF`.
 
-Docs tooling is colocated in `docs/website/`:
+## Database Migrations
+
+Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. To run manually:
 
 ```bash
-cd docs/website
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
-npm install
+  alembic upgrade head
-npm run build
+```
+
+To create a new migration:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  alembic revision --autogenerate -m "description of change"
 ```

docs/developer/overview.md

@@ -1,11 +1,65 @@
-# Developer Documentation
+---
+title: Developer Overview
+sidebar_position: 1
+---
 
-Use this section if you are contributing code or running full quality gates.
+# Developer Overview
 
-- [Documentation Standards](./documentation-standards.md)
+Scholarr is a Python 3.12+ FastAPI backend with an async SQLAlchemy ORM layer, PostgreSQL database, and a Vue 3 + TypeScript + Vite frontend.
-- [Local Development](./local-development.md)
+
-- [Architecture Boundaries](./architecture.md)
+## Stack
-- [Ingestion API & Backoff Strategies](./ingestion.md)
+
-- [API Contracts](./api-contract.md)
+| Layer | Technology |
-- [Contributing](./contributing.md)
+|-------|------------|
-- [Frontend Theme Inventory](./frontend-theme-inventory.md)
+| Backend | Python 3.12+, FastAPI, SQLAlchemy 2.0 (async/asyncpg), Alembic |
+| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS |
+| Database | PostgreSQL 15 |
+| Infrastructure | Multi-stage Docker, Docker Compose |
+| Linting | ruff (E, F, W, I, UP, B, SIM, RUF), mypy |
+| Testing | pytest, pytest-asyncio |
+| Versioning | python-semantic-release, conventional commits |
+
+## Quick Start
+
+```bash
+# Build the dev image and start services
+docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
+
+# Backend: http://localhost:8000 (hot reload enabled)
+# Frontend: http://localhost:5173 (Vite dev server with API proxy)
+```
+
+See [Local Development](local-development.md) for the full setup guide.
+
+## Project Layout
+
+```
+app/
+├── api/routers/          # FastAPI route handlers
+├── auth/                 # Session + CSRF middleware
+├── db/                   # SQLAlchemy models, session factory, migrations
+├── services/             # Domain service modules (see Architecture)
+│   ├── arxiv/            # arXiv API client, cache, rate limiting
+│   ├── crossref/         # Crossref DOI lookups
+│   ├── dbops/            # Database integrity + repair operations
+│   ├── ingestion/        # Run orchestration, scheduler, safety gates
+│   ├── openalex/         # OpenAlex metadata matching
+│   ├── portability/      # Import/export workflows
+│   ├── publication_identifiers/  # Multi-identifier resolution
+│   ├── publications/     # Listing, enrichment, dedup, PDF queue
+│   ├── runs/             # Run history, continuation queue
+│   ├── scholar/          # HTML parser, source fetch adapters
+│   ├── scholars/         # Scholar CRUD, image upload, name search
+│   └── unpaywall/        # Unpaywall PDF discovery
+├── main.py               # App factory, lifespan, middleware stack
+frontend/
+├── src/
+│   ├── components/       # Reusable Vue components
+│   ├── theme/presets/    # Color theme presets (light + dark)
+│   └── ...
+scripts/db/               # Operational database scripts
+tests/
+├── unit/                 # Fast, no-database tests
+├── integration/          # Tests requiring database + services
+└── fixtures/             # Shared test fixtures
+```

docs/developer/testing.md

@@ -0,0 +1,102 @@
+---
+title: Testing
+sidebar_position: 7
+---
+
+# Testing
+
+## Running Tests
+
+All tests must run inside containers:
+
+```bash
+# Unit tests (default: excludes integration markers)
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest
+
+# Integration tests
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest -m integration
+
+# Specific marker
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest -m db
+
+# Verbose output for a specific file
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python -m pytest tests/unit/test_fingerprints.py -v
+```
+
+## Test Configuration
+
+From `pyproject.toml`:
+
+```toml
+[tool.pytest.ini_options]
+addopts = "-q -m \"not integration\" --import-mode=importlib"
+asyncio_mode = "auto"
+testpaths = ["tests"]
+```
+
+- Default run excludes `integration` marked tests
+- Uses `importlib` import mode to resolve module name collisions
+- Async tests run automatically (no `@pytest.mark.asyncio` needed)
+
+## Markers
+
+| Marker | Description |
+|--------|-------------|
+| `integration` | Tests requiring external services (database, network) |
+| `db` | Tests that validate database behavior and constraints |
+| `migrations` | Tests focused on Alembic schema migration correctness |
+| `schema` | Tests focused on multi-tenant schema invariants |
+| `smoke` | Smoke tests for containerized runtime |
+
+## Test Tiers
+
+### Unit Tests (`tests/unit/`)
+
+Fast, no-database tests. Mock external dependencies. These run by default.
+
+Examples:
+- `test_fingerprints.py` - Publication fingerprinting logic
+- `test_scholar_parser.py` - HTML parsing without network calls
+- `test_doi_normalize.py` - DOI normalization rules
+- `test_ingestion_arxiv_rate_limit.py` - Rate limiter behavior
+- `test_publication_pdf_resolution_pipeline.py` - PDF pipeline logic
+
+Domain-specific unit tests are organized under `tests/unit/services/domains/`:
+- `arxiv/` - Cache, client, gateway, guards, parser, rate limit tests
+- `openalex/` - Client and matching tests
+- `publications/` - Dedup tests
+
+### Integration Tests (`tests/integration/`)
+
+Require a running database. Test full request/response flows and data consistency.
+
+Examples:
+- `test_api_v1.py` - API endpoint integration tests
+- `test_db_integrity.py` - Database integrity checks
+- `test_run_lifecycle_consistency.py` - Run state machine transitions
+- `test_deferred_enrichment.py` - Enrichment pipeline with real data
+- `test_fixture_probe_runs.py` - Fixture-based run probes
+
+### Smoke Tests
+
+Marked with `@pytest.mark.smoke`. Validate the containerized runtime starts and serves basic requests.
+
+## Fixtures
+
+Test fixtures live in `tests/fixtures/`:
+
+```
+tests/fixtures/
+└── scholar/
+    ├── profile_ok_amIMrIEAAAAJ.html          # Successful profile HTML
+    └── regression/
+        ├── profile_P1RwlvoAAAAJ.html         # Regression case
+        ├── profile_LZ5D_p4AAAAJ.html         # Regression case
+        └── profile_AAAAAAAAAAAA.html         # Regression case
+```
+
+Scholar HTML fixtures are real Google Scholar profile pages used to test parser robustness against DOM structure changes.

docs/index.md

@@ -1,33 +1,27 @@
 ---
-slug: /
+title: Scholarr Documentation
+sidebar_position: 1
 ---
 
-# scholarr Documentation
+# Scholarr
 
-This documentation is organized for both users and developers.
+Scholarr is a self-hosted academic publication tracker. It monitors Google Scholar profiles, discovers new publications, resolves open-access PDFs, and presents everything through a clean Vue 3 dashboard.
 
-## For Users
+## Quick Links
 
-- [User Overview](./user/overview.md)
+| Section | Description |
-- [Getting Started](./user/getting-started.md)
+|---------|-------------|
+| [User Guide](user/overview.md) | What scholarr does, installation, configuration |
+| [Developer Guide](developer/overview.md) | Architecture, local development, contributing |
+| [Operations](operations/overview.md) | Deployment, database runbook, scrape safety |
+| [Reference](reference/overview.md) | API contract, environment variables, changelog |
 
-## For Operators
+## Key Features
 
-- [Operations Overview](./operations/overview.md)
+- **Scholar Tracking** - Add Google Scholar profiles by ID, URL, or name search
-- [Scrape Safety Runbook](./operations/scrape-safety-runbook.md)
+- **Automated Ingestion** - Background scheduler fetches new publications on a configurable interval
-- [Database Runbook](./operations/database-runbook.md)
+- **Identifier Resolution** - Cross-references arXiv, Crossref, and OpenAlex for DOIs and metadata
-- [Migration Rollout Checklist](./operations/migration-checklist.md)
+- **PDF Discovery** - Resolves open-access PDFs via Unpaywall and arXiv
-
+- **Import/Export** - Portable scholar data with full publication state
-## For Developers
+- **Multi-User** - Session-based auth with admin user management
-
+- **Theming** - Multiple color presets with light/dark mode support
-- [Developer Overview](./developer/overview.md)
-- [Local Development](./developer/local-development.md)
-- [Architecture Boundaries](./developer/architecture.md)
-- [Contributing](./developer/contributing.md)
-- [Frontend Theme Inventory](./developer/frontend-theme-inventory.md)
-
-## Reference
-
-- [Reference Overview](./reference/overview.md)
-- [API Contract](./reference/api-contract.md)
-- [Environment Reference](./reference/environment.md)

docs/operations/arxiv-runbook.md

@@ -1,46 +1,80 @@
+---
+title: arXiv Runbook
+sidebar_position: 5
+---
+
 # arXiv Operations Runbook
 
 Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly.
 
-## Signals To Check
+## Signals to Check
-- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
+
+- Logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
 - `arxiv_runtime_state` row for cooldown and next-allowed timestamps
 - `arxiv_query_cache_entries` size and expiry churn
 
 ## Event Field Guide
-- `wait_seconds`: enforced pre-request delay from the global limiter.
+
-- `status_code`: upstream response code from arXiv.
+| Field | Description |
-- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429.
+|-------|-------------|
-- `source_path`: caller path (`search` or `lookup_ids`).
+| `wait_seconds` | Enforced pre-request delay from the global limiter |
+| `status_code` | Upstream response code from arXiv |
+| `cooldown_remaining_seconds` | Remaining cooldown when blocked or after 429 |
+| `source_path` | Caller path (`search` or `lookup_ids`) |
 
 ## Quick SQL Checks
+
+### Runtime State
+
 ```sql
 SELECT state_key, next_allowed_at, cooldown_until, updated_at
 FROM arxiv_runtime_state;
 ```
 
+### Cache Status
+
 ```sql
-SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry
+SELECT count(*) AS cache_rows,
+       min(expires_at) AS earliest_expiry,
+       max(expires_at) AS latest_expiry
 FROM arxiv_query_cache_entries;
 ```
 
 ## Common Scenarios
-1. Repeated `arxiv.cooldown_activated` events:
+
-- Confirm recent `429` statuses in `arxiv.request_completed`.
+### 1. Repeated `arxiv.cooldown_activated` Events
-- Reduce caller pressure (check new title-quality/identifier guards are active).
+
+- Confirm recent `429` statuses in `arxiv.request_completed` logs.
+- Reduce caller pressure (check title-quality/identifier guards are active).
 - Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict.
 
-2. High request latency with few completions:
+### 2. High Request Latency with Few Completions
+
 - Inspect `wait_seconds` in `arxiv.request_scheduled`.
 - Verify only one process path is repeatedly hitting arXiv (`source_path`).
 - Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears).
 
-3. Low cache effectiveness:
+### 3. Low Cache Effectiveness
+
 - Validate normalized query behavior and caller churn.
 - Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads.
 - Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed.
 
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `ARXIV_ENABLED` | `1` | Enable arXiv lookups |
+| `ARXIV_TIMEOUT_SECONDS` | `3.0` | Request timeout |
+| `ARXIV_MIN_INTERVAL_SECONDS` | `4.0` | Min interval between requests |
+| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | `60.0` | Cooldown after 429 |
+| `ARXIV_DEFAULT_MAX_RESULTS` | `3` | Max results per query |
+| `ARXIV_CACHE_TTL_SECONDS` | `900` | Cache TTL (15 min) |
+| `ARXIV_CACHE_MAX_ENTRIES` | `512` | Max cached queries |
+| `ARXIV_MAILTO` | *(empty)* | Contact email for API headers |
+
 ## Safe Recovery
+
 1. Pause automated ingestion if rate-limit storms persist.
 2. Let cooldown expire naturally; avoid manual burst retries.
 3. Resume and monitor event rates before restoring full load.

docs/operations/database-runbook.md

@@ -1,100 +1,195 @@
-# Database Operations Runbook
+---
+title: Database Runbook
+sidebar_position: 3
+---
 
-This runbook defines backup, restore, and verification procedures for the
+# Database Runbook
-PostgreSQL database used by `scholarr`.
 
-## Objectives
+## Backup
 
-- Keep data loss bounded via scheduled backups.
+Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service.
-- Provide repeatable restore procedures.
-- Verify backups by running regular restore drills.
 
-Recommended starting targets:
+### Custom-Format Backup (Default)
-
-- `RPO`: 24 hours (maximum acceptable data loss).
-- `RTO`: 60 minutes (maximum acceptable restore time).
-
-## Backup Strategy
-
-Use logical backups from the running `db` compose service.
-
-- Custom-format backup (recommended for restore flexibility):
 
 ```bash
 scripts/db/backup_full.sh
 ```
 
-- Plain SQL backup:
+Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`).
+
+### Plain SQL Backup
 
 ```bash
 scripts/db/backup_full.sh --plain
 ```
 
-Optional environment variables:
+Creates a `.sql` file.
 
-- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
+### Options
-- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
-- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
 
-## Restore Strategy
+| Option | Description |
+|--------|-------------|
+| `--plain` | Write plain SQL instead of custom-format dump |
 
-Restore from a `.dump` (custom format) or `.sql` file into the running `db`
+### Environment Variables
-compose service.
 
-- Restore without schema wipe:
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `BACKUP_DIR` | `<repo>/backups` | Destination directory |
+| `BACKUP_PREFIX` | `scholarr` | File prefix |
+| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` |
+
+## Restore
+
+Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service.
+
+### Restore a Custom-Format Dump
 
 ```bash
-scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
+scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump
 ```
 
-- Restore with full `public` schema reset:
+### Restore with Schema Wipe
 
 ```bash
-scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
+scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public
 ```
 
-## Safety Checklist Before Restore
+This drops and recreates the `public` schema before restoring. Use with caution.
 
-1. Pause writes (stop app/scheduler or set maintenance mode).
+### Options
-2. Create a fresh pre-restore backup.
-3. Validate target dump checksum/size.
-4. Confirm operator, scope, and rollback plan.
 
-## Post-Restore Verification
+| Option | Description |
+|--------|-------------|
+| `--file <path>` | Required. Path to `.dump` or `.sql` backup file |
+| `--wipe-public` | Drop and recreate `public` schema before restore |
 
-1. Run migrations head check.
+## Integrity Checks
-2. Run health endpoint checks.
+
-3. Verify table row counts for core tables:
+Use `scripts/db/check_integrity.py` to run database integrity checks.
-   - `users`
+
-   - `scholar_profiles`
+### Run Inside Container
-   - `publications`
-   - `scholar_publications`
-   - `crawl_runs`
-4. Run integrity report and require zero failures:
 
 ```bash
-python3 scripts/db/check_integrity.py
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python scripts/db/check_integrity.py
 ```
 
-5. Run API smoke checks for scholars/publications/runs pages.
+### With Strict Warnings
-
-## Data Cleanup and Repair
-
-Use the audited repair job for scholar-publication relinking cleanup:
 
 ```bash
-python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python scripts/db/check_integrity.py --strict-warnings
 ```
 
-Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
+Returns non-zero exit code if any warning is present.
 
-If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout.
+### Output Format
 
-## Restore Drill Cadence
+JSON report:
 
-- Run at least one full restore drill per month.
+```json
-- Record:
+{
-  - backup artifact used,
+  "status": "passed",
-  - restore start/end timestamps,
+  "failures": [],
-  - issues encountered,
+  "warnings": []
-  - achieved `RTO`.
+}
+```
+
+Exit codes:
+- `0` - All checks passed
+- `1` - Check failure
+- `2` - Warnings present (with `--strict-warnings`)
+
+### Via Admin API
+
+```
+GET /api/v1/admin/db/integrity
+```
+
+Returns the same integrity report through the API (admin auth required).
+
+## Publication Link Repair
+
+Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging.
+
+### Dry Run (Default)
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python scripts/db/repair_publication_links.py --user-id 1
+```
+
+### Apply Changes
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
+  python scripts/db/repair_publication_links.py --user-id 1 --apply \
+  --requested-by "admin@example.com"
+```
+
+### Options
+
+| Option | Description |
+|--------|-------------|
+| `--user-id <int>` | Required. Target user ID |
+| `--scholar-profile-id <int>` | Optional. Filter by scholar profile ID (repeatable) |
+| `--apply` | Apply changes (default is dry-run) |
+| `--gc-orphan-publications` | Delete publications with zero links after cleanup |
+| `--requested-by <string>` | Operator identifier for audit logs |
+
+### Via Admin API
+
+```
+POST /api/v1/admin/db/repairs/publication-links
+```
+
+## Near-Duplicate Repair
+
+Detect and merge near-duplicate publications via the admin API:
+
+```
+POST /api/v1/admin/db/repairs/publication-near-duplicates
+```
+
+## PDF Queue Management
+
+### List Queue
+
+```
+GET /api/v1/admin/db/pdf-queue
+```
+
+### Requeue Single Item
+
+```
+POST /api/v1/admin/db/pdf-queue/{id}/requeue
+```
+
+### Bulk Requeue
+
+```
+POST /api/v1/admin/db/pdf-queue/requeue-all
+```
+
+## Migration Procedures
+
+Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`.
+
+To run manually:
+
+```bash
+docker compose exec app alembic upgrade head
+```
+
+To check current migration status:
+
+```bash
+docker compose exec app alembic current
+```
+
+Recommended procedure for production:
+1. Backup the database before upgrading.
+2. Pull the new image.
+3. Start the container; migrations run on startup.
+4. Verify via `GET /healthz` and check logs for migration output.

docs/operations/deployment.md

@@ -0,0 +1,117 @@
+---
+title: Deployment
+sidebar_position: 2
+---
+
+# Deployment
+
+## Production Docker Compose
+
+The default `docker-compose.yml` runs two services:
+
+| Service | Image | Description |
+|---------|-------|-------------|
+| `db` | `postgres:15-alpine` | PostgreSQL database with persistent volume |
+| `app` | `justinzeus/scholarr:latest` | FastAPI application with embedded frontend |
+
+### Start
+
+```bash
+docker compose up -d
+```
+
+### Stop
+
+```bash
+docker compose down
+```
+
+### Update
+
+```bash
+docker compose pull
+docker compose up -d
+```
+
+### View Logs
+
+```bash
+docker compose logs -f app
+docker compose logs -f db
+```
+
+## Required Environment Variables
+
+These must be set in `.env` or the shell environment:
+
+```bash
+POSTGRES_PASSWORD=<secure-password>
+SESSION_SECRET_KEY=<random-string-32-chars-minimum>
+```
+
+## Volumes
+
+| Volume | Mount | Description |
+|--------|-------|-------------|
+| `postgres_data` | `/var/lib/postgresql/data` | Database files |
+| `scholar_uploads` | `/var/lib/scholarr/uploads` | Scholar profile images |
+
+## Health Checks
+
+### Database
+
+```yaml
+healthcheck:
+  test: ["CMD-SHELL", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"]
+  interval: 5s
+  timeout: 5s
+  retries: 20
+```
+
+### Application
+
+```yaml
+healthcheck:
+  test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"]
+  interval: 10s
+  timeout: 5s
+  retries: 12
+```
+
+The app service depends on `db` with `condition: service_healthy`, so it waits for the database to be ready.
+
+## Database Readiness Wait
+
+The app has built-in database readiness polling:
+
+- `DB_WAIT_TIMEOUT_SECONDS` (default: 60) - Max wait time
+- `DB_WAIT_INTERVAL_SECONDS` (default: 2) - Poll interval
+
+## Auto-Migration
+
+Set `MIGRATE_ON_START=1` (default) to run Alembic migrations automatically on startup.
+
+## Scaling Considerations
+
+- Run a single `app` instance to avoid scheduler conflicts (the scheduler is process-local).
+- arXiv requests are globally serialized via a PostgreSQL advisory lock, so multiple instances safely share the rate limiter.
+- Database pool defaults: 5 base connections + 10 overflow. Adjust `DATABASE_POOL_SIZE` and `DATABASE_POOL_MAX_OVERFLOW` for higher loads.
+
+## Admin Bootstrap
+
+For first-time setup without manual database access:
+
+```bash
+BOOTSTRAP_ADMIN_ON_START=1
+BOOTSTRAP_ADMIN_EMAIL=admin@example.com
+BOOTSTRAP_ADMIN_PASSWORD=<secure-password>
+```
+
+Set `BOOTSTRAP_ADMIN_FORCE_PASSWORD=1` to reset an existing admin password.
+
+## Security Hardening
+
+- Set `SESSION_COOKIE_SECURE=1` when serving over HTTPS.
+- Enable HSTS: `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED=1`.
+- Review CSP policy in `SECURITY_CSP_POLICY` for your domain.
+- Set `UNPAYWALL_EMAIL` and `ARXIV_MAILTO` for polite API pool access.

docs/operations/migration-checklist.md

@@ -1,48 +0,0 @@
-# Migration Checklist
-
-Use this checklist for every schema/data migration.
-
-## 1. Design Review
-
-- Define change type: `expand`, `backfill`, `contract`.
-- Document expected lock behavior and index impact.
-- Confirm backward compatibility with currently deployed app version.
-- Define rollback strategy before implementation.
-
-## 2. Pre-Migration Controls
-
-- Capture a fresh backup before applying migration.
-- Confirm migration head revision and working tree cleanliness.
-- Prepare validation queries for new/changed tables and indexes.
-- Identify high-risk tables (large row count, hot write paths).
-
-## 3. Implementation Standards
-
-- Keep migrations idempotent when feasible.
-- Prefer additive steps first (`nullable`, new index, new table).
-- For destructive changes, separate into later contract migration.
-- Avoid large blocking rewrites in a single deployment step.
-
-## 4. Verification
-
-- Apply migration in staging against production-like snapshot.
-- Verify:
-  - expected tables/columns/indexes,
-  - app startup and health endpoint,
-  - affected API flows,
-  - data consistency queries.
-
-## 5. Rollout and Recovery
-
-- Apply migration during planned window.
-- Monitor logs/errors and DB metrics during rollout.
-- If rollback needed:
-  - follow downgrade/recovery runbook,
-  - restore from backup if downgrade is unsafe,
-  - document incident timeline.
-
-## 6. Post-Migration Tasks
-
-- Update `README.md` / `.env.example` / ops docs.
-- Add/update integration tests for new schema assumptions.
-- Record migration notes in changelog.

docs/operations/overview.md

@@ -1,8 +1,30 @@
-# Operations Documentation
+---
+title: Operations Overview
+sidebar_position: 1
+---
 
-Use this section for production operations and incident/runbook workflows.
+# Operations Overview
 
-- [Scrape Safety Runbook](./scrape-safety-runbook.md)
+This section covers production deployment, database administration, and scrape safety operations.
-- [arXiv Runbook](./arxiv-runbook.md)
+
-- [Database Runbook](./database-runbook.md)
+## Quick Links
-- [Migration Rollout Checklist](./migration-checklist.md)
+
+| Guide | Description |
+|-------|-------------|
+| [Deployment](deployment.md) | Production Docker setup, scaling, health checks |
+| [Database Runbook](database-runbook.md) | Backup, restore, integrity checks, repair procedures |
+| [Scrape Safety Runbook](scrape-safety-runbook.md) | Rate limiting, cooldowns, CAPTCHA handling |
+| [arXiv Runbook](arxiv-runbook.md) | arXiv rate limits, cache tuning, query patterns |
+
+## Health Check
+
+The app exposes `GET /healthz` for container orchestration:
+
+```bash
+curl -fsS http://localhost:8000/healthz
+```
+
+Docker Compose healthcheck config:
+- Interval: 10s
+- Timeout: 5s
+- Retries: 12

docs/operations/scrape-safety-runbook.md

@@ -1,74 +1,76 @@
+---
+title: Scrape Safety Runbook
+sidebar_position: 4
+---
+
 # Scrape Safety Runbook
 
-This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
+Operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
 
 ## Key Signals
 
 Use structured log `event` fields (recommended with `LOG_FORMAT=json`):
 
-- `ingestion.safety_policy_blocked_run_start`
+| Event | Description |
-- `ingestion.safety_cooldown_entered`
+|-------|-------------|
-- `ingestion.safety_cooldown_cleared`
+| `ingestion.safety_policy_blocked_run_start` | Run blocked by safety policy |
-- `ingestion.alert_blocked_failure_threshold_exceeded`
+| `ingestion.safety_cooldown_entered` | Cooldown activated |
-- `ingestion.alert_network_failure_threshold_exceeded`
+| `ingestion.safety_cooldown_cleared` | Cooldown expired |
-- `ingestion.alert_retry_scheduled_threshold_exceeded`
+| `ingestion.alert_blocked_failure_threshold_exceeded` | Blocked failure threshold tripped |
-- `api.runs.manual_blocked_policy`
+| `ingestion.alert_network_failure_threshold_exceeded` | Network failure threshold tripped |
-- `api.runs.manual_blocked_safety`
+| `ingestion.alert_retry_scheduled_threshold_exceeded` | Retry threshold tripped |
-- `scheduler.run_skipped_safety_cooldown`
+| `api.runs.manual_blocked_policy` | Manual run blocked by policy |
-- `scheduler.run_skipped_safety_cooldown_precheck`
+| `api.runs.manual_blocked_safety` | Manual run blocked by safety cooldown |
-- `scheduler.queue_item_deferred_safety_cooldown`
+| `scheduler.run_skipped_safety_cooldown` | Scheduled run skipped due to cooldown |
+| `scheduler.run_skipped_safety_cooldown_precheck` | Scheduler precheck blocked |
+| `scheduler.queue_item_deferred_safety_cooldown` | Queue item deferred due to cooldown |
 
-Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules.
+Each event includes metric-style fields (`metric_name`, `metric_value`) for log-based alert rules.
 
 ## Recommended Alert Rules
 
-- Cooldown enters:
+- **Cooldown enters**: Trigger on `event=ingestion.safety_cooldown_entered`
-  - Trigger on `event=ingestion.safety_cooldown_entered`.
+- **Repeated start blocks**: High rate of `event=api.runs.manual_blocked_safety`
-- Repeated start blocks:
+- **Threshold trips**: Any of the `*_threshold_exceeded` events
-  - Trigger on high rate of `event=api.runs.manual_blocked_safety`.
+- **Scheduler pressure**: Sustained `scheduler.queue_item_deferred_safety_cooldown`
-- Threshold trips:
-  - Trigger on any of:
-    - `ingestion.alert_blocked_failure_threshold_exceeded`
-    - `ingestion.alert_network_failure_threshold_exceeded`
-    - `ingestion.alert_retry_scheduled_threshold_exceeded`
-- Scheduler pressure:
-  - Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`.
 
 ## If Your IP Appears Blocked
 
-Symptoms:
+### Symptoms
 
-- cooldown reason `blocked_failure_threshold_exceeded`
+- Cooldown reason: `blocked_failure_threshold_exceeded`
-- parse state `blocked_or_captcha`
+- Parse state: `blocked_or_captcha`
-- redirects toward Google account sign-in flows
+- Redirects toward Google account sign-in flows
 
-Actions:
+### Actions
 
 1. Stop manual retries immediately.
 2. Let cooldown expire; do not spam retriggers.
 3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values.
 4. Reduce concurrency pressure (keep one scheduler instance).
-5. Keep name-search disabled/WIP for now if login-gated responses persist.
+5. Keep name-search disabled if login-gated responses persist.
 6. Resume with a small monitored run and verify blocked rate drops.
 
-Avoid:
+### Avoid
 
-- aggressive rapid retries
+- Aggressive rapid retries
-- rotating through risky scraping patterns that increase challenge rates
+- Rotating through risky scraping patterns that increase challenge rates
-- bypass/captcha-solving workflows that may violate source platform rules
+- Bypass/CAPTCHA-solving workflows that may violate source platform rules
 
 ## Environment Controls
 
 Policy floors and safety controls:
 
-- `INGESTION_MIN_REQUEST_DELAY_SECONDS`
+| Variable | Default | Description |
-- `INGESTION_MIN_RUN_INTERVAL_MINUTES`
+|----------|---------|-------------|
-- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`
+| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | `2` | Floor delay between requests |
-- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`
+| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | `15` | Minimum time between runs |
-- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`
+| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `1` | Blocked failures before alert |
-- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`
+| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `2` | Network failures before alert |
-- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`
+| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `3` | Retries before alert |
-- `INGESTION_MANUAL_RUN_ALLOWED`
+| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `1800` | Cooldown after blocked threshold (30 min) |
-- `INGESTION_AUTOMATION_ALLOWED`
+| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `900` | Cooldown after network threshold (15 min) |
+| `INGESTION_MANUAL_RUN_ALLOWED` | `1` | Enable manual runs |
+| `INGESTION_AUTOMATION_ALLOWED` | `1` | Enable automated runs |
 
 Apply stricter values first, then relax slowly only after sustained healthy runs.

docs/reference/api-contract.md

@@ -1,31 +0,0 @@
-# API Contract
-
-## Envelope Invariant
-
-All API responses under `/api/v1` use one of these envelopes:
-
-- Success: `{"data": ..., "meta": {"request_id": "..."}}`
-- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
-
-`meta.request_id` must be present for success and error responses.
-
-Binary media assets are served outside `/api/v1` (for example, `GET /scholar-images/{scholar_profile_id}/upload`).
-
-## Publications Semantics
-
-- `GET /api/v1/publications` supports `mode=all|unread|latest`.
-- `mode=new` is currently accepted as a compatibility alias for `latest`.
-- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
-- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`.
-- `unread` represents read-state (`is_read=false`).
-- `latest` represents discovery-state (`first seen in the latest completed run`).
-
-Publication payloads expose:
-- `pub_url` (canonical scholar detail URL)
-- `doi` (normalized DOI)
-- `pdf_url` (resolved OA PDF when available)
-
-## Scholar Portability
-
-- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state.
-- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication.

docs/reference/api.md

@@ -0,0 +1,177 @@
+---
+title: API Contract
+sidebar_position: 2
+---
+
+# API Contract
+
+## Envelope Invariant
+
+All API responses under `/api/v1` use one of these envelopes:
+
+### Success
+
+```json
+{
+  "data": "...",
+  "meta": {
+    "request_id": "..."
+  }
+}
+```
+
+### Error
+
+```json
+{
+  "error": {
+    "code": "...",
+    "message": "...",
+    "details": "..."
+  },
+  "meta": {
+    "request_id": "..."
+  }
+}
+```
+
+`meta.request_id` is present on both success and error responses.
+
+### Binary Assets
+
+Binary media assets are served outside `/api/v1`:
+
+```
+GET /scholar-images/{scholar_profile_id}/upload
+```
+
+## DTO Structure
+
+Scholarr uses strictly typed Pydantic V2 models serialized through OpenAPI v3 via FastAPI.
+
+- `PublicationListItem` exposes a `.display_identifier` property that resolves the highest-confidence identifier regardless of backend origin, rather than a hardcoded `.doi`.
+- Frontend TypeScript types are compiled from the OpenAPI spec to ensure type safety across the stack.
+
+## Endpoints
+
+### Auth
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/api/v1/auth/login` | Login (rate-limited sliding window) |
+| `GET` | `/api/v1/auth/me` | Current session user |
+| `GET` | `/api/v1/auth/csrf` | Bootstrap CSRF token |
+| `POST` | `/api/v1/auth/change-password` | Change password |
+| `POST` | `/api/v1/auth/logout` | Logout |
+
+### Scholars
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/scholars` | List tracked scholars |
+| `POST` | `/api/v1/scholars` | Create scholar (auto-enqueue, metadata hydration) |
+| `GET` | `/api/v1/scholars/search` | Search author candidates by name |
+| `PATCH` | `/api/v1/scholars/{id}/toggle` | Toggle enabled status |
+| `DELETE` | `/api/v1/scholars/{id}` | Delete scholar |
+| `PUT` | `/api/v1/scholars/{id}/image/url` | Update image URL |
+| `POST` | `/api/v1/scholars/{id}/image/upload` | Upload image |
+| `DELETE` | `/api/v1/scholars/{id}/image` | Clear image customization |
+
+### Scholar Portability
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/scholars/export` | Export tracked scholars and publication link state |
+| `POST` | `/api/v1/scholars/import` | Import scholars with global publication deduplication |
+
+The export payload includes scholar metadata, tracked publication data, and link state (read/unread, favorites). Import preserves global deduplication.
+
+### Publications
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/publications` | List publications (filtered, paginated) |
+| `POST` | `/api/v1/publications/mark-all-read` | Mark all as read |
+| `POST` | `/api/v1/publications/mark-read` | Mark selected as read |
+| `POST` | `/api/v1/publications/{id}/retry-pdf` | Retry PDF resolution |
+| `POST` | `/api/v1/publications/{id}/favorite` | Toggle favorite |
+
+#### Publication Modes
+
+`GET /api/v1/publications` supports a `mode` parameter:
+
+| Mode | Description |
+|------|-------------|
+| `all` | All publications |
+| `unread` | Publications with `is_read=false` |
+| `latest` | Publications first seen in the latest completed run |
+
+`mode=new` is accepted as a compatibility alias for `latest`.
+
+#### Pagination
+
+Query parameters: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
+
+Response pagination fields:
+
+```json
+{
+  "page": 1,
+  "page_size": 20,
+  "has_prev": false,
+  "has_next": true,
+  "total_count": 142
+}
+```
+
+#### Publication Payload Fields
+
+| Field | Description |
+|-------|-------------|
+| `pub_url` | Canonical scholar detail URL |
+| `doi` | Normalized DOI |
+| `pdf_url` | Resolved open-access PDF URL (when available) |
+| `display_identifier` | Highest-confidence identifier regardless of source |
+
+### Runs
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/runs` | List runs with safety state |
+| `GET` | `/api/v1/runs/{run_id}` | Run detail with scholar results |
+| `POST` | `/api/v1/runs/{run_id}/cancel` | Cancel active run |
+| `POST` | `/api/v1/runs/manual` | Trigger manual run (idempotent, safety-checked) |
+| `GET` | `/api/v1/runs/queue/items` | List queue items |
+| `POST` | `/api/v1/runs/queue/{id}/retry` | Retry queue item |
+| `POST` | `/api/v1/runs/queue/{id}/drop` | Drop queue item |
+| `DELETE` | `/api/v1/runs/queue/{id}` | Clear queue item |
+| `GET` | `/api/v1/runs/{run_id}/stream` | Stream run events (SSE) |
+
+### Settings
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/settings` | Get user settings (with cooldown expiry check) |
+| `PUT` | `/api/v1/settings` | Update settings (interval, delay, nav, API keys) |
+
+### Admin - User Management
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/admin/users` | List all users |
+| `POST` | `/api/v1/admin/users` | Create user |
+| `PATCH` | `/api/v1/admin/users/{id}/active` | Set user active status |
+| `POST` | `/api/v1/admin/users/{id}/reset-password` | Reset password |
+
+### Admin - Database Operations
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/api/v1/admin/db/integrity` | Get integrity report |
+| `GET` | `/api/v1/admin/db/repair-jobs` | List repair jobs |
+| `GET` | `/api/v1/admin/db/pdf-queue` | List PDF queue |
+| `POST` | `/api/v1/admin/db/pdf-queue/{id}/requeue` | Requeue single PDF |
+| `POST` | `/api/v1/admin/db/pdf-queue/requeue-all` | Bulk requeue missing PDFs |
+| `POST` | `/api/v1/admin/db/repairs/publication-links` | Trigger link repair |
+| `POST` | `/api/v1/admin/db/repairs/publication-near-duplicates` | Trigger dedup repair |
+| `POST` | `/api/v1/admin/db/drop-all-publications` | Drop all publications (destructive) |

docs/reference/changelog.md

@@ -0,0 +1,12 @@
+---
+title: Changelog
+sidebar_position: 4
+---
+
+# Changelog
+
+This changelog is auto-generated by [python-semantic-release](https://python-semantic-release.readthedocs.io/).
+
+Release versions follow [Semantic Versioning](https://semver.org/). Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/).
+
+<!-- semantic-release will insert entries here -->

docs/reference/environment.md

@@ -1,68 +1,32 @@
-# Environment Reference and Audit
+---
+title: Environment Variables
+sidebar_position: 3
+---
 
-This document is the source-of-truth audit for what remains configurable versus internal defaults.
+# Environment Variables Quick Reference
 
-## Decision Summary
+All environment variables are documented in detail in [Configuration](../user/configuration.md).
 
-- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras.
+## Required Variables
-- Move to internal defaults only: none at this time.
-- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted.
 
-## Configurable Variables (By Section)
+| Variable | Description |
+|----------|-------------|
+| `POSTGRES_PASSWORD` | PostgreSQL password |
+| `SESSION_SECRET_KEY` | Session cookie signing key (32+ characters) |
 
-### Compose + Database
+## Categories
 
-`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE`
+| Category | Key Variables |
+|----------|--------------|
+| Database | `POSTGRES_DB`, `POSTGRES_USER`, `DATABASE_URL`, `DATABASE_POOL_*` |
+| Runtime | `APP_HOST`, `APP_PORT`, `MIGRATE_ON_START`, `FRONTEND_ENABLED` |
+| Auth | `SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_*` |
+| Security | `SECURITY_HEADERS_ENABLED`, `SECURITY_CSP_*`, `SECURITY_STRICT_TRANSPORT_*` |
+| Logging | `LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS` |
+| Scheduler | `SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_*_BATCH_SIZE` |
+| Ingestion | `INGESTION_*` (safety floors, cooldowns, retry policies) |
+| Scholar | `SCHOLAR_IMAGE_*`, `SCHOLAR_NAME_SEARCH_*` |
+| Enrichment | `UNPAYWALL_*`, `ARXIV_*`, `CROSSREF_*`, `OPENALEX_*`, `PDF_AUTO_RETRY_*` |
+| Bootstrap | `BOOTSTRAP_ADMIN_*`, `DB_WAIT_*` |
 
-### App Runtime + Networking
+See [Configuration](../user/configuration.md) for the complete table with types, defaults, and descriptions.
-
-`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR`
-
-### Database Pool
-
-`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS`
-
-### Frontend Dev Overrides
-
-`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET`
-
-### Auth + Session
-
-`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS`
-
-### HTTP Security Headers + CSP
-
-`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD`
-
-### Logging
-
-`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS`
-
-### Scheduler + Ingestion Safety
-
-`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `SCHEDULER_PDF_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_RATE_LIMIT_RETRIES`, `INGESTION_RATE_LIMIT_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
-
-### Scholar Images + Name Search Safety
-
-`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD`
-
-### OA Enrichment + PDF Resolution
-
-`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `ARXIV_ENABLED`, `ARXIV_TIMEOUT_SECONDS`, `ARXIV_MIN_INTERVAL_SECONDS`, `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS`, `ARXIV_DEFAULT_MAX_RESULTS`, `ARXIV_CACHE_TTL_SECONDS`, `ARXIV_CACHE_MAX_ENTRIES`, `ARXIV_MAILTO`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`, `OPENALEX_API_KEY`, `CROSSREF_API_TOKEN`, `CROSSREF_API_MAILTO`
-
-### Startup Bootstrap + DB Wait
-
-`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS`
-
-## Drift Guard
-
-CI enforces env parity with:
-
-```bash
-python3 scripts/check_env_contract.py
-```
-
-The check fails when:
-- `app/settings.py` references a variable missing from `.env.example`.
-- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras).
-- `.env.example` contains duplicate keys.

docs/reference/overview.md

@@ -1,6 +1,14 @@
-# Reference Documentation
+---
+title: Reference
+sidebar_position: 1
+---
 
-Use this section for canonical contracts and configuration references.
+# Reference
 
-- [API Contract](./api-contract.md)
+Quick-reference documentation for the scholarr API, environment variables, and release changelog.
-- [Environment Reference](./environment.md)
+
+| Document | Description |
+|----------|-------------|
+| [API Contract](api.md) | Envelope spec, endpoints, DTO structure, publication semantics |
+| [Environment Variables](environment.md) | Quick-reference table linking to full configuration docs |
+| [Changelog](changelog.md) | Auto-generated release history |

docs/user/configuration.md

@@ -0,0 +1,178 @@
+---
+title: Configuration
+sidebar_position: 3
+---
+
+# Configuration
+
+All configuration is done through environment variables. Copy `.env.example` to `.env` and adjust values as needed.
+
+## Compose & Database
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `POSTGRES_DB` | string | `scholar` | PostgreSQL database name |
+| `POSTGRES_USER` | string | `scholar` | PostgreSQL user |
+| `POSTGRES_PASSWORD` | string | **required** | PostgreSQL password |
+| `DATABASE_URL` | string | derived | SQLAlchemy async connection string |
+| `TEST_DATABASE_URL` | string | derived | Override for test database. If empty, tests derive `<db_name>_test` |
+| `SCHOLARR_IMAGE` | string | `justinzeus/scholarr:latest` | Docker image for the app service |
+
+## App Runtime & Networking
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `APP_NAME` | string | `scholarr` | Application name used in logs and headers |
+| `APP_HOST` | string | `0.0.0.0` | Bind address |
+| `APP_PORT` | int | `8000` | Internal port |
+| `APP_HOST_PORT` | int | `8000` | Host-mapped port |
+| `APP_RELOAD` | bool | `0` | Enable uvicorn auto-reload (dev only) |
+| `MIGRATE_ON_START` | bool | `1` | Run Alembic migrations on startup |
+| `FRONTEND_ENABLED` | bool | `1` | Serve the built Vue frontend |
+| `FRONTEND_DIST_DIR` | string | `/app/frontend/dist` | Path to compiled frontend assets |
+
+## Database Pool
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `DATABASE_POOL_MODE` | string | `auto` | Pool mode (`auto`, `fixed`, `null`) |
+| `DATABASE_POOL_SIZE` | int | `5` | Base pool size |
+| `DATABASE_POOL_MAX_OVERFLOW` | int | `10` | Maximum overflow connections |
+| `DATABASE_POOL_TIMEOUT_SECONDS` | int | `30` | Connection acquisition timeout |
+
+## Frontend Dev Overrides
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `FRONTEND_HOST_PORT` | int | `5173` | Host port for Vite dev server |
+| `CHOKIDAR_USEPOLLING` | bool | `1` | Enable polling for file watchers in containers |
+| `VITE_DEV_API_PROXY_TARGET` | string | `http://app:8000` | Backend URL for Vite proxy |
+
+## Auth & Session
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `SESSION_SECRET_KEY` | string | **required** | Signing key for session cookies (32+ chars) |
+| `SESSION_COOKIE_SECURE` | bool | `1` | Set `Secure` flag on session cookie (disable for local HTTP dev) |
+| `LOGIN_RATE_LIMIT_ATTEMPTS` | int | `5` | Max login attempts per window |
+| `LOGIN_RATE_LIMIT_WINDOW_SECONDS` | int | `60` | Sliding window for login rate limiting |
+
+## HTTP Security Headers & CSP
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `SECURITY_HEADERS_ENABLED` | bool | `1` | Enable security response headers |
+| `SECURITY_X_CONTENT_TYPE_OPTIONS` | string | `nosniff` | X-Content-Type-Options header value |
+| `SECURITY_X_FRAME_OPTIONS` | string | `DENY` | X-Frame-Options header value |
+| `SECURITY_REFERRER_POLICY` | string | `strict-origin-when-cross-origin` | Referrer-Policy header value |
+| `SECURITY_PERMISSIONS_POLICY` | string | *(restrictive)* | Permissions-Policy header value |
+| `SECURITY_CROSS_ORIGIN_OPENER_POLICY` | string | `same-origin` | Cross-Origin-Opener-Policy header |
+| `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY` | string | `same-origin` | Cross-Origin-Resource-Policy header |
+| `SECURITY_CSP_ENABLED` | bool | `1` | Enable Content-Security-Policy header |
+| `SECURITY_CSP_POLICY` | string | *(restrictive)* | CSP for app routes |
+| `SECURITY_CSP_DOCS_POLICY` | string | *(docs-specific)* | CSP for documentation routes |
+| `SECURITY_CSP_REPORT_ONLY` | bool | `0` | Use report-only mode for CSP |
+| `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED` | bool | `0` | Enable HSTS header |
+| `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE` | int | `31536000` | HSTS max-age in seconds |
+| `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS` | bool | `1` | HSTS includeSubDomains directive |
+| `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD` | bool | `0` | HSTS preload directive |
+
+## Logging
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `LOG_LEVEL` | string | `INFO` | Root log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
+| `LOG_FORMAT` | string | `console` | Log format (`console` or `json`) |
+| `LOG_REQUESTS` | bool | `1` | Log HTTP requests |
+| `LOG_UVICORN_ACCESS` | bool | `0` | Enable uvicorn access log |
+| `LOG_REQUEST_SKIP_PATHS` | string | `/healthz` | Comma-separated paths to exclude from request logging |
+| `LOG_REDACT_FIELDS` | string | *(empty)* | Comma-separated field names to redact in logs |
+
+## Scheduler & Ingestion Safety
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `SCHEDULER_ENABLED` | bool | `1` | Enable the background scheduler |
+| `SCHEDULER_TICK_SECONDS` | int | `60` | Scheduler poll interval |
+| `SCHEDULER_QUEUE_BATCH_SIZE` | int | `10` | Max scholars processed per tick |
+| `SCHEDULER_PDF_QUEUE_BATCH_SIZE` | int | `15` | Max PDF resolutions per tick |
+| `INGESTION_AUTOMATION_ALLOWED` | bool | `1` | Allow automated (scheduled) runs |
+| `INGESTION_MANUAL_RUN_ALLOWED` | bool | `1` | Allow manually triggered runs |
+| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | int | `15` | Minimum time between runs |
+| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | int | `2` | Floor delay between external requests |
+| `INGESTION_NETWORK_ERROR_RETRIES` | int | `1` | Retries on network errors |
+| `INGESTION_RETRY_BACKOFF_SECONDS` | float | `1.0` | Base backoff for network retries |
+| `INGESTION_RATE_LIMIT_RETRIES` | int | `3` | Retries on 429 responses |
+| `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` | float | `30.0` | Backoff per 429 retry |
+| `INGESTION_MAX_PAGES_PER_SCHOLAR` | int | `30` | Max paginated pages per scholar |
+| `INGESTION_PAGE_SIZE` | int | `100` | Publications per page |
+| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | int | `1` | Blocked failures before alert |
+| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | int | `2` | Network failures before alert |
+| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | int | `3` | Scheduled retries before alert |
+| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | int | `1800` | Cooldown after blocked-failure threshold (30 min) |
+| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | int | `900` | Cooldown after network-failure threshold (15 min) |
+| `INGESTION_CONTINUATION_QUEUE_ENABLED` | bool | `1` | Enable continuation queue for multi-page ingestion |
+| `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` | int | `120` | Base delay for continuation queue items |
+| `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` | int | `3600` | Max delay for continuation queue items |
+| `INGESTION_CONTINUATION_MAX_ATTEMPTS` | int | `6` | Max continuation attempts per scholar |
+
+## Scholar Images & Name Search Safety
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `SCHOLAR_IMAGE_UPLOAD_DIR` | string | `/var/lib/scholarr/uploads` | Directory for uploaded scholar images |
+| `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES` | int | `2000000` | Max image upload size (2 MB) |
+| `SCHOLAR_NAME_SEARCH_ENABLED` | bool | `1` | Enable name-based scholar search |
+| `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS` | int | `21600` | Cache TTL for successful searches (6 hours) |
+| `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS` | int | `300` | Cache TTL for blocked search results (5 min) |
+| `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES` | int | `512` | Max cache entries for name search |
+| `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS` | float | `8.0` | Min interval between name searches |
+| `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS` | float | `2.0` | Random jitter added to search interval |
+| `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD` | int | `1` | Blocked results before cooldown |
+| `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS` | int | `1800` | Cooldown after blocked name search (30 min) |
+| `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD` | int | `2` | Retries before alert |
+| `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` | int | `3` | Cooldown rejections before alert |
+
+## OA Enrichment & PDF Resolution
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `UNPAYWALL_ENABLED` | bool | `1` | Enable Unpaywall DOI lookups |
+| `UNPAYWALL_EMAIL` | string | *(empty)* | Polite pool email for Unpaywall API |
+| `UNPAYWALL_TIMEOUT_SECONDS` | float | `4.0` | Request timeout |
+| `UNPAYWALL_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Unpaywall requests |
+| `UNPAYWALL_MAX_ITEMS_PER_REQUEST` | int | `20` | Max items per batch |
+| `UNPAYWALL_RETRY_COOLDOWN_SECONDS` | int | `1800` | Cooldown after repeated failures |
+| `UNPAYWALL_PDF_DISCOVERY_ENABLED` | bool | `1` | Enable HTML-based PDF link discovery |
+| `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES` | int | `5` | Max candidate URLs to probe |
+| `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES` | int | `500000` | Max HTML response size to parse |
+| `ARXIV_ENABLED` | bool | `1` | Enable arXiv API lookups |
+| `ARXIV_TIMEOUT_SECONDS` | float | `3.0` | Request timeout |
+| `ARXIV_MIN_INTERVAL_SECONDS` | float | `4.0` | Min interval between arXiv requests |
+| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | float | `60.0` | Cooldown after arXiv 429 |
+| `ARXIV_DEFAULT_MAX_RESULTS` | int | `3` | Default max results per query |
+| `ARXIV_CACHE_TTL_SECONDS` | int | `900` | Query cache TTL (15 min) |
+| `ARXIV_CACHE_MAX_ENTRIES` | int | `512` | Max cached queries |
+| `ARXIV_MAILTO` | string | *(empty)* | Contact email for arXiv API headers |
+| `PDF_AUTO_RETRY_INTERVAL_SECONDS` | int | `86400` | Auto-retry interval for failed PDFs (24 hours) |
+| `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS` | int | `3600` | First retry interval (1 hour) |
+| `PDF_AUTO_RETRY_MAX_ATTEMPTS` | int | `3` | Max auto-retry attempts |
+| `CROSSREF_ENABLED` | bool | `1` | Enable Crossref lookups |
+| `CROSSREF_MAX_ROWS` | int | `10` | Max rows per Crossref query |
+| `CROSSREF_TIMEOUT_SECONDS` | float | `8.0` | Request timeout |
+| `CROSSREF_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Crossref requests |
+| `CROSSREF_MAX_LOOKUPS_PER_REQUEST` | int | `8` | Max lookups per ingestion request |
+| `OPENALEX_API_KEY` | string | *(empty)* | OpenAlex API key (optional) |
+| `CROSSREF_API_TOKEN` | string | *(empty)* | Crossref Plus API token (optional) |
+| `CROSSREF_API_MAILTO` | string | *(empty)* | Crossref polite pool email |
+
+## Startup Bootstrap & DB Wait
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `BOOTSTRAP_ADMIN_ON_START` | bool | `0` | Create admin user on startup |
+| `BOOTSTRAP_ADMIN_EMAIL` | string | *(empty)* | Admin email address |
+| `BOOTSTRAP_ADMIN_PASSWORD` | string | *(empty)* | Admin password |
+| `BOOTSTRAP_ADMIN_FORCE_PASSWORD` | bool | `0` | Overwrite existing admin password |
+| `DB_WAIT_TIMEOUT_SECONDS` | int | `60` | Max seconds to wait for database readiness |
+| `DB_WAIT_INTERVAL_SECONDS` | int | `2` | Poll interval while waiting for database |

docs/user/getting-started.md

@@ -1,41 +1,86 @@
-# User Getting Started
+---
+title: Getting Started
+sidebar_position: 2
+---
 
-## Required Setup
+# Getting Started
 
-1. Copy `.env.example` to `.env`.
+## Prerequisites
-2. Set `POSTGRES_PASSWORD`.
-3. Set `SESSION_SECRET_KEY`.
 
-## Deploy Method A: Prebuilt Image
+- Docker and Docker Compose v2+
+- A machine with at least 512 MB RAM
+
+## Installation
+
+1. Clone the repository:
+
+```bash
+git clone https://github.com/JustinZeus/scholarr.git
+cd scholarr
+```
+
+2. Copy the example environment file:
+
+```bash
+cp .env.example .env
+```
+
+3. Edit `.env` and set the required secrets:
+
+```bash
+# Required: database password
+POSTGRES_PASSWORD=your-secure-password
+
+# Required: session signing key (32+ random characters)
+SESSION_SECRET_KEY=your-random-secret-key
+```
+
+4. Start the stack:
+
+```bash
+docker compose up -d
+```
+
+5. Verify the service is healthy:
+
+```bash
+docker compose ps
+curl http://localhost:8000/healthz
+```
+
+The app is now available at `http://localhost:8000`.
+
+## First Run
+
+### Bootstrap an Admin User
+
+Set these environment variables before first start (or add them to `.env`):
+
+```bash
+BOOTSTRAP_ADMIN_ON_START=1
+BOOTSTRAP_ADMIN_EMAIL=admin@example.com
+BOOTSTRAP_ADMIN_PASSWORD=your-admin-password
+```
+
+Restart the app container. The admin account is created on startup.
+
+### Add Your First Scholar
+
+1. Log in at `http://localhost:8000`.
+2. Navigate to the Scholars page.
+3. Click **Add Scholar**.
+4. Enter a Google Scholar profile URL (e.g., `https://scholar.google.com/citations?user=XXXXXXXXXX`) or a Scholar ID.
+5. The scheduler will begin fetching publications on the next tick (default: 60 seconds).
+
+### Manual Run
+
+To trigger an immediate ingestion run, use the **Manual Run** button on the Runs page. This respects the minimum run interval (`INGESTION_MIN_RUN_INTERVAL_MINUTES`, default 15 minutes).
+
+## Updating
 
 ```bash
 docker compose pull
 docker compose up -d
 ```
 
-Open:
+Database migrations run automatically on startup when `MIGRATE_ON_START=1` (the default).
-- App/API: `http://localhost:8000`
-- Health endpoint: `http://localhost:8000/healthz`
-
-Upgrade:
-
-```bash
-docker compose pull
-docker compose up -d
-```
-
-## Deploy Method B: Local Source + Dev Frontend
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
-```
-
-Open:
-- API: `http://localhost:8000`
-- Frontend dev server: `http://localhost:5173`
-
-Stop:
-
-```bash
-docker compose -f docker-compose.yml -f docker-compose.dev.yml down
-```

docs/user/overview.md

@@ -1,5 +1,29 @@
-# User Documentation
+---
+title: Overview
+sidebar_position: 1
+---
 
-Use this section if you are deploying and using scholarr.
+# What is Scholarr?
 
-- [Getting Started](./getting-started.md)
+Scholarr is a self-hosted service that tracks academic publications from Google Scholar. It runs as a Docker container with a PostgreSQL database and serves a Vue 3 frontend.
+
+## Key Concepts
+
+- **Scholar** - A tracked Google Scholar profile. Scholars are user-scoped: each user manages their own list.
+- **Publication** - A globally deduplicated academic work. Publications are shared across all users to avoid duplicate storage.
+- **Scholar-Publication Link** - Connects a scholar to a publication for a specific user. Read/unread state, favorites, and visibility live on this link, not on the publication itself.
+- **Run** - A single ingestion cycle that fetches new publications for one or more scholars.
+- **Identifier** - A DOI, arXiv ID, PMID, or other external key attached to a publication. Multiple identifiers can exist per publication.
+
+## How It Works
+
+1. You add a Google Scholar profile (by ID, URL, or name search).
+2. The scheduler periodically scrapes the profile for new publications.
+3. Each publication is fingerprinted and deduplicated against the global store.
+4. External APIs (arXiv, Crossref, OpenAlex) are queried for identifiers.
+5. Unpaywall and arXiv resolve open-access PDF URLs when a DOI is available.
+6. The dashboard shows new, unread, and all publications with filtering and search.
+
+## Safety Model
+
+Scholarr enforces rate limits and cooldowns to prevent IP bans from upstream sources. These are not configurable to zero; they are safety floors. See [Scrape Safety Runbook](../operations/scrape-safety-runbook.md) for details.

docs/website/docusaurus.config.js

@@ -2,8 +2,8 @@
 
 /** @type {import('@docusaurus/types').Config} */
 const config = {
-  title: "scholarr",
+  title: "Scholarr",
-  tagline: "Self-hosted scholar tracking docs",
+  tagline: "Self-hosted academic publication tracker",
   favicon: "img/favicon.ico",
 
   url: "https://justinzeus.github.io",
@@ -23,59 +23,52 @@ const config = {
   presets: [
     [
       "classic",
-      {
+      /** @type {import('@docusaurus/preset-classic').Options} */
+      ({
         docs: {
           path: "..",
-          exclude: ["website/**", "README.md"],
           routeBasePath: "/",
           sidebarPath: require.resolve("./sidebars.js"),
-          editUrl: "https://github.com/JustinZeus/scholarr/tree/main/",
+          exclude: ["website/**", "README.md"],
         },
         blog: false,
         theme: {
           customCss: require.resolve("./src/css/custom.css"),
         },
-      },
+      }),
     ],
   ],
 
-  themeConfig: {
+  themeConfig:
-    navbar: {
+    /** @type {import('@docusaurus/preset-classic').ThemeConfig} */
-      title: "scholarr",
+    ({
-      items: [
+      navbar: {
-        {
+        title: "Scholarr",
-          type: "docSidebar",
+        items: [
-          sidebarId: "docsSidebar",
+          { to: "/user/overview", label: "User Guide", position: "left" },
-          position: "left",
+          {
-          label: "Docs",
+            to: "/developer/overview",
-        },
+            label: "Developer",
-        {
+            position: "left",
-          href: "https://github.com/JustinZeus/scholarr",
+          },
-          label: "GitHub",
+          {
-          position: "right",
+            to: "/operations/overview",
-        },
+            label: "Operations",
-      ],
+            position: "left",
-    },
+          },
-    footer: {
+          { to: "/reference/overview", label: "Reference", position: "left" },
-      style: "dark",
+          {
-      links: [
+            href: "https://github.com/JustinZeus/scholarr",
-        {
+            label: "GitHub",
-          title: "Docs",
+            position: "right",
-          items: [{ label: "Index", to: "/" }],
+          },
-        },
+        ],
-        {
+      },
-          title: "Community",
+      footer: {
-          items: [
+        style: "dark",
-            {
+        copyright: `Copyright \u00a9 ${new Date().getFullYear()} Scholarr. Built with Docusaurus.`,
-              label: "GitHub",
+      },
-              href: "https://github.com/JustinZeus/scholarr",
+    }),
-            },
-          ],
-        },
-      ],
-      copyright: `Copyright © ${new Date().getFullYear()} scholarr.`,
-    },
-  },
 };
 
 module.exports = config;

docs/website/package.json

@@ -1,18 +1,27 @@
 {
   "name": "scholarr-docs",
-  "version": "0.0.1",
+  "version": "0.1.0",
   "private": true,
   "scripts": {
+    "docusaurus": "docusaurus",
     "start": "docusaurus start",
     "build": "docusaurus build",
+    "swizzle": "docusaurus swizzle",
+    "deploy": "docusaurus deploy",
+    "clear": "docusaurus clear",
     "serve": "docusaurus serve"
   },
   "dependencies": {
-    "@docusaurus/core": "3.9.2",
+    "@docusaurus/core": "^3.0.0",
-    "@docusaurus/preset-classic": "3.9.2",
+    "@docusaurus/preset-classic": "^3.0.0",
-    "clsx": "^2.1.1",
-    "prism-react-renderer": "^2.3.1",
     "react": "^18.2.0",
     "react-dom": "^18.2.0"
+  },
+  "devDependencies": {
+    "@docusaurus/module-type-aliases": "^3.0.0"
+  },
+  "browserslist": {
+    "production": [">0.5%", "not dead", "not op_mini all"],
+    "development": ["last 1 chrome version", "last 1 firefox version", "last 1 safari version"]
   }
 }

docs/website/sidebars.js

@@ -1,38 +1,49 @@
 /** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
 const sidebars = {
-  docsSidebar: [
+  docs: [
     "index",
     {
       type: "category",
-      label: "Users",
+      label: "User Guide",
-      items: ["user/overview", "user/getting-started"],
+      items: [
+        "user/overview",
+        "user/getting-started",
+        "user/configuration",
+      ],
+    },
+    {
+      type: "category",
+      label: "Developer",
+      items: [
+        "developer/overview",
+        "developer/architecture",
+        "developer/local-development",
+        "developer/contributing",
+        "developer/ingestion",
+        "developer/frontend-theme-inventory",
+        "developer/testing",
+      ],
     },
     {
       type: "category",
       label: "Operations",
       items: [
         "operations/overview",
-        "operations/scrape-safety-runbook",
+        "operations/deployment",
         "operations/database-runbook",
-        "operations/migration-checklist",
+        "operations/scrape-safety-runbook",
-      ],
+        "operations/arxiv-runbook",
-    },
-    {
-      type: "category",
-      label: "Developers",
-      items: [
-        "developer/overview",
-        "developer/documentation-standards",
-        "developer/local-development",
-        "developer/architecture",
-        "developer/contributing",
-        "developer/frontend-theme-inventory",
       ],
     },
     {
       type: "category",
       label: "Reference",
-      items: ["reference/overview", "reference/api-contract", "reference/environment"],
+      items: [
+        "reference/overview",
+        "reference/api",
+        "reference/environment",
+        "reference/changelog",
+      ],
     },
   ],
 };

docs/website/src/css/custom.css

@@ -1,9 +1,22 @@
 :root {
-  --ifm-color-primary: #2a6b4e;
+  --ifm-color-primary: #2e8555;
-  --ifm-color-primary-dark: #255f46;
+  --ifm-color-primary-dark: #29784c;
-  --ifm-color-primary-darker: #225942;
+  --ifm-color-primary-darker: #277148;
-  --ifm-color-primary-darkest: #1c4936;
+  --ifm-color-primary-darkest: #205d3b;
-  --ifm-color-primary-light: #2f7756;
+  --ifm-color-primary-light: #33925d;
-  --ifm-color-primary-lighter: #327d5a;
+  --ifm-color-primary-lighter: #359962;
-  --ifm-color-primary-lightest: #3b946b;
+  --ifm-color-primary-lightest: #3cad6e;
+  --ifm-code-font-size: 95%;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
+}
+
+[data-theme="dark"] {
+  --ifm-color-primary: #25c2a0;
+  --ifm-color-primary-dark: #21af90;
+  --ifm-color-primary-darker: #1fa588;
+  --ifm-color-primary-darkest: #1a8870;
+  --ifm-color-primary-light: #29d5b0;
+  --ifm-color-primary-lighter: #32d8b4;
+  --ifm-color-primary-lightest: #4fddbf;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
 }