diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index f6b8216..0000000 --- a/docs/README.md +++ /dev/null @@ -1,23 +0,0 @@ -# Documentation Index - -This directory contains both docs content and docs tooling. - -## Audience-Based Structure - -- `docs/user/`: onboarding and usage-oriented docs for self-hosters. -- `docs/operations/`: runbooks and rollout checklists for ongoing operations. -- `docs/developer/`: architecture, contribution standards, and local development workflows. -- `docs/reference/`: canonical API and environment contracts. - -## Docs Tooling - -- `docs/website/`: Docusaurus app used to publish docs to GitHub Pages. - -## Recommended Reading Paths - -- Users: `docs/user/getting-started.md` -- Operators: `docs/operations/overview.md` -- Developers: `docs/developer/overview.md` -- Contract consumers: `docs/reference/overview.md` - -Published site: https://justinzeus.github.io/scholarr/ diff --git a/docs/developer/api-contract.md b/docs/developer/api-contract.md deleted file mode 100644 index 6884bc6..0000000 --- a/docs/developer/api-contract.md +++ /dev/null @@ -1,12 +0,0 @@ -# API Contracts - -`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI. - -## DTO Models Structure -FastAPI routes dynamically ingest request/response Data Transfer Objects. -- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin. -- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend. - -## Frontend Contract Requirements -The UI exclusively calls routes exposed by FastAPI's `APIRouter`. -All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally. diff --git a/docs/developer/architecture.md b/docs/developer/architecture.md index 4330830..392670e 100644 --- a/docs/developer/architecture.md +++ b/docs/developer/architecture.md @@ -1,58 +1,173 @@ -# Domain Boundaries +--- +title: Architecture +sidebar_position: 2 +--- + +# Architecture ## Data Model Rules -- Scholar tracking is user-scoped. -- Publications are global/deduplicated records. -- Read/favorite/visibility state stays on scholar-publication link rows. +- **Scholar tracking is user-scoped.** Each user manages their own list of tracked scholars. Validate mapping/join tables; never assume global links between users and Scholar IDs. +- **Publications are global, deduplicated records.** Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion. +- **State lives on the link.** Read/unread, favorites, and visibility state exist exclusively on the scholar-publication link table, not the global publication table. -## Service Boundaries +## Domain Service Boundaries -Canonical business logic belongs in `app/services/domains/*`. +All business logic resides in `app/services//`. Flat files in the `app/services/` root are strictly prohibited. -- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety. -- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters. -- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls. -- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths. -- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing. -- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing. -- `app/services/domains/runs/*`: run history and continuation queue operations. -- `app/services/domains/portability/*`: import/export workflows. +### Ingestion (`app/services/ingestion/`) -## Frontend Behavior Notes +Run orchestration, continuation queue, scheduler integration, and scrape safety policy enforcement. The `ScholarIngestionService` drives the primary data acquisition loop. -- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout. -- Lists: Long list views use internal scroll containers to prevent viewport overflow. -- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible. +Key modules: +- `application.py` - Main ingestion orchestrator +- `scheduler.py` - Background tick loop, queue batch processing +- `constants.py` - Safety policy constants and floor values +- `fingerprints.py` - Publication fingerprinting for deduplication +- `types.py` - Ingestion result types and state enums -## Data Integration & Acquisition Flow +### Scholar Parsing (`app/services/scholar/`) + +Fail-fast Google Scholar HTML parsing and source fetch adapters. Handles paginated HTML feeds, extracts publication blocks via regex and DOM invariants (e.g., `gsc_vcd_cib`). + +Key modules: +- `parser.py` - HTML parser for publication extraction +- `parser_utils.py` - Parsing helpers and DOM selectors +- `source.py` - HTTP fetch adapters with browser headers +- `profile_rows.py` - Profile metadata extraction +- `author_rows.py` - Author citation row parsing +- `state_detection.py` - Blocked/CAPTCHA/rate-limit detection + +### Scholar Management (`app/services/scholars/`) + +Scholar CRUD, profile image management, and name-search controls with rate limiting. + +Key modules: +- `application.py` - Scholar lifecycle operations +- `uploads.py` - Image upload handling +- `search_hints.py` - Name search with caching and cooldowns +- `validators.py` - Input validation for scholar creation + +### Publications (`app/services/publications/`) + +Listing, read-state management, favorite toggles, enrichment scheduling, deduplication, and PDF queue management. + +Key modules: +- `application.py` - Publication service facade +- `listing.py` - Filtered listing with pagination (modes: all/unread/latest) +- `queries.py` - Database query builders +- `counts.py` - Aggregation counts for dashboard +- `dedup.py` - Duplicate detection and merging +- `enrichment.py` - Identifier and metadata enrichment orchestration +- `pdf_queue.py` - PDF resolution queue policy +- `pdf_resolution_pipeline.py` - Multi-source PDF resolution (Unpaywall, arXiv) +- `types.py` - Publication DTOs and response types + +### Publication Identifiers (`app/services/publication_identifiers/`) + +Multi-identifier resolution engine. A single publication can have multiple identifiers (DOI, arXiv ID, PMID, PMCID). This decoupled approach replaced the earlier hardcoded DOI-only model. + +Key modules: +- `application.py` - Identifier gathering orchestration +- `normalize.py` - Identifier normalization and validation + +### arXiv (`app/services/arxiv/`) + +Typed API client with global DB-backed throttle, query cache, and in-flight request coalescing. + +Key modules: +- `client.py` - HTTP client for arXiv export API +- `gateway.py` - Gateway with advisory lock, caching, and coalescing +- `cache.py` - Query cache with TTL and max-entry pruning +- `rate_limit.py` - Global rate limiter via `arxiv_runtime_state` table +- `guards.py` - Load-shedding guards (skip when DOI/arXiv evidence exists) +- `parser.py` - Atom XML response parser + +Safety features: +- Requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`) +- Identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + max-entry pruning +- Concurrent identical misses are coalesced in-process (one outbound call serves all waiters) + +### Crossref (`app/services/crossref/`) + +DOI lookup via Crossref REST API with bounded pacing and configurable batch limits. + +### Unpaywall (`app/services/unpaywall/`) + +Open-access PDF resolution via Unpaywall API, with HTML-based PDF link discovery as a fallback. + +Key modules: +- `application.py` - Unpaywall service facade +- `pdf_discovery.py` - HTML page scraping for PDF link candidates + +### OpenAlex (`app/services/openalex/`) + +Metadata matching via OpenAlex API for supplementary identifier resolution. + +Key modules: +- `client.py` - OpenAlex API client +- `matching.py` - Fuzzy title/author matching + +### Runs (`app/services/runs/`) + +Run history tracking and continuation queue operations. + +Key modules: +- `application.py` - Run lifecycle management +- `queue_service.py` - Continuation queue operations (retry, drop, clear) +- `queue_queries.py` - Queue item queries + +### Portability (`app/services/portability/`) + +Import/export workflows for scholar data with full publication state preservation. + +Key modules: +- `application.py` - Import/export orchestration +- `exporting.py` - Scholar export serialization +- `publication_import.py` - Publication import with deduplication +- `scholar_import.py` - Scholar import with link reconstruction +- `normalize.py` - Payload normalization and validation + +### Database Operations (`app/services/dbops/`) + +Integrity checking, link repair, and near-duplicate repair operations exposed via admin API and CLI scripts. + +Key modules: +- `__init__.py` - `collect_integrity_report`, `run_publication_link_repair` +- `near_duplicate_repair.py` - Near-duplicate publication detection and merging + +## Data Integration Flow ```mermaid graph TD UI[Frontend Vue App] --> API[FastAPI Backend] - API --> Scheduler[Background Celery/Async Scheduler] - + API --> Scheduler[Background Async Scheduler] + Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser] Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering] - + IdentifierModule -->|Search arXiv API| API1(arXiv) IdentifierModule -->|Search Crossref API| API2(Crossref) - + IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)] - + Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline] DB --> |Identified DOIs| PDFPipeline PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API) - + Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker] - PDFWorker --> |Store PDF Metadata| DB + PDFWorker --> |Store PDF Metadata| DB ``` -### Identifier Engine Philosophy -The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics. +## API Layer -### arXiv Safety and Efficiency -- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`). -- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning. -- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters. -- structured logs provide auditability for scheduling/completion/cooldown/cache behavior. +Routes live in `app/api/routers/`. All responses under `/api/v1` use a strict envelope format. See [API Reference](../reference/api.md) for the full contract. + +## Middleware Stack + +Applied in `app/main.py`: + +1. **CSRF Protection** - Token-based CSRF validation +2. **Session Middleware** - Cookie-based session management (itsdangerous signing) +3. **Request Logging** - Structured request/response logging with configurable skip paths +4. **Security Headers** - Configurable HTTP security headers and CSP diff --git a/docs/developer/contributing.md b/docs/developer/contributing.md index ab44797..ec5ad5e 100644 --- a/docs/developer/contributing.md +++ b/docs/developer/contributing.md @@ -1,24 +1,78 @@ +--- +title: Contributing +sidebar_position: 4 +--- + # Contributing -## Scope -This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe. +## PR Process -## Essential-File Policy -Commit only source-of-truth files required to build, run, test, or document the app. +1. Create a feature branch from `main`. +2. Make your changes following the code standards below. +3. Run tests inside the container (see [Testing](testing.md)). +4. Run `ruff check .` and `mypy app/` to catch lint and type errors. +5. Open a pull request with a clear title and description. -Do not commit generated or local-only artifacts, including: -- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/` -- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/` -- coverage outputs (`.coverage`, `htmlcov/`) -- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`) -- local probe/scratch material (`planning/`) +## Commit Conventions -CI enforces this with `scripts/check_no_generated_artifacts.sh`. +This project uses [Conventional Commits](https://www.conventionalcommits.org/) with [python-semantic-release](https://python-semantic-release.readthedocs.io/) for automated versioning. -## Merge Checklist -- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas. -- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed). -- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`). -- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change. -- [ ] `README.md`, `.env.example`, and deployment notes stay aligned. -- [ ] `scripts/check_no_generated_artifacts.sh` passes locally. +Commit message format: + +``` +(): + +[optional body] +``` + +Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`. + +Examples: +- `feat(scholars): add bulk import from CSV` +- `fix(ingestion): handle empty citation blocks` +- `docs: update configuration reference` + +## Code Standards + +### Function Length + +Maximum 50 lines per function. Break complex logic into small, testable, single-responsibility functions. + +### DRY + +Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling. + +### Negative Space Programming + +Use explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing. + +### Cyclomatic Complexity + +Flatten logic. Use early returns and guard clauses instead of deep nesting. No magic numbers. + +### Domain Service Boundaries + +All business logic resides in `app/services//`. Flat files in the `app/services/` root are strictly prohibited. Each domain owns its own application service, types, and helpers. + +### API Envelope + +All `/api/v1` responses use the strict envelope format: + +- Success: `{"data": ..., "meta": {"request_id": "..."}}` +- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}` + +### Data Isolation + +- Scholar tracking is user-scoped. +- Publications are global, deduplicated records. +- Read/favorite/visibility state lives on scholar-publication link rows. + +### Scrape Safety + +Rate limits and cooldowns are immutable constraints. They prevent IP bans and must not be optimized away or set to zero. + +## UI Standards + +- Integrate Tailwind with the preset theming system (`frontend/src/theme/presets/`). +- Every UI element must have a clear purpose. +- Clarity through both styling and language. diff --git a/docs/developer/documentation-standards.md b/docs/developer/documentation-standards.md deleted file mode 100644 index f3c25b9..0000000 --- a/docs/developer/documentation-standards.md +++ /dev/null @@ -1,26 +0,0 @@ -# Documentation Standards - -This project keeps docs organized by audience and document type. - -## Audience Split - -- `user/`: onboarding and usage tasks for self-hosted users. -- `operations/`: runbooks and checklists for production operation. -- `developer/`: architecture, contribution workflow, and implementation guides. -- `reference/`: stable contracts (API, env variables, configuration behavior). - -## Document Quality Rules - -- One primary audience per page. -- Task pages must include prerequisites, exact commands, and verification steps. -- Reference pages should be contract-first and avoid procedural noise. -- Runbooks should include recovery steps and rollback/safety notes. -- Prefer concise pages with strong cross-links over long mixed-purpose pages. - -## Required Top-Level Entrypoints - -- `index.md`: user/developer/operator navigation hub. -- `user/getting-started.md`: first-run path. -- `developer/local-development.md`: contributor setup and validation path. -- `operations/overview.md`: operational playbook index. -- `reference/overview.md`: contract index. diff --git a/docs/developer/frontend-theme-inventory.md b/docs/developer/frontend-theme-inventory.md index b8d6571..0233071 100644 --- a/docs/developer/frontend-theme-inventory.md +++ b/docs/developer/frontend-theme-inventory.md @@ -1,89 +1,105 @@ -# Theme Inventory (Phase 0) +--- +title: Frontend Theme Inventory +sidebar_position: 6 +--- -This file captures the semantic color system baseline for the theme refactor. +# Frontend Theme Inventory ## Token Domains -- `scale`: - - `brand` (`50..950`) - - `info` (`50..950`) - - `success` (`50..950`) - - `warning` (`50..950`) - - `danger` (`50..950`) -- `surface`: - - `app` - - `nav` - - `nav_active` - - `card` - - `card_muted` - - `table` - - `table_header` - - `input` - - `overlay` -- `text`: - - `primary` - - `secondary` - - `muted` - - `inverse` - - `link` -- `border`: - - `default` - - `strong` - - `subtle` - - `interactive` -- `focus`: - - `ring` - - `ring_offset` -- `action` variants (`primary`, `secondary`, `ghost`, `danger`): - - `bg` - - `border` - - `text` - - `hover_bg` - - `hover_border` - - `hover_text` -- `state` variants (`info`, `success`, `warning`, `danger`): - - `bg` - - `border` - - `text` +The theme system uses semantic tokens organized into domains: -## Theme Sources +### Scale Colors -Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`. +Color ramps from `50` to `950` for each intent: -Current preset files: +- `brand` - Primary brand color +- `info` - Informational elements +- `success` - Success states +- `warning` - Warning states +- `danger` - Error/destructive states -- `frontend/src/theme/presets/parchment.js` -- `frontend/src/theme/presets/lilac.js` -- `frontend/src/theme/presets/dune.js` -- `frontend/src/theme/presets/oatmeal.js` -- `frontend/src/theme/presets/scholarly.json` -- `frontend/src/theme/presets/graphite.json` -- `frontend/src/theme/presets/tide.json` +### Surface -Each preset contains both `light` and `dark` mode token definitions. +Background colors for major UI areas: -## Adoption Status (Phase 0-3 complete baseline) +| Token | Usage | +|-------|-------| +| `app` | Application background | +| `nav` | Navigation bar | +| `nav_active` | Active navigation item | +| `card` | Card backgrounds | +| `card_muted` | Muted card variant | +| `table` | Table body | +| `table_header` | Table header | +| `input` | Form inputs | +| `overlay` | Modal/dialog overlays | -Tokenized foundation components: +### Text -- `AppButton` -- `AppCard` -- `AppCheckbox` -- `AppEmptyState` -- `AppHelpHint` -- `AppInput` -- `AppSelect` -- `AppTable` -- `AppModal` -- `AppHeader` -- `AppNav` -- `AppAlert` -- `AppBadge` -- `RunStatusBadge` -- `QueueHealthBadge` +| Token | Usage | +|-------|-------| +| `primary` | Default text | +| `secondary` | Supporting text | +| `muted` | Disabled/placeholder text | +| `inverse` | Text on dark backgrounds | +| `link` | Hyperlinks | -Hardening in place: +### Border -- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs` -- CI enforcement step in `frontend-quality` workflow -- Theme preset integrity tests in `frontend/src/theme/presets.test.ts` +| Token | Usage | +|-------|-------| +| `default` | Standard borders | +| `strong` | Emphasized borders | +| `subtle` | Light separators | +| `interactive` | Hover/focus borders | + +### Focus + +| Token | Usage | +|-------|-------| +| `ring` | Focus ring color | +| `ring_offset` | Focus ring offset color | + +### Action Variants + +Each action type (`primary`, `secondary`, `ghost`, `danger`) provides: + +`bg`, `border`, `text`, `hover_bg`, `hover_border`, `hover_text` + +### State Variants + +Each state (`info`, `success`, `warning`, `danger`) provides: + +`bg`, `border`, `text` + +## Theme Presets + +Presets are loaded from `frontend/src/theme/presets/*.{json,js}`. Each contains both `light` and `dark` mode token definitions. + +Current presets: + +| Preset | Format | +|--------|--------| +| `parchment` | JS | +| `lilac` | JS | +| `dune` | JS | +| `oatmeal` | JS | +| `scholarly` | JSON | +| `graphite` | JSON | +| `tide` | JSON | + +## Tokenized Components + +Foundation components using the token system: + +- `AppButton`, `AppCard`, `AppCheckbox`, `AppEmptyState` +- `AppHelpHint`, `AppInput`, `AppSelect`, `AppTable` +- `AppModal`, `AppHeader`, `AppNav`, `AppAlert` +- `AppBadge`, `RunStatusBadge`, `QueueHealthBadge` + +## Enforcement + +- Token policy check: `frontend/scripts/check_theme_tokens.mjs` +- CI step in `frontend-quality` workflow +- Preset integrity tests: `frontend/src/theme/presets.test.ts` diff --git a/docs/developer/ingestion.md b/docs/developer/ingestion.md index 26ee8e7..12eb014 100644 --- a/docs/developer/ingestion.md +++ b/docs/developer/ingestion.md @@ -1,38 +1,89 @@ -# Ingestion System & Backoff Strategies +--- +title: Ingestion Pipeline +sidebar_position: 5 +--- -The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans. +# Ingestion Pipeline -## Ingestion Overview -The underlying ingestion process: -1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`. -2. Connects asynchronously using configured HTTPX adapters with strict browser headers. -3. Downloads the paginated HTML feed for a user across multiple page iterations. -4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks. +The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans. -## Handling 429 Too Many Requests -Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`). +## Pipeline Overview -Scholarr treats these distinct from random network timeouts. -- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s). -- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead. +1. The scheduler (or a manual trigger) starts a **run** for one or more scholars. +2. The service connects via HTTPX with strict browser headers. +3. Paginated HTML feeds are downloaded for each scholar profile. +4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks. +5. Publications are fingerprinted and deduplicated against the global store. +6. External APIs resolve additional identifiers. +7. The PDF resolution pipeline runs asynchronously for publications with known DOIs. -## Publication Identifiers Loop -Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly. -- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes). -- **API Fetching**: Queries secondary bibliographic platforms sequentially: - - `export.arxiv.org/api/query` (Queries by Title and Author strings). - - `crossref.restful` APIs (Queries by Title and Author strings). +## Rate Limiting & Backoff -These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage. +### Network Errors + +Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s). + +### HTTP 429 (Rate Limited) + +When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies: + +- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3) +- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s) + +This pauses the pipeline gracefully instead of failing the entire run. + +### Safety Cooldowns + +Threshold-based cooldowns halt all ingestion after repeated failures: + +| Threshold | Variable | Default | Cooldown | +|-----------|----------|---------|----------| +| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) | +| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) | + +## Continuation Queue + +Multi-page ingestion uses a continuation queue to spread load over time: + +- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`) +- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s) +- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s) +- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6) + +Each continuation item is re-enqueued with exponential backoff. + +## Identifier Resolution + +After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers: + +1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches. +2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings. +3. **Crossref API** - Queries Crossref REST API by title and author strings. + +Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage. + +## PDF Resolution + +Publications with resolved DOIs enter the PDF resolution pipeline: + +1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs. +2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates. +3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly. + +Auto-retry is configured via `PDF_AUTO_RETRY_*` variables. ## arXiv Request Controls -- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock. -- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`. -- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out. -- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold. -## arXiv Observability Events -- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. -- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. -- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown. -- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`. +- **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock. +- **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`. +- **In-flight coalescing**: Duplicate concurrent misses join one outbound request. +- **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold. + +### Observability Events + +| Event | Description | +|-------|-------------| +| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. | +| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. | +| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. | +| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. | diff --git a/docs/developer/local-development.md b/docs/developer/local-development.md index af89f3f..2adfab0 100644 --- a/docs/developer/local-development.md +++ b/docs/developer/local-development.md @@ -1,52 +1,100 @@ -# Developer Local Development +--- +title: Local Development +sidebar_position: 3 +--- -## Start the Dev Stack +# Local Development + +## Prerequisites + +- Docker and Docker Compose v2+ +- Python 3.12+ (for IDE support and local linting) +- Node.js 20+ (for frontend development) + +## Starting the Dev Stack + +The development compose file overlays the production config with hot-reload and a separate Vite dev server: ```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build +docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build ``` -Open: -- API: `http://localhost:8000` -- Frontend dev server: `http://localhost:5173` +This starts three services: -Stop: +| Service | Port | Description | +|---------|------|-------------| +| `db` | 5432 (internal) | PostgreSQL 15 | +| `app` | 8000 | FastAPI backend with `APP_RELOAD=1` | +| `frontend` | 5173 | Vite dev server proxying API calls to `app:8000` | + +### Dev-Specific Overrides + +The `docker-compose.dev.yml` file applies these changes: + +- **`app`**: Uses `scholarr-dev:local` image built from the `dev` stage. Mounts the project root as `/app` for hot reload. Disables `SESSION_COOKIE_SECURE` and the built frontend. +- **`frontend`**: Node 20 container running `npm install && npm run dev`. Mounts `./frontend` with a named volume for `node_modules`. Uses polling for file watching (`CHOKIDAR_USEPOLLING=1`). + +## Environment Setup ```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml down +cp .env.example .env ``` -## Backend Validation +Set at minimum: ```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration +POSTGRES_PASSWORD=localdev +SESSION_SECRET_KEY=local-dev-secret-at-least-32-characters +SESSION_COOKIE_SECURE=0 ``` -## Frontend Validation +## Running Tests + +All tests run inside containers: ```bash -cd frontend -npm install -npm run typecheck -npm run test:run -npm run build +# Run unit tests (default: excludes integration markers) +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest + +# Run integration tests +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest -m integration + +# Run a specific test file +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest tests/unit/test_fingerprints.py -v ``` -## Repository Gates +See [Testing](testing.md) for markers, fixtures, and conventions. + +## Linting and Type Checking ```bash -python3 scripts/check_frontend_api_contract.py -python3 scripts/check_env_contract.py -./scripts/check_no_generated_artifacts.sh +# Ruff linting +ruff check . + +# Ruff formatting check +ruff format --check . + +# Mypy type checking +mypy app/ ``` -## Docs Site (Contributor Workflow) +Ruff config is in `pyproject.toml`: target Python 3.12, line length 120, rules `E F W I UP B SIM RUF`. -Docs tooling is colocated in `docs/website/`: +## Database Migrations + +Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. To run manually: ```bash -cd docs/website -npm install -npm run build +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + alembic upgrade head +``` + +To create a new migration: + +```bash +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + alembic revision --autogenerate -m "description of change" ``` diff --git a/docs/developer/overview.md b/docs/developer/overview.md index 7ab91cc..9974236 100644 --- a/docs/developer/overview.md +++ b/docs/developer/overview.md @@ -1,11 +1,65 @@ -# Developer Documentation +--- +title: Developer Overview +sidebar_position: 1 +--- -Use this section if you are contributing code or running full quality gates. +# Developer Overview -- [Documentation Standards](./documentation-standards.md) -- [Local Development](./local-development.md) -- [Architecture Boundaries](./architecture.md) -- [Ingestion API & Backoff Strategies](./ingestion.md) -- [API Contracts](./api-contract.md) -- [Contributing](./contributing.md) -- [Frontend Theme Inventory](./frontend-theme-inventory.md) +Scholarr is a Python 3.12+ FastAPI backend with an async SQLAlchemy ORM layer, PostgreSQL database, and a Vue 3 + TypeScript + Vite frontend. + +## Stack + +| Layer | Technology | +|-------|------------| +| Backend | Python 3.12+, FastAPI, SQLAlchemy 2.0 (async/asyncpg), Alembic | +| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS | +| Database | PostgreSQL 15 | +| Infrastructure | Multi-stage Docker, Docker Compose | +| Linting | ruff (E, F, W, I, UP, B, SIM, RUF), mypy | +| Testing | pytest, pytest-asyncio | +| Versioning | python-semantic-release, conventional commits | + +## Quick Start + +```bash +# Build the dev image and start services +docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build + +# Backend: http://localhost:8000 (hot reload enabled) +# Frontend: http://localhost:5173 (Vite dev server with API proxy) +``` + +See [Local Development](local-development.md) for the full setup guide. + +## Project Layout + +``` +app/ +├── api/routers/ # FastAPI route handlers +├── auth/ # Session + CSRF middleware +├── db/ # SQLAlchemy models, session factory, migrations +├── services/ # Domain service modules (see Architecture) +│ ├── arxiv/ # arXiv API client, cache, rate limiting +│ ├── crossref/ # Crossref DOI lookups +│ ├── dbops/ # Database integrity + repair operations +│ ├── ingestion/ # Run orchestration, scheduler, safety gates +│ ├── openalex/ # OpenAlex metadata matching +│ ├── portability/ # Import/export workflows +│ ├── publication_identifiers/ # Multi-identifier resolution +│ ├── publications/ # Listing, enrichment, dedup, PDF queue +│ ├── runs/ # Run history, continuation queue +│ ├── scholar/ # HTML parser, source fetch adapters +│ ├── scholars/ # Scholar CRUD, image upload, name search +│ └── unpaywall/ # Unpaywall PDF discovery +├── main.py # App factory, lifespan, middleware stack +frontend/ +├── src/ +│ ├── components/ # Reusable Vue components +│ ├── theme/presets/ # Color theme presets (light + dark) +│ └── ... +scripts/db/ # Operational database scripts +tests/ +├── unit/ # Fast, no-database tests +├── integration/ # Tests requiring database + services +└── fixtures/ # Shared test fixtures +``` diff --git a/docs/developer/testing.md b/docs/developer/testing.md new file mode 100644 index 0000000..6440a3d --- /dev/null +++ b/docs/developer/testing.md @@ -0,0 +1,102 @@ +--- +title: Testing +sidebar_position: 7 +--- + +# Testing + +## Running Tests + +All tests must run inside containers: + +```bash +# Unit tests (default: excludes integration markers) +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest + +# Integration tests +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest -m integration + +# Specific marker +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest -m db + +# Verbose output for a specific file +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest tests/unit/test_fingerprints.py -v +``` + +## Test Configuration + +From `pyproject.toml`: + +```toml +[tool.pytest.ini_options] +addopts = "-q -m \"not integration\" --import-mode=importlib" +asyncio_mode = "auto" +testpaths = ["tests"] +``` + +- Default run excludes `integration` marked tests +- Uses `importlib` import mode to resolve module name collisions +- Async tests run automatically (no `@pytest.mark.asyncio` needed) + +## Markers + +| Marker | Description | +|--------|-------------| +| `integration` | Tests requiring external services (database, network) | +| `db` | Tests that validate database behavior and constraints | +| `migrations` | Tests focused on Alembic schema migration correctness | +| `schema` | Tests focused on multi-tenant schema invariants | +| `smoke` | Smoke tests for containerized runtime | + +## Test Tiers + +### Unit Tests (`tests/unit/`) + +Fast, no-database tests. Mock external dependencies. These run by default. + +Examples: +- `test_fingerprints.py` - Publication fingerprinting logic +- `test_scholar_parser.py` - HTML parsing without network calls +- `test_doi_normalize.py` - DOI normalization rules +- `test_ingestion_arxiv_rate_limit.py` - Rate limiter behavior +- `test_publication_pdf_resolution_pipeline.py` - PDF pipeline logic + +Domain-specific unit tests are organized under `tests/unit/services/domains/`: +- `arxiv/` - Cache, client, gateway, guards, parser, rate limit tests +- `openalex/` - Client and matching tests +- `publications/` - Dedup tests + +### Integration Tests (`tests/integration/`) + +Require a running database. Test full request/response flows and data consistency. + +Examples: +- `test_api_v1.py` - API endpoint integration tests +- `test_db_integrity.py` - Database integrity checks +- `test_run_lifecycle_consistency.py` - Run state machine transitions +- `test_deferred_enrichment.py` - Enrichment pipeline with real data +- `test_fixture_probe_runs.py` - Fixture-based run probes + +### Smoke Tests + +Marked with `@pytest.mark.smoke`. Validate the containerized runtime starts and serves basic requests. + +## Fixtures + +Test fixtures live in `tests/fixtures/`: + +``` +tests/fixtures/ +└── scholar/ + ├── profile_ok_amIMrIEAAAAJ.html # Successful profile HTML + └── regression/ + ├── profile_P1RwlvoAAAAJ.html # Regression case + ├── profile_LZ5D_p4AAAAJ.html # Regression case + └── profile_AAAAAAAAAAAA.html # Regression case +``` + +Scholar HTML fixtures are real Google Scholar profile pages used to test parser robustness against DOM structure changes. diff --git a/docs/index.md b/docs/index.md index 4a6574c..94b563b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,33 +1,27 @@ --- -slug: / +title: Scholarr Documentation +sidebar_position: 1 --- -# scholarr Documentation +# Scholarr -This documentation is organized for both users and developers. +Scholarr is a self-hosted academic publication tracker. It monitors Google Scholar profiles, discovers new publications, resolves open-access PDFs, and presents everything through a clean Vue 3 dashboard. -## For Users +## Quick Links -- [User Overview](./user/overview.md) -- [Getting Started](./user/getting-started.md) +| Section | Description | +|---------|-------------| +| [User Guide](user/overview.md) | What scholarr does, installation, configuration | +| [Developer Guide](developer/overview.md) | Architecture, local development, contributing | +| [Operations](operations/overview.md) | Deployment, database runbook, scrape safety | +| [Reference](reference/overview.md) | API contract, environment variables, changelog | -## For Operators +## Key Features -- [Operations Overview](./operations/overview.md) -- [Scrape Safety Runbook](./operations/scrape-safety-runbook.md) -- [Database Runbook](./operations/database-runbook.md) -- [Migration Rollout Checklist](./operations/migration-checklist.md) - -## For Developers - -- [Developer Overview](./developer/overview.md) -- [Local Development](./developer/local-development.md) -- [Architecture Boundaries](./developer/architecture.md) -- [Contributing](./developer/contributing.md) -- [Frontend Theme Inventory](./developer/frontend-theme-inventory.md) - -## Reference - -- [Reference Overview](./reference/overview.md) -- [API Contract](./reference/api-contract.md) -- [Environment Reference](./reference/environment.md) +- **Scholar Tracking** - Add Google Scholar profiles by ID, URL, or name search +- **Automated Ingestion** - Background scheduler fetches new publications on a configurable interval +- **Identifier Resolution** - Cross-references arXiv, Crossref, and OpenAlex for DOIs and metadata +- **PDF Discovery** - Resolves open-access PDFs via Unpaywall and arXiv +- **Import/Export** - Portable scholar data with full publication state +- **Multi-User** - Session-based auth with admin user management +- **Theming** - Multiple color presets with light/dark mode support diff --git a/docs/operations/arxiv-runbook.md b/docs/operations/arxiv-runbook.md index b3183be..b49da90 100644 --- a/docs/operations/arxiv-runbook.md +++ b/docs/operations/arxiv-runbook.md @@ -1,46 +1,80 @@ +--- +title: arXiv Runbook +sidebar_position: 5 +--- + # arXiv Operations Runbook Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly. -## Signals To Check -- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss` +## Signals to Check + +- Logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss` - `arxiv_runtime_state` row for cooldown and next-allowed timestamps - `arxiv_query_cache_entries` size and expiry churn ## Event Field Guide -- `wait_seconds`: enforced pre-request delay from the global limiter. -- `status_code`: upstream response code from arXiv. -- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429. -- `source_path`: caller path (`search` or `lookup_ids`). + +| Field | Description | +|-------|-------------| +| `wait_seconds` | Enforced pre-request delay from the global limiter | +| `status_code` | Upstream response code from arXiv | +| `cooldown_remaining_seconds` | Remaining cooldown when blocked or after 429 | +| `source_path` | Caller path (`search` or `lookup_ids`) | ## Quick SQL Checks + +### Runtime State + ```sql SELECT state_key, next_allowed_at, cooldown_until, updated_at FROM arxiv_runtime_state; ``` +### Cache Status + ```sql -SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry +SELECT count(*) AS cache_rows, + min(expires_at) AS earliest_expiry, + max(expires_at) AS latest_expiry FROM arxiv_query_cache_entries; ``` ## Common Scenarios -1. Repeated `arxiv.cooldown_activated` events: -- Confirm recent `429` statuses in `arxiv.request_completed`. -- Reduce caller pressure (check new title-quality/identifier guards are active). + +### 1. Repeated `arxiv.cooldown_activated` Events + +- Confirm recent `429` statuses in `arxiv.request_completed` logs. +- Reduce caller pressure (check title-quality/identifier guards are active). - Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict. -2. High request latency with few completions: +### 2. High Request Latency with Few Completions + - Inspect `wait_seconds` in `arxiv.request_scheduled`. - Verify only one process path is repeatedly hitting arXiv (`source_path`). - Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears). -3. Low cache effectiveness: +### 3. Low Cache Effectiveness + - Validate normalized query behavior and caller churn. - Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads. - Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed. +## Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `ARXIV_ENABLED` | `1` | Enable arXiv lookups | +| `ARXIV_TIMEOUT_SECONDS` | `3.0` | Request timeout | +| `ARXIV_MIN_INTERVAL_SECONDS` | `4.0` | Min interval between requests | +| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | `60.0` | Cooldown after 429 | +| `ARXIV_DEFAULT_MAX_RESULTS` | `3` | Max results per query | +| `ARXIV_CACHE_TTL_SECONDS` | `900` | Cache TTL (15 min) | +| `ARXIV_CACHE_MAX_ENTRIES` | `512` | Max cached queries | +| `ARXIV_MAILTO` | *(empty)* | Contact email for API headers | + ## Safe Recovery + 1. Pause automated ingestion if rate-limit storms persist. 2. Let cooldown expire naturally; avoid manual burst retries. 3. Resume and monitor event rates before restoring full load. diff --git a/docs/operations/database-runbook.md b/docs/operations/database-runbook.md index abe320d..434bc51 100644 --- a/docs/operations/database-runbook.md +++ b/docs/operations/database-runbook.md @@ -1,100 +1,195 @@ -# Database Operations Runbook +--- +title: Database Runbook +sidebar_position: 3 +--- -This runbook defines backup, restore, and verification procedures for the -PostgreSQL database used by `scholarr`. +# Database Runbook -## Objectives +## Backup -- Keep data loss bounded via scheduled backups. -- Provide repeatable restore procedures. -- Verify backups by running regular restore drills. +Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service. -Recommended starting targets: - -- `RPO`: 24 hours (maximum acceptable data loss). -- `RTO`: 60 minutes (maximum acceptable restore time). - -## Backup Strategy - -Use logical backups from the running `db` compose service. - -- Custom-format backup (recommended for restore flexibility): +### Custom-Format Backup (Default) ```bash scripts/db/backup_full.sh ``` -- Plain SQL backup: +Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`). + +### Plain SQL Backup ```bash scripts/db/backup_full.sh --plain ``` -Optional environment variables: +Creates a `.sql` file. -- `BACKUP_DIR`: destination directory (default: `/backups`) -- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`) -- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml` +### Options -## Restore Strategy +| Option | Description | +|--------|-------------| +| `--plain` | Write plain SQL instead of custom-format dump | -Restore from a `.dump` (custom format) or `.sql` file into the running `db` -compose service. +### Environment Variables -- Restore without schema wipe: +| Variable | Default | Description | +|----------|---------|-------------| +| `BACKUP_DIR` | `/backups` | Destination directory | +| `BACKUP_PREFIX` | `scholarr` | File prefix | +| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` | + +## Restore + +Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service. + +### Restore a Custom-Format Dump ```bash -scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump +scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump ``` -- Restore with full `public` schema reset: +### Restore with Schema Wipe ```bash -scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public +scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public ``` -## Safety Checklist Before Restore +This drops and recreates the `public` schema before restoring. Use with caution. -1. Pause writes (stop app/scheduler or set maintenance mode). -2. Create a fresh pre-restore backup. -3. Validate target dump checksum/size. -4. Confirm operator, scope, and rollback plan. +### Options -## Post-Restore Verification +| Option | Description | +|--------|-------------| +| `--file ` | Required. Path to `.dump` or `.sql` backup file | +| `--wipe-public` | Drop and recreate `public` schema before restore | -1. Run migrations head check. -2. Run health endpoint checks. -3. Verify table row counts for core tables: - - `users` - - `scholar_profiles` - - `publications` - - `scholar_publications` - - `crawl_runs` -4. Run integrity report and require zero failures: +## Integrity Checks + +Use `scripts/db/check_integrity.py` to run database integrity checks. + +### Run Inside Container ```bash -python3 scripts/db/check_integrity.py +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python scripts/db/check_integrity.py ``` -5. Run API smoke checks for scholars/publications/runs pages. - -## Data Cleanup and Repair - -Use the audited repair job for scholar-publication relinking cleanup: +### With Strict Warnings ```bash -python3 scripts/db/repair_publication_links.py --user-id --requested-by "" --apply +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python scripts/db/check_integrity.py --strict-warnings ``` -Dry-run first, then re-run with `--apply` once scope and summary counts match expectation. +Returns non-zero exit code if any warning is present. -If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout. +### Output Format -## Restore Drill Cadence +JSON report: -- Run at least one full restore drill per month. -- Record: - - backup artifact used, - - restore start/end timestamps, - - issues encountered, - - achieved `RTO`. +```json +{ + "status": "passed", + "failures": [], + "warnings": [] +} +``` + +Exit codes: +- `0` - All checks passed +- `1` - Check failure +- `2` - Warnings present (with `--strict-warnings`) + +### Via Admin API + +``` +GET /api/v1/admin/db/integrity +``` + +Returns the same integrity report through the API (admin auth required). + +## Publication Link Repair + +Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging. + +### Dry Run (Default) + +```bash +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python scripts/db/repair_publication_links.py --user-id 1 +``` + +### Apply Changes + +```bash +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python scripts/db/repair_publication_links.py --user-id 1 --apply \ + --requested-by "admin@example.com" +``` + +### Options + +| Option | Description | +|--------|-------------| +| `--user-id ` | Required. Target user ID | +| `--scholar-profile-id ` | Optional. Filter by scholar profile ID (repeatable) | +| `--apply` | Apply changes (default is dry-run) | +| `--gc-orphan-publications` | Delete publications with zero links after cleanup | +| `--requested-by ` | Operator identifier for audit logs | + +### Via Admin API + +``` +POST /api/v1/admin/db/repairs/publication-links +``` + +## Near-Duplicate Repair + +Detect and merge near-duplicate publications via the admin API: + +``` +POST /api/v1/admin/db/repairs/publication-near-duplicates +``` + +## PDF Queue Management + +### List Queue + +``` +GET /api/v1/admin/db/pdf-queue +``` + +### Requeue Single Item + +``` +POST /api/v1/admin/db/pdf-queue/{id}/requeue +``` + +### Bulk Requeue + +``` +POST /api/v1/admin/db/pdf-queue/requeue-all +``` + +## Migration Procedures + +Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. + +To run manually: + +```bash +docker compose exec app alembic upgrade head +``` + +To check current migration status: + +```bash +docker compose exec app alembic current +``` + +Recommended procedure for production: +1. Backup the database before upgrading. +2. Pull the new image. +3. Start the container; migrations run on startup. +4. Verify via `GET /healthz` and check logs for migration output. diff --git a/docs/operations/deployment.md b/docs/operations/deployment.md new file mode 100644 index 0000000..48e06d4 --- /dev/null +++ b/docs/operations/deployment.md @@ -0,0 +1,117 @@ +--- +title: Deployment +sidebar_position: 2 +--- + +# Deployment + +## Production Docker Compose + +The default `docker-compose.yml` runs two services: + +| Service | Image | Description | +|---------|-------|-------------| +| `db` | `postgres:15-alpine` | PostgreSQL database with persistent volume | +| `app` | `justinzeus/scholarr:latest` | FastAPI application with embedded frontend | + +### Start + +```bash +docker compose up -d +``` + +### Stop + +```bash +docker compose down +``` + +### Update + +```bash +docker compose pull +docker compose up -d +``` + +### View Logs + +```bash +docker compose logs -f app +docker compose logs -f db +``` + +## Required Environment Variables + +These must be set in `.env` or the shell environment: + +```bash +POSTGRES_PASSWORD= +SESSION_SECRET_KEY= +``` + +## Volumes + +| Volume | Mount | Description | +|--------|-------|-------------| +| `postgres_data` | `/var/lib/postgresql/data` | Database files | +| `scholar_uploads` | `/var/lib/scholarr/uploads` | Scholar profile images | + +## Health Checks + +### Database + +```yaml +healthcheck: + test: ["CMD-SHELL", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"] + interval: 5s + timeout: 5s + retries: 20 +``` + +### Application + +```yaml +healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"] + interval: 10s + timeout: 5s + retries: 12 +``` + +The app service depends on `db` with `condition: service_healthy`, so it waits for the database to be ready. + +## Database Readiness Wait + +The app has built-in database readiness polling: + +- `DB_WAIT_TIMEOUT_SECONDS` (default: 60) - Max wait time +- `DB_WAIT_INTERVAL_SECONDS` (default: 2) - Poll interval + +## Auto-Migration + +Set `MIGRATE_ON_START=1` (default) to run Alembic migrations automatically on startup. + +## Scaling Considerations + +- Run a single `app` instance to avoid scheduler conflicts (the scheduler is process-local). +- arXiv requests are globally serialized via a PostgreSQL advisory lock, so multiple instances safely share the rate limiter. +- Database pool defaults: 5 base connections + 10 overflow. Adjust `DATABASE_POOL_SIZE` and `DATABASE_POOL_MAX_OVERFLOW` for higher loads. + +## Admin Bootstrap + +For first-time setup without manual database access: + +```bash +BOOTSTRAP_ADMIN_ON_START=1 +BOOTSTRAP_ADMIN_EMAIL=admin@example.com +BOOTSTRAP_ADMIN_PASSWORD= +``` + +Set `BOOTSTRAP_ADMIN_FORCE_PASSWORD=1` to reset an existing admin password. + +## Security Hardening + +- Set `SESSION_COOKIE_SECURE=1` when serving over HTTPS. +- Enable HSTS: `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED=1`. +- Review CSP policy in `SECURITY_CSP_POLICY` for your domain. +- Set `UNPAYWALL_EMAIL` and `ARXIV_MAILTO` for polite API pool access. diff --git a/docs/operations/migration-checklist.md b/docs/operations/migration-checklist.md deleted file mode 100644 index 106b9fc..0000000 --- a/docs/operations/migration-checklist.md +++ /dev/null @@ -1,48 +0,0 @@ -# Migration Checklist - -Use this checklist for every schema/data migration. - -## 1. Design Review - -- Define change type: `expand`, `backfill`, `contract`. -- Document expected lock behavior and index impact. -- Confirm backward compatibility with currently deployed app version. -- Define rollback strategy before implementation. - -## 2. Pre-Migration Controls - -- Capture a fresh backup before applying migration. -- Confirm migration head revision and working tree cleanliness. -- Prepare validation queries for new/changed tables and indexes. -- Identify high-risk tables (large row count, hot write paths). - -## 3. Implementation Standards - -- Keep migrations idempotent when feasible. -- Prefer additive steps first (`nullable`, new index, new table). -- For destructive changes, separate into later contract migration. -- Avoid large blocking rewrites in a single deployment step. - -## 4. Verification - -- Apply migration in staging against production-like snapshot. -- Verify: - - expected tables/columns/indexes, - - app startup and health endpoint, - - affected API flows, - - data consistency queries. - -## 5. Rollout and Recovery - -- Apply migration during planned window. -- Monitor logs/errors and DB metrics during rollout. -- If rollback needed: - - follow downgrade/recovery runbook, - - restore from backup if downgrade is unsafe, - - document incident timeline. - -## 6. Post-Migration Tasks - -- Update `README.md` / `.env.example` / ops docs. -- Add/update integration tests for new schema assumptions. -- Record migration notes in changelog. diff --git a/docs/operations/overview.md b/docs/operations/overview.md index f208a7d..7d2f997 100644 --- a/docs/operations/overview.md +++ b/docs/operations/overview.md @@ -1,8 +1,30 @@ -# Operations Documentation +--- +title: Operations Overview +sidebar_position: 1 +--- -Use this section for production operations and incident/runbook workflows. +# Operations Overview -- [Scrape Safety Runbook](./scrape-safety-runbook.md) -- [arXiv Runbook](./arxiv-runbook.md) -- [Database Runbook](./database-runbook.md) -- [Migration Rollout Checklist](./migration-checklist.md) +This section covers production deployment, database administration, and scrape safety operations. + +## Quick Links + +| Guide | Description | +|-------|-------------| +| [Deployment](deployment.md) | Production Docker setup, scaling, health checks | +| [Database Runbook](database-runbook.md) | Backup, restore, integrity checks, repair procedures | +| [Scrape Safety Runbook](scrape-safety-runbook.md) | Rate limiting, cooldowns, CAPTCHA handling | +| [arXiv Runbook](arxiv-runbook.md) | arXiv rate limits, cache tuning, query patterns | + +## Health Check + +The app exposes `GET /healthz` for container orchestration: + +```bash +curl -fsS http://localhost:8000/healthz +``` + +Docker Compose healthcheck config: +- Interval: 10s +- Timeout: 5s +- Retries: 12 diff --git a/docs/operations/scrape-safety-runbook.md b/docs/operations/scrape-safety-runbook.md index db01064..71f6769 100644 --- a/docs/operations/scrape-safety-runbook.md +++ b/docs/operations/scrape-safety-runbook.md @@ -1,74 +1,76 @@ +--- +title: Scrape Safety Runbook +sidebar_position: 4 +--- + # Scrape Safety Runbook -This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events. +Operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events. ## Key Signals Use structured log `event` fields (recommended with `LOG_FORMAT=json`): -- `ingestion.safety_policy_blocked_run_start` -- `ingestion.safety_cooldown_entered` -- `ingestion.safety_cooldown_cleared` -- `ingestion.alert_blocked_failure_threshold_exceeded` -- `ingestion.alert_network_failure_threshold_exceeded` -- `ingestion.alert_retry_scheduled_threshold_exceeded` -- `api.runs.manual_blocked_policy` -- `api.runs.manual_blocked_safety` -- `scheduler.run_skipped_safety_cooldown` -- `scheduler.run_skipped_safety_cooldown_precheck` -- `scheduler.queue_item_deferred_safety_cooldown` +| Event | Description | +|-------|-------------| +| `ingestion.safety_policy_blocked_run_start` | Run blocked by safety policy | +| `ingestion.safety_cooldown_entered` | Cooldown activated | +| `ingestion.safety_cooldown_cleared` | Cooldown expired | +| `ingestion.alert_blocked_failure_threshold_exceeded` | Blocked failure threshold tripped | +| `ingestion.alert_network_failure_threshold_exceeded` | Network failure threshold tripped | +| `ingestion.alert_retry_scheduled_threshold_exceeded` | Retry threshold tripped | +| `api.runs.manual_blocked_policy` | Manual run blocked by policy | +| `api.runs.manual_blocked_safety` | Manual run blocked by safety cooldown | +| `scheduler.run_skipped_safety_cooldown` | Scheduled run skipped due to cooldown | +| `scheduler.run_skipped_safety_cooldown_precheck` | Scheduler precheck blocked | +| `scheduler.queue_item_deferred_safety_cooldown` | Queue item deferred due to cooldown | -Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules. +Each event includes metric-style fields (`metric_name`, `metric_value`) for log-based alert rules. ## Recommended Alert Rules -- Cooldown enters: - - Trigger on `event=ingestion.safety_cooldown_entered`. -- Repeated start blocks: - - Trigger on high rate of `event=api.runs.manual_blocked_safety`. -- Threshold trips: - - Trigger on any of: - - `ingestion.alert_blocked_failure_threshold_exceeded` - - `ingestion.alert_network_failure_threshold_exceeded` - - `ingestion.alert_retry_scheduled_threshold_exceeded` -- Scheduler pressure: - - Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`. +- **Cooldown enters**: Trigger on `event=ingestion.safety_cooldown_entered` +- **Repeated start blocks**: High rate of `event=api.runs.manual_blocked_safety` +- **Threshold trips**: Any of the `*_threshold_exceeded` events +- **Scheduler pressure**: Sustained `scheduler.queue_item_deferred_safety_cooldown` ## If Your IP Appears Blocked -Symptoms: +### Symptoms -- cooldown reason `blocked_failure_threshold_exceeded` -- parse state `blocked_or_captcha` -- redirects toward Google account sign-in flows +- Cooldown reason: `blocked_failure_threshold_exceeded` +- Parse state: `blocked_or_captcha` +- Redirects toward Google account sign-in flows -Actions: +### Actions 1. Stop manual retries immediately. 2. Let cooldown expire; do not spam retriggers. 3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values. 4. Reduce concurrency pressure (keep one scheduler instance). -5. Keep name-search disabled/WIP for now if login-gated responses persist. +5. Keep name-search disabled if login-gated responses persist. 6. Resume with a small monitored run and verify blocked rate drops. -Avoid: +### Avoid -- aggressive rapid retries -- rotating through risky scraping patterns that increase challenge rates -- bypass/captcha-solving workflows that may violate source platform rules +- Aggressive rapid retries +- Rotating through risky scraping patterns that increase challenge rates +- Bypass/CAPTCHA-solving workflows that may violate source platform rules ## Environment Controls Policy floors and safety controls: -- `INGESTION_MIN_REQUEST_DELAY_SECONDS` -- `INGESTION_MIN_RUN_INTERVAL_MINUTES` -- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` -- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` -- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` -- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` -- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` -- `INGESTION_MANUAL_RUN_ALLOWED` -- `INGESTION_AUTOMATION_ALLOWED` +| Variable | Default | Description | +|----------|---------|-------------| +| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | `2` | Floor delay between requests | +| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | `15` | Minimum time between runs | +| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `1` | Blocked failures before alert | +| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `2` | Network failures before alert | +| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `3` | Retries before alert | +| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `1800` | Cooldown after blocked threshold (30 min) | +| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `900` | Cooldown after network threshold (15 min) | +| `INGESTION_MANUAL_RUN_ALLOWED` | `1` | Enable manual runs | +| `INGESTION_AUTOMATION_ALLOWED` | `1` | Enable automated runs | Apply stricter values first, then relax slowly only after sustained healthy runs. diff --git a/docs/reference/api-contract.md b/docs/reference/api-contract.md deleted file mode 100644 index 8b72186..0000000 --- a/docs/reference/api-contract.md +++ /dev/null @@ -1,31 +0,0 @@ -# API Contract - -## Envelope Invariant - -All API responses under `/api/v1` use one of these envelopes: - -- Success: `{"data": ..., "meta": {"request_id": "..."}}` -- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}` - -`meta.request_id` must be present for success and error responses. - -Binary media assets are served outside `/api/v1` (for example, `GET /scholar-images/{scholar_profile_id}/upload`). - -## Publications Semantics - -- `GET /api/v1/publications` supports `mode=all|unread|latest`. -- `mode=new` is currently accepted as a compatibility alias for `latest`. -- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support). -- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`. -- `unread` represents read-state (`is_read=false`). -- `latest` represents discovery-state (`first seen in the latest completed run`). - -Publication payloads expose: -- `pub_url` (canonical scholar detail URL) -- `doi` (normalized DOI) -- `pdf_url` (resolved OA PDF when available) - -## Scholar Portability - -- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state. -- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication. diff --git a/docs/reference/api.md b/docs/reference/api.md new file mode 100644 index 0000000..668ab51 --- /dev/null +++ b/docs/reference/api.md @@ -0,0 +1,177 @@ +--- +title: API Contract +sidebar_position: 2 +--- + +# API Contract + +## Envelope Invariant + +All API responses under `/api/v1` use one of these envelopes: + +### Success + +```json +{ + "data": "...", + "meta": { + "request_id": "..." + } +} +``` + +### Error + +```json +{ + "error": { + "code": "...", + "message": "...", + "details": "..." + }, + "meta": { + "request_id": "..." + } +} +``` + +`meta.request_id` is present on both success and error responses. + +### Binary Assets + +Binary media assets are served outside `/api/v1`: + +``` +GET /scholar-images/{scholar_profile_id}/upload +``` + +## DTO Structure + +Scholarr uses strictly typed Pydantic V2 models serialized through OpenAPI v3 via FastAPI. + +- `PublicationListItem` exposes a `.display_identifier` property that resolves the highest-confidence identifier regardless of backend origin, rather than a hardcoded `.doi`. +- Frontend TypeScript types are compiled from the OpenAPI spec to ensure type safety across the stack. + +## Endpoints + +### Auth + +| Method | Path | Description | +|--------|------|-------------| +| `POST` | `/api/v1/auth/login` | Login (rate-limited sliding window) | +| `GET` | `/api/v1/auth/me` | Current session user | +| `GET` | `/api/v1/auth/csrf` | Bootstrap CSRF token | +| `POST` | `/api/v1/auth/change-password` | Change password | +| `POST` | `/api/v1/auth/logout` | Logout | + +### Scholars + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/scholars` | List tracked scholars | +| `POST` | `/api/v1/scholars` | Create scholar (auto-enqueue, metadata hydration) | +| `GET` | `/api/v1/scholars/search` | Search author candidates by name | +| `PATCH` | `/api/v1/scholars/{id}/toggle` | Toggle enabled status | +| `DELETE` | `/api/v1/scholars/{id}` | Delete scholar | +| `PUT` | `/api/v1/scholars/{id}/image/url` | Update image URL | +| `POST` | `/api/v1/scholars/{id}/image/upload` | Upload image | +| `DELETE` | `/api/v1/scholars/{id}/image` | Clear image customization | + +### Scholar Portability + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/scholars/export` | Export tracked scholars and publication link state | +| `POST` | `/api/v1/scholars/import` | Import scholars with global publication deduplication | + +The export payload includes scholar metadata, tracked publication data, and link state (read/unread, favorites). Import preserves global deduplication. + +### Publications + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/publications` | List publications (filtered, paginated) | +| `POST` | `/api/v1/publications/mark-all-read` | Mark all as read | +| `POST` | `/api/v1/publications/mark-read` | Mark selected as read | +| `POST` | `/api/v1/publications/{id}/retry-pdf` | Retry PDF resolution | +| `POST` | `/api/v1/publications/{id}/favorite` | Toggle favorite | + +#### Publication Modes + +`GET /api/v1/publications` supports a `mode` parameter: + +| Mode | Description | +|------|-------------| +| `all` | All publications | +| `unread` | Publications with `is_read=false` | +| `latest` | Publications first seen in the latest completed run | + +`mode=new` is accepted as a compatibility alias for `latest`. + +#### Pagination + +Query parameters: `page`, `page_size` (with backward-compatible `limit`/`offset` support). + +Response pagination fields: + +```json +{ + "page": 1, + "page_size": 20, + "has_prev": false, + "has_next": true, + "total_count": 142 +} +``` + +#### Publication Payload Fields + +| Field | Description | +|-------|-------------| +| `pub_url` | Canonical scholar detail URL | +| `doi` | Normalized DOI | +| `pdf_url` | Resolved open-access PDF URL (when available) | +| `display_identifier` | Highest-confidence identifier regardless of source | + +### Runs + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/runs` | List runs with safety state | +| `GET` | `/api/v1/runs/{run_id}` | Run detail with scholar results | +| `POST` | `/api/v1/runs/{run_id}/cancel` | Cancel active run | +| `POST` | `/api/v1/runs/manual` | Trigger manual run (idempotent, safety-checked) | +| `GET` | `/api/v1/runs/queue/items` | List queue items | +| `POST` | `/api/v1/runs/queue/{id}/retry` | Retry queue item | +| `POST` | `/api/v1/runs/queue/{id}/drop` | Drop queue item | +| `DELETE` | `/api/v1/runs/queue/{id}` | Clear queue item | +| `GET` | `/api/v1/runs/{run_id}/stream` | Stream run events (SSE) | + +### Settings + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/settings` | Get user settings (with cooldown expiry check) | +| `PUT` | `/api/v1/settings` | Update settings (interval, delay, nav, API keys) | + +### Admin - User Management + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/admin/users` | List all users | +| `POST` | `/api/v1/admin/users` | Create user | +| `PATCH` | `/api/v1/admin/users/{id}/active` | Set user active status | +| `POST` | `/api/v1/admin/users/{id}/reset-password` | Reset password | + +### Admin - Database Operations + +| Method | Path | Description | +|--------|------|-------------| +| `GET` | `/api/v1/admin/db/integrity` | Get integrity report | +| `GET` | `/api/v1/admin/db/repair-jobs` | List repair jobs | +| `GET` | `/api/v1/admin/db/pdf-queue` | List PDF queue | +| `POST` | `/api/v1/admin/db/pdf-queue/{id}/requeue` | Requeue single PDF | +| `POST` | `/api/v1/admin/db/pdf-queue/requeue-all` | Bulk requeue missing PDFs | +| `POST` | `/api/v1/admin/db/repairs/publication-links` | Trigger link repair | +| `POST` | `/api/v1/admin/db/repairs/publication-near-duplicates` | Trigger dedup repair | +| `POST` | `/api/v1/admin/db/drop-all-publications` | Drop all publications (destructive) | diff --git a/docs/reference/changelog.md b/docs/reference/changelog.md new file mode 100644 index 0000000..23a7a11 --- /dev/null +++ b/docs/reference/changelog.md @@ -0,0 +1,12 @@ +--- +title: Changelog +sidebar_position: 4 +--- + +# Changelog + +This changelog is auto-generated by [python-semantic-release](https://python-semantic-release.readthedocs.io/). + +Release versions follow [Semantic Versioning](https://semver.org/). Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/). + + diff --git a/docs/reference/environment.md b/docs/reference/environment.md index 75cc192..fc7e29a 100644 --- a/docs/reference/environment.md +++ b/docs/reference/environment.md @@ -1,68 +1,32 @@ -# Environment Reference and Audit +--- +title: Environment Variables +sidebar_position: 3 +--- -This document is the source-of-truth audit for what remains configurable versus internal defaults. +# Environment Variables Quick Reference -## Decision Summary +All environment variables are documented in detail in [Configuration](../user/configuration.md). -- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras. -- Move to internal defaults only: none at this time. -- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted. +## Required Variables -## Configurable Variables (By Section) +| Variable | Description | +|----------|-------------| +| `POSTGRES_PASSWORD` | PostgreSQL password | +| `SESSION_SECRET_KEY` | Session cookie signing key (32+ characters) | -### Compose + Database +## Categories -`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE` +| Category | Key Variables | +|----------|--------------| +| Database | `POSTGRES_DB`, `POSTGRES_USER`, `DATABASE_URL`, `DATABASE_POOL_*` | +| Runtime | `APP_HOST`, `APP_PORT`, `MIGRATE_ON_START`, `FRONTEND_ENABLED` | +| Auth | `SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_*` | +| Security | `SECURITY_HEADERS_ENABLED`, `SECURITY_CSP_*`, `SECURITY_STRICT_TRANSPORT_*` | +| Logging | `LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS` | +| Scheduler | `SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_*_BATCH_SIZE` | +| Ingestion | `INGESTION_*` (safety floors, cooldowns, retry policies) | +| Scholar | `SCHOLAR_IMAGE_*`, `SCHOLAR_NAME_SEARCH_*` | +| Enrichment | `UNPAYWALL_*`, `ARXIV_*`, `CROSSREF_*`, `OPENALEX_*`, `PDF_AUTO_RETRY_*` | +| Bootstrap | `BOOTSTRAP_ADMIN_*`, `DB_WAIT_*` | -### App Runtime + Networking - -`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR` - -### Database Pool - -`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS` - -### Frontend Dev Overrides - -`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET` - -### Auth + Session - -`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS` - -### HTTP Security Headers + CSP - -`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD` - -### Logging - -`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS` - -### Scheduler + Ingestion Safety - -`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `SCHEDULER_PDF_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_RATE_LIMIT_RETRIES`, `INGESTION_RATE_LIMIT_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS` - -### Scholar Images + Name Search Safety - -`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` - -### OA Enrichment + PDF Resolution - -`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `ARXIV_ENABLED`, `ARXIV_TIMEOUT_SECONDS`, `ARXIV_MIN_INTERVAL_SECONDS`, `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS`, `ARXIV_DEFAULT_MAX_RESULTS`, `ARXIV_CACHE_TTL_SECONDS`, `ARXIV_CACHE_MAX_ENTRIES`, `ARXIV_MAILTO`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`, `OPENALEX_API_KEY`, `CROSSREF_API_TOKEN`, `CROSSREF_API_MAILTO` - -### Startup Bootstrap + DB Wait - -`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS` - -## Drift Guard - -CI enforces env parity with: - -```bash -python3 scripts/check_env_contract.py -``` - -The check fails when: -- `app/settings.py` references a variable missing from `.env.example`. -- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras). -- `.env.example` contains duplicate keys. +See [Configuration](../user/configuration.md) for the complete table with types, defaults, and descriptions. diff --git a/docs/reference/overview.md b/docs/reference/overview.md index 240683d..c8249d9 100644 --- a/docs/reference/overview.md +++ b/docs/reference/overview.md @@ -1,6 +1,14 @@ -# Reference Documentation +--- +title: Reference +sidebar_position: 1 +--- -Use this section for canonical contracts and configuration references. +# Reference -- [API Contract](./api-contract.md) -- [Environment Reference](./environment.md) +Quick-reference documentation for the scholarr API, environment variables, and release changelog. + +| Document | Description | +|----------|-------------| +| [API Contract](api.md) | Envelope spec, endpoints, DTO structure, publication semantics | +| [Environment Variables](environment.md) | Quick-reference table linking to full configuration docs | +| [Changelog](changelog.md) | Auto-generated release history | diff --git a/docs/user/configuration.md b/docs/user/configuration.md new file mode 100644 index 0000000..d79a1b7 --- /dev/null +++ b/docs/user/configuration.md @@ -0,0 +1,178 @@ +--- +title: Configuration +sidebar_position: 3 +--- + +# Configuration + +All configuration is done through environment variables. Copy `.env.example` to `.env` and adjust values as needed. + +## Compose & Database + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `POSTGRES_DB` | string | `scholar` | PostgreSQL database name | +| `POSTGRES_USER` | string | `scholar` | PostgreSQL user | +| `POSTGRES_PASSWORD` | string | **required** | PostgreSQL password | +| `DATABASE_URL` | string | derived | SQLAlchemy async connection string | +| `TEST_DATABASE_URL` | string | derived | Override for test database. If empty, tests derive `_test` | +| `SCHOLARR_IMAGE` | string | `justinzeus/scholarr:latest` | Docker image for the app service | + +## App Runtime & Networking + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `APP_NAME` | string | `scholarr` | Application name used in logs and headers | +| `APP_HOST` | string | `0.0.0.0` | Bind address | +| `APP_PORT` | int | `8000` | Internal port | +| `APP_HOST_PORT` | int | `8000` | Host-mapped port | +| `APP_RELOAD` | bool | `0` | Enable uvicorn auto-reload (dev only) | +| `MIGRATE_ON_START` | bool | `1` | Run Alembic migrations on startup | +| `FRONTEND_ENABLED` | bool | `1` | Serve the built Vue frontend | +| `FRONTEND_DIST_DIR` | string | `/app/frontend/dist` | Path to compiled frontend assets | + +## Database Pool + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `DATABASE_POOL_MODE` | string | `auto` | Pool mode (`auto`, `fixed`, `null`) | +| `DATABASE_POOL_SIZE` | int | `5` | Base pool size | +| `DATABASE_POOL_MAX_OVERFLOW` | int | `10` | Maximum overflow connections | +| `DATABASE_POOL_TIMEOUT_SECONDS` | int | `30` | Connection acquisition timeout | + +## Frontend Dev Overrides + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `FRONTEND_HOST_PORT` | int | `5173` | Host port for Vite dev server | +| `CHOKIDAR_USEPOLLING` | bool | `1` | Enable polling for file watchers in containers | +| `VITE_DEV_API_PROXY_TARGET` | string | `http://app:8000` | Backend URL for Vite proxy | + +## Auth & Session + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `SESSION_SECRET_KEY` | string | **required** | Signing key for session cookies (32+ chars) | +| `SESSION_COOKIE_SECURE` | bool | `1` | Set `Secure` flag on session cookie (disable for local HTTP dev) | +| `LOGIN_RATE_LIMIT_ATTEMPTS` | int | `5` | Max login attempts per window | +| `LOGIN_RATE_LIMIT_WINDOW_SECONDS` | int | `60` | Sliding window for login rate limiting | + +## HTTP Security Headers & CSP + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `SECURITY_HEADERS_ENABLED` | bool | `1` | Enable security response headers | +| `SECURITY_X_CONTENT_TYPE_OPTIONS` | string | `nosniff` | X-Content-Type-Options header value | +| `SECURITY_X_FRAME_OPTIONS` | string | `DENY` | X-Frame-Options header value | +| `SECURITY_REFERRER_POLICY` | string | `strict-origin-when-cross-origin` | Referrer-Policy header value | +| `SECURITY_PERMISSIONS_POLICY` | string | *(restrictive)* | Permissions-Policy header value | +| `SECURITY_CROSS_ORIGIN_OPENER_POLICY` | string | `same-origin` | Cross-Origin-Opener-Policy header | +| `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY` | string | `same-origin` | Cross-Origin-Resource-Policy header | +| `SECURITY_CSP_ENABLED` | bool | `1` | Enable Content-Security-Policy header | +| `SECURITY_CSP_POLICY` | string | *(restrictive)* | CSP for app routes | +| `SECURITY_CSP_DOCS_POLICY` | string | *(docs-specific)* | CSP for documentation routes | +| `SECURITY_CSP_REPORT_ONLY` | bool | `0` | Use report-only mode for CSP | +| `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED` | bool | `0` | Enable HSTS header | +| `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE` | int | `31536000` | HSTS max-age in seconds | +| `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS` | bool | `1` | HSTS includeSubDomains directive | +| `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD` | bool | `0` | HSTS preload directive | + +## Logging + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `LOG_LEVEL` | string | `INFO` | Root log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`) | +| `LOG_FORMAT` | string | `console` | Log format (`console` or `json`) | +| `LOG_REQUESTS` | bool | `1` | Log HTTP requests | +| `LOG_UVICORN_ACCESS` | bool | `0` | Enable uvicorn access log | +| `LOG_REQUEST_SKIP_PATHS` | string | `/healthz` | Comma-separated paths to exclude from request logging | +| `LOG_REDACT_FIELDS` | string | *(empty)* | Comma-separated field names to redact in logs | + +## Scheduler & Ingestion Safety + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `SCHEDULER_ENABLED` | bool | `1` | Enable the background scheduler | +| `SCHEDULER_TICK_SECONDS` | int | `60` | Scheduler poll interval | +| `SCHEDULER_QUEUE_BATCH_SIZE` | int | `10` | Max scholars processed per tick | +| `SCHEDULER_PDF_QUEUE_BATCH_SIZE` | int | `15` | Max PDF resolutions per tick | +| `INGESTION_AUTOMATION_ALLOWED` | bool | `1` | Allow automated (scheduled) runs | +| `INGESTION_MANUAL_RUN_ALLOWED` | bool | `1` | Allow manually triggered runs | +| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | int | `15` | Minimum time between runs | +| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | int | `2` | Floor delay between external requests | +| `INGESTION_NETWORK_ERROR_RETRIES` | int | `1` | Retries on network errors | +| `INGESTION_RETRY_BACKOFF_SECONDS` | float | `1.0` | Base backoff for network retries | +| `INGESTION_RATE_LIMIT_RETRIES` | int | `3` | Retries on 429 responses | +| `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` | float | `30.0` | Backoff per 429 retry | +| `INGESTION_MAX_PAGES_PER_SCHOLAR` | int | `30` | Max paginated pages per scholar | +| `INGESTION_PAGE_SIZE` | int | `100` | Publications per page | +| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | int | `1` | Blocked failures before alert | +| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | int | `2` | Network failures before alert | +| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | int | `3` | Scheduled retries before alert | +| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | int | `1800` | Cooldown after blocked-failure threshold (30 min) | +| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | int | `900` | Cooldown after network-failure threshold (15 min) | +| `INGESTION_CONTINUATION_QUEUE_ENABLED` | bool | `1` | Enable continuation queue for multi-page ingestion | +| `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` | int | `120` | Base delay for continuation queue items | +| `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` | int | `3600` | Max delay for continuation queue items | +| `INGESTION_CONTINUATION_MAX_ATTEMPTS` | int | `6` | Max continuation attempts per scholar | + +## Scholar Images & Name Search Safety + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `SCHOLAR_IMAGE_UPLOAD_DIR` | string | `/var/lib/scholarr/uploads` | Directory for uploaded scholar images | +| `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES` | int | `2000000` | Max image upload size (2 MB) | +| `SCHOLAR_NAME_SEARCH_ENABLED` | bool | `1` | Enable name-based scholar search | +| `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS` | int | `21600` | Cache TTL for successful searches (6 hours) | +| `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS` | int | `300` | Cache TTL for blocked search results (5 min) | +| `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES` | int | `512` | Max cache entries for name search | +| `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS` | float | `8.0` | Min interval between name searches | +| `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS` | float | `2.0` | Random jitter added to search interval | +| `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD` | int | `1` | Blocked results before cooldown | +| `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS` | int | `1800` | Cooldown after blocked name search (30 min) | +| `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD` | int | `2` | Retries before alert | +| `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` | int | `3` | Cooldown rejections before alert | + +## OA Enrichment & PDF Resolution + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `UNPAYWALL_ENABLED` | bool | `1` | Enable Unpaywall DOI lookups | +| `UNPAYWALL_EMAIL` | string | *(empty)* | Polite pool email for Unpaywall API | +| `UNPAYWALL_TIMEOUT_SECONDS` | float | `4.0` | Request timeout | +| `UNPAYWALL_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Unpaywall requests | +| `UNPAYWALL_MAX_ITEMS_PER_REQUEST` | int | `20` | Max items per batch | +| `UNPAYWALL_RETRY_COOLDOWN_SECONDS` | int | `1800` | Cooldown after repeated failures | +| `UNPAYWALL_PDF_DISCOVERY_ENABLED` | bool | `1` | Enable HTML-based PDF link discovery | +| `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES` | int | `5` | Max candidate URLs to probe | +| `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES` | int | `500000` | Max HTML response size to parse | +| `ARXIV_ENABLED` | bool | `1` | Enable arXiv API lookups | +| `ARXIV_TIMEOUT_SECONDS` | float | `3.0` | Request timeout | +| `ARXIV_MIN_INTERVAL_SECONDS` | float | `4.0` | Min interval between arXiv requests | +| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | float | `60.0` | Cooldown after arXiv 429 | +| `ARXIV_DEFAULT_MAX_RESULTS` | int | `3` | Default max results per query | +| `ARXIV_CACHE_TTL_SECONDS` | int | `900` | Query cache TTL (15 min) | +| `ARXIV_CACHE_MAX_ENTRIES` | int | `512` | Max cached queries | +| `ARXIV_MAILTO` | string | *(empty)* | Contact email for arXiv API headers | +| `PDF_AUTO_RETRY_INTERVAL_SECONDS` | int | `86400` | Auto-retry interval for failed PDFs (24 hours) | +| `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS` | int | `3600` | First retry interval (1 hour) | +| `PDF_AUTO_RETRY_MAX_ATTEMPTS` | int | `3` | Max auto-retry attempts | +| `CROSSREF_ENABLED` | bool | `1` | Enable Crossref lookups | +| `CROSSREF_MAX_ROWS` | int | `10` | Max rows per Crossref query | +| `CROSSREF_TIMEOUT_SECONDS` | float | `8.0` | Request timeout | +| `CROSSREF_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Crossref requests | +| `CROSSREF_MAX_LOOKUPS_PER_REQUEST` | int | `8` | Max lookups per ingestion request | +| `OPENALEX_API_KEY` | string | *(empty)* | OpenAlex API key (optional) | +| `CROSSREF_API_TOKEN` | string | *(empty)* | Crossref Plus API token (optional) | +| `CROSSREF_API_MAILTO` | string | *(empty)* | Crossref polite pool email | + +## Startup Bootstrap & DB Wait + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `BOOTSTRAP_ADMIN_ON_START` | bool | `0` | Create admin user on startup | +| `BOOTSTRAP_ADMIN_EMAIL` | string | *(empty)* | Admin email address | +| `BOOTSTRAP_ADMIN_PASSWORD` | string | *(empty)* | Admin password | +| `BOOTSTRAP_ADMIN_FORCE_PASSWORD` | bool | `0` | Overwrite existing admin password | +| `DB_WAIT_TIMEOUT_SECONDS` | int | `60` | Max seconds to wait for database readiness | +| `DB_WAIT_INTERVAL_SECONDS` | int | `2` | Poll interval while waiting for database | diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md index bc15032..1090f86 100644 --- a/docs/user/getting-started.md +++ b/docs/user/getting-started.md @@ -1,41 +1,86 @@ -# User Getting Started +--- +title: Getting Started +sidebar_position: 2 +--- -## Required Setup +# Getting Started -1. Copy `.env.example` to `.env`. -2. Set `POSTGRES_PASSWORD`. -3. Set `SESSION_SECRET_KEY`. +## Prerequisites -## Deploy Method A: Prebuilt Image +- Docker and Docker Compose v2+ +- A machine with at least 512 MB RAM + +## Installation + +1. Clone the repository: + +```bash +git clone https://github.com/JustinZeus/scholarr.git +cd scholarr +``` + +2. Copy the example environment file: + +```bash +cp .env.example .env +``` + +3. Edit `.env` and set the required secrets: + +```bash +# Required: database password +POSTGRES_PASSWORD=your-secure-password + +# Required: session signing key (32+ random characters) +SESSION_SECRET_KEY=your-random-secret-key +``` + +4. Start the stack: + +```bash +docker compose up -d +``` + +5. Verify the service is healthy: + +```bash +docker compose ps +curl http://localhost:8000/healthz +``` + +The app is now available at `http://localhost:8000`. + +## First Run + +### Bootstrap an Admin User + +Set these environment variables before first start (or add them to `.env`): + +```bash +BOOTSTRAP_ADMIN_ON_START=1 +BOOTSTRAP_ADMIN_EMAIL=admin@example.com +BOOTSTRAP_ADMIN_PASSWORD=your-admin-password +``` + +Restart the app container. The admin account is created on startup. + +### Add Your First Scholar + +1. Log in at `http://localhost:8000`. +2. Navigate to the Scholars page. +3. Click **Add Scholar**. +4. Enter a Google Scholar profile URL (e.g., `https://scholar.google.com/citations?user=XXXXXXXXXX`) or a Scholar ID. +5. The scheduler will begin fetching publications on the next tick (default: 60 seconds). + +### Manual Run + +To trigger an immediate ingestion run, use the **Manual Run** button on the Runs page. This respects the minimum run interval (`INGESTION_MIN_RUN_INTERVAL_MINUTES`, default 15 minutes). + +## Updating ```bash docker compose pull docker compose up -d ``` -Open: -- App/API: `http://localhost:8000` -- Health endpoint: `http://localhost:8000/healthz` - -Upgrade: - -```bash -docker compose pull -docker compose up -d -``` - -## Deploy Method B: Local Source + Dev Frontend - -```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build -``` - -Open: -- API: `http://localhost:8000` -- Frontend dev server: `http://localhost:5173` - -Stop: - -```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml down -``` +Database migrations run automatically on startup when `MIGRATE_ON_START=1` (the default). diff --git a/docs/user/overview.md b/docs/user/overview.md index 29ee003..50af970 100644 --- a/docs/user/overview.md +++ b/docs/user/overview.md @@ -1,5 +1,29 @@ -# User Documentation +--- +title: Overview +sidebar_position: 1 +--- -Use this section if you are deploying and using scholarr. +# What is Scholarr? -- [Getting Started](./getting-started.md) +Scholarr is a self-hosted service that tracks academic publications from Google Scholar. It runs as a Docker container with a PostgreSQL database and serves a Vue 3 frontend. + +## Key Concepts + +- **Scholar** - A tracked Google Scholar profile. Scholars are user-scoped: each user manages their own list. +- **Publication** - A globally deduplicated academic work. Publications are shared across all users to avoid duplicate storage. +- **Scholar-Publication Link** - Connects a scholar to a publication for a specific user. Read/unread state, favorites, and visibility live on this link, not on the publication itself. +- **Run** - A single ingestion cycle that fetches new publications for one or more scholars. +- **Identifier** - A DOI, arXiv ID, PMID, or other external key attached to a publication. Multiple identifiers can exist per publication. + +## How It Works + +1. You add a Google Scholar profile (by ID, URL, or name search). +2. The scheduler periodically scrapes the profile for new publications. +3. Each publication is fingerprinted and deduplicated against the global store. +4. External APIs (arXiv, Crossref, OpenAlex) are queried for identifiers. +5. Unpaywall and arXiv resolve open-access PDF URLs when a DOI is available. +6. The dashboard shows new, unread, and all publications with filtering and search. + +## Safety Model + +Scholarr enforces rate limits and cooldowns to prevent IP bans from upstream sources. These are not configurable to zero; they are safety floors. See [Scrape Safety Runbook](../operations/scrape-safety-runbook.md) for details. diff --git a/docs/website/docusaurus.config.js b/docs/website/docusaurus.config.js index f32aeaa..6156ad9 100644 --- a/docs/website/docusaurus.config.js +++ b/docs/website/docusaurus.config.js @@ -2,8 +2,8 @@ /** @type {import('@docusaurus/types').Config} */ const config = { - title: "scholarr", - tagline: "Self-hosted scholar tracking docs", + title: "Scholarr", + tagline: "Self-hosted academic publication tracker", favicon: "img/favicon.ico", url: "https://justinzeus.github.io", @@ -23,59 +23,52 @@ const config = { presets: [ [ "classic", - { + /** @type {import('@docusaurus/preset-classic').Options} */ + ({ docs: { path: "..", - exclude: ["website/**", "README.md"], routeBasePath: "/", sidebarPath: require.resolve("./sidebars.js"), - editUrl: "https://github.com/JustinZeus/scholarr/tree/main/", + exclude: ["website/**", "README.md"], }, blog: false, theme: { customCss: require.resolve("./src/css/custom.css"), }, - }, + }), ], ], - themeConfig: { - navbar: { - title: "scholarr", - items: [ - { - type: "docSidebar", - sidebarId: "docsSidebar", - position: "left", - label: "Docs", - }, - { - href: "https://github.com/JustinZeus/scholarr", - label: "GitHub", - position: "right", - }, - ], - }, - footer: { - style: "dark", - links: [ - { - title: "Docs", - items: [{ label: "Index", to: "/" }], - }, - { - title: "Community", - items: [ - { - label: "GitHub", - href: "https://github.com/JustinZeus/scholarr", - }, - ], - }, - ], - copyright: `Copyright © ${new Date().getFullYear()} scholarr.`, - }, - }, + themeConfig: + /** @type {import('@docusaurus/preset-classic').ThemeConfig} */ + ({ + navbar: { + title: "Scholarr", + items: [ + { to: "/user/overview", label: "User Guide", position: "left" }, + { + to: "/developer/overview", + label: "Developer", + position: "left", + }, + { + to: "/operations/overview", + label: "Operations", + position: "left", + }, + { to: "/reference/overview", label: "Reference", position: "left" }, + { + href: "https://github.com/JustinZeus/scholarr", + label: "GitHub", + position: "right", + }, + ], + }, + footer: { + style: "dark", + copyright: `Copyright \u00a9 ${new Date().getFullYear()} Scholarr. Built with Docusaurus.`, + }, + }), }; module.exports = config; diff --git a/docs/website/package.json b/docs/website/package.json index 383161f..85208b0 100644 --- a/docs/website/package.json +++ b/docs/website/package.json @@ -1,18 +1,27 @@ { "name": "scholarr-docs", - "version": "0.0.1", + "version": "0.1.0", "private": true, "scripts": { + "docusaurus": "docusaurus", "start": "docusaurus start", "build": "docusaurus build", + "swizzle": "docusaurus swizzle", + "deploy": "docusaurus deploy", + "clear": "docusaurus clear", "serve": "docusaurus serve" }, "dependencies": { - "@docusaurus/core": "3.9.2", - "@docusaurus/preset-classic": "3.9.2", - "clsx": "^2.1.1", - "prism-react-renderer": "^2.3.1", + "@docusaurus/core": "^3.0.0", + "@docusaurus/preset-classic": "^3.0.0", "react": "^18.2.0", "react-dom": "^18.2.0" + }, + "devDependencies": { + "@docusaurus/module-type-aliases": "^3.0.0" + }, + "browserslist": { + "production": [">0.5%", "not dead", "not op_mini all"], + "development": ["last 1 chrome version", "last 1 firefox version", "last 1 safari version"] } } diff --git a/docs/website/sidebars.js b/docs/website/sidebars.js index 3189871..2b97349 100644 --- a/docs/website/sidebars.js +++ b/docs/website/sidebars.js @@ -1,38 +1,49 @@ /** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */ const sidebars = { - docsSidebar: [ + docs: [ "index", { type: "category", - label: "Users", - items: ["user/overview", "user/getting-started"], + label: "User Guide", + items: [ + "user/overview", + "user/getting-started", + "user/configuration", + ], + }, + { + type: "category", + label: "Developer", + items: [ + "developer/overview", + "developer/architecture", + "developer/local-development", + "developer/contributing", + "developer/ingestion", + "developer/frontend-theme-inventory", + "developer/testing", + ], }, { type: "category", label: "Operations", items: [ "operations/overview", - "operations/scrape-safety-runbook", + "operations/deployment", "operations/database-runbook", - "operations/migration-checklist", - ], - }, - { - type: "category", - label: "Developers", - items: [ - "developer/overview", - "developer/documentation-standards", - "developer/local-development", - "developer/architecture", - "developer/contributing", - "developer/frontend-theme-inventory", + "operations/scrape-safety-runbook", + "operations/arxiv-runbook", ], }, { type: "category", label: "Reference", - items: ["reference/overview", "reference/api-contract", "reference/environment"], + items: [ + "reference/overview", + "reference/api", + "reference/environment", + "reference/changelog", + ], }, ], }; diff --git a/docs/website/src/css/custom.css b/docs/website/src/css/custom.css index 6f9b7ae..c875358 100644 --- a/docs/website/src/css/custom.css +++ b/docs/website/src/css/custom.css @@ -1,9 +1,22 @@ :root { - --ifm-color-primary: #2a6b4e; - --ifm-color-primary-dark: #255f46; - --ifm-color-primary-darker: #225942; - --ifm-color-primary-darkest: #1c4936; - --ifm-color-primary-light: #2f7756; - --ifm-color-primary-lighter: #327d5a; - --ifm-color-primary-lightest: #3b946b; + --ifm-color-primary: #2e8555; + --ifm-color-primary-dark: #29784c; + --ifm-color-primary-darker: #277148; + --ifm-color-primary-darkest: #205d3b; + --ifm-color-primary-light: #33925d; + --ifm-color-primary-lighter: #359962; + --ifm-color-primary-lightest: #3cad6e; + --ifm-code-font-size: 95%; + --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1); +} + +[data-theme="dark"] { + --ifm-color-primary: #25c2a0; + --ifm-color-primary-dark: #21af90; + --ifm-color-primary-darker: #1fa588; + --ifm-color-primary-darkest: #1a8870; + --ifm-color-primary-light: #29d5b0; + --ifm-color-primary-lighter: #32d8b4; + --ifm-color-primary-lightest: #4fddbf; + --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3); } diff --git a/docs/website/static/img/favicon.ico b/docs/website/static/img/favicon.ico deleted file mode 100644 index 783fa02..0000000 Binary files a/docs/website/static/img/favicon.ico and /dev/null differ