justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

docs: rebuild documentation from scratch with clean IA

Browse source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 09:39:27 +01:00
parent 6c31b331f7
commit 0c5b199b76
30 changed files with 1647 additions and 649 deletions
Download patch file Download diff file Expand all files Collapse all files

23
docs/README.md
View file

@ -1,23 +0,0 @@
# Documentation Index
This directory contains both docs content and docs tooling.
## Audience-Based Structure
- `docs/user/`: onboarding and usage-oriented docs for self-hosters.
- `docs/operations/`: runbooks and rollout checklists for ongoing operations.
- `docs/developer/`: architecture, contribution standards, and local development workflows.
- `docs/reference/`: canonical API and environment contracts.
## Docs Tooling
- `docs/website/`: Docusaurus app used to publish docs to GitHub Pages.
## Recommended Reading Paths
- Users: `docs/user/getting-started.md`
- Operators: `docs/operations/overview.md`
- Developers: `docs/developer/overview.md`
- Contract consumers: `docs/reference/overview.md`
Published site: https://justinzeus.github.io/scholarr/

12
docs/developer/api-contract.md
View file

@ -1,12 +0,0 @@
# API Contracts
`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI.
## DTO Models Structure
FastAPI routes dynamically ingest request/response Data Transfer Objects.
- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin.
- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend.
## Frontend Contract Requirements
The UI exclusively calls routes exposed by FastAPI's `APIRouter`.
All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally.

169
docs/developer/architecture.md
View file

@ -1,36 +1,147 @@
# Domain Boundaries ---
title: Architecture
sidebar_position: 2
---
# Architecture
## Data Model Rules ## Data Model Rules
- Scholar tracking is user-scoped. - **Scholar tracking is user-scoped.** Each user manages their own list of tracked scholars. Validate mapping/join tables; never assume global links between users and Scholar IDs.
- Publications are global/deduplicated records. - **Publications are global, deduplicated records.** Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
- Read/favorite/visibility state stays on scholar-publication link rows. - **State lives on the link.** Read/unread, favorites, and visibility state exist exclusively on the scholar-publication link table, not the global publication table.
## Service Boundaries ## Domain Service Boundaries
Canonical business logic belongs in `app/services/domains/*`. All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited.
- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety. ### Ingestion (`app/services/ingestion/`)
- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
- `app/services/domains/runs/*`: run history and continuation queue operations.
- `app/services/domains/portability/*`: import/export workflows.
## Frontend Behavior Notes Run orchestration, continuation queue, scheduler integration, and scrape safety policy enforcement. The `ScholarIngestionService` drives the primary data acquisition loop.
- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout. Key modules:
- Lists: Long list views use internal scroll containers to prevent viewport overflow. - `application.py` - Main ingestion orchestrator
- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible. - `scheduler.py` - Background tick loop, queue batch processing
- `constants.py` - Safety policy constants and floor values
- `fingerprints.py` - Publication fingerprinting for deduplication
- `types.py` - Ingestion result types and state enums
## Data Integration & Acquisition Flow ### Scholar Parsing (`app/services/scholar/`)
Fail-fast Google Scholar HTML parsing and source fetch adapters. Handles paginated HTML feeds, extracts publication blocks via regex and DOM invariants (e.g., `gsc_vcd_cib`).
Key modules:
- `parser.py` - HTML parser for publication extraction
- `parser_utils.py` - Parsing helpers and DOM selectors
- `source.py` - HTTP fetch adapters with browser headers
- `profile_rows.py` - Profile metadata extraction
- `author_rows.py` - Author citation row parsing
- `state_detection.py` - Blocked/CAPTCHA/rate-limit detection
### Scholar Management (`app/services/scholars/`)
Scholar CRUD, profile image management, and name-search controls with rate limiting.
Key modules:
- `application.py` - Scholar lifecycle operations
- `uploads.py` - Image upload handling
- `search_hints.py` - Name search with caching and cooldowns
- `validators.py` - Input validation for scholar creation
### Publications (`app/services/publications/`)
Listing, read-state management, favorite toggles, enrichment scheduling, deduplication, and PDF queue management.
Key modules:
- `application.py` - Publication service facade
- `listing.py` - Filtered listing with pagination (modes: all/unread/latest)
- `queries.py` - Database query builders
- `counts.py` - Aggregation counts for dashboard
- `dedup.py` - Duplicate detection and merging
- `enrichment.py` - Identifier and metadata enrichment orchestration
- `pdf_queue.py` - PDF resolution queue policy
- `pdf_resolution_pipeline.py` - Multi-source PDF resolution (Unpaywall, arXiv)
- `types.py` - Publication DTOs and response types
### Publication Identifiers (`app/services/publication_identifiers/`)
Multi-identifier resolution engine. A single publication can have multiple identifiers (DOI, arXiv ID, PMID, PMCID). This decoupled approach replaced the earlier hardcoded DOI-only model.
Key modules:
- `application.py` - Identifier gathering orchestration
- `normalize.py` - Identifier normalization and validation
### arXiv (`app/services/arxiv/`)
Typed API client with global DB-backed throttle, query cache, and in-flight request coalescing.
Key modules:
- `client.py` - HTTP client for arXiv export API
- `gateway.py` - Gateway with advisory lock, caching, and coalescing
- `cache.py` - Query cache with TTL and max-entry pruning
- `rate_limit.py` - Global rate limiter via `arxiv_runtime_state` table
- `guards.py` - Load-shedding guards (skip when DOI/arXiv evidence exists)
- `parser.py` - Atom XML response parser
Safety features:
- Requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`)
- Identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + max-entry pruning
- Concurrent identical misses are coalesced in-process (one outbound call serves all waiters)
### Crossref (`app/services/crossref/`)
DOI lookup via Crossref REST API with bounded pacing and configurable batch limits.
### Unpaywall (`app/services/unpaywall/`)
Open-access PDF resolution via Unpaywall API, with HTML-based PDF link discovery as a fallback.
Key modules:
- `application.py` - Unpaywall service facade
- `pdf_discovery.py` - HTML page scraping for PDF link candidates
### OpenAlex (`app/services/openalex/`)
Metadata matching via OpenAlex API for supplementary identifier resolution.
Key modules:
- `client.py` - OpenAlex API client
- `matching.py` - Fuzzy title/author matching
### Runs (`app/services/runs/`)
Run history tracking and continuation queue operations.
Key modules:
- `application.py` - Run lifecycle management
- `queue_service.py` - Continuation queue operations (retry, drop, clear)
- `queue_queries.py` - Queue item queries
### Portability (`app/services/portability/`)
Import/export workflows for scholar data with full publication state preservation.
Key modules:
- `application.py` - Import/export orchestration
- `exporting.py` - Scholar export serialization
- `publication_import.py` - Publication import with deduplication
- `scholar_import.py` - Scholar import with link reconstruction
- `normalize.py` - Payload normalization and validation
### Database Operations (`app/services/dbops/`)
Integrity checking, link repair, and near-duplicate repair operations exposed via admin API and CLI scripts.
Key modules:
- `__init__.py` - `collect_integrity_report`, `run_publication_link_repair`
- `near_duplicate_repair.py` - Near-duplicate publication detection and merging
## Data Integration Flow
```mermaid ```mermaid
graph TD graph TD
UI[Frontend Vue App] --> API[FastAPI Backend] UI[Frontend Vue App] --> API[FastAPI Backend]
API --> Scheduler[Background Celery/Async Scheduler] API --> Scheduler[Background Async Scheduler]
Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser] Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering] Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
@ -48,11 +159,15 @@ graph TD
PDFWorker --> |Store PDF Metadata| DB PDFWorker --> |Store PDF Metadata| DB
``` ```
### Identifier Engine Philosophy ## API Layer
The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.
### arXiv Safety and Efficiency Routes live in `app/api/routers/`. All responses under `/api/v1` use a strict envelope format. See [API Reference](../reference/api.md) for the full contract.
- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`).
- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning. ## Middleware Stack
- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
- structured logs provide auditability for scheduling/completion/cooldown/cache behavior. Applied in `app/main.py`:
1. **CSRF Protection** - Token-based CSRF validation
2. **Session Middleware** - Cookie-based session management (itsdangerous signing)
3. **Request Logging** - Structured request/response logging with configurable skip paths
4. **Security Headers** - Configurable HTTP security headers and CSP

90
docs/developer/contributing.md
View file

@ -1,24 +1,78 @@
---
title: Contributing
sidebar_position: 4
---
# Contributing # Contributing
## Scope ## PR Process
This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe.
## Essential-File Policy 1. Create a feature branch from `main`.
Commit only source-of-truth files required to build, run, test, or document the app. 2. Make your changes following the code standards below.
3. Run tests inside the container (see [Testing](testing.md)).
4. Run `ruff check .` and `mypy app/` to catch lint and type errors.
5. Open a pull request with a clear title and description.
Do not commit generated or local-only artifacts, including: ## Commit Conventions
- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`
- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/`
- coverage outputs (`.coverage`, `htmlcov/`)
- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`)
- local probe/scratch material (`planning/`)
CI enforces this with `scripts/check_no_generated_artifacts.sh`. This project uses [Conventional Commits](https://www.conventionalcommits.org/) with [python-semantic-release](https://python-semantic-release.readthedocs.io/) for automated versioning.
## Merge Checklist Commit message format:
- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas.
- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed). ```
- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`). <type>(<scope>): <description>
- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change.
- [ ] `README.md`, `.env.example`, and deployment notes stay aligned. [optional body]
- [ ] `scripts/check_no_generated_artifacts.sh` passes locally. ```
Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`.
Examples:
- `feat(scholars): add bulk import from CSV`
- `fix(ingestion): handle empty citation blocks`
- `docs: update configuration reference`
## Code Standards
### Function Length
Maximum 50 lines per function. Break complex logic into small, testable, single-responsibility functions.
### DRY
Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
### Negative Space Programming
Use explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
### Cyclomatic Complexity
Flatten logic. Use early returns and guard clauses instead of deep nesting. No magic numbers.
### Domain Service Boundaries
All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited. Each domain owns its own application service, types, and helpers.
### API Envelope
All `/api/v1` responses use the strict envelope format:
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
### Data Isolation
- Scholar tracking is user-scoped.
- Publications are global, deduplicated records.
- Read/favorite/visibility state lives on scholar-publication link rows.
### Scrape Safety
Rate limits and cooldowns are immutable constraints. They prevent IP bans and must not be optimized away or set to zero.
## UI Standards
- Integrate Tailwind with the preset theming system (`frontend/src/theme/presets/`).
- Every UI element must have a clear purpose.
- Clarity through both styling and language.

26
docs/developer/documentation-standards.md
View file

@ -1,26 +0,0 @@
# Documentation Standards
This project keeps docs organized by audience and document type.
## Audience Split
- `user/`: onboarding and usage tasks for self-hosted users.
- `operations/`: runbooks and checklists for production operation.
- `developer/`: architecture, contribution workflow, and implementation guides.
- `reference/`: stable contracts (API, env variables, configuration behavior).
## Document Quality Rules
- One primary audience per page.
- Task pages must include prerequisites, exact commands, and verification steps.
- Reference pages should be contract-first and avoid procedural noise.
- Runbooks should include recovery steps and rollback/safety notes.
- Prefer concise pages with strong cross-links over long mixed-purpose pages.
## Required Top-Level Entrypoints
- `index.md`: user/developer/operator navigation hub.
- `user/getting-started.md`: first-run path.
- `developer/local-development.md`: contributor setup and validation path.
- `operations/overview.md`: operational playbook index.
- `reference/overview.md`: contract index.

166
docs/developer/frontend-theme-inventory.md
View file

@ -1,89 +1,105 @@
# Theme Inventory (Phase 0) ---
title: Frontend Theme Inventory
sidebar_position: 6
---
This file captures the semantic color system baseline for the theme refactor. # Frontend Theme Inventory
## Token Domains ## Token Domains
- `scale`: The theme system uses semantic tokens organized into domains:
- `brand` (`50..950`)
- `info` (`50..950`)
- `success` (`50..950`)
- `warning` (`50..950`)
- `danger` (`50..950`)
- `surface`:
- `app`
- `nav`
- `nav_active`
- `card`
- `card_muted`
- `table`
- `table_header`
- `input`
- `overlay`
- `text`:
- `primary`
- `secondary`
- `muted`
- `inverse`
- `link`
- `border`:
- `default`
- `strong`
- `subtle`
- `interactive`
- `focus`:
- `ring`
- `ring_offset`
- `action` variants (`primary`, `secondary`, `ghost`, `danger`):
- `bg`
- `border`
- `text`
- `hover_bg`
- `hover_border`
- `hover_text`
- `state` variants (`info`, `success`, `warning`, `danger`):
- `bg`
- `border`
- `text`
## Theme Sources ### Scale Colors
Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`. Color ramps from `50` to `950` for each intent:
Current preset files: - `brand` - Primary brand color
- `info` - Informational elements
- `success` - Success states
- `warning` - Warning states
- `danger` - Error/destructive states
- `frontend/src/theme/presets/parchment.js` ### Surface
- `frontend/src/theme/presets/lilac.js`
- `frontend/src/theme/presets/dune.js`
- `frontend/src/theme/presets/oatmeal.js`
- `frontend/src/theme/presets/scholarly.json`
- `frontend/src/theme/presets/graphite.json`
- `frontend/src/theme/presets/tide.json`
Each preset contains both `light` and `dark` mode token definitions. Background colors for major UI areas:
## Adoption Status (Phase 0-3 complete baseline) | Token | Usage |
|-------|-------|
| `app` | Application background |
| `nav` | Navigation bar |
| `nav_active` | Active navigation item |
| `card` | Card backgrounds |
| `card_muted` | Muted card variant |
| `table` | Table body |
| `table_header` | Table header |
| `input` | Form inputs |
| `overlay` | Modal/dialog overlays |
Tokenized foundation components: ### Text
- `AppButton` | Token | Usage |
- `AppCard` |-------|-------|
- `AppCheckbox` | `primary` | Default text |
- `AppEmptyState` | `secondary` | Supporting text |
- `AppHelpHint` | `muted` | Disabled/placeholder text |
- `AppInput` | `inverse` | Text on dark backgrounds |
- `AppSelect` | `link` | Hyperlinks |
- `AppTable`
- `AppModal`
- `AppHeader`
- `AppNav`
- `AppAlert`
- `AppBadge`
- `RunStatusBadge`
- `QueueHealthBadge`
Hardening in place: ### Border
- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs` | Token | Usage |
- CI enforcement step in `frontend-quality` workflow |-------|-------|
- Theme preset integrity tests in `frontend/src/theme/presets.test.ts` | `default` | Standard borders |
| `strong` | Emphasized borders |
| `subtle` | Light separators |
| `interactive` | Hover/focus borders |
### Focus
| Token | Usage |
|-------|-------|
| `ring` | Focus ring color |
| `ring_offset` | Focus ring offset color |
### Action Variants
Each action type (`primary`, `secondary`, `ghost`, `danger`) provides:
`bg`, `border`, `text`, `hover_bg`, `hover_border`, `hover_text`
### State Variants
Each state (`info`, `success`, `warning`, `danger`) provides:
`bg`, `border`, `text`
## Theme Presets
Presets are loaded from `frontend/src/theme/presets/*.{json,js}`. Each contains both `light` and `dark` mode token definitions.
Current presets:
| Preset | Format |
|--------|--------|
| `parchment` | JS |
| `lilac` | JS |
| `dune` | JS |
| `oatmeal` | JS |
| `scholarly` | JSON |
| `graphite` | JSON |
| `tide` | JSON |
## Tokenized Components
Foundation components using the token system:
- `AppButton`, `AppCard`, `AppCheckbox`, `AppEmptyState`
- `AppHelpHint`, `AppInput`, `AppSelect`, `AppTable`
- `AppModal`, `AppHeader`, `AppNav`, `AppAlert`
- `AppBadge`, `RunStatusBadge`, `QueueHealthBadge`
## Enforcement
- Token policy check: `frontend/scripts/check_theme_tokens.mjs`
- CI step in `frontend-quality` workflow
- Preset integrity tests: `frontend/src/theme/presets.test.ts`

109
docs/developer/ingestion.md
View file

@ -1,38 +1,89 @@
# Ingestion System & Backoff Strategies ---
title: Ingestion Pipeline
sidebar_position: 5
---
The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans. # Ingestion Pipeline
## Ingestion Overview The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans.
The underlying ingestion process:
1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
3. Downloads the paginated HTML feed for a user across multiple page iterations.
4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
## Handling 429 Too Many Requests ## Pipeline Overview
Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`).
Scholarr treats these distinct from random network timeouts. 1. The scheduler (or a manual trigger) starts a **run** for one or more scholars.
- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s). 2. The service connects via HTTPX with strict browser headers.
- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead. 3. Paginated HTML feeds are downloaded for each scholar profile.
4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks.
5. Publications are fingerprinted and deduplicated against the global store.
6. External APIs resolve additional identifiers.
7. The PDF resolution pipeline runs asynchronously for publications with known DOIs.
## Publication Identifiers Loop ## Rate Limiting & Backoff
Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
- **API Fetching**: Queries secondary bibliographic platforms sequentially:
- `export.arxiv.org/api/query` (Queries by Title and Author strings).
- `crossref.restful` APIs (Queries by Title and Author strings).
These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage. ### Network Errors
Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s).
### HTTP 429 (Rate Limited)
When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies:
- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3)
- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s)
This pauses the pipeline gracefully instead of failing the entire run.
### Safety Cooldowns
Threshold-based cooldowns halt all ingestion after repeated failures:
| Threshold | Variable | Default | Cooldown |
|-----------|----------|---------|----------|
| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) |
| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) |
## Continuation Queue
Multi-page ingestion uses a continuation queue to spread load over time:
- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`)
- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s)
- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s)
- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6)
Each continuation item is re-enqueued with exponential backoff.
## Identifier Resolution
After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers:
1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches.
2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings.
3. **Crossref API** - Queries Crossref REST API by title and author strings.
Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage.
## PDF Resolution
Publications with resolved DOIs enter the PDF resolution pipeline:
1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs.
2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates.
3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly.
Auto-retry is configured via `PDF_AUTO_RETRY_*` variables.
## arXiv Request Controls ## arXiv Request Controls
- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
## arXiv Observability Events - **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. - **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`.
- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. - **In-flight coalescing**: Duplicate concurrent misses join one outbound request.
- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown. - **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.
### Observability Events
| Event | Description |
|-------|-------------|
| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. |
| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. |
| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. |
| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. |

100
docs/developer/local-development.md
View file

@ -1,52 +1,100 @@
# Developer Local Development ---
title: Local Development
sidebar_position: 3
---
## Start the Dev Stack # Local Development
## Prerequisites
- Docker and Docker Compose v2+
- Python 3.12+ (for IDE support and local linting)
- Node.js 20+ (for frontend development)
## Starting the Dev Stack
The development compose file overlays the production config with hot-reload and a separate Vite dev server:
```bash ```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
``` ```
Open: This starts three services:
- API: `http://localhost:8000`
- Frontend dev server: `http://localhost:5173`
Stop: | Service | Port | Description |
|---------|------|-------------|
| `db` | 5432 (internal) | PostgreSQL 15 |
| `app` | 8000 | FastAPI backend with `APP_RELOAD=1` |
| `frontend` | 5173 | Vite dev server proxying API calls to `app:8000` |
### Dev-Specific Overrides
The `docker-compose.dev.yml` file applies these changes:
- **`app`**: Uses `scholarr-dev:local` image built from the `dev` stage. Mounts the project root as `/app` for hot reload. Disables `SESSION_COOKIE_SECURE` and the built frontend.
- **`frontend`**: Node 20 container running `npm install && npm run dev`. Mounts `./frontend` with a named volume for `node_modules`. Uses polling for file watching (`CHOKIDAR_USEPOLLING=1`).
## Environment Setup
```bash ```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml down cp .env.example .env
``` ```
## Backend Validation Set at minimum:
```bash ```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit POSTGRES_PASSWORD=localdev
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration SESSION_SECRET_KEY=local-dev-secret-at-least-32-characters
SESSION_COOKIE_SECURE=0
``` ```
## Frontend Validation ## Running Tests
All tests run inside containers:
```bash ```bash
cd frontend # Run unit tests (default: excludes integration markers)
npm install docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
npm run typecheck python -m pytest
npm run test:run
npm run build # Run integration tests
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m integration
# Run a specific test file
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest tests/unit/test_fingerprints.py -v
``` ```
## Repository Gates See [Testing](testing.md) for markers, fixtures, and conventions.
## Linting and Type Checking
```bash ```bash
python3 scripts/check_frontend_api_contract.py # Ruff linting
python3 scripts/check_env_contract.py ruff check .
./scripts/check_no_generated_artifacts.sh
# Ruff formatting check
ruff format --check .
# Mypy type checking
mypy app/
``` ```
## Docs Site (Contributor Workflow) Ruff config is in `pyproject.toml`: target Python 3.12, line length 120, rules `E F W I UP B SIM RUF`.
Docs tooling is colocated in `docs/website/`: ## Database Migrations
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. To run manually:
```bash ```bash
cd docs/website docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
npm install alembic upgrade head
npm run build ```
To create a new migration:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
alembic revision --autogenerate -m "description of change"
``` ```

72
docs/developer/overview.md
View file

@ -1,11 +1,65 @@
# Developer Documentation ---
title: Developer Overview
sidebar_position: 1
---
Use this section if you are contributing code or running full quality gates. # Developer Overview
- [Documentation Standards](./documentation-standards.md) Scholarr is a Python 3.12+ FastAPI backend with an async SQLAlchemy ORM layer, PostgreSQL database, and a Vue 3 + TypeScript + Vite frontend.
- [Local Development](./local-development.md)
- [Architecture Boundaries](./architecture.md) ## Stack
- [Ingestion API & Backoff Strategies](./ingestion.md)
- [API Contracts](./api-contract.md) | Layer | Technology |
- [Contributing](./contributing.md) |-------|------------|
- [Frontend Theme Inventory](./frontend-theme-inventory.md) | Backend | Python 3.12+, FastAPI, SQLAlchemy 2.0 (async/asyncpg), Alembic |
| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS |
| Database | PostgreSQL 15 |
| Infrastructure | Multi-stage Docker, Docker Compose |
| Linting | ruff (E, F, W, I, UP, B, SIM, RUF), mypy |
| Testing | pytest, pytest-asyncio |
| Versioning | python-semantic-release, conventional commits |
## Quick Start
```bash
# Build the dev image and start services
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
# Backend: http://localhost:8000 (hot reload enabled)
# Frontend: http://localhost:5173 (Vite dev server with API proxy)
```
See [Local Development](local-development.md) for the full setup guide.
## Project Layout
```
app/
├── api/routers/ # FastAPI route handlers
├── auth/ # Session + CSRF middleware
├── db/ # SQLAlchemy models, session factory, migrations
├── services/ # Domain service modules (see Architecture)
│ ├── arxiv/ # arXiv API client, cache, rate limiting
│ ├── crossref/ # Crossref DOI lookups
│ ├── dbops/ # Database integrity + repair operations
│ ├── ingestion/ # Run orchestration, scheduler, safety gates
│ ├── openalex/ # OpenAlex metadata matching
│ ├── portability/ # Import/export workflows
│ ├── publication_identifiers/ # Multi-identifier resolution
│ ├── publications/ # Listing, enrichment, dedup, PDF queue
│ ├── runs/ # Run history, continuation queue
│ ├── scholar/ # HTML parser, source fetch adapters
│ ├── scholars/ # Scholar CRUD, image upload, name search
│ └── unpaywall/ # Unpaywall PDF discovery
├── main.py # App factory, lifespan, middleware stack
frontend/
├── src/
│ ├── components/ # Reusable Vue components
│ ├── theme/presets/ # Color theme presets (light + dark)
│ └── ...
scripts/db/ # Operational database scripts
tests/
├── unit/ # Fast, no-database tests
├── integration/ # Tests requiring database + services
└── fixtures/ # Shared test fixtures
```

102
docs/developer/testing.md Normal file
View file

@ -0,0 +1,102 @@
---
title: Testing
sidebar_position: 7
---
# Testing
## Running Tests
All tests must run inside containers:
```bash
# Unit tests (default: excludes integration markers)
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest
# Integration tests
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m integration
# Specific marker
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m db
# Verbose output for a specific file
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest tests/unit/test_fingerprints.py -v
```
## Test Configuration
From `pyproject.toml`:
```toml
[tool.pytest.ini_options]
addopts = "-q -m \"not integration\" --import-mode=importlib"
asyncio_mode = "auto"
testpaths = ["tests"]
```
- Default run excludes `integration` marked tests
- Uses `importlib` import mode to resolve module name collisions
- Async tests run automatically (no `@pytest.mark.asyncio` needed)
## Markers
| Marker | Description |
|--------|-------------|
| `integration` | Tests requiring external services (database, network) |
| `db` | Tests that validate database behavior and constraints |
| `migrations` | Tests focused on Alembic schema migration correctness |
| `schema` | Tests focused on multi-tenant schema invariants |
| `smoke` | Smoke tests for containerized runtime |
## Test Tiers
### Unit Tests (`tests/unit/`)
Fast, no-database tests. Mock external dependencies. These run by default.
Examples:
- `test_fingerprints.py` - Publication fingerprinting logic
- `test_scholar_parser.py` - HTML parsing without network calls
- `test_doi_normalize.py` - DOI normalization rules
- `test_ingestion_arxiv_rate_limit.py` - Rate limiter behavior
- `test_publication_pdf_resolution_pipeline.py` - PDF pipeline logic
Domain-specific unit tests are organized under `tests/unit/services/domains/`:
- `arxiv/` - Cache, client, gateway, guards, parser, rate limit tests
- `openalex/` - Client and matching tests
- `publications/` - Dedup tests
### Integration Tests (`tests/integration/`)
Require a running database. Test full request/response flows and data consistency.
Examples:
- `test_api_v1.py` - API endpoint integration tests
- `test_db_integrity.py` - Database integrity checks
- `test_run_lifecycle_consistency.py` - Run state machine transitions
- `test_deferred_enrichment.py` - Enrichment pipeline with real data
- `test_fixture_probe_runs.py` - Fixture-based run probes
### Smoke Tests
Marked with `@pytest.mark.smoke`. Validate the containerized runtime starts and serves basic requests.
## Fixtures
Test fixtures live in `tests/fixtures/`:
```
tests/fixtures/
└── scholar/
├── profile_ok_amIMrIEAAAAJ.html # Successful profile HTML
└── regression/
├── profile_P1RwlvoAAAAJ.html # Regression case
├── profile_LZ5D_p4AAAAJ.html # Regression case
└── profile_AAAAAAAAAAAA.html # Regression case
```
Scholar HTML fixtures are real Google Scholar profile pages used to test parser robustness against DOM structure changes.

44
docs/index.md
View file

@ -1,33 +1,27 @@
--- ---
slug: / title: Scholarr Documentation
sidebar_position: 1
--- ---
# scholarr Documentation # Scholarr
This documentation is organized for both users and developers. Scholarr is a self-hosted academic publication tracker. It monitors Google Scholar profiles, discovers new publications, resolves open-access PDFs, and presents everything through a clean Vue 3 dashboard.
## For Users ## Quick Links
- [User Overview](./user/overview.md) | Section | Description |
- [Getting Started](./user/getting-started.md) |---------|-------------|
| [User Guide](user/overview.md) | What scholarr does, installation, configuration |
| [Developer Guide](developer/overview.md) | Architecture, local development, contributing |
| [Operations](operations/overview.md) | Deployment, database runbook, scrape safety |
| [Reference](reference/overview.md) | API contract, environment variables, changelog |
## For Operators ## Key Features
- [Operations Overview](./operations/overview.md) - **Scholar Tracking** - Add Google Scholar profiles by ID, URL, or name search
- [Scrape Safety Runbook](./operations/scrape-safety-runbook.md) - **Automated Ingestion** - Background scheduler fetches new publications on a configurable interval
- [Database Runbook](./operations/database-runbook.md) - **Identifier Resolution** - Cross-references arXiv, Crossref, and OpenAlex for DOIs and metadata
- [Migration Rollout Checklist](./operations/migration-checklist.md) - **PDF Discovery** - Resolves open-access PDFs via Unpaywall and arXiv
- **Import/Export** - Portable scholar data with full publication state
## For Developers - **Multi-User** - Session-based auth with admin user management
- **Theming** - Multiple color presets with light/dark mode support
- [Developer Overview](./developer/overview.md)
- [Local Development](./developer/local-development.md)
- [Architecture Boundaries](./developer/architecture.md)
- [Contributing](./developer/contributing.md)
- [Frontend Theme Inventory](./developer/frontend-theme-inventory.md)
## Reference
- [Reference Overview](./reference/overview.md)
- [API Contract](./reference/api-contract.md)
- [Environment Reference](./reference/environment.md)

58
docs/operations/arxiv-runbook.md
View file

@ -1,46 +1,80 @@
---
title: arXiv Runbook
sidebar_position: 5
---
# arXiv Operations Runbook # arXiv Operations Runbook
Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly. Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly.
## Signals To Check ## Signals to Check
- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
- Logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
- `arxiv_runtime_state` row for cooldown and next-allowed timestamps - `arxiv_runtime_state` row for cooldown and next-allowed timestamps
- `arxiv_query_cache_entries` size and expiry churn - `arxiv_query_cache_entries` size and expiry churn
## Event Field Guide ## Event Field Guide
- `wait_seconds`: enforced pre-request delay from the global limiter.
- `status_code`: upstream response code from arXiv. | Field | Description |
- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429. |-------|-------------|
- `source_path`: caller path (`search` or `lookup_ids`). | `wait_seconds` | Enforced pre-request delay from the global limiter |
| `status_code` | Upstream response code from arXiv |
| `cooldown_remaining_seconds` | Remaining cooldown when blocked or after 429 |
| `source_path` | Caller path (`search` or `lookup_ids`) |
## Quick SQL Checks ## Quick SQL Checks
### Runtime State
```sql ```sql
SELECT state_key, next_allowed_at, cooldown_until, updated_at SELECT state_key, next_allowed_at, cooldown_until, updated_at
FROM arxiv_runtime_state; FROM arxiv_runtime_state;
``` ```
### Cache Status
```sql ```sql
SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry SELECT count(*) AS cache_rows,
min(expires_at) AS earliest_expiry,
max(expires_at) AS latest_expiry
FROM arxiv_query_cache_entries; FROM arxiv_query_cache_entries;
``` ```
## Common Scenarios ## Common Scenarios
1. Repeated `arxiv.cooldown_activated` events:
- Confirm recent `429` statuses in `arxiv.request_completed`. ### 1. Repeated `arxiv.cooldown_activated` Events
- Reduce caller pressure (check new title-quality/identifier guards are active).
- Confirm recent `429` statuses in `arxiv.request_completed` logs.
- Reduce caller pressure (check title-quality/identifier guards are active).
- Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict. - Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict.
2. High request latency with few completions: ### 2. High Request Latency with Few Completions
- Inspect `wait_seconds` in `arxiv.request_scheduled`. - Inspect `wait_seconds` in `arxiv.request_scheduled`.
- Verify only one process path is repeatedly hitting arXiv (`source_path`). - Verify only one process path is repeatedly hitting arXiv (`source_path`).
- Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears). - Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears).
3. Low cache effectiveness: ### 3. Low Cache Effectiveness
- Validate normalized query behavior and caller churn. - Validate normalized query behavior and caller churn.
- Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads. - Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads.
- Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed. - Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed.
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `ARXIV_ENABLED` | `1` | Enable arXiv lookups |
| `ARXIV_TIMEOUT_SECONDS` | `3.0` | Request timeout |
| `ARXIV_MIN_INTERVAL_SECONDS` | `4.0` | Min interval between requests |
| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | `60.0` | Cooldown after 429 |
| `ARXIV_DEFAULT_MAX_RESULTS` | `3` | Max results per query |
| `ARXIV_CACHE_TTL_SECONDS` | `900` | Cache TTL (15 min) |
| `ARXIV_CACHE_MAX_ENTRIES` | `512` | Max cached queries |
| `ARXIV_MAILTO` | *(empty)* | Contact email for API headers |
## Safe Recovery ## Safe Recovery
1. Pause automated ingestion if rate-limit storms persist. 1. Pause automated ingestion if rate-limit storms persist.
2. Let cooldown expire naturally; avoid manual burst retries. 2. Let cooldown expire naturally; avoid manual burst retries.
3. Resume and monitor event rates before restoring full load. 3. Resume and monitor event rates before restoring full load.

215
docs/operations/database-runbook.md
View file

@ -1,100 +1,195 @@
# Database Operations Runbook ---
title: Database Runbook
sidebar_position: 3
---
This runbook defines backup, restore, and verification procedures for the # Database Runbook
PostgreSQL database used by `scholarr`.
## Objectives ## Backup
- Keep data loss bounded via scheduled backups. Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service.
- Provide repeatable restore procedures.
- Verify backups by running regular restore drills.
Recommended starting targets: ### Custom-Format Backup (Default)
- `RPO`: 24 hours (maximum acceptable data loss).
- `RTO`: 60 minutes (maximum acceptable restore time).
## Backup Strategy
Use logical backups from the running `db` compose service.
- Custom-format backup (recommended for restore flexibility):
```bash ```bash
scripts/db/backup_full.sh scripts/db/backup_full.sh
``` ```
- Plain SQL backup: Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`).
### Plain SQL Backup
```bash ```bash
scripts/db/backup_full.sh --plain scripts/db/backup_full.sh --plain
``` ```
Optional environment variables: Creates a `.sql` file.
- `BACKUP_DIR`: destination directory (default: `<repo>/backups`) ### Options
- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
## Restore Strategy | Option | Description |
|--------|-------------|
| `--plain` | Write plain SQL instead of custom-format dump |
Restore from a `.dump` (custom format) or `.sql` file into the running `db` ### Environment Variables
compose service.
- Restore without schema wipe: | Variable | Default | Description |
|----------|---------|-------------|
| `BACKUP_DIR` | `<repo>/backups` | Destination directory |
| `BACKUP_PREFIX` | `scholarr` | File prefix |
| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` |
## Restore
Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service.
### Restore a Custom-Format Dump
```bash ```bash
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump
``` ```
- Restore with full `public` schema reset: ### Restore with Schema Wipe
```bash ```bash
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public
``` ```
## Safety Checklist Before Restore This drops and recreates the `public` schema before restoring. Use with caution.
1. Pause writes (stop app/scheduler or set maintenance mode). ### Options
2. Create a fresh pre-restore backup.
3. Validate target dump checksum/size.
4. Confirm operator, scope, and rollback plan.
## Post-Restore Verification | Option | Description |
|--------|-------------|
| `--file <path>` | Required. Path to `.dump` or `.sql` backup file |
| `--wipe-public` | Drop and recreate `public` schema before restore |
1. Run migrations head check. ## Integrity Checks
2. Run health endpoint checks.
3. Verify table row counts for core tables: Use `scripts/db/check_integrity.py` to run database integrity checks.
- `users`
- `scholar_profiles` ### Run Inside Container
- `publications`
- `scholar_publications`
- `crawl_runs`
4. Run integrity report and require zero failures:
```bash ```bash
python3 scripts/db/check_integrity.py docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/check_integrity.py
``` ```
5. Run API smoke checks for scholars/publications/runs pages. ### With Strict Warnings
## Data Cleanup and Repair
Use the audited repair job for scholar-publication relinking cleanup:
```bash ```bash
python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/check_integrity.py --strict-warnings
``` ```
Dry-run first, then re-run with `--apply` once scope and summary counts match expectation. Returns non-zero exit code if any warning is present.
If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout. ### Output Format
## Restore Drill Cadence JSON report:
- Run at least one full restore drill per month. ```json
- Record: {
- backup artifact used, "status": "passed",
- restore start/end timestamps, "failures": [],
- issues encountered, "warnings": []
- achieved `RTO`. }
```
Exit codes:
- `0` - All checks passed
- `1` - Check failure
- `2` - Warnings present (with `--strict-warnings`)
### Via Admin API
```
GET /api/v1/admin/db/integrity
```
Returns the same integrity report through the API (admin auth required).
## Publication Link Repair
Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging.
### Dry Run (Default)
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/repair_publication_links.py --user-id 1
```
### Apply Changes
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python scripts/db/repair_publication_links.py --user-id 1 --apply \
--requested-by "admin@example.com"
```
### Options
| Option | Description |
|--------|-------------|
| `--user-id <int>` | Required. Target user ID |
| `--scholar-profile-id <int>` | Optional. Filter by scholar profile ID (repeatable) |
| `--apply` | Apply changes (default is dry-run) |
| `--gc-orphan-publications` | Delete publications with zero links after cleanup |
| `--requested-by <string>` | Operator identifier for audit logs |
### Via Admin API
```
POST /api/v1/admin/db/repairs/publication-links
```
## Near-Duplicate Repair
Detect and merge near-duplicate publications via the admin API:
```
POST /api/v1/admin/db/repairs/publication-near-duplicates
```
## PDF Queue Management
### List Queue
```
GET /api/v1/admin/db/pdf-queue
```
### Requeue Single Item
```
POST /api/v1/admin/db/pdf-queue/{id}/requeue
```
### Bulk Requeue
```
POST /api/v1/admin/db/pdf-queue/requeue-all
```
## Migration Procedures
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`.
To run manually:
```bash
docker compose exec app alembic upgrade head
```
To check current migration status:
```bash
docker compose exec app alembic current
```
Recommended procedure for production:
1. Backup the database before upgrading.
2. Pull the new image.
3. Start the container; migrations run on startup.
4. Verify via `GET /healthz` and check logs for migration output.

117
docs/operations/deployment.md Normal file
View file

@ -0,0 +1,117 @@
---
title: Deployment
sidebar_position: 2
---
# Deployment
## Production Docker Compose
The default `docker-compose.yml` runs two services:
| Service | Image | Description |
|---------|-------|-------------|
| `db` | `postgres:15-alpine` | PostgreSQL database with persistent volume |
| `app` | `justinzeus/scholarr:latest` | FastAPI application with embedded frontend |
### Start
```bash
docker compose up -d
```
### Stop
```bash
docker compose down
```
### Update
```bash
docker compose pull
docker compose up -d
```
### View Logs
```bash
docker compose logs -f app
docker compose logs -f db
```
## Required Environment Variables
These must be set in `.env` or the shell environment:
```bash
POSTGRES_PASSWORD=<secure-password>
SESSION_SECRET_KEY=<random-string-32-chars-minimum>
```
## Volumes
| Volume | Mount | Description |
|--------|-------|-------------|
| `postgres_data` | `/var/lib/postgresql/data` | Database files |
| `scholar_uploads` | `/var/lib/scholarr/uploads` | Scholar profile images |
## Health Checks
### Database
```yaml
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"]
interval: 5s
timeout: 5s
retries: 20
```
### Application
```yaml
healthcheck:
test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"]
interval: 10s
timeout: 5s
retries: 12
```
The app service depends on `db` with `condition: service_healthy`, so it waits for the database to be ready.
## Database Readiness Wait
The app has built-in database readiness polling:
- `DB_WAIT_TIMEOUT_SECONDS` (default: 60) - Max wait time
- `DB_WAIT_INTERVAL_SECONDS` (default: 2) - Poll interval
## Auto-Migration
Set `MIGRATE_ON_START=1` (default) to run Alembic migrations automatically on startup.
## Scaling Considerations
- Run a single `app` instance to avoid scheduler conflicts (the scheduler is process-local).
- arXiv requests are globally serialized via a PostgreSQL advisory lock, so multiple instances safely share the rate limiter.
- Database pool defaults: 5 base connections + 10 overflow. Adjust `DATABASE_POOL_SIZE` and `DATABASE_POOL_MAX_OVERFLOW` for higher loads.
## Admin Bootstrap
For first-time setup without manual database access:
```bash
BOOTSTRAP_ADMIN_ON_START=1
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
BOOTSTRAP_ADMIN_PASSWORD=<secure-password>
```
Set `BOOTSTRAP_ADMIN_FORCE_PASSWORD=1` to reset an existing admin password.
## Security Hardening
- Set `SESSION_COOKIE_SECURE=1` when serving over HTTPS.
- Enable HSTS: `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED=1`.
- Review CSP policy in `SECURITY_CSP_POLICY` for your domain.
- Set `UNPAYWALL_EMAIL` and `ARXIV_MAILTO` for polite API pool access.

48
docs/operations/migration-checklist.md
View file

@ -1,48 +0,0 @@
# Migration Checklist
Use this checklist for every schema/data migration.
## 1. Design Review
- Define change type: `expand`, `backfill`, `contract`.
- Document expected lock behavior and index impact.
- Confirm backward compatibility with currently deployed app version.
- Define rollback strategy before implementation.
## 2. Pre-Migration Controls
- Capture a fresh backup before applying migration.
- Confirm migration head revision and working tree cleanliness.
- Prepare validation queries for new/changed tables and indexes.
- Identify high-risk tables (large row count, hot write paths).
## 3. Implementation Standards
- Keep migrations idempotent when feasible.
- Prefer additive steps first (`nullable`, new index, new table).
- For destructive changes, separate into later contract migration.
- Avoid large blocking rewrites in a single deployment step.
## 4. Verification
- Apply migration in staging against production-like snapshot.
- Verify:
- expected tables/columns/indexes,
- app startup and health endpoint,
- affected API flows,
- data consistency queries.
## 5. Rollout and Recovery
- Apply migration during planned window.
- Monitor logs/errors and DB metrics during rollout.
- If rollback needed:
- follow downgrade/recovery runbook,
- restore from backup if downgrade is unsafe,
- document incident timeline.
## 6. Post-Migration Tasks
- Update `README.md` / `.env.example` / ops docs.
- Add/update integration tests for new schema assumptions.
- Record migration notes in changelog.

34
docs/operations/overview.md
View file

@ -1,8 +1,30 @@
# Operations Documentation ---
title: Operations Overview
sidebar_position: 1
---
Use this section for production operations and incident/runbook workflows. # Operations Overview
- [Scrape Safety Runbook](./scrape-safety-runbook.md) This section covers production deployment, database administration, and scrape safety operations.
- [arXiv Runbook](./arxiv-runbook.md)
- [Database Runbook](./database-runbook.md) ## Quick Links
- [Migration Rollout Checklist](./migration-checklist.md)
| Guide | Description |
|-------|-------------|
| [Deployment](deployment.md) | Production Docker setup, scaling, health checks |
| [Database Runbook](database-runbook.md) | Backup, restore, integrity checks, repair procedures |
| [Scrape Safety Runbook](scrape-safety-runbook.md) | Rate limiting, cooldowns, CAPTCHA handling |
| [arXiv Runbook](arxiv-runbook.md) | arXiv rate limits, cache tuning, query patterns |
## Health Check
The app exposes `GET /healthz` for container orchestration:
```bash
curl -fsS http://localhost:8000/healthz
```
Docker Compose healthcheck config:
- Interval: 10s
- Timeout: 5s
- Retries: 12

88
docs/operations/scrape-safety-runbook.md
View file

@ -1,74 +1,76 @@
---
title: Scrape Safety Runbook
sidebar_position: 4
---
# Scrape Safety Runbook # Scrape Safety Runbook
This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events. Operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
## Key Signals ## Key Signals
Use structured log `event` fields (recommended with `LOG_FORMAT=json`): Use structured log `event` fields (recommended with `LOG_FORMAT=json`):
- `ingestion.safety_policy_blocked_run_start` | Event | Description |
- `ingestion.safety_cooldown_entered` |-------|-------------|
- `ingestion.safety_cooldown_cleared` | `ingestion.safety_policy_blocked_run_start` | Run blocked by safety policy |
- `ingestion.alert_blocked_failure_threshold_exceeded` | `ingestion.safety_cooldown_entered` | Cooldown activated |
- `ingestion.alert_network_failure_threshold_exceeded` | `ingestion.safety_cooldown_cleared` | Cooldown expired |
- `ingestion.alert_retry_scheduled_threshold_exceeded` | `ingestion.alert_blocked_failure_threshold_exceeded` | Blocked failure threshold tripped |
- `api.runs.manual_blocked_policy` | `ingestion.alert_network_failure_threshold_exceeded` | Network failure threshold tripped |
- `api.runs.manual_blocked_safety` | `ingestion.alert_retry_scheduled_threshold_exceeded` | Retry threshold tripped |
- `scheduler.run_skipped_safety_cooldown` | `api.runs.manual_blocked_policy` | Manual run blocked by policy |
- `scheduler.run_skipped_safety_cooldown_precheck` | `api.runs.manual_blocked_safety` | Manual run blocked by safety cooldown |
- `scheduler.queue_item_deferred_safety_cooldown` | `scheduler.run_skipped_safety_cooldown` | Scheduled run skipped due to cooldown |
| `scheduler.run_skipped_safety_cooldown_precheck` | Scheduler precheck blocked |
| `scheduler.queue_item_deferred_safety_cooldown` | Queue item deferred due to cooldown |
Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules. Each event includes metric-style fields (`metric_name`, `metric_value`) for log-based alert rules.
## Recommended Alert Rules ## Recommended Alert Rules
- Cooldown enters: - **Cooldown enters**: Trigger on `event=ingestion.safety_cooldown_entered`
- Trigger on `event=ingestion.safety_cooldown_entered`. - **Repeated start blocks**: High rate of `event=api.runs.manual_blocked_safety`
- Repeated start blocks: - **Threshold trips**: Any of the `*_threshold_exceeded` events
- Trigger on high rate of `event=api.runs.manual_blocked_safety`. - **Scheduler pressure**: Sustained `scheduler.queue_item_deferred_safety_cooldown`
- Threshold trips:
- Trigger on any of:
- `ingestion.alert_blocked_failure_threshold_exceeded`
- `ingestion.alert_network_failure_threshold_exceeded`
- `ingestion.alert_retry_scheduled_threshold_exceeded`
- Scheduler pressure:
- Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`.
## If Your IP Appears Blocked ## If Your IP Appears Blocked
Symptoms: ### Symptoms
- cooldown reason `blocked_failure_threshold_exceeded` - Cooldown reason: `blocked_failure_threshold_exceeded`
- parse state `blocked_or_captcha` - Parse state: `blocked_or_captcha`
- redirects toward Google account sign-in flows - Redirects toward Google account sign-in flows
Actions: ### Actions
1. Stop manual retries immediately. 1. Stop manual retries immediately.
2. Let cooldown expire; do not spam retriggers. 2. Let cooldown expire; do not spam retriggers.
3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values. 3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values.
4. Reduce concurrency pressure (keep one scheduler instance). 4. Reduce concurrency pressure (keep one scheduler instance).
5. Keep name-search disabled/WIP for now if login-gated responses persist. 5. Keep name-search disabled if login-gated responses persist.
6. Resume with a small monitored run and verify blocked rate drops. 6. Resume with a small monitored run and verify blocked rate drops.
Avoid: ### Avoid
- aggressive rapid retries - Aggressive rapid retries
- rotating through risky scraping patterns that increase challenge rates - Rotating through risky scraping patterns that increase challenge rates
- bypass/captcha-solving workflows that may violate source platform rules - Bypass/CAPTCHA-solving workflows that may violate source platform rules
## Environment Controls ## Environment Controls
Policy floors and safety controls: Policy floors and safety controls:
- `INGESTION_MIN_REQUEST_DELAY_SECONDS` | Variable | Default | Description |
- `INGESTION_MIN_RUN_INTERVAL_MINUTES` |----------|---------|-------------|
- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `INGESTION_MIN_REQUEST_DELAY_SECONDS` | `2` | Floor delay between requests |
- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `INGESTION_MIN_RUN_INTERVAL_MINUTES` | `15` | Minimum time between runs |
- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `1` | Blocked failures before alert |
- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `2` | Network failures before alert |
- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `3` | Retries before alert |
- `INGESTION_MANUAL_RUN_ALLOWED` | `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `1800` | Cooldown after blocked threshold (30 min) |
- `INGESTION_AUTOMATION_ALLOWED` | `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `900` | Cooldown after network threshold (15 min) |
| `INGESTION_MANUAL_RUN_ALLOWED` | `1` | Enable manual runs |
| `INGESTION_AUTOMATION_ALLOWED` | `1` | Enable automated runs |
Apply stricter values first, then relax slowly only after sustained healthy runs. Apply stricter values first, then relax slowly only after sustained healthy runs.

31
docs/reference/api-contract.md
View file

@ -1,31 +0,0 @@
# API Contract
## Envelope Invariant
All API responses under `/api/v1` use one of these envelopes:
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
`meta.request_id` must be present for success and error responses.
Binary media assets are served outside `/api/v1` (for example, `GET /scholar-images/{scholar_profile_id}/upload`).
## Publications Semantics
- `GET /api/v1/publications` supports `mode=all|unread|latest`.
- `mode=new` is currently accepted as a compatibility alias for `latest`.
- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`.
- `unread` represents read-state (`is_read=false`).
- `latest` represents discovery-state (`first seen in the latest completed run`).
Publication payloads expose:
- `pub_url` (canonical scholar detail URL)
- `doi` (normalized DOI)
- `pdf_url` (resolved OA PDF when available)
## Scholar Portability
- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state.
- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication.

177
docs/reference/api.md Normal file
View file

@ -0,0 +1,177 @@
---
title: API Contract
sidebar_position: 2
---
# API Contract
## Envelope Invariant
All API responses under `/api/v1` use one of these envelopes:
### Success
```json
{
"data": "...",
"meta": {
"request_id": "..."
}
}
```
### Error
```json
{
"error": {
"code": "...",
"message": "...",
"details": "..."
},
"meta": {
"request_id": "..."
}
}
```
`meta.request_id` is present on both success and error responses.
### Binary Assets
Binary media assets are served outside `/api/v1`:
```
GET /scholar-images/{scholar_profile_id}/upload
```
## DTO Structure
Scholarr uses strictly typed Pydantic V2 models serialized through OpenAPI v3 via FastAPI.
- `PublicationListItem` exposes a `.display_identifier` property that resolves the highest-confidence identifier regardless of backend origin, rather than a hardcoded `.doi`.
- Frontend TypeScript types are compiled from the OpenAPI spec to ensure type safety across the stack.
## Endpoints
### Auth
| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/api/v1/auth/login` | Login (rate-limited sliding window) |
| `GET` | `/api/v1/auth/me` | Current session user |
| `GET` | `/api/v1/auth/csrf` | Bootstrap CSRF token |
| `POST` | `/api/v1/auth/change-password` | Change password |
| `POST` | `/api/v1/auth/logout` | Logout |
### Scholars
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/scholars` | List tracked scholars |
| `POST` | `/api/v1/scholars` | Create scholar (auto-enqueue, metadata hydration) |
| `GET` | `/api/v1/scholars/search` | Search author candidates by name |
| `PATCH` | `/api/v1/scholars/{id}/toggle` | Toggle enabled status |
| `DELETE` | `/api/v1/scholars/{id}` | Delete scholar |
| `PUT` | `/api/v1/scholars/{id}/image/url` | Update image URL |
| `POST` | `/api/v1/scholars/{id}/image/upload` | Upload image |
| `DELETE` | `/api/v1/scholars/{id}/image` | Clear image customization |
### Scholar Portability
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/scholars/export` | Export tracked scholars and publication link state |
| `POST` | `/api/v1/scholars/import` | Import scholars with global publication deduplication |
The export payload includes scholar metadata, tracked publication data, and link state (read/unread, favorites). Import preserves global deduplication.
### Publications
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/publications` | List publications (filtered, paginated) |
| `POST` | `/api/v1/publications/mark-all-read` | Mark all as read |
| `POST` | `/api/v1/publications/mark-read` | Mark selected as read |
| `POST` | `/api/v1/publications/{id}/retry-pdf` | Retry PDF resolution |
| `POST` | `/api/v1/publications/{id}/favorite` | Toggle favorite |
#### Publication Modes
`GET /api/v1/publications` supports a `mode` parameter:
| Mode | Description |
|------|-------------|
| `all` | All publications |
| `unread` | Publications with `is_read=false` |
| `latest` | Publications first seen in the latest completed run |
`mode=new` is accepted as a compatibility alias for `latest`.
#### Pagination
Query parameters: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
Response pagination fields:
```json
{
"page": 1,
"page_size": 20,
"has_prev": false,
"has_next": true,
"total_count": 142
}
```
#### Publication Payload Fields
| Field | Description |
|-------|-------------|
| `pub_url` | Canonical scholar detail URL |
| `doi` | Normalized DOI |
| `pdf_url` | Resolved open-access PDF URL (when available) |
| `display_identifier` | Highest-confidence identifier regardless of source |
### Runs
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/runs` | List runs with safety state |
| `GET` | `/api/v1/runs/{run_id}` | Run detail with scholar results |
| `POST` | `/api/v1/runs/{run_id}/cancel` | Cancel active run |
| `POST` | `/api/v1/runs/manual` | Trigger manual run (idempotent, safety-checked) |
| `GET` | `/api/v1/runs/queue/items` | List queue items |
| `POST` | `/api/v1/runs/queue/{id}/retry` | Retry queue item |
| `POST` | `/api/v1/runs/queue/{id}/drop` | Drop queue item |
| `DELETE` | `/api/v1/runs/queue/{id}` | Clear queue item |
| `GET` | `/api/v1/runs/{run_id}/stream` | Stream run events (SSE) |
### Settings
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/settings` | Get user settings (with cooldown expiry check) |
| `PUT` | `/api/v1/settings` | Update settings (interval, delay, nav, API keys) |
### Admin - User Management
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/admin/users` | List all users |
| `POST` | `/api/v1/admin/users` | Create user |
| `PATCH` | `/api/v1/admin/users/{id}/active` | Set user active status |
| `POST` | `/api/v1/admin/users/{id}/reset-password` | Reset password |
### Admin - Database Operations
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/admin/db/integrity` | Get integrity report |
| `GET` | `/api/v1/admin/db/repair-jobs` | List repair jobs |
| `GET` | `/api/v1/admin/db/pdf-queue` | List PDF queue |
| `POST` | `/api/v1/admin/db/pdf-queue/{id}/requeue` | Requeue single PDF |
| `POST` | `/api/v1/admin/db/pdf-queue/requeue-all` | Bulk requeue missing PDFs |
| `POST` | `/api/v1/admin/db/repairs/publication-links` | Trigger link repair |
| `POST` | `/api/v1/admin/db/repairs/publication-near-duplicates` | Trigger dedup repair |
| `POST` | `/api/v1/admin/db/drop-all-publications` | Drop all publications (destructive) |

12
docs/reference/changelog.md Normal file
View file

@ -0,0 +1,12 @@
---
title: Changelog
sidebar_position: 4
---
# Changelog
This changelog is auto-generated by [python-semantic-release](https://python-semantic-release.readthedocs.io/).
Release versions follow [Semantic Versioning](https://semver.org/). Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/).
<!-- semantic-release will insert entries here -->

86
docs/reference/environment.md
View file

@ -1,68 +1,32 @@
# Environment Reference and Audit ---
title: Environment Variables
sidebar_position: 3
---
This document is the source-of-truth audit for what remains configurable versus internal defaults. # Environment Variables Quick Reference
## Decision Summary All environment variables are documented in detail in [Configuration](../user/configuration.md).
- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras. ## Required Variables
- Move to internal defaults only: none at this time.
- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted.
## Configurable Variables (By Section) | Variable | Description |
|----------|-------------|
| `POSTGRES_PASSWORD` | PostgreSQL password |
| `SESSION_SECRET_KEY` | Session cookie signing key (32+ characters) |
### Compose + Database ## Categories
`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE` | Category | Key Variables |
|----------|--------------|
| Database | `POSTGRES_DB`, `POSTGRES_USER`, `DATABASE_URL`, `DATABASE_POOL_*` |
| Runtime | `APP_HOST`, `APP_PORT`, `MIGRATE_ON_START`, `FRONTEND_ENABLED` |
| Auth | `SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_*` |
| Security | `SECURITY_HEADERS_ENABLED`, `SECURITY_CSP_*`, `SECURITY_STRICT_TRANSPORT_*` |
| Logging | `LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS` |
| Scheduler | `SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_*_BATCH_SIZE` |
| Ingestion | `INGESTION_*` (safety floors, cooldowns, retry policies) |
| Scholar | `SCHOLAR_IMAGE_*`, `SCHOLAR_NAME_SEARCH_*` |
| Enrichment | `UNPAYWALL_*`, `ARXIV_*`, `CROSSREF_*`, `OPENALEX_*`, `PDF_AUTO_RETRY_*` |
| Bootstrap | `BOOTSTRAP_ADMIN_*`, `DB_WAIT_*` |
### App Runtime + Networking See [Configuration](../user/configuration.md) for the complete table with types, defaults, and descriptions.
`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR`
### Database Pool
`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS`
### Frontend Dev Overrides
`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET`
### Auth + Session
`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS`
### HTTP Security Headers + CSP
`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD`
### Logging
`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS`
### Scheduler + Ingestion Safety
`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `SCHEDULER_PDF_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_RATE_LIMIT_RETRIES`, `INGESTION_RATE_LIMIT_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
### Scholar Images + Name Search Safety
`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD`
### OA Enrichment + PDF Resolution
`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `ARXIV_ENABLED`, `ARXIV_TIMEOUT_SECONDS`, `ARXIV_MIN_INTERVAL_SECONDS`, `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS`, `ARXIV_DEFAULT_MAX_RESULTS`, `ARXIV_CACHE_TTL_SECONDS`, `ARXIV_CACHE_MAX_ENTRIES`, `ARXIV_MAILTO`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`, `OPENALEX_API_KEY`, `CROSSREF_API_TOKEN`, `CROSSREF_API_MAILTO`
### Startup Bootstrap + DB Wait
`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS`
## Drift Guard
CI enforces env parity with:
```bash
python3 scripts/check_env_contract.py
```
The check fails when:
- `app/settings.py` references a variable missing from `.env.example`.
- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras).
- `.env.example` contains duplicate keys.

16
docs/reference/overview.md
View file

@ -1,6 +1,14 @@
# Reference Documentation ---
title: Reference
sidebar_position: 1
---
Use this section for canonical contracts and configuration references. # Reference
- [API Contract](./api-contract.md) Quick-reference documentation for the scholarr API, environment variables, and release changelog.
- [Environment Reference](./environment.md)
| Document | Description |
|----------|-------------|
| [API Contract](api.md) | Envelope spec, endpoints, DTO structure, publication semantics |
| [Environment Variables](environment.md) | Quick-reference table linking to full configuration docs |
| [Changelog](changelog.md) | Auto-generated release history |

178
docs/user/configuration.md Normal file
View file

@ -0,0 +1,178 @@
---
title: Configuration
sidebar_position: 3
---
# Configuration
All configuration is done through environment variables. Copy `.env.example` to `.env` and adjust values as needed.
## Compose & Database
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `POSTGRES_DB` | string | `scholar` | PostgreSQL database name |
| `POSTGRES_USER` | string | `scholar` | PostgreSQL user |
| `POSTGRES_PASSWORD` | string | **required** | PostgreSQL password |
| `DATABASE_URL` | string | derived | SQLAlchemy async connection string |
| `TEST_DATABASE_URL` | string | derived | Override for test database. If empty, tests derive `<db_name>_test` |
| `SCHOLARR_IMAGE` | string | `justinzeus/scholarr:latest` | Docker image for the app service |
## App Runtime & Networking
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `APP_NAME` | string | `scholarr` | Application name used in logs and headers |
| `APP_HOST` | string | `0.0.0.0` | Bind address |
| `APP_PORT` | int | `8000` | Internal port |
| `APP_HOST_PORT` | int | `8000` | Host-mapped port |
| `APP_RELOAD` | bool | `0` | Enable uvicorn auto-reload (dev only) |
| `MIGRATE_ON_START` | bool | `1` | Run Alembic migrations on startup |
| `FRONTEND_ENABLED` | bool | `1` | Serve the built Vue frontend |
| `FRONTEND_DIST_DIR` | string | `/app/frontend/dist` | Path to compiled frontend assets |
## Database Pool
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `DATABASE_POOL_MODE` | string | `auto` | Pool mode (`auto`, `fixed`, `null`) |
| `DATABASE_POOL_SIZE` | int | `5` | Base pool size |
| `DATABASE_POOL_MAX_OVERFLOW` | int | `10` | Maximum overflow connections |
| `DATABASE_POOL_TIMEOUT_SECONDS` | int | `30` | Connection acquisition timeout |
## Frontend Dev Overrides
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `FRONTEND_HOST_PORT` | int | `5173` | Host port for Vite dev server |
| `CHOKIDAR_USEPOLLING` | bool | `1` | Enable polling for file watchers in containers |
| `VITE_DEV_API_PROXY_TARGET` | string | `http://app:8000` | Backend URL for Vite proxy |
## Auth & Session
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `SESSION_SECRET_KEY` | string | **required** | Signing key for session cookies (32+ chars) |
| `SESSION_COOKIE_SECURE` | bool | `1` | Set `Secure` flag on session cookie (disable for local HTTP dev) |
| `LOGIN_RATE_LIMIT_ATTEMPTS` | int | `5` | Max login attempts per window |
| `LOGIN_RATE_LIMIT_WINDOW_SECONDS` | int | `60` | Sliding window for login rate limiting |
## HTTP Security Headers & CSP
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `SECURITY_HEADERS_ENABLED` | bool | `1` | Enable security response headers |
| `SECURITY_X_CONTENT_TYPE_OPTIONS` | string | `nosniff` | X-Content-Type-Options header value |
| `SECURITY_X_FRAME_OPTIONS` | string | `DENY` | X-Frame-Options header value |
| `SECURITY_REFERRER_POLICY` | string | `strict-origin-when-cross-origin` | Referrer-Policy header value |
| `SECURITY_PERMISSIONS_POLICY` | string | *(restrictive)* | Permissions-Policy header value |
| `SECURITY_CROSS_ORIGIN_OPENER_POLICY` | string | `same-origin` | Cross-Origin-Opener-Policy header |
| `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY` | string | `same-origin` | Cross-Origin-Resource-Policy header |
| `SECURITY_CSP_ENABLED` | bool | `1` | Enable Content-Security-Policy header |
| `SECURITY_CSP_POLICY` | string | *(restrictive)* | CSP for app routes |
| `SECURITY_CSP_DOCS_POLICY` | string | *(docs-specific)* | CSP for documentation routes |
| `SECURITY_CSP_REPORT_ONLY` | bool | `0` | Use report-only mode for CSP |
| `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED` | bool | `0` | Enable HSTS header |
| `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE` | int | `31536000` | HSTS max-age in seconds |
| `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS` | bool | `1` | HSTS includeSubDomains directive |
| `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD` | bool | `0` | HSTS preload directive |
## Logging
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `LOG_LEVEL` | string | `INFO` | Root log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
| `LOG_FORMAT` | string | `console` | Log format (`console` or `json`) |
| `LOG_REQUESTS` | bool | `1` | Log HTTP requests |
| `LOG_UVICORN_ACCESS` | bool | `0` | Enable uvicorn access log |
| `LOG_REQUEST_SKIP_PATHS` | string | `/healthz` | Comma-separated paths to exclude from request logging |
| `LOG_REDACT_FIELDS` | string | *(empty)* | Comma-separated field names to redact in logs |
## Scheduler & Ingestion Safety
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `SCHEDULER_ENABLED` | bool | `1` | Enable the background scheduler |
| `SCHEDULER_TICK_SECONDS` | int | `60` | Scheduler poll interval |
| `SCHEDULER_QUEUE_BATCH_SIZE` | int | `10` | Max scholars processed per tick |
| `SCHEDULER_PDF_QUEUE_BATCH_SIZE` | int | `15` | Max PDF resolutions per tick |
| `INGESTION_AUTOMATION_ALLOWED` | bool | `1` | Allow automated (scheduled) runs |
| `INGESTION_MANUAL_RUN_ALLOWED` | bool | `1` | Allow manually triggered runs |
| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | int | `15` | Minimum time between runs |
| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | int | `2` | Floor delay between external requests |
| `INGESTION_NETWORK_ERROR_RETRIES` | int | `1` | Retries on network errors |
| `INGESTION_RETRY_BACKOFF_SECONDS` | float | `1.0` | Base backoff for network retries |
| `INGESTION_RATE_LIMIT_RETRIES` | int | `3` | Retries on 429 responses |
| `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` | float | `30.0` | Backoff per 429 retry |
| `INGESTION_MAX_PAGES_PER_SCHOLAR` | int | `30` | Max paginated pages per scholar |
| `INGESTION_PAGE_SIZE` | int | `100` | Publications per page |
| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | int | `1` | Blocked failures before alert |
| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | int | `2` | Network failures before alert |
| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | int | `3` | Scheduled retries before alert |
| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | int | `1800` | Cooldown after blocked-failure threshold (30 min) |
| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | int | `900` | Cooldown after network-failure threshold (15 min) |
| `INGESTION_CONTINUATION_QUEUE_ENABLED` | bool | `1` | Enable continuation queue for multi-page ingestion |
| `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` | int | `120` | Base delay for continuation queue items |
| `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` | int | `3600` | Max delay for continuation queue items |
| `INGESTION_CONTINUATION_MAX_ATTEMPTS` | int | `6` | Max continuation attempts per scholar |
## Scholar Images & Name Search Safety
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `SCHOLAR_IMAGE_UPLOAD_DIR` | string | `/var/lib/scholarr/uploads` | Directory for uploaded scholar images |
| `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES` | int | `2000000` | Max image upload size (2 MB) |
| `SCHOLAR_NAME_SEARCH_ENABLED` | bool | `1` | Enable name-based scholar search |
| `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS` | int | `21600` | Cache TTL for successful searches (6 hours) |
| `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS` | int | `300` | Cache TTL for blocked search results (5 min) |
| `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES` | int | `512` | Max cache entries for name search |
| `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS` | float | `8.0` | Min interval between name searches |
| `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS` | float | `2.0` | Random jitter added to search interval |
| `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD` | int | `1` | Blocked results before cooldown |
| `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS` | int | `1800` | Cooldown after blocked name search (30 min) |
| `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD` | int | `2` | Retries before alert |
| `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` | int | `3` | Cooldown rejections before alert |
## OA Enrichment & PDF Resolution
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `UNPAYWALL_ENABLED` | bool | `1` | Enable Unpaywall DOI lookups |
| `UNPAYWALL_EMAIL` | string | *(empty)* | Polite pool email for Unpaywall API |
| `UNPAYWALL_TIMEOUT_SECONDS` | float | `4.0` | Request timeout |
| `UNPAYWALL_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Unpaywall requests |
| `UNPAYWALL_MAX_ITEMS_PER_REQUEST` | int | `20` | Max items per batch |
| `UNPAYWALL_RETRY_COOLDOWN_SECONDS` | int | `1800` | Cooldown after repeated failures |
| `UNPAYWALL_PDF_DISCOVERY_ENABLED` | bool | `1` | Enable HTML-based PDF link discovery |
| `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES` | int | `5` | Max candidate URLs to probe |
| `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES` | int | `500000` | Max HTML response size to parse |
| `ARXIV_ENABLED` | bool | `1` | Enable arXiv API lookups |
| `ARXIV_TIMEOUT_SECONDS` | float | `3.0` | Request timeout |
| `ARXIV_MIN_INTERVAL_SECONDS` | float | `4.0` | Min interval between arXiv requests |
| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | float | `60.0` | Cooldown after arXiv 429 |
| `ARXIV_DEFAULT_MAX_RESULTS` | int | `3` | Default max results per query |
| `ARXIV_CACHE_TTL_SECONDS` | int | `900` | Query cache TTL (15 min) |
| `ARXIV_CACHE_MAX_ENTRIES` | int | `512` | Max cached queries |
| `ARXIV_MAILTO` | string | *(empty)* | Contact email for arXiv API headers |
| `PDF_AUTO_RETRY_INTERVAL_SECONDS` | int | `86400` | Auto-retry interval for failed PDFs (24 hours) |
| `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS` | int | `3600` | First retry interval (1 hour) |
| `PDF_AUTO_RETRY_MAX_ATTEMPTS` | int | `3` | Max auto-retry attempts |
| `CROSSREF_ENABLED` | bool | `1` | Enable Crossref lookups |
| `CROSSREF_MAX_ROWS` | int | `10` | Max rows per Crossref query |
| `CROSSREF_TIMEOUT_SECONDS` | float | `8.0` | Request timeout |
| `CROSSREF_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Crossref requests |
| `CROSSREF_MAX_LOOKUPS_PER_REQUEST` | int | `8` | Max lookups per ingestion request |
| `OPENALEX_API_KEY` | string | *(empty)* | OpenAlex API key (optional) |
| `CROSSREF_API_TOKEN` | string | *(empty)* | Crossref Plus API token (optional) |
| `CROSSREF_API_MAILTO` | string | *(empty)* | Crossref polite pool email |
## Startup Bootstrap & DB Wait
| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `BOOTSTRAP_ADMIN_ON_START` | bool | `0` | Create admin user on startup |
| `BOOTSTRAP_ADMIN_EMAIL` | string | *(empty)* | Admin email address |
| `BOOTSTRAP_ADMIN_PASSWORD` | string | *(empty)* | Admin password |
| `BOOTSTRAP_ADMIN_FORCE_PASSWORD` | bool | `0` | Overwrite existing admin password |
| `DB_WAIT_TIMEOUT_SECONDS` | int | `60` | Max seconds to wait for database readiness |
| `DB_WAIT_INTERVAL_SECONDS` | int | `2` | Poll interval while waiting for database |

109
docs/user/getting-started.md
View file

@ -1,41 +1,86 @@
# User Getting Started ---
title: Getting Started
sidebar_position: 2
---
## Required Setup # Getting Started
1. Copy `.env.example` to `.env`. ## Prerequisites
2. Set `POSTGRES_PASSWORD`.
3. Set `SESSION_SECRET_KEY`.
## Deploy Method A: Prebuilt Image - Docker and Docker Compose v2+
- A machine with at least 512 MB RAM
## Installation
1. Clone the repository:
```bash
git clone https://github.com/JustinZeus/scholarr.git
cd scholarr
```
2. Copy the example environment file:
```bash
cp .env.example .env
```
3. Edit `.env` and set the required secrets:
```bash
# Required: database password
POSTGRES_PASSWORD=your-secure-password
# Required: session signing key (32+ random characters)
SESSION_SECRET_KEY=your-random-secret-key
```
4. Start the stack:
```bash
docker compose up -d
```
5. Verify the service is healthy:
```bash
docker compose ps
curl http://localhost:8000/healthz
```
The app is now available at `http://localhost:8000`.
## First Run
### Bootstrap an Admin User
Set these environment variables before first start (or add them to `.env`):
```bash
BOOTSTRAP_ADMIN_ON_START=1
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
BOOTSTRAP_ADMIN_PASSWORD=your-admin-password
```
Restart the app container. The admin account is created on startup.
### Add Your First Scholar
1. Log in at `http://localhost:8000`.
2. Navigate to the Scholars page.
3. Click **Add Scholar**.
4. Enter a Google Scholar profile URL (e.g., `https://scholar.google.com/citations?user=XXXXXXXXXX`) or a Scholar ID.
5. The scheduler will begin fetching publications on the next tick (default: 60 seconds).
### Manual Run
To trigger an immediate ingestion run, use the **Manual Run** button on the Runs page. This respects the minimum run interval (`INGESTION_MIN_RUN_INTERVAL_MINUTES`, default 15 minutes).
## Updating
```bash ```bash
docker compose pull docker compose pull
docker compose up -d docker compose up -d
``` ```
Open: Database migrations run automatically on startup when `MIGRATE_ON_START=1` (the default).
- App/API: `http://localhost:8000`
- Health endpoint: `http://localhost:8000/healthz`
Upgrade:
```bash
docker compose pull
docker compose up -d
```
## Deploy Method B: Local Source + Dev Frontend
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
```
Open:
- API: `http://localhost:8000`
- Frontend dev server: `http://localhost:5173`
Stop:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
```

30
docs/user/overview.md
View file

@ -1,5 +1,29 @@
# User Documentation ---
title: Overview
sidebar_position: 1
---
Use this section if you are deploying and using scholarr. # What is Scholarr?
- [Getting Started](./getting-started.md) Scholarr is a self-hosted service that tracks academic publications from Google Scholar. It runs as a Docker container with a PostgreSQL database and serves a Vue 3 frontend.
## Key Concepts
- **Scholar** - A tracked Google Scholar profile. Scholars are user-scoped: each user manages their own list.
- **Publication** - A globally deduplicated academic work. Publications are shared across all users to avoid duplicate storage.
- **Scholar-Publication Link** - Connects a scholar to a publication for a specific user. Read/unread state, favorites, and visibility live on this link, not on the publication itself.
- **Run** - A single ingestion cycle that fetches new publications for one or more scholars.
- **Identifier** - A DOI, arXiv ID, PMID, or other external key attached to a publication. Multiple identifiers can exist per publication.
## How It Works
1. You add a Google Scholar profile (by ID, URL, or name search).
2. The scheduler periodically scrapes the profile for new publications.
3. Each publication is fingerprinted and deduplicated against the global store.
4. External APIs (arXiv, Crossref, OpenAlex) are queried for identifiers.
5. Unpaywall and arXiv resolve open-access PDF URLs when a DOI is available.
6. The dashboard shows new, unread, and all publications with filtering and search.
## Safety Model
Scholarr enforces rate limits and cooldowns to prevent IP bans from upstream sources. These are not configurable to zero; they are safety floors. See [Scrape Safety Runbook](../operations/scrape-safety-runbook.md) for details.

79
docs/website/docusaurus.config.js
View file

@ -2,8 +2,8 @@
/** @type {import('@docusaurus/types').Config} */ /** @type {import('@docusaurus/types').Config} */
const config = { const config = {
title: "scholarr", title: "Scholarr",
tagline: "Self-hosted scholar tracking docs", tagline: "Self-hosted academic publication tracker",
favicon: "img/favicon.ico", favicon: "img/favicon.ico",
url: "https://justinzeus.github.io", url: "https://justinzeus.github.io",
@ -23,59 +23,52 @@ const config = {
presets: [ presets: [
[ [
"classic", "classic",
{ /** @type {import('@docusaurus/preset-classic').Options} */
({
docs: { docs: {
path: "..", path: "..",
exclude: ["website/**", "README.md"],
routeBasePath: "/", routeBasePath: "/",
sidebarPath: require.resolve("./sidebars.js"), sidebarPath: require.resolve("./sidebars.js"),
editUrl: "https://github.com/JustinZeus/scholarr/tree/main/", exclude: ["website/**", "README.md"],
}, },
blog: false, blog: false,
theme: { theme: {
customCss: require.resolve("./src/css/custom.css"), customCss: require.resolve("./src/css/custom.css"),
}, },
}, }),
], ],
], ],
themeConfig: { themeConfig:
navbar: { /** @type {import('@docusaurus/preset-classic').ThemeConfig} */
title: "scholarr", ({
items: [ navbar: {
{ title: "Scholarr",
type: "docSidebar", items: [
sidebarId: "docsSidebar", { to: "/user/overview", label: "User Guide", position: "left" },
position: "left", {
label: "Docs", to: "/developer/overview",
}, label: "Developer",
{ position: "left",
href: "https://github.com/JustinZeus/scholarr", },
label: "GitHub", {
position: "right", to: "/operations/overview",
}, label: "Operations",
], position: "left",
}, },
footer: { { to: "/reference/overview", label: "Reference", position: "left" },
style: "dark", {
links: [ href: "https://github.com/JustinZeus/scholarr",
{ label: "GitHub",
title: "Docs", position: "right",
items: [{ label: "Index", to: "/" }], },
}, ],
{ },
title: "Community", footer: {
items: [ style: "dark",
{ copyright: `Copyright \u00a9 ${new Date().getFullYear()} Scholarr. Built with Docusaurus.`,
label: "GitHub", },
href: "https://github.com/JustinZeus/scholarr", }),
},
],
},
],
copyright: `Copyright © ${new Date().getFullYear()} scholarr.`,
},
},
}; };
module.exports = config; module.exports = config;

19
docs/website/package.json
View file

@ -1,18 +1,27 @@
{ {
"name": "scholarr-docs", "name": "scholarr-docs",
"version": "0.0.1", "version": "0.1.0",
"private": true, "private": true,
"scripts": { "scripts": {
"docusaurus": "docusaurus",
"start": "docusaurus start", "start": "docusaurus start",
"build": "docusaurus build", "build": "docusaurus build",
"swizzle": "docusaurus swizzle",
"deploy": "docusaurus deploy",
"clear": "docusaurus clear",
"serve": "docusaurus serve" "serve": "docusaurus serve"
}, },
"dependencies": { "dependencies": {
"@docusaurus/core": "3.9.2", "@docusaurus/core": "^3.0.0",
"@docusaurus/preset-classic": "3.9.2", "@docusaurus/preset-classic": "^3.0.0",
"clsx": "^2.1.1",
"prism-react-renderer": "^2.3.1",
"react": "^18.2.0", "react": "^18.2.0",
"react-dom": "^18.2.0" "react-dom": "^18.2.0"
},
"devDependencies": {
"@docusaurus/module-type-aliases": "^3.0.0"
},
"browserslist": {
"production": [">0.5%", "not dead", "not op_mini all"],
"development": ["last 1 chrome version", "last 1 firefox version", "last 1 safari version"]
} }
} }

47
docs/website/sidebars.js
View file

@ -1,38 +1,49 @@
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */ /** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
const sidebars = { const sidebars = {
docsSidebar: [ docs: [
"index", "index",
{ {
type: "category", type: "category",
label: "Users", label: "User Guide",
items: ["user/overview", "user/getting-started"], items: [
"user/overview",
"user/getting-started",
"user/configuration",
],
},
{
type: "category",
label: "Developer",
items: [
"developer/overview",
"developer/architecture",
"developer/local-development",
"developer/contributing",
"developer/ingestion",
"developer/frontend-theme-inventory",
"developer/testing",
],
}, },
{ {
type: "category", type: "category",
label: "Operations", label: "Operations",
items: [ items: [
"operations/overview", "operations/overview",
"operations/scrape-safety-runbook", "operations/deployment",
"operations/database-runbook", "operations/database-runbook",
"operations/migration-checklist", "operations/scrape-safety-runbook",
], "operations/arxiv-runbook",
},
{
type: "category",
label: "Developers",
items: [
"developer/overview",
"developer/documentation-standards",
"developer/local-development",
"developer/architecture",
"developer/contributing",
"developer/frontend-theme-inventory",
], ],
}, },
{ {
type: "category", type: "category",
label: "Reference", label: "Reference",
items: ["reference/overview", "reference/api-contract", "reference/environment"], items: [
"reference/overview",
"reference/api",
"reference/environment",
"reference/changelog",
],
}, },
], ],
}; };

27
docs/website/src/css/custom.css
View file

@ -1,9 +1,22 @@
:root { :root {
--ifm-color-primary: #2a6b4e; --ifm-color-primary: #2e8555;
--ifm-color-primary-dark: #255f46; --ifm-color-primary-dark: #29784c;
--ifm-color-primary-darker: #225942; --ifm-color-primary-darker: #277148;
--ifm-color-primary-darkest: #1c4936; --ifm-color-primary-darkest: #205d3b;
--ifm-color-primary-light: #2f7756; --ifm-color-primary-light: #33925d;
--ifm-color-primary-lighter: #327d5a; --ifm-color-primary-lighter: #359962;
--ifm-color-primary-lightest: #3b946b; --ifm-color-primary-lightest: #3cad6e;
--ifm-code-font-size: 95%;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
}
[data-theme="dark"] {
--ifm-color-primary: #25c2a0;
--ifm-color-primary-dark: #21af90;
--ifm-color-primary-darker: #1fa588;
--ifm-color-primary-darkest: #1a8870;
--ifm-color-primary-light: #29d5b0;
--ifm-color-primary-lighter: #32d8b4;
--ifm-color-primary-lightest: #4fddbf;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
} }

BIN
docs/website/static/img/favicon.ico
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 15 KiB