docs: rebuild documentation from scratch with clean IA
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
6c31b331f7
commit
0c5b199b76
30 changed files with 1647 additions and 649 deletions
|
|
@ -1,23 +0,0 @@
|
|||
# Documentation Index
|
||||
|
||||
This directory contains both docs content and docs tooling.
|
||||
|
||||
## Audience-Based Structure
|
||||
|
||||
- `docs/user/`: onboarding and usage-oriented docs for self-hosters.
|
||||
- `docs/operations/`: runbooks and rollout checklists for ongoing operations.
|
||||
- `docs/developer/`: architecture, contribution standards, and local development workflows.
|
||||
- `docs/reference/`: canonical API and environment contracts.
|
||||
|
||||
## Docs Tooling
|
||||
|
||||
- `docs/website/`: Docusaurus app used to publish docs to GitHub Pages.
|
||||
|
||||
## Recommended Reading Paths
|
||||
|
||||
- Users: `docs/user/getting-started.md`
|
||||
- Operators: `docs/operations/overview.md`
|
||||
- Developers: `docs/developer/overview.md`
|
||||
- Contract consumers: `docs/reference/overview.md`
|
||||
|
||||
Published site: https://justinzeus.github.io/scholarr/
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
# API Contracts
|
||||
|
||||
`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI.
|
||||
|
||||
## DTO Models Structure
|
||||
FastAPI routes dynamically ingest request/response Data Transfer Objects.
|
||||
- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin.
|
||||
- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend.
|
||||
|
||||
## Frontend Contract Requirements
|
||||
The UI exclusively calls routes exposed by FastAPI's `APIRouter`.
|
||||
All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally.
|
||||
|
|
@ -1,36 +1,147 @@
|
|||
# Domain Boundaries
|
||||
---
|
||||
title: Architecture
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Architecture
|
||||
|
||||
## Data Model Rules
|
||||
|
||||
- Scholar tracking is user-scoped.
|
||||
- Publications are global/deduplicated records.
|
||||
- Read/favorite/visibility state stays on scholar-publication link rows.
|
||||
- **Scholar tracking is user-scoped.** Each user manages their own list of tracked scholars. Validate mapping/join tables; never assume global links between users and Scholar IDs.
|
||||
- **Publications are global, deduplicated records.** Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
|
||||
- **State lives on the link.** Read/unread, favorites, and visibility state exist exclusively on the scholar-publication link table, not the global publication table.
|
||||
|
||||
## Service Boundaries
|
||||
## Domain Service Boundaries
|
||||
|
||||
Canonical business logic belongs in `app/services/domains/*`.
|
||||
All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited.
|
||||
|
||||
- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety.
|
||||
- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
|
||||
- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
|
||||
- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
|
||||
- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
|
||||
- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
|
||||
- `app/services/domains/runs/*`: run history and continuation queue operations.
|
||||
- `app/services/domains/portability/*`: import/export workflows.
|
||||
### Ingestion (`app/services/ingestion/`)
|
||||
|
||||
## Frontend Behavior Notes
|
||||
Run orchestration, continuation queue, scheduler integration, and scrape safety policy enforcement. The `ScholarIngestionService` drives the primary data acquisition loop.
|
||||
|
||||
- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
|
||||
- Lists: Long list views use internal scroll containers to prevent viewport overflow.
|
||||
- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.
|
||||
Key modules:
|
||||
- `application.py` - Main ingestion orchestrator
|
||||
- `scheduler.py` - Background tick loop, queue batch processing
|
||||
- `constants.py` - Safety policy constants and floor values
|
||||
- `fingerprints.py` - Publication fingerprinting for deduplication
|
||||
- `types.py` - Ingestion result types and state enums
|
||||
|
||||
## Data Integration & Acquisition Flow
|
||||
### Scholar Parsing (`app/services/scholar/`)
|
||||
|
||||
Fail-fast Google Scholar HTML parsing and source fetch adapters. Handles paginated HTML feeds, extracts publication blocks via regex and DOM invariants (e.g., `gsc_vcd_cib`).
|
||||
|
||||
Key modules:
|
||||
- `parser.py` - HTML parser for publication extraction
|
||||
- `parser_utils.py` - Parsing helpers and DOM selectors
|
||||
- `source.py` - HTTP fetch adapters with browser headers
|
||||
- `profile_rows.py` - Profile metadata extraction
|
||||
- `author_rows.py` - Author citation row parsing
|
||||
- `state_detection.py` - Blocked/CAPTCHA/rate-limit detection
|
||||
|
||||
### Scholar Management (`app/services/scholars/`)
|
||||
|
||||
Scholar CRUD, profile image management, and name-search controls with rate limiting.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Scholar lifecycle operations
|
||||
- `uploads.py` - Image upload handling
|
||||
- `search_hints.py` - Name search with caching and cooldowns
|
||||
- `validators.py` - Input validation for scholar creation
|
||||
|
||||
### Publications (`app/services/publications/`)
|
||||
|
||||
Listing, read-state management, favorite toggles, enrichment scheduling, deduplication, and PDF queue management.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Publication service facade
|
||||
- `listing.py` - Filtered listing with pagination (modes: all/unread/latest)
|
||||
- `queries.py` - Database query builders
|
||||
- `counts.py` - Aggregation counts for dashboard
|
||||
- `dedup.py` - Duplicate detection and merging
|
||||
- `enrichment.py` - Identifier and metadata enrichment orchestration
|
||||
- `pdf_queue.py` - PDF resolution queue policy
|
||||
- `pdf_resolution_pipeline.py` - Multi-source PDF resolution (Unpaywall, arXiv)
|
||||
- `types.py` - Publication DTOs and response types
|
||||
|
||||
### Publication Identifiers (`app/services/publication_identifiers/`)
|
||||
|
||||
Multi-identifier resolution engine. A single publication can have multiple identifiers (DOI, arXiv ID, PMID, PMCID). This decoupled approach replaced the earlier hardcoded DOI-only model.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Identifier gathering orchestration
|
||||
- `normalize.py` - Identifier normalization and validation
|
||||
|
||||
### arXiv (`app/services/arxiv/`)
|
||||
|
||||
Typed API client with global DB-backed throttle, query cache, and in-flight request coalescing.
|
||||
|
||||
Key modules:
|
||||
- `client.py` - HTTP client for arXiv export API
|
||||
- `gateway.py` - Gateway with advisory lock, caching, and coalescing
|
||||
- `cache.py` - Query cache with TTL and max-entry pruning
|
||||
- `rate_limit.py` - Global rate limiter via `arxiv_runtime_state` table
|
||||
- `guards.py` - Load-shedding guards (skip when DOI/arXiv evidence exists)
|
||||
- `parser.py` - Atom XML response parser
|
||||
|
||||
Safety features:
|
||||
- Requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`)
|
||||
- Identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + max-entry pruning
|
||||
- Concurrent identical misses are coalesced in-process (one outbound call serves all waiters)
|
||||
|
||||
### Crossref (`app/services/crossref/`)
|
||||
|
||||
DOI lookup via Crossref REST API with bounded pacing and configurable batch limits.
|
||||
|
||||
### Unpaywall (`app/services/unpaywall/`)
|
||||
|
||||
Open-access PDF resolution via Unpaywall API, with HTML-based PDF link discovery as a fallback.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Unpaywall service facade
|
||||
- `pdf_discovery.py` - HTML page scraping for PDF link candidates
|
||||
|
||||
### OpenAlex (`app/services/openalex/`)
|
||||
|
||||
Metadata matching via OpenAlex API for supplementary identifier resolution.
|
||||
|
||||
Key modules:
|
||||
- `client.py` - OpenAlex API client
|
||||
- `matching.py` - Fuzzy title/author matching
|
||||
|
||||
### Runs (`app/services/runs/`)
|
||||
|
||||
Run history tracking and continuation queue operations.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Run lifecycle management
|
||||
- `queue_service.py` - Continuation queue operations (retry, drop, clear)
|
||||
- `queue_queries.py` - Queue item queries
|
||||
|
||||
### Portability (`app/services/portability/`)
|
||||
|
||||
Import/export workflows for scholar data with full publication state preservation.
|
||||
|
||||
Key modules:
|
||||
- `application.py` - Import/export orchestration
|
||||
- `exporting.py` - Scholar export serialization
|
||||
- `publication_import.py` - Publication import with deduplication
|
||||
- `scholar_import.py` - Scholar import with link reconstruction
|
||||
- `normalize.py` - Payload normalization and validation
|
||||
|
||||
### Database Operations (`app/services/dbops/`)
|
||||
|
||||
Integrity checking, link repair, and near-duplicate repair operations exposed via admin API and CLI scripts.
|
||||
|
||||
Key modules:
|
||||
- `__init__.py` - `collect_integrity_report`, `run_publication_link_repair`
|
||||
- `near_duplicate_repair.py` - Near-duplicate publication detection and merging
|
||||
|
||||
## Data Integration Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
UI[Frontend Vue App] --> API[FastAPI Backend]
|
||||
API --> Scheduler[Background Celery/Async Scheduler]
|
||||
API --> Scheduler[Background Async Scheduler]
|
||||
|
||||
Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
|
||||
Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
|
||||
|
|
@ -48,11 +159,15 @@ graph TD
|
|||
PDFWorker --> |Store PDF Metadata| DB
|
||||
```
|
||||
|
||||
### Identifier Engine Philosophy
|
||||
The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.
|
||||
## API Layer
|
||||
|
||||
### arXiv Safety and Efficiency
|
||||
- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`).
|
||||
- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning.
|
||||
- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
|
||||
- structured logs provide auditability for scheduling/completion/cooldown/cache behavior.
|
||||
Routes live in `app/api/routers/`. All responses under `/api/v1` use a strict envelope format. See [API Reference](../reference/api.md) for the full contract.
|
||||
|
||||
## Middleware Stack
|
||||
|
||||
Applied in `app/main.py`:
|
||||
|
||||
1. **CSRF Protection** - Token-based CSRF validation
|
||||
2. **Session Middleware** - Cookie-based session management (itsdangerous signing)
|
||||
3. **Request Logging** - Structured request/response logging with configurable skip paths
|
||||
4. **Security Headers** - Configurable HTTP security headers and CSP
|
||||
|
|
|
|||
|
|
@ -1,24 +1,78 @@
|
|||
---
|
||||
title: Contributing
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Contributing
|
||||
|
||||
## Scope
|
||||
This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe.
|
||||
## PR Process
|
||||
|
||||
## Essential-File Policy
|
||||
Commit only source-of-truth files required to build, run, test, or document the app.
|
||||
1. Create a feature branch from `main`.
|
||||
2. Make your changes following the code standards below.
|
||||
3. Run tests inside the container (see [Testing](testing.md)).
|
||||
4. Run `ruff check .` and `mypy app/` to catch lint and type errors.
|
||||
5. Open a pull request with a clear title and description.
|
||||
|
||||
Do not commit generated or local-only artifacts, including:
|
||||
- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`
|
||||
- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/`
|
||||
- coverage outputs (`.coverage`, `htmlcov/`)
|
||||
- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`)
|
||||
- local probe/scratch material (`planning/`)
|
||||
## Commit Conventions
|
||||
|
||||
CI enforces this with `scripts/check_no_generated_artifacts.sh`.
|
||||
This project uses [Conventional Commits](https://www.conventionalcommits.org/) with [python-semantic-release](https://python-semantic-release.readthedocs.io/) for automated versioning.
|
||||
|
||||
## Merge Checklist
|
||||
- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas.
|
||||
- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed).
|
||||
- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`).
|
||||
- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change.
|
||||
- [ ] `README.md`, `.env.example`, and deployment notes stay aligned.
|
||||
- [ ] `scripts/check_no_generated_artifacts.sh` passes locally.
|
||||
Commit message format:
|
||||
|
||||
```
|
||||
<type>(<scope>): <description>
|
||||
|
||||
[optional body]
|
||||
```
|
||||
|
||||
Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`.
|
||||
|
||||
Examples:
|
||||
- `feat(scholars): add bulk import from CSV`
|
||||
- `fix(ingestion): handle empty citation blocks`
|
||||
- `docs: update configuration reference`
|
||||
|
||||
## Code Standards
|
||||
|
||||
### Function Length
|
||||
|
||||
Maximum 50 lines per function. Break complex logic into small, testable, single-responsibility functions.
|
||||
|
||||
### DRY
|
||||
|
||||
Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
|
||||
|
||||
### Negative Space Programming
|
||||
|
||||
Use explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
|
||||
|
||||
### Cyclomatic Complexity
|
||||
|
||||
Flatten logic. Use early returns and guard clauses instead of deep nesting. No magic numbers.
|
||||
|
||||
### Domain Service Boundaries
|
||||
|
||||
All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited. Each domain owns its own application service, types, and helpers.
|
||||
|
||||
### API Envelope
|
||||
|
||||
All `/api/v1` responses use the strict envelope format:
|
||||
|
||||
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
|
||||
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
|
||||
|
||||
### Data Isolation
|
||||
|
||||
- Scholar tracking is user-scoped.
|
||||
- Publications are global, deduplicated records.
|
||||
- Read/favorite/visibility state lives on scholar-publication link rows.
|
||||
|
||||
### Scrape Safety
|
||||
|
||||
Rate limits and cooldowns are immutable constraints. They prevent IP bans and must not be optimized away or set to zero.
|
||||
|
||||
## UI Standards
|
||||
|
||||
- Integrate Tailwind with the preset theming system (`frontend/src/theme/presets/`).
|
||||
- Every UI element must have a clear purpose.
|
||||
- Clarity through both styling and language.
|
||||
|
|
|
|||
|
|
@ -1,26 +0,0 @@
|
|||
# Documentation Standards
|
||||
|
||||
This project keeps docs organized by audience and document type.
|
||||
|
||||
## Audience Split
|
||||
|
||||
- `user/`: onboarding and usage tasks for self-hosted users.
|
||||
- `operations/`: runbooks and checklists for production operation.
|
||||
- `developer/`: architecture, contribution workflow, and implementation guides.
|
||||
- `reference/`: stable contracts (API, env variables, configuration behavior).
|
||||
|
||||
## Document Quality Rules
|
||||
|
||||
- One primary audience per page.
|
||||
- Task pages must include prerequisites, exact commands, and verification steps.
|
||||
- Reference pages should be contract-first and avoid procedural noise.
|
||||
- Runbooks should include recovery steps and rollback/safety notes.
|
||||
- Prefer concise pages with strong cross-links over long mixed-purpose pages.
|
||||
|
||||
## Required Top-Level Entrypoints
|
||||
|
||||
- `index.md`: user/developer/operator navigation hub.
|
||||
- `user/getting-started.md`: first-run path.
|
||||
- `developer/local-development.md`: contributor setup and validation path.
|
||||
- `operations/overview.md`: operational playbook index.
|
||||
- `reference/overview.md`: contract index.
|
||||
|
|
@ -1,89 +1,105 @@
|
|||
# Theme Inventory (Phase 0)
|
||||
---
|
||||
title: Frontend Theme Inventory
|
||||
sidebar_position: 6
|
||||
---
|
||||
|
||||
This file captures the semantic color system baseline for the theme refactor.
|
||||
# Frontend Theme Inventory
|
||||
|
||||
## Token Domains
|
||||
|
||||
- `scale`:
|
||||
- `brand` (`50..950`)
|
||||
- `info` (`50..950`)
|
||||
- `success` (`50..950`)
|
||||
- `warning` (`50..950`)
|
||||
- `danger` (`50..950`)
|
||||
- `surface`:
|
||||
- `app`
|
||||
- `nav`
|
||||
- `nav_active`
|
||||
- `card`
|
||||
- `card_muted`
|
||||
- `table`
|
||||
- `table_header`
|
||||
- `input`
|
||||
- `overlay`
|
||||
- `text`:
|
||||
- `primary`
|
||||
- `secondary`
|
||||
- `muted`
|
||||
- `inverse`
|
||||
- `link`
|
||||
- `border`:
|
||||
- `default`
|
||||
- `strong`
|
||||
- `subtle`
|
||||
- `interactive`
|
||||
- `focus`:
|
||||
- `ring`
|
||||
- `ring_offset`
|
||||
- `action` variants (`primary`, `secondary`, `ghost`, `danger`):
|
||||
- `bg`
|
||||
- `border`
|
||||
- `text`
|
||||
- `hover_bg`
|
||||
- `hover_border`
|
||||
- `hover_text`
|
||||
- `state` variants (`info`, `success`, `warning`, `danger`):
|
||||
- `bg`
|
||||
- `border`
|
||||
- `text`
|
||||
The theme system uses semantic tokens organized into domains:
|
||||
|
||||
## Theme Sources
|
||||
### Scale Colors
|
||||
|
||||
Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`.
|
||||
Color ramps from `50` to `950` for each intent:
|
||||
|
||||
Current preset files:
|
||||
- `brand` - Primary brand color
|
||||
- `info` - Informational elements
|
||||
- `success` - Success states
|
||||
- `warning` - Warning states
|
||||
- `danger` - Error/destructive states
|
||||
|
||||
- `frontend/src/theme/presets/parchment.js`
|
||||
- `frontend/src/theme/presets/lilac.js`
|
||||
- `frontend/src/theme/presets/dune.js`
|
||||
- `frontend/src/theme/presets/oatmeal.js`
|
||||
- `frontend/src/theme/presets/scholarly.json`
|
||||
- `frontend/src/theme/presets/graphite.json`
|
||||
- `frontend/src/theme/presets/tide.json`
|
||||
### Surface
|
||||
|
||||
Each preset contains both `light` and `dark` mode token definitions.
|
||||
Background colors for major UI areas:
|
||||
|
||||
## Adoption Status (Phase 0-3 complete baseline)
|
||||
| Token | Usage |
|
||||
|-------|-------|
|
||||
| `app` | Application background |
|
||||
| `nav` | Navigation bar |
|
||||
| `nav_active` | Active navigation item |
|
||||
| `card` | Card backgrounds |
|
||||
| `card_muted` | Muted card variant |
|
||||
| `table` | Table body |
|
||||
| `table_header` | Table header |
|
||||
| `input` | Form inputs |
|
||||
| `overlay` | Modal/dialog overlays |
|
||||
|
||||
Tokenized foundation components:
|
||||
### Text
|
||||
|
||||
- `AppButton`
|
||||
- `AppCard`
|
||||
- `AppCheckbox`
|
||||
- `AppEmptyState`
|
||||
- `AppHelpHint`
|
||||
- `AppInput`
|
||||
- `AppSelect`
|
||||
- `AppTable`
|
||||
- `AppModal`
|
||||
- `AppHeader`
|
||||
- `AppNav`
|
||||
- `AppAlert`
|
||||
- `AppBadge`
|
||||
- `RunStatusBadge`
|
||||
- `QueueHealthBadge`
|
||||
| Token | Usage |
|
||||
|-------|-------|
|
||||
| `primary` | Default text |
|
||||
| `secondary` | Supporting text |
|
||||
| `muted` | Disabled/placeholder text |
|
||||
| `inverse` | Text on dark backgrounds |
|
||||
| `link` | Hyperlinks |
|
||||
|
||||
Hardening in place:
|
||||
### Border
|
||||
|
||||
- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs`
|
||||
- CI enforcement step in `frontend-quality` workflow
|
||||
- Theme preset integrity tests in `frontend/src/theme/presets.test.ts`
|
||||
| Token | Usage |
|
||||
|-------|-------|
|
||||
| `default` | Standard borders |
|
||||
| `strong` | Emphasized borders |
|
||||
| `subtle` | Light separators |
|
||||
| `interactive` | Hover/focus borders |
|
||||
|
||||
### Focus
|
||||
|
||||
| Token | Usage |
|
||||
|-------|-------|
|
||||
| `ring` | Focus ring color |
|
||||
| `ring_offset` | Focus ring offset color |
|
||||
|
||||
### Action Variants
|
||||
|
||||
Each action type (`primary`, `secondary`, `ghost`, `danger`) provides:
|
||||
|
||||
`bg`, `border`, `text`, `hover_bg`, `hover_border`, `hover_text`
|
||||
|
||||
### State Variants
|
||||
|
||||
Each state (`info`, `success`, `warning`, `danger`) provides:
|
||||
|
||||
`bg`, `border`, `text`
|
||||
|
||||
## Theme Presets
|
||||
|
||||
Presets are loaded from `frontend/src/theme/presets/*.{json,js}`. Each contains both `light` and `dark` mode token definitions.
|
||||
|
||||
Current presets:
|
||||
|
||||
| Preset | Format |
|
||||
|--------|--------|
|
||||
| `parchment` | JS |
|
||||
| `lilac` | JS |
|
||||
| `dune` | JS |
|
||||
| `oatmeal` | JS |
|
||||
| `scholarly` | JSON |
|
||||
| `graphite` | JSON |
|
||||
| `tide` | JSON |
|
||||
|
||||
## Tokenized Components
|
||||
|
||||
Foundation components using the token system:
|
||||
|
||||
- `AppButton`, `AppCard`, `AppCheckbox`, `AppEmptyState`
|
||||
- `AppHelpHint`, `AppInput`, `AppSelect`, `AppTable`
|
||||
- `AppModal`, `AppHeader`, `AppNav`, `AppAlert`
|
||||
- `AppBadge`, `RunStatusBadge`, `QueueHealthBadge`
|
||||
|
||||
## Enforcement
|
||||
|
||||
- Token policy check: `frontend/scripts/check_theme_tokens.mjs`
|
||||
- CI step in `frontend-quality` workflow
|
||||
- Preset integrity tests: `frontend/src/theme/presets.test.ts`
|
||||
|
|
|
|||
|
|
@ -1,38 +1,89 @@
|
|||
# Ingestion System & Backoff Strategies
|
||||
---
|
||||
title: Ingestion Pipeline
|
||||
sidebar_position: 5
|
||||
---
|
||||
|
||||
The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans.
|
||||
# Ingestion Pipeline
|
||||
|
||||
## Ingestion Overview
|
||||
The underlying ingestion process:
|
||||
1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
|
||||
2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
|
||||
3. Downloads the paginated HTML feed for a user across multiple page iterations.
|
||||
4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
|
||||
The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans.
|
||||
|
||||
## Handling 429 Too Many Requests
|
||||
Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`).
|
||||
## Pipeline Overview
|
||||
|
||||
Scholarr treats these distinct from random network timeouts.
|
||||
- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s).
|
||||
- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead.
|
||||
1. The scheduler (or a manual trigger) starts a **run** for one or more scholars.
|
||||
2. The service connects via HTTPX with strict browser headers.
|
||||
3. Paginated HTML feeds are downloaded for each scholar profile.
|
||||
4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks.
|
||||
5. Publications are fingerprinted and deduplicated against the global store.
|
||||
6. External APIs resolve additional identifiers.
|
||||
7. The PDF resolution pipeline runs asynchronously for publications with known DOIs.
|
||||
|
||||
## Publication Identifiers Loop
|
||||
Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
|
||||
- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
|
||||
- **API Fetching**: Queries secondary bibliographic platforms sequentially:
|
||||
- `export.arxiv.org/api/query` (Queries by Title and Author strings).
|
||||
- `crossref.restful` APIs (Queries by Title and Author strings).
|
||||
## Rate Limiting & Backoff
|
||||
|
||||
These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.
|
||||
### Network Errors
|
||||
|
||||
Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s).
|
||||
|
||||
### HTTP 429 (Rate Limited)
|
||||
|
||||
When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies:
|
||||
|
||||
- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3)
|
||||
- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s)
|
||||
|
||||
This pauses the pipeline gracefully instead of failing the entire run.
|
||||
|
||||
### Safety Cooldowns
|
||||
|
||||
Threshold-based cooldowns halt all ingestion after repeated failures:
|
||||
|
||||
| Threshold | Variable | Default | Cooldown |
|
||||
|-----------|----------|---------|----------|
|
||||
| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) |
|
||||
| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) |
|
||||
|
||||
## Continuation Queue
|
||||
|
||||
Multi-page ingestion uses a continuation queue to spread load over time:
|
||||
|
||||
- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`)
|
||||
- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s)
|
||||
- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s)
|
||||
- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6)
|
||||
|
||||
Each continuation item is re-enqueued with exponential backoff.
|
||||
|
||||
## Identifier Resolution
|
||||
|
||||
After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers:
|
||||
|
||||
1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches.
|
||||
2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings.
|
||||
3. **Crossref API** - Queries Crossref REST API by title and author strings.
|
||||
|
||||
Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage.
|
||||
|
||||
## PDF Resolution
|
||||
|
||||
Publications with resolved DOIs enter the PDF resolution pipeline:
|
||||
|
||||
1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs.
|
||||
2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates.
|
||||
3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly.
|
||||
|
||||
Auto-retry is configured via `PDF_AUTO_RETRY_*` variables.
|
||||
|
||||
## arXiv Request Controls
|
||||
- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
|
||||
- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
|
||||
- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
|
||||
- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
|
||||
|
||||
## arXiv Observability Events
|
||||
- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
|
||||
- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
|
||||
- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown.
|
||||
- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.
|
||||
- **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
|
||||
- **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`.
|
||||
- **In-flight coalescing**: Duplicate concurrent misses join one outbound request.
|
||||
- **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
|
||||
|
||||
### Observability Events
|
||||
|
||||
| Event | Description |
|
||||
|-------|-------------|
|
||||
| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. |
|
||||
| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. |
|
||||
| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. |
|
||||
| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. |
|
||||
|
|
|
|||
|
|
@ -1,52 +1,100 @@
|
|||
# Developer Local Development
|
||||
---
|
||||
title: Local Development
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
## Start the Dev Stack
|
||||
# Local Development
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker and Docker Compose v2+
|
||||
- Python 3.12+ (for IDE support and local linting)
|
||||
- Node.js 20+ (for frontend development)
|
||||
|
||||
## Starting the Dev Stack
|
||||
|
||||
The development compose file overlays the production config with hot-reload and a separate Vite dev server:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
|
||||
```
|
||||
|
||||
Open:
|
||||
- API: `http://localhost:8000`
|
||||
- Frontend dev server: `http://localhost:5173`
|
||||
This starts three services:
|
||||
|
||||
Stop:
|
||||
| Service | Port | Description |
|
||||
|---------|------|-------------|
|
||||
| `db` | 5432 (internal) | PostgreSQL 15 |
|
||||
| `app` | 8000 | FastAPI backend with `APP_RELOAD=1` |
|
||||
| `frontend` | 5173 | Vite dev server proxying API calls to `app:8000` |
|
||||
|
||||
### Dev-Specific Overrides
|
||||
|
||||
The `docker-compose.dev.yml` file applies these changes:
|
||||
|
||||
- **`app`**: Uses `scholarr-dev:local` image built from the `dev` stage. Mounts the project root as `/app` for hot reload. Disables `SESSION_COOKIE_SECURE` and the built frontend.
|
||||
- **`frontend`**: Node 20 container running `npm install && npm run dev`. Mounts `./frontend` with a named volume for `node_modules`. Uses polling for file watching (`CHOKIDAR_USEPOLLING=1`).
|
||||
|
||||
## Environment Setup
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
## Backend Validation
|
||||
Set at minimum:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration
|
||||
POSTGRES_PASSWORD=localdev
|
||||
SESSION_SECRET_KEY=local-dev-secret-at-least-32-characters
|
||||
SESSION_COOKIE_SECURE=0
|
||||
```
|
||||
|
||||
## Frontend Validation
|
||||
## Running Tests
|
||||
|
||||
All tests run inside containers:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
npm run typecheck
|
||||
npm run test:run
|
||||
npm run build
|
||||
# Run unit tests (default: excludes integration markers)
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest
|
||||
|
||||
# Run integration tests
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest -m integration
|
||||
|
||||
# Run a specific test file
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest tests/unit/test_fingerprints.py -v
|
||||
```
|
||||
|
||||
## Repository Gates
|
||||
See [Testing](testing.md) for markers, fixtures, and conventions.
|
||||
|
||||
## Linting and Type Checking
|
||||
|
||||
```bash
|
||||
python3 scripts/check_frontend_api_contract.py
|
||||
python3 scripts/check_env_contract.py
|
||||
./scripts/check_no_generated_artifacts.sh
|
||||
# Ruff linting
|
||||
ruff check .
|
||||
|
||||
# Ruff formatting check
|
||||
ruff format --check .
|
||||
|
||||
# Mypy type checking
|
||||
mypy app/
|
||||
```
|
||||
|
||||
## Docs Site (Contributor Workflow)
|
||||
Ruff config is in `pyproject.toml`: target Python 3.12, line length 120, rules `E F W I UP B SIM RUF`.
|
||||
|
||||
Docs tooling is colocated in `docs/website/`:
|
||||
## Database Migrations
|
||||
|
||||
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. To run manually:
|
||||
|
||||
```bash
|
||||
cd docs/website
|
||||
npm install
|
||||
npm run build
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
To create a new migration:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
alembic revision --autogenerate -m "description of change"
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,11 +1,65 @@
|
|||
# Developer Documentation
|
||||
---
|
||||
title: Developer Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
Use this section if you are contributing code or running full quality gates.
|
||||
# Developer Overview
|
||||
|
||||
- [Documentation Standards](./documentation-standards.md)
|
||||
- [Local Development](./local-development.md)
|
||||
- [Architecture Boundaries](./architecture.md)
|
||||
- [Ingestion API & Backoff Strategies](./ingestion.md)
|
||||
- [API Contracts](./api-contract.md)
|
||||
- [Contributing](./contributing.md)
|
||||
- [Frontend Theme Inventory](./frontend-theme-inventory.md)
|
||||
Scholarr is a Python 3.12+ FastAPI backend with an async SQLAlchemy ORM layer, PostgreSQL database, and a Vue 3 + TypeScript + Vite frontend.
|
||||
|
||||
## Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|------------|
|
||||
| Backend | Python 3.12+, FastAPI, SQLAlchemy 2.0 (async/asyncpg), Alembic |
|
||||
| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS |
|
||||
| Database | PostgreSQL 15 |
|
||||
| Infrastructure | Multi-stage Docker, Docker Compose |
|
||||
| Linting | ruff (E, F, W, I, UP, B, SIM, RUF), mypy |
|
||||
| Testing | pytest, pytest-asyncio |
|
||||
| Versioning | python-semantic-release, conventional commits |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Build the dev image and start services
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
|
||||
|
||||
# Backend: http://localhost:8000 (hot reload enabled)
|
||||
# Frontend: http://localhost:5173 (Vite dev server with API proxy)
|
||||
```
|
||||
|
||||
See [Local Development](local-development.md) for the full setup guide.
|
||||
|
||||
## Project Layout
|
||||
|
||||
```
|
||||
app/
|
||||
├── api/routers/ # FastAPI route handlers
|
||||
├── auth/ # Session + CSRF middleware
|
||||
├── db/ # SQLAlchemy models, session factory, migrations
|
||||
├── services/ # Domain service modules (see Architecture)
|
||||
│ ├── arxiv/ # arXiv API client, cache, rate limiting
|
||||
│ ├── crossref/ # Crossref DOI lookups
|
||||
│ ├── dbops/ # Database integrity + repair operations
|
||||
│ ├── ingestion/ # Run orchestration, scheduler, safety gates
|
||||
│ ├── openalex/ # OpenAlex metadata matching
|
||||
│ ├── portability/ # Import/export workflows
|
||||
│ ├── publication_identifiers/ # Multi-identifier resolution
|
||||
│ ├── publications/ # Listing, enrichment, dedup, PDF queue
|
||||
│ ├── runs/ # Run history, continuation queue
|
||||
│ ├── scholar/ # HTML parser, source fetch adapters
|
||||
│ ├── scholars/ # Scholar CRUD, image upload, name search
|
||||
│ └── unpaywall/ # Unpaywall PDF discovery
|
||||
├── main.py # App factory, lifespan, middleware stack
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/ # Reusable Vue components
|
||||
│ ├── theme/presets/ # Color theme presets (light + dark)
|
||||
│ └── ...
|
||||
scripts/db/ # Operational database scripts
|
||||
tests/
|
||||
├── unit/ # Fast, no-database tests
|
||||
├── integration/ # Tests requiring database + services
|
||||
└── fixtures/ # Shared test fixtures
|
||||
```
|
||||
|
|
|
|||
102
docs/developer/testing.md
Normal file
102
docs/developer/testing.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
---
|
||||
title: Testing
|
||||
sidebar_position: 7
|
||||
---
|
||||
|
||||
# Testing
|
||||
|
||||
## Running Tests
|
||||
|
||||
All tests must run inside containers:
|
||||
|
||||
```bash
|
||||
# Unit tests (default: excludes integration markers)
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest
|
||||
|
||||
# Integration tests
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest -m integration
|
||||
|
||||
# Specific marker
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest -m db
|
||||
|
||||
# Verbose output for a specific file
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python -m pytest tests/unit/test_fingerprints.py -v
|
||||
```
|
||||
|
||||
## Test Configuration
|
||||
|
||||
From `pyproject.toml`:
|
||||
|
||||
```toml
|
||||
[tool.pytest.ini_options]
|
||||
addopts = "-q -m \"not integration\" --import-mode=importlib"
|
||||
asyncio_mode = "auto"
|
||||
testpaths = ["tests"]
|
||||
```
|
||||
|
||||
- Default run excludes `integration` marked tests
|
||||
- Uses `importlib` import mode to resolve module name collisions
|
||||
- Async tests run automatically (no `@pytest.mark.asyncio` needed)
|
||||
|
||||
## Markers
|
||||
|
||||
| Marker | Description |
|
||||
|--------|-------------|
|
||||
| `integration` | Tests requiring external services (database, network) |
|
||||
| `db` | Tests that validate database behavior and constraints |
|
||||
| `migrations` | Tests focused on Alembic schema migration correctness |
|
||||
| `schema` | Tests focused on multi-tenant schema invariants |
|
||||
| `smoke` | Smoke tests for containerized runtime |
|
||||
|
||||
## Test Tiers
|
||||
|
||||
### Unit Tests (`tests/unit/`)
|
||||
|
||||
Fast, no-database tests. Mock external dependencies. These run by default.
|
||||
|
||||
Examples:
|
||||
- `test_fingerprints.py` - Publication fingerprinting logic
|
||||
- `test_scholar_parser.py` - HTML parsing without network calls
|
||||
- `test_doi_normalize.py` - DOI normalization rules
|
||||
- `test_ingestion_arxiv_rate_limit.py` - Rate limiter behavior
|
||||
- `test_publication_pdf_resolution_pipeline.py` - PDF pipeline logic
|
||||
|
||||
Domain-specific unit tests are organized under `tests/unit/services/domains/`:
|
||||
- `arxiv/` - Cache, client, gateway, guards, parser, rate limit tests
|
||||
- `openalex/` - Client and matching tests
|
||||
- `publications/` - Dedup tests
|
||||
|
||||
### Integration Tests (`tests/integration/`)
|
||||
|
||||
Require a running database. Test full request/response flows and data consistency.
|
||||
|
||||
Examples:
|
||||
- `test_api_v1.py` - API endpoint integration tests
|
||||
- `test_db_integrity.py` - Database integrity checks
|
||||
- `test_run_lifecycle_consistency.py` - Run state machine transitions
|
||||
- `test_deferred_enrichment.py` - Enrichment pipeline with real data
|
||||
- `test_fixture_probe_runs.py` - Fixture-based run probes
|
||||
|
||||
### Smoke Tests
|
||||
|
||||
Marked with `@pytest.mark.smoke`. Validate the containerized runtime starts and serves basic requests.
|
||||
|
||||
## Fixtures
|
||||
|
||||
Test fixtures live in `tests/fixtures/`:
|
||||
|
||||
```
|
||||
tests/fixtures/
|
||||
└── scholar/
|
||||
├── profile_ok_amIMrIEAAAAJ.html # Successful profile HTML
|
||||
└── regression/
|
||||
├── profile_P1RwlvoAAAAJ.html # Regression case
|
||||
├── profile_LZ5D_p4AAAAJ.html # Regression case
|
||||
└── profile_AAAAAAAAAAAA.html # Regression case
|
||||
```
|
||||
|
||||
Scholar HTML fixtures are real Google Scholar profile pages used to test parser robustness against DOM structure changes.
|
||||
|
|
@ -1,33 +1,27 @@
|
|||
---
|
||||
slug: /
|
||||
title: Scholarr Documentation
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
# scholarr Documentation
|
||||
# Scholarr
|
||||
|
||||
This documentation is organized for both users and developers.
|
||||
Scholarr is a self-hosted academic publication tracker. It monitors Google Scholar profiles, discovers new publications, resolves open-access PDFs, and presents everything through a clean Vue 3 dashboard.
|
||||
|
||||
## For Users
|
||||
## Quick Links
|
||||
|
||||
- [User Overview](./user/overview.md)
|
||||
- [Getting Started](./user/getting-started.md)
|
||||
| Section | Description |
|
||||
|---------|-------------|
|
||||
| [User Guide](user/overview.md) | What scholarr does, installation, configuration |
|
||||
| [Developer Guide](developer/overview.md) | Architecture, local development, contributing |
|
||||
| [Operations](operations/overview.md) | Deployment, database runbook, scrape safety |
|
||||
| [Reference](reference/overview.md) | API contract, environment variables, changelog |
|
||||
|
||||
## For Operators
|
||||
## Key Features
|
||||
|
||||
- [Operations Overview](./operations/overview.md)
|
||||
- [Scrape Safety Runbook](./operations/scrape-safety-runbook.md)
|
||||
- [Database Runbook](./operations/database-runbook.md)
|
||||
- [Migration Rollout Checklist](./operations/migration-checklist.md)
|
||||
|
||||
## For Developers
|
||||
|
||||
- [Developer Overview](./developer/overview.md)
|
||||
- [Local Development](./developer/local-development.md)
|
||||
- [Architecture Boundaries](./developer/architecture.md)
|
||||
- [Contributing](./developer/contributing.md)
|
||||
- [Frontend Theme Inventory](./developer/frontend-theme-inventory.md)
|
||||
|
||||
## Reference
|
||||
|
||||
- [Reference Overview](./reference/overview.md)
|
||||
- [API Contract](./reference/api-contract.md)
|
||||
- [Environment Reference](./reference/environment.md)
|
||||
- **Scholar Tracking** - Add Google Scholar profiles by ID, URL, or name search
|
||||
- **Automated Ingestion** - Background scheduler fetches new publications on a configurable interval
|
||||
- **Identifier Resolution** - Cross-references arXiv, Crossref, and OpenAlex for DOIs and metadata
|
||||
- **PDF Discovery** - Resolves open-access PDFs via Unpaywall and arXiv
|
||||
- **Import/Export** - Portable scholar data with full publication state
|
||||
- **Multi-User** - Session-based auth with admin user management
|
||||
- **Theming** - Multiple color presets with light/dark mode support
|
||||
|
|
|
|||
|
|
@ -1,46 +1,80 @@
|
|||
---
|
||||
title: arXiv Runbook
|
||||
sidebar_position: 5
|
||||
---
|
||||
|
||||
# arXiv Operations Runbook
|
||||
|
||||
Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly.
|
||||
|
||||
## Signals To Check
|
||||
- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
|
||||
## Signals to Check
|
||||
|
||||
- Logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
|
||||
- `arxiv_runtime_state` row for cooldown and next-allowed timestamps
|
||||
- `arxiv_query_cache_entries` size and expiry churn
|
||||
|
||||
## Event Field Guide
|
||||
- `wait_seconds`: enforced pre-request delay from the global limiter.
|
||||
- `status_code`: upstream response code from arXiv.
|
||||
- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429.
|
||||
- `source_path`: caller path (`search` or `lookup_ids`).
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `wait_seconds` | Enforced pre-request delay from the global limiter |
|
||||
| `status_code` | Upstream response code from arXiv |
|
||||
| `cooldown_remaining_seconds` | Remaining cooldown when blocked or after 429 |
|
||||
| `source_path` | Caller path (`search` or `lookup_ids`) |
|
||||
|
||||
## Quick SQL Checks
|
||||
|
||||
### Runtime State
|
||||
|
||||
```sql
|
||||
SELECT state_key, next_allowed_at, cooldown_until, updated_at
|
||||
FROM arxiv_runtime_state;
|
||||
```
|
||||
|
||||
### Cache Status
|
||||
|
||||
```sql
|
||||
SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry
|
||||
SELECT count(*) AS cache_rows,
|
||||
min(expires_at) AS earliest_expiry,
|
||||
max(expires_at) AS latest_expiry
|
||||
FROM arxiv_query_cache_entries;
|
||||
```
|
||||
|
||||
## Common Scenarios
|
||||
1. Repeated `arxiv.cooldown_activated` events:
|
||||
- Confirm recent `429` statuses in `arxiv.request_completed`.
|
||||
- Reduce caller pressure (check new title-quality/identifier guards are active).
|
||||
|
||||
### 1. Repeated `arxiv.cooldown_activated` Events
|
||||
|
||||
- Confirm recent `429` statuses in `arxiv.request_completed` logs.
|
||||
- Reduce caller pressure (check title-quality/identifier guards are active).
|
||||
- Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict.
|
||||
|
||||
2. High request latency with few completions:
|
||||
### 2. High Request Latency with Few Completions
|
||||
|
||||
- Inspect `wait_seconds` in `arxiv.request_scheduled`.
|
||||
- Verify only one process path is repeatedly hitting arXiv (`source_path`).
|
||||
- Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears).
|
||||
|
||||
3. Low cache effectiveness:
|
||||
### 3. Low Cache Effectiveness
|
||||
|
||||
- Validate normalized query behavior and caller churn.
|
||||
- Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads.
|
||||
- Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `ARXIV_ENABLED` | `1` | Enable arXiv lookups |
|
||||
| `ARXIV_TIMEOUT_SECONDS` | `3.0` | Request timeout |
|
||||
| `ARXIV_MIN_INTERVAL_SECONDS` | `4.0` | Min interval between requests |
|
||||
| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | `60.0` | Cooldown after 429 |
|
||||
| `ARXIV_DEFAULT_MAX_RESULTS` | `3` | Max results per query |
|
||||
| `ARXIV_CACHE_TTL_SECONDS` | `900` | Cache TTL (15 min) |
|
||||
| `ARXIV_CACHE_MAX_ENTRIES` | `512` | Max cached queries |
|
||||
| `ARXIV_MAILTO` | *(empty)* | Contact email for API headers |
|
||||
|
||||
## Safe Recovery
|
||||
|
||||
1. Pause automated ingestion if rate-limit storms persist.
|
||||
2. Let cooldown expire naturally; avoid manual burst retries.
|
||||
3. Resume and monitor event rates before restoring full load.
|
||||
|
|
|
|||
|
|
@ -1,100 +1,195 @@
|
|||
# Database Operations Runbook
|
||||
---
|
||||
title: Database Runbook
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
This runbook defines backup, restore, and verification procedures for the
|
||||
PostgreSQL database used by `scholarr`.
|
||||
# Database Runbook
|
||||
|
||||
## Objectives
|
||||
## Backup
|
||||
|
||||
- Keep data loss bounded via scheduled backups.
|
||||
- Provide repeatable restore procedures.
|
||||
- Verify backups by running regular restore drills.
|
||||
Use `scripts/db/backup_full.sh` to create logical backups from the running `db` compose service.
|
||||
|
||||
Recommended starting targets:
|
||||
|
||||
- `RPO`: 24 hours (maximum acceptable data loss).
|
||||
- `RTO`: 60 minutes (maximum acceptable restore time).
|
||||
|
||||
## Backup Strategy
|
||||
|
||||
Use logical backups from the running `db` compose service.
|
||||
|
||||
- Custom-format backup (recommended for restore flexibility):
|
||||
### Custom-Format Backup (Default)
|
||||
|
||||
```bash
|
||||
scripts/db/backup_full.sh
|
||||
```
|
||||
|
||||
- Plain SQL backup:
|
||||
Creates a `.dump` file in `./backups/` (e.g., `scholarr_20260227T120000Z.dump`).
|
||||
|
||||
### Plain SQL Backup
|
||||
|
||||
```bash
|
||||
scripts/db/backup_full.sh --plain
|
||||
```
|
||||
|
||||
Optional environment variables:
|
||||
Creates a `.sql` file.
|
||||
|
||||
- `BACKUP_DIR`: destination directory (default: `<repo>/backups`)
|
||||
- `BACKUP_PREFIX`: backup filename prefix (default: `scholarr`)
|
||||
- `USE_DEV_COMPOSE=1`: include `docker-compose.dev.yml`
|
||||
### Options
|
||||
|
||||
## Restore Strategy
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--plain` | Write plain SQL instead of custom-format dump |
|
||||
|
||||
Restore from a `.dump` (custom format) or `.sql` file into the running `db`
|
||||
compose service.
|
||||
### Environment Variables
|
||||
|
||||
- Restore without schema wipe:
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BACKUP_DIR` | `<repo>/backups` | Destination directory |
|
||||
| `BACKUP_PREFIX` | `scholarr` | File prefix |
|
||||
| `USE_DEV_COMPOSE` | `0` | Set to `1` to include `docker-compose.dev.yml` |
|
||||
|
||||
## Restore
|
||||
|
||||
Use `scripts/db/restore_dump.sh` to restore a backup into the running `db` service.
|
||||
|
||||
### Restore a Custom-Format Dump
|
||||
|
||||
```bash
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump
|
||||
```
|
||||
|
||||
- Restore with full `public` schema reset:
|
||||
### Restore with Schema Wipe
|
||||
|
||||
```bash
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260220T120000Z.dump --wipe-public
|
||||
scripts/db/restore_dump.sh --file backups/scholarr_20260227T120000Z.dump --wipe-public
|
||||
```
|
||||
|
||||
## Safety Checklist Before Restore
|
||||
This drops and recreates the `public` schema before restoring. Use with caution.
|
||||
|
||||
1. Pause writes (stop app/scheduler or set maintenance mode).
|
||||
2. Create a fresh pre-restore backup.
|
||||
3. Validate target dump checksum/size.
|
||||
4. Confirm operator, scope, and rollback plan.
|
||||
### Options
|
||||
|
||||
## Post-Restore Verification
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--file <path>` | Required. Path to `.dump` or `.sql` backup file |
|
||||
| `--wipe-public` | Drop and recreate `public` schema before restore |
|
||||
|
||||
1. Run migrations head check.
|
||||
2. Run health endpoint checks.
|
||||
3. Verify table row counts for core tables:
|
||||
- `users`
|
||||
- `scholar_profiles`
|
||||
- `publications`
|
||||
- `scholar_publications`
|
||||
- `crawl_runs`
|
||||
4. Run integrity report and require zero failures:
|
||||
## Integrity Checks
|
||||
|
||||
Use `scripts/db/check_integrity.py` to run database integrity checks.
|
||||
|
||||
### Run Inside Container
|
||||
|
||||
```bash
|
||||
python3 scripts/db/check_integrity.py
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/check_integrity.py
|
||||
```
|
||||
|
||||
5. Run API smoke checks for scholars/publications/runs pages.
|
||||
|
||||
## Data Cleanup and Repair
|
||||
|
||||
Use the audited repair job for scholar-publication relinking cleanup:
|
||||
### With Strict Warnings
|
||||
|
||||
```bash
|
||||
python3 scripts/db/repair_publication_links.py --user-id <id> --requested-by "<operator>" --apply
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/check_integrity.py --strict-warnings
|
||||
```
|
||||
|
||||
Dry-run first, then re-run with `--apply` once scope and summary counts match expectation.
|
||||
Returns non-zero exit code if any warning is present.
|
||||
|
||||
If cleanup changes schema assumptions, follow `docs/operations/migration-checklist.md` before any migration rollout.
|
||||
### Output Format
|
||||
|
||||
## Restore Drill Cadence
|
||||
JSON report:
|
||||
|
||||
- Run at least one full restore drill per month.
|
||||
- Record:
|
||||
- backup artifact used,
|
||||
- restore start/end timestamps,
|
||||
- issues encountered,
|
||||
- achieved `RTO`.
|
||||
```json
|
||||
{
|
||||
"status": "passed",
|
||||
"failures": [],
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
Exit codes:
|
||||
- `0` - All checks passed
|
||||
- `1` - Check failure
|
||||
- `2` - Warnings present (with `--strict-warnings`)
|
||||
|
||||
### Via Admin API
|
||||
|
||||
```
|
||||
GET /api/v1/admin/db/integrity
|
||||
```
|
||||
|
||||
Returns the same integrity report through the API (admin auth required).
|
||||
|
||||
## Publication Link Repair
|
||||
|
||||
Use `scripts/db/repair_publication_links.py` to repair scholar-publication links with audit logging.
|
||||
|
||||
### Dry Run (Default)
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/repair_publication_links.py --user-id 1
|
||||
```
|
||||
|
||||
### Apply Changes
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
|
||||
python scripts/db/repair_publication_links.py --user-id 1 --apply \
|
||||
--requested-by "admin@example.com"
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `--user-id <int>` | Required. Target user ID |
|
||||
| `--scholar-profile-id <int>` | Optional. Filter by scholar profile ID (repeatable) |
|
||||
| `--apply` | Apply changes (default is dry-run) |
|
||||
| `--gc-orphan-publications` | Delete publications with zero links after cleanup |
|
||||
| `--requested-by <string>` | Operator identifier for audit logs |
|
||||
|
||||
### Via Admin API
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/repairs/publication-links
|
||||
```
|
||||
|
||||
## Near-Duplicate Repair
|
||||
|
||||
Detect and merge near-duplicate publications via the admin API:
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/repairs/publication-near-duplicates
|
||||
```
|
||||
|
||||
## PDF Queue Management
|
||||
|
||||
### List Queue
|
||||
|
||||
```
|
||||
GET /api/v1/admin/db/pdf-queue
|
||||
```
|
||||
|
||||
### Requeue Single Item
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/pdf-queue/{id}/requeue
|
||||
```
|
||||
|
||||
### Bulk Requeue
|
||||
|
||||
```
|
||||
POST /api/v1/admin/db/pdf-queue/requeue-all
|
||||
```
|
||||
|
||||
## Migration Procedures
|
||||
|
||||
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`.
|
||||
|
||||
To run manually:
|
||||
|
||||
```bash
|
||||
docker compose exec app alembic upgrade head
|
||||
```
|
||||
|
||||
To check current migration status:
|
||||
|
||||
```bash
|
||||
docker compose exec app alembic current
|
||||
```
|
||||
|
||||
Recommended procedure for production:
|
||||
1. Backup the database before upgrading.
|
||||
2. Pull the new image.
|
||||
3. Start the container; migrations run on startup.
|
||||
4. Verify via `GET /healthz` and check logs for migration output.
|
||||
|
|
|
|||
117
docs/operations/deployment.md
Normal file
117
docs/operations/deployment.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
---
|
||||
title: Deployment
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# Deployment
|
||||
|
||||
## Production Docker Compose
|
||||
|
||||
The default `docker-compose.yml` runs two services:
|
||||
|
||||
| Service | Image | Description |
|
||||
|---------|-------|-------------|
|
||||
| `db` | `postgres:15-alpine` | PostgreSQL database with persistent volume |
|
||||
| `app` | `justinzeus/scholarr:latest` | FastAPI application with embedded frontend |
|
||||
|
||||
### Start
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Stop
|
||||
|
||||
```bash
|
||||
docker compose down
|
||||
```
|
||||
|
||||
### Update
|
||||
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### View Logs
|
||||
|
||||
```bash
|
||||
docker compose logs -f app
|
||||
docker compose logs -f db
|
||||
```
|
||||
|
||||
## Required Environment Variables
|
||||
|
||||
These must be set in `.env` or the shell environment:
|
||||
|
||||
```bash
|
||||
POSTGRES_PASSWORD=<secure-password>
|
||||
SESSION_SECRET_KEY=<random-string-32-chars-minimum>
|
||||
```
|
||||
|
||||
## Volumes
|
||||
|
||||
| Volume | Mount | Description |
|
||||
|--------|-------|-------------|
|
||||
| `postgres_data` | `/var/lib/postgresql/data` | Database files |
|
||||
| `scholar_uploads` | `/var/lib/scholarr/uploads` | Scholar profile images |
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Database
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $POSTGRES_USER -d $POSTGRES_DB"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
```
|
||||
|
||||
### Application
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS http://localhost:8000/healthz >/dev/null || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 12
|
||||
```
|
||||
|
||||
The app service depends on `db` with `condition: service_healthy`, so it waits for the database to be ready.
|
||||
|
||||
## Database Readiness Wait
|
||||
|
||||
The app has built-in database readiness polling:
|
||||
|
||||
- `DB_WAIT_TIMEOUT_SECONDS` (default: 60) - Max wait time
|
||||
- `DB_WAIT_INTERVAL_SECONDS` (default: 2) - Poll interval
|
||||
|
||||
## Auto-Migration
|
||||
|
||||
Set `MIGRATE_ON_START=1` (default) to run Alembic migrations automatically on startup.
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
- Run a single `app` instance to avoid scheduler conflicts (the scheduler is process-local).
|
||||
- arXiv requests are globally serialized via a PostgreSQL advisory lock, so multiple instances safely share the rate limiter.
|
||||
- Database pool defaults: 5 base connections + 10 overflow. Adjust `DATABASE_POOL_SIZE` and `DATABASE_POOL_MAX_OVERFLOW` for higher loads.
|
||||
|
||||
## Admin Bootstrap
|
||||
|
||||
For first-time setup without manual database access:
|
||||
|
||||
```bash
|
||||
BOOTSTRAP_ADMIN_ON_START=1
|
||||
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
|
||||
BOOTSTRAP_ADMIN_PASSWORD=<secure-password>
|
||||
```
|
||||
|
||||
Set `BOOTSTRAP_ADMIN_FORCE_PASSWORD=1` to reset an existing admin password.
|
||||
|
||||
## Security Hardening
|
||||
|
||||
- Set `SESSION_COOKIE_SECURE=1` when serving over HTTPS.
|
||||
- Enable HSTS: `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED=1`.
|
||||
- Review CSP policy in `SECURITY_CSP_POLICY` for your domain.
|
||||
- Set `UNPAYWALL_EMAIL` and `ARXIV_MAILTO` for polite API pool access.
|
||||
|
|
@ -1,48 +0,0 @@
|
|||
# Migration Checklist
|
||||
|
||||
Use this checklist for every schema/data migration.
|
||||
|
||||
## 1. Design Review
|
||||
|
||||
- Define change type: `expand`, `backfill`, `contract`.
|
||||
- Document expected lock behavior and index impact.
|
||||
- Confirm backward compatibility with currently deployed app version.
|
||||
- Define rollback strategy before implementation.
|
||||
|
||||
## 2. Pre-Migration Controls
|
||||
|
||||
- Capture a fresh backup before applying migration.
|
||||
- Confirm migration head revision and working tree cleanliness.
|
||||
- Prepare validation queries for new/changed tables and indexes.
|
||||
- Identify high-risk tables (large row count, hot write paths).
|
||||
|
||||
## 3. Implementation Standards
|
||||
|
||||
- Keep migrations idempotent when feasible.
|
||||
- Prefer additive steps first (`nullable`, new index, new table).
|
||||
- For destructive changes, separate into later contract migration.
|
||||
- Avoid large blocking rewrites in a single deployment step.
|
||||
|
||||
## 4. Verification
|
||||
|
||||
- Apply migration in staging against production-like snapshot.
|
||||
- Verify:
|
||||
- expected tables/columns/indexes,
|
||||
- app startup and health endpoint,
|
||||
- affected API flows,
|
||||
- data consistency queries.
|
||||
|
||||
## 5. Rollout and Recovery
|
||||
|
||||
- Apply migration during planned window.
|
||||
- Monitor logs/errors and DB metrics during rollout.
|
||||
- If rollback needed:
|
||||
- follow downgrade/recovery runbook,
|
||||
- restore from backup if downgrade is unsafe,
|
||||
- document incident timeline.
|
||||
|
||||
## 6. Post-Migration Tasks
|
||||
|
||||
- Update `README.md` / `.env.example` / ops docs.
|
||||
- Add/update integration tests for new schema assumptions.
|
||||
- Record migration notes in changelog.
|
||||
|
|
@ -1,8 +1,30 @@
|
|||
# Operations Documentation
|
||||
---
|
||||
title: Operations Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
Use this section for production operations and incident/runbook workflows.
|
||||
# Operations Overview
|
||||
|
||||
- [Scrape Safety Runbook](./scrape-safety-runbook.md)
|
||||
- [arXiv Runbook](./arxiv-runbook.md)
|
||||
- [Database Runbook](./database-runbook.md)
|
||||
- [Migration Rollout Checklist](./migration-checklist.md)
|
||||
This section covers production deployment, database administration, and scrape safety operations.
|
||||
|
||||
## Quick Links
|
||||
|
||||
| Guide | Description |
|
||||
|-------|-------------|
|
||||
| [Deployment](deployment.md) | Production Docker setup, scaling, health checks |
|
||||
| [Database Runbook](database-runbook.md) | Backup, restore, integrity checks, repair procedures |
|
||||
| [Scrape Safety Runbook](scrape-safety-runbook.md) | Rate limiting, cooldowns, CAPTCHA handling |
|
||||
| [arXiv Runbook](arxiv-runbook.md) | arXiv rate limits, cache tuning, query patterns |
|
||||
|
||||
## Health Check
|
||||
|
||||
The app exposes `GET /healthz` for container orchestration:
|
||||
|
||||
```bash
|
||||
curl -fsS http://localhost:8000/healthz
|
||||
```
|
||||
|
||||
Docker Compose healthcheck config:
|
||||
- Interval: 10s
|
||||
- Timeout: 5s
|
||||
- Retries: 12
|
||||
|
|
|
|||
|
|
@ -1,74 +1,76 @@
|
|||
---
|
||||
title: Scrape Safety Runbook
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Scrape Safety Runbook
|
||||
|
||||
This runbook covers operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
|
||||
Operational handling for scrape safety cooldowns, threshold alerts, and blocked-IP events.
|
||||
|
||||
## Key Signals
|
||||
|
||||
Use structured log `event` fields (recommended with `LOG_FORMAT=json`):
|
||||
|
||||
- `ingestion.safety_policy_blocked_run_start`
|
||||
- `ingestion.safety_cooldown_entered`
|
||||
- `ingestion.safety_cooldown_cleared`
|
||||
- `ingestion.alert_blocked_failure_threshold_exceeded`
|
||||
- `ingestion.alert_network_failure_threshold_exceeded`
|
||||
- `ingestion.alert_retry_scheduled_threshold_exceeded`
|
||||
- `api.runs.manual_blocked_policy`
|
||||
- `api.runs.manual_blocked_safety`
|
||||
- `scheduler.run_skipped_safety_cooldown`
|
||||
- `scheduler.run_skipped_safety_cooldown_precheck`
|
||||
- `scheduler.queue_item_deferred_safety_cooldown`
|
||||
| Event | Description |
|
||||
|-------|-------------|
|
||||
| `ingestion.safety_policy_blocked_run_start` | Run blocked by safety policy |
|
||||
| `ingestion.safety_cooldown_entered` | Cooldown activated |
|
||||
| `ingestion.safety_cooldown_cleared` | Cooldown expired |
|
||||
| `ingestion.alert_blocked_failure_threshold_exceeded` | Blocked failure threshold tripped |
|
||||
| `ingestion.alert_network_failure_threshold_exceeded` | Network failure threshold tripped |
|
||||
| `ingestion.alert_retry_scheduled_threshold_exceeded` | Retry threshold tripped |
|
||||
| `api.runs.manual_blocked_policy` | Manual run blocked by policy |
|
||||
| `api.runs.manual_blocked_safety` | Manual run blocked by safety cooldown |
|
||||
| `scheduler.run_skipped_safety_cooldown` | Scheduled run skipped due to cooldown |
|
||||
| `scheduler.run_skipped_safety_cooldown_precheck` | Scheduler precheck blocked |
|
||||
| `scheduler.queue_item_deferred_safety_cooldown` | Queue item deferred due to cooldown |
|
||||
|
||||
Each event includes metric-style fields (`metric_name`, `metric_value`) for straightforward log-based alert rules.
|
||||
Each event includes metric-style fields (`metric_name`, `metric_value`) for log-based alert rules.
|
||||
|
||||
## Recommended Alert Rules
|
||||
|
||||
- Cooldown enters:
|
||||
- Trigger on `event=ingestion.safety_cooldown_entered`.
|
||||
- Repeated start blocks:
|
||||
- Trigger on high rate of `event=api.runs.manual_blocked_safety`.
|
||||
- Threshold trips:
|
||||
- Trigger on any of:
|
||||
- `ingestion.alert_blocked_failure_threshold_exceeded`
|
||||
- `ingestion.alert_network_failure_threshold_exceeded`
|
||||
- `ingestion.alert_retry_scheduled_threshold_exceeded`
|
||||
- Scheduler pressure:
|
||||
- Trigger on sustained `scheduler.queue_item_deferred_safety_cooldown`.
|
||||
- **Cooldown enters**: Trigger on `event=ingestion.safety_cooldown_entered`
|
||||
- **Repeated start blocks**: High rate of `event=api.runs.manual_blocked_safety`
|
||||
- **Threshold trips**: Any of the `*_threshold_exceeded` events
|
||||
- **Scheduler pressure**: Sustained `scheduler.queue_item_deferred_safety_cooldown`
|
||||
|
||||
## If Your IP Appears Blocked
|
||||
|
||||
Symptoms:
|
||||
### Symptoms
|
||||
|
||||
- cooldown reason `blocked_failure_threshold_exceeded`
|
||||
- parse state `blocked_or_captcha`
|
||||
- redirects toward Google account sign-in flows
|
||||
- Cooldown reason: `blocked_failure_threshold_exceeded`
|
||||
- Parse state: `blocked_or_captcha`
|
||||
- Redirects toward Google account sign-in flows
|
||||
|
||||
Actions:
|
||||
### Actions
|
||||
|
||||
1. Stop manual retries immediately.
|
||||
2. Let cooldown expire; do not spam retriggers.
|
||||
3. Increase `INGESTION_MIN_REQUEST_DELAY_SECONDS` and user request delay values.
|
||||
4. Reduce concurrency pressure (keep one scheduler instance).
|
||||
5. Keep name-search disabled/WIP for now if login-gated responses persist.
|
||||
5. Keep name-search disabled if login-gated responses persist.
|
||||
6. Resume with a small monitored run and verify blocked rate drops.
|
||||
|
||||
Avoid:
|
||||
### Avoid
|
||||
|
||||
- aggressive rapid retries
|
||||
- rotating through risky scraping patterns that increase challenge rates
|
||||
- bypass/captcha-solving workflows that may violate source platform rules
|
||||
- Aggressive rapid retries
|
||||
- Rotating through risky scraping patterns that increase challenge rates
|
||||
- Bypass/CAPTCHA-solving workflows that may violate source platform rules
|
||||
|
||||
## Environment Controls
|
||||
|
||||
Policy floors and safety controls:
|
||||
|
||||
- `INGESTION_MIN_REQUEST_DELAY_SECONDS`
|
||||
- `INGESTION_MIN_RUN_INTERVAL_MINUTES`
|
||||
- `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`
|
||||
- `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`
|
||||
- `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`
|
||||
- `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`
|
||||
- `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`
|
||||
- `INGESTION_MANUAL_RUN_ALLOWED`
|
||||
- `INGESTION_AUTOMATION_ALLOWED`
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | `2` | Floor delay between requests |
|
||||
| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | `15` | Minimum time between runs |
|
||||
| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | `1` | Blocked failures before alert |
|
||||
| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | `2` | Network failures before alert |
|
||||
| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | `3` | Retries before alert |
|
||||
| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | `1800` | Cooldown after blocked threshold (30 min) |
|
||||
| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | `900` | Cooldown after network threshold (15 min) |
|
||||
| `INGESTION_MANUAL_RUN_ALLOWED` | `1` | Enable manual runs |
|
||||
| `INGESTION_AUTOMATION_ALLOWED` | `1` | Enable automated runs |
|
||||
|
||||
Apply stricter values first, then relax slowly only after sustained healthy runs.
|
||||
|
|
|
|||
|
|
@ -1,31 +0,0 @@
|
|||
# API Contract
|
||||
|
||||
## Envelope Invariant
|
||||
|
||||
All API responses under `/api/v1` use one of these envelopes:
|
||||
|
||||
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
|
||||
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
|
||||
|
||||
`meta.request_id` must be present for success and error responses.
|
||||
|
||||
Binary media assets are served outside `/api/v1` (for example, `GET /scholar-images/{scholar_profile_id}/upload`).
|
||||
|
||||
## Publications Semantics
|
||||
|
||||
- `GET /api/v1/publications` supports `mode=all|unread|latest`.
|
||||
- `mode=new` is currently accepted as a compatibility alias for `latest`.
|
||||
- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
|
||||
- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`.
|
||||
- `unread` represents read-state (`is_read=false`).
|
||||
- `latest` represents discovery-state (`first seen in the latest completed run`).
|
||||
|
||||
Publication payloads expose:
|
||||
- `pub_url` (canonical scholar detail URL)
|
||||
- `doi` (normalized DOI)
|
||||
- `pdf_url` (resolved OA PDF when available)
|
||||
|
||||
## Scholar Portability
|
||||
|
||||
- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state.
|
||||
- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication.
|
||||
177
docs/reference/api.md
Normal file
177
docs/reference/api.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
---
|
||||
title: API Contract
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
# API Contract
|
||||
|
||||
## Envelope Invariant
|
||||
|
||||
All API responses under `/api/v1` use one of these envelopes:
|
||||
|
||||
### Success
|
||||
|
||||
```json
|
||||
{
|
||||
"data": "...",
|
||||
"meta": {
|
||||
"request_id": "..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Error
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"code": "...",
|
||||
"message": "...",
|
||||
"details": "..."
|
||||
},
|
||||
"meta": {
|
||||
"request_id": "..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`meta.request_id` is present on both success and error responses.
|
||||
|
||||
### Binary Assets
|
||||
|
||||
Binary media assets are served outside `/api/v1`:
|
||||
|
||||
```
|
||||
GET /scholar-images/{scholar_profile_id}/upload
|
||||
```
|
||||
|
||||
## DTO Structure
|
||||
|
||||
Scholarr uses strictly typed Pydantic V2 models serialized through OpenAPI v3 via FastAPI.
|
||||
|
||||
- `PublicationListItem` exposes a `.display_identifier` property that resolves the highest-confidence identifier regardless of backend origin, rather than a hardcoded `.doi`.
|
||||
- Frontend TypeScript types are compiled from the OpenAPI spec to ensure type safety across the stack.
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Auth
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `POST` | `/api/v1/auth/login` | Login (rate-limited sliding window) |
|
||||
| `GET` | `/api/v1/auth/me` | Current session user |
|
||||
| `GET` | `/api/v1/auth/csrf` | Bootstrap CSRF token |
|
||||
| `POST` | `/api/v1/auth/change-password` | Change password |
|
||||
| `POST` | `/api/v1/auth/logout` | Logout |
|
||||
|
||||
### Scholars
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/scholars` | List tracked scholars |
|
||||
| `POST` | `/api/v1/scholars` | Create scholar (auto-enqueue, metadata hydration) |
|
||||
| `GET` | `/api/v1/scholars/search` | Search author candidates by name |
|
||||
| `PATCH` | `/api/v1/scholars/{id}/toggle` | Toggle enabled status |
|
||||
| `DELETE` | `/api/v1/scholars/{id}` | Delete scholar |
|
||||
| `PUT` | `/api/v1/scholars/{id}/image/url` | Update image URL |
|
||||
| `POST` | `/api/v1/scholars/{id}/image/upload` | Upload image |
|
||||
| `DELETE` | `/api/v1/scholars/{id}/image` | Clear image customization |
|
||||
|
||||
### Scholar Portability
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/scholars/export` | Export tracked scholars and publication link state |
|
||||
| `POST` | `/api/v1/scholars/import` | Import scholars with global publication deduplication |
|
||||
|
||||
The export payload includes scholar metadata, tracked publication data, and link state (read/unread, favorites). Import preserves global deduplication.
|
||||
|
||||
### Publications
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/publications` | List publications (filtered, paginated) |
|
||||
| `POST` | `/api/v1/publications/mark-all-read` | Mark all as read |
|
||||
| `POST` | `/api/v1/publications/mark-read` | Mark selected as read |
|
||||
| `POST` | `/api/v1/publications/{id}/retry-pdf` | Retry PDF resolution |
|
||||
| `POST` | `/api/v1/publications/{id}/favorite` | Toggle favorite |
|
||||
|
||||
#### Publication Modes
|
||||
|
||||
`GET /api/v1/publications` supports a `mode` parameter:
|
||||
|
||||
| Mode | Description |
|
||||
|------|-------------|
|
||||
| `all` | All publications |
|
||||
| `unread` | Publications with `is_read=false` |
|
||||
| `latest` | Publications first seen in the latest completed run |
|
||||
|
||||
`mode=new` is accepted as a compatibility alias for `latest`.
|
||||
|
||||
#### Pagination
|
||||
|
||||
Query parameters: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
|
||||
|
||||
Response pagination fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"page": 1,
|
||||
"page_size": 20,
|
||||
"has_prev": false,
|
||||
"has_next": true,
|
||||
"total_count": 142
|
||||
}
|
||||
```
|
||||
|
||||
#### Publication Payload Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `pub_url` | Canonical scholar detail URL |
|
||||
| `doi` | Normalized DOI |
|
||||
| `pdf_url` | Resolved open-access PDF URL (when available) |
|
||||
| `display_identifier` | Highest-confidence identifier regardless of source |
|
||||
|
||||
### Runs
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/runs` | List runs with safety state |
|
||||
| `GET` | `/api/v1/runs/{run_id}` | Run detail with scholar results |
|
||||
| `POST` | `/api/v1/runs/{run_id}/cancel` | Cancel active run |
|
||||
| `POST` | `/api/v1/runs/manual` | Trigger manual run (idempotent, safety-checked) |
|
||||
| `GET` | `/api/v1/runs/queue/items` | List queue items |
|
||||
| `POST` | `/api/v1/runs/queue/{id}/retry` | Retry queue item |
|
||||
| `POST` | `/api/v1/runs/queue/{id}/drop` | Drop queue item |
|
||||
| `DELETE` | `/api/v1/runs/queue/{id}` | Clear queue item |
|
||||
| `GET` | `/api/v1/runs/{run_id}/stream` | Stream run events (SSE) |
|
||||
|
||||
### Settings
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/settings` | Get user settings (with cooldown expiry check) |
|
||||
| `PUT` | `/api/v1/settings` | Update settings (interval, delay, nav, API keys) |
|
||||
|
||||
### Admin - User Management
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/admin/users` | List all users |
|
||||
| `POST` | `/api/v1/admin/users` | Create user |
|
||||
| `PATCH` | `/api/v1/admin/users/{id}/active` | Set user active status |
|
||||
| `POST` | `/api/v1/admin/users/{id}/reset-password` | Reset password |
|
||||
|
||||
### Admin - Database Operations
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| `GET` | `/api/v1/admin/db/integrity` | Get integrity report |
|
||||
| `GET` | `/api/v1/admin/db/repair-jobs` | List repair jobs |
|
||||
| `GET` | `/api/v1/admin/db/pdf-queue` | List PDF queue |
|
||||
| `POST` | `/api/v1/admin/db/pdf-queue/{id}/requeue` | Requeue single PDF |
|
||||
| `POST` | `/api/v1/admin/db/pdf-queue/requeue-all` | Bulk requeue missing PDFs |
|
||||
| `POST` | `/api/v1/admin/db/repairs/publication-links` | Trigger link repair |
|
||||
| `POST` | `/api/v1/admin/db/repairs/publication-near-duplicates` | Trigger dedup repair |
|
||||
| `POST` | `/api/v1/admin/db/drop-all-publications` | Drop all publications (destructive) |
|
||||
12
docs/reference/changelog.md
Normal file
12
docs/reference/changelog.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
---
|
||||
title: Changelog
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
# Changelog
|
||||
|
||||
This changelog is auto-generated by [python-semantic-release](https://python-semantic-release.readthedocs.io/).
|
||||
|
||||
Release versions follow [Semantic Versioning](https://semver.org/). Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/).
|
||||
|
||||
<!-- semantic-release will insert entries here -->
|
||||
|
|
@ -1,68 +1,32 @@
|
|||
# Environment Reference and Audit
|
||||
---
|
||||
title: Environment Variables
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
This document is the source-of-truth audit for what remains configurable versus internal defaults.
|
||||
# Environment Variables Quick Reference
|
||||
|
||||
## Decision Summary
|
||||
All environment variables are documented in detail in [Configuration](../user/configuration.md).
|
||||
|
||||
- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras.
|
||||
- Move to internal defaults only: none at this time.
|
||||
- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted.
|
||||
## Required Variables
|
||||
|
||||
## Configurable Variables (By Section)
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `POSTGRES_PASSWORD` | PostgreSQL password |
|
||||
| `SESSION_SECRET_KEY` | Session cookie signing key (32+ characters) |
|
||||
|
||||
### Compose + Database
|
||||
## Categories
|
||||
|
||||
`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE`
|
||||
| Category | Key Variables |
|
||||
|----------|--------------|
|
||||
| Database | `POSTGRES_DB`, `POSTGRES_USER`, `DATABASE_URL`, `DATABASE_POOL_*` |
|
||||
| Runtime | `APP_HOST`, `APP_PORT`, `MIGRATE_ON_START`, `FRONTEND_ENABLED` |
|
||||
| Auth | `SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_*` |
|
||||
| Security | `SECURITY_HEADERS_ENABLED`, `SECURITY_CSP_*`, `SECURITY_STRICT_TRANSPORT_*` |
|
||||
| Logging | `LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS` |
|
||||
| Scheduler | `SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_*_BATCH_SIZE` |
|
||||
| Ingestion | `INGESTION_*` (safety floors, cooldowns, retry policies) |
|
||||
| Scholar | `SCHOLAR_IMAGE_*`, `SCHOLAR_NAME_SEARCH_*` |
|
||||
| Enrichment | `UNPAYWALL_*`, `ARXIV_*`, `CROSSREF_*`, `OPENALEX_*`, `PDF_AUTO_RETRY_*` |
|
||||
| Bootstrap | `BOOTSTRAP_ADMIN_*`, `DB_WAIT_*` |
|
||||
|
||||
### App Runtime + Networking
|
||||
|
||||
`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR`
|
||||
|
||||
### Database Pool
|
||||
|
||||
`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS`
|
||||
|
||||
### Frontend Dev Overrides
|
||||
|
||||
`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET`
|
||||
|
||||
### Auth + Session
|
||||
|
||||
`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS`
|
||||
|
||||
### HTTP Security Headers + CSP
|
||||
|
||||
`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD`
|
||||
|
||||
### Logging
|
||||
|
||||
`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS`
|
||||
|
||||
### Scheduler + Ingestion Safety
|
||||
|
||||
`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `SCHEDULER_PDF_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_RATE_LIMIT_RETRIES`, `INGESTION_RATE_LIMIT_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
|
||||
|
||||
### Scholar Images + Name Search Safety
|
||||
|
||||
`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD`
|
||||
|
||||
### OA Enrichment + PDF Resolution
|
||||
|
||||
`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `ARXIV_ENABLED`, `ARXIV_TIMEOUT_SECONDS`, `ARXIV_MIN_INTERVAL_SECONDS`, `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS`, `ARXIV_DEFAULT_MAX_RESULTS`, `ARXIV_CACHE_TTL_SECONDS`, `ARXIV_CACHE_MAX_ENTRIES`, `ARXIV_MAILTO`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`, `OPENALEX_API_KEY`, `CROSSREF_API_TOKEN`, `CROSSREF_API_MAILTO`
|
||||
|
||||
### Startup Bootstrap + DB Wait
|
||||
|
||||
`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS`
|
||||
|
||||
## Drift Guard
|
||||
|
||||
CI enforces env parity with:
|
||||
|
||||
```bash
|
||||
python3 scripts/check_env_contract.py
|
||||
```
|
||||
|
||||
The check fails when:
|
||||
- `app/settings.py` references a variable missing from `.env.example`.
|
||||
- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras).
|
||||
- `.env.example` contains duplicate keys.
|
||||
See [Configuration](../user/configuration.md) for the complete table with types, defaults, and descriptions.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,14 @@
|
|||
# Reference Documentation
|
||||
---
|
||||
title: Reference
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
Use this section for canonical contracts and configuration references.
|
||||
# Reference
|
||||
|
||||
- [API Contract](./api-contract.md)
|
||||
- [Environment Reference](./environment.md)
|
||||
Quick-reference documentation for the scholarr API, environment variables, and release changelog.
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [API Contract](api.md) | Envelope spec, endpoints, DTO structure, publication semantics |
|
||||
| [Environment Variables](environment.md) | Quick-reference table linking to full configuration docs |
|
||||
| [Changelog](changelog.md) | Auto-generated release history |
|
||||
|
|
|
|||
178
docs/user/configuration.md
Normal file
178
docs/user/configuration.md
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
---
|
||||
title: Configuration
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
# Configuration
|
||||
|
||||
All configuration is done through environment variables. Copy `.env.example` to `.env` and adjust values as needed.
|
||||
|
||||
## Compose & Database
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `POSTGRES_DB` | string | `scholar` | PostgreSQL database name |
|
||||
| `POSTGRES_USER` | string | `scholar` | PostgreSQL user |
|
||||
| `POSTGRES_PASSWORD` | string | **required** | PostgreSQL password |
|
||||
| `DATABASE_URL` | string | derived | SQLAlchemy async connection string |
|
||||
| `TEST_DATABASE_URL` | string | derived | Override for test database. If empty, tests derive `<db_name>_test` |
|
||||
| `SCHOLARR_IMAGE` | string | `justinzeus/scholarr:latest` | Docker image for the app service |
|
||||
|
||||
## App Runtime & Networking
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `APP_NAME` | string | `scholarr` | Application name used in logs and headers |
|
||||
| `APP_HOST` | string | `0.0.0.0` | Bind address |
|
||||
| `APP_PORT` | int | `8000` | Internal port |
|
||||
| `APP_HOST_PORT` | int | `8000` | Host-mapped port |
|
||||
| `APP_RELOAD` | bool | `0` | Enable uvicorn auto-reload (dev only) |
|
||||
| `MIGRATE_ON_START` | bool | `1` | Run Alembic migrations on startup |
|
||||
| `FRONTEND_ENABLED` | bool | `1` | Serve the built Vue frontend |
|
||||
| `FRONTEND_DIST_DIR` | string | `/app/frontend/dist` | Path to compiled frontend assets |
|
||||
|
||||
## Database Pool
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `DATABASE_POOL_MODE` | string | `auto` | Pool mode (`auto`, `fixed`, `null`) |
|
||||
| `DATABASE_POOL_SIZE` | int | `5` | Base pool size |
|
||||
| `DATABASE_POOL_MAX_OVERFLOW` | int | `10` | Maximum overflow connections |
|
||||
| `DATABASE_POOL_TIMEOUT_SECONDS` | int | `30` | Connection acquisition timeout |
|
||||
|
||||
## Frontend Dev Overrides
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `FRONTEND_HOST_PORT` | int | `5173` | Host port for Vite dev server |
|
||||
| `CHOKIDAR_USEPOLLING` | bool | `1` | Enable polling for file watchers in containers |
|
||||
| `VITE_DEV_API_PROXY_TARGET` | string | `http://app:8000` | Backend URL for Vite proxy |
|
||||
|
||||
## Auth & Session
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `SESSION_SECRET_KEY` | string | **required** | Signing key for session cookies (32+ chars) |
|
||||
| `SESSION_COOKIE_SECURE` | bool | `1` | Set `Secure` flag on session cookie (disable for local HTTP dev) |
|
||||
| `LOGIN_RATE_LIMIT_ATTEMPTS` | int | `5` | Max login attempts per window |
|
||||
| `LOGIN_RATE_LIMIT_WINDOW_SECONDS` | int | `60` | Sliding window for login rate limiting |
|
||||
|
||||
## HTTP Security Headers & CSP
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `SECURITY_HEADERS_ENABLED` | bool | `1` | Enable security response headers |
|
||||
| `SECURITY_X_CONTENT_TYPE_OPTIONS` | string | `nosniff` | X-Content-Type-Options header value |
|
||||
| `SECURITY_X_FRAME_OPTIONS` | string | `DENY` | X-Frame-Options header value |
|
||||
| `SECURITY_REFERRER_POLICY` | string | `strict-origin-when-cross-origin` | Referrer-Policy header value |
|
||||
| `SECURITY_PERMISSIONS_POLICY` | string | *(restrictive)* | Permissions-Policy header value |
|
||||
| `SECURITY_CROSS_ORIGIN_OPENER_POLICY` | string | `same-origin` | Cross-Origin-Opener-Policy header |
|
||||
| `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY` | string | `same-origin` | Cross-Origin-Resource-Policy header |
|
||||
| `SECURITY_CSP_ENABLED` | bool | `1` | Enable Content-Security-Policy header |
|
||||
| `SECURITY_CSP_POLICY` | string | *(restrictive)* | CSP for app routes |
|
||||
| `SECURITY_CSP_DOCS_POLICY` | string | *(docs-specific)* | CSP for documentation routes |
|
||||
| `SECURITY_CSP_REPORT_ONLY` | bool | `0` | Use report-only mode for CSP |
|
||||
| `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED` | bool | `0` | Enable HSTS header |
|
||||
| `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE` | int | `31536000` | HSTS max-age in seconds |
|
||||
| `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS` | bool | `1` | HSTS includeSubDomains directive |
|
||||
| `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD` | bool | `0` | HSTS preload directive |
|
||||
|
||||
## Logging
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `LOG_LEVEL` | string | `INFO` | Root log level (`DEBUG`, `INFO`, `WARNING`, `ERROR`) |
|
||||
| `LOG_FORMAT` | string | `console` | Log format (`console` or `json`) |
|
||||
| `LOG_REQUESTS` | bool | `1` | Log HTTP requests |
|
||||
| `LOG_UVICORN_ACCESS` | bool | `0` | Enable uvicorn access log |
|
||||
| `LOG_REQUEST_SKIP_PATHS` | string | `/healthz` | Comma-separated paths to exclude from request logging |
|
||||
| `LOG_REDACT_FIELDS` | string | *(empty)* | Comma-separated field names to redact in logs |
|
||||
|
||||
## Scheduler & Ingestion Safety
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `SCHEDULER_ENABLED` | bool | `1` | Enable the background scheduler |
|
||||
| `SCHEDULER_TICK_SECONDS` | int | `60` | Scheduler poll interval |
|
||||
| `SCHEDULER_QUEUE_BATCH_SIZE` | int | `10` | Max scholars processed per tick |
|
||||
| `SCHEDULER_PDF_QUEUE_BATCH_SIZE` | int | `15` | Max PDF resolutions per tick |
|
||||
| `INGESTION_AUTOMATION_ALLOWED` | bool | `1` | Allow automated (scheduled) runs |
|
||||
| `INGESTION_MANUAL_RUN_ALLOWED` | bool | `1` | Allow manually triggered runs |
|
||||
| `INGESTION_MIN_RUN_INTERVAL_MINUTES` | int | `15` | Minimum time between runs |
|
||||
| `INGESTION_MIN_REQUEST_DELAY_SECONDS` | int | `2` | Floor delay between external requests |
|
||||
| `INGESTION_NETWORK_ERROR_RETRIES` | int | `1` | Retries on network errors |
|
||||
| `INGESTION_RETRY_BACKOFF_SECONDS` | float | `1.0` | Base backoff for network retries |
|
||||
| `INGESTION_RATE_LIMIT_RETRIES` | int | `3` | Retries on 429 responses |
|
||||
| `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` | float | `30.0` | Backoff per 429 retry |
|
||||
| `INGESTION_MAX_PAGES_PER_SCHOLAR` | int | `30` | Max paginated pages per scholar |
|
||||
| `INGESTION_PAGE_SIZE` | int | `100` | Publications per page |
|
||||
| `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | int | `1` | Blocked failures before alert |
|
||||
| `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | int | `2` | Network failures before alert |
|
||||
| `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD` | int | `3` | Scheduled retries before alert |
|
||||
| `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` | int | `1800` | Cooldown after blocked-failure threshold (30 min) |
|
||||
| `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` | int | `900` | Cooldown after network-failure threshold (15 min) |
|
||||
| `INGESTION_CONTINUATION_QUEUE_ENABLED` | bool | `1` | Enable continuation queue for multi-page ingestion |
|
||||
| `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` | int | `120` | Base delay for continuation queue items |
|
||||
| `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` | int | `3600` | Max delay for continuation queue items |
|
||||
| `INGESTION_CONTINUATION_MAX_ATTEMPTS` | int | `6` | Max continuation attempts per scholar |
|
||||
|
||||
## Scholar Images & Name Search Safety
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `SCHOLAR_IMAGE_UPLOAD_DIR` | string | `/var/lib/scholarr/uploads` | Directory for uploaded scholar images |
|
||||
| `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES` | int | `2000000` | Max image upload size (2 MB) |
|
||||
| `SCHOLAR_NAME_SEARCH_ENABLED` | bool | `1` | Enable name-based scholar search |
|
||||
| `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS` | int | `21600` | Cache TTL for successful searches (6 hours) |
|
||||
| `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS` | int | `300` | Cache TTL for blocked search results (5 min) |
|
||||
| `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES` | int | `512` | Max cache entries for name search |
|
||||
| `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS` | float | `8.0` | Min interval between name searches |
|
||||
| `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS` | float | `2.0` | Random jitter added to search interval |
|
||||
| `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD` | int | `1` | Blocked results before cooldown |
|
||||
| `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS` | int | `1800` | Cooldown after blocked name search (30 min) |
|
||||
| `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD` | int | `2` | Retries before alert |
|
||||
| `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` | int | `3` | Cooldown rejections before alert |
|
||||
|
||||
## OA Enrichment & PDF Resolution
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `UNPAYWALL_ENABLED` | bool | `1` | Enable Unpaywall DOI lookups |
|
||||
| `UNPAYWALL_EMAIL` | string | *(empty)* | Polite pool email for Unpaywall API |
|
||||
| `UNPAYWALL_TIMEOUT_SECONDS` | float | `4.0` | Request timeout |
|
||||
| `UNPAYWALL_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Unpaywall requests |
|
||||
| `UNPAYWALL_MAX_ITEMS_PER_REQUEST` | int | `20` | Max items per batch |
|
||||
| `UNPAYWALL_RETRY_COOLDOWN_SECONDS` | int | `1800` | Cooldown after repeated failures |
|
||||
| `UNPAYWALL_PDF_DISCOVERY_ENABLED` | bool | `1` | Enable HTML-based PDF link discovery |
|
||||
| `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES` | int | `5` | Max candidate URLs to probe |
|
||||
| `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES` | int | `500000` | Max HTML response size to parse |
|
||||
| `ARXIV_ENABLED` | bool | `1` | Enable arXiv API lookups |
|
||||
| `ARXIV_TIMEOUT_SECONDS` | float | `3.0` | Request timeout |
|
||||
| `ARXIV_MIN_INTERVAL_SECONDS` | float | `4.0` | Min interval between arXiv requests |
|
||||
| `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` | float | `60.0` | Cooldown after arXiv 429 |
|
||||
| `ARXIV_DEFAULT_MAX_RESULTS` | int | `3` | Default max results per query |
|
||||
| `ARXIV_CACHE_TTL_SECONDS` | int | `900` | Query cache TTL (15 min) |
|
||||
| `ARXIV_CACHE_MAX_ENTRIES` | int | `512` | Max cached queries |
|
||||
| `ARXIV_MAILTO` | string | *(empty)* | Contact email for arXiv API headers |
|
||||
| `PDF_AUTO_RETRY_INTERVAL_SECONDS` | int | `86400` | Auto-retry interval for failed PDFs (24 hours) |
|
||||
| `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS` | int | `3600` | First retry interval (1 hour) |
|
||||
| `PDF_AUTO_RETRY_MAX_ATTEMPTS` | int | `3` | Max auto-retry attempts |
|
||||
| `CROSSREF_ENABLED` | bool | `1` | Enable Crossref lookups |
|
||||
| `CROSSREF_MAX_ROWS` | int | `10` | Max rows per Crossref query |
|
||||
| `CROSSREF_TIMEOUT_SECONDS` | float | `8.0` | Request timeout |
|
||||
| `CROSSREF_MIN_INTERVAL_SECONDS` | float | `0.6` | Min interval between Crossref requests |
|
||||
| `CROSSREF_MAX_LOOKUPS_PER_REQUEST` | int | `8` | Max lookups per ingestion request |
|
||||
| `OPENALEX_API_KEY` | string | *(empty)* | OpenAlex API key (optional) |
|
||||
| `CROSSREF_API_TOKEN` | string | *(empty)* | Crossref Plus API token (optional) |
|
||||
| `CROSSREF_API_MAILTO` | string | *(empty)* | Crossref polite pool email |
|
||||
|
||||
## Startup Bootstrap & DB Wait
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `BOOTSTRAP_ADMIN_ON_START` | bool | `0` | Create admin user on startup |
|
||||
| `BOOTSTRAP_ADMIN_EMAIL` | string | *(empty)* | Admin email address |
|
||||
| `BOOTSTRAP_ADMIN_PASSWORD` | string | *(empty)* | Admin password |
|
||||
| `BOOTSTRAP_ADMIN_FORCE_PASSWORD` | bool | `0` | Overwrite existing admin password |
|
||||
| `DB_WAIT_TIMEOUT_SECONDS` | int | `60` | Max seconds to wait for database readiness |
|
||||
| `DB_WAIT_INTERVAL_SECONDS` | int | `2` | Poll interval while waiting for database |
|
||||
|
|
@ -1,41 +1,86 @@
|
|||
# User Getting Started
|
||||
---
|
||||
title: Getting Started
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
## Required Setup
|
||||
# Getting Started
|
||||
|
||||
1. Copy `.env.example` to `.env`.
|
||||
2. Set `POSTGRES_PASSWORD`.
|
||||
3. Set `SESSION_SECRET_KEY`.
|
||||
## Prerequisites
|
||||
|
||||
## Deploy Method A: Prebuilt Image
|
||||
- Docker and Docker Compose v2+
|
||||
- A machine with at least 512 MB RAM
|
||||
|
||||
## Installation
|
||||
|
||||
1. Clone the repository:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/JustinZeus/scholarr.git
|
||||
cd scholarr
|
||||
```
|
||||
|
||||
2. Copy the example environment file:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
3. Edit `.env` and set the required secrets:
|
||||
|
||||
```bash
|
||||
# Required: database password
|
||||
POSTGRES_PASSWORD=your-secure-password
|
||||
|
||||
# Required: session signing key (32+ random characters)
|
||||
SESSION_SECRET_KEY=your-random-secret-key
|
||||
```
|
||||
|
||||
4. Start the stack:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
5. Verify the service is healthy:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
curl http://localhost:8000/healthz
|
||||
```
|
||||
|
||||
The app is now available at `http://localhost:8000`.
|
||||
|
||||
## First Run
|
||||
|
||||
### Bootstrap an Admin User
|
||||
|
||||
Set these environment variables before first start (or add them to `.env`):
|
||||
|
||||
```bash
|
||||
BOOTSTRAP_ADMIN_ON_START=1
|
||||
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
|
||||
BOOTSTRAP_ADMIN_PASSWORD=your-admin-password
|
||||
```
|
||||
|
||||
Restart the app container. The admin account is created on startup.
|
||||
|
||||
### Add Your First Scholar
|
||||
|
||||
1. Log in at `http://localhost:8000`.
|
||||
2. Navigate to the Scholars page.
|
||||
3. Click **Add Scholar**.
|
||||
4. Enter a Google Scholar profile URL (e.g., `https://scholar.google.com/citations?user=XXXXXXXXXX`) or a Scholar ID.
|
||||
5. The scheduler will begin fetching publications on the next tick (default: 60 seconds).
|
||||
|
||||
### Manual Run
|
||||
|
||||
To trigger an immediate ingestion run, use the **Manual Run** button on the Runs page. This respects the minimum run interval (`INGESTION_MIN_RUN_INTERVAL_MINUTES`, default 15 minutes).
|
||||
|
||||
## Updating
|
||||
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Open:
|
||||
- App/API: `http://localhost:8000`
|
||||
- Health endpoint: `http://localhost:8000/healthz`
|
||||
|
||||
Upgrade:
|
||||
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Deploy Method B: Local Source + Dev Frontend
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
|
||||
```
|
||||
|
||||
Open:
|
||||
- API: `http://localhost:8000`
|
||||
- Frontend dev server: `http://localhost:5173`
|
||||
|
||||
Stop:
|
||||
|
||||
```bash
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
|
||||
```
|
||||
Database migrations run automatically on startup when `MIGRATE_ON_START=1` (the default).
|
||||
|
|
|
|||
|
|
@ -1,5 +1,29 @@
|
|||
# User Documentation
|
||||
---
|
||||
title: Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
Use this section if you are deploying and using scholarr.
|
||||
# What is Scholarr?
|
||||
|
||||
- [Getting Started](./getting-started.md)
|
||||
Scholarr is a self-hosted service that tracks academic publications from Google Scholar. It runs as a Docker container with a PostgreSQL database and serves a Vue 3 frontend.
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Scholar** - A tracked Google Scholar profile. Scholars are user-scoped: each user manages their own list.
|
||||
- **Publication** - A globally deduplicated academic work. Publications are shared across all users to avoid duplicate storage.
|
||||
- **Scholar-Publication Link** - Connects a scholar to a publication for a specific user. Read/unread state, favorites, and visibility live on this link, not on the publication itself.
|
||||
- **Run** - A single ingestion cycle that fetches new publications for one or more scholars.
|
||||
- **Identifier** - A DOI, arXiv ID, PMID, or other external key attached to a publication. Multiple identifiers can exist per publication.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. You add a Google Scholar profile (by ID, URL, or name search).
|
||||
2. The scheduler periodically scrapes the profile for new publications.
|
||||
3. Each publication is fingerprinted and deduplicated against the global store.
|
||||
4. External APIs (arXiv, Crossref, OpenAlex) are queried for identifiers.
|
||||
5. Unpaywall and arXiv resolve open-access PDF URLs when a DOI is available.
|
||||
6. The dashboard shows new, unread, and all publications with filtering and search.
|
||||
|
||||
## Safety Model
|
||||
|
||||
Scholarr enforces rate limits and cooldowns to prevent IP bans from upstream sources. These are not configurable to zero; they are safety floors. See [Scrape Safety Runbook](../operations/scrape-safety-runbook.md) for details.
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@
|
|||
|
||||
/** @type {import('@docusaurus/types').Config} */
|
||||
const config = {
|
||||
title: "scholarr",
|
||||
tagline: "Self-hosted scholar tracking docs",
|
||||
title: "Scholarr",
|
||||
tagline: "Self-hosted academic publication tracker",
|
||||
favicon: "img/favicon.ico",
|
||||
|
||||
url: "https://justinzeus.github.io",
|
||||
|
|
@ -23,32 +23,40 @@ const config = {
|
|||
presets: [
|
||||
[
|
||||
"classic",
|
||||
{
|
||||
/** @type {import('@docusaurus/preset-classic').Options} */
|
||||
({
|
||||
docs: {
|
||||
path: "..",
|
||||
exclude: ["website/**", "README.md"],
|
||||
routeBasePath: "/",
|
||||
sidebarPath: require.resolve("./sidebars.js"),
|
||||
editUrl: "https://github.com/JustinZeus/scholarr/tree/main/",
|
||||
exclude: ["website/**", "README.md"],
|
||||
},
|
||||
blog: false,
|
||||
theme: {
|
||||
customCss: require.resolve("./src/css/custom.css"),
|
||||
},
|
||||
},
|
||||
}),
|
||||
],
|
||||
],
|
||||
|
||||
themeConfig: {
|
||||
themeConfig:
|
||||
/** @type {import('@docusaurus/preset-classic').ThemeConfig} */
|
||||
({
|
||||
navbar: {
|
||||
title: "scholarr",
|
||||
title: "Scholarr",
|
||||
items: [
|
||||
{ to: "/user/overview", label: "User Guide", position: "left" },
|
||||
{
|
||||
type: "docSidebar",
|
||||
sidebarId: "docsSidebar",
|
||||
to: "/developer/overview",
|
||||
label: "Developer",
|
||||
position: "left",
|
||||
label: "Docs",
|
||||
},
|
||||
{
|
||||
to: "/operations/overview",
|
||||
label: "Operations",
|
||||
position: "left",
|
||||
},
|
||||
{ to: "/reference/overview", label: "Reference", position: "left" },
|
||||
{
|
||||
href: "https://github.com/JustinZeus/scholarr",
|
||||
label: "GitHub",
|
||||
|
|
@ -58,24 +66,9 @@ const config = {
|
|||
},
|
||||
footer: {
|
||||
style: "dark",
|
||||
links: [
|
||||
{
|
||||
title: "Docs",
|
||||
items: [{ label: "Index", to: "/" }],
|
||||
},
|
||||
{
|
||||
title: "Community",
|
||||
items: [
|
||||
{
|
||||
label: "GitHub",
|
||||
href: "https://github.com/JustinZeus/scholarr",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
copyright: `Copyright © ${new Date().getFullYear()} scholarr.`,
|
||||
},
|
||||
copyright: `Copyright \u00a9 ${new Date().getFullYear()} Scholarr. Built with Docusaurus.`,
|
||||
},
|
||||
}),
|
||||
};
|
||||
|
||||
module.exports = config;
|
||||
|
|
|
|||
|
|
@ -1,18 +1,27 @@
|
|||
{
|
||||
"name": "scholarr-docs",
|
||||
"version": "0.0.1",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"docusaurus": "docusaurus",
|
||||
"start": "docusaurus start",
|
||||
"build": "docusaurus build",
|
||||
"swizzle": "docusaurus swizzle",
|
||||
"deploy": "docusaurus deploy",
|
||||
"clear": "docusaurus clear",
|
||||
"serve": "docusaurus serve"
|
||||
},
|
||||
"dependencies": {
|
||||
"@docusaurus/core": "3.9.2",
|
||||
"@docusaurus/preset-classic": "3.9.2",
|
||||
"clsx": "^2.1.1",
|
||||
"prism-react-renderer": "^2.3.1",
|
||||
"@docusaurus/core": "^3.0.0",
|
||||
"@docusaurus/preset-classic": "^3.0.0",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@docusaurus/module-type-aliases": "^3.0.0"
|
||||
},
|
||||
"browserslist": {
|
||||
"production": [">0.5%", "not dead", "not op_mini all"],
|
||||
"development": ["last 1 chrome version", "last 1 firefox version", "last 1 safari version"]
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,38 +1,49 @@
|
|||
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */
|
||||
const sidebars = {
|
||||
docsSidebar: [
|
||||
docs: [
|
||||
"index",
|
||||
{
|
||||
type: "category",
|
||||
label: "Users",
|
||||
items: ["user/overview", "user/getting-started"],
|
||||
label: "User Guide",
|
||||
items: [
|
||||
"user/overview",
|
||||
"user/getting-started",
|
||||
"user/configuration",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Developer",
|
||||
items: [
|
||||
"developer/overview",
|
||||
"developer/architecture",
|
||||
"developer/local-development",
|
||||
"developer/contributing",
|
||||
"developer/ingestion",
|
||||
"developer/frontend-theme-inventory",
|
||||
"developer/testing",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Operations",
|
||||
items: [
|
||||
"operations/overview",
|
||||
"operations/scrape-safety-runbook",
|
||||
"operations/deployment",
|
||||
"operations/database-runbook",
|
||||
"operations/migration-checklist",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Developers",
|
||||
items: [
|
||||
"developer/overview",
|
||||
"developer/documentation-standards",
|
||||
"developer/local-development",
|
||||
"developer/architecture",
|
||||
"developer/contributing",
|
||||
"developer/frontend-theme-inventory",
|
||||
"operations/scrape-safety-runbook",
|
||||
"operations/arxiv-runbook",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Reference",
|
||||
items: ["reference/overview", "reference/api-contract", "reference/environment"],
|
||||
items: [
|
||||
"reference/overview",
|
||||
"reference/api",
|
||||
"reference/environment",
|
||||
"reference/changelog",
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,9 +1,22 @@
|
|||
:root {
|
||||
--ifm-color-primary: #2a6b4e;
|
||||
--ifm-color-primary-dark: #255f46;
|
||||
--ifm-color-primary-darker: #225942;
|
||||
--ifm-color-primary-darkest: #1c4936;
|
||||
--ifm-color-primary-light: #2f7756;
|
||||
--ifm-color-primary-lighter: #327d5a;
|
||||
--ifm-color-primary-lightest: #3b946b;
|
||||
--ifm-color-primary: #2e8555;
|
||||
--ifm-color-primary-dark: #29784c;
|
||||
--ifm-color-primary-darker: #277148;
|
||||
--ifm-color-primary-darkest: #205d3b;
|
||||
--ifm-color-primary-light: #33925d;
|
||||
--ifm-color-primary-lighter: #359962;
|
||||
--ifm-color-primary-lightest: #3cad6e;
|
||||
--ifm-code-font-size: 95%;
|
||||
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
|
||||
}
|
||||
|
||||
[data-theme="dark"] {
|
||||
--ifm-color-primary: #25c2a0;
|
||||
--ifm-color-primary-dark: #21af90;
|
||||
--ifm-color-primary-darker: #1fa588;
|
||||
--ifm-color-primary-darkest: #1a8870;
|
||||
--ifm-color-primary-light: #29d5b0;
|
||||
--ifm-color-primary-lighter: #32d8b4;
|
||||
--ifm-color-primary-lightest: #4fddbf;
|
||||
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 15 KiB |
Loading…
Add table
Add a link
Reference in a new issue