justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

docs: rebuild documentation from scratch with clean IA

Browse source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 09:39:27 +01:00
parent 6c31b331f7
commit 0c5b199b76
30 changed files with 1647 additions and 649 deletions
Download patch file Download diff file Expand all files Collapse all files

12
docs/developer/api-contract.md
View file

@ -1,12 +0,0 @@
# API Contracts
`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI.
## DTO Models Structure
FastAPI routes dynamically ingest request/response Data Transfer Objects.
- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin.
- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend.
## Frontend Contract Requirements
The UI exclusively calls routes exposed by FastAPI's `APIRouter`.
All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally.

181
docs/developer/architecture.md
View file

@ -1,58 +1,173 @@
# Domain Boundaries
---
title: Architecture
sidebar_position: 2
---
# Architecture
## Data Model Rules
- Scholar tracking is user-scoped.
- Publications are global/deduplicated records.
- Read/favorite/visibility state stays on scholar-publication link rows.
- **Scholar tracking is user-scoped.** Each user manages their own list of tracked scholars. Validate mapping/join tables; never assume global links between users and Scholar IDs.
- **Publications are global, deduplicated records.** Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
- **State lives on the link.** Read/unread, favorites, and visibility state exist exclusively on the scholar-publication link table, not the global publication table.
## Service Boundaries
## Domain Service Boundaries
Canonical business logic belongs in `app/services/domains/*`.
All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited.
- `app/services/domains/ingestion/*`: run orchestration, continuation queue, scheduler, and scrape safety.
- `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
- `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
- `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
- `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
- `app/services/domains/runs/*`: run history and continuation queue operations.
- `app/services/domains/portability/*`: import/export workflows.
### Ingestion (`app/services/ingestion/`)
## Frontend Behavior Notes
Run orchestration, continuation queue, scheduler integration, and scrape safety policy enforcement. The `ScholarIngestionService` drives the primary data acquisition loop.
- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
- Lists: Long list views use internal scroll containers to prevent viewport overflow.
- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.
Key modules:
- `application.py` - Main ingestion orchestrator
- `scheduler.py` - Background tick loop, queue batch processing
- `constants.py` - Safety policy constants and floor values
- `fingerprints.py` - Publication fingerprinting for deduplication
- `types.py` - Ingestion result types and state enums
## Data Integration & Acquisition Flow
### Scholar Parsing (`app/services/scholar/`)
Fail-fast Google Scholar HTML parsing and source fetch adapters. Handles paginated HTML feeds, extracts publication blocks via regex and DOM invariants (e.g., `gsc_vcd_cib`).
Key modules:
- `parser.py` - HTML parser for publication extraction
- `parser_utils.py` - Parsing helpers and DOM selectors
- `source.py` - HTTP fetch adapters with browser headers
- `profile_rows.py` - Profile metadata extraction
- `author_rows.py` - Author citation row parsing
- `state_detection.py` - Blocked/CAPTCHA/rate-limit detection
### Scholar Management (`app/services/scholars/`)
Scholar CRUD, profile image management, and name-search controls with rate limiting.
Key modules:
- `application.py` - Scholar lifecycle operations
- `uploads.py` - Image upload handling
- `search_hints.py` - Name search with caching and cooldowns
- `validators.py` - Input validation for scholar creation
### Publications (`app/services/publications/`)
Listing, read-state management, favorite toggles, enrichment scheduling, deduplication, and PDF queue management.
Key modules:
- `application.py` - Publication service facade
- `listing.py` - Filtered listing with pagination (modes: all/unread/latest)
- `queries.py` - Database query builders
- `counts.py` - Aggregation counts for dashboard
- `dedup.py` - Duplicate detection and merging
- `enrichment.py` - Identifier and metadata enrichment orchestration
- `pdf_queue.py` - PDF resolution queue policy
- `pdf_resolution_pipeline.py` - Multi-source PDF resolution (Unpaywall, arXiv)
- `types.py` - Publication DTOs and response types
### Publication Identifiers (`app/services/publication_identifiers/`)
Multi-identifier resolution engine. A single publication can have multiple identifiers (DOI, arXiv ID, PMID, PMCID). This decoupled approach replaced the earlier hardcoded DOI-only model.
Key modules:
- `application.py` - Identifier gathering orchestration
- `normalize.py` - Identifier normalization and validation
### arXiv (`app/services/arxiv/`)
Typed API client with global DB-backed throttle, query cache, and in-flight request coalescing.
Key modules:
- `client.py` - HTTP client for arXiv export API
- `gateway.py` - Gateway with advisory lock, caching, and coalescing
- `cache.py` - Query cache with TTL and max-entry pruning
- `rate_limit.py` - Global rate limiter via `arxiv_runtime_state` table
- `guards.py` - Load-shedding guards (skip when DOI/arXiv evidence exists)
- `parser.py` - Atom XML response parser
Safety features:
- Requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`)
- Identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + max-entry pruning
- Concurrent identical misses are coalesced in-process (one outbound call serves all waiters)
### Crossref (`app/services/crossref/`)
DOI lookup via Crossref REST API with bounded pacing and configurable batch limits.
### Unpaywall (`app/services/unpaywall/`)
Open-access PDF resolution via Unpaywall API, with HTML-based PDF link discovery as a fallback.
Key modules:
- `application.py` - Unpaywall service facade
- `pdf_discovery.py` - HTML page scraping for PDF link candidates
### OpenAlex (`app/services/openalex/`)
Metadata matching via OpenAlex API for supplementary identifier resolution.
Key modules:
- `client.py` - OpenAlex API client
- `matching.py` - Fuzzy title/author matching
### Runs (`app/services/runs/`)
Run history tracking and continuation queue operations.
Key modules:
- `application.py` - Run lifecycle management
- `queue_service.py` - Continuation queue operations (retry, drop, clear)
- `queue_queries.py` - Queue item queries
### Portability (`app/services/portability/`)
Import/export workflows for scholar data with full publication state preservation.
Key modules:
- `application.py` - Import/export orchestration
- `exporting.py` - Scholar export serialization
- `publication_import.py` - Publication import with deduplication
- `scholar_import.py` - Scholar import with link reconstruction
- `normalize.py` - Payload normalization and validation
### Database Operations (`app/services/dbops/`)
Integrity checking, link repair, and near-duplicate repair operations exposed via admin API and CLI scripts.
Key modules:
- `__init__.py` - `collect_integrity_report`, `run_publication_link_repair`
- `near_duplicate_repair.py` - Near-duplicate publication detection and merging
## Data Integration Flow
```mermaid
graph TD
UI[Frontend Vue App] --> API[FastAPI Backend]
API --> Scheduler[Background Celery/Async Scheduler]
API --> Scheduler[Background Async Scheduler]
Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
IdentifierModule -->|Search arXiv API| API1(arXiv)
IdentifierModule -->|Search Crossref API| API2(Crossref)
IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)]
Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline]
DB --> |Identified DOIs| PDFPipeline
PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API)
Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker]
PDFWorker --> |Store PDF Metadata| DB
PDFWorker --> |Store PDF Metadata| DB
```
### Identifier Engine Philosophy
The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.
## API Layer
### arXiv Safety and Efficiency
- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`).
- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning.
- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
- structured logs provide auditability for scheduling/completion/cooldown/cache behavior.
Routes live in `app/api/routers/`. All responses under `/api/v1` use a strict envelope format. See [API Reference](../reference/api.md) for the full contract.
## Middleware Stack
Applied in `app/main.py`:
1. **CSRF Protection** - Token-based CSRF validation
2. **Session Middleware** - Cookie-based session management (itsdangerous signing)
3. **Request Logging** - Structured request/response logging with configurable skip paths
4. **Security Headers** - Configurable HTTP security headers and CSP

90
docs/developer/contributing.md
View file

@ -1,24 +1,78 @@
---
title: Contributing
sidebar_position: 4
---
# Contributing
## Scope
This project favors small, reviewable pull requests that keep runtime behavior clear and operationally safe.
## PR Process
## Essential-File Policy
Commit only source-of-truth files required to build, run, test, or document the app.
1. Create a feature branch from `main`.
2. Make your changes following the code standards below.
3. Run tests inside the container (see [Testing](testing.md)).
4. Run `ruff check .` and `mypy app/` to catch lint and type errors.
5. Open a pull request with a clear title and description.
Do not commit generated or local-only artifacts, including:
- `__pycache__/`, `*.pyc`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`
- frontend build/install outputs like `frontend/dist/`, `frontend/node_modules/`, `frontend/.vite/`
- coverage outputs (`.coverage`, `htmlcov/`)
- packaging/build leftovers (`*.egg-info/`, `build/`, `dist/`)
- local probe/scratch material (`planning/`)
## Commit Conventions
CI enforces this with `scripts/check_no_generated_artifacts.sh`.
This project uses [Conventional Commits](https://www.conventionalcommits.org/) with [python-semantic-release](https://python-semantic-release.readthedocs.io/) for automated versioning.
## Merge Checklist
- [ ] Changes are minimal, purposeful, and remove obsolete/dead code in touched areas.
- [ ] Backend tests pass (`uv run pytest tests/unit` and integration scope as needed).
- [ ] Frontend checks pass (`npm run typecheck`, `npm run test:run`, `npm run build`).
- [ ] API/behavior docs are updated when env vars, endpoints, or payloads change.
- [ ] `README.md`, `.env.example`, and deployment notes stay aligned.
- [ ] `scripts/check_no_generated_artifacts.sh` passes locally.
Commit message format:
```
<type>(<scope>): <description>
[optional body]
```
Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`.
Examples:
- `feat(scholars): add bulk import from CSV`
- `fix(ingestion): handle empty citation blocks`
- `docs: update configuration reference`
## Code Standards
### Function Length
Maximum 50 lines per function. Break complex logic into small, testable, single-responsibility functions.
### DRY
Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
### Negative Space Programming
Use explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
### Cyclomatic Complexity
Flatten logic. Use early returns and guard clauses instead of deep nesting. No magic numbers.
### Domain Service Boundaries
All business logic resides in `app/services/<domain>/`. Flat files in the `app/services/` root are strictly prohibited. Each domain owns its own application service, types, and helpers.
### API Envelope
All `/api/v1` responses use the strict envelope format:
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
### Data Isolation
- Scholar tracking is user-scoped.
- Publications are global, deduplicated records.
- Read/favorite/visibility state lives on scholar-publication link rows.
### Scrape Safety
Rate limits and cooldowns are immutable constraints. They prevent IP bans and must not be optimized away or set to zero.
## UI Standards
- Integrate Tailwind with the preset theming system (`frontend/src/theme/presets/`).
- Every UI element must have a clear purpose.
- Clarity through both styling and language.

26
docs/developer/documentation-standards.md
View file

@ -1,26 +0,0 @@
# Documentation Standards
This project keeps docs organized by audience and document type.
## Audience Split
- `user/`: onboarding and usage tasks for self-hosted users.
- `operations/`: runbooks and checklists for production operation.
- `developer/`: architecture, contribution workflow, and implementation guides.
- `reference/`: stable contracts (API, env variables, configuration behavior).
## Document Quality Rules
- One primary audience per page.
- Task pages must include prerequisites, exact commands, and verification steps.
- Reference pages should be contract-first and avoid procedural noise.
- Runbooks should include recovery steps and rollback/safety notes.
- Prefer concise pages with strong cross-links over long mixed-purpose pages.
## Required Top-Level Entrypoints
- `index.md`: user/developer/operator navigation hub.
- `user/getting-started.md`: first-run path.
- `developer/local-development.md`: contributor setup and validation path.
- `operations/overview.md`: operational playbook index.
- `reference/overview.md`: contract index.

166
docs/developer/frontend-theme-inventory.md
View file

@ -1,89 +1,105 @@
# Theme Inventory (Phase 0)
---
title: Frontend Theme Inventory
sidebar_position: 6
---
This file captures the semantic color system baseline for the theme refactor.
# Frontend Theme Inventory
## Token Domains
- `scale`:
- `brand` (`50..950`)
- `info` (`50..950`)
- `success` (`50..950`)
- `warning` (`50..950`)
- `danger` (`50..950`)
- `surface`:
- `app`
- `nav`
- `nav_active`
- `card`
- `card_muted`
- `table`
- `table_header`
- `input`
- `overlay`
- `text`:
- `primary`
- `secondary`
- `muted`
- `inverse`
- `link`
- `border`:
- `default`
- `strong`
- `subtle`
- `interactive`
- `focus`:
- `ring`
- `ring_offset`
- `action` variants (`primary`, `secondary`, `ghost`, `danger`):
- `bg`
- `border`
- `text`
- `hover_bg`
- `hover_border`
- `hover_text`
- `state` variants (`info`, `success`, `warning`, `danger`):
- `bg`
- `border`
- `text`
The theme system uses semantic tokens organized into domains:
## Theme Sources
### Scale Colors
Theme presets are dynamically loaded from `frontend/src/theme/presets/*.{json,js}`.
Color ramps from `50` to `950` for each intent:
Current preset files:
- `brand` - Primary brand color
- `info` - Informational elements
- `success` - Success states
- `warning` - Warning states
- `danger` - Error/destructive states
- `frontend/src/theme/presets/parchment.js`
- `frontend/src/theme/presets/lilac.js`
- `frontend/src/theme/presets/dune.js`
- `frontend/src/theme/presets/oatmeal.js`
- `frontend/src/theme/presets/scholarly.json`
- `frontend/src/theme/presets/graphite.json`
- `frontend/src/theme/presets/tide.json`
### Surface
Each preset contains both `light` and `dark` mode token definitions.
Background colors for major UI areas:
## Adoption Status (Phase 0-3 complete baseline)
| Token | Usage |
|-------|-------|
| `app` | Application background |
| `nav` | Navigation bar |
| `nav_active` | Active navigation item |
| `card` | Card backgrounds |
| `card_muted` | Muted card variant |
| `table` | Table body |
| `table_header` | Table header |
| `input` | Form inputs |
| `overlay` | Modal/dialog overlays |
Tokenized foundation components:
### Text
- `AppButton`
- `AppCard`
- `AppCheckbox`
- `AppEmptyState`
- `AppHelpHint`
- `AppInput`
- `AppSelect`
- `AppTable`
- `AppModal`
- `AppHeader`
- `AppNav`
- `AppAlert`
- `AppBadge`
- `RunStatusBadge`
- `QueueHealthBadge`
| Token | Usage |
|-------|-------|
| `primary` | Default text |
| `secondary` | Supporting text |
| `muted` | Disabled/placeholder text |
| `inverse` | Text on dark backgrounds |
| `link` | Hyperlinks |
Hardening in place:
### Border
- Frontend token policy check script: `frontend/scripts/check_theme_tokens.mjs`
- CI enforcement step in `frontend-quality` workflow
- Theme preset integrity tests in `frontend/src/theme/presets.test.ts`
| Token | Usage |
|-------|-------|
| `default` | Standard borders |
| `strong` | Emphasized borders |
| `subtle` | Light separators |
| `interactive` | Hover/focus borders |
### Focus
| Token | Usage |
|-------|-------|
| `ring` | Focus ring color |
| `ring_offset` | Focus ring offset color |
### Action Variants
Each action type (`primary`, `secondary`, `ghost`, `danger`) provides:
`bg`, `border`, `text`, `hover_bg`, `hover_border`, `hover_text`
### State Variants
Each state (`info`, `success`, `warning`, `danger`) provides:
`bg`, `border`, `text`
## Theme Presets
Presets are loaded from `frontend/src/theme/presets/*.{json,js}`. Each contains both `light` and `dark` mode token definitions.
Current presets:
| Preset | Format |
|--------|--------|
| `parchment` | JS |
| `lilac` | JS |
| `dune` | JS |
| `oatmeal` | JS |
| `scholarly` | JSON |
| `graphite` | JSON |
| `tide` | JSON |
## Tokenized Components
Foundation components using the token system:
- `AppButton`, `AppCard`, `AppCheckbox`, `AppEmptyState`
- `AppHelpHint`, `AppInput`, `AppSelect`, `AppTable`
- `AppModal`, `AppHeader`, `AppNav`, `AppAlert`
- `AppBadge`, `RunStatusBadge`, `QueueHealthBadge`
## Enforcement
- Token policy check: `frontend/scripts/check_theme_tokens.mjs`
- CI step in `frontend-quality` workflow
- Preset integrity tests: `frontend/src/theme/presets.test.ts`

109
docs/developer/ingestion.md
View file

@ -1,38 +1,89 @@
# Ingestion System & Backoff Strategies
---
title: Ingestion Pipeline
sidebar_position: 5
---
The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans.
# Ingestion Pipeline
## Ingestion Overview
The underlying ingestion process:
1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
3. Downloads the paginated HTML feed for a user across multiple page iterations.
4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans.
## Handling 429 Too Many Requests
Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`).
## Pipeline Overview
Scholarr treats these distinct from random network timeouts.
- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s).
- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead.
1. The scheduler (or a manual trigger) starts a **run** for one or more scholars.
2. The service connects via HTTPX with strict browser headers.
3. Paginated HTML feeds are downloaded for each scholar profile.
4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks.
5. Publications are fingerprinted and deduplicated against the global store.
6. External APIs resolve additional identifiers.
7. The PDF resolution pipeline runs asynchronously for publications with known DOIs.
## Publication Identifiers Loop
Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
- **API Fetching**: Queries secondary bibliographic platforms sequentially:
- `export.arxiv.org/api/query` (Queries by Title and Author strings).
- `crossref.restful` APIs (Queries by Title and Author strings).
## Rate Limiting & Backoff
These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.
### Network Errors
Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s).
### HTTP 429 (Rate Limited)
When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies:
- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3)
- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s)
This pauses the pipeline gracefully instead of failing the entire run.
### Safety Cooldowns
Threshold-based cooldowns halt all ingestion after repeated failures:
| Threshold | Variable | Default | Cooldown |
|-----------|----------|---------|----------|
| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) |
| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) |
## Continuation Queue
Multi-page ingestion uses a continuation queue to spread load over time:
- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`)
- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s)
- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s)
- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6)
Each continuation item is re-enqueued with exponential backoff.
## Identifier Resolution
After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers:
1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches.
2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings.
3. **Crossref API** - Queries Crossref REST API by title and author strings.
Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage.
## PDF Resolution
Publications with resolved DOIs enter the PDF resolution pipeline:
1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs.
2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates.
3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly.
Auto-retry is configured via `PDF_AUTO_RETRY_*` variables.
## arXiv Request Controls
- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
## arXiv Observability Events
- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown.
- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.
- **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`.
- **In-flight coalescing**: Duplicate concurrent misses join one outbound request.
- **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
### Observability Events
| Event | Description |
|-------|-------------|
| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. |
| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. |
| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. |
| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. |

100
docs/developer/local-development.md
View file

@ -1,52 +1,100 @@
# Developer Local Development
---
title: Local Development
sidebar_position: 3
---
## Start the Dev Stack
# Local Development
## Prerequisites
- Docker and Docker Compose v2+
- Python 3.12+ (for IDE support and local linting)
- Node.js 20+ (for frontend development)
## Starting the Dev Stack
The development compose file overlays the production config with hot-reload and a separate Vite dev server:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
```
Open:
- API: `http://localhost:8000`
- Frontend dev server: `http://localhost:5173`
This starts three services:
Stop:
| Service | Port | Description |
|---------|------|-------------|
| `db` | 5432 (internal) | PostgreSQL 15 |
| `app` | 8000 | FastAPI backend with `APP_RELOAD=1` |
| `frontend` | 5173 | Vite dev server proxying API calls to `app:8000` |
### Dev-Specific Overrides
The `docker-compose.dev.yml` file applies these changes:
- **`app`**: Uses `scholarr-dev:local` image built from the `dev` stage. Mounts the project root as `/app` for hot reload. Disables `SESSION_COOKIE_SECURE` and the built frontend.
- **`frontend`**: Node 20 container running `npm install && npm run dev`. Mounts `./frontend` with a named volume for `node_modules`. Uses polling for file watching (`CHOKIDAR_USEPOLLING=1`).
## Environment Setup
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml down
cp .env.example .env
```
## Backend Validation
Set at minimum:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration
POSTGRES_PASSWORD=localdev
SESSION_SECRET_KEY=local-dev-secret-at-least-32-characters
SESSION_COOKIE_SECURE=0
```
## Frontend Validation
## Running Tests
All tests run inside containers:
```bash
cd frontend
npm install
npm run typecheck
npm run test:run
npm run build
# Run unit tests (default: excludes integration markers)
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest
# Run integration tests
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m integration
# Run a specific test file
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest tests/unit/test_fingerprints.py -v
```
## Repository Gates
See [Testing](testing.md) for markers, fixtures, and conventions.
## Linting and Type Checking
```bash
python3 scripts/check_frontend_api_contract.py
python3 scripts/check_env_contract.py
./scripts/check_no_generated_artifacts.sh
# Ruff linting
ruff check .
# Ruff formatting check
ruff format --check .
# Mypy type checking
mypy app/
```
## Docs Site (Contributor Workflow)
Ruff config is in `pyproject.toml`: target Python 3.12, line length 120, rules `E F W I UP B SIM RUF`.
Docs tooling is colocated in `docs/website/`:
## Database Migrations
Alembic migrations run automatically on startup when `MIGRATE_ON_START=1`. To run manually:
```bash
cd docs/website
npm install
npm run build
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
alembic upgrade head
```
To create a new migration:
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
alembic revision --autogenerate -m "description of change"
```

72
docs/developer/overview.md
View file

@ -1,11 +1,65 @@
# Developer Documentation
---
title: Developer Overview
sidebar_position: 1
---
Use this section if you are contributing code or running full quality gates.
# Developer Overview
- [Documentation Standards](./documentation-standards.md)
- [Local Development](./local-development.md)
- [Architecture Boundaries](./architecture.md)
- [Ingestion API & Backoff Strategies](./ingestion.md)
- [API Contracts](./api-contract.md)
- [Contributing](./contributing.md)
- [Frontend Theme Inventory](./frontend-theme-inventory.md)
Scholarr is a Python 3.12+ FastAPI backend with an async SQLAlchemy ORM layer, PostgreSQL database, and a Vue 3 + TypeScript + Vite frontend.
## Stack
| Layer | Technology |
|-------|------------|
| Backend | Python 3.12+, FastAPI, SQLAlchemy 2.0 (async/asyncpg), Alembic |
| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS |
| Database | PostgreSQL 15 |
| Infrastructure | Multi-stage Docker, Docker Compose |
| Linting | ruff (E, F, W, I, UP, B, SIM, RUF), mypy |
| Testing | pytest, pytest-asyncio |
| Versioning | python-semantic-release, conventional commits |
## Quick Start
```bash
# Build the dev image and start services
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
# Backend: http://localhost:8000 (hot reload enabled)
# Frontend: http://localhost:5173 (Vite dev server with API proxy)
```
See [Local Development](local-development.md) for the full setup guide.
## Project Layout
```
app/
├── api/routers/ # FastAPI route handlers
├── auth/ # Session + CSRF middleware
├── db/ # SQLAlchemy models, session factory, migrations
├── services/ # Domain service modules (see Architecture)
│ ├── arxiv/ # arXiv API client, cache, rate limiting
│ ├── crossref/ # Crossref DOI lookups
│ ├── dbops/ # Database integrity + repair operations
│ ├── ingestion/ # Run orchestration, scheduler, safety gates
│ ├── openalex/ # OpenAlex metadata matching
│ ├── portability/ # Import/export workflows
│ ├── publication_identifiers/ # Multi-identifier resolution
│ ├── publications/ # Listing, enrichment, dedup, PDF queue
│ ├── runs/ # Run history, continuation queue
│ ├── scholar/ # HTML parser, source fetch adapters
│ ├── scholars/ # Scholar CRUD, image upload, name search
│ └── unpaywall/ # Unpaywall PDF discovery
├── main.py # App factory, lifespan, middleware stack
frontend/
├── src/
│ ├── components/ # Reusable Vue components
│ ├── theme/presets/ # Color theme presets (light + dark)
│ └── ...
scripts/db/ # Operational database scripts
tests/
├── unit/ # Fast, no-database tests
├── integration/ # Tests requiring database + services
└── fixtures/ # Shared test fixtures
```

102
docs/developer/testing.md Normal file
View file

@ -0,0 +1,102 @@
---
title: Testing
sidebar_position: 7
---
# Testing
## Running Tests
All tests must run inside containers:
```bash
# Unit tests (default: excludes integration markers)
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest
# Integration tests
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m integration
# Specific marker
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest -m db
# Verbose output for a specific file
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest tests/unit/test_fingerprints.py -v
```
## Test Configuration
From `pyproject.toml`:
```toml
[tool.pytest.ini_options]
addopts = "-q -m \"not integration\" --import-mode=importlib"
asyncio_mode = "auto"
testpaths = ["tests"]
```
- Default run excludes `integration` marked tests
- Uses `importlib` import mode to resolve module name collisions
- Async tests run automatically (no `@pytest.mark.asyncio` needed)
## Markers
| Marker | Description |
|--------|-------------|
| `integration` | Tests requiring external services (database, network) |
| `db` | Tests that validate database behavior and constraints |
| `migrations` | Tests focused on Alembic schema migration correctness |
| `schema` | Tests focused on multi-tenant schema invariants |
| `smoke` | Smoke tests for containerized runtime |
## Test Tiers
### Unit Tests (`tests/unit/`)
Fast, no-database tests. Mock external dependencies. These run by default.
Examples:
- `test_fingerprints.py` - Publication fingerprinting logic
- `test_scholar_parser.py` - HTML parsing without network calls
- `test_doi_normalize.py` - DOI normalization rules
- `test_ingestion_arxiv_rate_limit.py` - Rate limiter behavior
- `test_publication_pdf_resolution_pipeline.py` - PDF pipeline logic
Domain-specific unit tests are organized under `tests/unit/services/domains/`:
- `arxiv/` - Cache, client, gateway, guards, parser, rate limit tests
- `openalex/` - Client and matching tests
- `publications/` - Dedup tests
### Integration Tests (`tests/integration/`)
Require a running database. Test full request/response flows and data consistency.
Examples:
- `test_api_v1.py` - API endpoint integration tests
- `test_db_integrity.py` - Database integrity checks
- `test_run_lifecycle_consistency.py` - Run state machine transitions
- `test_deferred_enrichment.py` - Enrichment pipeline with real data
- `test_fixture_probe_runs.py` - Fixture-based run probes
### Smoke Tests
Marked with `@pytest.mark.smoke`. Validate the containerized runtime starts and serves basic requests.
## Fixtures
Test fixtures live in `tests/fixtures/`:
```
tests/fixtures/
└── scholar/
├── profile_ok_amIMrIEAAAAJ.html # Successful profile HTML
└── regression/
├── profile_P1RwlvoAAAAJ.html # Regression case
├── profile_LZ5D_p4AAAAJ.html # Regression case
└── profile_AAAAAAAAAAAA.html # Regression case
```
Scholar HTML fixtures are real Google Scholar profile pages used to test parser robustness against DOM structure changes.