justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

docs: rebuild documentation from scratch with clean IA

Browse source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 09:39:27 +01:00
parent 6c31b331f7
commit 0c5b199b76
30 changed files with 1647 additions and 649 deletions
Download patch file Download diff file Expand all files Collapse all files

109
docs/developer/ingestion.md
View file

@ -1,38 +1,89 @@
# Ingestion System & Backoff Strategies
---
title: Ingestion Pipeline
sidebar_position: 5
---
The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans.
# Ingestion Pipeline
## Ingestion Overview
The underlying ingestion process:
1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
3. Downloads the paginated HTML feed for a user across multiple page iterations.
4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
The `ScholarIngestionService` drives the primary data acquisition loop. Google Scholar uses heavy bot protection, so the pipeline includes nuanced backoff strategies to protect user networks from IP bans.
## Handling 429 Too Many Requests
Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`).
## Pipeline Overview
Scholarr treats these distinct from random network timeouts.
- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s).
- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead.
1. The scheduler (or a manual trigger) starts a **run** for one or more scholars.
2. The service connects via HTTPX with strict browser headers.
3. Paginated HTML feeds are downloaded for each scholar profile.
4. A regex + DOM-invariant parser (`gsc_vcd_cib` selectors) extracts publication blocks.
5. Publications are fingerprinted and deduplicated against the global store.
6. External APIs resolve additional identifiers.
7. The PDF resolution pipeline runs asynchronously for publications with known DOIs.
## Publication Identifiers Loop
Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
- **API Fetching**: Queries secondary bibliographic platforms sequentially:
- `export.arxiv.org/api/query` (Queries by Title and Author strings).
- `crossref.restful` APIs (Queries by Title and Author strings).
## Rate Limiting & Backoff
These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.
### Network Errors
Handled via `INGESTION_NETWORK_ERROR_RETRIES` (default: 1) with base backoff of `INGESTION_RETRY_BACKOFF_SECONDS` (default: 1.0s).
### HTTP 429 (Rate Limited)
When the parser detects `BLOCKED_OR_CAPTCHA` with `blocked_http_429_rate_limited`, a dedicated cooldown applies:
- Retries: `INGESTION_RATE_LIMIT_RETRIES` (default: 3)
- Backoff per retry: `INGESTION_RATE_LIMIT_BACKOFF_SECONDS` (default: 30s)
This pauses the pipeline gracefully instead of failing the entire run.
### Safety Cooldowns
Threshold-based cooldowns halt all ingestion after repeated failures:
| Threshold | Variable | Default | Cooldown |
|-----------|----------|---------|----------|
| Blocked failures | `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD` | 1 | 1800s (30 min) |
| Network failures | `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD` | 2 | 900s (15 min) |
## Continuation Queue
Multi-page ingestion uses a continuation queue to spread load over time:
- `INGESTION_CONTINUATION_QUEUE_ENABLED` (default: `1`)
- Base delay: `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default: 120s)
- Max delay: `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default: 3600s)
- Max attempts: `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default: 6)
Each continuation item is re-enqueued with exponential backoff.
## Identifier Resolution
After publication extraction, the `gather_identifiers_for_publication` module resolves identifiers:
1. **Local parsing** - Searches HTML parameters for DOI patterns and arXiv regex matches.
2. **arXiv API** - Queries `export.arxiv.org/api/query` by title and author strings.
3. **Crossref API** - Queries Crossref REST API by title and author strings.
Identifiers are stored in the `publication_identifiers` table rather than as hardcoded properties, maximizing matching resilience for the PDF resolution stage.
## PDF Resolution
Publications with resolved DOIs enter the PDF resolution pipeline:
1. **Unpaywall** - Queries the Unpaywall API for open-access PDF URLs.
2. **PDF Discovery** - If Unpaywall returns an OA page URL without a direct PDF link, the service fetches the HTML and searches for PDF link candidates.
3. **arXiv Direct** - If an arXiv ID is known, the PDF URL is derived directly.
Auto-retry is configured via `PDF_AUTO_RETRY_*` variables.
## arXiv Request Controls
- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
## arXiv Observability Events
- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown.
- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.
- **Global throttle**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
- **Query cache**: Identical request parameters are fingerprinted and cached in `arxiv_query_cache_entries`.
- **In-flight coalescing**: Duplicate concurrent misses join one outbound request.
- **Load shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
### Observability Events
| Event | Description |
|-------|-------------|
| `arxiv.request_scheduled` | Emitted before a gated request. Includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`. |
| `arxiv.request_completed` | Emitted after response. Includes `status_code`, `wait_seconds`, `source_path`. |
| `arxiv.cooldown_activated` | Emitted when status `429` triggers cooldown. |
| `arxiv.cache_hit` / `arxiv.cache_miss` | Emitted on query cache lookup with `source_path`. |