Intermediate commit

docs/developer/architecture.md

@@ -14,6 +14,7 @@ Canonical business logic belongs in `app/services/domains/*`.
 - `app/services/domains/scholar/*`: fail-fast scholar parsing and source fetch adapters.
 - `app/services/domains/scholars/*`: scholar CRUD, profile image, and name-search controls.
 - `app/services/domains/publications/*`: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
+- `app/services/domains/arxiv/*`: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
 - `app/services/domains/crossref/*` + `app/services/domains/unpaywall/*`: DOI/OA enrichment with bounded pacing.
 - `app/services/domains/runs/*`: run history and continuation queue operations.
 - `app/services/domains/portability/*`: import/export workflows.
@@ -49,3 +50,9 @@ graph TD
 
 ### Identifier Engine Philosophy
 The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.
+
+### arXiv Safety and Efficiency
+- arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (`arxiv_runtime_state`).
+- identical request payloads are fingerprinted and cached in `arxiv_query_cache_entries` with TTL + optional max-entry pruning.
+- concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
+- structured logs provide auditability for scheduling/completion/cooldown/cache behavior.

docs/developer/ingestion.md

@@ -24,3 +24,15 @@ Once a publication is built, the `gather_identifiers_for_publication` module iso
   - `crossref.restful` APIs (Queries by Title and Author strings).
 
 These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.
+
+## arXiv Request Controls
+- **Global throttle state**: arXiv calls share `arxiv_runtime_state` so all workers respect one cooldown/interval clock.
+- **Query cache**: identical request parameters map to a stable fingerprint and are stored in `arxiv_query_cache_entries`.
+- **In-flight coalescing**: duplicate concurrent misses join one outbound request instead of fan-out.
+- **Caller load-shedding**: arXiv lookups are skipped when high-confidence DOI/arXiv evidence already exists, or when title quality is below threshold.
+
+## arXiv Observability Events
+- `arxiv.request_scheduled`: emitted before a gated request; includes `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
+- `arxiv.request_completed`: emitted after response; includes `status_code`, `wait_seconds`, `cooldown_remaining_seconds`, `source_path`.
+- `arxiv.cooldown_activated`: emitted when status `429` triggers cooldown.
+- `arxiv.cache_hit` / `arxiv.cache_miss`: emitted on query cache lookup with `source_path`.

docs/operations/arxiv-runbook.md

@@ -0,0 +1,46 @@
+# arXiv Operations Runbook
+
+Use this runbook when arXiv lookups are slow, rate-limited, or behaving unexpectedly.
+
+## Signals To Check
+- logs for `arxiv.request_scheduled`, `arxiv.request_completed`, `arxiv.cooldown_activated`, `arxiv.cache_hit`, `arxiv.cache_miss`
+- `arxiv_runtime_state` row for cooldown and next-allowed timestamps
+- `arxiv_query_cache_entries` size and expiry churn
+
+## Event Field Guide
+- `wait_seconds`: enforced pre-request delay from the global limiter.
+- `status_code`: upstream response code from arXiv.
+- `cooldown_remaining_seconds`: remaining cooldown when blocked or after 429.
+- `source_path`: caller path (`search` or `lookup_ids`).
+
+## Quick SQL Checks
+```sql
+SELECT state_key, next_allowed_at, cooldown_until, updated_at
+FROM arxiv_runtime_state;
+```
+
+```sql
+SELECT count(*) AS cache_rows, min(expires_at) AS earliest_expiry, max(expires_at) AS latest_expiry
+FROM arxiv_query_cache_entries;
+```
+
+## Common Scenarios
+1. Repeated `arxiv.cooldown_activated` events:
+- Confirm recent `429` statuses in `arxiv.request_completed`.
+- Reduce caller pressure (check new title-quality/identifier guards are active).
+- Temporarily raise `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS` if upstream remains strict.
+
+2. High request latency with few completions:
+- Inspect `wait_seconds` in `arxiv.request_scheduled`.
+- Verify only one process path is repeatedly hitting arXiv (`source_path`).
+- Confirm cache is enabled (`ARXIV_CACHE_TTL_SECONDS > 0`) and effective (`cache_hit` appears).
+
+3. Low cache effectiveness:
+- Validate normalized query behavior and caller churn.
+- Increase `ARXIV_CACHE_TTL_SECONDS` for stable workloads.
+- Increase `ARXIV_CACHE_MAX_ENTRIES` if heavy eviction is observed.
+
+## Safe Recovery
+1. Pause automated ingestion if rate-limit storms persist.
+2. Let cooldown expire naturally; avoid manual burst retries.
+3. Resume and monitor event rates before restoring full load.

docs/operations/overview.md

@@ -3,5 +3,6 @@
 Use this section for production operations and incident/runbook workflows.
 
 - [Scrape Safety Runbook](./scrape-safety-runbook.md)
+- [arXiv Runbook](./arxiv-runbook.md)
 - [Database Runbook](./database-runbook.md)
 - [Migration Rollout Checklist](./migration-checklist.md)

docs/reference/environment.md

@@ -40,7 +40,7 @@ This document is the source-of-truth audit for what remains configurable versus
 
 ### Scheduler + Ingestion Safety
 
-`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
+`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `SCHEDULER_PDF_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_RATE_LIMIT_RETRIES`, `INGESTION_RATE_LIMIT_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
 
 ### Scholar Images + Name Search Safety
 
@@ -48,7 +48,7 @@ This document is the source-of-truth audit for what remains configurable versus
 
 ### OA Enrichment + PDF Resolution
 
-`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`
+`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `ARXIV_ENABLED`, `ARXIV_TIMEOUT_SECONDS`, `ARXIV_MIN_INTERVAL_SECONDS`, `ARXIV_RATE_LIMIT_COOLDOWN_SECONDS`, `ARXIV_DEFAULT_MAX_RESULTS`, `ARXIV_CACHE_TTL_SECONDS`, `ARXIV_CACHE_MAX_ENTRIES`, `ARXIV_MAILTO`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`, `OPENALEX_API_KEY`, `CROSSREF_API_TOKEN`, `CROSSREF_API_MAILTO`
 
 ### Startup Bootstrap + DB Wait