Domain Boundaries

Data Model Rules

Scholar tracking is user-scoped.
Publications are global/deduplicated records.
Read/favorite/visibility state stays on scholar-publication link rows.

Service Boundaries

Canonical business logic belongs in app/services/domains/*.

app/services/domains/ingestion/*: run orchestration, continuation queue, scheduler, and scrape safety.
app/services/domains/scholar/*: fail-fast scholar parsing and source fetch adapters.
app/services/domains/scholars/*: scholar CRUD, profile image, and name-search controls.
app/services/domains/publications/*: listing/read-state, favorite toggles, enrichment scheduling, and retry paths.
app/services/domains/arxiv/*: typed API client, global DB-backed throttle, query cache, and in-flight request coalescing.
app/services/domains/crossref/* + app/services/domains/unpaywall/*: DOI/OA enrichment with bounded pacing.
app/services/domains/runs/*: run history and continuation queue operations.
app/services/domains/portability/*: import/export workflows.

Frontend Behavior Notes

Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
Lists: Long list views use internal scroll containers to prevent viewport overflow.
Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.

Data Integration & Acquisition Flow

graph TD
    UI[Frontend Vue App] --> API[FastAPI Backend]
    API --> Scheduler[Background Celery/Async Scheduler]
    
    Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
    Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
    
    IdentifierModule -->|Search arXiv API| API1(arXiv)
    IdentifierModule -->|Search Crossref API| API2(Crossref)
    
    IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)]
    
    Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline]
    DB --> |Identified DOIs| PDFPipeline
    PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API)
    
    Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker]
    PDFWorker --> |Store PDF Metadata| DB

Identifier Engine Philosophy

The platform previously treated the DOI as a hardcoded 1:1 property of a publication. It now utilizes a decoupled Identifier Gathering module (PublicationIdentifier table). A single publication can have multiple identifiers (ex. doi, arxiv, pmid, pmcid). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.

arXiv Safety and Efficiency

arXiv requests are globally serialized via a PostgreSQL advisory lock and shared runtime row (arxiv_runtime_state).
identical request payloads are fingerprinted and cached in arxiv_query_cache_entries with TTL + optional max-entry pruning.
concurrent identical misses are coalesced in-process, so one outbound call serves all waiters.
structured logs provide auditability for scheduling/completion/cooldown/cache behavior.

3.2 KiB Raw Blame History