justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

temp commit

Browse source
This commit is contained in:
Justin Visser 2026-02-26 12:54:19 +01:00
parent 8760f27b51
commit 0e9e49df16
193 changed files with 23228 additions and 935 deletions
Download patch file Download diff file Expand all files Collapse all files

32
docs/developer/architecture.md
View file

@ -20,6 +20,32 @@ Canonical business logic belongs in `app/services/domains/*`.
## Frontend Behavior Notes
- Mobile primary nav is in a left drawer and closes on route change or logout.
- Long list views use internal scroll containers.
- Name search remains intentionally constrained due upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URL.
- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
- Lists: Long list views use internal scroll containers to prevent viewport overflow.
- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.
## Data Integration & Acquisition Flow
```mermaid
graph TD
UI[Frontend Vue App] --> API[FastAPI Backend]
API --> Scheduler[Background Celery/Async Scheduler]
Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
IdentifierModule -->|Search arXiv API| API1(arXiv)
IdentifierModule -->|Search Crossref API| API2(Crossref)
IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)]
Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline]
DB --> |Identified DOIs| PDFPipeline
PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API)
Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker]
PDFWorker --> |Store PDF Metadata| DB
```
### Identifier Engine Philosophy
The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.