temp commit

docs/developer/api-contract.md

@@ -0,0 +1,12 @@
+# API Contracts
+
+`Scholarr` is designed with strictly typed Pydantic V2 models, serialized across the network through standard OpenAPI `v3` specs via FastAPI.
+
+## DTO Models Structure
+FastAPI routes dynamically ingest request/response Data Transfer Objects.
+- **Example**: `PublicationListItem` no longer contains a hard-coded `.doi`. It contains a `.display_identifier` property resolving the highest confidence identifier regardless of backend origin.
+- **Testing**: Run `./scripts/check_frontend_api_contract.py` or equivalent integration steps mapped in CI to ensure that backend python routes strictly map to the TypeScript types compiled by the Frontend.
+
+## Frontend Contract Requirements
+The UI exclusively calls routes exposed by FastAPI's `APIRouter`. 
+All frontend integration tests enforce that Vue 3 Components do not assume hard properties directly outside the bounded TS schemas. This avoids type mismatch runtime explosions if the database is scaled horizontally.

docs/developer/architecture.md

@@ -20,6 +20,32 @@ Canonical business logic belongs in `app/services/domains/*`.
 
 ## Frontend Behavior Notes
 
-- Mobile primary nav is in a left drawer and closes on route change or logout.
-- Long list views use internal scroll containers.
-- Name search remains intentionally constrained due upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URL.
+- Navigation: Mobile primary nav is in a left drawer and closes on route change or logout.
+- Lists: Long list views use internal scroll containers to prevent viewport overflow.
+- Rate Limiting: Name search remains intentionally constrained due to upstream anti-bot behavior; production onboarding should prefer scholar ID/profile URLs directly if possible.
+
+## Data Integration & Acquisition Flow
+
+```mermaid
+graph TD
+    UI[Frontend Vue App] --> API[FastAPI Backend]
+    API --> Scheduler[Background Celery/Async Scheduler]
+    
+    Scheduler -->|1. Fetch HTML| Scholar[Google Scholar HTML Parser]
+    Scholar -->|2. Extract Metadata| IdentifierModule[Identifier Gathering]
+    
+    IdentifierModule -->|Search arXiv API| API1(arXiv)
+    IdentifierModule -->|Search Crossref API| API2(Crossref)
+    
+    IdentifierModule -->|3. Save Identifiers| DB[(PostgreSQL)]
+    
+    Scheduler -->|4. Resolve PDF| PDFPipeline[PDF Resolution Pipeline]
+    DB --> |Identified DOIs| PDFPipeline
+    PDFPipeline -->|Search Open Access APIs| Unpaywall(Unpaywall API)
+    
+    Unpaywall --> |Acquire PDF URL| PDFWorker[PDF Download Worker]
+    PDFWorker --> |Store PDF Metadata| DB 
+```
+
+### Identifier Engine Philosophy
+The platform previously treated the `DOI` as a hardcoded 1:1 property of a publication. It now utilizes a decoupled *Identifier Gathering* module (`PublicationIdentifier` table). A single publication can have multiple identifiers (ex. `doi`, `arxiv`, `pmid`, `pmcid`). This creates high resilience when integrating with external APIs, allowing systems like Unpaywall to be fed explicitly with high-confidence DOIs, rather than relying on unstructured search heuristics.

docs/developer/ingestion.md

@@ -0,0 +1,26 @@
+# Ingestion System & Backoff Strategies
+
+The `ScholarIngestionService` drives the primary data acquisition loop. Since Google Scholar utilizes heavy bot protection and rate limits, this package contains nuanced backoffs to protect user networks from automated IP bans.
+
+## Ingestion Overview
+The underlying ingestion process:
+1. Receives an explicit request or background cron trigger to resolve a `scholar_profile_id`.
+2. Connects asynchronously using configured HTTPX adapters with strict browser headers.
+3. Downloads the paginated HTML feed for a user across multiple page iterations.
+4. Uses `regex` and DOM-invariants (e.g. `gsc_vcd_cib`) to pull individual publication blocks.
+
+## Handling 429 Too Many Requests
+Google Scholar aggressively throws HTTP 429 responses if multiple concurrent tabs or rapidly sequential commands query the same IP address for specific API endpoints (like `citations?view_op=view_citation...`). 
+
+Scholarr treats these distinct from random network timeouts.
+- **Network Error Retries**: Handled via `ingestion_network_error_retries` with a base backoff of `ingestion_retry_backoff_seconds` (Default 1.0s).
+- **Rate Limit 429 Retries**: When `ParseState.BLOCKED_OR_CAPTCHA` captures `blocked_http_429_rate_limited`, the system applies a dedicated cooldown. It respects `ingestion_rate_limit_retries` multiplied by `ingestion_rate_limit_backoff_seconds` (Default 30.0s). This prevents the pipeline from fatalizing a user's job completely, pausing operations seamlessly instead.
+
+## Publication Identifiers Loop
+Once a publication is built, the `gather_identifiers_for_publication` module isolates keys explicitly.
+- **Local Parsing**: Searches for direct identifiers within the HTML parameters (DOI patterns, arXiv regexes).
+- **API Fetching**: Queries secondary bibliographic platforms sequentially:
+  - `export.arxiv.org/api/query` (Queries by Title and Author strings).
+  - `crossref.restful` APIs (Queries by Title and Author strings).
+
+These identifiers are accumulated in `publication_identifiers` instead of being bound as hard-coded properties, maximizing matching resilience in the automated Unpaywall PDF acquisition stage.

docs/developer/overview.md

@@ -5,5 +5,7 @@ Use this section if you are contributing code or running full quality gates.
 - [Documentation Standards](./documentation-standards.md)
 - [Local Development](./local-development.md)
 - [Architecture Boundaries](./architecture.md)
+- [Ingestion API & Backoff Strategies](./ingestion.md)
+- [API Contracts](./api-contract.md)
 - [Contributing](./contributing.md)
 - [Frontend Theme Inventory](./frontend-theme-inventory.md)