temp commit

README.md

@@ -2,7 +2,7 @@
 
 <div align="center">
 
-Self-hosted scholar tracking with a single app image (API + frontend).
+Self-hosted scholar tracking with a single app image (API + frontend). It automates publication discovery, parses Google Scholar, gathers external metadata via Crossref & arXiv APIs, and autonomously downloads Open Access PDFs.
 
 [![CI](https://img.shields.io/github/actions/workflow/status/justinzeus/scholarr/ci.yml?style=for-the-badge)](https://github.com/JustinZeus/scholarr/actions/workflows/ci.yml)
 [![Docker Pulls](https://img.shields.io/docker/pulls/justinzeus/scholarr?style=for-the-badge&logo=docker)](https://hub.docker.com/r/justinzeus/scholarr)
@@ -10,6 +10,25 @@ Self-hosted scholar tracking with a single app image (API + frontend).
 
 </div>
 
+## Key Features
+- **Automated Ingestion**: Background tasks regularly scrape author profiles to pull the latest publications.
+- **Resilient Parsing**: Custom DOM parsing logic falls back heavily across changes, backed by a dynamic `429 Too Many Requests` backoff cycle.
+- **Identifier Subsystem**: Isolates DOIs, PMIDs, and arXiv IDs independently of PDFs mapping fallback external APIs sequentially.
+- **Unpaywall PDF Search**: Leverages a fully automated resolution queue to grab public PDFs via the Unpaywall API on identified entries.
+- **Beautiful UI**: Polished Vue 3 frontend that ships natively inside the same Docker container as the FastAPI backend.
+
+## Architecture Data Flow
+```mermaid
+graph LR
+    HTML(Google Scholar) --> |Scraper| Ingestion
+    Ingestion --> |Extract DOI| DB[(Database)]
+    Ingestion -.-> |Fallback API Search| Crossref/arXiv
+    Crossref/arXiv --> DB
+    
+    DB --> |Identified DOIs| PDFQueue(Unpaywall/PDF Workers)
+    PDFQueue --> |Download & Store File| Storage
+```
+
 ## Quick Start
 
 1. Copy env template:
@@ -36,10 +55,12 @@ Open:
 ## Documentation
 
 Complete documentation is published at:
-
 - https://justinzeus.github.io/scholarr/
 
-Source markdown and docs tooling live under `docs/`.
+You can also read the developer/architecture breakdowns directly within the repository under:
+- `docs/developer/architecture.md`
+- `docs/developer/ingestion.md`
+- `docs/developer/api-contract.md`
 
 ## Quality Gates
 
@@ -53,11 +74,14 @@ docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv r
 Frontend:
 
 ```bash
-cd frontend
+# ALL tests/builds MUST be run within the docker container. Do not run raw npm commands on the host.
-npm install
+# For automated agents checking outputs, redirect stdout/stderr to files (e.g. > /tmp/npm_typecheck.log 2>&1) 
-npm run typecheck
+# The temp files /tmp/npm_test.log, /tmp/npm_lint.log, /tmp/npm_typecheck.log, and /tmp/pytest_eval.log are pre-authorized for overwrite.
-npm run test:run
+
-npm run build
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm install
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run typecheck
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run test:run
+docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run build
 ```
 
 Contract and env checks:
@@ -67,3 +91,4 @@ python3 scripts/check_frontend_api_contract.py
 python3 scripts/check_env_contract.py
 ./scripts/check_no_generated_artifacts.sh
 ```
+thanks arxiv

agents.md

@@ -3,35 +3,45 @@
 Adhere strictly to these constraints.
 
 ## 1. Coding Standards (Strict Enforcement)
-* **Function Length:** Maximum 50 lines of code per function. Break down complex logic into small, testable, single-responsibility functions.
+
-* **DRY (Don't Repeat Yourself):** Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
+- **Function Length:** Maximum 50 lines of code per function. Break down complex logic into small, testable, single-responsibility functions.
-* **Negative Space Programming:** Utilize explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
+- **DRY (Don't Repeat Yourself):** Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
-* **Cyclomatic Complexity:** Flatten logic. Use early returns and guard clauses instead of deep nesting.
+- **Negative Space Programming:** Utilize explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
+- **Cyclomatic Complexity:** Flatten logic. Use early returns and guard clauses instead of deep nesting.
   no magic numbers
 
 ## 2. Domain Architecture & Data Model
-* **Data Isolation:** Scholar tracking is **user-scoped**. Validate mapping/join tables; never assume global links between users and Scholar IDs.
+
-* **Data Deduplication:** Publications are **global records**. Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
+- **Data Isolation:** Scholar tracking is **user-scoped**. Validate mapping/join tables; never assume global links between users and Scholar IDs.
-* **State Management:** Visibility and "read/unread" states exist exclusively on the scholar-publication link table, not the global publication table.
+- **Data Deduplication:** Publications are **global records**. Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
-* **API Contract:** Exact envelope format required:
+- **State Management:** Visibility and "read/unread" states exist exclusively on the scholar-publication link table, not the global publication table.
-    * Success: `{"data": ..., "meta": {"request_id": "..."}}`
+- **API Contract:** Exact envelope format required:
-    * Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
+  - Success: `{"data": ..., "meta": {"request_id": "..."}}`
+  - Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
 
 ## 3. Scrape Safety & Rate Limiting (Immutable)
+
 These limits prevent IP bans and are not to be optimized away.
-* **Minimum Delay:** Enforce `INGESTION_MIN_REQUEST_DELAY_SECONDS` (default 2s) between all external requests.
+
-* **Anti-Detection:** Default to direct ID or profile URL ingestion. Name searches trigger CAPTCHAs.
+- **Minimum Delay:** Enforce `INGESTION_MIN_REQUEST_DELAY_SECONDS` (default 2s) between all external requests.
-* **Cooldowns:** Respect `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` (1800s) and `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` (900s) upon threshold breaches.
+- **Anti-Detection:** Default to direct ID or profile URL ingestion. Name searches trigger CAPTCHAs.
+- **Cooldowns:** Respect `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` (1800s) and `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` (900s) upon threshold breaches.
 
 ## 4. Current Environment & Stack
-* **Backend:** Python 3.12+, FastAPI, SQLAlchemy (Async/asyncpg), Alembic.
+
-* **Frontend:** TypeScript, Vue 3, Vite.
+- **Backend:** Python 3.12+, FastAPI, SQLAlchemy (Async/asyncpg), Alembic.
-* **Infrastructure:** Multi-stage Docker.
+- **Frontend:** TypeScript, Vue 3, Vite.
+- **Infrastructure:** Multi-stage Docker.
 
 ## 5. Domain Service Boundaries
-* **Strict Modularity:** Flat files in the `app/services/` root are strictly prohibited. All business logic and routing must reside exclusively within `app/services/domains/`.
 
+- **Strict Modularity:** Flat files in the `app/services/` root are strictly prohibited. All business logic and routing must reside exclusively within `app/services/domains/`.
 
 ## 6. UI rules
+
 Make sure to properly integrate tailwind in combination with the preset theming
 Clarity through both styling and language are a priority. all UI elements need to have a proper reason for existing.
+
+## 7. Testing
+
+All tests need to be ran using containers. `docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app`

alembic/versions/20260224_0018_add_openalex_enriched_to_publications.py

@@ -0,0 +1,30 @@
+"""Add openalex_enriched to publications
+
+Revision ID: 20260224_0018
+Revises: fa0c5e3262d5
+Create Date: 2026-02-24 19:10:00.000000
+
+"""
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '20260224_0018'
+down_revision: str | Sequence[str] | None = 'fa0c5e3262d5'
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column('publications', sa.Column('openalex_enriched', sa.Boolean(), server_default=sa.text('false'), nullable=False))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('publications', 'openalex_enriched')
+    # ### end Alembic commands ###

alembic/versions/20260224_0019_add_resolving_to_runstatus.py

@@ -0,0 +1,31 @@
+"""Add resolving to RunStatus enum
+
+Revision ID: 20260224_0019
+Revises: 20260224_0018
+Create Date: 2026-02-24 22:20:00.000000
+
+"""
+from collections.abc import Sequence
+
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision: str = '20260224_0019'
+down_revision: str | Sequence[str] | None = '20260224_0018'
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    # Use COMMIT to break out of the transaction block that Alembic creates in env.py.
+    # Postgres restricts ALTER TYPE ... ADD VALUE inside transactions when the type is in use.
+    # This is the robust, non-patchwork way to ensure the schema evolves correctly.
+    op.execute("COMMIT")
+    op.execute("ALTER TYPE run_status ADD VALUE IF NOT EXISTS 'resolving' AFTER 'running'")
+
+
+def downgrade() -> None:
+    # Enum values cannot be easily removed in Postgres without recreating the type,
+    # which is risky and usually avoided in simple migrations.
+    pass

alembic/versions/20260224_0020_add_openalex_last_attempt_at.py

@@ -0,0 +1,26 @@
+"""Add openalex_last_attempt_at to publications
+
+Revision ID: 20260224_0020
+Revises: 20260224_0019
+Create Date: 2026-02-24 22:50:00.000000
+
+"""
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '20260224_0020'
+down_revision: str | Sequence[str] | None = '20260224_0019'
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.add_column('publications', sa.Column('openalex_last_attempt_at', sa.DateTime(timezone=True), nullable=True))
+
+
+def downgrade() -> None:
+    op.drop_column('publications', 'openalex_last_attempt_at')

alembic/versions/20260225_0021_enforce_single_active_run_per_user.py

@@ -0,0 +1,70 @@
+"""Enforce one active crawl run per user.
+
+Revision ID: 20260225_0021
+Revises: 20260224_0020
+Create Date: 2026-02-25 09:10:00.000000
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "20260225_0021"
+down_revision: str | Sequence[str] | None = "20260224_0020"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+INDEX_NAME = "uq_crawl_runs_user_active"
+ACTIVE_STATUSES = ("running", "resolving")
+
+
+def upgrade() -> None:
+    bind = op.get_bind()
+    inspector = sa.inspect(bind)
+    indexes = {index["name"] for index in inspector.get_indexes("crawl_runs")}
+
+    op.execute("ALTER TYPE run_status ADD VALUE IF NOT EXISTS 'canceled'")
+
+    op.execute(
+        """
+        WITH ranked AS (
+            SELECT
+                id,
+                ROW_NUMBER() OVER (
+                    PARTITION BY user_id
+                    ORDER BY start_dt DESC, id DESC
+                ) AS rn
+            FROM crawl_runs
+            WHERE status IN ('running', 'resolving')
+        )
+        UPDATE crawl_runs AS runs
+        SET
+            status = 'failed',
+            end_dt = COALESCE(runs.end_dt, NOW())
+        FROM ranked
+        WHERE runs.id = ranked.id
+          AND ranked.rn > 1
+        """
+    )
+
+    if INDEX_NAME not in indexes:
+        op.create_index(
+            INDEX_NAME,
+            "crawl_runs",
+            ["user_id"],
+            unique=True,
+            postgresql_where=sa.text(
+                "status IN ('running'::run_status, 'resolving'::run_status)"
+            ),
+        )
+
+
+def downgrade() -> None:
+    bind = op.get_bind()
+    inspector = sa.inspect(bind)
+    indexes = {index["name"] for index in inspector.get_indexes("crawl_runs")}
+    if INDEX_NAME in indexes:
+        op.drop_index(INDEX_NAME, table_name="crawl_runs")

alembic/versions/20260225_0022_add_canonical_title_hash_to_publications.py

@@ -0,0 +1,35 @@
+"""Add canonical_title_hash to publications for cross-scholar dedup.
+
+Revision ID: 20260225_0022
+Revises: 20260225_0021
+Create Date: 2026-02-25 10:00:00.000000
+"""
+
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = "20260225_0022"
+down_revision: str | Sequence[str] | None = "20260225_0021"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "publications",
+        sa.Column("canonical_title_hash", sa.String(64), nullable=True),
+    )
+    op.create_index(
+        "ix_publications_canonical_title_hash",
+        "publications",
+        ["canonical_title_hash"],
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_publications_canonical_title_hash", table_name="publications")
+    op.drop_column("publications", "canonical_title_hash")

alembic/versions/44f7e10ef777_remove_doi_from_publications_and_.py

@@ -0,0 +1,27 @@
+"""remove_doi_from_publications_and_migrate_identifiers
+
+Revision ID: 44f7e10ef777
+Revises: 20260222_0016
+Create Date: 2026-02-22 17:03:56.261936
+
+"""
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = '44f7e10ef777'
+down_revision: str | Sequence[str] | None = '20260222_0016'
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    pass
+
+
+def downgrade() -> None:
+    pass
+

alembic/versions/fa0c5e3262d5_add_api_integration_keys_to_usersetting.py

@@ -0,0 +1,39 @@
+"""Add API integration keys to UserSetting
+
+Revision ID: fa0c5e3262d5
+Revises: 44f7e10ef777
+Create Date: 2026-02-23 14:25:45.021512
+
+"""
+from collections.abc import Sequence
+
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision: str = 'fa0c5e3262d5'
+down_revision: str | Sequence[str] | None = '44f7e10ef777'
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_index(op.f('ix_publications_doi'), table_name='publications')
+    op.drop_column('publications', 'doi')
+    op.add_column('user_settings', sa.Column('openalex_api_key', sa.String(length=255), nullable=True))
+    op.add_column('user_settings', sa.Column('crossref_api_token', sa.String(length=255), nullable=True))
+    op.add_column('user_settings', sa.Column('crossref_api_mailto', sa.String(length=255), nullable=True))
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column('user_settings', 'crossref_api_mailto')
+    op.drop_column('user_settings', 'crossref_api_token')
+    op.drop_column('user_settings', 'openalex_api_key')
+    op.add_column('publications', sa.Column('doi', sa.VARCHAR(length=255), autoincrement=False, nullable=True))
+    op.create_index(op.f('ix_publications_doi'), 'publications', ['doi'], unique=False)
+    # ### end Alembic commands ###
+

app/api/routers/admin_dbops.py

@@ -57,7 +57,6 @@ def _serialize_pdf_queue_item(item) -> dict[str, object]:
     return {
         "publication_id": item.publication_id,
         "title": item.title,
-        "doi": item.doi,
         "display_identifier": _serialize_display_identifier(item.display_identifier),
         "pdf_url": item.pdf_url,
         "status": item.status,
@@ -348,3 +347,64 @@ async def trigger_publication_link_repair(
         },
     )
     return success_payload(request, data=result)
+
+
+DROP_PUBLICATIONS_CONFIRMATION = "DROP ALL PUBLICATIONS"
+
+
+@router.post("/drop-all-publications")
+async def drop_all_publications(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    body = await request.json()
+    confirmation_text = (body.get("confirmation_text") or "").strip()
+    if confirmation_text != DROP_PUBLICATIONS_CONFIRMATION:
+        raise ApiException(
+            status_code=400,
+            code="confirmation_required",
+            message=f"Type '{DROP_PUBLICATIONS_CONFIRMATION}' to confirm this destructive action.",
+        )
+
+    from sqlalchemy import delete, func, select, update
+
+    from app.db.models import (
+        Publication,
+        PublicationIdentifier,
+        PublicationPdfJob,
+        PublicationPdfJobEvent,
+        ScholarProfile,
+        ScholarPublication,
+    )
+
+    count_result = await db_session.execute(select(func.count()).select_from(Publication))
+    total_publications = count_result.scalar_one()
+
+    await db_session.execute(delete(ScholarPublication))
+    await db_session.execute(delete(PublicationIdentifier))
+    await db_session.execute(delete(PublicationPdfJobEvent))
+    await db_session.execute(delete(PublicationPdfJob))
+    await db_session.execute(delete(Publication))
+    await db_session.execute(
+        update(ScholarProfile).values(baseline_completed=False)
+    )
+    await db_session.commit()
+
+    logger.warning(
+        "api.admin.db.all_publications_dropped",
+        extra={
+            "event": "api.admin.db.all_publications_dropped",
+            "admin_user_id": int(admin_user.id),
+            "admin_email": admin_user.email,
+            "deleted_count": int(total_publications),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "deleted_count": int(total_publications),
+            "message": f"Dropped {total_publications} publication(s) and all related data. "
+            "Scholar baselines have been reset; the next run will re-discover all publications.",
+        },
+    )

app/api/routers/publications.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+from datetime import datetime, timezone
 import logging
 from typing import Literal
 
@@ -62,7 +63,6 @@ def _serialize_publication_item(item) -> dict[str, object]:
         "citation_count": item.citation_count,
         "venue_text": item.venue_text,
         "pub_url": item.pub_url,
-        "doi": item.doi,
         "display_identifier": _serialize_display_identifier(item.display_identifier),
         "pdf_url": item.pdf_url,
         "pdf_status": item.pdf_status,
@@ -94,23 +94,27 @@ async def _publication_counts(
     user_id: int,
     selected_scholar_id: int | None,
     favorite_only: bool,
+    snapshot_before: datetime | None,
 ) -> tuple[int, int, int, int]:
     unread_count = await publication_service.count_unread_for_user(
         db_session,
         user_id=user_id,
         scholar_profile_id=selected_scholar_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
     favorites_count = await publication_service.count_favorite_for_user(
         db_session,
         user_id=user_id,
         scholar_profile_id=selected_scholar_id,
+        snapshot_before=snapshot_before,
     )
     latest_count = await publication_service.count_latest_for_user(
         db_session,
         user_id=user_id,
         scholar_profile_id=selected_scholar_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
     total_count = await publication_service.count_for_user(
         db_session,
@@ -118,6 +122,7 @@ async def _publication_counts(
         mode=publication_service.MODE_ALL,
         scholar_profile_id=selected_scholar_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
     return unread_count, favorites_count, latest_count, total_count
 
@@ -129,8 +134,12 @@ async def _list_publications_for_request(
     mode: Literal["all", "unread", "latest", "new"] | None,
     favorite_only: bool,
     scholar_profile_id: int | None,
+    search: str | None,
+    sort_by: str,
+    sort_dir: str,
     limit: int,
     offset: int,
+    snapshot_before: datetime | None,
 ) -> tuple[str, int | None, list]:
     resolved_mode = publication_service.resolve_publication_view_mode(mode)
     selected_scholar_id = scholar_profile_id
@@ -145,8 +154,12 @@ async def _list_publications_for_request(
         mode=resolved_mode,
         scholar_profile_id=selected_scholar_id,
         favorite_only=favorite_only,
+        search=search,
+        sort_by=sort_by,
+        sort_dir=sort_dir,
         limit=limit,
         offset=offset,
+        snapshot_before=snapshot_before,
     )
     await publication_service.schedule_missing_pdf_enrichment_for_user(
         db_session,
@@ -162,6 +175,27 @@ async def _list_publications_for_request(
     return resolved_mode, selected_scholar_id, hydrated
 
 
+def _resolve_publications_snapshot(
+    *,
+    snapshot: str | None,
+) -> tuple[datetime, str]:
+    if snapshot is None:
+        now_utc = datetime.now(timezone.utc)
+        return now_utc, now_utc.isoformat()
+    try:
+        parsed = datetime.fromisoformat(snapshot)
+    except ValueError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_snapshot",
+            message="Invalid publications snapshot cursor.",
+        ) from exc
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    normalized = parsed.astimezone(timezone.utc)
+    return normalized, normalized.isoformat()
+
+
 def _resolve_publications_paging(
     *,
     page: int,
@@ -188,6 +222,7 @@ def _publications_list_data(
     page: int,
     page_size: int,
     offset: int,
+    snapshot: str,
 ) -> dict[str, object]:
     return {
         "mode": mode,
@@ -200,6 +235,7 @@ def _publications_list_data(
         "total_count": total_count,
         "page": int(page),
         "page_size": int(page_size),
+        "snapshot": snapshot,
         "has_prev": int(offset) > 0,
         "has_next": int(offset) + int(page_size) < int(total_count),
         "publications": [_serialize_publication_item(item) for item in publications],
@@ -322,7 +358,6 @@ def _log_retry_pdf_result(
             "queued": queued,
             "resolved_pdf": resolved_pdf,
             "pdf_status": pdf_status,
-            "has_doi": bool(doi),
         },
     )
 
@@ -336,10 +371,14 @@ async def list_publications(
     mode: Literal["all", "unread", "latest", "new"] | None = Query(default=None),
     favorite_only: bool = Query(default=False),
     scholar_profile_id: int | None = Query(default=None, ge=1),
+    search: str | None = Query(default=None, min_length=1, max_length=200),
+    sort_by: Literal["first_seen", "title", "year", "citations", "scholar"] = Query(default="first_seen"),
+    sort_dir: Literal["asc", "desc"] = Query(default="desc"),
     page: int = Query(default=1, ge=1),
     page_size: int = Query(default=100, ge=1, le=500),
     limit: int | None = Query(default=None, ge=1, le=1000),
     offset: int | None = Query(default=None, ge=0),
+    snapshot: str | None = Query(default=None, min_length=1, max_length=64),
     db_session: AsyncSession = Depends(get_db_session),
     current_user: User = Depends(get_api_current_user),
 ):
@@ -349,20 +388,27 @@ async def list_publications(
         limit=limit,
         offset=offset,
     )
+    snapshot_before, snapshot_cursor = _resolve_publications_snapshot(snapshot=snapshot)
+    normalized_search = (search or "").strip() or None
     resolved_mode, selected_scholar_id, publications = await _list_publications_for_request(
         db_session,
         current_user=current_user,
         mode=mode,
         favorite_only=favorite_only,
         scholar_profile_id=scholar_profile_id,
+        search=normalized_search,
+        sort_by=sort_by,
+        sort_dir=sort_dir,
         limit=resolved_limit,
         offset=resolved_offset,
+        snapshot_before=snapshot_before,
     )
     unread_count, favorites_count, latest_count, total_count = await _publication_counts(
         db_session,
         user_id=current_user.id,
         selected_scholar_id=selected_scholar_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
     data = _publications_list_data(
         mode=resolved_mode,
@@ -376,6 +422,7 @@ async def list_publications(
         page=resolved_page,
         page_size=resolved_limit,
         offset=resolved_offset,
+        snapshot=snapshot_cursor,
     )
     return success_payload(request, data=data)
 
@@ -480,7 +527,7 @@ async def retry_publication_pdf(
         queued=queued,
         resolved_pdf=resolved_pdf,
         pdf_status=current.pdf_status,
-        doi=current.doi,
+        doi=None,
     )
     return success_payload(
         request,

app/api/routers/runs.py

@@ -11,6 +11,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from app.api.deps import get_api_current_user
 from app.api.errors import ApiException
 from app.api.responses import success_payload
+from fastapi.responses import StreamingResponse
 from app.api.schemas import (
     ManualRunEnvelope,
     QueueClearEnvelope,
@@ -24,6 +25,7 @@ from app.db.session import get_db_session
 from app.services.domains.ingestion import application as ingestion_service
 from app.services.domains.ingestion import safety as run_safety_service
 from app.services.domains.runs import application as run_service
+from app.services.domains.runs.events import event_generator
 from app.services.domains.settings import application as user_settings_service
 from app.settings import settings
 from app.api.runtime_deps import get_ingestion_service
@@ -31,6 +33,7 @@ from app.api.runtime_deps import get_ingestion_service
 logger = logging.getLogger(__name__)
 
 router = APIRouter(prefix="/runs", tags=["api-runs"])
+ACTIVE_RUN_STATUSES = {RunStatus.RUNNING, RunStatus.RESOLVING}
 
 IDEMPOTENCY_HEADER = "Idempotency-Key"
 IDEMPOTENCY_MAX_LENGTH = 128
@@ -338,7 +341,7 @@ async def _reused_manual_run_payload(
     )
     if previous_run is None:
         return None
-    if previous_run.status == RunStatus.RUNNING:
+    if previous_run.status in (RunStatus.RUNNING, RunStatus.RESOLVING):
         raise ApiException(
             status_code=409,
             code="run_in_progress",
@@ -410,7 +413,7 @@ async def _recover_integrity_error(
             extra={"event": "api.runs.manual_integrity_error", "user_id": user_id},
         )
         raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
-    if existing_run.status == RunStatus.RUNNING:
+    if existing_run.status in (RunStatus.RUNNING, RunStatus.RESOLVING):
         raise ApiException(
             status_code=409,
             code="run_in_progress",
@@ -586,6 +589,61 @@ async def get_run(
     )
 
 
+@router.post(
+    "/{run_id}/cancel",
+    response_model=RunDetailEnvelope,
+)
+async def cancel_run(
+    run_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    run = await run_service.get_run_for_user(
+        db_session,
+        user_id=current_user.id,
+        run_id=run_id,
+    )
+    if run is None:
+        raise ApiException(
+            status_code=404,
+            code="run_not_found",
+            message="Run not found.",
+        )
+
+    if run.status in ACTIVE_RUN_STATUSES:
+        run.status = RunStatus.CANCELED
+        await db_session.commit()
+        await db_session.refresh(run)
+    else:
+        raise ApiException(
+            status_code=409,
+            code="run_not_cancelable",
+            message="Run is already terminal and cannot be canceled.",
+            details={"run_id": int(run.id), "status": run.status.value},
+        )
+
+    error_log = run.error_log if isinstance(run.error_log, dict) else {}
+    scholar_results = error_log.get("scholar_results")
+    if not isinstance(scholar_results, list):
+        scholar_results = []
+        
+    safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    
+    return success_payload(
+        request,
+        data={
+            "run": _serialize_run(run),
+            "summary": run_service.extract_run_summary(error_log),
+            "scholar_results": [_normalize_scholar_result(item) for item in scholar_results],
+            "safety_state": safety_state,
+        },
+    )
+
+
 @router.post(
     "/manual",
     response_model=ManualRunEnvelope,
@@ -596,10 +654,7 @@ async def run_manual(
     current_user: User = Depends(get_api_current_user),
     ingest_service: ingestion_service.ScholarIngestionService = Depends(get_ingestion_service),
 ):
-    safety_state = await _load_safety_state(
+    safety_state = await _load_safety_state(db_session, user_id=current_user.id)
-        db_session,
-        user_id=current_user.id,
-    )
     if not settings.ingestion_manual_run_allowed:
         _raise_manual_runs_disabled(user_id=current_user.id, safety_state=safety_state)
 
@@ -614,28 +669,90 @@ async def run_manual(
     if reused_payload is not None:
         return reused_payload
 
-    run_summary = await _execute_manual_run(
+    try:
+        user_settings = await user_settings_service.get_or_create_settings(db_session, user_id=current_user.id)
+        
+        # Initialize run (creates the record and performs safety checks)
+        run, scholars, start_cstart_map = await ingest_service.initialize_run(
             db_session,
-        request=request,
-        ingest_service=ingest_service,
             user_id=current_user.id,
+            trigger_type=RunTriggerType.MANUAL,
+            request_delay_seconds=user_settings.request_delay_seconds,
+            network_error_retries=settings.ingestion_network_error_retries,
+            retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
+            max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
+            page_size=settings.ingestion_page_size,
+            idempotency_key=idempotency_key,
+            alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
+            alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
+            alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
+        )
+        
+        await db_session.commit()
+        
+        # Kick off background execution
+        from app.db.session import get_session_factory
+        import asyncio
+        
+        asyncio.create_task(
+            ingest_service.execute_run(
+                session_factory=get_session_factory(),
+                run_id=run.id,
+                user_id=current_user.id,
+                scholars=scholars,
+                start_cstart_map=start_cstart_map,
+                request_delay_seconds=user_settings.request_delay_seconds,
+                network_error_retries=settings.ingestion_network_error_retries,
+                retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
+                max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
+                page_size=settings.ingestion_page_size,
+                auto_queue_continuations=settings.ingestion_continuation_queue_enabled,
+                queue_delay_seconds=settings.ingestion_continuation_base_delay_seconds,
+                alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
+                alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
+                alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
                 idempotency_key=idempotency_key,
             )
-    if isinstance(run_summary, dict):
-        return run_summary
-
-    current_safety_state = await _load_safety_state(
-        db_session,
-        user_id=current_user.id,
         )
+        
         return success_payload(
             request,
-        data=_manual_run_success_payload(
+            data={
-            run_summary=run_summary,
+                "run_id": int(run.id),
-            idempotency_key=idempotency_key,
+                "status": run.status.value,
-            safety_state=current_safety_state,
+                "scholar_count": int(run.scholar_count or 0),
-        ),
+                "succeeded_count": 0,
+                "failed_count": 0,
+                "partial_count": 0,
+                "new_publication_count": 0,
+                "reused_existing_run": False,
+                "idempotency_key": idempotency_key,
+                "safety_state": await _load_safety_state(db_session, user_id=current_user.id),
+            }
         )
+    except ingestion_service.RunBlockedBySafetyPolicyError as exc:
+        await db_session.rollback()
+        _raise_manual_blocked_safety(exc=exc, user_id=current_user.id)
+    except ingestion_service.RunAlreadyInProgressError as exc:
+        await db_session.rollback()
+        raise ApiException(
+            status_code=409,
+            code="run_in_progress",
+            message="A run is already in progress for this account.",
+        ) from exc
+    except IntegrityError as exc:
+        await db_session.rollback()
+        return await _recover_integrity_error(
+            db_session,
+            request=request,
+            user_id=current_user.id,
+            idempotency_key=idempotency_key,
+            original_exc=exc,
+        )
+    except Exception as exc:
+        await db_session.rollback()
+        _raise_manual_failed(exc=exc, user_id=current_user.id)
+
 
 
 @router.get(
@@ -769,3 +886,26 @@ async def clear_queue_item(
             "message": "Queue item cleared.",
         },
     )
+
+
+@router.get("/{run_id}/stream")
+async def stream_run_events(
+    run_id: int,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    run = await run_service.get_run_for_user(
+        db_session,
+        user_id=current_user.id,
+        run_id=run_id,
+    )
+    if run is None:
+        raise ApiException(
+            status_code=404,
+            code="run_not_found",
+            message="Run not found.",
+        )
+    return StreamingResponse(
+        event_generator(run_id),
+        media_type="text/event-stream"
+    )

app/api/routers/settings.py

@@ -38,6 +38,9 @@ def _serialize_settings(user_settings) -> dict[str, object]:
             "cooldown_network_seconds": max(60, int(settings_module.ingestion_safety_cooldown_network_seconds)),
         },
         "safety_state": run_safety_service.get_safety_state_payload(user_settings),
+        "openalex_api_key": user_settings.openalex_api_key,
+        "crossref_api_token": user_settings.crossref_api_token,
+        "crossref_api_mailto": user_settings.crossref_api_mailto,
     }
 
 
@@ -107,6 +110,7 @@ def _log_settings_update(*, user_id: int, updated) -> None:
             "run_interval_minutes": updated.run_interval_minutes,
             "request_delay_seconds": updated.request_delay_seconds,
             "nav_visible_pages": updated.nav_visible_pages,
+            "openalex_api_key": "SET" if updated.openalex_api_key else "UNSET",
         },
     )
 
@@ -169,6 +173,9 @@ async def update_settings(
         run_interval_minutes=parsed_interval,
         request_delay_seconds=parsed_delay,
         nav_visible_pages=parsed_nav_visible_pages,
+        openalex_api_key=payload.openalex_api_key,
+        crossref_api_token=payload.crossref_api_token,
+        crossref_api_mailto=payload.crossref_api_mailto,
     )
     await _clear_expired_cooldown_with_log(
         db_session,

app/api/schemas.py

@@ -205,7 +205,6 @@ class PublicationExportItemData(BaseModel):
     author_text: str | None = None
     venue_text: str | None = None
     pub_url: str | None = None
-    doi: str | None = None
     pdf_url: str | None = None
     is_read: bool = False
 
@@ -594,7 +593,6 @@ class DisplayIdentifierData(BaseModel):
 class AdminPdfQueueItemData(BaseModel):
     publication_id: int
     title: str
-    doi: str | None
     display_identifier: DisplayIdentifierData | None = None
     pdf_url: str | None
     status: str
@@ -726,6 +724,10 @@ class SettingsData(BaseModel):
     policy: SettingsPolicyData
     safety_state: ScrapeSafetyStateData
     
+    openalex_api_key: str | None = None
+    crossref_api_token: str | None = None
+    crossref_api_mailto: str | None = None
+
     model_config = ConfigDict(extra="forbid")
 
 
@@ -742,6 +744,10 @@ class SettingsUpdateRequest(BaseModel):
     request_delay_seconds: int
     nav_visible_pages: list[str] | None = None
 
+    openalex_api_key: str | None = None
+    crossref_api_token: str | None = None
+    crossref_api_mailto: str | None = None
+
     model_config = ConfigDict(extra="forbid")
 
 
@@ -754,7 +760,6 @@ class PublicationItemData(BaseModel):
     citation_count: int
     venue_text: str | None
     pub_url: str | None
-    doi: str | None
     display_identifier: DisplayIdentifierData | None = None
     pdf_url: str | None
     pdf_status: str = "untracked"
@@ -780,6 +785,7 @@ class PublicationsListData(BaseModel):
     total_count: int
     page: int
     page_size: int
+    snapshot: str
     has_next: bool = False
     has_prev: bool = False
     publications: list[PublicationItemData]

app/db/models.py

@@ -31,9 +31,11 @@ class RunTriggerType(StrEnum):
 
 class RunStatus(StrEnum):
     RUNNING = "running"
+    RESOLVING = "resolving"
     SUCCESS = "success"
     PARTIAL_FAILURE = "partial_failure"
     FAILED = "failed"
+    CANCELED = "canceled"
 
 
 class QueueItemStatus(StrEnum):
@@ -113,6 +115,11 @@ class UserSetting(Base):
     )
     scrape_cooldown_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
     scrape_cooldown_reason: Mapped[str | None] = mapped_column(String(64))
+    
+    openalex_api_key: Mapped[str | None] = mapped_column(String(255))
+    crossref_api_token: Mapped[str | None] = mapped_column(String(255))
+    crossref_api_mailto: Mapped[str | None] = mapped_column(String(255))
+
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()
     )
@@ -161,6 +168,14 @@ class CrawlRun(Base):
     __tablename__ = "crawl_runs"
     __table_args__ = (
         Index("ix_crawl_runs_user_start", "user_id", "start_dt"),
+        Index(
+            "uq_crawl_runs_user_active",
+            "user_id",
+            unique=True,
+            postgresql_where=text(
+                "status IN ('running'::run_status, 'resolving'::run_status)"
+            ),
+        ),
         Index(
             "uq_crawl_runs_user_manual_idempotency_key",
             "user_id",
@@ -228,8 +243,16 @@ class Publication(Base):
     author_text: Mapped[str | None] = mapped_column(Text)
     venue_text: Mapped[str | None] = mapped_column(Text)
     pub_url: Mapped[str | None] = mapped_column(Text)
-    doi: Mapped[str | None] = mapped_column(String(255))
     pdf_url: Mapped[str | None] = mapped_column(Text)
+    canonical_title_hash: Mapped[str | None] = mapped_column(
+        String(64), nullable=True, index=True
+    )
+    openalex_enriched: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    openalex_last_attempt_at: Mapped[datetime | None] = mapped_column(
+        DateTime(timezone=True), nullable=True
+    )
     created_at: Mapped[datetime] = mapped_column(
         DateTime(timezone=True), nullable=False, server_default=func.now()
     )

app/main.py

@@ -66,6 +66,18 @@ def _log_startup_build_marker() -> None:
 @asynccontextmanager
 async def lifespan(_: FastAPI):
     _log_startup_build_marker()
+    
+    from app.db.session import get_session_factory
+    from sqlalchemy import text
+    try:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            await session.execute(text("UPDATE crawl_runs SET status = 'failed' WHERE status::text IN ('running', 'resolving')"))
+            await session.commit()
+            logger.info("app.startup_orphaned_runs_cleaned", extra={"event": "app.startup_orphaned_runs_cleaned"})
+    except Exception as e:
+        logger.error(f"Failed to clean orphaned runs at startup: {e}")
+
     await scheduler_service.start()
     yield
     await scheduler_service.stop()

app/services/domains/arxiv/application.py

@@ -0,0 +1,110 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import TYPE_CHECKING
+import xml.etree.ElementTree as ET
+
+import httpx
+
+from app.services.domains.publication_identifiers.normalize import normalize_arxiv_id
+from app.settings import settings
+
+if TYPE_CHECKING:
+    from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
+
+logger = logging.getLogger(__name__)
+
+# arXiv API terms: max 1 request per 3 seconds, single connection at a time.
+_ARXIV_LOCK = asyncio.Lock()
+_ARXIV_MIN_INTERVAL_SECONDS = 4.0
+# Global cooldown: when arXiv returns 429, all batches back off for this long.
+_ARXIV_RATE_LIMIT_COOLDOWN_SECONDS = 60.0
+_arxiv_rate_limited_until: float = 0.0  # asyncio monotonic time
+
+
+class ArxivRateLimitError(Exception):
+    """arXiv returned 429 — stop the batch to avoid hammering."""
+    pass
+
+
+def _build_arxiv_query(title: str, author_surname: str | None) -> str | None:
+    parts = []
+    if title:
+        clean_title = title.replace('"', '').replace("'", "")
+        parts.append(f'ti:"{clean_title}"')
+    if author_surname:
+        parts.append(f'au:"{author_surname}"')
+    if not parts:
+        return None
+    return " AND ".join(parts)
+
+
+async def discover_arxiv_id_for_publication(
+    *,
+    item: PublicationListItem | UnreadPublicationItem,
+    request_email: str | None = None,
+    timeout_seconds: float = 3.0,
+) -> str | None:
+    title = (item.title or "").strip()
+    if not title:
+        return None
+
+    author_surname = None
+    if item.scholar_label:
+        tokens = [t for t in item.scholar_label.strip().split() if t]
+        if tokens:
+            author_surname = tokens[-1].lower()
+
+    query = _build_arxiv_query(title, author_surname)
+    if not query:
+        return None
+
+    url = "https://export.arxiv.org/api/query"
+    params = {"search_query": query, "start": 0, "max_results": 3}
+    headers = {
+        "User-Agent": (
+            f"scholar-scraper/1.0 "
+            f"(mailto:{request_email or settings.crossref_api_mailto or 'unknown@example.com'})"
+        )
+    }
+
+    try:
+        async with _ARXIV_LOCK:
+            global _arxiv_rate_limited_until
+            now = asyncio.get_running_loop().time()
+            if now < _arxiv_rate_limited_until:
+                remaining = _arxiv_rate_limited_until - now
+                raise ArxivRateLimitError(f"arXiv global cooldown active ({remaining:.0f}s remaining)")
+
+            async with httpx.AsyncClient(timeout=timeout_seconds, follow_redirects=True, headers=headers) as client:
+                response = await client.get(url, params=params)
+
+            if response.status_code == 429:
+                _arxiv_rate_limited_until = asyncio.get_running_loop().time() + _ARXIV_RATE_LIMIT_COOLDOWN_SECONDS
+                raise ArxivRateLimitError("arXiv rate limit hit (429) — stopping batch")
+
+            await asyncio.sleep(_ARXIV_MIN_INTERVAL_SECONDS)
+
+        response.raise_for_status()
+
+        root = ET.fromstring(response.text)
+        namespace = {"atom": "http://www.w3.org/2005/Atom"}
+
+        for entry in root.findall("atom:entry", namespace):
+            id_elem = entry.find("atom:id", namespace)
+            if id_elem is not None and id_elem.text:
+                candidate = str(id_elem.text)
+                if "/abs/" in candidate:
+                    candidate = candidate.split("/abs/")[-1]
+                normalized = normalize_arxiv_id(candidate)
+                if normalized:
+                    logger.debug("arxiv.id_discovered", extra={"event": "arxiv.id_discovered", "arxiv_id": normalized})
+                    return normalized
+
+    except ArxivRateLimitError:
+        raise  # propagate so the batch loop can stop
+    except Exception as exc:
+        logger.debug(f"Failed to query arXiv API: {exc}")
+
+    return None

app/services/domains/ingestion/fingerprints.py

@@ -15,12 +15,59 @@ from app.services.domains.ingestion.constants import (
 )
 from app.services.domains.scholar.parser import ParseState, ParsedProfilePage, PublicationCandidate
 
+# Scholar-specific noise patterns stripped before canonical comparison.
+# Applied in order; each targets a different Scholar metadata injection style.
+_NOISE_DOI_RE = re.compile(r"[,.\s]+doi\s*:\s*\S+.*$", re.IGNORECASE)
+_NOISE_ARXIV_RE = re.compile(r"[,.\s]+arxiv\b.*$", re.IGNORECASE)
+_NOISE_PREPRINT_RE = re.compile(
+    r"[,\s]+(?:preprint|extended\s+version|technical\s+report|working\s+paper)\b.*$",
+    re.IGNORECASE,
+)
+_NOISE_TRAILING_YEAR_RE = re.compile(r"\s*[,(]\s*\d{4}\s*[),]?\s*$")
+# Strips ". Capitalised sentence" appended as venue: ". Comput. Sci…", ". Journal of…"
+_NOISE_VENUE_SENTENCE_RE = re.compile(r"(?<=\w{3})\.\s+[A-Z][a-z].*$")
+
+_CANONICAL_DEDUP_THRESHOLD = 0.82
+
 
 def normalize_title(value: str) -> str:
     lowered = value.lower()
     return TITLE_ALNUM_RE.sub("", lowered)
 
 
+def canonical_title_for_dedup(title: str) -> str:
+    """Strip Scholar-specific noise suffixes then normalize for dedup comparison."""
+    t = title.strip()
+    t = _NOISE_DOI_RE.sub("", t)
+    t = _NOISE_ARXIV_RE.sub("", t)
+    t = _NOISE_PREPRINT_RE.sub("", t)
+    t = _NOISE_TRAILING_YEAR_RE.sub("", t)
+    t = _NOISE_VENUE_SENTENCE_RE.sub("", t)
+    return normalize_title(t.strip())
+
+
+def _stripped_title_for_canonical(title: str) -> str:
+    """Apply noise-stripping and lowercase but PRESERVE spaces (for later tokenization)."""
+    t = title.strip()
+    t = _NOISE_DOI_RE.sub("", t)
+    t = _NOISE_ARXIV_RE.sub("", t)
+    t = _NOISE_PREPRINT_RE.sub("", t)
+    t = _NOISE_TRAILING_YEAR_RE.sub("", t)
+    t = _NOISE_VENUE_SENTENCE_RE.sub("", t)
+    return t.lower().strip()
+
+
+def _canonical_title_tokens(title: str) -> set[str]:
+    """Word tokens of the noise-stripped title (preserves token boundaries)."""
+    return set(WORD_RE.findall(_stripped_title_for_canonical(title)))
+
+
+def _jaccard(a: set[str], b: set[str]) -> float:
+    if not a or not b:
+        return 0.0
+    return len(a & b) / len(a | b)
+
+
 def _first_author_last_name(authors_text: str | None) -> str:
     if not authors_text:
         return ""
@@ -100,29 +147,89 @@ def _next_cstart_value(*, articles_range: str | None, fallback: int) -> int:
     return int(fallback)
 
 
+def _title_tokens(value: str) -> set[str]:
+    """Extract normalized word tokens for fuzzy title comparison."""
+    return set(WORD_RE.findall(value.lower()))
+
+
+def fuzzy_titles_match(
+    title_a: str,
+    title_b: str,
+    *,
+    threshold: float = 0.85,
+) -> bool:
+    """Return True if two titles are near-duplicates by token-level Jaccard similarity.
+
+    A threshold of 0.85 catches common academic duplicate patterns:
+    differences in punctuation, minor word variations, subtitle changes.
+    """
+    tokens_a = _title_tokens(title_a)
+    tokens_b = _title_tokens(title_b)
+    return _jaccard(tokens_a, tokens_b) >= threshold
+
+
 def _dedupe_publication_candidates(
     publications: list[PublicationCandidate],
+    *,
+    seen_canonical: set[str] | None = None,
 ) -> list[PublicationCandidate]:
+    """Deduplicate candidates using canonical title matching.
+
+    Args:
+        publications: candidates to filter
+        seen_canonical: optional mutable set shared across pages.  Stores the
+            noise-stripped *lowercased* (but space-preserved) canonical string
+            so it can be tokenized on the next page for cross-page fuzzy dedup.
+            Accepted canonicals are added; existing entries are consulted.
+    """
     deduped: list[PublicationCandidate] = []
-    seen: set[str] = set()
+    seen_exact: set[str] = set()
-    for publication in publications:
+    # Token sets for fuzzy comparison; seeded from cross-page state.
-        if publication.cluster_id:
+    seen_tokens: list[set[str]] = []
-            identity = f"cluster:{publication.cluster_id}"
+
-        else:
+    if seen_canonical:
-            identity = "|".join(
+        for stripped in seen_canonical:
+            seen_tokens.append(set(WORD_RE.findall(stripped)))
+
+    for pub in publications:
+        identity = _publication_identity(pub)
+        if identity in seen_exact:
+            continue
+
+        # Use space-preserving stripped form for token-level fuzzy match.
+        tokens = _canonical_title_tokens(pub.title)
+        if _is_fuzzy_dup(tokens, seen_tokens):
+            continue
+
+        seen_exact.add(identity)
+        seen_tokens.append(tokens)
+        if seen_canonical is not None:
+            # Store the noise-stripped lowercased (space-preserved) form.
+            seen_canonical.add(_stripped_title_for_canonical(pub.title))
+        deduped.append(pub)
+
+    return deduped
+
+
+def _publication_identity(pub: PublicationCandidate) -> str:
+    if pub.cluster_id:
+        return f"cluster:{pub.cluster_id}"
+    canonical = canonical_title_for_dedup(pub.title)
+    return "|".join(
         [
             "fallback",
-                    normalize_title(publication.title),
+            canonical,
-                    str(publication.year) if publication.year is not None else "",
+            str(pub.year) if pub.year is not None else "",
-                    publication.authors_text or "",
+            _first_author_last_name(pub.authors_text),
-                    publication.venue_text or "",
         ]
     )
-        if identity in seen:
+
-            continue
+
-        seen.add(identity)
+def _is_fuzzy_dup(tokens: set[str], seen: list[set[str]]) -> bool:
-        deduped.append(publication)
+    for existing in seen:
-    return deduped
+        if _jaccard(tokens, existing) >= _CANONICAL_DEDUP_THRESHOLD:
+            return True
+    return False
 
 
 def _build_body_excerpt(body: str, *, max_chars: int = 220) -> str | None:

app/services/domains/ingestion/scheduler.py

@@ -146,6 +146,9 @@ class SchedulerService:
     async def _tick_once(self) -> None:
         if self._continuation_queue_enabled:
             await self._drain_continuation_queue()
+        
+        await self._drain_pdf_queue()
+
         candidates = await self._load_candidates()
         if not candidates:
             return
@@ -488,6 +491,8 @@ class SchedulerService:
                     request_delay_seconds=request_delay_seconds,
                     network_error_retries=self._network_error_retries,
                     retry_backoff_seconds=self._retry_backoff_seconds,
+                    rate_limit_retries=settings.ingestion_rate_limit_retries,
+                    rate_limit_backoff_seconds=settings.ingestion_rate_limit_backoff_seconds,
                     max_pages_per_scholar=self._max_pages_per_scholar,
                     page_size=self._page_size,
                     scholar_profile_ids={job.scholar_profile_id},
@@ -583,6 +588,27 @@ class SchedulerService:
             return
         await self._finalize_queue_job_after_run(job, run_summary)
 
+    async def _drain_pdf_queue(self) -> None:
+        from app.services.domains.publications.pdf_queue import drain_ready_jobs
+        
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            try:
+                processed = await drain_ready_jobs(
+                    session,
+                    limit=settings.scheduler_pdf_queue_batch_size,
+                    max_attempts=settings.pdf_auto_retry_max_attempts,
+                )
+                if processed > 0:
+                    logger.info("scheduler.pdf_queue_drain_completed", extra={
+                        "event": "scheduler.pdf_queue_drain_completed",
+                        "processed_count": processed,
+                    })
+            except Exception:
+                logger.exception("scheduler.pdf_queue_drain_failed", extra={
+                    "event": "scheduler.pdf_queue_drain_failed",
+                })
+
     async def _load_request_delay_for_user(
         self,
         db_session: AsyncSession,

app/services/domains/ingestion/types.py

@@ -35,6 +35,7 @@ class PagedParseResult:
     pagination_truncated_reason: str | None
     continuation_cstart: int | None
     skipped_no_change: bool
+    discovered_publication_count: int
 
 
 @dataclass
@@ -88,6 +89,7 @@ class PagedLoopState:
     has_more_remaining: bool = False
     pagination_truncated_reason: str | None = None
     continuation_cstart: int | None = None
+    discovered_publication_count: int = 0
 
 
 class RunAlreadyInProgressError(RuntimeError):

app/services/domains/openalex/__init__.py

@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+import logging
+
+logger = logging.getLogger(__name__)

app/services/domains/openalex/client.py

@@ -0,0 +1,151 @@
+import asyncio
+import logging
+from typing import Any, Mapping
+
+import httpx
+from tenacity import (
+    retry,
+    retry_if_exception_type,
+    stop_after_attempt,
+    wait_exponential,
+)
+
+from app.services.domains.openalex.types import OpenAlexWork
+
+logger = logging.getLogger(__name__)
+
+OPENALEX_BASE_URL = "https://api.openalex.org"
+
+
+class OpenAlexClientError(Exception):
+    pass
+
+
+class OpenAlexRateLimitError(OpenAlexClientError):
+    """Transient rate limit (too many requests per second)."""
+    pass
+
+
+class OpenAlexBudgetExhaustedError(OpenAlexClientError):
+    """Daily API budget exhausted — retrying is futile until midnight UTC."""
+    pass
+
+
+class OpenAlexClient:
+    def __init__(
+        self,
+        api_key: str | None = None,
+        mailto: str | None = None,
+        timeout: float = 10.0,
+    ) -> None:
+        self.api_key = api_key
+        self.mailto = mailto
+        self.timeout = timeout
+
+    @property
+    def _base_params(self) -> dict[str, str]:
+        params = {}
+        if self.mailto:
+            params["mailto"] = self.mailto
+        if self.api_key:
+            params["api_key"] = self.api_key
+        return params
+
+    @retry(
+        retry=retry_if_exception_type((httpx.NetworkError, httpx.TimeoutException)),
+        stop=stop_after_attempt(3),
+        wait=wait_exponential(multiplier=1, min=2, max=10),
+        reraise=True,
+    )
+    async def get_work_by_doi(self, doi: str) -> OpenAlexWork | None:
+        """Fetch a single work by DOI directly."""
+        clean_doi = doi.replace("https://doi.org/", "")
+        if not clean_doi:
+            return None
+
+        url = f"{OPENALEX_BASE_URL}/works/{clean_doi}"
+        
+        headers = {}
+        if self.mailto:
+            headers["User-Agent"] = f"scholar-scraper/1.0 (mailto:{self.mailto})"
+        else:
+            headers["User-Agent"] = "scholar-scraper/1.0"
+
+        async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True, headers=headers) as client:
+            response = await client.get(url, params=self._base_params)
+
+        if response.status_code == 404:
+            return None
+        if response.status_code == 429:
+            remaining = response.headers.get("X-RateLimit-Remaining-USD", "")
+            if remaining == "0" or remaining.startswith("-"):
+                raise OpenAlexBudgetExhaustedError(
+                    "Daily API budget exhausted; retrying won't help until midnight UTC"
+                )
+            raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex work by DOI")
+        if response.status_code >= 400:
+            logger.warning("OpenAlex API error: %s %s", response.status_code, response.text)
+            raise OpenAlexClientError(f"API Error {response.status_code}")
+
+        data = response.json()
+        return OpenAlexWork.from_api_dict(data)
+
+    @retry(
+        retry=retry_if_exception_type((httpx.NetworkError, httpx.TimeoutException)),
+        stop=stop_after_attempt(3),
+        wait=wait_exponential(multiplier=1, min=2, max=10),
+        reraise=True,
+    )
+    async def get_works_by_filter(
+        self,
+        filters: dict[str, str],
+        limit: int = 50,
+    ) -> list[OpenAlexWork]:
+        """
+        Fetch works using the ?filter= query parameter.
+        Supports fetching multiple records by joining filters with | (OR logic).
+        """
+        if not filters:
+            return []
+
+        # Example: {"doi": "10.foo|10.bar", "title.search": "query"}
+        filter_str = ",".join(f"{k}:{v}" for k, v in filters.items())
+        
+        params = self._base_params.copy()
+        params["filter"] = filter_str
+        params["per-page"] = str(limit)
+
+        url = f"{OPENALEX_BASE_URL}/works"
+        
+        headers = {}
+        if self.mailto:
+            headers["User-Agent"] = f"scholar-scraper/1.0 (mailto:{self.mailto})"
+        else:
+            headers["User-Agent"] = "scholar-scraper/1.0"
+
+        async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True, headers=headers) as client:
+            response = await client.get(url, params=params)
+
+        if response.status_code == 429:
+            remaining = response.headers.get("X-RateLimit-Remaining-USD", "")
+            if remaining == "0" or remaining.startswith("-"):
+                raise OpenAlexBudgetExhaustedError(
+                    "Daily API budget exhausted; retrying won't help until midnight UTC"
+                )
+            raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex works list")
+        if response.status_code >= 400:
+            logger.warning("OpenAlex API error (filters=%s): %s %s", filters, response.status_code, response.text)
+            raise OpenAlexClientError(f"API Error {response.status_code}")
+
+        data = response.json()
+        results = data.get("results") or []
+        
+        parsed_works = []
+        for raw_work in results:
+            try:
+                parsed_works.append(OpenAlexWork.from_api_dict(raw_work))
+            except Exception as e:
+                logger.warning("Failed to parse OpenAlex raw dict: %s", e)
+                continue
+
+        return parsed_works

app/services/domains/openalex/matching.py

@@ -0,0 +1,108 @@
+import logging
+import re
+
+from rapidfuzz import fuzz
+
+from app.services.domains.openalex.types import OpenAlexWork
+
+logger = logging.getLogger(__name__)
+
+# A minimum similarity score out of 100 for a title to be considered a match candidate.
+TITLE_MATCH_THRESHOLD = 90.0
+# The margin within the top score where a secondary tiebreaker (author/year) is necessary.
+TIEBREAKER_MARGIN = 5.0
+
+
+def _clean_string(s: str | None) -> str:
+    if not s:
+        return ""
+    # Strip non-alphanumeric (keep spaces), lowercase, and collapse whitespace
+    cleaned = re.sub(r"[^a-z0-9\s]", " ", s.lower())
+    return " ".join(cleaned.split())
+
+
+def _author_overlap_score(target_authors: str | None, candidate_authors: list[str]) -> bool:
+    if not target_authors or not candidate_authors:
+        return False
+    
+    target_clean = _clean_string(target_authors)
+    if not target_clean:
+        return False
+        
+    for candidate in candidate_authors:
+        cand_clean = _clean_string(candidate)
+        if cand_clean and (cand_clean in target_clean or target_clean in cand_clean):
+            return True
+        # Alternatively check rapidfuzz token_set_ratio
+        if cand_clean and fuzz.token_set_ratio(target_clean, cand_clean) > 80:
+            return True
+            
+    return False
+
+
+def find_best_match(
+    target_title: str,
+    target_year: int | None,
+    target_authors: str | None,
+    candidates: list[OpenAlexWork],
+) -> OpenAlexWork | None:
+    """
+    Finds the best matching OpenAlexWork from a list of candidates, prioritizing title similarity (>90%)
+    with year and author overlap as tiebreakers for close candidates.
+    """
+    if not target_title or not candidates:
+        return None
+
+    clean_target = _clean_string(target_title)
+    if not clean_target:
+        return None
+
+    scored_candidates: list[tuple[float, OpenAlexWork]] = []
+
+    for cand in candidates:
+        if not cand.title:
+            continue
+            
+        clean_cand = _clean_string(cand.title)
+        
+        # Primary sort: string similarity ratio
+        score = fuzz.ratio(clean_target, clean_cand)
+        
+        if score >= TITLE_MATCH_THRESHOLD:
+            scored_candidates.append((score, cand))
+
+    if not scored_candidates:
+        return None
+
+    # Sort descending by score
+    scored_candidates.sort(key=lambda x: x[0], reverse=True)
+    
+    best_score = scored_candidates[0][0]
+    
+    # Extract all candidates within the tiebreaker margin
+    top_scored_candidates = [
+        (score, cand) for score, cand in scored_candidates 
+        if best_score - score <= TIEBREAKER_MARGIN
+    ]
+
+    if len(top_scored_candidates) == 1:
+        return top_scored_candidates[0][1]
+
+    # We have a tie or near-tie. Use year and author overlap to break the tie.
+    # Score candidates: +1 for year match (within 1 year), +1 for author overlap
+    tiebreaker_scores: list[tuple[int, float, OpenAlexWork]] = []
+    
+    for original_score, cand in top_scored_candidates:
+        tb_score = 0
+        if target_year is not None and cand.publication_year is not None:
+            if abs(target_year - cand.publication_year) <= 1:
+                tb_score += 1
+                
+        candidate_author_names = [a.display_name for a in cand.authors if a.display_name]
+        if _author_overlap_score(target_authors, candidate_author_names):
+            tb_score += 1
+            
+        tiebreaker_scores.append((tb_score, original_score, cand))
+        
+    tiebreaker_scores.sort(key=lambda x: (x[0], x[1]), reverse=True)
+    return tiebreaker_scores[0][2]

app/services/domains/openalex/types.py

@@ -0,0 +1,71 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Mapping
+
+
+@dataclass(frozen=True)
+class OpenAlexAuthor:
+    openalex_id: str | None
+    display_name: str | None
+
+
+@dataclass(frozen=True)
+class OpenAlexWork:
+    openalex_id: str
+    doi: str | None
+    pmid: str | None
+    pmcid: str | None
+    title: str | None
+    publication_year: int | None
+    cited_by_count: int
+    is_oa: bool
+    oa_url: str | None
+    authors: list[OpenAlexAuthor] = field(default_factory=list)
+    raw_data: Mapping[str, Any] = field(default_factory=dict, repr=False)
+
+    @classmethod
+    def from_api_dict(cls, data: Mapping[str, Any]) -> OpenAlexWork:
+        ids = data.get("ids") or {}
+        
+        # Extract DOI without the https://doi.org/ prefix
+        doi = ids.get("doi")
+        if doi and doi.startswith("https://doi.org/"):
+            doi = doi[16:]
+            
+        # Extract PMID without the url prefix
+        pmid = ids.get("pmid")
+        if pmid and pmid.startswith("https://pubmed.ncbi.nlm.nih.gov/"):
+            pmid = pmid[32:]
+            
+        # Extract PMCID without the url prefix
+        pmcid = ids.get("pmcid")
+        if pmcid and pmcid.startswith("https://www.ncbi.nlm.nih.gov/pmc/articles/"):
+            pmcid = pmcid[42:]
+
+        open_access = data.get("open_access") or {}
+        
+        authors = []
+        for authorship in data.get("authorships") or []:
+            author_data = authorship.get("author") or {}
+            authors.append(
+                OpenAlexAuthor(
+                    openalex_id=author_data.get("id"),
+                    display_name=author_data.get("display_name"),
+                )
+            )
+
+        return cls(
+            openalex_id=data.get("id", ""),
+            doi=doi,
+            pmid=pmid,
+            pmcid=pmcid,
+            title=data.get("title"),
+            publication_year=data.get("publication_year"),
+            cited_by_count=data.get("cited_by_count", 0),
+            is_oa=bool(open_access.get("is_oa")),
+            oa_url=open_access.get("oa_url"),
+            authors=authors,
+            raw_data=dict(data),
+        )

app/services/domains/portability/exporting.py

@@ -34,7 +34,6 @@ def _serialize_export_publication(row: tuple[Any, ...]) -> dict[str, Any]:
         author_text,
         venue_text,
         pub_url,
-        doi,
         pdf_url,
         is_read,
     ) = row
@@ -48,7 +47,6 @@ def _serialize_export_publication(row: tuple[Any, ...]) -> dict[str, Any]:
         "author_text": author_text,
         "venue_text": venue_text,
         "pub_url": pub_url,
-        "doi": doi,
         "pdf_url": pdf_url,
         "is_read": bool(is_read),
     }
@@ -75,7 +73,6 @@ async def export_user_data(
             Publication.author_text,
             Publication.venue_text,
             Publication.pub_url,
-            Publication.doi,
             Publication.pdf_url,
             ScholarPublication.is_read,
         )

app/services/domains/portability/publication_import.py

@@ -71,7 +71,6 @@ def _apply_imported_publication_values(
     author_text: str | None,
     venue_text: str | None,
     pub_url: str | None,
-    doi: str | None,
     pdf_url: str | None,
     cluster_id: str | None,
 ) -> bool:
@@ -98,9 +97,6 @@ def _apply_imported_publication_values(
     if pub_url and publication.pub_url != pub_url:
         publication.pub_url = pub_url
         updated = True
-    if doi and publication.doi != doi:
-        publication.doi = doi
-        updated = True
     if pdf_url and publication.pdf_url != pdf_url:
         publication.pdf_url = pdf_url
         updated = True
@@ -117,7 +113,6 @@ def _new_publication(
     author_text: str | None,
     venue_text: str | None,
     pub_url: str | None,
-    doi: str | None,
     pdf_url: str | None,
 ) -> Publication:
     return Publication(
@@ -130,7 +125,6 @@ def _new_publication(
         author_text=author_text,
         venue_text=venue_text,
         pub_url=pub_url,
-        doi=doi,
         pdf_url=pdf_url,
     )
 
@@ -285,7 +279,6 @@ async def _create_import_publication(
         author_text=payload.author_text,
         venue_text=payload.venue_text,
         pub_url=payload.pub_url,
-        doi=payload.doi,
         pdf_url=payload.pdf_url,
     )
     db_session.add(publication)
@@ -306,7 +299,6 @@ def _update_import_publication(
         author_text=payload.author_text,
         venue_text=payload.venue_text,
         pub_url=payload.pub_url,
-        doi=payload.doi,
         pdf_url=payload.pdf_url,
         cluster_id=payload.cluster_id,
     )

app/services/domains/publication_identifiers/application.py

@@ -21,7 +21,7 @@ from app.services.domains.publication_identifiers.types import (
 
 if TYPE_CHECKING:
     from app.services.domains.publications.pdf_queue import PdfQueueListItem
-    from app.services.domains.publications.types import PublicationListItem
+    from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
 
 CONFIDENCE_HIGH = 0.98
 CONFIDENCE_MEDIUM = 0.9
@@ -117,9 +117,76 @@ async def sync_identifiers_for_publication_fields(
     await _upsert_publication_candidates(db_session, publication_id=int(publication.id), candidates=candidates)
 
 
+async def discover_and_sync_identifiers_for_publication(
+    db_session: AsyncSession,
+    *,
+    publication: Publication,
+    scholar_label: str,
+) -> None:
+    await sync_identifiers_for_publication_fields(db_session, publication=publication)
+    
+    existing_doi = await _existing_identifier_by_kind(
+        db_session,
+        publication_id=int(publication.id),
+        kind=IdentifierKind.DOI.value,
+    )
+    if existing_doi is not None:
+        return
+
+    from app.services.domains.crossref import application as crossref_service
+    from app.services.domains.publications.types import UnreadPublicationItem
+    
+    item = UnreadPublicationItem(
+        publication_id=int(publication.id),
+        scholar_profile_id=0,
+        scholar_label=scholar_label,
+        title=str(publication.title_raw or ""),
+        year=publication.year,
+        citation_count=publication.citation_count,
+        venue_text=publication.venue_text,
+        pub_url=publication.pub_url,
+        pdf_url=publication.pdf_url,
+    )
+    
+    discovered_doi = await crossref_service.discover_doi_for_publication(item=item)
+    if discovered_doi:
+        normalized_doi = normalize_doi(discovered_doi)
+        if normalized_doi:
+            candidate = _candidate(
+                IdentifierKind.DOI,
+                discovered_doi,
+                normalized_doi,
+                "crossref_api",
+                CONFIDENCE_MEDIUM,
+                None,
+            )
+            await _upsert_publication_candidate(db_session, publication_id=int(publication.id), candidate=candidate)
+
+    existing_arxiv = await _existing_identifier_by_kind(
+        db_session,
+        publication_id=int(publication.id),
+        kind=IdentifierKind.ARXIV.value,
+    )
+    if existing_arxiv is None:
+        from app.services.domains.arxiv import application as arxiv_service
+        discovered_arxiv = await arxiv_service.discover_arxiv_id_for_publication(item=item)
+        if discovered_arxiv:
+            normalized_arxiv = normalize_arxiv_id(discovered_arxiv)
+            if normalized_arxiv:
+                candidate = _candidate(
+                    IdentifierKind.ARXIV,
+                    discovered_arxiv,
+                    normalized_arxiv,
+                    "arxiv_api",
+                    CONFIDENCE_MEDIUM,
+                    None,
+                )
+                await _upsert_publication_candidate(db_session, publication_id=int(publication.id), candidate=candidate)
+
+
 def _publication_field_candidates(publication: Publication) -> list[IdentifierCandidate]:
     return _fallback_candidates_from_values(
-        doi=publication.doi,
+        doi=None,
         pub_url=publication.pub_url,
         pdf_url=publication.pdf_url,
     )
@@ -194,6 +261,21 @@ async def _existing_identifier(
     return result.scalar_one_or_none()
 
 
+async def _existing_identifier_by_kind(
+    db_session: AsyncSession,
+    *,
+    publication_id: int,
+    kind: str,
+) -> PublicationIdentifier | None:
+    result = await db_session.execute(
+        select(PublicationIdentifier).where(
+            PublicationIdentifier.publication_id == publication_id,
+            PublicationIdentifier.kind == kind,
+        ).order_by(PublicationIdentifier.confidence_score.desc()).limit(1)
+    )
+    return result.scalar_one_or_none()
+
+
 def _new_identifier_row(
     *,
     publication_id: int,
@@ -234,7 +316,7 @@ def _overlay_publication_item(
     item: PublicationListItem,
     display_identifier: DisplayIdentifier | None,
 ) -> PublicationListItem:
-    fallback = display_identifier or derive_display_identifier_from_values(doi=item.doi, pub_url=item.pub_url, pdf_url=item.pdf_url)
+    fallback = display_identifier or derive_display_identifier_from_values(doi=None, pub_url=item.pub_url, pdf_url=item.pdf_url)
     return replace(item, display_identifier=fallback)
 
 
@@ -253,7 +335,7 @@ def _overlay_queue_item(
     item: PdfQueueListItem,
     display_identifier: DisplayIdentifier | None,
 ) -> PdfQueueListItem:
-    fallback = display_identifier or derive_display_identifier_from_values(doi=item.doi, pdf_url=item.pdf_url)
+    fallback = display_identifier or derive_display_identifier_from_values(doi=None, pdf_url=item.pdf_url)
     return replace(item, display_identifier=fallback)
 
 

app/services/domains/publication_identifiers/normalize.py

@@ -7,7 +7,7 @@ from app.services.domains.doi.normalize import normalize_doi
 from app.services.domains.publication_identifiers.types import IdentifierKind
 
 ARXIV_ABS_RE = re.compile(r"\barxiv:\s*([a-z-]+/\d{7}|\d{4}\.\d{4,5})(v\d+)?\b", re.I)
-ARXIV_PATH_RE = re.compile(r"^/(?:abs|pdf)/([a-z-]+/\d{7}|\d{4}\.\d{4,5})(v\d+)?(?:\.pdf)?/?$", re.I)
+ARXIV_PATH_RE = re.compile(r"^/(?:abs|pdf|html|ps|format)/([a-z-]+/\d{7}|\d{4}\.\d{4,5})(v\d+)?(?:\.pdf)?/?$", re.I)
 PMCID_RE = re.compile(r"\b(PMC\d+)\b", re.I)
 PUBMED_PATH_RE = re.compile(r"^/(\d+)/?$")
 

app/services/domains/publications/application.py

@@ -24,7 +24,7 @@ from app.services.domains.publications.modes import (
     resolve_publication_view_mode,
 )
 from app.services.domains.publications.queries import (
-    get_latest_completed_run_id_for_user,
+    get_latest_run_id_for_user,
     get_publication_item_for_user,
     publications_query,
 )
@@ -50,7 +50,7 @@ __all__ = [
     "PublicationListItem",
     "UnreadPublicationItem",
     "resolve_publication_view_mode",
-    "get_latest_completed_run_id_for_user",
+    "get_latest_run_id_for_user",
     "publications_query",
     "get_publication_item_for_user",
     "list_for_user",

app/services/domains/publications/counts.py

@@ -1,6 +1,8 @@
 from __future__ import annotations
 
-from sqlalchemy import func, select
+from datetime import datetime
+
+from sqlalchemy import distinct, func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.models import ScholarProfile, ScholarPublication
@@ -10,7 +12,7 @@ from app.services.domains.publications.modes import (
     MODE_UNREAD,
     resolve_publication_view_mode,
 )
-from app.services.domains.publications.queries import get_latest_completed_run_id_for_user
+from app.services.domains.publications.queries import get_latest_run_id_for_user
 
 
 async def count_for_user(
@@ -20,11 +22,12 @@ async def count_for_user(
     mode: str = MODE_ALL,
     scholar_profile_id: int | None = None,
     favorite_only: bool = False,
+    snapshot_before: datetime | None = None,
 ) -> int:
     resolved_mode = resolve_publication_view_mode(mode)
-    latest_run_id = await get_latest_completed_run_id_for_user(db_session, user_id=user_id)
+    latest_run_id = await get_latest_run_id_for_user(db_session, user_id=user_id)
     stmt = (
-        select(func.count())
+        select(func.count(distinct(ScholarPublication.publication_id)))
         .select_from(ScholarPublication)
         .join(ScholarProfile, ScholarProfile.id == ScholarPublication.scholar_profile_id)
         .where(ScholarProfile.user_id == user_id)
@@ -33,6 +36,8 @@ async def count_for_user(
         stmt = stmt.where(ScholarProfile.id == scholar_profile_id)
     if favorite_only:
         stmt = stmt.where(ScholarPublication.is_favorite.is_(True))
+    if snapshot_before is not None:
+        stmt = stmt.where(ScholarPublication.created_at <= snapshot_before)
     if resolved_mode == MODE_UNREAD:
         stmt = stmt.where(ScholarPublication.is_read.is_(False))
     if resolved_mode == MODE_LATEST:
@@ -49,6 +54,7 @@ async def count_unread_for_user(
     user_id: int,
     scholar_profile_id: int | None = None,
     favorite_only: bool = False,
+    snapshot_before: datetime | None = None,
 ) -> int:
     return await count_for_user(
         db_session,
@@ -56,6 +62,7 @@ async def count_unread_for_user(
         mode=MODE_UNREAD,
         scholar_profile_id=scholar_profile_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
 
 
@@ -65,6 +72,7 @@ async def count_latest_for_user(
     user_id: int,
     scholar_profile_id: int | None = None,
     favorite_only: bool = False,
+    snapshot_before: datetime | None = None,
 ) -> int:
     return await count_for_user(
         db_session,
@@ -72,6 +80,7 @@ async def count_latest_for_user(
         mode=MODE_LATEST,
         scholar_profile_id=scholar_profile_id,
         favorite_only=favorite_only,
+        snapshot_before=snapshot_before,
     )
 
 
@@ -80,6 +89,7 @@ async def count_favorite_for_user(
     *,
     user_id: int,
     scholar_profile_id: int | None = None,
+    snapshot_before: datetime | None = None,
 ) -> int:
     return await count_for_user(
         db_session,
@@ -87,4 +97,5 @@ async def count_favorite_for_user(
         mode=MODE_ALL,
         scholar_profile_id=scholar_profile_id,
         favorite_only=True,
+        snapshot_before=snapshot_before,
     )

app/services/domains/publications/dedup.py

@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+import logging
+
+from sqlalchemy import delete, select
+from sqlalchemy.orm import aliased
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import Publication, PublicationIdentifier, ScholarPublication
+
+logger = logging.getLogger(__name__)
+
+
+async def find_identifier_duplicate_pairs(
+    db_session: AsyncSession,
+) -> list[tuple[int, int]]:
+    """Return (winner_id, dup_id) pairs where two publications share the same identifier.
+
+    Winner is always the lower publication_id (earlier-created). Uses the existing
+    ix_publication_identifiers_kind_value index for the self-join.
+    """
+    pi1 = aliased(PublicationIdentifier, name="pi1")
+    pi2 = aliased(PublicationIdentifier, name="pi2")
+    rows = await db_session.execute(
+        select(pi1.publication_id, pi2.publication_id)
+        .join(
+            pi2,
+            (pi1.kind == pi2.kind)
+            & (pi1.value_normalized == pi2.value_normalized)
+            & (pi1.publication_id < pi2.publication_id),
+        )
+        .distinct()
+    )
+    return [(winner_id, dup_id) for winner_id, dup_id in rows]
+
+
+async def merge_duplicate_publication(
+    db_session: AsyncSession,
+    *,
+    winner_id: int,
+    dup_id: int,
+) -> None:
+    """Merge dup_id into winner_id: migrate scholar links, then delete the dup."""
+    await _migrate_scholar_links(db_session, winner_id=winner_id, dup_id=dup_id)
+    await db_session.execute(
+        delete(Publication).where(Publication.id == dup_id)
+    )
+    logger.info(
+        "publications.identifier_merge",
+        extra={
+            "event": "publications.identifier_merge",
+            "winner_id": winner_id,
+            "dup_id": dup_id,
+        },
+    )
+
+
+async def _migrate_scholar_links(
+    db_session: AsyncSession,
+    *,
+    winner_id: int,
+    dup_id: int,
+) -> None:
+    """Move ScholarPublication links from dup to winner, dropping conflicts."""
+    dup_links_result = await db_session.execute(
+        select(ScholarPublication).where(ScholarPublication.publication_id == dup_id)
+    )
+    dup_links = dup_links_result.scalars().all()
+
+    winner_profiles_result = await db_session.execute(
+        select(ScholarPublication.scholar_profile_id).where(
+            ScholarPublication.publication_id == winner_id
+        )
+    )
+    winner_profiles: set[int] = {row for (row,) in winner_profiles_result}
+
+    for link in dup_links:
+        if link.scholar_profile_id in winner_profiles:
+            await db_session.delete(link)
+        else:
+            link.publication_id = winner_id
+
+
+async def sweep_identifier_duplicates(db_session: AsyncSession) -> int:
+    """Find publications sharing an identifier and merge duplicates into the winner.
+
+    Returns the number of duplicate publications removed.
+    """
+    pairs = await find_identifier_duplicate_pairs(db_session)
+    if not pairs:
+        return 0
+
+    # Deduplicate the pairs — a dup may appear multiple times if it shares
+    # several identifiers with the winner; process each dup only once.
+    processed_dups: set[int] = set()
+    for winner_id, dup_id in pairs:
+        if dup_id in processed_dups:
+            continue
+        processed_dups.add(dup_id)
+        await merge_duplicate_publication(db_session, winner_id=winner_id, dup_id=dup_id)
+
+    await db_session.flush()
+    return len(processed_dups)

app/services/domains/publications/listing.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from datetime import datetime
+
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.services.domains.publication_identifiers import application as identifier_service
@@ -9,7 +11,7 @@ from app.services.domains.publications.modes import (
     resolve_publication_view_mode,
 )
 from app.services.domains.publications.queries import (
-    get_latest_completed_run_id_for_user,
+    get_latest_run_id_for_user,
     get_publication_item_for_user,
     publication_list_item_from_row,
     publications_query,
@@ -25,11 +27,15 @@ async def list_for_user(
     mode: str = MODE_ALL,
     scholar_profile_id: int | None = None,
     favorite_only: bool = False,
-    limit: int = 300,
+    search: str | None = None,
+    sort_by: str = "first_seen",
+    sort_dir: str = "desc",
+    limit: int = 100,
     offset: int = 0,
+    snapshot_before: datetime | None = None,
 ) -> list[PublicationListItem]:
     resolved_mode = resolve_publication_view_mode(mode)
-    latest_run_id = await get_latest_completed_run_id_for_user(db_session, user_id=user_id)
+    latest_run_id = await get_latest_run_id_for_user(db_session, user_id=user_id)
     result = await db_session.execute(
         publications_query(
             user_id=user_id,
@@ -37,8 +43,12 @@ async def list_for_user(
             latest_run_id=latest_run_id,
             scholar_profile_id=scholar_profile_id,
             favorite_only=favorite_only,
+            search=search,
+            sort_by=sort_by,
+            sort_dir=sort_dir,
             limit=limit,
             offset=offset,
+            snapshot_before=snapshot_before,
         )
     )
     rows = [

app/services/domains/publications/pdf_queue.py

@@ -2,10 +2,10 @@ from __future__ import annotations
 
 import asyncio
 from dataclasses import dataclass
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 import logging
 
-from sqlalchemy import Select, func, literal, or_, select, union_all
+from sqlalchemy import Select, and_, func, literal, or_, select, union_all
 from sqlalchemy.ext.asyncio import AsyncSession
 
 from app.db.models import (
@@ -20,7 +20,6 @@ from app.db.session import get_session_factory
 from app.services.domains.publication_identifiers import application as identifier_service
 from app.services.domains.publication_identifiers.types import DisplayIdentifier
 from app.services.domains.publications.pdf_resolution_pipeline import (
-    PDF_SOURCE_SCHOLAR_PUBLICATION_PAGE,
     resolve_publication_pdf_outcome_for_row,
 )
 from app.services.domains.publications.types import PublicationListItem
@@ -49,7 +48,6 @@ _scheduled_tasks: set[asyncio.Task[None]] = set()
 class PdfQueueListItem:
     publication_id: int
     title: str
-    doi: str | None
     pdf_url: str | None
     status: str
     attempt_count: int
@@ -114,7 +112,6 @@ def _item_from_row_and_job(
         citation_count=row.citation_count,
         venue_text=row.venue_text,
         pub_url=row.pub_url,
-        doi=row.doi,
         pdf_url=row.pdf_url,
         is_read=row.is_read,
         is_favorite=row.is_favorite,
@@ -360,7 +357,7 @@ def _failed_outcome(
 ) -> OaResolutionOutcome:
     return OaResolutionOutcome(
         publication_id=row.publication_id,
-        doi=row.doi,
+        doi=None,
         pdf_url=None,
         failure_reason=FAILURE_RESOLUTION_EXCEPTION,
         source=None,
@@ -372,10 +369,12 @@ async def _fetch_outcome_for_row(
     *,
     row: PublicationListItem,
     request_email: str | None,
+    openalex_api_key: str | None = None,
 ) -> OaResolutionOutcome:
     pipeline_result = await resolve_publication_pdf_outcome_for_row(
         row=row,
         request_email=request_email,
+        openalex_api_key=openalex_api_key,
     )
     outcome = pipeline_result.outcome
     if outcome is not None:
@@ -386,11 +385,8 @@ async def _fetch_outcome_for_row(
 def _apply_publication_update(
     publication: Publication,
     *,
-    doi: str | None,
     pdf_url: str | None,
 ) -> None:
-    if doi and publication.doi != doi:
-        publication.doi = doi
     if pdf_url and publication.pdf_url != pdf_url:
         publication.pdf_url = pdf_url
 
@@ -426,7 +422,7 @@ async def _persist_outcome(
         job = await db_session.get(PublicationPdfJob, publication_id)
         if publication is None or job is None:
             return
-        _apply_publication_update(publication, doi=outcome.doi, pdf_url=outcome.pdf_url)
+        _apply_publication_update(publication, pdf_url=outcome.pdf_url)
         await identifier_service.sync_identifiers_for_publication_resolution(
             db_session,
             publication=publication,
@@ -453,10 +449,26 @@ async def _resolve_publication_row(
     user_id: int,
     request_email: str | None,
     row: PublicationListItem,
+    openalex_api_key: str | None = None,
 ) -> None:
+    from app.services.domains.openalex.client import OpenAlexBudgetExhaustedError
+    from app.services.domains.arxiv.application import ArxivRateLimitError
     await _mark_attempt_started(publication_id=row.publication_id, user_id=user_id)
     try:
-        outcome = await _fetch_outcome_for_row(row=row, request_email=request_email)
+        outcome = await _fetch_outcome_for_row(
+            row=row,
+            request_email=request_email,
+            openalex_api_key=openalex_api_key,
+        )
+    except (OpenAlexBudgetExhaustedError, ArxivRateLimitError):
+        # Persist a terminal outcome so jobs do not remain stuck in "running".
+        await _persist_outcome(
+            publication_id=row.publication_id,
+            user_id=user_id,
+            outcome=_failed_outcome(row=row),
+        )
+        # Propagate upward so the batch loop can stop immediately.
+        raise
     except Exception as exc:  # pragma: no cover - defensive network boundary
         logger.warning(
             "publications.pdf_queue.resolve_failed",
@@ -480,12 +492,44 @@ async def _run_resolution_task(
     request_email: str | None,
     rows: list[PublicationListItem],
 ) -> None:
+    from app.services.domains.openalex.client import OpenAlexBudgetExhaustedError
+    from app.services.domains.arxiv.application import ArxivRateLimitError
+    from app.services.domains.settings import application as user_settings_service
+
+    # Resolve the best available API key: per-user setting → env var fallback.
+    openalex_api_key: str | None = None
+    try:
+        session_factory = get_session_factory()
+        async with session_factory() as key_session:
+            user_settings = await user_settings_service.get_or_create_settings(key_session, user_id=user_id)
+            openalex_api_key = getattr(user_settings, "openalex_api_key", None) or settings.openalex_api_key
+    except Exception:
+        openalex_api_key = settings.openalex_api_key
+
     for row in rows:
+        try:
             await _resolve_publication_row(
                 user_id=user_id,
                 request_email=request_email,
                 row=row,
+                openalex_api_key=openalex_api_key,
             )
+        except OpenAlexBudgetExhaustedError:
+            logger.warning(
+                "publications.pdf_queue.budget_exhausted",
+                extra={"event": "publications.pdf_queue.budget_exhausted",
+                       "detail": "Stopping PDF resolution batch — OpenAlex daily budget exhausted"},
+            )
+            break
+        except ArxivRateLimitError:
+            logger.warning(
+                "publications.pdf_queue.arxiv_rate_limited",
+                extra={
+                    "event": "publications.pdf_queue.arxiv_rate_limited",
+                    "detail": "Stopping PDF resolution batch — arXiv rate limit hit (429)",
+                },
+            )
+            break
 
 
 def _schedule_rows(
@@ -571,7 +615,6 @@ def _retry_item_from_publication(
         citation_count=int(publication.citation_count or 0),
         venue_text=publication.venue_text,
         pub_url=publication.pub_url,
-        doi=publication.doi,
         pdf_url=publication.pdf_url,
         is_read=is_read,
         first_seen_at=first_seen_at,
@@ -629,13 +672,12 @@ def _queue_candidate_from_publication(publication: Publication) -> PublicationLi
     return PublicationListItem(
         publication_id=int(publication.id),
         scholar_profile_id=0,
-        scholar_label="admin",
+        scholar_label="",
         title=publication.title_raw,
         year=publication.year,
         citation_count=int(publication.citation_count or 0),
         venue_text=publication.venue_text,
         pub_url=publication.pub_url,
-        doi=publication.doi,
         pdf_url=publication.pdf_url,
         is_read=True,
         first_seen_at=publication.created_at or _utcnow(),
@@ -649,6 +691,9 @@ async def _missing_pdf_candidates(
     limit: int,
 ) -> list[PublicationListItem]:
     bounded_limit = max(1, min(int(limit), 5000))
+    now = datetime.now(timezone.utc)
+    cooldown_threshold = now - timedelta(days=7)
+
     result = await db_session.execute(
         select(Publication)
         .outerjoin(PublicationPdfJob, PublicationPdfJob.publication_id == Publication.id)
@@ -656,7 +701,13 @@ async def _missing_pdf_candidates(
         .where(
             or_(
                 PublicationPdfJob.publication_id.is_(None),
+                and_(
                     PublicationPdfJob.status.notin_([PDF_STATUS_QUEUED, PDF_STATUS_RUNNING]),
+                    or_(
+                        PublicationPdfJob.last_attempt_at.is_(None),
+                        PublicationPdfJob.last_attempt_at < cooldown_threshold,
+                    ),
+                ),
             )
         )
         .order_by(Publication.updated_at.desc(), Publication.id.desc())
@@ -694,7 +745,6 @@ def _tracked_queue_select_base(*, status: str | None) -> Select[tuple]:
         select(
             PublicationPdfJob.publication_id,
             Publication.title_raw,
-            Publication.doi,
             Publication.pdf_url,
             PublicationPdfJob.status,
             PublicationPdfJob.attempt_count,
@@ -730,7 +780,6 @@ def _untracked_queue_select_base() -> Select[tuple]:
         select(
             Publication.id,
             Publication.title_raw,
-            Publication.doi,
             Publication.pdf_url,
             literal(PDF_STATUS_UNTRACKED),
             literal(0),
@@ -793,19 +842,18 @@ def _queue_item_from_row(row: tuple) -> PdfQueueListItem:
     return PdfQueueListItem(
         publication_id=int(row[0]),
         title=str(row[1] or ""),
-        doi=row[2],
+        pdf_url=row[2],
-        pdf_url=row[3],
+        status=str(row[3] or PDF_STATUS_UNTRACKED),
-        status=str(row[4] or PDF_STATUS_UNTRACKED),
+        attempt_count=int(row[4] or 0),
-        attempt_count=int(row[5] or 0),
+        last_failure_reason=row[5],
-        last_failure_reason=row[6],
+        last_failure_detail=row[6],
-        last_failure_detail=row[7],
+        last_source=row[7],
-        last_source=row[8],
+        requested_by_user_id=int(row[8]) if row[8] is not None else None,
-        requested_by_user_id=int(row[9]) if row[9] is not None else None,
+        requested_by_email=row[9],
-        requested_by_email=row[10],
+        queued_at=row[10],
-        queued_at=row[11],
+        last_attempt_at=row[11],
-        last_attempt_at=row[12],
+        resolved_at=row[12],
-        resolved_at=row[13],
+        updated_at=row[13],
-        updated_at=row[14],
     )
 
 
@@ -902,3 +950,25 @@ async def list_pdf_queue_page(
         limit=bounded_limit,
         offset=bounded_offset,
     )
+
+
+async def drain_ready_jobs(
+    db_session: AsyncSession,
+    *,
+    limit: int,
+    max_attempts: int,
+) -> int:
+    result = await db_session.execute(
+        select(User.id).where(User.is_active.is_(True)).order_by(User.id.asc()).limit(1)
+    )
+    system_user_id = result.scalar_one_or_none()
+    if system_user_id is None:
+        return 0
+
+    bulk_result = await enqueue_all_missing_pdf_jobs(
+        db_session,
+        user_id=system_user_id,
+        request_email=settings.unpaywall_email,
+        limit=limit,
+    )
+    return bulk_result.queued_count

app/services/domains/publications/pdf_resolution_pipeline.py

@@ -2,100 +2,114 @@ from __future__ import annotations
 
 from dataclasses import dataclass
 import logging
-from urllib.parse import urlparse
+from typing import Any
-
-import httpx
 
+from app.services.domains.arxiv.application import ArxivRateLimitError
+from app.services.domains.openalex.client import OpenAlexBudgetExhaustedError
 from app.services.domains.publications.types import PublicationListItem
-from app.services.domains.scholar.publication_pdf import (
-    ScholarPublicationLinkCandidate,
-    ScholarPublicationLinkCandidates,
-    fetch_link_candidates_from_scholar_publication_page,
-)
-from app.services.domains.unpaywall import pdf_discovery as pdf_discovery_service
 from app.services.domains.unpaywall.application import OaResolutionOutcome, resolve_publication_oa_outcomes
 from app.settings import settings
 
 logger = logging.getLogger(__name__)
 
-PDF_SOURCE_SCHOLAR_PUBLICATION_PAGE = "scholar_publication_page"
-PDF_SOURCE_SCHOLAR_PUBLICATION_PAGE_UNLABELED = "scholar_publication_page_unlabeled_fallback"
-PDF_PATH_TOKEN = "/pdf/"
-HTTP_TIMEOUT_FLOOR_SECONDS = 0.5
-
 
 @dataclass(frozen=True)
 class PipelineOutcome:
     outcome: OaResolutionOutcome | None
-    scholar_candidates: ScholarPublicationLinkCandidates | None
+    scholar_candidates: Any | None  # Kept for backward compatibility with calling signatures
 
 
 async def resolve_publication_pdf_outcome_for_row(
     *,
     row: PublicationListItem,
     request_email: str | None,
+    openalex_api_key: str | None = None,
 ) -> PipelineOutcome:
-    candidates = await _safe_scholar_candidates(row.pub_url)
+    # 1. OpenAlex OA — raises OpenAlexBudgetExhaustedError if budget is gone
-    labeled = _labeled_candidate(candidates)
+    openalex_outcome = await _openalex_outcome(row, request_email=request_email, openalex_api_key=openalex_api_key)
-    if labeled is not None:
+    if openalex_outcome and openalex_outcome.pdf_url:
-        return PipelineOutcome(_scholar_outcome(row=row, candidate=labeled), candidates)
+        return PipelineOutcome(openalex_outcome, None)
+
+    # 2. arXiv
+    arxiv_outcome = await _arxiv_outcome(row, request_email=request_email)
+    if arxiv_outcome and arxiv_outcome.pdf_url:
+        return PipelineOutcome(arxiv_outcome, None)
+
+    # 3. Unpaywall (which falls back to Crossref)
     oa_outcome = await _oa_outcome(row=row, request_email=request_email)
-    if _oa_has_pdf(oa_outcome):
+    return PipelineOutcome(oa_outcome, None)
-        return PipelineOutcome(oa_outcome, candidates)
-    unlabeled = _unlabeled_candidate(candidates)
-    if unlabeled is None:
-        return PipelineOutcome(oa_outcome, candidates)
-    fallback_outcome = await _unlabeled_fallback_outcome(row=row, candidate=unlabeled)
-    if fallback_outcome is not None:
-        return PipelineOutcome(fallback_outcome, candidates)
-    return PipelineOutcome(oa_outcome, candidates)
 
 
-async def _safe_scholar_candidates(pub_url: str | None) -> ScholarPublicationLinkCandidates | None:
+async def _openalex_outcome(
-    try:
-        return await fetch_link_candidates_from_scholar_publication_page(pub_url)
-    except Exception as exc:  # pragma: no cover - defensive boundary
-        logger.warning(
-            "publications.pdf_resolution.scholar_candidates_failed",
-            extra={"event": "publications.pdf_resolution.scholar_candidates_failed", "error": str(exc)},
-        )
-        return None
-
-
-def _labeled_candidate(
-    candidates: ScholarPublicationLinkCandidates | None,
-) -> ScholarPublicationLinkCandidate | None:
-    if candidates is None:
-        return None
-    return candidates.labeled_candidate
-
-
-def _unlabeled_candidate(
-    candidates: ScholarPublicationLinkCandidates | None,
-) -> ScholarPublicationLinkCandidate | None:
-    if candidates is None:
-        return None
-    return candidates.fallback_candidate
-
-
-def _scholar_outcome(
-    *,
     row: PublicationListItem,
-    candidate: ScholarPublicationLinkCandidate,
+    request_email: str | None,
-) -> OaResolutionOutcome:
+    openalex_api_key: str | None = None,
-    source = (
+) -> OaResolutionOutcome | None:
-        PDF_SOURCE_SCHOLAR_PUBLICATION_PAGE
+    from app.services.domains.openalex.client import OpenAlexClient
-        if candidate.label_present
+    from app.services.domains.openalex.matching import find_best_match
-        else PDF_SOURCE_SCHOLAR_PUBLICATION_PAGE_UNLABELED
+
+    if not row.title:
+        return None
+
+    import re
+    safe_title = re.sub(r"[^\w\s]", " ", row.title)
+    safe_title = " ".join(safe_title.split())
+    if not safe_title:
+        return None
+
+    api_key = openalex_api_key or settings.openalex_api_key
+    client = OpenAlexClient(api_key=api_key, mailto=request_email or settings.crossref_api_mailto)
+    try:
+        openalex_works = await client.get_works_by_filter({"title.search": safe_title}, limit=5)
+        match = find_best_match(
+            target_title=row.title,
+            target_year=row.year,
+            target_authors=row.scholar_label,
+            candidates=openalex_works,
         )
+        if match and match.oa_url:
             return OaResolutionOutcome(
                 publication_id=row.publication_id,
-        doi=row.doi,
+                doi=match.doi,
-        pdf_url=candidate.url,
+                pdf_url=match.oa_url,
                 failure_reason=None,
-        source=source,
+                source="openalex",
                 used_crossref=False,
             )
+    except OpenAlexBudgetExhaustedError:
+        # Re-raise so the caller's batch loop can stop hitting the API.
+        raise
+    except Exception as exc:
+        logger.warning(
+            "publications.pdf_resolution.openalex_failed",
+            extra={"event": "publications.pdf_resolution.openalex_failed", "error": str(exc)},
+        )
+    return None
+
+
+async def _arxiv_outcome(row: PublicationListItem, request_email: str | None) -> OaResolutionOutcome | None:
+    from app.services.domains.arxiv.application import discover_arxiv_id_for_publication
+
+    try:
+        arxiv_id = await discover_arxiv_id_for_publication(item=row, request_email=request_email)
+        if arxiv_id:
+            pdf_url = f"https://arxiv.org/pdf/{arxiv_id}.pdf"
+            return OaResolutionOutcome(
+                publication_id=row.publication_id,
+                doi=None,
+                pdf_url=pdf_url,
+                failure_reason=None,
+                source="arxiv",
+                used_crossref=False,
+            )
+    except ArxivRateLimitError:
+        raise  # propagate so the batch loop can stop
+    except Exception as exc:
+        logger.warning(
+            "publications.pdf_resolution.arxiv_failed",
+            extra={"event": "publications.pdf_resolution.arxiv_failed", "error": str(exc)},
+        )
+    return None
 
 
 async def _oa_outcome(
@@ -105,46 +119,3 @@ async def _oa_outcome(
 ) -> OaResolutionOutcome | None:
     outcomes = await resolve_publication_oa_outcomes([row], request_email=request_email)
     return outcomes.get(row.publication_id)
-
-
-def _oa_has_pdf(outcome: OaResolutionOutcome | None) -> bool:
-    return bool(outcome and outcome.pdf_url)
-
-
-async def _unlabeled_fallback_outcome(
-    *,
-    row: PublicationListItem,
-    candidate: ScholarPublicationLinkCandidate,
-) -> OaResolutionOutcome | None:
-    pdf_url = await _validated_pdf_url(candidate.url)
-    if pdf_url is None:
-        return None
-    return _scholar_outcome(row=row, candidate=ScholarPublicationLinkCandidate(
-        url=pdf_url,
-        confidence_score=candidate.confidence_score,
-        label_present=False,
-        reason=candidate.reason,
-    ))
-
-
-async def _validated_pdf_url(candidate_url: str) -> str | None:
-    if _looks_direct_pdf(candidate_url):
-        return candidate_url
-    timeout_seconds = _discovery_timeout_seconds()
-    async with httpx.AsyncClient(timeout=timeout_seconds) as client:
-        if await pdf_discovery_service._candidate_is_pdf(client, candidate_url=candidate_url):
-            return candidate_url
-        return await pdf_discovery_service.resolve_pdf_from_landing_page(client, page_url=candidate_url)
-
-
-def _looks_direct_pdf(url: str | None) -> bool:
-    if pdf_discovery_service.looks_like_pdf_url(url):
-        return True
-    if not isinstance(url, str):
-        return False
-    path = (urlparse(url).path or "").lower()
-    return PDF_PATH_TOKEN in path
-
-
-def _discovery_timeout_seconds() -> float:
-    return max(float(settings.unpaywall_timeout_seconds), HTTP_TIMEOUT_FLOOR_SECONDS)

app/services/domains/publications/queries.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from datetime import datetime
+
 from sqlalchemy import Select, func, select
 from sqlalchemy.ext.asyncio import AsyncSession
 
@@ -15,15 +17,24 @@ def _normalized_citation_count(value: object) -> int:
         return 0
 
 
-async def get_latest_completed_run_id_for_user(
+async def get_latest_run_id_for_user(
     db_session: AsyncSession,
     *,
     user_id: int,
 ) -> int | None:
+    # We include RUNNING and RESOLVING statuses so that the "New" tab shows
+    # results in real-time as they are discovered.
     result = await db_session.execute(
         select(func.max(CrawlRun.id)).where(
             CrawlRun.user_id == user_id,
-            CrawlRun.status != RunStatus.RUNNING,
+            CrawlRun.status.in_(
+                [
+                    RunStatus.RUNNING,
+                    RunStatus.RESOLVING,
+                    RunStatus.SUCCESS,
+                    RunStatus.PARTIAL_FAILURE,
+                ]
+            ),
         )
     )
     latest_run_id = result.scalar_one_or_none()
@@ -39,7 +50,19 @@ def publications_query(
     favorite_only: bool,
     limit: int,
     offset: int = 0,
+    search: str | None = None,
+    sort_by: str = "first_seen",
+    sort_dir: str = "desc",
+    snapshot_before: datetime | None = None,
 ) -> Select[tuple]:
+    _SORT_COLUMNS = {
+        "first_seen": ScholarPublication.created_at,
+        "title": Publication.title_raw,
+        "year": Publication.year,
+        "citations": Publication.citation_count,
+        "scholar": ScholarProfile.display_name,
+    }
+
     scholar_label = ScholarProfile.display_name
     stmt = (
         select(
@@ -52,7 +75,6 @@ def publications_query(
             Publication.citation_count,
             Publication.venue_text,
             Publication.pub_url,
-            Publication.doi,
             Publication.pdf_url,
             ScholarPublication.is_read,
             ScholarPublication.is_favorite,
@@ -62,9 +84,14 @@ def publications_query(
         .join(ScholarPublication, ScholarPublication.publication_id == Publication.id)
         .join(ScholarProfile, ScholarProfile.id == ScholarPublication.scholar_profile_id)
         .where(ScholarProfile.user_id == user_id)
-        .order_by(ScholarPublication.created_at.desc(), Publication.id.desc())
+    )
-        .offset(max(int(offset), 0))
+    if search:
-        .limit(limit)
+        safe_search = search.replace("%", r"\%").replace("_", r"\_")
+        pat = f"%{safe_search}%"
+        stmt = stmt.where(
+            Publication.title_raw.ilike(pat)
+            | ScholarProfile.display_name.ilike(pat)
+            | Publication.venue_text.ilike(pat)
         )
     if scholar_profile_id is not None:
         stmt = stmt.where(ScholarProfile.id == scholar_profile_id)
@@ -76,6 +103,16 @@ def publications_query(
         if latest_run_id is None:
             return stmt.where(False)
         stmt = stmt.where(ScholarPublication.first_seen_run_id == latest_run_id)
+    if snapshot_before is not None:
+        stmt = stmt.where(ScholarPublication.created_at <= snapshot_before)
+
+    sort_col = _SORT_COLUMNS.get(sort_by, ScholarPublication.created_at)
+    order = sort_col.desc() if sort_dir == "desc" else sort_col.asc()
+    stmt = stmt.order_by(order, Publication.id.desc())
+
+    if limit is not None:
+        stmt = stmt.offset(max(int(offset), 0)).limit(limit)
+
     return stmt
 
 
@@ -96,7 +133,6 @@ def publication_query_for_user(
             Publication.citation_count,
             Publication.venue_text,
             Publication.pub_url,
-            Publication.doi,
             Publication.pdf_url,
             ScholarPublication.is_read,
             ScholarPublication.is_favorite,
@@ -121,7 +157,7 @@ async def get_publication_item_for_user(
     scholar_profile_id: int,
     publication_id: int,
 ) -> PublicationListItem | None:
-    latest_run_id = await get_latest_completed_run_id_for_user(db_session, user_id=user_id)
+    latest_run_id = await get_latest_run_id_for_user(db_session, user_id=user_id)
     result = await db_session.execute(
         publication_query_for_user(
             user_id=user_id,
@@ -150,7 +186,6 @@ def publication_list_item_from_row(
         citation_count,
         venue_text,
         pub_url,
-        doi,
         pdf_url,
         is_read,
         is_favorite,
@@ -166,7 +201,6 @@ def publication_list_item_from_row(
         citation_count=_normalized_citation_count(citation_count),
         venue_text=venue_text,
         pub_url=pub_url,
-        doi=doi,
         pdf_url=pdf_url,
         is_read=bool(is_read),
         is_favorite=bool(is_favorite),
@@ -188,7 +222,6 @@ def unread_item_from_row(row: tuple) -> UnreadPublicationItem:
         citation_count,
         venue_text,
         pub_url,
-        doi,
         pdf_url,
         _is_read,
         _is_favorite,
@@ -204,6 +237,5 @@ def unread_item_from_row(row: tuple) -> UnreadPublicationItem:
         citation_count=_normalized_citation_count(citation_count),
         venue_text=venue_text,
         pub_url=pub_url,
-        doi=doi,
         pdf_url=pdf_url,
     )

app/services/domains/publications/types.py

@@ -16,7 +16,6 @@ class PublicationListItem:
     citation_count: int
     venue_text: str | None
     pub_url: str | None
-    doi: str | None
     pdf_url: str | None
     is_read: bool
     first_seen_at: datetime
@@ -39,5 +38,4 @@ class UnreadPublicationItem:
     citation_count: int
     venue_text: str | None
     pub_url: str | None
-    doi: str | None
     pdf_url: str | None

app/services/domains/runs/events.py

@@ -0,0 +1,59 @@
+import asyncio
+import json
+import logging
+from typing import Any, AsyncGenerator, Dict, Set
+
+logger = logging.getLogger(__name__)
+
+class RunEventPublisher:
+    def __init__(self) -> None:
+        # Maps run_id to a set of subscriber queues
+        self._subscribers: Dict[int, Set[asyncio.Queue]] = {}
+
+    def subscribe(self, run_id: int) -> asyncio.Queue:
+        if run_id not in self._subscribers:
+            self._subscribers[run_id] = set()
+        queue: asyncio.Queue[Any] = asyncio.Queue()
+        self._subscribers[run_id].add(queue)
+        logger.debug(f"New subscriber for run {run_id}. Total: {len(self._subscribers[run_id])}")
+        return queue
+
+    def unsubscribe(self, run_id: int, queue: asyncio.Queue) -> None:
+        if run_id in self._subscribers:
+            self._subscribers[run_id].discard(queue)
+            if not self._subscribers[run_id]:
+                self._subscribers.pop(run_id, None)
+
+    async def publish(self, run_id: int, event_type: str, data: dict[str, Any]) -> None:
+        if run_id not in self._subscribers:
+            return
+        
+        message = {
+            "type": event_type,
+            "data": data
+        }
+        
+        # Fan-out to all active subscribers for this run
+        for queue in list(self._subscribers[run_id]):
+            try:
+                queue.put_nowait(message)
+            except asyncio.QueueFull:
+                logger.warning(f"Subscriber queue full for run {run_id}, dropping message")
+
+run_events = RunEventPublisher()
+
+async def event_generator(run_id: int) -> AsyncGenerator[str, None]:
+    queue = run_events.subscribe(run_id)
+    try:
+        while True:
+            # Wait for a new event
+            message = await queue.get()
+            # Server-Sent Events format: "event: <type>\ndata: <json>\n\n"
+            event_type = message["type"]
+            data_str = json.dumps(message["data"])
+            yield f"event: {event_type}\ndata: {data_str}\n\n"
+    except asyncio.CancelledError:
+        logger.debug(f"Client disconnected from SSE stream for run {run_id}")
+        raise
+    finally:
+        run_events.unsubscribe(run_id, queue)

app/services/domains/scholar/publication_pdf.py

@@ -1,232 +0,0 @@
-from __future__ import annotations
-
-from dataclasses import dataclass
-from html.parser import HTMLParser
-from urllib.parse import parse_qs, urlparse
-
-from app.services.domains.scholar.parser_types import ScholarDomInvariantError
-from app.services.domains.scholar.parser_utils import attr_href, normalize_space
-from app.services.domains.scholar.rate_limit import wait_for_scholar_slot
-from app.services.domains.scholar.source import LiveScholarSource
-from app.services.domains.settings.application import resolve_request_delay_minimum
-from app.settings import settings
-
-CONTAINER_ID = "gsc_oci_title_gg"
-PDF_LABEL_TOKEN = "[pdf]"
-SCHOLAR_PDF_LABELED_CONFIDENCE = 0.98
-SCHOLAR_PDF_UNLABELED_CONFIDENCE = 0.2
-ALLOWED_URL_SCHEMES = frozenset({"http", "https"})
-
-
-@dataclass(frozen=True)
-class ScholarPublicationLinkCandidate:
-    url: str
-    confidence_score: float
-    label_present: bool
-    reason: str
-
-
-@dataclass(frozen=True)
-class ScholarPublicationLinkCandidates:
-    container_seen: bool
-    labeled_candidate: ScholarPublicationLinkCandidate | None
-    fallback_candidate: ScholarPublicationLinkCandidate | None
-    warnings: tuple[str, ...] = ()
-
-
-@dataclass(frozen=True)
-class _ParsedAnchor:
-    href: str | None
-    text: str
-
-
-class _ScholarPublicationPdfParser(HTMLParser):
-    def __init__(self) -> None:
-        super().__init__(convert_charrefs=True)
-        self.container_seen = False
-        self.anchors: list[_ParsedAnchor] = []
-        self._container_depth = 0
-        self._anchor_depth = 0
-        self._anchor_href: str | None = None
-        self._anchor_parts: list[str] = []
-
-    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
-        self._increment_depths()
-        if self._starts_container(tag, attrs):
-            self.container_seen = True
-            self._container_depth = 1
-            return
-        if self._container_depth <= 0 or tag != "a":
-            return
-        if self._anchor_depth > 0:
-            return
-        self._anchor_href = attr_href(attrs)
-        self._anchor_parts = []
-        self._anchor_depth = 1
-
-    def handle_data(self, data: str) -> None:
-        if self._anchor_depth > 0:
-            self._anchor_parts.append(data)
-
-    def handle_endtag(self, tag: str) -> None:
-        if self._anchor_depth > 0:
-            self._anchor_depth -= 1
-            if self._anchor_depth == 0:
-                self._finish_anchor()
-        if self._container_depth > 0:
-            self._container_depth -= 1
-
-    def _increment_depths(self) -> None:
-        if self._container_depth > 0:
-            self._container_depth += 1
-        if self._anchor_depth > 0:
-            self._anchor_depth += 1
-
-    def _starts_container(self, tag: str, attrs: list[tuple[str, str | None]]) -> bool:
-        if tag != "div":
-            return False
-        attrs_map = {name.lower(): (value or "") for name, value in attrs}
-        return attrs_map.get("id") == CONTAINER_ID
-
-    def _finish_anchor(self) -> None:
-        self.anchors.append(
-            _ParsedAnchor(
-                href=self._anchor_href,
-                text=normalize_space("".join(self._anchor_parts)),
-            )
-        )
-        self._anchor_href = None
-        self._anchor_parts = []
-
-
-def is_scholar_publication_detail_url(url: str | None) -> bool:
-    if not isinstance(url, str) or not url.strip():
-        return False
-    parsed = urlparse(url)
-    if parsed.scheme not in ALLOWED_URL_SCHEMES:
-        return False
-    if parsed.netloc.lower() != "scholar.google.com":
-        return False
-    query = parse_qs(parsed.query)
-    return _has_view_citation_params(query)
-
-
-def _has_view_citation_params(query: dict[str, list[str]]) -> bool:
-    view_op = (query.get("view_op") or [""])[0]
-    citation = (query.get("citation_for_view") or [""])[0].strip()
-    return view_op == "view_citation" and bool(citation)
-
-
-def extract_link_candidates_from_publication_detail_html(html: str) -> ScholarPublicationLinkCandidates:
-    parser = _parsed_publication_detail(html)
-    if not parser.container_seen:
-        return ScholarPublicationLinkCandidates(False, None, None)
-    anchors = _validated_container_anchors(parser.anchors)
-    labeled = _select_labeled_candidate(anchors)
-    fallback = _select_fallback_candidate(anchors, labeled=labeled)
-    warnings = _candidate_warnings(labeled=labeled, fallback=fallback)
-    return ScholarPublicationLinkCandidates(True, labeled, fallback, warnings)
-
-
-def _parsed_publication_detail(html: str) -> _ScholarPublicationPdfParser:
-    parser = _ScholarPublicationPdfParser()
-    parser.feed(html)
-    parser.close()
-    return parser
-
-
-def _validated_container_anchors(anchors: list[_ParsedAnchor]) -> list[_ParsedAnchor]:
-    if not anchors:
-        raise ScholarDomInvariantError(
-            code="layout_publication_link_container_missing_anchor",
-            message="Scholar publication link container was present without an anchor.",
-        )
-    validated: list[_ParsedAnchor] = []
-    for anchor in anchors:
-        validated.append(_validated_anchor(anchor))
-    return validated
-
-
-def _validated_anchor(anchor: _ParsedAnchor) -> _ParsedAnchor:
-    href = (anchor.href or "").strip()
-    if not href:
-        raise ScholarDomInvariantError(
-            code="layout_publication_link_missing_href",
-            message="Scholar publication link container anchor was missing href.",
-        )
-    parsed = urlparse(href)
-    if parsed.scheme not in ALLOWED_URL_SCHEMES:
-        raise ScholarDomInvariantError(
-            code="layout_publication_link_invalid_scheme",
-            message="Scholar publication link used a non-http URL.",
-        )
-    return _ParsedAnchor(href=href, text=anchor.text)
-
-
-def _select_labeled_candidate(anchors: list[_ParsedAnchor]) -> ScholarPublicationLinkCandidate | None:
-    for anchor in anchors:
-        if PDF_LABEL_TOKEN in anchor.text.lower():
-            return ScholarPublicationLinkCandidate(
-                url=str(anchor.href),
-                confidence_score=SCHOLAR_PDF_LABELED_CONFIDENCE,
-                label_present=True,
-                reason="scholar_link_labeled_pdf",
-            )
-    return None
-
-
-def _select_fallback_candidate(
-    anchors: list[_ParsedAnchor],
-    *,
-    labeled: ScholarPublicationLinkCandidate | None,
-) -> ScholarPublicationLinkCandidate | None:
-    for anchor in anchors:
-        if labeled and anchor.href == labeled.url:
-            continue
-        return ScholarPublicationLinkCandidate(
-            url=str(anchor.href),
-            confidence_score=SCHOLAR_PDF_UNLABELED_CONFIDENCE,
-            label_present=False,
-            reason="scholar_link_unlabeled_fallback",
-        )
-    if labeled is None and anchors:
-        anchor = anchors[0]
-        return ScholarPublicationLinkCandidate(
-            url=str(anchor.href),
-            confidence_score=SCHOLAR_PDF_UNLABELED_CONFIDENCE,
-            label_present=False,
-            reason="scholar_link_unlabeled_fallback",
-        )
-    return None
-
-
-def _candidate_warnings(
-    *,
-    labeled: ScholarPublicationLinkCandidate | None,
-    fallback: ScholarPublicationLinkCandidate | None,
-) -> tuple[str, ...]:
-    warnings: list[str] = []
-    if labeled is None and fallback is not None:
-        warnings.append("scholar_publication_link_unlabeled_only")
-    return tuple(warnings)
-
-
-def _scholar_request_delay_seconds() -> int:
-    return resolve_request_delay_minimum(settings.ingestion_min_request_delay_seconds)
-
-
-def _fetch_succeeded(fetch_result) -> bool:
-    return int(fetch_result.status_code or 0) == 200 and not fetch_result.error
-
-
-async def fetch_link_candidates_from_scholar_publication_page(
-    publication_url: str | None,
-) -> ScholarPublicationLinkCandidates | None:
-    if not is_scholar_publication_detail_url(publication_url):
-        return None
-    await wait_for_scholar_slot(min_interval_seconds=float(_scholar_request_delay_seconds()))
-    source = LiveScholarSource()
-    fetch_result = await source.fetch_publication_html(str(publication_url))
-    if not _fetch_succeeded(fetch_result):
-        return None
-    return extract_link_candidates_from_publication_detail_html(fetch_result.body)

app/services/domains/settings/application.py

@@ -109,6 +109,8 @@ def parse_nav_visible_pages(value: object) -> list[str]:
     return deduped
 
 
+from app.settings import settings as app_settings
+
 async def get_or_create_settings(
     db_session: AsyncSession,
     *,
@@ -121,7 +123,12 @@ async def get_or_create_settings(
     if settings is not None:
         return settings
 
-    settings = UserSetting(user_id=user_id)
+    settings = UserSetting(
+        user_id=user_id,
+        openalex_api_key=app_settings.openalex_api_key,
+        crossref_api_token=app_settings.crossref_api_token,
+        crossref_api_mailto=app_settings.crossref_api_mailto,
+    )
     db_session.add(settings)
     await db_session.commit()
     await db_session.refresh(settings)
@@ -136,11 +143,17 @@ async def update_settings(
     run_interval_minutes: int,
     request_delay_seconds: int,
     nav_visible_pages: list[str],
+    openalex_api_key: str | None,
+    crossref_api_token: str | None,
+    crossref_api_mailto: str | None,
 ) -> UserSetting:
     settings.auto_run_enabled = auto_run_enabled
     settings.run_interval_minutes = run_interval_minutes
     settings.request_delay_seconds = request_delay_seconds
     settings.nav_visible_pages = nav_visible_pages
+    settings.openalex_api_key = openalex_api_key
+    settings.crossref_api_token = crossref_api_token
+    settings.crossref_api_mailto = crossref_api_mailto
     await db_session.commit()
     await db_session.refresh(settings)
     return settings

app/services/domains/unpaywall/application.py

@@ -63,7 +63,10 @@ def _extract_explicit_doi(text: str | None) -> str | None:
 
 
 def _publication_doi(item: PublicationListItem | UnreadPublicationItem) -> str | None:
-    stored = normalize_doi(item.doi)
+    stored = None
+    if getattr(item, "display_identifier", None) and item.display_identifier.kind == "doi":
+        stored = normalize_doi(item.display_identifier.value)
+    
     explicit_doi = (
         _extract_explicit_doi(item.pub_url)
         or _extract_explicit_doi(item.venue_text)
@@ -169,9 +172,11 @@ async def _fetch_unpaywall_payload_by_doi(
     email: str,
 ) -> dict | None:
     await wait_for_unpaywall_slot(min_interval_seconds=settings.unpaywall_min_interval_seconds)
+    headers = {"User-Agent": f"scholar-scraper/1.0 (mailto:{email})"}
     response = await client.get(
         UNPAYWALL_URL_TEMPLATE.format(doi=doi),
         params={"email": email},
+        headers=headers,
     )
     if response.status_code != 200:
         return None
@@ -427,7 +432,8 @@ async def resolve_publication_oa_outcomes(
 
     timeout_seconds = max(float(settings.unpaywall_timeout_seconds), 0.5)
     targets = _resolution_targets(items)[: max(int(settings.unpaywall_max_items_per_request), 0)]
-    async with httpx.AsyncClient(timeout=timeout_seconds) as client:
+    headers = {"User-Agent": f"scholar-scraper/1.0 (mailto:{email})"}
+    async with httpx.AsyncClient(timeout=timeout_seconds, follow_redirects=True, headers=headers) as client:
         outcomes = await _resolve_outcomes_with_client(
             client=client,
             targets=targets,

app/settings.py

@@ -140,6 +140,11 @@ class Settings:
         "INGESTION_RETRY_BACKOFF_SECONDS",
         1.0,
     )
+    ingestion_rate_limit_retries: int = _env_int("INGESTION_RATE_LIMIT_RETRIES", 3)
+    ingestion_rate_limit_backoff_seconds: float = _env_float(
+        "INGESTION_RATE_LIMIT_BACKOFF_SECONDS",
+        30.0,
+    )
     ingestion_max_pages_per_scholar: int = _env_int(
         "INGESTION_MAX_PAGES_PER_SCHOLAR",
         30,
@@ -182,6 +187,7 @@ class Settings:
         6,
     )
     scheduler_queue_batch_size: int = _env_int("SCHEDULER_QUEUE_BATCH_SIZE", 10)
+    scheduler_pdf_queue_batch_size: int = _env_int("SCHEDULER_PDF_QUEUE_BATCH_SIZE", 15)
     frontend_enabled: bool = _env_bool("FRONTEND_ENABLED", True)
     frontend_dist_dir: str = _env_str("FRONTEND_DIST_DIR", "/app/frontend/dist")
     scholar_image_upload_dir: str = _env_str(
@@ -241,9 +247,9 @@ class Settings:
     )
     pdf_auto_retry_first_interval_seconds: int = _env_int(
         "PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS",
-        3_600,
+        86_400,
     )
-    pdf_auto_retry_max_attempts: int = _env_int("PDF_AUTO_RETRY_MAX_ATTEMPTS", 3)
+    pdf_auto_retry_max_attempts: int = _env_int("PDF_AUTO_RETRY_MAX_ATTEMPTS", 2)
     unpaywall_pdf_discovery_enabled: bool = _env_bool("UNPAYWALL_PDF_DISCOVERY_ENABLED", True)
     unpaywall_pdf_discovery_max_candidates: int = _env_int("UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES", 5)
     unpaywall_pdf_discovery_max_html_bytes: int = _env_int("UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES", 500_000)
@@ -253,5 +259,9 @@ class Settings:
     crossref_min_interval_seconds: float = _env_float("CROSSREF_MIN_INTERVAL_SECONDS", 0.6)
     crossref_max_lookups_per_request: int = _env_int("CROSSREF_MAX_LOOKUPS_PER_REQUEST", 8)
     
+    openalex_api_key: str | None = os.getenv("OPENALEX_API_KEY")
+    crossref_api_token: str | None = os.getenv("CROSSREF_API_TOKEN")
+    crossref_api_mailto: str | None = os.getenv("CROSSREF_API_MAILTO")
+
 
 settings = Settings()

build/lib/app/__init__.py

@@ -0,0 +1 @@
+

build/lib/app/api/__init__.py

@@ -0,0 +1,2 @@
+from __future__ import annotations
+

build/lib/app/api/deps.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+from fastapi import Depends, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.errors import ApiException
+from app.auth import runtime as auth_runtime
+from app.db.models import User
+from app.db.session import get_db_session
+
+
+async def get_api_current_user(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+) -> User:
+    current_user = await auth_runtime.get_authenticated_user(request, db_session)
+    if current_user is None:
+        raise ApiException(
+            status_code=401,
+            code="auth_required",
+            message="Authentication required.",
+        )
+    return current_user
+
+
+async def get_api_admin_user(
+    current_user: User = Depends(get_api_current_user),
+) -> User:
+    if not current_user.is_admin:
+        raise ApiException(
+            status_code=403,
+            code="forbidden",
+            message="Admin access required.",
+        )
+    return current_user

build/lib/app/api/errors.py

@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.exception_handlers import (
+    http_exception_handler as fastapi_http_exception_handler,
+)
+from fastapi.exception_handlers import (
+    request_validation_exception_handler as fastapi_validation_exception_handler,
+)
+from fastapi.exceptions import RequestValidationError
+
+from app.api.responses import error_response
+
+
+def _is_api_path(path: str) -> bool:
+    return path.startswith("/api/")
+
+
+def _status_code_to_error_code(status_code: int) -> str:
+    mapping = {
+        400: "bad_request",
+        401: "auth_required",
+        403: "forbidden",
+        404: "not_found",
+        409: "conflict",
+        422: "validation_error",
+        429: "rate_limited",
+        500: "internal_error",
+    }
+    return mapping.get(status_code, "error")
+
+
+class ApiException(Exception):
+    def __init__(
+        self,
+        *,
+        status_code: int,
+        code: str,
+        message: str,
+        details: Any | None = None,
+    ) -> None:
+        super().__init__(message)
+        self.status_code = status_code
+        self.code = code
+        self.message = message
+        self.details = details
+
+
+def register_api_exception_handlers(app: FastAPI) -> None:
+    @app.exception_handler(ApiException)
+    async def _handle_api_exception(request: Request, exc: ApiException):
+        return error_response(
+            request,
+            status_code=exc.status_code,
+            code=exc.code,
+            message=exc.message,
+            details=exc.details,
+        )
+
+    @app.exception_handler(HTTPException)
+    async def _handle_http_exception(request: Request, exc: HTTPException):
+        if not _is_api_path(request.url.path):
+            return await fastapi_http_exception_handler(request, exc)
+        return error_response(
+            request,
+            status_code=exc.status_code,
+            code=_status_code_to_error_code(exc.status_code),
+            message=str(exc.detail) if exc.detail is not None else "Request failed.",
+            details=exc.detail if isinstance(exc.detail, (dict, list)) else None,
+        )
+
+    @app.exception_handler(RequestValidationError)
+    async def _handle_validation_exception(request: Request, exc: RequestValidationError):
+        if not _is_api_path(request.url.path):
+            return await fastapi_validation_exception_handler(request, exc)
+        return error_response(
+            request,
+            status_code=422,
+            code="validation_error",
+            message="Request validation failed.",
+            details=exc.errors(),
+        )
+

build/lib/app/api/media.py

@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+import mimetypes
+
+from fastapi import APIRouter, Depends
+from fastapi.responses import FileResponse
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_current_user
+from app.api.errors import ApiException
+from app.db.models import User
+from app.db.session import get_db_session
+from app.services.domains.scholars import application as scholar_service
+from app.settings import settings
+
+router = APIRouter(tags=["media"])
+
+
+@router.get("/scholar-images/{scholar_profile_id}/upload")
+async def get_uploaded_scholar_image(
+    scholar_profile_id: int,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+    if not profile.profile_image_upload_path:
+        raise ApiException(
+            status_code=404,
+            code="scholar_image_not_found",
+            message="Scholar image not found.",
+        )
+
+    try:
+        image_path = scholar_service.resolve_upload_file_path(
+            upload_dir=settings.scholar_image_upload_dir,
+            relative_path=profile.profile_image_upload_path,
+        )
+    except scholar_service.ScholarServiceError as exc:
+        raise ApiException(
+            status_code=404,
+            code="scholar_image_not_found",
+            message="Scholar image not found.",
+        ) from exc
+
+    if not image_path.exists() or not image_path.is_file():
+        raise ApiException(
+            status_code=404,
+            code="scholar_image_not_found",
+            message="Scholar image not found.",
+        )
+
+    media_type = mimetypes.guess_type(str(image_path))[0] or "application/octet-stream"
+    return FileResponse(path=image_path, media_type=media_type)

build/lib/app/api/responses.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi.encoders import jsonable_encoder
+from starlette.requests import Request
+from starlette.responses import JSONResponse
+
+
+def _request_id(request: Request) -> str | None:
+    request_state = getattr(request, "state", None)
+    if request_state is None:
+        return None
+    return getattr(request_state, "request_id", None)
+
+
+def success_payload(
+    request: Request,
+    *,
+    data: Any,
+) -> dict[str, Any]:
+    return {
+        "data": data,
+        "meta": {
+            "request_id": _request_id(request),
+        },
+    }
+
+
+def success_response(
+    request: Request,
+    *,
+    data: Any,
+    status_code: int = 200,
+) -> JSONResponse:
+    payload = success_payload(request, data=data)
+    return JSONResponse(
+        status_code=status_code,
+        content=jsonable_encoder(payload),
+    )
+
+
+def error_response(
+    request: Request,
+    *,
+    status_code: int,
+    code: str,
+    message: str,
+    details: Any | None = None,
+) -> JSONResponse:
+    payload = {
+        "error": {
+            "code": code,
+            "message": message,
+            "details": details,
+        },
+        "meta": {
+            "request_id": _request_id(request),
+        },
+    }
+    return JSONResponse(
+        status_code=status_code,
+        content=jsonable_encoder(payload),
+    )

build/lib/app/api/router.py

@@ -0,0 +1,14 @@
+from __future__ import annotations
+
+from fastapi import APIRouter
+
+from app.api.routers import admin, admin_dbops, auth, publications, runs, scholars, settings
+
+router = APIRouter(prefix="/api/v1")
+router.include_router(auth.router)
+router.include_router(admin.router)
+router.include_router(admin_dbops.router)
+router.include_router(scholars.router)
+router.include_router(settings.router)
+router.include_router(runs.router)
+router.include_router(publications.router)

build/lib/app/api/routers/__init__.py

@@ -0,0 +1,2 @@
+from __future__ import annotations
+

build/lib/app/api/routers/admin.py

@@ -0,0 +1,202 @@
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Depends, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_admin_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from app.api.schemas import (
+    AdminResetPasswordRequest,
+    AdminUserActiveUpdateRequest,
+    AdminUserCreateRequest,
+    AdminUserEnvelope,
+    AdminUsersListEnvelope,
+    MessageEnvelope,
+)
+from app.auth.deps import get_auth_service
+from app.auth.service import AuthService
+from app.db.models import User
+from app.db.session import get_db_session
+from app.services.domains.users import application as user_service
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/admin/users", tags=["api-admin-users"])
+
+
+def _serialize_user(user: User) -> dict[str, object]:
+    return {
+        "id": int(user.id),
+        "email": user.email,
+        "is_active": bool(user.is_active),
+        "is_admin": bool(user.is_admin),
+        "created_at": user.created_at,
+        "updated_at": user.updated_at,
+    }
+
+
+@router.get(
+    "",
+    response_model=AdminUsersListEnvelope,
+)
+async def list_users(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    users = await user_service.list_users(db_session)
+    logger.info(
+        "api.admin.users_listed",
+        extra={
+            "event": "api.admin.users_listed",
+            "admin_user_id": int(admin_user.id),
+            "user_count": len(users),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "users": [_serialize_user(user) for user in users],
+        },
+    )
+
+
+@router.post(
+    "",
+    response_model=AdminUserEnvelope,
+    status_code=201,
+)
+async def create_user(
+    payload: AdminUserCreateRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    auth_service: AuthService = Depends(get_auth_service),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    try:
+        validated_email = user_service.validate_email(payload.email)
+        validated_password = user_service.validate_password(payload.password)
+        created_user = await user_service.create_user(
+            db_session,
+            email=validated_email,
+            password_hash=auth_service.hash_password(validated_password),
+            is_admin=bool(payload.is_admin),
+        )
+    except user_service.UserServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_user_input",
+            message=str(exc),
+        ) from exc
+
+    logger.info(
+        "api.admin.user_created",
+        extra={
+            "event": "api.admin.user_created",
+            "admin_user_id": int(admin_user.id),
+            "target_user_id": int(created_user.id),
+            "target_is_admin": bool(created_user.is_admin),
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_user(created_user),
+    )
+
+
+@router.patch(
+    "/{user_id}/active",
+    response_model=AdminUserEnvelope,
+)
+async def set_user_active(
+    user_id: int,
+    payload: AdminUserActiveUpdateRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    target_user = await user_service.get_user_by_id(db_session, user_id)
+    if target_user is None:
+        raise ApiException(
+            status_code=404,
+            code="user_not_found",
+            message="User not found.",
+        )
+    if (
+        int(target_user.id) == int(admin_user.id)
+        and bool(target_user.is_active)
+        and not bool(payload.is_active)
+    ):
+        raise ApiException(
+            status_code=400,
+            code="cannot_deactivate_self",
+            message="You cannot deactivate your own account.",
+        )
+    updated_user = await user_service.set_user_active(
+        db_session,
+        user=target_user,
+        is_active=bool(payload.is_active),
+    )
+    logger.info(
+        "api.admin.user_active_updated",
+        extra={
+            "event": "api.admin.user_active_updated",
+            "admin_user_id": int(admin_user.id),
+            "target_user_id": int(updated_user.id),
+            "is_active": bool(updated_user.is_active),
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_user(updated_user),
+    )
+
+
+@router.post(
+    "/{user_id}/reset-password",
+    response_model=MessageEnvelope,
+)
+async def reset_user_password(
+    user_id: int,
+    payload: AdminResetPasswordRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    auth_service: AuthService = Depends(get_auth_service),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    target_user = await user_service.get_user_by_id(db_session, user_id)
+    if target_user is None:
+        raise ApiException(
+            status_code=404,
+            code="user_not_found",
+            message="User not found.",
+        )
+    try:
+        validated_password = user_service.validate_password(payload.new_password)
+    except user_service.UserServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_password",
+            message=str(exc),
+        ) from exc
+
+    await user_service.set_user_password_hash(
+        db_session,
+        user=target_user,
+        password_hash=auth_service.hash_password(validated_password),
+    )
+    logger.info(
+        "api.admin.user_password_reset",
+        extra={
+            "event": "api.admin.user_password_reset",
+            "admin_user_id": int(admin_user.id),
+            "target_user_id": int(target_user.id),
+        },
+    )
+    return success_payload(
+        request,
+        data={"message": f"Password reset: {target_user.email}"},
+    )

build/lib/app/api/routers/admin_dbops.py

@@ -0,0 +1,349 @@
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Depends, Path, Query, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_admin_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from app.api.schemas import (
+    AdminDbIntegrityEnvelope,
+    AdminPdfQueueBulkEnqueueEnvelope,
+    AdminPdfQueueRequeueEnvelope,
+    AdminPdfQueueEnvelope,
+    AdminDbRepairJobsEnvelope,
+    AdminRepairPublicationLinksEnvelope,
+    AdminRepairPublicationLinksRequest,
+)
+from app.db.models import DataRepairJob, User
+from app.db.session import get_db_session
+from app.services.domains.dbops import (
+    collect_integrity_report,
+    list_repair_jobs,
+    run_publication_link_repair,
+)
+from app.services.domains.publications import application as publication_service
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/admin/db", tags=["api-admin-dbops"])
+
+
+def _serialize_repair_job(job: DataRepairJob) -> dict[str, object]:
+    return {
+        "id": int(job.id),
+        "job_name": job.job_name,
+        "requested_by": job.requested_by,
+        "dry_run": bool(job.dry_run),
+        "status": job.status,
+        "scope": dict(job.scope or {}),
+        "summary": dict(job.summary or {}),
+        "error_text": job.error_text,
+        "started_at": job.started_at,
+        "finished_at": job.finished_at,
+        "created_at": job.created_at,
+        "updated_at": job.updated_at,
+    }
+
+
+def _requested_by_value(*, payload: AdminRepairPublicationLinksRequest, admin_user: User) -> str:
+    from_payload = (payload.requested_by or "").strip()
+    return from_payload or admin_user.email
+
+
+def _serialize_pdf_queue_item(item) -> dict[str, object]:
+    return {
+        "publication_id": item.publication_id,
+        "title": item.title,
+        "display_identifier": _serialize_display_identifier(item.display_identifier),
+        "pdf_url": item.pdf_url,
+        "status": item.status,
+        "attempt_count": item.attempt_count,
+        "last_failure_reason": item.last_failure_reason,
+        "last_failure_detail": item.last_failure_detail,
+        "last_source": item.last_source,
+        "requested_by_user_id": item.requested_by_user_id,
+        "requested_by_email": item.requested_by_email,
+        "queued_at": item.queued_at,
+        "last_attempt_at": item.last_attempt_at,
+        "resolved_at": item.resolved_at,
+        "updated_at": item.updated_at,
+    }
+
+
+def _serialize_display_identifier(value) -> dict[str, object] | None:
+    if value is None:
+        return None
+    return {
+        "kind": value.kind,
+        "value": value.value,
+        "label": value.label,
+        "url": value.url,
+        "confidence_score": float(value.confidence_score),
+    }
+
+
+def _requeue_response_state(*, queued: bool) -> tuple[str, str]:
+    if queued:
+        return "queued", "PDF lookup queued."
+    return "blocked", "PDF lookup is already queued or currently running."
+
+
+def _bulk_enqueue_message(*, queued_count: int, requested_count: int) -> str:
+    if requested_count == 0:
+        return "No missing-PDF publications were found."
+    if queued_count == 0:
+        return "No publications were queued; all candidates were already in-flight."
+    return f"Queued {queued_count} publication(s) for PDF lookup."
+
+
+def _resolve_pdf_queue_paging(
+    *,
+    page: int,
+    page_size: int,
+    limit: int | None,
+    offset: int | None,
+) -> tuple[int, int, int]:
+    resolved_limit = int(limit) if limit is not None else int(page_size)
+    resolved_offset = int(offset) if offset is not None else max((int(page) - 1) * resolved_limit, 0)
+    resolved_page = max((resolved_offset // max(resolved_limit, 1)) + 1, 1)
+    return resolved_page, max(resolved_limit, 1), max(resolved_offset, 0)
+
+
+def _pdf_queue_page_data(*, total_count: int, page: int, page_size: int, offset: int) -> dict[str, object]:
+    return {
+        "total_count": int(total_count),
+        "page": int(page),
+        "page_size": int(page_size),
+        "has_prev": int(offset) > 0,
+        "has_next": int(offset) + int(page_size) < int(total_count),
+    }
+
+
+@router.get(
+    "/integrity",
+    response_model=AdminDbIntegrityEnvelope,
+)
+async def get_integrity_report(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    report = await collect_integrity_report(db_session)
+    logger.info(
+        "api.admin.db.integrity_checked",
+        extra={
+            "event": "api.admin.db.integrity_checked",
+            "admin_user_id": int(admin_user.id),
+            "status": report.get("status"),
+            "failure_count": len(report.get("failures", [])),
+            "warning_count": len(report.get("warnings", [])),
+        },
+    )
+    return success_payload(request, data=report)
+
+
+@router.get(
+    "/repair-jobs",
+    response_model=AdminDbRepairJobsEnvelope,
+)
+async def get_repair_jobs(
+    request: Request,
+    limit: int = Query(default=50, ge=1, le=200),
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    jobs = await list_repair_jobs(db_session, limit=limit)
+    logger.info(
+        "api.admin.db.repair_jobs_listed",
+        extra={
+            "event": "api.admin.db.repair_jobs_listed",
+            "admin_user_id": int(admin_user.id),
+            "limit": int(limit),
+            "job_count": len(jobs),
+        },
+    )
+    return success_payload(
+        request,
+        data={"jobs": [_serialize_repair_job(job) for job in jobs]},
+    )
+
+
+@router.get(
+    "/pdf-queue",
+    response_model=AdminPdfQueueEnvelope,
+)
+async def get_pdf_queue(
+    request: Request,
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=100, ge=1, le=500),
+    limit: int | None = Query(default=None, ge=1, le=500),
+    offset: int | None = Query(default=None, ge=0),
+    status: str | None = Query(default=None),
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    normalized_status = (status or "").strip().lower() or None
+    resolved_page, resolved_limit, resolved_offset = _resolve_pdf_queue_paging(
+        page=page,
+        page_size=page_size,
+        limit=limit,
+        offset=offset,
+    )
+    queue_page = await publication_service.list_pdf_queue_page(
+        db_session,
+        limit=resolved_limit,
+        offset=resolved_offset,
+        status=normalized_status,
+    )
+    logger.info(
+        "api.admin.db.pdf_queue_listed",
+        extra={
+            "event": "api.admin.db.pdf_queue_listed",
+            "admin_user_id": int(admin_user.id),
+            "page": int(resolved_page),
+            "page_size": int(resolved_limit),
+            "offset": int(resolved_offset),
+            "status": normalized_status,
+            "item_count": len(queue_page.items),
+            "total_count": int(queue_page.total_count),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "items": [_serialize_pdf_queue_item(item) for item in queue_page.items],
+            **_pdf_queue_page_data(
+                total_count=queue_page.total_count,
+                page=resolved_page,
+                page_size=resolved_limit,
+                offset=resolved_offset,
+            ),
+        },
+    )
+
+
+@router.post(
+    "/pdf-queue/{publication_id}/requeue",
+    response_model=AdminPdfQueueRequeueEnvelope,
+)
+async def requeue_pdf_lookup(
+    request: Request,
+    publication_id: int = Path(ge=1),
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    result = await publication_service.enqueue_retry_pdf_job_for_publication_id(
+        db_session,
+        user_id=int(admin_user.id),
+        request_email=admin_user.email,
+        publication_id=publication_id,
+    )
+    if not result.publication_exists:
+        raise ApiException(
+            status_code=404,
+            code="publication_not_found",
+            message="Publication not found.",
+        )
+    status, message = _requeue_response_state(queued=result.queued)
+    logger.info(
+        "api.admin.db.pdf_queue_requeue_requested",
+        extra={
+            "event": "api.admin.db.pdf_queue_requeue_requested",
+            "admin_user_id": int(admin_user.id),
+            "publication_id": int(publication_id),
+            "queued": bool(result.queued),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "publication_id": int(publication_id),
+            "queued": bool(result.queued),
+            "status": status,
+            "message": message,
+        },
+    )
+
+
+@router.post(
+    "/pdf-queue/requeue-all",
+    response_model=AdminPdfQueueBulkEnqueueEnvelope,
+)
+async def requeue_all_missing_pdfs(
+    request: Request,
+    limit: int = Query(default=1000, ge=1, le=5000),
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    result = await publication_service.enqueue_all_missing_pdf_jobs(
+        db_session,
+        user_id=int(admin_user.id),
+        request_email=admin_user.email,
+        limit=limit,
+    )
+    logger.info(
+        "api.admin.db.pdf_queue_requeue_all",
+        extra={
+            "event": "api.admin.db.pdf_queue_requeue_all",
+            "admin_user_id": int(admin_user.id),
+            "limit": int(limit),
+            "requested_count": int(result.requested_count),
+            "queued_count": int(result.queued_count),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "requested_count": int(result.requested_count),
+            "queued_count": int(result.queued_count),
+            "message": _bulk_enqueue_message(
+                queued_count=int(result.queued_count),
+                requested_count=int(result.requested_count),
+            ),
+        },
+    )
+
+
+@router.post(
+    "/repairs/publication-links",
+    response_model=AdminRepairPublicationLinksEnvelope,
+)
+async def trigger_publication_link_repair(
+    payload: AdminRepairPublicationLinksRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    admin_user: User = Depends(get_api_admin_user),
+):
+    try:
+        result = await run_publication_link_repair(
+            db_session,
+            scope_mode=payload.scope_mode,
+            user_id=payload.user_id,
+            scholar_profile_ids=payload.scholar_profile_ids,
+            dry_run=bool(payload.dry_run),
+            gc_orphan_publications=bool(payload.gc_orphan_publications),
+            requested_by=_requested_by_value(payload=payload, admin_user=admin_user),
+        )
+    except ValueError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_repair_scope",
+            message=str(exc),
+        ) from exc
+    logger.info(
+        "api.admin.db.publication_link_repair_triggered",
+        extra={
+            "event": "api.admin.db.publication_link_repair_triggered",
+            "admin_user_id": int(admin_user.id),
+            "scope_mode": payload.scope_mode,
+            "target_user_id": int(payload.user_id) if payload.user_id is not None else None,
+            "dry_run": bool(payload.dry_run),
+            "gc_orphan_publications": bool(payload.gc_orphan_publications),
+            "job_id": int(result["job_id"]),
+            "status": result["status"],
+        },
+    )
+    return success_payload(request, data=result)

build/lib/app/api/routers/auth.py

@@ -0,0 +1,237 @@
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Depends, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.errors import ApiException
+from app.api.deps import get_api_current_user
+from app.api.responses import success_payload
+from app.api.schemas import (
+    AuthMeEnvelope,
+    ChangePasswordRequest,
+    CsrfBootstrapEnvelope,
+    LoginEnvelope,
+    LoginRequest,
+    MessageEnvelope,
+)
+from app.auth.deps import get_auth_service, get_login_rate_limiter
+from app.auth.rate_limit import SlidingWindowRateLimiter
+from app.auth import runtime as auth_runtime
+from app.auth.service import AuthService
+from app.auth.session import set_session_user
+from app.db.models import User
+from app.db.session import get_db_session
+from app.security.csrf import ensure_csrf_token
+from app.services.domains.users import application as user_service
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/auth", tags=["api-auth"])
+
+
+def _login_limiter_key_and_email(request: Request, payload: LoginRequest) -> tuple[str, str]:
+    return auth_runtime.login_rate_limit_key(request, payload.email), payload.email.strip().lower()
+
+
+def _raise_rate_limited(normalized_email: str, retry_after_seconds: int) -> None:
+    logger.warning(
+        "api.auth.login_rate_limited",
+        extra={
+            "event": "api.auth.login_rate_limited",
+            "email": normalized_email,
+            "retry_after_seconds": retry_after_seconds,
+        },
+    )
+    raise ApiException(
+        status_code=429,
+        code="rate_limited",
+        message="Too many login attempts. Please try again later.",
+        details={"retry_after_seconds": retry_after_seconds},
+    )
+
+
+def _serialize_user_payload(user: User) -> dict[str, object]:
+    return {
+        "id": int(user.id),
+        "email": user.email,
+        "is_admin": bool(user.is_admin),
+        "is_active": bool(user.is_active),
+    }
+
+
+def _raise_invalid_credentials(*, normalized_email: str) -> None:
+    logger.info(
+        "api.auth.login_failed",
+        extra={"event": "api.auth.login_failed", "email": normalized_email},
+    )
+    raise ApiException(
+        status_code=401,
+        code="invalid_credentials",
+        message="Invalid email or password.",
+    )
+
+
+@router.post(
+    "/login",
+    response_model=LoginEnvelope,
+)
+async def login(
+    payload: LoginRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    auth_service: AuthService = Depends(get_auth_service),
+    rate_limiter: SlidingWindowRateLimiter = Depends(get_login_rate_limiter),
+):
+    limiter_key, normalized_email = _login_limiter_key_and_email(request, payload)
+    decision = rate_limiter.check(limiter_key)
+    if not decision.allowed:
+        _raise_rate_limited(normalized_email, int(decision.retry_after_seconds))
+
+    user = await auth_service.authenticate_user(
+        db_session,
+        email=payload.email,
+        password=payload.password,
+    )
+    if user is None:
+        rate_limiter.record_failure(limiter_key)
+        _raise_invalid_credentials(normalized_email=normalized_email)
+
+    rate_limiter.reset(limiter_key)
+    set_session_user(
+        request,
+        user_id=user.id,
+        email=user.email,
+        is_admin=user.is_admin,
+    )
+    logger.info(
+        "api.auth.login_succeeded",
+        extra={
+            "event": "api.auth.login_succeeded",
+            "user_id": user.id,
+            "is_admin": user.is_admin,
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "authenticated": True,
+            "csrf_token": ensure_csrf_token(request),
+            "user": _serialize_user_payload(user),
+        },
+    )
+
+
+@router.get(
+    "/me",
+    response_model=AuthMeEnvelope,
+)
+async def get_current_session(
+    request: Request,
+    current_user: User = Depends(get_api_current_user),
+):
+    return success_payload(
+        request,
+        data={
+            "authenticated": True,
+            "csrf_token": ensure_csrf_token(request),
+            "user": _serialize_user_payload(current_user),
+        },
+    )
+
+
+@router.get(
+    "/csrf",
+    response_model=CsrfBootstrapEnvelope,
+)
+async def get_csrf_bootstrap(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+):
+    current_user = await auth_runtime.get_authenticated_user(request, db_session)
+    return success_payload(
+        request,
+        data={
+            "csrf_token": ensure_csrf_token(request),
+            "authenticated": current_user is not None,
+        },
+    )
+
+
+@router.post(
+    "/change-password",
+    response_model=MessageEnvelope,
+)
+async def change_password(
+    payload: ChangePasswordRequest,
+    request: Request,
+    current_user: User = Depends(get_api_current_user),
+    db_session: AsyncSession = Depends(get_db_session),
+    auth_service: AuthService = Depends(get_auth_service),
+):
+    if not auth_service.verify_password(
+        password_hash=current_user.password_hash,
+        password=payload.current_password,
+    ):
+        raise ApiException(
+            status_code=400,
+            code="invalid_current_password",
+            message="Current password is incorrect.",
+        )
+    if payload.new_password != payload.confirm_password:
+        raise ApiException(
+            status_code=400,
+            code="password_confirmation_mismatch",
+            message="New password and confirmation do not match.",
+        )
+    try:
+        validated_password = user_service.validate_password(payload.new_password)
+    except user_service.UserServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_password",
+            message=str(exc),
+        ) from exc
+
+    await user_service.set_user_password_hash(
+        db_session,
+        user=current_user,
+        password_hash=auth_service.hash_password(validated_password),
+    )
+    logger.info(
+        "api.auth.password_changed",
+        extra={
+            "event": "api.auth.password_changed",
+            "user_id": int(current_user.id),
+        },
+    )
+    return success_payload(
+        request,
+        data={"message": "Password updated successfully."},
+    )
+
+
+@router.post(
+    "/logout",
+    response_model=MessageEnvelope,
+)
+async def logout(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+):
+    current_user = await auth_runtime.get_authenticated_user(request, db_session)
+    auth_runtime.invalidate_session(request)
+    logger.info(
+        "api.auth.logout",
+        extra={
+            "event": "api.auth.logout",
+            "user_id": int(current_user.id) if current_user is not None else None,
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "message": "Logged out.",
+        },
+    )

build/lib/app/api/routers/publications.py

@@ -0,0 +1,528 @@
+from __future__ import annotations
+
+import logging
+from typing import Literal
+
+from fastapi import APIRouter, Depends, Path, Query, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_current_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from app.api.schemas import (
+    MarkAllReadEnvelope,
+    MarkSelectedReadEnvelope,
+    MarkSelectedReadRequest,
+    PublicationsListEnvelope,
+    RetryPublicationPdfEnvelope,
+    RetryPublicationPdfRequest,
+    TogglePublicationFavoriteEnvelope,
+    TogglePublicationFavoriteRequest,
+)
+from app.db.models import User
+from app.db.session import get_db_session
+from app.services.domains.publication_identifiers import application as identifier_service
+from app.services.domains.publications import application as publication_service
+from app.services.domains.scholars import application as scholar_service
+from app.settings import settings
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/publications", tags=["api-publications"])
+
+
+async def _require_selected_profile(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    selected_scholar_id: int | None,
+) -> None:
+    if selected_scholar_id is None:
+        return
+    selected_profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=selected_scholar_id,
+    )
+    if selected_profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar filter not found.",
+        )
+
+
+def _serialize_publication_item(item) -> dict[str, object]:
+    return {
+        "publication_id": item.publication_id,
+        "scholar_profile_id": item.scholar_profile_id,
+        "scholar_label": item.scholar_label,
+        "title": item.title,
+        "year": item.year,
+        "citation_count": item.citation_count,
+        "venue_text": item.venue_text,
+        "pub_url": item.pub_url,
+        "display_identifier": _serialize_display_identifier(item.display_identifier),
+        "pdf_url": item.pdf_url,
+        "pdf_status": item.pdf_status,
+        "pdf_attempt_count": item.pdf_attempt_count,
+        "pdf_failure_reason": item.pdf_failure_reason,
+        "pdf_failure_detail": item.pdf_failure_detail,
+        "is_read": item.is_read,
+        "is_favorite": item.is_favorite,
+        "first_seen_at": item.first_seen_at,
+        "is_new_in_latest_run": item.is_new_in_latest_run,
+    }
+
+
+def _serialize_display_identifier(value) -> dict[str, object] | None:
+    if value is None:
+        return None
+    return {
+        "kind": value.kind,
+        "value": value.value,
+        "label": value.label,
+        "url": value.url,
+        "confidence_score": float(value.confidence_score),
+    }
+
+
+async def _publication_counts(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    selected_scholar_id: int | None,
+    favorite_only: bool,
+) -> tuple[int, int, int, int]:
+    unread_count = await publication_service.count_unread_for_user(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=selected_scholar_id,
+        favorite_only=favorite_only,
+    )
+    favorites_count = await publication_service.count_favorite_for_user(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=selected_scholar_id,
+    )
+    latest_count = await publication_service.count_latest_for_user(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=selected_scholar_id,
+        favorite_only=favorite_only,
+    )
+    total_count = await publication_service.count_for_user(
+        db_session,
+        user_id=user_id,
+        mode=publication_service.MODE_ALL,
+        scholar_profile_id=selected_scholar_id,
+        favorite_only=favorite_only,
+    )
+    return unread_count, favorites_count, latest_count, total_count
+
+
+async def _list_publications_for_request(
+    db_session: AsyncSession,
+    *,
+    current_user: User,
+    mode: Literal["all", "unread", "latest", "new"] | None,
+    favorite_only: bool,
+    scholar_profile_id: int | None,
+    limit: int,
+    offset: int,
+) -> tuple[str, int | None, list]:
+    resolved_mode = publication_service.resolve_publication_view_mode(mode)
+    selected_scholar_id = scholar_profile_id
+    await _require_selected_profile(
+        db_session,
+        user_id=current_user.id,
+        selected_scholar_id=selected_scholar_id,
+    )
+    publications = await publication_service.list_for_user(
+        db_session,
+        user_id=current_user.id,
+        mode=resolved_mode,
+        scholar_profile_id=selected_scholar_id,
+        favorite_only=favorite_only,
+        limit=limit,
+        offset=offset,
+    )
+    await publication_service.schedule_missing_pdf_enrichment_for_user(
+        db_session,
+        user_id=current_user.id,
+        request_email=current_user.email,
+        items=publications,
+        max_items=settings.unpaywall_max_items_per_request,
+    )
+    hydrated = await publication_service.hydrate_pdf_enrichment_state(
+        db_session,
+        items=publications,
+    )
+    return resolved_mode, selected_scholar_id, hydrated
+
+
+def _resolve_publications_paging(
+    *,
+    page: int,
+    page_size: int,
+    limit: int | None,
+    offset: int | None,
+) -> tuple[int, int, int]:
+    resolved_limit = int(limit) if limit is not None else int(page_size)
+    resolved_offset = int(offset) if offset is not None else max((int(page) - 1) * resolved_limit, 0)
+    resolved_page = max((resolved_offset // max(resolved_limit, 1)) + 1, 1)
+    return resolved_page, max(resolved_limit, 1), max(resolved_offset, 0)
+
+
+def _publications_list_data(
+    *,
+    mode: str,
+    favorite_only: bool,
+    selected_scholar_id: int | None,
+    unread_count: int,
+    favorites_count: int,
+    latest_count: int,
+    total_count: int,
+    publications: list,
+    page: int,
+    page_size: int,
+    offset: int,
+) -> dict[str, object]:
+    return {
+        "mode": mode,
+        "favorite_only": favorite_only,
+        "selected_scholar_profile_id": selected_scholar_id,
+        "unread_count": unread_count,
+        "favorites_count": favorites_count,
+        "latest_count": latest_count,
+        "new_count": latest_count,
+        "total_count": total_count,
+        "page": int(page),
+        "page_size": int(page_size),
+        "has_prev": int(offset) > 0,
+        "has_next": int(offset) + int(page_size) < int(total_count),
+        "publications": [_serialize_publication_item(item) for item in publications],
+    }
+
+
+def _retry_pdf_message(*, queued: bool, resolved_pdf: bool, pdf_status: str) -> str:
+    if resolved_pdf:
+        return "Open-access PDF link already resolved."
+    if queued:
+        return "PDF lookup queued."
+    if pdf_status in {"queued", "running"}:
+        return "PDF lookup is already queued."
+    return "No open-access PDF link found."
+
+
+def _favorite_message(*, is_favorite: bool) -> str:
+    if is_favorite:
+        return "Publication marked as favorite."
+    return "Publication removed from favorites."
+
+
+async def _retry_publication_state(
+    db_session: AsyncSession,
+    *,
+    current_user: User,
+    scholar_profile_id: int,
+    publication_id: int,
+):
+    publication = await publication_service.retry_pdf_for_user(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+        publication_id=publication_id,
+    )
+    if publication is None:
+        raise ApiException(
+            status_code=404,
+            code="publication_not_found",
+            message="Publication not found.",
+        )
+    queued = False
+    if not publication.pdf_url:
+        queued = await publication_service.schedule_retry_pdf_enrichment_for_row(
+            db_session,
+            user_id=current_user.id,
+            request_email=current_user.email,
+            item=publication,
+        )
+    hydrated = await publication_service.hydrate_pdf_enrichment_state(
+        db_session,
+        items=[publication],
+    )
+    current = hydrated[0] if hydrated else publication
+    return current, queued
+
+
+async def _favorite_publication_state(
+    db_session: AsyncSession,
+    *,
+    current_user: User,
+    scholar_profile_id: int,
+    publication_id: int,
+    is_favorite: bool,
+):
+    updated_count = await publication_service.set_publication_favorite_for_user(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+        publication_id=publication_id,
+        is_favorite=is_favorite,
+    )
+    if updated_count <= 0:
+        raise ApiException(
+            status_code=404,
+            code="publication_not_found",
+            message="Publication not found.",
+        )
+    publication = await publication_service.get_publication_item_for_user(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+        publication_id=publication_id,
+    )
+    if publication is None:
+        raise ApiException(
+            status_code=404,
+            code="publication_not_found",
+            message="Publication not found.",
+        )
+    hydrated = await publication_service.hydrate_pdf_enrichment_state(
+        db_session,
+        items=[publication],
+    )
+    current = hydrated[0] if hydrated else publication
+    identifiers = await identifier_service.overlay_publication_items_with_display_identifiers(
+        db_session,
+        items=[current],
+    )
+    return identifiers[0] if identifiers else current
+
+
+def _log_retry_pdf_result(
+    *,
+    current_user: User,
+    scholar_profile_id: int,
+    publication_id: int,
+    queued: bool,
+    resolved_pdf: bool,
+    pdf_status: str,
+    doi: str | None,
+) -> None:
+    logger.info(
+        "api.publications.retry_pdf",
+        extra={
+            "event": "api.publications.retry_pdf",
+            "user_id": current_user.id,
+            "scholar_profile_id": scholar_profile_id,
+            "publication_id": publication_id,
+            "queued": queued,
+            "resolved_pdf": resolved_pdf,
+            "pdf_status": pdf_status,
+        },
+    )
+
+
+@router.get(
+    "",
+    response_model=PublicationsListEnvelope,
+)
+async def list_publications(
+    request: Request,
+    mode: Literal["all", "unread", "latest", "new"] | None = Query(default=None),
+    favorite_only: bool = Query(default=False),
+    scholar_profile_id: int | None = Query(default=None, ge=1),
+    page: int = Query(default=1, ge=1),
+    page_size: int = Query(default=100, ge=1, le=500),
+    limit: int | None = Query(default=None, ge=1, le=1000),
+    offset: int | None = Query(default=None, ge=0),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    resolved_page, resolved_limit, resolved_offset = _resolve_publications_paging(
+        page=page,
+        page_size=page_size,
+        limit=limit,
+        offset=offset,
+    )
+    resolved_mode, selected_scholar_id, publications = await _list_publications_for_request(
+        db_session,
+        current_user=current_user,
+        mode=mode,
+        favorite_only=favorite_only,
+        scholar_profile_id=scholar_profile_id,
+        limit=resolved_limit,
+        offset=resolved_offset,
+    )
+    unread_count, favorites_count, latest_count, total_count = await _publication_counts(
+        db_session,
+        user_id=current_user.id,
+        selected_scholar_id=selected_scholar_id,
+        favorite_only=favorite_only,
+    )
+    data = _publications_list_data(
+        mode=resolved_mode,
+        favorite_only=favorite_only,
+        selected_scholar_id=selected_scholar_id,
+        unread_count=unread_count,
+        favorites_count=favorites_count,
+        latest_count=latest_count,
+        total_count=total_count,
+        publications=publications,
+        page=resolved_page,
+        page_size=resolved_limit,
+        offset=resolved_offset,
+    )
+    return success_payload(request, data=data)
+
+
+@router.post(
+    "/mark-all-read",
+    response_model=MarkAllReadEnvelope,
+)
+async def mark_all_publications_read(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    updated_count = await publication_service.mark_all_unread_as_read_for_user(
+        db_session,
+        user_id=current_user.id,
+    )
+    logger.info(
+        "api.publications.mark_all_read",
+        extra={
+            "event": "api.publications.mark_all_read",
+            "user_id": current_user.id,
+            "updated_count": updated_count,
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "message": "Marked all unread publications as read.",
+            "updated_count": updated_count,
+        },
+    )
+
+
+@router.post(
+    "/mark-read",
+    response_model=MarkSelectedReadEnvelope,
+)
+async def mark_selected_publications_read(
+    payload: MarkSelectedReadRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    selection_pairs = sorted(
+        {
+            (int(item.scholar_profile_id), int(item.publication_id))
+            for item in payload.selections
+        }
+    )
+    updated_count = await publication_service.mark_selected_as_read_for_user(
+        db_session,
+        user_id=current_user.id,
+        selections=selection_pairs,
+    )
+    logger.info(
+        "api.publications.mark_selected_read",
+        extra={
+            "event": "api.publications.mark_selected_read",
+            "user_id": current_user.id,
+            "requested_count": len(selection_pairs),
+            "updated_count": updated_count,
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "message": "Marked selected publications as read.",
+            "requested_count": len(selection_pairs),
+            "updated_count": updated_count,
+        },
+    )
+
+
+@router.post(
+    "/{publication_id}/retry-pdf",
+    response_model=RetryPublicationPdfEnvelope,
+)
+async def retry_publication_pdf(
+    payload: RetryPublicationPdfRequest,
+    request: Request,
+    publication_id: int = Path(ge=1),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    current, queued = await _retry_publication_state(
+        db_session,
+        current_user=current_user,
+        scholar_profile_id=payload.scholar_profile_id,
+        publication_id=publication_id,
+    )
+    resolved_pdf = bool(current.pdf_url)
+    message = _retry_pdf_message(
+        queued=queued,
+        resolved_pdf=resolved_pdf,
+        pdf_status=current.pdf_status,
+    )
+    _log_retry_pdf_result(
+        current_user=current_user,
+        scholar_profile_id=payload.scholar_profile_id,
+        publication_id=publication_id,
+        queued=queued,
+        resolved_pdf=resolved_pdf,
+        pdf_status=current.pdf_status,
+        doi=None,
+    )
+    return success_payload(
+        request,
+        data={
+            "message": message,
+            "queued": queued,
+            "resolved_pdf": resolved_pdf,
+            "publication": _serialize_publication_item(current),
+        },
+    )
+
+
+@router.post(
+    "/{publication_id}/favorite",
+    response_model=TogglePublicationFavoriteEnvelope,
+)
+async def toggle_publication_favorite(
+    payload: TogglePublicationFavoriteRequest,
+    request: Request,
+    publication_id: int = Path(ge=1),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    current = await _favorite_publication_state(
+        db_session,
+        current_user=current_user,
+        scholar_profile_id=payload.scholar_profile_id,
+        publication_id=publication_id,
+        is_favorite=payload.is_favorite,
+    )
+    logger.info(
+        "api.publications.favorite",
+        extra={
+            "event": "api.publications.favorite",
+            "user_id": current_user.id,
+            "scholar_profile_id": payload.scholar_profile_id,
+            "publication_id": publication_id,
+            "is_favorite": bool(payload.is_favorite),
+        },
+    )
+    return success_payload(
+        request,
+        data={
+            "message": _favorite_message(is_favorite=bool(payload.is_favorite)),
+            "publication": _serialize_publication_item(current),
+        },
+    )

build/lib/app/api/routers/runs.py

@@ -0,0 +1,845 @@
+from __future__ import annotations
+
+import logging
+import re
+from typing import Any
+
+from fastapi import APIRouter, Depends, Query, Request
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_current_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from fastapi.responses import StreamingResponse
+from app.api.schemas import (
+    ManualRunEnvelope,
+    QueueClearEnvelope,
+    QueueItemEnvelope,
+    QueueListEnvelope,
+    RunDetailEnvelope,
+    RunsListEnvelope,
+)
+from app.db.models import RunStatus, RunTriggerType, User
+from app.db.session import get_db_session
+from app.services.domains.ingestion import application as ingestion_service
+from app.services.domains.ingestion import safety as run_safety_service
+from app.services.domains.runs import application as run_service
+from app.services.domains.runs.events import event_generator
+from app.services.domains.settings import application as user_settings_service
+from app.settings import settings
+from app.api.runtime_deps import get_ingestion_service
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/runs", tags=["api-runs"])
+
+IDEMPOTENCY_HEADER = "Idempotency-Key"
+IDEMPOTENCY_MAX_LENGTH = 128
+IDEMPOTENCY_PATTERN = re.compile(r"^[A-Za-z0-9._:-]{8,128}$")
+
+
+def _int_value(value: Any, default: int = 0) -> int:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _str_value(value: Any) -> str | None:
+    if value is None:
+        return None
+    text = str(value).strip()
+    return text if text else None
+
+
+def _bool_value(value: Any, default: bool = False) -> bool:
+    if isinstance(value, bool):
+        return value
+    if isinstance(value, str):
+        lowered = value.strip().lower()
+        if lowered in {"1", "true", "yes", "on"}:
+            return True
+        if lowered in {"0", "false", "no", "off"}:
+            return False
+    return default
+
+
+def _normalize_idempotency_key(raw_value: str | None) -> str | None:
+    if raw_value is None:
+        return None
+    candidate = raw_value.strip()
+    if not candidate:
+        return None
+    if len(candidate) > IDEMPOTENCY_MAX_LENGTH or not IDEMPOTENCY_PATTERN.match(candidate):
+        raise ApiException(
+            status_code=400,
+            code="invalid_idempotency_key",
+            message=(
+                "Invalid Idempotency-Key. Use 8-128 characters from: "
+                "A-Z a-z 0-9 . _ : -"
+            ),
+        )
+    return candidate
+
+
+def _serialize_run(run) -> dict[str, Any]:
+    summary = run_service.extract_run_summary(run.error_log)
+    return {
+        "id": int(run.id),
+        "trigger_type": run.trigger_type.value,
+        "status": run.status.value,
+        "start_dt": run.start_dt,
+        "end_dt": run.end_dt,
+        "scholar_count": int(run.scholar_count or 0),
+        "new_publication_count": int(run.new_pub_count or 0),
+        "failed_count": int(summary["failed_count"]),
+        "partial_count": int(summary["partial_count"]),
+    }
+
+
+def _serialize_queue_item(item) -> dict[str, Any]:
+    return {
+        "id": int(item.id),
+        "scholar_profile_id": int(item.scholar_profile_id),
+        "scholar_label": item.scholar_label,
+        "status": item.status,
+        "reason": item.reason,
+        "dropped_reason": item.dropped_reason,
+        "attempt_count": int(item.attempt_count),
+        "resume_cstart": int(item.resume_cstart),
+        "next_attempt_dt": item.next_attempt_dt,
+        "updated_at": item.updated_at,
+        "last_error": item.last_error,
+        "last_run_id": item.last_run_id,
+    }
+
+
+def _normalize_attempt_log(value: Any) -> list[dict[str, Any]]:
+    if not isinstance(value, list):
+        return []
+    normalized: list[dict[str, Any]] = []
+    for item in value:
+        if not isinstance(item, dict):
+            continue
+        normalized.append(
+            {
+                "attempt": _int_value(item.get("attempt"), 0),
+                "cstart": _int_value(item.get("cstart"), 0),
+                "state": _str_value(item.get("state")),
+                "state_reason": _str_value(item.get("state_reason")),
+                "status_code": (
+                    _int_value(item.get("status_code"))
+                    if item.get("status_code") is not None
+                    else None
+                ),
+                "fetch_error": _str_value(item.get("fetch_error")),
+            }
+        )
+    return normalized
+
+
+def _normalize_page_logs(value: Any) -> list[dict[str, Any]]:
+    if not isinstance(value, list):
+        return []
+    normalized: list[dict[str, Any]] = []
+    for item in value:
+        if not isinstance(item, dict):
+            continue
+        warning_codes = item.get("warning_codes")
+        normalized.append(
+            {
+                "page": _int_value(item.get("page"), 0),
+                "cstart": _int_value(item.get("cstart"), 0),
+                "state": _str_value(item.get("state")) or "unknown",
+                "state_reason": _str_value(item.get("state_reason")),
+                "status_code": (
+                    _int_value(item.get("status_code"))
+                    if item.get("status_code") is not None
+                    else None
+                ),
+                "publication_count": _int_value(item.get("publication_count"), 0),
+                "attempt_count": _int_value(item.get("attempt_count"), 0),
+                "has_show_more_button": _bool_value(item.get("has_show_more_button"), False),
+                "articles_range": _str_value(item.get("articles_range")),
+                "warning_codes": [
+                    str(code)
+                    for code in (warning_codes if isinstance(warning_codes, list) else [])
+                ],
+            }
+        )
+    return normalized
+
+
+def _normalize_debug(value: Any) -> dict[str, Any] | None:
+    if not isinstance(value, dict):
+        return None
+    marker_counts = value.get("marker_counts_nonzero")
+    warning_codes = value.get("warning_codes")
+    return {
+        "status_code": (
+            _int_value(value.get("status_code"))
+            if value.get("status_code") is not None
+            else None
+        ),
+        "final_url": _str_value(value.get("final_url")),
+        "fetch_error": _str_value(value.get("fetch_error")),
+        "body_sha256": _str_value(value.get("body_sha256")),
+        "body_length": (
+            _int_value(value.get("body_length"))
+            if value.get("body_length") is not None
+            else None
+        ),
+        "has_show_more_button": (
+            _bool_value(value.get("has_show_more_button"), False)
+            if value.get("has_show_more_button") is not None
+            else None
+        ),
+        "articles_range": _str_value(value.get("articles_range")),
+        "state_reason": _str_value(value.get("state_reason")),
+        "warning_codes": [
+            str(code)
+            for code in (warning_codes if isinstance(warning_codes, list) else [])
+        ],
+        "marker_counts_nonzero": {
+            str(key): _int_value(count, 0)
+            for key, count in (marker_counts.items() if isinstance(marker_counts, dict) else [])
+        },
+        "page_logs": _normalize_page_logs(value.get("page_logs")),
+        "attempt_log": _normalize_attempt_log(value.get("attempt_log")),
+    }
+
+
+def _normalize_scholar_result(value: Any) -> dict[str, Any]:
+    if not isinstance(value, dict):
+        return {
+            "scholar_profile_id": 0,
+            "scholar_id": "unknown",
+            "state": "unknown",
+            "state_reason": None,
+            "outcome": "failed",
+            "attempt_count": 0,
+            "publication_count": 0,
+            "start_cstart": 0,
+            "continuation_cstart": None,
+            "continuation_enqueued": False,
+            "continuation_cleared": False,
+            "warnings": [],
+            "error": None,
+            "debug": None,
+        }
+    warnings = value.get("warnings")
+    return {
+        "scholar_profile_id": _int_value(value.get("scholar_profile_id"), 0),
+        "scholar_id": _str_value(value.get("scholar_id")) or "unknown",
+        "state": _str_value(value.get("state")) or "unknown",
+        "state_reason": _str_value(value.get("state_reason")),
+        "outcome": _str_value(value.get("outcome")) or "failed",
+        "attempt_count": _int_value(value.get("attempt_count"), 0),
+        "publication_count": _int_value(value.get("publication_count"), 0),
+        "start_cstart": _int_value(value.get("start_cstart"), 0),
+        "continuation_cstart": (
+            _int_value(value.get("continuation_cstart"))
+            if value.get("continuation_cstart") is not None
+            else None
+        ),
+        "continuation_enqueued": _bool_value(value.get("continuation_enqueued"), False),
+        "continuation_cleared": _bool_value(value.get("continuation_cleared"), False),
+        "warnings": [str(item) for item in (warnings if isinstance(warnings, list) else [])],
+        "error": _str_value(value.get("error")),
+        "debug": _normalize_debug(value.get("debug")),
+    }
+
+
+def _manual_run_payload_from_run(
+    run,
+    *,
+    idempotency_key: str | None,
+    reused_existing_run: bool,
+    safety_state: dict[str, Any],
+) -> dict[str, Any]:
+    summary = run_service.extract_run_summary(run.error_log)
+    return {
+        "run_id": int(run.id),
+        "status": run.status.value,
+        "scholar_count": int(run.scholar_count or 0),
+        "succeeded_count": int(summary["succeeded_count"]),
+        "failed_count": int(summary["failed_count"]),
+        "partial_count": int(summary["partial_count"]),
+        "new_publication_count": int(run.new_pub_count or 0),
+        "reused_existing_run": reused_existing_run,
+        "idempotency_key": idempotency_key,
+        "safety_state": safety_state,
+    }
+
+
+async def _load_safety_state(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+) -> dict[str, Any]:
+    user_settings = await user_settings_service.get_or_create_settings(
+        db_session,
+        user_id=user_id,
+    )
+    previous_safety_state = run_safety_service.get_safety_event_context(user_settings)
+    if run_safety_service.clear_expired_cooldown(user_settings):
+        await db_session.commit()
+        await db_session.refresh(user_settings)
+        logger.info(
+            "api.runs.safety_cooldown_cleared",
+            extra={
+                "event": "api.runs.safety_cooldown_cleared",
+                "user_id": user_id,
+                "reason": previous_safety_state.get("cooldown_reason"),
+                "cooldown_until": previous_safety_state.get("cooldown_until"),
+                "metric_name": "api_runs_safety_cooldown_cleared_total",
+                "metric_value": 1,
+            },
+    )
+    return run_safety_service.get_safety_state_payload(user_settings)
+
+
+def _raise_manual_runs_disabled(*, user_id: int, safety_state: dict[str, Any]) -> None:
+    logger.warning(
+        "api.runs.manual_blocked_policy",
+        extra={
+            "event": "api.runs.manual_blocked_policy",
+            "user_id": user_id,
+            "policy": {"manual_run_allowed": False},
+            "safety_state": safety_state,
+            "metric_name": "api_runs_manual_blocked_policy_total",
+            "metric_value": 1,
+        },
+    )
+    raise ApiException(
+        status_code=403,
+        code="manual_runs_disabled",
+        message="Manual checks are disabled by server policy.",
+        details={
+            "policy": {"manual_run_allowed": False},
+            "safety_state": safety_state,
+        },
+    )
+
+
+async def _reused_manual_run_payload(
+    db_session: AsyncSession,
+    *,
+    request: Request,
+    user_id: int,
+    idempotency_key: str | None,
+    safety_state: dict[str, Any],
+) -> dict[str, Any] | None:
+    if idempotency_key is None:
+        return None
+    previous_run = await run_service.get_manual_run_by_idempotency_key(
+        db_session,
+        user_id=user_id,
+        idempotency_key=idempotency_key,
+    )
+    if previous_run is None:
+        return None
+    if previous_run.status == RunStatus.RUNNING:
+        raise ApiException(
+            status_code=409,
+            code="run_in_progress",
+            message="A run with this idempotency key is still in progress.",
+            details={"run_id": int(previous_run.id), "idempotency_key": idempotency_key},
+        )
+    return success_payload(
+        request,
+        data=_manual_run_payload_from_run(
+            previous_run,
+            idempotency_key=idempotency_key,
+            reused_existing_run=True,
+            safety_state=safety_state,
+        ),
+    )
+
+
+async def _run_ingestion_for_manual(
+    db_session: AsyncSession,
+    *,
+    ingest_service: ingestion_service.ScholarIngestionService,
+    user_id: int,
+    idempotency_key: str | None,
+):
+    user_settings = await user_settings_service.get_or_create_settings(
+        db_session,
+        user_id=user_id,
+    )
+    return await ingest_service.run_for_user(
+        db_session,
+        user_id=user_id,
+        trigger_type=RunTriggerType.MANUAL,
+        request_delay_seconds=user_settings.request_delay_seconds,
+        network_error_retries=settings.ingestion_network_error_retries,
+        retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
+        max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
+        page_size=settings.ingestion_page_size,
+        auto_queue_continuations=settings.ingestion_continuation_queue_enabled,
+        queue_delay_seconds=settings.ingestion_continuation_base_delay_seconds,
+        idempotency_key=idempotency_key,
+        alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
+        alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
+        alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
+    )
+
+
+async def _recover_integrity_error(
+    db_session: AsyncSession,
+    *,
+    request: Request,
+    user_id: int,
+    idempotency_key: str | None,
+    original_exc: IntegrityError,
+) -> dict[str, Any]:
+    if idempotency_key is None:
+        logger.exception(
+            "api.runs.manual_integrity_error",
+            extra={"event": "api.runs.manual_integrity_error", "user_id": user_id},
+        )
+        raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
+    existing_run = await run_service.get_manual_run_by_idempotency_key(
+        db_session,
+        user_id=user_id,
+        idempotency_key=idempotency_key,
+    )
+    if existing_run is None:
+        logger.exception(
+            "api.runs.manual_integrity_error",
+            extra={"event": "api.runs.manual_integrity_error", "user_id": user_id},
+        )
+        raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
+    if existing_run.status == RunStatus.RUNNING:
+        raise ApiException(
+            status_code=409,
+            code="run_in_progress",
+            message="A run with this idempotency key is still in progress.",
+            details={"run_id": int(existing_run.id), "idempotency_key": idempotency_key},
+        ) from original_exc
+    return success_payload(
+        request,
+        data=_manual_run_payload_from_run(
+            existing_run,
+            idempotency_key=idempotency_key,
+            reused_existing_run=True,
+            safety_state=await _load_safety_state(db_session, user_id=user_id),
+        ),
+    )
+
+
+async def _execute_manual_run(
+    db_session: AsyncSession,
+    *,
+    request: Request,
+    ingest_service: ingestion_service.ScholarIngestionService,
+    user_id: int,
+    idempotency_key: str | None,
+):
+    try:
+        return await _run_ingestion_for_manual(
+            db_session,
+            ingest_service=ingest_service,
+            user_id=user_id,
+            idempotency_key=idempotency_key,
+        )
+    except ingestion_service.RunAlreadyInProgressError as exc:
+        await db_session.rollback()
+        raise ApiException(
+            status_code=409,
+            code="run_in_progress",
+            message="A run is already in progress for this account.",
+        ) from exc
+    except ingestion_service.RunBlockedBySafetyPolicyError as exc:
+        await db_session.rollback()
+        _raise_manual_blocked_safety(exc=exc, user_id=user_id)
+    except IntegrityError as exc:
+        await db_session.rollback()
+        return await _recover_integrity_error(
+            db_session,
+            request=request,
+            user_id=user_id,
+            idempotency_key=idempotency_key,
+            original_exc=exc,
+        )
+    except Exception as exc:
+        await db_session.rollback()
+        _raise_manual_failed(exc=exc, user_id=user_id)
+
+
+def _raise_manual_blocked_safety(*, exc, user_id: int) -> None:
+    logger.info(
+        "api.runs.manual_blocked_safety",
+        extra={
+            "event": "api.runs.manual_blocked_safety",
+            "user_id": user_id,
+            "reason": exc.safety_state.get("cooldown_reason"),
+            "cooldown_until": exc.safety_state.get("cooldown_until"),
+            "cooldown_remaining_seconds": exc.safety_state.get("cooldown_remaining_seconds"),
+            "metric_name": "api_runs_manual_blocked_safety_total",
+            "metric_value": 1,
+        },
+    )
+    raise ApiException(
+        status_code=429,
+        code=exc.code,
+        message=exc.message,
+        details={"safety_state": exc.safety_state},
+    ) from exc
+
+
+def _raise_manual_failed(*, exc: Exception, user_id: int) -> None:
+    logger.exception(
+        "api.runs.manual_failed",
+        extra={"event": "api.runs.manual_failed", "user_id": user_id},
+    )
+    raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from exc
+
+
+def _manual_run_success_payload(
+    *,
+    run_summary,
+    idempotency_key: str | None,
+    safety_state: dict[str, Any],
+) -> dict[str, Any]:
+    return {
+        "run_id": run_summary.crawl_run_id,
+        "status": run_summary.status.value,
+        "scholar_count": run_summary.scholar_count,
+        "succeeded_count": run_summary.succeeded_count,
+        "failed_count": run_summary.failed_count,
+        "partial_count": run_summary.partial_count,
+        "new_publication_count": run_summary.new_publication_count,
+        "reused_existing_run": False,
+        "idempotency_key": idempotency_key,
+        "safety_state": safety_state,
+    }
+
+
+@router.get(
+    "",
+    response_model=RunsListEnvelope,
+)
+async def list_runs(
+    request: Request,
+    failed_only: bool = Query(default=False),
+    limit: int = Query(default=100, ge=1, le=500),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    runs = await run_service.list_runs_for_user(
+        db_session,
+        user_id=current_user.id,
+        limit=limit,
+        failed_only=failed_only,
+    )
+    safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    return success_payload(
+        request,
+        data={
+            "runs": [_serialize_run(run) for run in runs],
+            "safety_state": safety_state,
+        },
+    )
+
+
+@router.get(
+    "/{run_id}",
+    response_model=RunDetailEnvelope,
+)
+async def get_run(
+    run_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    run = await run_service.get_run_for_user(
+        db_session,
+        user_id=current_user.id,
+        run_id=run_id,
+    )
+    if run is None:
+        raise ApiException(
+            status_code=404,
+            code="run_not_found",
+            message="Run not found.",
+        )
+    error_log = run.error_log if isinstance(run.error_log, dict) else {}
+    scholar_results = error_log.get("scholar_results")
+    if not isinstance(scholar_results, list):
+        scholar_results = []
+    safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    return success_payload(
+        request,
+        data={
+            "run": _serialize_run(run),
+            "summary": run_service.extract_run_summary(error_log),
+            "scholar_results": [_normalize_scholar_result(item) for item in scholar_results],
+            "safety_state": safety_state,
+        },
+    )
+
+
+@router.post(
+    "/{run_id}/cancel",
+    response_model=RunDetailEnvelope,
+)
+async def cancel_run(
+    run_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    run = await run_service.get_run_for_user(
+        db_session,
+        user_id=current_user.id,
+        run_id=run_id,
+    )
+    if run is None:
+        raise ApiException(
+            status_code=404,
+            code="run_not_found",
+            message="Run not found.",
+        )
+        
+    if run.status == RunStatus.RUNNING:
+        run.status = RunStatus.CANCELED
+        await db_session.commit()
+        await db_session.refresh(run)
+
+    error_log = run.error_log if isinstance(run.error_log, dict) else {}
+    scholar_results = error_log.get("scholar_results")
+    if not isinstance(scholar_results, list):
+        scholar_results = []
+        
+    safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    
+    return success_payload(
+        request,
+        data={
+            "run": _serialize_run(run),
+            "summary": run_service.extract_run_summary(error_log),
+            "scholar_results": [_normalize_scholar_result(item) for item in scholar_results],
+            "safety_state": safety_state,
+        },
+    )
+
+
+@router.post(
+    "/manual",
+    response_model=ManualRunEnvelope,
+)
+async def run_manual(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+    ingest_service: ingestion_service.ScholarIngestionService = Depends(get_ingestion_service),
+):
+    safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    if not settings.ingestion_manual_run_allowed:
+        _raise_manual_runs_disabled(user_id=current_user.id, safety_state=safety_state)
+
+    idempotency_key = _normalize_idempotency_key(request.headers.get(IDEMPOTENCY_HEADER))
+    reused_payload = await _reused_manual_run_payload(
+        db_session,
+        request=request,
+        user_id=current_user.id,
+        idempotency_key=idempotency_key,
+        safety_state=safety_state,
+    )
+    if reused_payload is not None:
+        return reused_payload
+
+    run_summary = await _execute_manual_run(
+        db_session,
+        request=request,
+        ingest_service=ingest_service,
+        user_id=current_user.id,
+        idempotency_key=idempotency_key,
+    )
+    if isinstance(run_summary, dict):
+        return run_summary
+
+    current_safety_state = await _load_safety_state(
+        db_session,
+        user_id=current_user.id,
+    )
+    return success_payload(
+        request,
+        data=_manual_run_success_payload(
+            run_summary=run_summary,
+            idempotency_key=idempotency_key,
+            safety_state=current_safety_state,
+        ),
+    )
+
+
+
+@router.get(
+    "/queue/items",
+    response_model=QueueListEnvelope,
+)
+async def list_queue_items(
+    request: Request,
+    limit: int = Query(default=200, ge=1, le=500),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    items = await run_service.list_queue_items_for_user(
+        db_session,
+        user_id=current_user.id,
+        limit=limit,
+    )
+    return success_payload(
+        request,
+        data={
+            "queue_items": [_serialize_queue_item(item) for item in items],
+        },
+    )
+
+
+@router.post(
+    "/queue/{queue_item_id}/retry",
+    response_model=QueueItemEnvelope,
+)
+async def retry_queue_item(
+    queue_item_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    try:
+        queue_item = await run_service.retry_queue_item_for_user(
+            db_session,
+            user_id=current_user.id,
+            queue_item_id=queue_item_id,
+        )
+    except run_service.QueueTransitionError as exc:
+        raise ApiException(
+            status_code=409,
+            code=exc.code,
+            message=exc.message,
+            details={"current_status": exc.current_status},
+        ) from exc
+    if queue_item is None:
+        raise ApiException(
+            status_code=404,
+            code="queue_item_not_found",
+            message="Queue item not found.",
+        )
+    return success_payload(
+        request,
+        data=_serialize_queue_item(queue_item),
+    )
+
+
+@router.post(
+    "/queue/{queue_item_id}/drop",
+    response_model=QueueItemEnvelope,
+)
+async def drop_queue_item(
+    queue_item_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    try:
+        dropped = await run_service.drop_queue_item_for_user(
+            db_session,
+            user_id=current_user.id,
+            queue_item_id=queue_item_id,
+        )
+    except run_service.QueueTransitionError as exc:
+        raise ApiException(
+            status_code=409,
+            code=exc.code,
+            message=exc.message,
+            details={"current_status": exc.current_status},
+        ) from exc
+    if dropped is None:
+        raise ApiException(
+            status_code=404,
+            code="queue_item_not_found",
+            message="Queue item not found.",
+        )
+    return success_payload(
+        request,
+        data=_serialize_queue_item(dropped),
+    )
+
+
+@router.delete(
+    "/queue/{queue_item_id}",
+    response_model=QueueClearEnvelope,
+)
+async def clear_queue_item(
+    queue_item_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    try:
+        deleted = await run_service.clear_queue_item_for_user(
+            db_session,
+            user_id=current_user.id,
+            queue_item_id=queue_item_id,
+        )
+    except run_service.QueueTransitionError as exc:
+        raise ApiException(
+            status_code=409,
+            code=exc.code,
+            message=exc.message,
+            details={"current_status": exc.current_status},
+        ) from exc
+    if deleted is None:
+        raise ApiException(
+            status_code=404,
+            code="queue_item_not_found",
+            message="Queue item not found.",
+        )
+    return success_payload(
+        request,
+        data={
+            "queue_item_id": deleted.queue_item_id,
+            "previous_status": deleted.previous_status,
+            "status": "cleared",
+            "message": "Queue item cleared.",
+        },
+    )
+
+
+@router.get("/{run_id}/stream")
+async def stream_run_events(
+    run_id: int,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    run = await run_service.get_run_for_user(
+        db_session,
+        user_id=current_user.id,
+        run_id=run_id,
+    )
+    if run is None:
+        raise ApiException(
+            status_code=404,
+            code="run_not_found",
+            message="Run not found.",
+        )
+    return StreamingResponse(
+        event_generator(run_id),
+        media_type="text/event-stream"
+    )

build/lib/app/api/routers/scholars.py

@@ -0,0 +1,557 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+
+from fastapi import APIRouter, Depends, File, Query, Request, UploadFile
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_current_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from app.api.runtime_deps import get_scholar_source
+from app.api.schemas import (
+    DataExportEnvelope,
+    DataImportEnvelope,
+    DataImportRequest,
+    MessageEnvelope,
+    ScholarCreateRequest,
+    ScholarEnvelope,
+    ScholarImageUrlUpdateRequest,
+    ScholarSearchEnvelope,
+    ScholarsListEnvelope,
+)
+from app.db.models import User
+from app.db.session import get_db_session
+from app.services.domains.portability import application as import_export_service
+from app.services.domains.scholars import application as scholar_service
+from app.services.domains.scholar.source import ScholarSource
+from app.settings import settings
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/scholars", tags=["api-scholars"])
+
+
+def _uploaded_image_media_path(scholar_profile_id: int) -> str:
+    return f"/scholar-images/{scholar_profile_id}/upload"
+
+
+def _serialize_scholar(profile) -> dict[str, object]:
+    uploaded_image_url = None
+    if profile.profile_image_upload_path:
+        uploaded_image_url = _uploaded_image_media_path(int(profile.id))
+
+    profile_image_url, profile_image_source = scholar_service.resolve_profile_image(
+        profile,
+        uploaded_image_url=uploaded_image_url,
+    )
+
+    return {
+        "id": int(profile.id),
+        "scholar_id": profile.scholar_id,
+        "display_name": profile.display_name,
+        "profile_image_url": profile_image_url,
+        "profile_image_source": profile_image_source,
+        "is_enabled": bool(profile.is_enabled),
+        "baseline_completed": bool(profile.baseline_completed),
+        "last_run_dt": profile.last_run_dt,
+        "last_run_status": (
+            profile.last_run_status.value if profile.last_run_status is not None else None
+        ),
+    }
+
+
+async def _hydrate_scholar_metadata_if_needed(
+    db_session: AsyncSession,
+    *,
+    profile,
+    source: ScholarSource,
+    user_id: int,
+):
+    try:
+        if not profile.profile_image_url or not (profile.display_name or "").strip():
+            return await asyncio.wait_for(
+                scholar_service.hydrate_profile_metadata(
+                    db_session,
+                    profile=profile,
+                    source=source,
+                ),
+                timeout=5.0,
+            )
+    except Exception:
+        logger.warning(
+            "api.scholars.create_metadata_hydration_failed",
+            extra={
+                "event": "api.scholars.create_metadata_hydration_failed",
+                "user_id": user_id,
+                "scholar_profile_id": profile.id,
+            },
+        )
+    return profile
+
+
+def _search_kwargs() -> dict[str, object]:
+    return {
+        "network_error_retries": settings.ingestion_network_error_retries,
+        "retry_backoff_seconds": settings.ingestion_retry_backoff_seconds,
+        "search_enabled": settings.scholar_name_search_enabled,
+        "cache_ttl_seconds": settings.scholar_name_search_cache_ttl_seconds,
+        "blocked_cache_ttl_seconds": settings.scholar_name_search_blocked_cache_ttl_seconds,
+        "cache_max_entries": settings.scholar_name_search_cache_max_entries,
+        "min_interval_seconds": settings.scholar_name_search_min_interval_seconds,
+        "interval_jitter_seconds": settings.scholar_name_search_interval_jitter_seconds,
+        "cooldown_block_threshold": settings.scholar_name_search_cooldown_block_threshold,
+        "cooldown_seconds": settings.scholar_name_search_cooldown_seconds,
+        "retry_alert_threshold": settings.scholar_name_search_alert_retry_count_threshold,
+        "cooldown_rejection_alert_threshold": (
+            settings.scholar_name_search_alert_cooldown_rejections_threshold
+        ),
+    }
+
+
+def _search_response_data(query: str, parsed) -> dict[str, object]:
+    return {
+        "query": query.strip(),
+        "state": parsed.state.value,
+        "state_reason": parsed.state_reason,
+        "action_hint": scholar_service.scrape_state_hint(
+            state=parsed.state,
+            state_reason=parsed.state_reason,
+        ),
+        "candidates": [
+            {
+                "scholar_id": item.scholar_id,
+                "display_name": item.display_name,
+                "affiliation": item.affiliation,
+                "email_domain": item.email_domain,
+                "cited_by_count": item.cited_by_count,
+                "interests": item.interests,
+                "profile_url": item.profile_url,
+                "profile_image_url": item.profile_image_url,
+            }
+            for item in parsed.candidates
+        ],
+        "warnings": parsed.warnings,
+    }
+
+
+async def _read_uploaded_image(image: UploadFile) -> bytes:
+    try:
+        return await image.read()
+    finally:
+        await image.close()
+
+
+async def _require_user_profile(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    scholar_profile_id: int,
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+    return profile
+
+
+@router.get(
+    "",
+    response_model=ScholarsListEnvelope,
+)
+async def list_scholars(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    scholars = await scholar_service.list_scholars_for_user(
+        db_session,
+        user_id=current_user.id,
+    )
+    return success_payload(
+        request,
+        data={
+            "scholars": [_serialize_scholar(profile) for profile in scholars],
+        },
+    )
+
+
+@router.get(
+    "/export",
+    response_model=DataExportEnvelope,
+)
+async def export_scholars_and_publications(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    data = await import_export_service.export_user_data(
+        db_session,
+        user_id=current_user.id,
+    )
+    return success_payload(request, data=data)
+
+
+@router.post(
+    "/import",
+    response_model=DataImportEnvelope,
+)
+async def import_scholars_and_publications(
+    payload: DataImportRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    if (
+        payload.schema_version is not None
+        and int(payload.schema_version) != import_export_service.EXPORT_SCHEMA_VERSION
+    ):
+        raise ApiException(
+            status_code=400,
+            code="invalid_import_schema_version",
+            message=(
+                "Import schema version is not supported. "
+                f"Expected {import_export_service.EXPORT_SCHEMA_VERSION}."
+            ),
+        )
+    try:
+        result = await import_export_service.import_user_data(
+            db_session,
+            user_id=current_user.id,
+            scholars=[item.model_dump() for item in payload.scholars],
+            publications=[item.model_dump() for item in payload.publications],
+        )
+    except import_export_service.ImportExportError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_import_payload",
+            message=str(exc),
+        ) from exc
+    logger.info(
+        "api.scholars.imported",
+        extra={
+            "event": "api.scholars.imported",
+            "user_id": current_user.id,
+            **result,
+        },
+    )
+    return success_payload(request, data=result)
+
+
+@router.post(
+    "",
+    response_model=ScholarEnvelope,
+    status_code=201,
+)
+async def create_scholar(
+    payload: ScholarCreateRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    source: ScholarSource = Depends(get_scholar_source),
+    current_user: User = Depends(get_api_current_user),
+):
+    try:
+        created = await scholar_service.create_scholar_for_user(
+            db_session,
+            user_id=current_user.id,
+            scholar_id=payload.scholar_id,
+            display_name="",
+            profile_image_url=payload.profile_image_url,
+        )
+    except scholar_service.ScholarServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_scholar",
+            message=str(exc),
+        ) from exc
+    logger.info(
+        "api.scholars.created",
+        extra={
+            "event": "api.scholars.created",
+            "user_id": current_user.id,
+            "scholar_profile_id": created.id,
+        },
+    )
+    created = await _hydrate_scholar_metadata_if_needed(
+        db_session,
+        profile=created,
+        source=source,
+        user_id=current_user.id,
+    )
+
+    return success_payload(
+        request,
+        data=_serialize_scholar(created),
+    )
+
+
+@router.get(
+    "/search",
+    response_model=ScholarSearchEnvelope,
+)
+async def search_scholars(
+    request: Request,
+    query: str = Query(..., min_length=2, max_length=120),
+    limit: int = Query(10, ge=1, le=25),
+    db_session: AsyncSession = Depends(get_db_session),
+    source: ScholarSource = Depends(get_scholar_source),
+    current_user: User = Depends(get_api_current_user),
+):
+    try:
+        parsed = await scholar_service.search_author_candidates(
+            source=source,
+            db_session=db_session,
+            query=query,
+            limit=limit,
+            **_search_kwargs(),
+        )
+    except scholar_service.ScholarServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_scholar_search",
+            message=str(exc),
+        ) from exc
+
+    logger.info(
+        "api.scholars.search.completed",
+        extra={
+            "event": "api.scholars.search.completed",
+            "user_id": current_user.id,
+            "query": query.strip(),
+            "candidate_count": len(parsed.candidates),
+            "state": parsed.state.value,
+        },
+    )
+    return success_payload(
+        request,
+        data=_search_response_data(query, parsed),
+    )
+
+
+@router.patch(
+    "/{scholar_profile_id}/toggle",
+    response_model=ScholarEnvelope,
+)
+async def toggle_scholar(
+    scholar_profile_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+    updated = await scholar_service.toggle_scholar_enabled(db_session, profile=profile)
+    logger.info(
+        "api.scholars.toggled",
+        extra={
+            "event": "api.scholars.toggled",
+            "user_id": current_user.id,
+            "scholar_profile_id": updated.id,
+            "is_enabled": updated.is_enabled,
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_scholar(updated),
+    )
+
+
+@router.delete(
+    "/{scholar_profile_id}",
+    response_model=MessageEnvelope,
+)
+async def delete_scholar(
+    scholar_profile_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+    await scholar_service.delete_scholar(
+        db_session,
+        profile=profile,
+        upload_dir=settings.scholar_image_upload_dir,
+    )
+    logger.info(
+        "api.scholars.deleted",
+        extra={
+            "event": "api.scholars.deleted",
+            "user_id": current_user.id,
+            "scholar_profile_id": scholar_profile_id,
+        },
+    )
+    return success_payload(
+        request,
+        data={"message": "Scholar deleted."},
+    )
+
+
+@router.put(
+    "/{scholar_profile_id}/image/url",
+    response_model=ScholarEnvelope,
+)
+async def update_scholar_image_url(
+    scholar_profile_id: int,
+    payload: ScholarImageUrlUpdateRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+
+    try:
+        updated = await scholar_service.set_profile_image_override_url(
+            db_session,
+            profile=profile,
+            image_url=payload.image_url,
+            upload_dir=settings.scholar_image_upload_dir,
+        )
+    except scholar_service.ScholarServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_scholar_image",
+            message=str(exc),
+        ) from exc
+
+    logger.info(
+        "api.scholars.image_url_updated",
+        extra={
+            "event": "api.scholars.image_url_updated",
+            "user_id": current_user.id,
+            "scholar_profile_id": updated.id,
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_scholar(updated),
+    )
+
+
+@router.post(
+    "/{scholar_profile_id}/image/upload",
+    response_model=ScholarEnvelope,
+)
+async def upload_scholar_image(
+    scholar_profile_id: int,
+    request: Request,
+    image: UploadFile = File(...),
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await _require_user_profile(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+
+    image_bytes = await _read_uploaded_image(image)
+    try:
+        updated = await scholar_service.set_profile_image_upload(
+            db_session,
+            profile=profile,
+            content_type=image.content_type,
+            image_bytes=image_bytes,
+            upload_dir=settings.scholar_image_upload_dir,
+            max_upload_bytes=settings.scholar_image_upload_max_bytes,
+        )
+    except scholar_service.ScholarServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_scholar_image",
+            message=str(exc),
+        ) from exc
+
+    image_size = len(image_bytes)
+    logger.info(
+        "api.scholars.image_uploaded",
+        extra={
+            "event": "api.scholars.image_uploaded",
+            "user_id": current_user.id,
+            "scholar_profile_id": updated.id,
+            "content_type": image.content_type,
+            "size_bytes": image_size,
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_scholar(updated),
+    )
+
+
+@router.delete(
+    "/{scholar_profile_id}/image",
+    response_model=ScholarEnvelope,
+)
+async def clear_scholar_image_customization(
+    scholar_profile_id: int,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    profile = await scholar_service.get_user_scholar_by_id(
+        db_session,
+        user_id=current_user.id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    if profile is None:
+        raise ApiException(
+            status_code=404,
+            code="scholar_not_found",
+            message="Scholar not found.",
+        )
+
+    updated = await scholar_service.clear_profile_image_customization(
+        db_session,
+        profile=profile,
+        upload_dir=settings.scholar_image_upload_dir,
+    )
+    logger.info(
+        "api.scholars.image_customization_cleared",
+        extra={
+            "event": "api.scholars.image_customization_cleared",
+            "user_id": current_user.id,
+            "scholar_profile_id": updated.id,
+        },
+    )
+    return success_payload(
+        request,
+        data=_serialize_scholar(updated),
+    )
+

build/lib/app/api/routers/settings.py

@@ -0,0 +1,189 @@
+from __future__ import annotations
+
+import logging
+
+from fastapi import APIRouter, Depends, Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.api.deps import get_api_current_user
+from app.api.errors import ApiException
+from app.api.responses import success_payload
+from app.api.schemas import SettingsEnvelope, SettingsUpdateRequest
+from app.db.models import User
+from app.db.session import get_db_session
+from app.services.domains.ingestion import safety as run_safety_service
+from app.services.domains.settings import application as user_settings_service
+from app.settings import settings as settings_module
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/settings", tags=["api-settings"])
+
+
+def _serialize_settings(user_settings) -> dict[str, object]:
+    min_run_interval_minutes, min_request_delay_seconds = _minimum_policy()
+    return {
+        "auto_run_enabled": bool(user_settings.auto_run_enabled) and settings_module.ingestion_automation_allowed,
+        "run_interval_minutes": int(user_settings.run_interval_minutes),
+        "request_delay_seconds": int(user_settings.request_delay_seconds),
+        "nav_visible_pages": list(user_settings.nav_visible_pages or []),
+        "policy": {
+            "min_run_interval_minutes": min_run_interval_minutes,
+            "min_request_delay_seconds": min_request_delay_seconds,
+            "automation_allowed": bool(settings_module.ingestion_automation_allowed),
+            "manual_run_allowed": bool(settings_module.ingestion_manual_run_allowed),
+            "blocked_failure_threshold": max(1, int(settings_module.ingestion_alert_blocked_failure_threshold)),
+            "network_failure_threshold": max(1, int(settings_module.ingestion_alert_network_failure_threshold)),
+            "cooldown_blocked_seconds": max(60, int(settings_module.ingestion_safety_cooldown_blocked_seconds)),
+            "cooldown_network_seconds": max(60, int(settings_module.ingestion_safety_cooldown_network_seconds)),
+        },
+        "safety_state": run_safety_service.get_safety_state_payload(user_settings),
+        "openalex_api_key": user_settings.openalex_api_key,
+        "crossref_api_token": user_settings.crossref_api_token,
+        "crossref_api_mailto": user_settings.crossref_api_mailto,
+    }
+
+
+def _minimum_policy() -> tuple[int, int]:
+    min_run_interval_minutes = user_settings_service.resolve_run_interval_minimum(
+        settings_module.ingestion_min_run_interval_minutes
+    )
+    min_request_delay_seconds = user_settings_service.resolve_request_delay_minimum(
+        settings_module.ingestion_min_request_delay_seconds
+    )
+    return min_run_interval_minutes, min_request_delay_seconds
+
+
+def _parse_settings_payload(payload: SettingsUpdateRequest, user_settings) -> tuple[int, int, list[str]]:
+    min_run_interval_minutes, min_request_delay_seconds = _minimum_policy()
+    parsed_interval = user_settings_service.parse_run_interval_minutes(
+        str(payload.run_interval_minutes),
+        minimum=min_run_interval_minutes,
+    )
+    parsed_delay = user_settings_service.parse_request_delay_seconds(
+        str(payload.request_delay_seconds),
+        minimum=min_request_delay_seconds,
+    )
+    parsed_nav_visible_pages = user_settings_service.parse_nav_visible_pages(
+        payload.nav_visible_pages
+        if payload.nav_visible_pages is not None
+        else list(user_settings.nav_visible_pages or user_settings_service.DEFAULT_NAV_VISIBLE_PAGES)
+    )
+    return parsed_interval, parsed_delay, parsed_nav_visible_pages
+
+
+async def _clear_expired_cooldown_with_log(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    user_settings,
+) -> None:
+    previous_safety_state = run_safety_service.get_safety_event_context(user_settings)
+    if not run_safety_service.clear_expired_cooldown(user_settings):
+        return
+    await db_session.commit()
+    await db_session.refresh(user_settings)
+    logger.info(
+        "api.settings.safety_cooldown_cleared",
+        extra={
+            "event": "api.settings.safety_cooldown_cleared",
+            "user_id": user_id,
+            "reason": previous_safety_state.get("cooldown_reason"),
+            "cooldown_until": previous_safety_state.get("cooldown_until"),
+            "metric_name": "api_settings_safety_cooldown_cleared_total",
+            "metric_value": 1,
+        },
+    )
+
+
+def _effective_auto_run_enabled(payload: SettingsUpdateRequest) -> bool:
+    return bool(payload.auto_run_enabled) and settings_module.ingestion_automation_allowed
+
+
+def _log_settings_update(*, user_id: int, updated) -> None:
+    logger.info(
+        "api.settings.updated",
+        extra={
+            "event": "api.settings.updated",
+            "user_id": user_id,
+            "auto_run_enabled": updated.auto_run_enabled,
+            "run_interval_minutes": updated.run_interval_minutes,
+            "request_delay_seconds": updated.request_delay_seconds,
+            "nav_visible_pages": updated.nav_visible_pages,
+            "openalex_api_key": "SET" if updated.openalex_api_key else "UNSET",
+        },
+    )
+
+
+@router.get(
+    "",
+    response_model=SettingsEnvelope,
+)
+async def get_settings(
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    user_settings = await user_settings_service.get_or_create_settings(
+        db_session,
+        user_id=current_user.id,
+    )
+    await _clear_expired_cooldown_with_log(
+        db_session,
+        user_id=current_user.id,
+        user_settings=user_settings,
+    )
+    return success_payload(
+        request,
+        data=_serialize_settings(user_settings),
+    )
+
+
+@router.put(
+    "",
+    response_model=SettingsEnvelope,
+)
+async def update_settings(
+    payload: SettingsUpdateRequest,
+    request: Request,
+    db_session: AsyncSession = Depends(get_db_session),
+    current_user: User = Depends(get_api_current_user),
+):
+    user_settings = await user_settings_service.get_or_create_settings(
+        db_session,
+        user_id=current_user.id,
+    )
+
+    try:
+        parsed_interval, parsed_delay, parsed_nav_visible_pages = _parse_settings_payload(
+            payload,
+            user_settings,
+        )
+    except user_settings_service.UserSettingsServiceError as exc:
+        raise ApiException(
+            status_code=400,
+            code="invalid_settings",
+            message=str(exc),
+        ) from exc
+
+    updated = await user_settings_service.update_settings(
+        db_session,
+        settings=user_settings,
+        auto_run_enabled=_effective_auto_run_enabled(payload),
+        run_interval_minutes=parsed_interval,
+        request_delay_seconds=parsed_delay,
+        nav_visible_pages=parsed_nav_visible_pages,
+        openalex_api_key=payload.openalex_api_key,
+        crossref_api_token=payload.crossref_api_token,
+        crossref_api_mailto=payload.crossref_api_mailto,
+    )
+    await _clear_expired_cooldown_with_log(
+        db_session,
+        user_id=current_user.id,
+        user_settings=updated,
+    )
+    _log_settings_update(user_id=current_user.id, updated=updated)
+    return success_payload(
+        request,
+        data=_serialize_settings(updated),
+    )

build/lib/app/api/runtime_deps.py

@@ -0,0 +1,16 @@
+from __future__ import annotations
+
+from fastapi import Depends
+
+from app.services.domains.ingestion import application as ingestion_service
+from app.services.domains.scholar import source as scholar_source_service
+
+
+def get_scholar_source() -> scholar_source_service.ScholarSource:
+    return scholar_source_service.LiveScholarSource()
+
+
+def get_ingestion_service(
+    source: scholar_source_service.ScholarSource = Depends(get_scholar_source),
+) -> ingestion_service.ScholarIngestionService:
+    return ingestion_service.ScholarIngestionService(source=source)

build/lib/app/api/schemas.py

@@ -0,0 +1,884 @@
+from __future__ import annotations
+
+from datetime import datetime
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+
+class ApiMeta(BaseModel):
+    request_id: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ApiErrorData(BaseModel):
+    code: str
+    message: str
+    details: Any | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ApiErrorEnvelope(BaseModel):
+    error: ApiErrorData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MessageData(BaseModel):
+    message: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MessageEnvelope(BaseModel):
+    data: MessageData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class SessionUserData(BaseModel):
+    id: int
+    email: str
+    is_admin: bool
+    is_active: bool
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AuthMeData(BaseModel):
+    authenticated: bool
+    csrf_token: str
+    user: SessionUserData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AuthMeEnvelope(BaseModel):
+    data: AuthMeData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class CsrfBootstrapData(BaseModel):
+    csrf_token: str
+    authenticated: bool
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class CsrfBootstrapEnvelope(BaseModel):
+    data: CsrfBootstrapData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class LoginRequest(BaseModel):
+    email: str
+    password: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class LoginData(BaseModel):
+    authenticated: bool
+    csrf_token: str
+    user: SessionUserData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class LoginEnvelope(BaseModel):
+    data: LoginData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ChangePasswordRequest(BaseModel):
+    current_password: str
+    new_password: str
+    confirm_password: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarItemData(BaseModel):
+    id: int
+    scholar_id: str
+    display_name: str | None
+    profile_image_url: str | None
+    profile_image_source: str
+    is_enabled: bool
+    baseline_completed: bool
+    last_run_dt: datetime | None
+    last_run_status: str | None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarsListData(BaseModel):
+    scholars: list[ScholarItemData]
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarsListEnvelope(BaseModel):
+    data: ScholarsListData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarCreateRequest(BaseModel):
+    scholar_id: str
+    profile_image_url: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarEnvelope(BaseModel):
+    data: ScholarItemData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarSearchCandidateData(BaseModel):
+    scholar_id: str
+    display_name: str
+    affiliation: str | None
+    email_domain: str | None
+    cited_by_count: int | None
+    interests: list[str] = Field(default_factory=list)
+    profile_url: str
+    profile_image_url: str | None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarSearchData(BaseModel):
+    query: str
+    state: str
+    state_reason: str
+    action_hint: str | None = None
+    candidates: list[ScholarSearchCandidateData]
+    warnings: list[str] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarSearchEnvelope(BaseModel):
+    data: ScholarSearchData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarImageUrlUpdateRequest(BaseModel):
+    image_url: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScholarExportItemData(BaseModel):
+    scholar_id: str
+    display_name: str | None = None
+    is_enabled: bool = True
+    profile_image_override_url: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class PublicationExportItemData(BaseModel):
+    scholar_id: str
+    cluster_id: str | None = None
+    fingerprint_sha256: str | None = None
+    title: str
+    year: int | None = None
+    citation_count: int = 0
+    author_text: str | None = None
+    venue_text: str | None = None
+    pub_url: str | None = None
+    pdf_url: str | None = None
+    is_read: bool = False
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DataExportData(BaseModel):
+    schema_version: int
+    exported_at: str
+    scholars: list[ScholarExportItemData]
+    publications: list[PublicationExportItemData]
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DataExportEnvelope(BaseModel):
+    data: DataExportData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DataImportRequest(BaseModel):
+    schema_version: int | None = None
+    scholars: list[ScholarExportItemData] = Field(default_factory=list)
+    publications: list[PublicationExportItemData] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DataImportResultData(BaseModel):
+    scholars_created: int
+    scholars_updated: int
+    publications_created: int
+    publications_updated: int
+    links_created: int
+    links_updated: int
+    skipped_records: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DataImportEnvelope(BaseModel):
+    data: DataImportResultData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunListItemData(BaseModel):
+    id: int
+    trigger_type: str
+    status: str
+    start_dt: datetime
+    end_dt: datetime | None
+    scholar_count: int
+    new_publication_count: int
+    failed_count: int
+    partial_count: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunsListData(BaseModel):
+    runs: list[RunListItemData]
+    safety_state: ScrapeSafetyStateData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunsListEnvelope(BaseModel):
+    data: RunsListData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunSummaryData(BaseModel):
+    succeeded_count: int
+    failed_count: int
+    partial_count: int
+    failed_state_counts: dict[str, int] = Field(default_factory=dict)
+    failed_reason_counts: dict[str, int] = Field(default_factory=dict)
+    scrape_failure_counts: dict[str, int] = Field(default_factory=dict)
+    retry_counts: dict[str, int] = Field(default_factory=dict)
+    alert_thresholds: dict[str, int] = Field(default_factory=dict)
+    alert_flags: dict[str, bool] = Field(default_factory=dict)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScrapeSafetyCountersData(BaseModel):
+    consecutive_blocked_runs: int = 0
+    consecutive_network_runs: int = 0
+    cooldown_entry_count: int = 0
+    blocked_start_count: int = 0
+    last_blocked_failure_count: int = 0
+    last_network_failure_count: int = 0
+    last_evaluated_run_id: int | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ScrapeSafetyStateData(BaseModel):
+    cooldown_active: bool
+    cooldown_reason: str | None = None
+    cooldown_reason_label: str | None = None
+    cooldown_until: datetime | None = None
+    cooldown_remaining_seconds: int = 0
+    recommended_action: str | None = None
+    counters: ScrapeSafetyCountersData = Field(default_factory=ScrapeSafetyCountersData)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunAttemptLogData(BaseModel):
+    attempt: int
+    cstart: int
+    state: str | None = None
+    state_reason: str | None = None
+    status_code: int | None = None
+    fetch_error: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunPageLogData(BaseModel):
+    page: int
+    cstart: int
+    state: str
+    state_reason: str | None = None
+    status_code: int | None = None
+    publication_count: int = 0
+    attempt_count: int = 0
+    has_show_more_button: bool = False
+    articles_range: str | None = None
+    warning_codes: list[str] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunDebugData(BaseModel):
+    status_code: int | None = None
+    final_url: str | None = None
+    fetch_error: str | None = None
+    body_sha256: str | None = None
+    body_length: int | None = None
+    has_show_more_button: bool | None = None
+    articles_range: str | None = None
+    state_reason: str | None = None
+    warning_codes: list[str] = Field(default_factory=list)
+    marker_counts_nonzero: dict[str, int] = Field(default_factory=dict)
+    page_logs: list[RunPageLogData] = Field(default_factory=list)
+    attempt_log: list[RunAttemptLogData] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunScholarResultData(BaseModel):
+    scholar_profile_id: int
+    scholar_id: str
+    state: str
+    state_reason: str | None = None
+    outcome: str
+    attempt_count: int = 0
+    publication_count: int = 0
+    start_cstart: int = 0
+    continuation_cstart: int | None = None
+    continuation_enqueued: bool = False
+    continuation_cleared: bool = False
+    warnings: list[str] = Field(default_factory=list)
+    error: str | None = None
+    debug: RunDebugData | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunDetailData(BaseModel):
+    run: RunListItemData
+    summary: RunSummaryData
+    scholar_results: list[RunScholarResultData]
+    safety_state: ScrapeSafetyStateData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RunDetailEnvelope(BaseModel):
+    data: RunDetailData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ManualRunData(BaseModel):
+    run_id: int
+    status: str
+    scholar_count: int
+    succeeded_count: int
+    failed_count: int
+    partial_count: int
+    new_publication_count: int
+    reused_existing_run: bool
+    idempotency_key: str | None = None
+    safety_state: ScrapeSafetyStateData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class ManualRunEnvelope(BaseModel):
+    data: ManualRunData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueItemData(BaseModel):
+    id: int
+    scholar_profile_id: int
+    scholar_label: str
+    status: str
+    reason: str
+    dropped_reason: str | None
+    attempt_count: int
+    resume_cstart: int
+    next_attempt_dt: datetime | None
+    updated_at: datetime
+    last_error: str | None
+    last_run_id: int | None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueListData(BaseModel):
+    queue_items: list[QueueItemData]
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueListEnvelope(BaseModel):
+    data: QueueListData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueItemEnvelope(BaseModel):
+    data: QueueItemData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueClearData(BaseModel):
+    queue_item_id: int
+    previous_status: str
+    status: str
+    message: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class QueueClearEnvelope(BaseModel):
+    data: QueueClearData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUserData(BaseModel):
+    id: int
+    email: str
+    is_active: bool
+    is_admin: bool
+    created_at: datetime
+    updated_at: datetime
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUsersListData(BaseModel):
+    users: list[AdminUserData]
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUsersListEnvelope(BaseModel):
+    data: AdminUsersListData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUserCreateRequest(BaseModel):
+    email: str
+    password: str
+    is_admin: bool = False
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUserActiveUpdateRequest(BaseModel):
+    is_active: bool
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminUserEnvelope(BaseModel):
+    data: AdminUserData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminResetPasswordRequest(BaseModel):
+    new_password: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbIntegrityCheckData(BaseModel):
+    name: str
+    count: int
+    severity: str
+    message: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbIntegrityData(BaseModel):
+    status: str
+    checked_at: datetime
+    failures: list[str] = Field(default_factory=list)
+    warnings: list[str] = Field(default_factory=list)
+    checks: list[AdminDbIntegrityCheckData] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbIntegrityEnvelope(BaseModel):
+    data: AdminDbIntegrityData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbRepairJobData(BaseModel):
+    id: int
+    job_name: str
+    requested_by: str | None
+    dry_run: bool
+    status: str
+    scope: dict[str, Any] = Field(default_factory=dict)
+    summary: dict[str, Any] = Field(default_factory=dict)
+    error_text: str | None
+    started_at: datetime | None
+    finished_at: datetime | None
+    created_at: datetime
+    updated_at: datetime
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbRepairJobsData(BaseModel):
+    jobs: list[AdminDbRepairJobData] = Field(default_factory=list)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminDbRepairJobsEnvelope(BaseModel):
+    data: AdminDbRepairJobsData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class DisplayIdentifierData(BaseModel):
+    kind: str
+    value: str
+    label: str
+    url: str | None
+    confidence_score: float = Field(ge=0.0, le=1.0)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueItemData(BaseModel):
+    publication_id: int
+    title: str
+    display_identifier: DisplayIdentifierData | None = None
+    pdf_url: str | None
+    status: str
+    attempt_count: int
+    last_failure_reason: str | None
+    last_failure_detail: str | None
+    last_source: str | None
+    requested_by_user_id: int | None
+    requested_by_email: str | None
+    queued_at: datetime | None
+    last_attempt_at: datetime | None
+    resolved_at: datetime | None
+    updated_at: datetime
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueData(BaseModel):
+    items: list[AdminPdfQueueItemData] = Field(default_factory=list)
+    total_count: int = 0
+    page: int = 1
+    page_size: int = 100
+    has_next: bool = False
+    has_prev: bool = False
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueEnvelope(BaseModel):
+    data: AdminPdfQueueData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueRequeueData(BaseModel):
+    publication_id: int
+    queued: bool
+    status: str
+    message: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueRequeueEnvelope(BaseModel):
+    data: AdminPdfQueueRequeueData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueBulkEnqueueData(BaseModel):
+    requested_count: int
+    queued_count: int
+    message: str
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminPdfQueueBulkEnqueueEnvelope(BaseModel):
+    data: AdminPdfQueueBulkEnqueueData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminRepairPublicationLinksRequest(BaseModel):
+    scope_mode: Literal["single_user", "all_users"] = "single_user"
+    user_id: int | None = Field(default=None, ge=1)
+    scholar_profile_ids: list[int] = Field(default_factory=list, max_length=200)
+    dry_run: bool = True
+    gc_orphan_publications: bool = False
+    requested_by: str | None = None
+    confirmation_text: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+    @model_validator(mode="after")
+    def validate_scope(self) -> "AdminRepairPublicationLinksRequest":
+        if self.scope_mode == "single_user" and self.user_id is None:
+            raise ValueError("user_id is required when scope_mode=single_user.")
+        if self.scope_mode == "all_users" and self.user_id is not None:
+            raise ValueError("user_id must be omitted when scope_mode=all_users.")
+        if not self.dry_run and self.scope_mode == "all_users":
+            expected = "REPAIR ALL USERS"
+            provided = (self.confirmation_text or "").strip()
+            if provided != expected:
+                raise ValueError(
+                    "confirmation_text must equal 'REPAIR ALL USERS' "
+                    "when applying a repair to all users."
+                )
+        return self
+
+
+class AdminRepairPublicationLinksResultData(BaseModel):
+    job_id: int
+    status: str
+    scope: dict[str, Any] = Field(default_factory=dict)
+    summary: dict[str, Any] = Field(default_factory=dict)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class AdminRepairPublicationLinksEnvelope(BaseModel):
+    data: AdminRepairPublicationLinksResultData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class SettingsPolicyData(BaseModel):
+    min_run_interval_minutes: int
+    min_request_delay_seconds: int
+    automation_allowed: bool
+    manual_run_allowed: bool
+    blocked_failure_threshold: int
+    network_failure_threshold: int
+    cooldown_blocked_seconds: int
+    cooldown_network_seconds: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class SettingsData(BaseModel):
+    auto_run_enabled: bool
+    run_interval_minutes: int
+    request_delay_seconds: int
+    nav_visible_pages: list[str]
+    policy: SettingsPolicyData
+    safety_state: ScrapeSafetyStateData
+    
+    openalex_api_key: str | None = None
+    crossref_api_token: str | None = None
+    crossref_api_mailto: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class SettingsEnvelope(BaseModel):
+    data: SettingsData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class SettingsUpdateRequest(BaseModel):
+    auto_run_enabled: bool
+    run_interval_minutes: int
+    request_delay_seconds: int
+    nav_visible_pages: list[str] | None = None
+
+    openalex_api_key: str | None = None
+    crossref_api_token: str | None = None
+    crossref_api_mailto: str | None = None
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class PublicationItemData(BaseModel):
+    publication_id: int
+    scholar_profile_id: int
+    scholar_label: str
+    title: str
+    year: int | None
+    citation_count: int
+    venue_text: str | None
+    pub_url: str | None
+    display_identifier: DisplayIdentifierData | None = None
+    pdf_url: str | None
+    pdf_status: str = "untracked"
+    pdf_attempt_count: int = 0
+    pdf_failure_reason: str | None = None
+    pdf_failure_detail: str | None = None
+    is_read: bool
+    is_favorite: bool = False
+    first_seen_at: datetime
+    is_new_in_latest_run: bool
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class PublicationsListData(BaseModel):
+    mode: str
+    favorite_only: bool = False
+    selected_scholar_profile_id: int | None
+    unread_count: int
+    favorites_count: int
+    latest_count: int
+    new_count: int
+    total_count: int
+    page: int
+    page_size: int
+    has_next: bool = False
+    has_prev: bool = False
+    publications: list[PublicationItemData]
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class PublicationsListEnvelope(BaseModel):
+    data: PublicationsListData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MarkAllReadData(BaseModel):
+    message: str
+    updated_count: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MarkAllReadEnvelope(BaseModel):
+    data: MarkAllReadData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class PublicationSelectionItem(BaseModel):
+    scholar_profile_id: int
+    publication_id: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MarkSelectedReadRequest(BaseModel):
+    selections: list[PublicationSelectionItem] = Field(min_length=1, max_length=500)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MarkSelectedReadData(BaseModel):
+    message: str
+    requested_count: int
+    updated_count: int
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class MarkSelectedReadEnvelope(BaseModel):
+    data: MarkSelectedReadData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RetryPublicationPdfRequest(BaseModel):
+    scholar_profile_id: int = Field(ge=1)
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RetryPublicationPdfData(BaseModel):
+    message: str
+    queued: bool
+    resolved_pdf: bool
+    publication: PublicationItemData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class RetryPublicationPdfEnvelope(BaseModel):
+    data: RetryPublicationPdfData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class TogglePublicationFavoriteRequest(BaseModel):
+    scholar_profile_id: int = Field(ge=1)
+    is_favorite: bool
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class TogglePublicationFavoriteData(BaseModel):
+    message: str
+    publication: PublicationItemData
+
+    model_config = ConfigDict(extra="forbid")
+
+
+class TogglePublicationFavoriteEnvelope(BaseModel):
+    data: TogglePublicationFavoriteData
+    meta: ApiMeta
+
+    model_config = ConfigDict(extra="forbid")

build/lib/app/auth/__init__.py

@@ -0,0 +1,2 @@
+"""Authentication package for scholarr."""
+

build/lib/app/auth/deps.py

@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+from functools import lru_cache
+
+from app.auth.rate_limit import SlidingWindowRateLimiter
+from app.auth.security import PasswordService
+from app.auth.service import AuthService
+from app.settings import settings
+
+
+@lru_cache
+def get_password_service() -> PasswordService:
+    return PasswordService()
+
+
+@lru_cache
+def get_auth_service() -> AuthService:
+    return AuthService(password_service=get_password_service())
+
+
+@lru_cache
+def get_login_rate_limiter() -> SlidingWindowRateLimiter:
+    return SlidingWindowRateLimiter(
+        max_attempts=settings.login_rate_limit_attempts,
+        window_seconds=settings.login_rate_limit_window_seconds,
+    )
+

build/lib/app/auth/rate_limit.py

@@ -0,0 +1,69 @@
+from __future__ import annotations
+
+from collections import deque
+from collections.abc import Callable
+from dataclasses import dataclass
+from math import ceil
+from threading import Lock
+from time import monotonic
+
+
+@dataclass(frozen=True)
+class RateLimitDecision:
+    allowed: bool
+    retry_after_seconds: int = 0
+
+
+class SlidingWindowRateLimiter:
+    def __init__(
+        self,
+        *,
+        max_attempts: int,
+        window_seconds: int,
+        now: Callable[[], float] = monotonic,
+    ) -> None:
+        self._max_attempts = max_attempts
+        self._window_seconds = window_seconds
+        self._now = now
+        self._attempts: dict[str, deque[float]] = {}
+        self._lock = Lock()
+
+    def check(self, key: str) -> RateLimitDecision:
+        with self._lock:
+            now_value = self._now()
+            attempts = self._attempts.get(key)
+            if attempts is None:
+                return RateLimitDecision(allowed=True)
+            self._trim_expired(attempts, now_value)
+            if not attempts:
+                self._attempts.pop(key, None)
+                return RateLimitDecision(allowed=True)
+            if len(attempts) >= self._max_attempts:
+                retry_after = self._window_seconds - (now_value - attempts[0])
+                return RateLimitDecision(
+                    allowed=False,
+                    retry_after_seconds=max(1, ceil(retry_after)),
+                )
+            return RateLimitDecision(allowed=True)
+
+    def record_failure(self, key: str) -> None:
+        with self._lock:
+            now_value = self._now()
+            attempts = self._attempts.get(key)
+            if attempts is None:
+                attempts = deque()
+                self._attempts[key] = attempts
+            self._trim_expired(attempts, now_value)
+            attempts.append(now_value)
+
+    def reset(self, key: str) -> None:
+        with self._lock:
+            self._attempts.pop(key, None)
+
+    def clear_all(self) -> None:
+        with self._lock:
+            self._attempts.clear()
+
+    def _trim_expired(self, attempts: deque[float], now_value: float) -> None:
+        while attempts and now_value - attempts[0] >= self._window_seconds:
+            attempts.popleft()

build/lib/app/auth/runtime.py

@@ -0,0 +1,55 @@
+from __future__ import annotations
+
+import logging
+
+from fastapi import Request
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.auth.session import clear_session_user, get_session_user, set_session_user
+from app.db.models import User
+from app.security.csrf import CSRF_SESSION_KEY
+from app.services.domains.users import application as user_service
+
+logger = logging.getLogger(__name__)
+
+
+def invalidate_session(request: Request) -> None:
+    clear_session_user(request)
+    request.session.pop(CSRF_SESSION_KEY, None)
+
+
+async def get_authenticated_user(
+    request: Request,
+    db_session: AsyncSession,
+) -> User | None:
+    session_user = get_session_user(request)
+    if session_user is None:
+        return None
+
+    user = await user_service.get_user_by_id(db_session, session_user.id)
+    if user is None or not user.is_active:
+        logger.info(
+            "auth.session_invalidated",
+            extra={
+                "event": "auth.session_invalidated",
+                "session_user_id": session_user.id,
+            },
+        )
+        invalidate_session(request)
+        return None
+
+    if user.email != session_user.email or user.is_admin != session_user.is_admin:
+        set_session_user(
+            request,
+            user_id=user.id,
+            email=user.email,
+            is_admin=user.is_admin,
+        )
+
+    return user
+
+
+def login_rate_limit_key(request: Request, email: str) -> str:
+    client_host = request.client.host if request.client is not None else "unknown"
+    normalized_email = email.strip().lower()
+    return f"{client_host}:{normalized_email or '<empty>'}"

build/lib/app/auth/security.py

@@ -0,0 +1,19 @@
+from __future__ import annotations
+
+from argon2 import PasswordHasher
+from argon2.exceptions import InvalidHashError, VerificationError
+
+
+class PasswordService:
+    def __init__(self, hasher: PasswordHasher | None = None) -> None:
+        self._hasher = hasher or PasswordHasher()
+
+    def hash_password(self, password: str) -> str:
+        return self._hasher.hash(password)
+
+    def verify_password(self, password_hash: str, password: str) -> bool:
+        try:
+            return bool(self._hasher.verify(password_hash, password))
+        except (InvalidHashError, VerificationError):
+            return False
+

build/lib/app/auth/service.py

@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.auth.security import PasswordService
+from app.db.models import User
+
+
+class AuthService:
+    def __init__(self, password_service: PasswordService) -> None:
+        self._password_service = password_service
+
+    async def authenticate_user(
+        self,
+        db_session: AsyncSession,
+        *,
+        email: str,
+        password: str,
+    ) -> User | None:
+        normalized_email = email.strip().lower()
+        if not normalized_email or not password:
+            return None
+        result = await db_session.execute(
+            select(User).where(User.email == normalized_email)
+        )
+        user = result.scalar_one_or_none()
+        if user is None or not user.is_active:
+            return None
+        if not self._password_service.verify_password(user.password_hash, password):
+            return None
+        return user
+
+    def hash_password(self, password: str) -> str:
+        return self._password_service.hash_password(password)
+
+    def verify_password(self, *, password_hash: str, password: str) -> bool:
+        return self._password_service.verify_password(password_hash, password)

build/lib/app/auth/session.py

@@ -0,0 +1,50 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from starlette.requests import Request
+
+
+SESSION_USER_ID_KEY = "auth_user_id"
+SESSION_USER_EMAIL_KEY = "auth_user_email"
+SESSION_USER_IS_ADMIN_KEY = "auth_user_is_admin"
+
+
+@dataclass(frozen=True)
+class SessionUser:
+    id: int
+    email: str
+    is_admin: bool
+
+
+def get_session_user(request: Request) -> SessionUser | None:
+    user_id = request.session.get(SESSION_USER_ID_KEY)
+    email = request.session.get(SESSION_USER_EMAIL_KEY)
+    is_admin = request.session.get(SESSION_USER_IS_ADMIN_KEY)
+    if user_id is None or email is None or is_admin is None:
+        return None
+    try:
+        parsed_user_id = int(user_id)
+    except (TypeError, ValueError):
+        return None
+    if not isinstance(email, str):
+        return None
+    return SessionUser(id=parsed_user_id, email=email, is_admin=bool(is_admin))
+
+
+def set_session_user(
+    request: Request,
+    *,
+    user_id: int,
+    email: str,
+    is_admin: bool,
+) -> None:
+    request.session[SESSION_USER_ID_KEY] = int(user_id)
+    request.session[SESSION_USER_EMAIL_KEY] = email
+    request.session[SESSION_USER_IS_ADMIN_KEY] = bool(is_admin)
+
+
+def clear_session_user(request: Request) -> None:
+    request.session.pop(SESSION_USER_ID_KEY, None)
+    request.session.pop(SESSION_USER_EMAIL_KEY, None)
+    request.session.pop(SESSION_USER_IS_ADMIN_KEY, None)

build/lib/app/db/__init__.py

@@ -0,0 +1 @@
+

build/lib/app/db/base.py

@@ -0,0 +1,27 @@
+from datetime import datetime, timezone
+
+from sqlalchemy import MetaData, event
+from sqlalchemy.orm import DeclarativeBase
+
+
+NAMING_CONVENTION = {
+    "ix": "ix_%(column_0_label)s",
+    "uq": "uq_%(table_name)s_%(column_0_name)s",
+    "ck": "ck_%(table_name)s_%(constraint_name)s",
+    "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
+    "pk": "pk_%(table_name)s",
+}
+
+
+class Base(DeclarativeBase):
+    metadata = MetaData(naming_convention=NAMING_CONVENTION)
+
+
+@event.listens_for(Base, "before_update", propagate=True)
+def _set_updated_at_before_update(_mapper, _connection, target) -> None:
+    # Keep audit timestamps current for ORM-managed updates.
+    if hasattr(target, "updated_at"):
+        target.updated_at = datetime.now(timezone.utc)
+
+
+metadata = Base.metadata

build/lib/app/db/models.py

@@ -0,0 +1,546 @@
+from __future__ import annotations
+
+from datetime import datetime
+from enum import StrEnum
+
+from sqlalchemy import (
+    Boolean,
+    CheckConstraint,
+    DateTime,
+    Enum,
+    Float,
+    ForeignKey,
+    Index,
+    Integer,
+    String,
+    Text,
+    UniqueConstraint,
+    func,
+    text,
+)
+from sqlalchemy.dialects.postgresql import JSONB
+from sqlalchemy.orm import Mapped, mapped_column
+
+from app.db.base import Base
+
+
+class RunTriggerType(StrEnum):
+    MANUAL = "manual"
+    SCHEDULED = "scheduled"
+
+
+class RunStatus(StrEnum):
+    RUNNING = "running"
+    SUCCESS = "success"
+    PARTIAL_FAILURE = "partial_failure"
+    FAILED = "failed"
+    CANCELED = "canceled"
+
+
+class QueueItemStatus(StrEnum):
+    QUEUED = "queued"
+    RETRYING = "retrying"
+    DROPPED = "dropped"
+
+
+RUN_STATUS_DB_ENUM = Enum(
+    RunStatus,
+    name="run_status",
+    values_callable=lambda members: [member.value for member in members],
+)
+RUN_TRIGGER_TYPE_DB_ENUM = Enum(
+    RunTriggerType,
+    name="run_trigger_type",
+    values_callable=lambda members: [member.value for member in members],
+)
+
+
+class User(Base):
+    __tablename__ = "users"
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    email: Mapped[str] = mapped_column(String(255), nullable=False, unique=True)
+    password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
+    is_active: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("true")
+    )
+    is_admin: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class UserSetting(Base):
+    __tablename__ = "user_settings"
+    __table_args__ = (
+        CheckConstraint(
+            "run_interval_minutes >= 15",
+            name="run_interval_minutes_min_15",
+        ),
+        CheckConstraint(
+            "request_delay_seconds >= 2",
+            name="request_delay_seconds_min_2",
+        ),
+    )
+
+    user_id: Mapped[int] = mapped_column(
+        ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
+    )
+    auto_run_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    run_interval_minutes: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("1440")
+    )
+    request_delay_seconds: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("10")
+    )
+    nav_visible_pages: Mapped[list[str]] = mapped_column(
+        JSONB,
+        nullable=False,
+        server_default=text(
+            '\'["dashboard","scholars","publications","settings","style-guide","runs","users"]\'::jsonb'
+        ),
+    )
+    scrape_safety_state: Mapped[dict] = mapped_column(
+        JSONB,
+        nullable=False,
+        server_default=text("'{}'::jsonb"),
+    )
+    scrape_cooldown_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    scrape_cooldown_reason: Mapped[str | None] = mapped_column(String(64))
+    
+    openalex_api_key: Mapped[str | None] = mapped_column(String(255))
+    crossref_api_token: Mapped[str | None] = mapped_column(String(255))
+    crossref_api_mailto: Mapped[str | None] = mapped_column(String(255))
+
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class ScholarProfile(Base):
+    __tablename__ = "scholar_profiles"
+    __table_args__ = (
+        UniqueConstraint("user_id", "scholar_id", name="uq_scholar_profiles_user_scholar"),
+        Index("ix_scholar_profiles_user_enabled", "user_id", "is_enabled"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(
+        ForeignKey("users.id", ondelete="CASCADE"), nullable=False
+    )
+    scholar_id: Mapped[str] = mapped_column(String(64), nullable=False)
+    display_name: Mapped[str | None] = mapped_column(String(255))
+    profile_image_url: Mapped[str | None] = mapped_column(Text)
+    profile_image_override_url: Mapped[str | None] = mapped_column(Text)
+    profile_image_upload_path: Mapped[str | None] = mapped_column(Text)
+    last_initial_page_fingerprint_sha256: Mapped[str | None] = mapped_column(String(64))
+    last_initial_page_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    is_enabled: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("true")
+    )
+    baseline_completed: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    last_run_dt: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    last_run_status: Mapped[RunStatus | None] = mapped_column(
+        RUN_STATUS_DB_ENUM,
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class CrawlRun(Base):
+    __tablename__ = "crawl_runs"
+    __table_args__ = (
+        Index("ix_crawl_runs_user_start", "user_id", "start_dt"),
+        Index(
+            "uq_crawl_runs_user_manual_idempotency_key",
+            "user_id",
+            "idempotency_key",
+            unique=True,
+            postgresql_where=text(
+                "idempotency_key IS NOT NULL AND trigger_type = 'manual'::run_trigger_type"
+            ),
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(
+        ForeignKey("users.id", ondelete="CASCADE"), nullable=False
+    )
+    trigger_type: Mapped[RunTriggerType] = mapped_column(
+        RUN_TRIGGER_TYPE_DB_ENUM, nullable=False
+    )
+    status: Mapped[RunStatus] = mapped_column(
+        RUN_STATUS_DB_ENUM, nullable=False
+    )
+    start_dt: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    end_dt: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    scholar_count: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("0")
+    )
+    new_pub_count: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("0")
+    )
+    idempotency_key: Mapped[str | None] = mapped_column(String(128))
+    error_log: Mapped[dict] = mapped_column(
+        JSONB, nullable=False, server_default=text("'{}'::jsonb")
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class Publication(Base):
+    __tablename__ = "publications"
+    __table_args__ = (
+        UniqueConstraint("fingerprint_sha256", name="uq_publications_fingerprint"),
+        Index(
+            "uq_publications_cluster_id_not_null",
+            "cluster_id",
+            unique=True,
+            postgresql_where=text("cluster_id IS NOT NULL"),
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    cluster_id: Mapped[str | None] = mapped_column(String(64))
+    fingerprint_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
+    title_raw: Mapped[str] = mapped_column(Text, nullable=False)
+    title_normalized: Mapped[str] = mapped_column(Text, nullable=False)
+    year: Mapped[int | None] = mapped_column(Integer)
+    citation_count: Mapped[int] = mapped_column(
+        Integer, nullable=False, server_default=text("0")
+    )
+    author_text: Mapped[str | None] = mapped_column(Text)
+    venue_text: Mapped[str | None] = mapped_column(Text)
+    pub_url: Mapped[str | None] = mapped_column(Text)
+    pdf_url: Mapped[str | None] = mapped_column(Text)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class PublicationIdentifier(Base):
+    __tablename__ = "publication_identifiers"
+    __table_args__ = (
+        UniqueConstraint(
+            "publication_id",
+            "kind",
+            "value_normalized",
+            name="uq_publication_identifiers_publication_kind_value",
+        ),
+        CheckConstraint(
+            "confidence_score >= 0 AND confidence_score <= 1",
+            name="publication_identifiers_confidence_score_range",
+        ),
+        Index(
+            "ix_publication_identifiers_kind_value",
+            "kind",
+            "value_normalized",
+        ),
+        Index(
+            "ix_publication_identifiers_publication_id",
+            "publication_id",
+        ),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    publication_id: Mapped[int] = mapped_column(
+        ForeignKey("publications.id", ondelete="CASCADE"),
+        nullable=False,
+    )
+    kind: Mapped[str] = mapped_column(String(32), nullable=False)
+    value_raw: Mapped[str] = mapped_column(Text, nullable=False)
+    value_normalized: Mapped[str] = mapped_column(String(255), nullable=False)
+    source: Mapped[str] = mapped_column(String(64), nullable=False)
+    confidence_score: Mapped[float] = mapped_column(
+        Float,
+        nullable=False,
+        server_default=text("0"),
+    )
+    evidence_url: Mapped[str | None] = mapped_column(Text)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class PublicationPdfJob(Base):
+    __tablename__ = "publication_pdf_jobs"
+    __table_args__ = (
+        CheckConstraint(
+            "status IN ('queued', 'running', 'resolved', 'failed')",
+            name="publication_pdf_jobs_status_valid",
+        ),
+        Index("ix_publication_pdf_jobs_status", "status"),
+        Index("ix_publication_pdf_jobs_updated_at", "updated_at"),
+        Index("ix_publication_pdf_jobs_queued_at", "queued_at"),
+    )
+
+    publication_id: Mapped[int] = mapped_column(
+        ForeignKey("publications.id", ondelete="CASCADE"),
+        primary_key=True,
+    )
+    status: Mapped[str] = mapped_column(
+        String(16),
+        nullable=False,
+        server_default=text("'queued'"),
+    )
+    attempt_count: Mapped[int] = mapped_column(
+        Integer,
+        nullable=False,
+        server_default=text("0"),
+    )
+    queued_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    last_attempt_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    resolved_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    last_failure_reason: Mapped[str | None] = mapped_column(String(64))
+    last_failure_detail: Mapped[str | None] = mapped_column(Text)
+    last_source: Mapped[str | None] = mapped_column(String(32))
+    last_requested_by_user_id: Mapped[int | None] = mapped_column(
+        ForeignKey("users.id", ondelete="SET NULL"),
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class PublicationPdfJobEvent(Base):
+    __tablename__ = "publication_pdf_job_events"
+    __table_args__ = (
+        Index("ix_publication_pdf_job_events_publication_created", "publication_id", "created_at"),
+        Index("ix_publication_pdf_job_events_created_at", "created_at"),
+        Index("ix_publication_pdf_job_events_event_type", "event_type"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    publication_id: Mapped[int] = mapped_column(
+        ForeignKey("publications.id", ondelete="CASCADE"),
+        nullable=False,
+    )
+    user_id: Mapped[int | None] = mapped_column(
+        ForeignKey("users.id", ondelete="SET NULL"),
+    )
+    event_type: Mapped[str] = mapped_column(String(32), nullable=False)
+    status: Mapped[str | None] = mapped_column(String(16))
+    source: Mapped[str | None] = mapped_column(String(32))
+    failure_reason: Mapped[str | None] = mapped_column(String(64))
+    message: Mapped[str | None] = mapped_column(Text)
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class ScholarPublication(Base):
+    __tablename__ = "scholar_publications"
+    __table_args__ = (
+        Index("ix_scholar_publications_is_read", "is_read"),
+        Index("ix_scholar_publications_is_favorite", "is_favorite"),
+    )
+
+    scholar_profile_id: Mapped[int] = mapped_column(
+        ForeignKey("scholar_profiles.id", ondelete="CASCADE"),
+        primary_key=True,
+    )
+    publication_id: Mapped[int] = mapped_column(
+        ForeignKey("publications.id", ondelete="CASCADE"),
+        primary_key=True,
+    )
+    is_read: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    is_favorite: Mapped[bool] = mapped_column(
+        Boolean, nullable=False, server_default=text("false")
+    )
+    first_seen_run_id: Mapped[int | None] = mapped_column(
+        ForeignKey("crawl_runs.id", ondelete="SET NULL")
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class IngestionQueueItem(Base):
+    __tablename__ = "ingestion_queue_items"
+    __table_args__ = (
+        UniqueConstraint(
+            "user_id",
+            "scholar_profile_id",
+            name="uq_ingestion_queue_user_scholar",
+        ),
+        CheckConstraint(
+            "status IN ('queued', 'retrying', 'dropped')",
+            name="ingestion_queue_status_valid",
+        ),
+        Index("ix_ingestion_queue_next_attempt", "next_attempt_dt"),
+        Index("ix_ingestion_queue_status_next_attempt", "status", "next_attempt_dt"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    user_id: Mapped[int] = mapped_column(
+        ForeignKey("users.id", ondelete="CASCADE"),
+        nullable=False,
+    )
+    scholar_profile_id: Mapped[int] = mapped_column(
+        ForeignKey("scholar_profiles.id", ondelete="CASCADE"),
+        nullable=False,
+    )
+    resume_cstart: Mapped[int] = mapped_column(
+        Integer,
+        nullable=False,
+        server_default=text("0"),
+    )
+    reason: Mapped[str] = mapped_column(String(64), nullable=False)
+    status: Mapped[str] = mapped_column(
+        String(16),
+        nullable=False,
+        server_default=text("'queued'"),
+    )
+    attempt_count: Mapped[int] = mapped_column(
+        Integer,
+        nullable=False,
+        server_default=text("0"),
+    )
+    next_attempt_dt: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=False,
+        server_default=func.now(),
+    )
+    last_run_id: Mapped[int | None] = mapped_column(
+        ForeignKey("crawl_runs.id", ondelete="SET NULL"),
+    )
+    last_error: Mapped[str | None] = mapped_column(Text)
+    dropped_reason: Mapped[str | None] = mapped_column(String(128))
+    dropped_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class DataRepairJob(Base):
+    __tablename__ = "data_repair_jobs"
+    __table_args__ = (
+        CheckConstraint(
+            "status IN ('planned', 'running', 'completed', 'failed')",
+            name="data_repair_jobs_status_valid",
+        ),
+        Index("ix_data_repair_jobs_created_at", "created_at"),
+        Index("ix_data_repair_jobs_job_name_created_at", "job_name", "created_at"),
+    )
+
+    id: Mapped[int] = mapped_column(primary_key=True)
+    job_name: Mapped[str] = mapped_column(String(64), nullable=False)
+    requested_by: Mapped[str | None] = mapped_column(String(255))
+    scope: Mapped[dict] = mapped_column(
+        JSONB,
+        nullable=False,
+        server_default=text("'{}'::jsonb"),
+    )
+    dry_run: Mapped[bool] = mapped_column(
+        Boolean,
+        nullable=False,
+        server_default=text("true"),
+    )
+    status: Mapped[str] = mapped_column(
+        String(16),
+        nullable=False,
+        server_default=text("'planned'"),
+    )
+    summary: Mapped[dict] = mapped_column(
+        JSONB,
+        nullable=False,
+        server_default=text("'{}'::jsonb"),
+    )
+    error_text: Mapped[str | None] = mapped_column(Text)
+    started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class AuthorSearchRuntimeState(Base):
+    __tablename__ = "author_search_runtime_state"
+
+    state_key: Mapped[str] = mapped_column(String(64), primary_key=True)
+    last_live_request_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    cooldown_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
+    consecutive_blocked_count: Mapped[int] = mapped_column(
+        Integer,
+        nullable=False,
+        server_default=text("0"),
+    )
+    cooldown_rejection_count: Mapped[int] = mapped_column(
+        Integer,
+        nullable=False,
+        server_default=text("0"),
+    )
+    cooldown_alert_emitted: Mapped[bool] = mapped_column(
+        Boolean,
+        nullable=False,
+        server_default=text("false"),
+    )
+    created_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+
+
+class AuthorSearchCacheEntry(Base):
+    __tablename__ = "author_search_cache_entries"
+    __table_args__ = (
+        Index("ix_author_search_cache_expires_at", "expires_at"),
+        Index("ix_author_search_cache_cached_at", "cached_at"),
+    )
+
+    query_key: Mapped[str] = mapped_column(String(256), primary_key=True)
+    payload: Mapped[dict] = mapped_column(
+        JSONB,
+        nullable=False,
+    )
+    expires_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True),
+        nullable=False,
+    )
+    cached_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), nullable=False, server_default=func.now()
+    )

build/lib/app/db/session.py

@@ -0,0 +1,99 @@
+from collections.abc import AsyncIterator
+import logging
+import os
+
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import (
+    AsyncEngine,
+    AsyncSession,
+    async_sessionmaker,
+    create_async_engine,
+)
+from sqlalchemy.pool import NullPool
+
+from app.settings import settings
+
+logger = logging.getLogger(__name__)
+
+_engine: AsyncEngine | None = None
+_session_factory: async_sessionmaker[AsyncSession] | None = None
+
+
+def _normalized_pool_mode(raw_mode: str) -> str:
+    mode = (raw_mode or "").strip().lower()
+    if mode == "auto":
+        if os.getenv("PYTEST_CURRENT_TEST"):
+            return "null"
+        app_env = (os.getenv("APP_ENV") or "").strip().lower()
+        if app_env in {"test", "development", "dev", "local"}:
+            return "null"
+        return "queue"
+    if mode in {"null", "queue"}:
+        return mode
+    logger.warning(
+        "db.invalid_pool_mode_fallback",
+        extra={
+            "event": "db.invalid_pool_mode_fallback",
+            "database_pool_mode": raw_mode,
+            "fallback_mode": "queue",
+        },
+    )
+    return "queue"
+
+
+def get_engine() -> AsyncEngine:
+    global _engine
+    if _engine is None:
+        pool_mode = _normalized_pool_mode(settings.database_pool_mode)
+        engine_kwargs: dict[str, object] = {
+            "pool_pre_ping": True,
+        }
+        if pool_mode == "null":
+            engine_kwargs["poolclass"] = NullPool
+        else:
+            engine_kwargs["pool_size"] = max(1, int(settings.database_pool_size))
+            engine_kwargs["max_overflow"] = max(0, int(settings.database_pool_max_overflow))
+            engine_kwargs["pool_timeout"] = max(1, int(settings.database_pool_timeout_seconds))
+
+        _engine = create_async_engine(settings.database_url, **engine_kwargs)
+        logger.info(
+            "db.engine_initialized",
+            extra={
+                "event": "db.engine_initialized",
+                "pool_mode": pool_mode,
+            },
+        )
+    return _engine
+
+
+def get_session_factory() -> async_sessionmaker[AsyncSession]:
+    global _session_factory
+    if _session_factory is None:
+        _session_factory = async_sessionmaker(get_engine(), expire_on_commit=False)
+    return _session_factory
+
+
+async def get_db_session() -> AsyncIterator[AsyncSession]:
+    session_factory = get_session_factory()
+    async with session_factory() as session:
+        yield session
+
+
+async def check_database() -> bool:
+    engine = get_engine()
+    try:
+        async with engine.connect() as conn:
+            result = await conn.execute(text("SELECT 1"))
+            return result.scalar_one() == 1
+    except Exception:
+        logger.exception("db.healthcheck_failed", extra={"event": "db.healthcheck_failed"})
+        return False
+
+
+async def close_engine() -> None:
+    global _engine, _session_factory
+    if _engine is not None:
+        await _engine.dispose()
+        logger.info("db.engine_disposed", extra={"event": "db.engine_disposed"})
+        _engine = None
+        _session_factory = None

build/lib/app/http/__init__.py

@@ -0,0 +1 @@
+from __future__ import annotations

build/lib/app/http/middleware.py

@@ -0,0 +1,205 @@
+from __future__ import annotations
+
+from secrets import token_urlsafe
+import logging
+import time
+
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.requests import Request
+from starlette.responses import Response
+
+from app.logging_context import set_request_id
+
+REQUEST_ID_HEADER = "X-Request-ID"
+DEFAULT_PERMISSIONS_POLICY = (
+    "accelerometer=(), autoplay=(), camera=(), display-capture=(), "
+    "geolocation=(), gyroscope=(), microphone=(), payment=(), usb=()"
+)
+DEFAULT_CSP_POLICY = (
+    "default-src 'self'; "
+    "base-uri 'self'; "
+    "form-action 'self'; "
+    "frame-ancestors 'none'; "
+    "img-src 'self' data: https:; "
+    "script-src 'self'; "
+    "style-src 'self' 'unsafe-inline'; "
+    "font-src 'self' data:; "
+    "connect-src 'self'; "
+    "object-src 'none'"
+)
+DEFAULT_CSP_DOCS_POLICY = (
+    "default-src 'self'; "
+    "img-src 'self' data: https:; "
+    "script-src 'self' 'unsafe-inline'; "
+    "style-src 'self' 'unsafe-inline'; "
+    "font-src 'self' data:; "
+    "connect-src 'self' https:; "
+    "object-src 'none'; "
+    "frame-ancestors 'none'"
+)
+
+logger = logging.getLogger(__name__)
+
+
+class RequestLoggingMiddleware(BaseHTTPMiddleware):
+    def __init__(
+        self,
+        app,
+        *,
+        log_requests: bool = True,
+        skip_paths: tuple[str, ...] = (),
+    ) -> None:
+        super().__init__(app)
+        self._log_requests = log_requests
+        self._skip_paths = tuple(path for path in skip_paths if path)
+
+    async def dispatch(self, request: Request, call_next) -> Response:
+        request_id = request.headers.get(REQUEST_ID_HEADER) or token_urlsafe(12)
+        request.state.request_id = request_id
+        set_request_id(request_id)
+
+        start = time.perf_counter()
+        should_log = self._log_requests and not self._is_skipped_path(request.url.path)
+        if should_log:
+            logger.debug(
+                "request.started",
+                extra={
+                    "event": "request.started",
+                    "method": request.method,
+                    "path": request.url.path,
+                },
+            )
+
+        try:
+            response = await call_next(request)
+        except Exception:
+            duration_ms = int((time.perf_counter() - start) * 1000)
+            logger.exception(
+                "request.failed",
+                extra={
+                    "event": "request.failed",
+                    "method": request.method,
+                    "path": request.url.path,
+                    "duration_ms": duration_ms,
+                },
+            )
+            raise
+        else:
+            duration_ms = int((time.perf_counter() - start) * 1000)
+            response.headers[REQUEST_ID_HEADER] = request_id
+            if should_log:
+                logger.debug(
+                    "request.completed",
+                    extra={
+                        "event": "request.completed",
+                        "method": request.method,
+                        "path": request.url.path,
+                        "status_code": response.status_code,
+                        "duration_ms": duration_ms,
+                    },
+                )
+            return response
+        finally:
+            set_request_id(None)
+
+    def _is_skipped_path(self, path: str) -> bool:
+        return any(path.startswith(prefix) for prefix in self._skip_paths)
+
+
+class SecurityHeadersMiddleware(BaseHTTPMiddleware):
+    def __init__(
+        self,
+        app,
+        *,
+        enabled: bool = True,
+        x_content_type_options: str = "nosniff",
+        x_frame_options: str = "DENY",
+        referrer_policy: str = "strict-origin-when-cross-origin",
+        permissions_policy: str = DEFAULT_PERMISSIONS_POLICY,
+        cross_origin_opener_policy: str = "same-origin",
+        cross_origin_resource_policy: str = "same-origin",
+        content_security_policy_enabled: bool = True,
+        content_security_policy: str = DEFAULT_CSP_POLICY,
+        content_security_policy_docs: str = DEFAULT_CSP_DOCS_POLICY,
+        content_security_policy_report_only: bool = False,
+        strict_transport_security_enabled: bool = False,
+        strict_transport_security_max_age: int = 31_536_000,
+        strict_transport_security_include_subdomains: bool = True,
+        strict_transport_security_preload: bool = False,
+    ) -> None:
+        super().__init__(app)
+        self._enabled = enabled
+        self._x_content_type_options = x_content_type_options.strip()
+        self._x_frame_options = x_frame_options.strip()
+        self._referrer_policy = referrer_policy.strip()
+        self._permissions_policy = permissions_policy.strip()
+        self._cross_origin_opener_policy = cross_origin_opener_policy.strip()
+        self._cross_origin_resource_policy = cross_origin_resource_policy.strip()
+        self._csp_enabled = content_security_policy_enabled
+        self._csp_policy = content_security_policy.strip()
+        self._csp_docs_policy = content_security_policy_docs.strip()
+        self._csp_report_only = content_security_policy_report_only
+        self._hsts_enabled = strict_transport_security_enabled
+        self._hsts_max_age = max(0, strict_transport_security_max_age)
+        self._hsts_include_subdomains = strict_transport_security_include_subdomains
+        self._hsts_preload = strict_transport_security_preload
+
+    async def dispatch(self, request: Request, call_next) -> Response:
+        response = await call_next(request)
+        if not self._enabled:
+            return response
+
+        if self._x_content_type_options:
+            response.headers.setdefault("X-Content-Type-Options", self._x_content_type_options)
+        if self._x_frame_options:
+            response.headers.setdefault("X-Frame-Options", self._x_frame_options)
+        if self._referrer_policy:
+            response.headers.setdefault("Referrer-Policy", self._referrer_policy)
+        if self._permissions_policy:
+            response.headers.setdefault("Permissions-Policy", self._permissions_policy)
+        if self._cross_origin_opener_policy:
+            response.headers.setdefault(
+                "Cross-Origin-Opener-Policy",
+                self._cross_origin_opener_policy,
+            )
+        if self._cross_origin_resource_policy:
+            response.headers.setdefault(
+                "Cross-Origin-Resource-Policy",
+                self._cross_origin_resource_policy,
+            )
+
+        csp_policy = self._csp_policy_for_path(request.url.path)
+        if self._csp_enabled and csp_policy:
+            csp_header = (
+                "Content-Security-Policy-Report-Only"
+                if self._csp_report_only
+                else "Content-Security-Policy"
+            )
+            response.headers.setdefault(csp_header, csp_policy)
+
+        hsts = self._strict_transport_security_value()
+        if hsts:
+            response.headers.setdefault("Strict-Transport-Security", hsts)
+
+        return response
+
+    def _csp_policy_for_path(self, path: str) -> str:
+        if path.startswith("/docs") or path.startswith("/redoc"):
+            return self._csp_docs_policy or self._csp_policy
+        return self._csp_policy
+
+    def _strict_transport_security_value(self) -> str:
+        if not self._hsts_enabled:
+            return ""
+
+        directives = [f"max-age={self._hsts_max_age}"]
+        if self._hsts_include_subdomains:
+            directives.append("includeSubDomains")
+        if self._hsts_preload:
+            directives.append("preload")
+        return "; ".join(directives)
+
+
+def parse_skip_paths(raw_value: str) -> tuple[str, ...]:
+    parts = [part.strip() for part in raw_value.split(",")]
+    return tuple(part for part in parts if part)

build/lib/app/logging_config.py

@@ -0,0 +1,183 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+import json
+import logging
+import sys
+from typing import Any
+
+from app.logging_context import get_request_id
+
+DEFAULT_REDACT_FIELDS = {
+    "authorization",
+    "cookie",
+    "csrf_token",
+    "new_password",
+    "password",
+    "password_hash",
+    "session",
+    "session_secret_key",
+}
+
+_BASE_RECORD = logging.makeLogRecord({})
+_STANDARD_RECORD_FIELDS = set(_BASE_RECORD.__dict__.keys()) | {"message", "asctime"}
+_NOISY_RECORD_FIELDS = {"color_message"}
+
+
+def parse_redact_fields(raw: str | None) -> set[str]:
+    fields = {field.strip().lower() for field in (raw or "").split(",") if field.strip()}
+    return DEFAULT_REDACT_FIELDS | fields
+
+
+def configure_logging(
+    *,
+    level: str,
+    log_format: str,
+    redact_fields: set[str],
+    include_uvicorn_access: bool,
+) -> None:
+    normalized_level = _normalize_level(level)
+    root_logger = logging.getLogger()
+    root_logger.handlers.clear()
+    root_logger.setLevel(normalized_level)
+
+    handler = logging.StreamHandler(stream=sys.stdout)
+    handler.setLevel(normalized_level)
+    handler.addFilter(RequestContextFilter())
+
+    normalized_format = log_format.strip().lower()
+    if normalized_format == "json":
+        handler.setFormatter(JsonLogFormatter(redact_fields=redact_fields))
+    else:
+        handler.setFormatter(ConsoleLogFormatter(redact_fields=redact_fields))
+
+    root_logger.addHandler(handler)
+
+    # Route server/framework logs through our single root handler.
+    for logger_name in ("uvicorn", "uvicorn.error"):
+        framework_logger = logging.getLogger(logger_name)
+        framework_logger.handlers.clear()
+        framework_logger.propagate = True
+        framework_logger.setLevel(normalized_level)
+
+    access_logger = logging.getLogger("uvicorn.access")
+    access_logger.handlers.clear()
+    access_logger.propagate = True
+    access_logger.setLevel(normalized_level if include_uvicorn_access else logging.WARNING)
+
+
+class RequestContextFilter(logging.Filter):
+    def filter(self, record: logging.LogRecord) -> bool:
+        if not getattr(record, "request_id", None):
+            request_id = get_request_id()
+            if request_id:
+                record.request_id = request_id
+        return True
+
+
+class JsonLogFormatter(logging.Formatter):
+    def __init__(self, *, redact_fields: set[str]) -> None:
+        super().__init__()
+        self._redact_fields = {field.lower() for field in redact_fields}
+
+    def format(self, record: logging.LogRecord) -> str:
+        payload: dict[str, Any] = {
+            "timestamp": _format_timestamp(record.created),
+            "level": record.levelname.lower(),
+            "logger": record.name,
+            "event": getattr(record, "event", record.getMessage()),
+        }
+        request_id = getattr(record, "request_id", None)
+        if request_id:
+            payload["request_id"] = request_id
+        payload.update(self._redact_mapping(_extra_fields(record)))
+        if record.exc_info:
+            payload["exception"] = self.formatException(record.exc_info)
+        return json.dumps(payload, ensure_ascii=True, default=str)
+
+    def _redact_mapping(self, value: dict[str, Any]) -> dict[str, Any]:
+        return {key: self._redact_value(key, item) for key, item in value.items()}
+
+    def _redact_value(self, key: str, value: Any) -> Any:
+        if key.lower() in self._redact_fields:
+            return "[REDACTED]"
+        if isinstance(value, dict):
+            return {nested_key: self._redact_value(nested_key, nested_value) for nested_key, nested_value in value.items()}
+        if isinstance(value, (list, tuple)):
+            return [self._redact_value(key, item) for item in value]
+        return value
+
+
+class ConsoleLogFormatter(logging.Formatter):
+    def __init__(self, *, redact_fields: set[str]) -> None:
+        super().__init__()
+        self._json_formatter = JsonLogFormatter(redact_fields=redact_fields)
+
+    def format(self, record: logging.LogRecord) -> str:
+        payload = json.loads(self._json_formatter.format(record))
+        timestamp = payload.get("timestamp", "")
+        level = _short_level(payload.get("level", "info"))
+        logger_name = str(payload.get("logger", "app"))
+        event = str(payload.get("event", ""))
+
+        parts = [timestamp, level, logger_name, event]
+
+        request_id = payload.pop("request_id", None)
+        method = payload.pop("method", None)
+        path = payload.pop("path", None)
+        status_code = payload.pop("status_code", None)
+        duration_ms = payload.pop("duration_ms", None)
+
+        if request_id:
+            parts.append(f"rid={request_id}")
+        if method and path:
+            parts.append(f"{method} {path}")
+        if status_code is not None:
+            parts.append(str(status_code))
+        if duration_ms is not None:
+            parts.append(f"{duration_ms}ms")
+
+        for key in sorted(payload.keys()):
+            if key in {"timestamp", "level", "logger", "event", "exception"}:
+                continue
+            parts.append(f"{key}={payload[key]}")
+
+        if "exception" in payload:
+            parts.append(f"exception={payload['exception']}")
+
+        return " | ".join(str(part) for part in parts if part)
+
+
+def _extra_fields(record: logging.LogRecord) -> dict[str, Any]:
+    extras: dict[str, Any] = {}
+    for key, value in record.__dict__.items():
+        if key in _STANDARD_RECORD_FIELDS or key.startswith("_"):
+            continue
+        if key in _NOISY_RECORD_FIELDS:
+            continue
+        extras[key] = value
+    return extras
+
+
+def _normalize_level(level: str) -> int:
+    normalized = level.strip().upper()
+    mapping = logging.getLevelNamesMapping()
+    if normalized not in mapping:
+        return logging.INFO
+    return mapping[normalized]
+
+
+def _format_timestamp(created_ts: float) -> str:
+    dt = datetime.fromtimestamp(created_ts, tz=timezone.utc)
+    return dt.strftime("%Y-%m-%d %H:%M:%SZ")
+
+
+def _short_level(level: str) -> str:
+    mapping = {
+        "debug": "DBG",
+        "info": "INF",
+        "warning": "WRN",
+        "error": "ERR",
+        "critical": "CRT",
+    }
+    return mapping.get(level.lower(), level[:3].upper())

build/lib/app/logging_context.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from contextvars import ContextVar
+
+
+_request_id_ctx: ContextVar[str | None] = ContextVar("request_id", default=None)
+
+
+def get_request_id() -> str | None:
+    return _request_id_ctx.get()
+
+
+def set_request_id(value: str | None) -> None:
+    _request_id_ctx.set(value)
+

build/lib/app/main.py

@@ -0,0 +1,168 @@
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+import logging
+from pathlib import Path
+
+from fastapi import FastAPI, HTTPException
+from fastapi.responses import FileResponse
+from fastapi.staticfiles import StaticFiles
+from starlette.middleware.sessions import SessionMiddleware
+
+from app.api.errors import register_api_exception_handlers
+from app.api.media import router as media_router
+from app.api.router import router as api_router
+from app.api.runtime_deps import get_ingestion_service, get_scholar_source
+from app.db.session import check_database
+from app.db.session import close_engine
+from app.http.middleware import (
+    RequestLoggingMiddleware,
+    SecurityHeadersMiddleware,
+    parse_skip_paths,
+)
+from app.logging_config import configure_logging, parse_redact_fields
+from app.security.csrf import CSRFMiddleware
+from app.services.domains.ingestion.scheduler import SchedulerService
+from app.settings import settings
+
+logger = logging.getLogger(__name__)
+BUILD_MARKER = "2026-02-19.phase2.direct-pdf-import-export-dashboard-sync"
+
+configure_logging(
+    level=settings.log_level,
+    log_format=settings.log_format,
+    redact_fields=parse_redact_fields(settings.log_redact_fields),
+    include_uvicorn_access=settings.log_uvicorn_access,
+)
+
+scheduler_service = SchedulerService(
+    enabled=settings.scheduler_enabled,
+    tick_seconds=settings.scheduler_tick_seconds,
+    network_error_retries=settings.ingestion_network_error_retries,
+    retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
+    max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
+    page_size=settings.ingestion_page_size,
+    continuation_queue_enabled=settings.ingestion_continuation_queue_enabled,
+    continuation_base_delay_seconds=settings.ingestion_continuation_base_delay_seconds,
+    continuation_max_delay_seconds=settings.ingestion_continuation_max_delay_seconds,
+    continuation_max_attempts=settings.ingestion_continuation_max_attempts,
+    queue_batch_size=settings.scheduler_queue_batch_size,
+)
+
+
+def _log_startup_build_marker() -> None:
+    logger.info(
+        "app.startup_build_marker",
+        extra={
+            "event": "app.startup_build_marker",
+            "build_marker": BUILD_MARKER,
+            "frontend_enabled": settings.frontend_enabled,
+            "scheduler_enabled": settings.scheduler_enabled,
+            "log_format": settings.log_format,
+        },
+    )
+
+
+@asynccontextmanager
+async def lifespan(_: FastAPI):
+    _log_startup_build_marker()
+    
+    from app.db.session import get_session_factory
+    from sqlalchemy import text
+    try:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            await session.execute(text("UPDATE crawl_runs SET status = 'failed' WHERE status = 'running'"))
+            await session.commit()
+            logger.info("app.startup_orphaned_runs_cleaned", extra={"event": "app.startup_orphaned_runs_cleaned"})
+    except Exception as e:
+        logger.error(f"Failed to clean orphaned runs at startup: {e}")
+
+    await scheduler_service.start()
+    yield
+    await scheduler_service.stop()
+    await close_engine()
+
+
+app = FastAPI(title=settings.app_name, lifespan=lifespan)
+register_api_exception_handlers(app)
+app.add_middleware(CSRFMiddleware)
+app.add_middleware(
+    SessionMiddleware,
+    secret_key=settings.session_secret_key,
+    same_site="lax",
+    https_only=settings.session_cookie_secure,
+)
+app.add_middleware(
+    RequestLoggingMiddleware,
+    log_requests=settings.log_requests,
+    skip_paths=parse_skip_paths(settings.log_request_skip_paths),
+)
+app.add_middleware(
+    SecurityHeadersMiddleware,
+    enabled=settings.security_headers_enabled,
+    x_content_type_options=settings.security_x_content_type_options,
+    x_frame_options=settings.security_x_frame_options,
+    referrer_policy=settings.security_referrer_policy,
+    permissions_policy=settings.security_permissions_policy,
+    cross_origin_opener_policy=settings.security_cross_origin_opener_policy,
+    cross_origin_resource_policy=settings.security_cross_origin_resource_policy,
+    content_security_policy_enabled=settings.security_csp_enabled,
+    content_security_policy=settings.security_csp_policy,
+    content_security_policy_docs=settings.security_csp_docs_policy,
+    content_security_policy_report_only=settings.security_csp_report_only,
+    strict_transport_security_enabled=settings.security_strict_transport_security_enabled,
+    strict_transport_security_max_age=settings.security_strict_transport_security_max_age,
+    strict_transport_security_include_subdomains=(
+        settings.security_strict_transport_security_include_subdomains
+    ),
+    strict_transport_security_preload=settings.security_strict_transport_security_preload,
+)
+app.include_router(api_router)
+app.include_router(media_router)
+
+
+@app.get("/healthz")
+async def healthz() -> dict[str, str]:
+    if await check_database():
+        return {"status": "ok"}
+    raise HTTPException(status_code=500, detail="database unavailable")
+
+
+def _configure_frontend_routes(application: FastAPI) -> None:
+    if not settings.frontend_enabled:
+        return
+
+    dist_dir = Path(settings.frontend_dist_dir)
+    index_file = dist_dir / "index.html"
+
+    if not index_file.is_file():
+        return
+
+    assets_dir = dist_dir / "assets"
+    if assets_dir.is_dir():
+        application.mount("/assets", StaticFiles(directory=assets_dir), name="frontend-assets")
+
+    @application.get("/", include_in_schema=False)
+    async def frontend_index() -> FileResponse:
+        return FileResponse(index_file)
+
+    @application.get("/{full_path:path}", include_in_schema=False)
+    async def frontend_entry(full_path: str) -> FileResponse:
+        normalized = full_path.lstrip("/")
+        if normalized in {"healthz", "api"} or normalized.startswith("api/"):
+            raise HTTPException(status_code=404, detail="Not Found")
+
+        candidate = (dist_dir / normalized).resolve()
+        try:
+            candidate.relative_to(dist_dir.resolve())
+        except ValueError as error:
+            raise HTTPException(status_code=404, detail="Not Found") from error
+
+        if candidate.is_file():
+            return FileResponse(candidate)
+
+        return FileResponse(index_file)
+
+
+_configure_frontend_routes(app)

build/lib/app/security/__init__.py

@@ -0,0 +1,2 @@
+"""Security middleware and helpers for scholarr."""
+

build/lib/app/security/csrf.py

@@ -0,0 +1,133 @@
+from __future__ import annotations
+
+import logging
+from secrets import compare_digest, token_urlsafe
+from urllib.parse import parse_qs
+
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.requests import Request
+from starlette.responses import JSONResponse, PlainTextResponse, Response
+from starlette.types import Message
+
+
+CSRF_SESSION_KEY = "csrf_token"
+CSRF_FORM_FIELD = "csrf_token"
+CSRF_HEADER_NAME = "x-csrf-token"
+SAFE_METHODS = {"GET", "HEAD", "OPTIONS", "TRACE"}
+logger = logging.getLogger(__name__)
+
+
+def ensure_csrf_token(request: Request) -> str:
+    token = request.session.get(CSRF_SESSION_KEY)
+    if token is None:
+        token = token_urlsafe(32)
+        request.session[CSRF_SESSION_KEY] = token
+    return str(token)
+
+
+class CSRFMiddleware(BaseHTTPMiddleware):
+    def __init__(self, app, *, exempt_paths: set[str] | None = None) -> None:
+        super().__init__(app)
+        self._exempt_paths = exempt_paths or set()
+
+    async def dispatch(self, request: Request, call_next) -> Response:
+        if self._should_skip(request):
+            return await call_next(request)
+
+        session_token = request.session.get(CSRF_SESSION_KEY)
+        if not session_token:
+            logger.warning(
+                "csrf.missing_session_token",
+                extra={
+                    "event": "csrf.missing_session_token",
+                    "method": request.method,
+                    "path": request.url.path,
+                },
+            )
+            return self._csrf_error_response(
+                request,
+                code="csrf_missing",
+                message="CSRF token missing.",
+            )
+
+        request_token = request.headers.get(CSRF_HEADER_NAME)
+        if request_token is None and self._is_form_payload(request):
+            body = await request.body()
+            request_token = self._token_from_form_body(request, body)
+            request._receive = self._build_receive(body)  # type: ignore[attr-defined]
+
+        if not request_token or not compare_digest(str(session_token), str(request_token)):
+            logger.warning(
+                "csrf.invalid_token",
+                extra={
+                    "event": "csrf.invalid_token",
+                    "method": request.method,
+                    "path": request.url.path,
+                },
+            )
+            return self._csrf_error_response(
+                request,
+                code="csrf_invalid",
+                message="CSRF token invalid.",
+            )
+
+        return await call_next(request)
+
+    def _should_skip(self, request: Request) -> bool:
+        if request.method in SAFE_METHODS:
+            return True
+        path = request.url.path
+        return path in self._exempt_paths
+
+    def _is_form_payload(self, request: Request) -> bool:
+        content_type = request.headers.get("content-type", "")
+        return content_type.startswith("application/x-www-form-urlencoded") or (
+            content_type.startswith("multipart/form-data")
+        )
+
+    def _token_from_form_body(self, request: Request, body: bytes) -> str | None:
+        content_type = request.headers.get("content-type", "")
+        if not content_type.startswith("application/x-www-form-urlencoded"):
+            return None
+        parsed = parse_qs(body.decode("utf-8"), keep_blank_values=True)
+        values = parsed.get(CSRF_FORM_FIELD)
+        if not values:
+            return None
+        return values[0]
+
+    def _build_receive(self, body: bytes):
+        consumed = False
+
+        async def receive() -> Message:
+            nonlocal consumed
+            if consumed:
+                return {"type": "http.request", "body": b"", "more_body": False}
+            consumed = True
+            return {"type": "http.request", "body": body, "more_body": False}
+
+        return receive
+
+    def _csrf_error_response(
+        self,
+        request: Request,
+        *,
+        code: str,
+        message: str,
+    ) -> Response:
+        if request.url.path.startswith("/api/"):
+            request_state = getattr(request, "state", None)
+            request_id = getattr(request_state, "request_id", None) if request_state else None
+            return JSONResponse(
+                {
+                    "error": {
+                        "code": code,
+                        "message": message,
+                        "details": None,
+                    },
+                    "meta": {
+                        "request_id": request_id,
+                    },
+                },
+                status_code=403,
+            )
+        return PlainTextResponse(message, status_code=403)

build/lib/app/services/__init__.py

@@ -0,0 +1,2 @@
+"""Service layer for scholarr application workflows."""
+

build/lib/app/services/domains/__init__.py

@@ -0,0 +1 @@
+from __future__ import annotations

build/lib/app/services/domains/arxiv/application.py

@@ -0,0 +1,84 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import TYPE_CHECKING
+import xml.etree.ElementTree as ET
+
+import httpx
+
+from app.services.domains.publication_identifiers.normalize import normalize_arxiv_id
+from app.settings import settings
+
+if TYPE_CHECKING:
+    from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
+
+logger = logging.getLogger(__name__)
+
+
+def _build_arxiv_query(title: str, author_surname: str | None) -> str | None:
+    parts = []
+    if title:
+        # arXiv api allows strict title searching using ti:
+        clean_title = title.replace('"', '').replace("'", "")
+        parts.append(f'ti:"{clean_title}"')
+    if author_surname:
+        parts.append(f'au:"{author_surname}"')
+    
+    if not parts:
+        return None
+    return " AND ".join(parts)
+
+
+async def discover_arxiv_id_for_publication(
+    *,
+    item: PublicationListItem | UnreadPublicationItem,
+    request_email: str | None = None,
+    timeout_seconds: float = 3.0,
+) -> str | None:
+    title = (item.title or "").strip()
+    if not title:
+        return None
+        
+    author_surname = None
+    if item.scholar_label:
+        tokens = [t for t in item.scholar_label.strip().split() if t]
+        if tokens:
+            author_surname = tokens[-1].lower()
+
+    query = _build_arxiv_query(title, author_surname)
+    if not query:
+        return None
+
+    url = "https://export.arxiv.org/api/query"
+    params = {
+        "search_query": query,
+        "start": 0,
+        "max_results": 3,
+    }
+
+    headers = {"User-Agent": f"scholar-scraper/1.0 (mailto:{request_email or settings.crossref_api_mailto or 'unknown@example.com'})"}
+    
+    try:
+        async with httpx.AsyncClient(timeout=timeout_seconds, follow_redirects=True, headers=headers) as client:
+            response = await client.get(url, params=params)
+            response.raise_for_status()
+            
+        root = ET.fromstring(response.text)
+        namespace = {'atom': 'http://www.w3.org/2005/Atom'}
+        
+        for entry in root.findall('atom:entry', namespace):
+            id_elem = entry.find('atom:id', namespace)
+            if id_elem is not None and id_elem.text:
+                candidate = str(id_elem.text)
+                if '/abs/' in candidate:
+                    candidate = candidate.split('/abs/')[-1]
+                normalized = normalize_arxiv_id(candidate)
+                if normalized:
+                    logger.debug("arxiv.id_discovered", extra={"event": "arxiv.id_discovered", "arxiv_id": normalized})
+                    return normalized
+                    
+    except Exception as exc:
+        logger.debug(f"Failed to query arXiv API: {exc}")
+        
+    return None

build/lib/app/services/domains/crossref/__init__.py

@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+from app.services.domains.crossref.application import discover_doi_for_publication
+
+__all__ = ["discover_doi_for_publication"]

build/lib/app/services/domains/crossref/application.py

@@ -0,0 +1,382 @@
+from __future__ import annotations
+
+import asyncio
+import logging
+import re
+import threading
+import time
+from typing import TYPE_CHECKING
+
+from crossref.restful import Etiquette, Works
+
+from app.services.domains.doi.normalize import normalize_doi
+from app.settings import settings
+
+if TYPE_CHECKING:
+    from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
+
+TOKEN_RE = re.compile(r"[a-z0-9]+")
+NON_ALNUM_RE = re.compile(r"[^a-z0-9\\s]+")
+STOP_WORDS = {"the", "and", "for", "with", "from", "method", "study", "analysis"}
+_RATE_LOCK = threading.Lock()
+_LAST_REQUEST_AT = 0.0
+logger = logging.getLogger(__name__)
+STRICT_TITLE_MATCH_THRESHOLD = 0.75
+RELAXED_TITLE_MATCH_THRESHOLD = 0.85
+
+
+def _rate_limit_wait(min_interval_seconds: float) -> None:
+    global _LAST_REQUEST_AT
+    interval = max(float(min_interval_seconds), 0.0)
+    with _RATE_LOCK:
+        elapsed = time.monotonic() - _LAST_REQUEST_AT
+        remaining = interval - elapsed
+        if remaining > 0:
+            time.sleep(remaining)
+        _LAST_REQUEST_AT = time.monotonic()
+
+
+def _normalized_tokens(value: str) -> list[str]:
+    lowered = value.lower().replace("’", "'").replace("“", "\"").replace("”", "\"")
+    lowered = NON_ALNUM_RE.sub(" ", lowered)
+    return [token for token in TOKEN_RE.findall(lowered) if len(token) >= 3]
+
+
+def _normalized_query(value: str) -> str:
+    tokens = [token for token in _normalized_tokens(value) if token not in STOP_WORDS]
+    if len(tokens) < 3:
+        tokens = _normalized_tokens(value)
+    if len(tokens) < 3:
+        return ""
+    return " ".join(tokens[:12]).strip()
+
+
+def _query_author(value: str) -> str | None:
+    tokens = [token for token in value.strip().split() if token]
+    if len(tokens) < 2:
+        return None
+    return " ".join(tokens[:2])[:64]
+
+
+def _author_surname(value: str) -> str | None:
+    tokens = [token for token in value.strip().split() if token]
+    if not tokens:
+        return None
+    return NON_ALNUM_RE.sub("", tokens[-1].lower()) or None
+
+
+def _query_filters(year: int | None) -> list[tuple[str, str] | None]:
+    if year is None:
+        return [None]
+    return [
+        (f"{year - 1}-01-01", f"{year + 1}-12-31"),
+        (f"{year}-01-01", f"{year}-12-31"),
+        None,
+    ]
+
+
+def _candidate_title(item: dict) -> str:
+    titles = item.get("title")
+    if isinstance(titles, list) and titles:
+        return str(titles[0] or "")
+    return str(item.get("title") or "")
+
+
+def _title_match_score(source: str, candidate: str) -> float:
+    source_tokens = {token for token in _normalized_tokens(source) if len(token) >= 3}
+    candidate_tokens = {token for token in _normalized_tokens(candidate) if len(token) >= 3}
+    if not source_tokens or not candidate_tokens:
+        return 0.0
+    return len(source_tokens & candidate_tokens) / float(len(source_tokens))
+
+
+def _candidate_year(item: dict) -> int | None:
+    issued = item.get("issued")
+    if not isinstance(issued, dict):
+        return None
+    date_parts = issued.get("date-parts")
+    if not isinstance(date_parts, list) or not date_parts:
+        return None
+    first = date_parts[0]
+    if not isinstance(first, list) or not first:
+        return None
+    try:
+        return int(first[0])
+    except (TypeError, ValueError):
+        return None
+
+
+def _candidate_author_match(item: dict, surname: str | None) -> bool:
+    if not surname:
+        return True
+    authors = item.get("author")
+    if not isinstance(authors, list):
+        return False
+    for author in authors:
+        if not isinstance(author, dict):
+            continue
+        family = NON_ALNUM_RE.sub("", str(author.get("family") or "").lower())
+        if family and family == surname:
+            return True
+    return False
+
+
+def _candidate_rank(*, title: str, year: int | None, item: dict) -> tuple[float, str | None]:
+    doi = normalize_doi(str(item.get("DOI") or ""))
+    if doi is None:
+        return 0.0, None
+    score = _title_match_score(title, _candidate_title(item))
+    candidate_year = _candidate_year(item)
+    if year is not None and candidate_year is not None:
+        if abs(year - candidate_year) > 1:
+            return 0.0, None
+        score += 0.1
+    return score, doi
+
+
+def _year_delta(source_year: int | None, candidate_year: int | None) -> int | None:
+    if source_year is None or candidate_year is None:
+        return None
+    return abs(int(source_year) - int(candidate_year))
+
+
+def _candidate_rank_relaxed(
+    *,
+    title: str,
+    year: int | None,
+    item: dict,
+    author_surname: str | None,
+) -> tuple[float, str | None]:
+    doi = normalize_doi(str(item.get("DOI") or ""))
+    if doi is None:
+        return 0.0, None
+    score = _title_match_score(title, _candidate_title(item))
+    if score <= 0:
+        return 0.0, None
+    candidate_year = _candidate_year(item)
+    delta = _year_delta(year, candidate_year)
+    if delta is not None:
+        if delta <= 1:
+            score += 0.05
+        elif delta <= 3:
+            score += 0.0
+        elif delta <= 5:
+            score -= 0.03
+        else:
+            score -= 0.08
+    if _candidate_author_match(item, author_surname):
+        score += 0.03
+    return score, doi
+
+
+def _best_candidate_doi_strict(
+    *,
+    title: str,
+    year: int | None,
+    items: list[dict],
+    author_surname: str | None,
+) -> str | None:
+    best_score = 0.0
+    best_doi: str | None = None
+    best_year: int | None = None
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        if not _candidate_author_match(item, author_surname):
+            continue
+        score, doi = _candidate_rank(title=title, year=year, item=item)
+        candidate_year = _candidate_year(item)
+        if doi is None or score < STRICT_TITLE_MATCH_THRESHOLD:
+            continue
+        if score > best_score:
+            best_score = score
+            best_doi = doi
+            best_year = candidate_year
+            continue
+        if abs(score - best_score) > 0.02:
+            continue
+        if best_year is None or candidate_year is None:
+            continue
+        if candidate_year < best_year:
+            best_doi = doi
+            best_year = candidate_year
+    return best_doi
+
+
+def _best_candidate_doi_relaxed(
+    *,
+    title: str,
+    year: int | None,
+    items: list[dict],
+    author_surname: str | None,
+) -> str | None:
+    best_score = 0.0
+    best_doi: str | None = None
+    best_author_match = False
+    best_delta: int | None = None
+    best_year: int | None = None
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        score, doi = _candidate_rank_relaxed(
+            title=title,
+            year=year,
+            item=item,
+            author_surname=author_surname,
+        )
+        if doi is None or score < RELAXED_TITLE_MATCH_THRESHOLD:
+            continue
+        candidate_year = _candidate_year(item)
+        candidate_author_match = _candidate_author_match(item, author_surname)
+        candidate_delta = _year_delta(year, candidate_year)
+        if score > best_score:
+            best_score = score
+            best_doi = doi
+            best_author_match = candidate_author_match
+            best_delta = candidate_delta
+            best_year = candidate_year
+            continue
+        if abs(score - best_score) > 0.02:
+            continue
+        if candidate_author_match and not best_author_match:
+            best_doi = doi
+            best_author_match = True
+            best_delta = candidate_delta
+            best_year = candidate_year
+            continue
+        if best_delta is None and candidate_delta is not None:
+            best_doi = doi
+            best_author_match = candidate_author_match
+            best_delta = candidate_delta
+            best_year = candidate_year
+            continue
+        if best_delta is not None and candidate_delta is not None and candidate_delta < best_delta:
+            best_doi = doi
+            best_author_match = candidate_author_match
+            best_delta = candidate_delta
+            best_year = candidate_year
+            continue
+        if best_year is None or candidate_year is None:
+            continue
+        if candidate_year < best_year:
+            best_doi = doi
+            best_author_match = candidate_author_match
+            best_delta = candidate_delta
+            best_year = candidate_year
+    return best_doi
+
+
+def _best_candidate_doi(
+    *,
+    title: str,
+    year: int | None,
+    items: list[dict],
+    author_surname: str | None,
+) -> str | None:
+    strict_match = _best_candidate_doi_strict(
+        title=title,
+        year=year,
+        items=items,
+        author_surname=author_surname,
+    )
+    if strict_match:
+        return strict_match
+    return _best_candidate_doi_relaxed(
+        title=title,
+        year=year,
+        items=items,
+        author_surname=author_surname,
+    )
+
+
+def _works_client(email: str | None) -> Works:
+    if email:
+        etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
+        return Works(etiquette=etiquette)
+    return Works()
+
+
+def _fetch_items_sync(
+    *,
+    query: str,
+    author: str | None,
+    date_range: tuple[str, str] | None,
+    max_rows: int,
+    email: str | None,
+    min_interval_seconds: float,
+) -> list[dict]:
+    _rate_limit_wait(min_interval_seconds)
+    works = _works_client(email)
+    params = {"bibliographic": query}
+    if author:
+        params["author"] = author
+    request = works.query(**params)
+    if date_range is not None:
+        from_date, until_date = date_range
+        request = request.filter(from_pub_date=from_date, until_pub_date=until_date)
+    request = request.select(["DOI", "title", "issued", "score", "author"])
+    items: list[dict] = []
+    for entry in request:
+        if isinstance(entry, dict):
+            items.append(entry)
+        if len(items) >= max(max_rows, 1):
+            break
+    return items
+
+
+async def _fetch_items(
+    *,
+    query: str,
+    author: str | None,
+    date_range: tuple[str, str] | None,
+    max_rows: int,
+    email: str | None,
+) -> list[dict]:
+    timeout = max(float(settings.crossref_timeout_seconds), 0.5)
+    try:
+        return await asyncio.wait_for(
+            asyncio.to_thread(
+                _fetch_items_sync,
+                query=query,
+                author=author,
+                date_range=date_range,
+                max_rows=max_rows,
+                email=email,
+                min_interval_seconds=settings.crossref_min_interval_seconds,
+            ),
+            timeout=timeout,
+        )
+    except Exception:
+        return []
+
+
+async def discover_doi_for_publication(
+    *,
+    item: PublicationListItem | UnreadPublicationItem,
+    max_rows: int = 10,
+    email: str | None = None,
+) -> str | None:
+    title = (item.title or "").strip()
+    query = _normalized_query(title)
+    if not query:
+        return None
+    author = _query_author(item.scholar_label)
+    author_surname = _author_surname(item.scholar_label)
+    for date_range in _query_filters(item.year):
+        items = await _fetch_items(
+            query=query,
+            author=author,
+            date_range=date_range,
+            max_rows=max_rows,
+            email=email,
+        )
+        doi = _best_candidate_doi(
+            title=title,
+            year=item.year,
+            items=items,
+            author_surname=author_surname,
+        )
+        if doi:
+            logger.debug("crossref.doi_discovered", extra={"event": "crossref.doi_discovered"})
+            return doi
+    return None

build/lib/app/services/domains/dbops/__init__.py

@@ -0,0 +1,5 @@
+from app.services.domains.dbops.application import run_publication_link_repair
+from app.services.domains.dbops.integrity import collect_integrity_report
+from app.services.domains.dbops.query import list_repair_jobs
+
+__all__ = ["collect_integrity_report", "list_repair_jobs", "run_publication_link_repair"]

build/lib/app/services/domains/dbops/application.py

@@ -0,0 +1,364 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Any
+
+from sqlalchemy import delete, exists, func, select, update
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import DataRepairJob, IngestionQueueItem, Publication, ScholarProfile, ScholarPublication
+
+
+REPAIR_STATUS_PLANNED = "planned"
+REPAIR_STATUS_RUNNING = "running"
+REPAIR_STATUS_COMPLETED = "completed"
+REPAIR_STATUS_FAILED = "failed"
+SCOPE_MODE_SINGLE_USER = "single_user"
+SCOPE_MODE_ALL_USERS = "all_users"
+
+
+def _utcnow() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _normalize_scope_mode(scope_mode: str) -> str:
+    normalized = scope_mode.strip().lower()
+    if normalized in {SCOPE_MODE_SINGLE_USER, SCOPE_MODE_ALL_USERS}:
+        return normalized
+    raise ValueError("Unknown scope mode.")
+
+
+def _scope_user_id(*, scope_mode: str, user_id: int | None) -> int | None:
+    if scope_mode == SCOPE_MODE_SINGLE_USER:
+        if user_id is None:
+            raise ValueError("user_id is required when scope_mode=single_user.")
+        return int(user_id)
+    if user_id is not None:
+        raise ValueError("user_id must be omitted when scope_mode=all_users.")
+    return None
+
+
+def _scope_payload(
+    *,
+    scope_mode: str,
+    user_id: int | None,
+    target_scholar_profile_ids: list[int],
+    orphan_gc: bool,
+) -> dict[str, Any]:
+    payload: dict[str, Any] = {
+        "scope_mode": scope_mode,
+        "scholar_profile_ids": [int(value) for value in target_scholar_profile_ids],
+        "gc_orphan_publications": bool(orphan_gc),
+    }
+    if user_id is not None:
+        payload["user_id"] = int(user_id)
+    return payload
+
+
+async def _target_scholar_profile_ids(
+    db_session: AsyncSession,
+    *,
+    scope_mode: str,
+    user_id: int | None,
+    scholar_profile_ids: list[int] | None,
+) -> list[int]:
+    stmt = select(ScholarProfile.id)
+    if scope_mode == SCOPE_MODE_SINGLE_USER:
+        stmt = stmt.where(ScholarProfile.user_id == user_id)
+    if scholar_profile_ids:
+        normalized_ids = [int(value) for value in scholar_profile_ids]
+        stmt = stmt.where(ScholarProfile.id.in_(normalized_ids))
+    result = await db_session.execute(stmt.order_by(ScholarProfile.id.asc()))
+    ids = [int(row[0]) for row in result.all()]
+    if not ids:
+        raise ValueError("No target scholar profiles found for the requested scope.")
+    return ids
+
+
+async def _count_scope(
+    db_session: AsyncSession,
+    *,
+    user_id: int | None,
+    target_scholar_profile_ids: list[int],
+) -> dict[str, int]:
+    links_result = await db_session.execute(
+        select(func.count())
+        .select_from(ScholarPublication)
+        .where(ScholarPublication.scholar_profile_id.in_(target_scholar_profile_ids))
+    )
+    queue_stmt = (
+        select(func.count())
+        .select_from(IngestionQueueItem)
+        .where(IngestionQueueItem.scholar_profile_id.in_(target_scholar_profile_ids))
+    )
+    if user_id is not None:
+        queue_stmt = queue_stmt.where(IngestionQueueItem.user_id == user_id)
+    queue_result = await db_session.execute(queue_stmt)
+    return {
+        "target_scholar_count": len(target_scholar_profile_ids),
+        "links_in_scope": int(links_result.scalar_one() or 0),
+        "queue_items_in_scope": int(queue_result.scalar_one() or 0),
+    }
+
+
+async def _count_orphan_publications(db_session: AsyncSession) -> int:
+    stmt = (
+        select(func.count())
+        .select_from(Publication)
+        .where(
+            ~exists(
+                select(1).where(ScholarPublication.publication_id == Publication.id)
+            )
+        )
+    )
+    result = await db_session.execute(stmt)
+    return int(result.scalar_one() or 0)
+
+
+async def _create_job(
+    db_session: AsyncSession,
+    *,
+    requested_by: str | None,
+    scope: dict[str, Any],
+    dry_run: bool,
+) -> DataRepairJob:
+    job = DataRepairJob(
+        job_name="repair_publication_links",
+        requested_by=(requested_by or "").strip() or None,
+        scope=scope,
+        dry_run=dry_run,
+        status=REPAIR_STATUS_PLANNED,
+        summary={},
+    )
+    db_session.add(job)
+    await db_session.flush()
+    return job
+
+
+def _job_summary(
+    *,
+    counts: dict[str, int],
+    dry_run: bool,
+    links_deleted: int,
+    queue_items_deleted: int,
+    scholars_reset: int,
+    orphan_publications_before: int,
+    orphan_publications_deleted: int,
+) -> dict[str, Any]:
+    return {
+        **counts,
+        "dry_run": bool(dry_run),
+        "links_deleted": int(links_deleted),
+        "queue_items_deleted": int(queue_items_deleted),
+        "scholars_reset": int(scholars_reset),
+        "orphan_publications_before": int(orphan_publications_before),
+        "orphan_publications_deleted": int(orphan_publications_deleted),
+    }
+
+
+async def _delete_links_for_targets(db_session: AsyncSession, *, target_scholar_profile_ids: list[int]) -> int:
+    result = await db_session.execute(
+        delete(ScholarPublication).where(
+            ScholarPublication.scholar_profile_id.in_(target_scholar_profile_ids)
+        )
+    )
+    return int(result.rowcount or 0)
+
+
+async def _delete_queue_for_targets(
+    db_session: AsyncSession,
+    *,
+    user_id: int | None,
+    target_scholar_profile_ids: list[int],
+) -> int:
+    stmt = delete(IngestionQueueItem).where(
+        IngestionQueueItem.scholar_profile_id.in_(target_scholar_profile_ids)
+    )
+    if user_id is not None:
+        stmt = stmt.where(IngestionQueueItem.user_id == user_id)
+    result = await db_session.execute(stmt)
+    return int(result.rowcount or 0)
+
+
+async def _reset_scholar_tracking_state(
+    db_session: AsyncSession,
+    *,
+    user_id: int | None,
+    target_scholar_profile_ids: list[int],
+) -> int:
+    stmt = update(ScholarProfile).where(ScholarProfile.id.in_(target_scholar_profile_ids))
+    if user_id is not None:
+        stmt = stmt.where(ScholarProfile.user_id == user_id)
+    result = await db_session.execute(
+        stmt.values(
+            baseline_completed=False,
+            last_initial_page_fingerprint_sha256=None,
+            last_initial_page_checked_at=None,
+            last_run_dt=None,
+            last_run_status=None,
+        )
+    )
+    return int(result.rowcount or 0)
+
+
+async def _delete_orphan_publications(db_session: AsyncSession) -> int:
+    result = await db_session.execute(
+        delete(Publication).where(
+            ~exists(select(1).where(ScholarPublication.publication_id == Publication.id))
+        )
+    )
+    return int(result.rowcount or 0)
+
+
+async def _mutation_counts(
+    db_session: AsyncSession,
+    *,
+    user_id: int | None,
+    target_ids: list[int],
+    dry_run: bool,
+    gc_orphan_publications: bool,
+) -> tuple[int, int, int, int]:
+    if dry_run:
+        return 0, 0, 0, 0
+    links_deleted = await _delete_links_for_targets(
+        db_session,
+        target_scholar_profile_ids=target_ids,
+    )
+    queue_deleted = await _delete_queue_for_targets(
+        db_session,
+        user_id=user_id,
+        target_scholar_profile_ids=target_ids,
+    )
+    scholars_reset = await _reset_scholar_tracking_state(
+        db_session,
+        user_id=user_id,
+        target_scholar_profile_ids=target_ids,
+    )
+    orphan_deleted = 0
+    if gc_orphan_publications:
+        orphan_deleted = await _delete_orphan_publications(db_session)
+    return links_deleted, queue_deleted, scholars_reset, orphan_deleted
+
+
+def _result_payload(*, job: DataRepairJob, scope: dict[str, Any], summary: dict[str, Any]) -> dict[str, Any]:
+    return {
+        "job_id": int(job.id),
+        "status": job.status,
+        "scope": scope,
+        "summary": summary,
+    }
+
+
+async def _complete_job(
+    db_session: AsyncSession,
+    *,
+    job: DataRepairJob,
+    summary: dict[str, Any],
+    scope: dict[str, Any],
+) -> dict[str, Any]:
+    job.summary = summary
+    job.status = REPAIR_STATUS_COMPLETED
+    job.finished_at = _utcnow()
+    await db_session.commit()
+    return _result_payload(job=job, scope=scope, summary=summary)
+
+
+async def _fail_job(db_session: AsyncSession, *, job: DataRepairJob, error: Exception) -> None:
+    await db_session.rollback()
+    job.status = REPAIR_STATUS_FAILED
+    job.error_text = str(error)
+    job.finished_at = _utcnow()
+    db_session.add(job)
+    await db_session.commit()
+
+
+async def _prepare_repair_job(
+    db_session: AsyncSession,
+    *,
+    scope_mode: str,
+    user_id: int | None,
+    scholar_profile_ids: list[int] | None,
+    dry_run: bool,
+    gc_orphan_publications: bool,
+    requested_by: str | None,
+) -> tuple[int | None, list[int], dict[str, Any], DataRepairJob]:
+    normalized_scope = _normalize_scope_mode(scope_mode)
+    scope_user_id = _scope_user_id(scope_mode=normalized_scope, user_id=user_id)
+    target_ids = await _target_scholar_profile_ids(
+        db_session,
+        scope_mode=normalized_scope,
+        user_id=scope_user_id,
+        scholar_profile_ids=scholar_profile_ids,
+    )
+    scope = _scope_payload(
+        scope_mode=normalized_scope,
+        user_id=scope_user_id,
+        target_scholar_profile_ids=target_ids,
+        orphan_gc=gc_orphan_publications,
+    )
+    job = await _create_job(db_session, requested_by=requested_by, scope=scope, dry_run=dry_run)
+    job.status = REPAIR_STATUS_RUNNING
+    job.started_at = _utcnow()
+    return scope_user_id, target_ids, scope, job
+
+
+async def _build_repair_summary(
+    db_session: AsyncSession,
+    *,
+    scope_user_id: int | None,
+    target_ids: list[int],
+    dry_run: bool,
+    gc_orphan_publications: bool,
+) -> dict[str, Any]:
+    counts = await _count_scope(db_session, user_id=scope_user_id, target_scholar_profile_ids=target_ids)
+    orphan_before = await _count_orphan_publications(db_session)
+    links_deleted, queue_deleted, scholars_reset, orphan_deleted = await _mutation_counts(
+        db_session,
+        user_id=scope_user_id,
+        target_ids=target_ids,
+        dry_run=dry_run,
+        gc_orphan_publications=gc_orphan_publications,
+    )
+    return _job_summary(
+        counts=counts,
+        dry_run=dry_run,
+        links_deleted=links_deleted,
+        queue_items_deleted=queue_deleted,
+        scholars_reset=scholars_reset,
+        orphan_publications_before=orphan_before,
+        orphan_publications_deleted=orphan_deleted,
+    )
+
+
+async def run_publication_link_repair(
+    db_session: AsyncSession,
+    *,
+    scope_mode: str = SCOPE_MODE_SINGLE_USER,
+    user_id: int | None = None,
+    scholar_profile_ids: list[int] | None = None,
+    dry_run: bool = True,
+    gc_orphan_publications: bool = False,
+    requested_by: str | None = None,
+) -> dict[str, Any]:
+    scope_user_id, target_ids, scope, job = await _prepare_repair_job(
+        db_session,
+        scope_mode=scope_mode,
+        user_id=user_id,
+        scholar_profile_ids=scholar_profile_ids,
+        dry_run=dry_run,
+        gc_orphan_publications=gc_orphan_publications,
+        requested_by=requested_by,
+    )
+
+    try:
+        summary = await _build_repair_summary(
+            db_session,
+            scope_user_id=scope_user_id,
+            target_ids=target_ids,
+            dry_run=dry_run,
+            gc_orphan_publications=gc_orphan_publications,
+        )
+        return await _complete_job(db_session, job=job, summary=summary, scope=scope)
+    except Exception as exc:
+        await _fail_job(db_session, job=job, error=exc)
+        raise

build/lib/app/services/domains/dbops/integrity.py

@@ -0,0 +1,175 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import Any
+
+from sqlalchemy import func, select, text
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import Publication
+
+INTEGRITY_CHECK_DEFS = (
+    (
+        "legacy_cluster_id_format",
+        "warning",
+        "Publications with non-namespaced cluster IDs.",
+    ),
+    (
+        "negative_citation_count",
+        "failure",
+        "Publications with negative citation counts.",
+    ),
+    (
+        "orphan_publications",
+        "warning",
+        "Publications with no scholar links.",
+    ),
+    (
+        "orphan_scholar_publication_links",
+        "failure",
+        "Link rows missing parent scholar/publication.",
+    ),
+    (
+        "duplicate_fingerprint_keys",
+        "failure",
+        "Duplicate publication fingerprint keys.",
+    ),
+    (
+        "duplicate_cluster_ids",
+        "failure",
+        "Duplicate non-null publication cluster IDs.",
+    ),
+    (
+        "missing_pdf_url",
+        "metric",
+        "Publications without a resolved PDF URL.",
+    ),
+)
+
+
+async def _legacy_cluster_id_count(db_session: AsyncSession) -> int:
+    stmt = (
+        select(func.count())
+        .select_from(Publication)
+        .where(Publication.cluster_id.is_not(None))
+        .where(~Publication.cluster_id.like("cfv:%"))
+        .where(~Publication.cluster_id.like("cluster:%"))
+    )
+    result = await db_session.execute(stmt)
+    return int(result.scalar_one() or 0)
+
+
+async def _negative_citation_count(db_session: AsyncSession) -> int:
+    result = await db_session.execute(
+        select(func.count()).select_from(Publication).where(Publication.citation_count < 0)
+    )
+    return int(result.scalar_one() or 0)
+
+
+async def _missing_pdf_url_count(db_session: AsyncSession) -> int:
+    result = await db_session.execute(
+        select(func.count()).select_from(Publication).where(Publication.pdf_url.is_(None))
+    )
+    return int(result.scalar_one() or 0)
+
+
+async def _count_from_sql(db_session: AsyncSession, *, sql: str) -> int:
+    result = await db_session.execute(text(sql))
+    return int(result.scalar_one() or 0)
+
+
+def _issues_for_severity(*, checks: list[dict[str, Any]], severity: str) -> list[str]:
+    return [row["name"] for row in checks if row["severity"] == severity and row["count"] > 0]
+
+
+def _check_row(*, name: str, count: int, severity: str, message: str) -> dict[str, Any]:
+    return {
+        "name": name,
+        "count": int(count),
+        "severity": severity,
+        "message": message,
+    }
+
+
+async def _collect_counts(db_session: AsyncSession) -> dict[str, int]:
+    orphan_publications = await _count_from_sql(
+        db_session,
+        sql=(
+            "SELECT count(*) FROM publications p "
+            "LEFT JOIN scholar_publications sp ON sp.publication_id = p.id "
+            "WHERE sp.publication_id IS NULL"
+        ),
+    )
+    orphan_links = await _count_from_sql(
+        db_session,
+        sql=(
+            "SELECT count(*) FROM scholar_publications sp "
+            "LEFT JOIN publications p ON p.id = sp.publication_id "
+            "LEFT JOIN scholar_profiles s ON s.id = sp.scholar_profile_id "
+            "WHERE p.id IS NULL OR s.id IS NULL"
+        ),
+    )
+    duplicate_fingerprints = await _count_from_sql(
+        db_session,
+        sql=(
+            "SELECT count(*) FROM ("
+            "SELECT fingerprint_sha256 FROM publications "
+            "GROUP BY fingerprint_sha256 HAVING count(*) > 1"
+            ") dup"
+        ),
+    )
+    duplicate_cluster_ids = await _count_from_sql(
+        db_session,
+        sql=(
+            "SELECT count(*) FROM ("
+            "SELECT cluster_id FROM publications "
+            "WHERE cluster_id IS NOT NULL "
+            "GROUP BY cluster_id HAVING count(*) > 1"
+            ") dup"
+        ),
+    )
+    return {
+        "legacy_cluster_id_format": await _legacy_cluster_id_count(db_session),
+        "negative_citation_count": await _negative_citation_count(db_session),
+        "orphan_publications": orphan_publications,
+        "orphan_scholar_publication_links": orphan_links,
+        "duplicate_fingerprint_keys": duplicate_fingerprints,
+        "duplicate_cluster_ids": duplicate_cluster_ids,
+        "missing_pdf_url": await _missing_pdf_url_count(db_session),
+    }
+
+
+def _build_checks(*, counts: dict[str, int]) -> list[dict[str, Any]]:
+    return [
+        _check_row(
+            name=name,
+            count=counts[name],
+            severity=severity,
+            message=message,
+        )
+        for name, severity, message in INTEGRITY_CHECK_DEFS
+    ]
+
+
+def _status_from_issues(*, failures: list[str], warnings: list[str]) -> str:
+    status = "ok"
+    if failures:
+        status = "failed"
+    elif warnings:
+        status = "warning"
+    return status
+
+
+async def collect_integrity_report(db_session: AsyncSession) -> dict[str, Any]:
+    counts = await _collect_counts(db_session)
+    checks = _build_checks(counts=counts)
+    failures = _issues_for_severity(checks=checks, severity="failure")
+    warnings = _issues_for_severity(checks=checks, severity="warning")
+
+    return {
+        "status": _status_from_issues(failures=failures, warnings=warnings),
+        "checked_at": datetime.now(timezone.utc).isoformat(),
+        "failures": failures,
+        "warnings": warnings,
+        "checks": checks,
+    }

build/lib/app/services/domains/dbops/query.py

@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import DataRepairJob
+
+
+def _bounded_limit(limit: int) -> int:
+    return max(1, min(int(limit), 200))
+
+
+async def list_repair_jobs(
+    db_session: AsyncSession,
+    *,
+    limit: int = 50,
+) -> list[DataRepairJob]:
+    bounded = _bounded_limit(limit)
+    result = await db_session.execute(
+        select(DataRepairJob).order_by(DataRepairJob.created_at.desc()).limit(bounded)
+    )
+    return list(result.scalars())

build/lib/app/services/domains/doi/normalize.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+
+import re
+from urllib.parse import unquote
+
+DOI_RE = re.compile(r"10\.\d{4,9}/[-._;()/:A-Z0-9]+", re.I)
+
+
+def normalize_doi(value: str | None) -> str | None:
+    if not value:
+        return None
+    match = DOI_RE.search(unquote(value))
+    if not match:
+        return None
+    return match.group(0).rstrip(" .;,)").lower()
+
+
+def first_doi_from_texts(*values: str | None) -> str | None:
+    for value in values:
+        doi = normalize_doi(value)
+        if doi:
+            return doi
+    return None

build/lib/app/services/domains/ingestion/__init__.py

@@ -0,0 +1 @@
+from __future__ import annotations

build/lib/app/services/domains/ingestion/constants.py

@@ -0,0 +1,31 @@
+from __future__ import annotations
+
+import re
+
+from app.services.domains.scholar.parser import ParseState
+
+TITLE_ALNUM_RE = re.compile(r"[^a-z0-9]+")
+WORD_RE = re.compile(r"[a-z0-9]+")
+HTML_TAG_RE = re.compile(r"<[^>]+>", re.S)
+SPACE_RE = re.compile(r"\s+")
+
+FAILED_STATES = {
+    ParseState.BLOCKED_OR_CAPTCHA.value,
+    ParseState.LAYOUT_CHANGED.value,
+    ParseState.NETWORK_ERROR.value,
+    "ingestion_error",
+}
+
+FAILURE_BUCKET_BLOCKED = "blocked_or_captcha"
+FAILURE_BUCKET_NETWORK = "network_error"
+FAILURE_BUCKET_LAYOUT = "layout_changed"
+FAILURE_BUCKET_INGESTION = "ingestion_error"
+FAILURE_BUCKET_OTHER = "other_failure"
+
+RUN_LOCK_NAMESPACE = 8217
+RESUMABLE_PARTIAL_REASONS = {
+    "max_pages_reached",
+    "pagination_cursor_stalled",
+}
+RESUMABLE_PARTIAL_REASON_PREFIXES = ("page_state_network_error",)
+INITIAL_PAGE_FINGERPRINT_MAX_PUBLICATIONS = 30

build/lib/app/services/domains/ingestion/fingerprints.py

@@ -0,0 +1,175 @@
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+from typing import Any
+from urllib.parse import urljoin
+
+from app.services.domains.ingestion.constants import (
+    HTML_TAG_RE,
+    INITIAL_PAGE_FINGERPRINT_MAX_PUBLICATIONS,
+    SPACE_RE,
+    TITLE_ALNUM_RE,
+    WORD_RE,
+)
+from app.services.domains.scholar.parser import ParseState, ParsedProfilePage, PublicationCandidate
+
+
+def normalize_title(value: str) -> str:
+    lowered = value.lower()
+    return TITLE_ALNUM_RE.sub("", lowered)
+
+
+def _first_author_last_name(authors_text: str | None) -> str:
+    if not authors_text:
+        return ""
+    first_author = authors_text.split(",", maxsplit=1)[0].strip().lower()
+    words = WORD_RE.findall(first_author)
+    if not words:
+        return ""
+    return words[-1]
+
+
+def _first_venue_word(venue_text: str | None) -> str:
+    if not venue_text:
+        return ""
+    words = WORD_RE.findall(venue_text.lower())
+    if not words:
+        return ""
+    return words[0]
+
+
+def build_publication_fingerprint(candidate: PublicationCandidate) -> str:
+    canonical = "|".join(
+        [
+            normalize_title(candidate.title),
+            str(candidate.year) if candidate.year is not None else "",
+            _first_author_last_name(candidate.authors_text),
+            _first_venue_word(candidate.venue_text),
+        ]
+    )
+    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
+
+
+def build_initial_page_fingerprint(parsed_page: ParsedProfilePage) -> str | None:
+    if parsed_page.state not in {ParseState.OK, ParseState.NO_RESULTS}:
+        return None
+
+    normalized_rows: list[dict[str, Any]] = []
+    for publication in parsed_page.publications[:INITIAL_PAGE_FINGERPRINT_MAX_PUBLICATIONS]:
+        normalized_rows.append(
+            {
+                "cluster_id": publication.cluster_id or "",
+                "title_normalized": normalize_title(publication.title),
+                "year": publication.year,
+                "citation_count": publication.citation_count,
+            }
+        )
+
+    payload = {
+        "state": parsed_page.state.value,
+        "articles_range": parsed_page.articles_range or "",
+        "has_show_more_button": parsed_page.has_show_more_button,
+        "profile_name": parsed_page.profile_name or "",
+        "publications": normalized_rows,
+    }
+    canonical = json.dumps(
+        payload,
+        sort_keys=True,
+        separators=(",", ":"),
+        ensure_ascii=True,
+    )
+    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
+
+
+def build_publication_url(path_or_url: str | None) -> str | None:
+    if not path_or_url:
+        return None
+    return urljoin("https://scholar.google.com", path_or_url)
+
+
+def _next_cstart_value(*, articles_range: str | None, fallback: int) -> int:
+    if articles_range:
+        numbers = re.findall(r"\d+", articles_range)
+        if len(numbers) >= 2:
+            try:
+                return int(numbers[1])
+            except ValueError:
+                pass
+    return int(fallback)
+
+
+def _title_tokens(value: str) -> set[str]:
+    """Extract normalized word tokens for fuzzy title comparison."""
+    return set(WORD_RE.findall(value.lower()))
+
+
+def fuzzy_titles_match(
+    title_a: str,
+    title_b: str,
+    *,
+    threshold: float = 0.85,
+) -> bool:
+    """Return True if two titles are near-duplicates by token-level Jaccard similarity.
+
+    A threshold of 0.85 catches common academic duplicate patterns:
+    differences in punctuation, minor word variations, subtitle changes.
+    """
+    tokens_a = _title_tokens(title_a)
+    tokens_b = _title_tokens(title_b)
+    if not tokens_a or not tokens_b:
+        return False
+    intersection = tokens_a & tokens_b
+    union = tokens_a | tokens_b
+    return (len(intersection) / len(union)) >= threshold
+
+
+def _dedupe_publication_candidates(
+    publications: list[PublicationCandidate],
+) -> list[PublicationCandidate]:
+    deduped: list[PublicationCandidate] = []
+    seen: set[str] = set()
+    seen_titles: list[tuple[str, int]] = []  # (normalized_title, index into deduped)
+
+    for publication in publications:
+        if publication.cluster_id:
+            identity = f"cluster:{publication.cluster_id}"
+        else:
+            identity = "|".join(
+                [
+                    "fallback",
+                    normalize_title(publication.title),
+                    str(publication.year) if publication.year is not None else "",
+                    _first_author_last_name(publication.authors_text),
+                    _first_venue_word(publication.venue_text),
+                ]
+            )
+        if identity in seen:
+            continue
+
+        # Fuzzy title check — catch near-identical titles not caught by exact fingerprint
+        norm_title = normalize_title(publication.title)
+        is_fuzzy_dup = False
+        for existing_title, _idx in seen_titles:
+            if fuzzy_titles_match(norm_title, existing_title):
+                is_fuzzy_dup = True
+                break
+        if is_fuzzy_dup:
+            continue
+
+        seen.add(identity)
+        seen_titles.append((norm_title, len(deduped)))
+        deduped.append(publication)
+    return deduped
+
+
+def _build_body_excerpt(body: str, *, max_chars: int = 220) -> str | None:
+    if not body:
+        return None
+    flattened = SPACE_RE.sub(" ", HTML_TAG_RE.sub(" ", body)).strip()
+    if not flattened:
+        return None
+    if len(flattened) <= max_chars:
+        return flattened
+    return f"{flattened[:max_chars - 1]}..."

build/lib/app/services/domains/ingestion/queue.py

@@ -0,0 +1,348 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import IngestionQueueItem, QueueItemStatus
+
+
+@dataclass(frozen=True)
+class ContinuationQueueJob:
+    id: int
+    user_id: int
+    scholar_profile_id: int
+    resume_cstart: int
+    reason: str
+    status: str
+    attempt_count: int
+    next_attempt_dt: datetime
+
+
+ACTIVE_QUEUE_STATUSES: tuple[str, ...] = (
+    QueueItemStatus.QUEUED.value,
+    QueueItemStatus.RETRYING.value,
+)
+
+
+def normalize_cstart(value: int | None) -> int:
+    if value is None:
+        return 0
+    return max(0, int(value))
+
+
+def compute_backoff_seconds(*, base_seconds: int, attempt_count: int, max_seconds: int) -> int:
+    base = max(1, int(base_seconds))
+    attempts = max(1, int(attempt_count))
+    maximum = max(base, int(max_seconds))
+    seconds = base * (2 ** max(0, attempts - 1))
+    return min(seconds, maximum)
+
+
+async def _get_item_for_user_scholar(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    scholar_profile_id: int,
+) -> IngestionQueueItem | None:
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(
+            IngestionQueueItem.user_id == user_id,
+            IngestionQueueItem.scholar_profile_id == scholar_profile_id,
+        )
+    )
+    return result.scalar_one_or_none()
+
+
+def _build_queue_item(
+    *,
+    now: datetime,
+    user_id: int,
+    scholar_profile_id: int,
+    normalized_cstart: int,
+    reason: str,
+    run_id: int | None,
+    next_attempt_dt: datetime,
+) -> IngestionQueueItem:
+    return IngestionQueueItem(
+        user_id=user_id,
+        scholar_profile_id=scholar_profile_id,
+        resume_cstart=normalized_cstart,
+        reason=reason,
+        status=QueueItemStatus.QUEUED.value,
+        attempt_count=0,
+        next_attempt_dt=next_attempt_dt,
+        last_run_id=run_id,
+        last_error=None,
+        dropped_reason=None,
+        dropped_at=None,
+        created_at=now,
+        updated_at=now,
+    )
+
+
+def _update_existing_queue_item(
+    *,
+    item: IngestionQueueItem,
+    now: datetime,
+    normalized_cstart: int,
+    reason: str,
+    run_id: int | None,
+    next_attempt_dt: datetime,
+) -> None:
+    item.resume_cstart = normalized_cstart
+    item.reason = reason
+    if item.status == QueueItemStatus.DROPPED.value:
+        item.attempt_count = 0
+    item.status = QueueItemStatus.QUEUED.value
+    item.next_attempt_dt = next_attempt_dt
+    item.last_run_id = run_id
+    item.last_error = None
+    item.dropped_reason = None
+    item.dropped_at = None
+    item.updated_at = now
+
+
+async def upsert_job(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    scholar_profile_id: int,
+    resume_cstart: int,
+    reason: str,
+    run_id: int | None,
+    delay_seconds: int,
+) -> IngestionQueueItem:
+    now = datetime.now(timezone.utc)
+    next_attempt_dt = now + timedelta(seconds=max(0, int(delay_seconds)))
+    item = await _get_item_for_user_scholar(
+        db_session,
+        user_id=user_id,
+        scholar_profile_id=scholar_profile_id,
+    )
+    normalized_cstart = normalize_cstart(resume_cstart)
+    if item is None:
+        item = _build_queue_item(
+            now=now,
+            user_id=user_id,
+            scholar_profile_id=scholar_profile_id,
+            normalized_cstart=normalized_cstart,
+            reason=reason,
+            run_id=run_id,
+            next_attempt_dt=next_attempt_dt,
+        )
+        db_session.add(item)
+        return item
+
+    _update_existing_queue_item(
+        item=item,
+        now=now,
+        normalized_cstart=normalized_cstart,
+        reason=reason,
+        run_id=run_id,
+        next_attempt_dt=next_attempt_dt,
+    )
+    return item
+
+
+async def clear_job_for_scholar(
+    db_session: AsyncSession,
+    *,
+    user_id: int,
+    scholar_profile_id: int,
+) -> bool:
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(
+            IngestionQueueItem.user_id == user_id,
+            IngestionQueueItem.scholar_profile_id == scholar_profile_id,
+        )
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return False
+    await db_session.delete(item)
+    return True
+
+
+async def delete_job_by_id(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+) -> bool:
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return False
+    await db_session.delete(item)
+    return True
+
+
+async def list_due_jobs(
+    db_session: AsyncSession,
+    *,
+    now: datetime,
+    limit: int,
+) -> list[ContinuationQueueJob]:
+    result = await db_session.execute(
+        select(IngestionQueueItem)
+        .where(
+            IngestionQueueItem.next_attempt_dt <= now,
+            IngestionQueueItem.status.in_(ACTIVE_QUEUE_STATUSES),
+        )
+        .order_by(
+            IngestionQueueItem.next_attempt_dt.asc(),
+            IngestionQueueItem.id.asc(),
+        )
+        .limit(limit)
+    )
+    rows = list(result.scalars().all())
+    jobs: list[ContinuationQueueJob] = []
+    for row in rows:
+        jobs.append(
+            ContinuationQueueJob(
+                id=int(row.id),
+                user_id=int(row.user_id),
+                scholar_profile_id=int(row.scholar_profile_id),
+                resume_cstart=normalize_cstart(row.resume_cstart),
+                reason=row.reason,
+                status=row.status,
+                attempt_count=int(row.attempt_count),
+                next_attempt_dt=row.next_attempt_dt,
+            )
+        )
+    return jobs
+
+
+async def increment_attempt_count(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    item.attempt_count = int(item.attempt_count or 0) + 1
+    item.updated_at = now
+    return item
+
+
+async def reset_attempt_count(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    item.attempt_count = 0
+    item.updated_at = now
+    return item
+
+
+async def reschedule_job(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+    delay_seconds: int,
+    reason: str,
+    error: str | None = None,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    item.next_attempt_dt = now + timedelta(seconds=max(1, int(delay_seconds)))
+    item.status = QueueItemStatus.QUEUED.value
+    item.reason = reason
+    item.last_error = error
+    item.dropped_reason = None
+    item.dropped_at = None
+    item.updated_at = now
+    return item
+
+
+async def mark_retrying(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+    reason: str | None = None,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    if item.status == QueueItemStatus.DROPPED.value:
+        return item
+    item.status = QueueItemStatus.RETRYING.value
+    if reason:
+        item.reason = reason
+    item.updated_at = now
+    return item
+
+
+async def mark_dropped(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+    reason: str,
+    error: str | None = None,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    item.status = QueueItemStatus.DROPPED.value
+    item.reason = "dropped"
+    item.dropped_reason = reason
+    item.dropped_at = now
+    if error is not None:
+        item.last_error = error
+    item.updated_at = now
+    return item
+
+
+async def mark_queued_now(
+    db_session: AsyncSession,
+    *,
+    job_id: int,
+    reason: str,
+    reset_attempt_count: bool = False,
+) -> IngestionQueueItem | None:
+    now = datetime.now(timezone.utc)
+    result = await db_session.execute(
+        select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
+    )
+    item = result.scalar_one_or_none()
+    if item is None:
+        return None
+    item.status = QueueItemStatus.QUEUED.value
+    item.reason = reason
+    item.next_attempt_dt = now
+    if reset_attempt_count:
+        item.attempt_count = 0
+    item.last_error = None
+    item.dropped_reason = None
+    item.dropped_at = None
+    item.updated_at = now
+    return item

build/lib/app/services/domains/ingestion/safety.py

@@ -0,0 +1,282 @@
+from __future__ import annotations
+
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+from app.db.models import UserSetting
+
+COOLDOWN_REASON_BLOCKED_FAILURE_THRESHOLD = "blocked_failure_threshold_exceeded"
+COOLDOWN_REASON_NETWORK_FAILURE_THRESHOLD = "network_failure_threshold_exceeded"
+
+_COUNTER_CONSECUTIVE_BLOCKED_RUNS = "consecutive_blocked_runs"
+_COUNTER_CONSECUTIVE_NETWORK_RUNS = "consecutive_network_runs"
+_COUNTER_COOLDOWN_ENTRY_COUNT = "cooldown_entry_count"
+_COUNTER_BLOCKED_START_COUNT = "blocked_start_count"
+_COUNTER_LAST_BLOCKED_FAILURE_COUNT = "last_blocked_failure_count"
+_COUNTER_LAST_NETWORK_FAILURE_COUNT = "last_network_failure_count"
+_COUNTER_LAST_EVALUATED_RUN_ID = "last_evaluated_run_id"
+
+
+def _utcnow() -> datetime:
+    return datetime.now(timezone.utc)
+
+
+def _safe_int(value: Any, default: int = 0) -> int:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return default
+
+
+def _safe_optional_int(value: Any) -> int | None:
+    if value is None:
+        return None
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def _normalize_datetime(value: datetime | None) -> datetime | None:
+    if value is None:
+        return None
+    if value.tzinfo is None:
+        return value.replace(tzinfo=timezone.utc)
+    return value.astimezone(timezone.utc)
+
+
+def _state_dict(settings: UserSetting) -> dict[str, Any]:
+    state = settings.scrape_safety_state
+    if isinstance(state, dict):
+        return state
+    return {}
+
+
+def _counters_from_state(settings: UserSetting) -> dict[str, Any]:
+    state = _state_dict(settings)
+    return {
+        _COUNTER_CONSECUTIVE_BLOCKED_RUNS: max(
+            0,
+            _safe_int(state.get(_COUNTER_CONSECUTIVE_BLOCKED_RUNS), 0),
+        ),
+        _COUNTER_CONSECUTIVE_NETWORK_RUNS: max(
+            0,
+            _safe_int(state.get(_COUNTER_CONSECUTIVE_NETWORK_RUNS), 0),
+        ),
+        _COUNTER_COOLDOWN_ENTRY_COUNT: max(
+            0,
+            _safe_int(state.get(_COUNTER_COOLDOWN_ENTRY_COUNT), 0),
+        ),
+        _COUNTER_BLOCKED_START_COUNT: max(
+            0,
+            _safe_int(state.get(_COUNTER_BLOCKED_START_COUNT), 0),
+        ),
+        _COUNTER_LAST_BLOCKED_FAILURE_COUNT: max(
+            0,
+            _safe_int(state.get(_COUNTER_LAST_BLOCKED_FAILURE_COUNT), 0),
+        ),
+        _COUNTER_LAST_NETWORK_FAILURE_COUNT: max(
+            0,
+            _safe_int(state.get(_COUNTER_LAST_NETWORK_FAILURE_COUNT), 0),
+        ),
+        _COUNTER_LAST_EVALUATED_RUN_ID: _safe_optional_int(
+            state.get(_COUNTER_LAST_EVALUATED_RUN_ID),
+        ),
+    }
+
+
+def _cooldown_reason_label(reason: str | None) -> str | None:
+    if reason == COOLDOWN_REASON_BLOCKED_FAILURE_THRESHOLD:
+        return "Blocked responses exceeded safety threshold"
+    if reason == COOLDOWN_REASON_NETWORK_FAILURE_THRESHOLD:
+        return "Network failures exceeded safety threshold"
+    return None
+
+
+def _recommended_action(reason: str | None) -> str | None:
+    if reason == COOLDOWN_REASON_BLOCKED_FAILURE_THRESHOLD:
+        return (
+            "Google Scholar appears to be blocking requests. Wait for cooldown to expire, "
+            "increase request delay, and avoid repeated manual retries."
+        )
+    if reason == COOLDOWN_REASON_NETWORK_FAILURE_THRESHOLD:
+        return (
+            "Network failures crossed the threshold. Verify connectivity and retry after cooldown."
+        )
+    return None
+
+
+def is_cooldown_active(
+    settings: UserSetting,
+    *,
+    now_utc: datetime | None = None,
+) -> bool:
+    now = now_utc or _utcnow()
+    cooldown_until = _normalize_datetime(settings.scrape_cooldown_until)
+    if cooldown_until is None:
+        return False
+    return cooldown_until > now
+
+
+def clear_expired_cooldown(
+    settings: UserSetting,
+    *,
+    now_utc: datetime | None = None,
+) -> bool:
+    now = now_utc or _utcnow()
+    cooldown_until = _normalize_datetime(settings.scrape_cooldown_until)
+    if cooldown_until is None:
+        return False
+    if cooldown_until > now:
+        return False
+    settings.scrape_cooldown_until = None
+    settings.scrape_cooldown_reason = None
+    return True
+
+
+def register_cooldown_blocked_start(
+    settings: UserSetting,
+    *,
+    now_utc: datetime | None = None,
+) -> dict[str, Any]:
+    now = now_utc or _utcnow()
+    counters = _counters_from_state(settings)
+    counters[_COUNTER_BLOCKED_START_COUNT] = int(counters[_COUNTER_BLOCKED_START_COUNT]) + 1
+    settings.scrape_safety_state = counters
+    return get_safety_state_payload(settings, now_utc=now)
+
+
+def _update_run_counters(
+    *,
+    counters: dict[str, Any],
+    run_id: int,
+    blocked_failure_count: int,
+    network_failure_count: int,
+) -> tuple[int, int]:
+    bounded_blocked_failures = max(0, int(blocked_failure_count))
+    bounded_network_failures = max(0, int(network_failure_count))
+    counters[_COUNTER_LAST_BLOCKED_FAILURE_COUNT] = bounded_blocked_failures
+    counters[_COUNTER_LAST_NETWORK_FAILURE_COUNT] = bounded_network_failures
+    counters[_COUNTER_LAST_EVALUATED_RUN_ID] = int(run_id)
+    counters[_COUNTER_CONSECUTIVE_BLOCKED_RUNS] = (
+        int(counters[_COUNTER_CONSECUTIVE_BLOCKED_RUNS]) + 1
+        if bounded_blocked_failures > 0
+        else 0
+    )
+    counters[_COUNTER_CONSECUTIVE_NETWORK_RUNS] = (
+        int(counters[_COUNTER_CONSECUTIVE_NETWORK_RUNS]) + 1
+        if bounded_network_failures > 0
+        else 0
+    )
+    return bounded_blocked_failures, bounded_network_failures
+
+
+def _resolve_cooldown_trigger(
+    *,
+    blocked_failures: int,
+    network_failures: int,
+    blocked_failure_threshold: int,
+    network_failure_threshold: int,
+    blocked_cooldown_seconds: int,
+    network_cooldown_seconds: int,
+) -> tuple[str | None, int]:
+    if blocked_failures >= max(1, int(blocked_failure_threshold)):
+        return COOLDOWN_REASON_BLOCKED_FAILURE_THRESHOLD, max(60, int(blocked_cooldown_seconds))
+    if network_failures >= max(1, int(network_failure_threshold)):
+        return COOLDOWN_REASON_NETWORK_FAILURE_THRESHOLD, max(60, int(network_cooldown_seconds))
+    return None, 0
+
+
+def _apply_cooldown_decision(
+    *,
+    settings: UserSetting,
+    counters: dict[str, Any],
+    now: datetime,
+    reason: str | None,
+    cooldown_seconds: int,
+) -> None:
+    if reason is None:
+        clear_expired_cooldown(settings, now_utc=now)
+        return
+    settings.scrape_cooldown_reason = reason
+    settings.scrape_cooldown_until = now + timedelta(seconds=max(60, int(cooldown_seconds)))
+    counters[_COUNTER_COOLDOWN_ENTRY_COUNT] = int(counters[_COUNTER_COOLDOWN_ENTRY_COUNT]) + 1
+
+
+def apply_run_safety_outcome(
+    settings: UserSetting,
+    *,
+    run_id: int,
+    blocked_failure_count: int,
+    network_failure_count: int,
+    blocked_failure_threshold: int,
+    network_failure_threshold: int,
+    blocked_cooldown_seconds: int,
+    network_cooldown_seconds: int,
+    now_utc: datetime | None = None,
+) -> tuple[dict[str, Any], str | None]:
+    now = now_utc or _utcnow()
+    counters = _counters_from_state(settings)
+    blocked_failures, network_failures = _update_run_counters(
+        counters=counters,
+        run_id=run_id,
+        blocked_failure_count=blocked_failure_count,
+        network_failure_count=network_failure_count,
+    )
+    reason, cooldown_seconds = _resolve_cooldown_trigger(
+        blocked_failures=blocked_failures,
+        network_failures=network_failures,
+        blocked_failure_threshold=blocked_failure_threshold,
+        network_failure_threshold=network_failure_threshold,
+        blocked_cooldown_seconds=blocked_cooldown_seconds,
+        network_cooldown_seconds=network_cooldown_seconds,
+    )
+    _apply_cooldown_decision(
+        settings=settings,
+        counters=counters,
+        now=now,
+        reason=reason,
+        cooldown_seconds=cooldown_seconds,
+    )
+    settings.scrape_safety_state = counters
+    return get_safety_state_payload(settings, now_utc=now), reason
+
+
+def get_safety_state_payload(
+    settings: UserSetting,
+    *,
+    now_utc: datetime | None = None,
+) -> dict[str, Any]:
+    now = now_utc or _utcnow()
+    cooldown_until = _normalize_datetime(settings.scrape_cooldown_until)
+    cooldown_active = bool(cooldown_until is not None and cooldown_until > now)
+    cooldown_remaining_seconds = 0
+    if cooldown_active and cooldown_until is not None:
+        cooldown_remaining_seconds = max(0, int((cooldown_until - now).total_seconds()))
+
+    reason = settings.scrape_cooldown_reason if cooldown_active else None
+
+    return {
+        "cooldown_active": cooldown_active,
+        "cooldown_reason": reason,
+        "cooldown_reason_label": _cooldown_reason_label(reason),
+        "cooldown_until": cooldown_until,
+        "cooldown_remaining_seconds": cooldown_remaining_seconds,
+        "recommended_action": _recommended_action(reason),
+        "counters": _counters_from_state(settings),
+    }
+
+
+def get_safety_event_context(
+    settings: UserSetting,
+    *,
+    now_utc: datetime | None = None,
+) -> dict[str, Any]:
+    payload = get_safety_state_payload(settings, now_utc=now_utc)
+    return {
+        "cooldown_active": bool(payload.get("cooldown_active")),
+        "cooldown_reason": payload.get("cooldown_reason"),
+        "cooldown_until": payload.get("cooldown_until"),
+        "cooldown_remaining_seconds": int(payload.get("cooldown_remaining_seconds") or 0),
+        "safety_counters": payload.get("counters", {}),
+    }

build/lib/app/services/domains/ingestion/scheduler.py

@@ -0,0 +1,622 @@
+from __future__ import annotations
+
+import asyncio
+from dataclasses import dataclass
+from datetime import datetime, timedelta, timezone
+import logging
+
+from sqlalchemy import select
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from app.db.models import (
+    CrawlRun,
+    QueueItemStatus,
+    RunTriggerType,
+    ScholarProfile,
+    User,
+    UserSetting,
+)
+from app.db.session import get_session_factory
+from app.services.domains.ingestion import queue as queue_service
+from app.services.domains.ingestion.application import (
+    RunAlreadyInProgressError,
+    RunBlockedBySafetyPolicyError,
+    ScholarIngestionService,
+)
+from app.services.domains.settings import application as user_settings_service
+from app.services.domains.scholar.source import LiveScholarSource
+from app.settings import settings
+
+logger = logging.getLogger(__name__)
+
+
+def _request_delay_floor_seconds() -> int:
+    return user_settings_service.resolve_request_delay_minimum(
+        settings.ingestion_min_request_delay_seconds
+    )
+
+
+def _effective_request_delay_seconds(value: int | None) -> int:
+    floor = _request_delay_floor_seconds()
+    try:
+        parsed = int(value) if value is not None else floor
+    except (TypeError, ValueError):
+        parsed = floor
+    return max(floor, parsed)
+
+
+@dataclass(frozen=True)
+class _AutoRunCandidate:
+    user_id: int
+    run_interval_minutes: int
+    request_delay_seconds: int
+    cooldown_until: datetime | None
+    cooldown_reason: str | None
+
+
+class SchedulerService:
+    def __init__(
+        self,
+        *,
+        enabled: bool,
+        tick_seconds: int,
+        network_error_retries: int,
+        retry_backoff_seconds: float,
+        max_pages_per_scholar: int,
+        page_size: int,
+        continuation_queue_enabled: bool,
+        continuation_base_delay_seconds: int,
+        continuation_max_delay_seconds: int,
+        continuation_max_attempts: int,
+        queue_batch_size: int,
+    ) -> None:
+        self._enabled = enabled
+        self._tick_seconds = max(5, int(tick_seconds))
+        self._network_error_retries = max(0, int(network_error_retries))
+        self._retry_backoff_seconds = max(0.0, float(retry_backoff_seconds))
+        self._max_pages_per_scholar = max(1, int(max_pages_per_scholar))
+        self._page_size = max(1, int(page_size))
+        self._continuation_queue_enabled = bool(continuation_queue_enabled)
+        self._continuation_base_delay_seconds = max(1, int(continuation_base_delay_seconds))
+        self._continuation_max_delay_seconds = max(
+            self._continuation_base_delay_seconds,
+            int(continuation_max_delay_seconds),
+        )
+        self._continuation_max_attempts = max(1, int(continuation_max_attempts))
+        self._queue_batch_size = max(1, int(queue_batch_size))
+        self._task: asyncio.Task[None] | None = None
+        self._source = LiveScholarSource()
+
+    async def start(self) -> None:
+        if not self._enabled:
+            logger.info(
+                "scheduler.disabled",
+                extra={
+                    "event": "scheduler.disabled",
+                },
+            )
+            return
+        if self._task is not None:
+            return
+        self._task = asyncio.create_task(self._run_loop(), name="scholarr-scheduler")
+        logger.info(
+            "scheduler.started",
+            extra={
+                "event": "scheduler.started",
+                "tick_seconds": self._tick_seconds,
+                "network_error_retries": self._network_error_retries,
+                "retry_backoff_seconds": self._retry_backoff_seconds,
+                "max_pages_per_scholar": self._max_pages_per_scholar,
+                "page_size": self._page_size,
+                "continuation_queue_enabled": self._continuation_queue_enabled,
+                "continuation_base_delay_seconds": self._continuation_base_delay_seconds,
+                "continuation_max_delay_seconds": self._continuation_max_delay_seconds,
+                "continuation_max_attempts": self._continuation_max_attempts,
+                "queue_batch_size": self._queue_batch_size,
+            },
+        )
+
+    async def stop(self) -> None:
+        if self._task is None:
+            return
+        self._task.cancel()
+        try:
+            await self._task
+        except asyncio.CancelledError:
+            pass
+        finally:
+            self._task = None
+        logger.info("scheduler.stopped", extra={"event": "scheduler.stopped"})
+
+    async def _run_loop(self) -> None:
+        while True:
+            try:
+                await self._tick_once()
+            except asyncio.CancelledError:
+                raise
+            except Exception:
+                logger.exception(
+                    "scheduler.tick_failed",
+                    extra={
+                        "event": "scheduler.tick_failed",
+                    },
+                )
+            await asyncio.sleep(float(self._tick_seconds))
+
+    async def _tick_once(self) -> None:
+        if self._continuation_queue_enabled:
+            await self._drain_continuation_queue()
+        
+        await self._drain_pdf_queue()
+
+        candidates = await self._load_candidates()
+        if not candidates:
+            return
+        now = datetime.now(timezone.utc)
+        for candidate in candidates:
+            if not await self._is_due(candidate, now=now):
+                continue
+            await self._run_candidate(candidate)
+
+    async def _load_candidate_rows(self) -> list[tuple]:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            result = await session.execute(
+                select(
+                    UserSetting.user_id,
+                    UserSetting.run_interval_minutes,
+                    UserSetting.request_delay_seconds,
+                    UserSetting.scrape_cooldown_until,
+                    UserSetting.scrape_cooldown_reason,
+                )
+                .join(User, User.id == UserSetting.user_id)
+                .where(User.is_active.is_(True), UserSetting.auto_run_enabled.is_(True))
+                .order_by(UserSetting.user_id.asc())
+            )
+            return result.all()
+
+    @staticmethod
+    def _candidate_from_row(row: tuple, *, now_utc: datetime) -> _AutoRunCandidate | None:
+        user_id, run_interval_minutes, request_delay_seconds, cooldown_until, cooldown_reason = row
+        if cooldown_until is not None and cooldown_until.tzinfo is None:
+            cooldown_until = cooldown_until.replace(tzinfo=timezone.utc)
+        if cooldown_until is not None and cooldown_until > now_utc:
+            logger.info(
+                "scheduler.run_skipped_safety_cooldown_precheck",
+                extra={
+                    "event": "scheduler.run_skipped_safety_cooldown_precheck",
+                    "user_id": int(user_id),
+                    "reason": cooldown_reason,
+                    "cooldown_until": cooldown_until,
+                    "cooldown_remaining_seconds": int((cooldown_until - now_utc).total_seconds()),
+                    "metric_name": "scheduler_run_skipped_safety_cooldown_total",
+                    "metric_value": 1,
+                },
+            )
+            return None
+        return _AutoRunCandidate(
+            user_id=int(user_id),
+            run_interval_minutes=int(run_interval_minutes),
+            request_delay_seconds=_effective_request_delay_seconds(request_delay_seconds),
+            cooldown_until=cooldown_until,
+            cooldown_reason=(str(cooldown_reason).strip() if cooldown_reason else None),
+        )
+
+    async def _load_candidates(self) -> list[_AutoRunCandidate]:
+        if not settings.ingestion_automation_allowed:
+            return []
+        rows = await self._load_candidate_rows()
+        now_utc = datetime.now(timezone.utc)
+        candidates: list[_AutoRunCandidate] = []
+        for row in rows:
+            candidate = self._candidate_from_row(row, now_utc=now_utc)
+            if candidate is not None:
+                candidates.append(candidate)
+        return candidates
+
+    async def _is_due(self, candidate: _AutoRunCandidate, *, now: datetime) -> bool:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            result = await session.execute(
+                select(CrawlRun.start_dt)
+                .where(
+                    CrawlRun.user_id == candidate.user_id,
+                )
+                .order_by(CrawlRun.start_dt.desc(), CrawlRun.id.desc())
+                .limit(1)
+            )
+            last_run = result.scalar_one_or_none()
+
+        if last_run is None:
+            return True
+
+        next_due_dt = last_run + timedelta(
+            minutes=candidate.run_interval_minutes
+        )
+        return now >= next_due_dt
+
+    async def _run_candidate_ingestion(
+        self,
+        *,
+        candidate: _AutoRunCandidate,
+    ):
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            ingestion = ScholarIngestionService(source=self._source)
+            try:
+                return await ingestion.run_for_user(
+                    session,
+                    user_id=candidate.user_id,
+                    trigger_type=RunTriggerType.SCHEDULED,
+                    request_delay_seconds=candidate.request_delay_seconds,
+                    network_error_retries=self._network_error_retries,
+                    retry_backoff_seconds=self._retry_backoff_seconds,
+                    max_pages_per_scholar=self._max_pages_per_scholar,
+                    page_size=self._page_size,
+                    auto_queue_continuations=self._continuation_queue_enabled,
+                    queue_delay_seconds=self._continuation_base_delay_seconds,
+                    alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
+                    alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
+                    alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
+                )
+            except RunAlreadyInProgressError:
+                await session.rollback()
+                logger.info("scheduler.run_skipped_locked", extra={"event": "scheduler.run_skipped_locked", "user_id": candidate.user_id})
+                return None
+            except RunBlockedBySafetyPolicyError as exc:
+                await session.rollback()
+                logger.info(
+                    "scheduler.run_skipped_safety_cooldown",
+                    extra={
+                        "event": "scheduler.run_skipped_safety_cooldown",
+                        "user_id": candidate.user_id,
+                        "reason": exc.safety_state.get("cooldown_reason"),
+                        "cooldown_until": exc.safety_state.get("cooldown_until"),
+                        "cooldown_remaining_seconds": exc.safety_state.get("cooldown_remaining_seconds"),
+                        "metric_name": "scheduler_run_skipped_safety_cooldown_total",
+                        "metric_value": 1,
+                    },
+                )
+                return None
+            except Exception:
+                await session.rollback()
+                logger.exception("scheduler.run_failed", extra={"event": "scheduler.run_failed", "user_id": candidate.user_id})
+                return None
+
+    async def _run_candidate(self, candidate: _AutoRunCandidate) -> None:
+        run_summary = await self._run_candidate_ingestion(candidate=candidate)
+        if run_summary is None:
+            return
+        logger.info(
+            "scheduler.run_completed",
+            extra={
+                "event": "scheduler.run_completed",
+                "user_id": candidate.user_id,
+                "run_id": run_summary.crawl_run_id,
+                "status": run_summary.status.value,
+                "scholar_count": run_summary.scholar_count,
+                "new_publication_count": run_summary.new_publication_count,
+            },
+        )
+
+    async def _drain_continuation_queue(self) -> None:
+        now = datetime.now(timezone.utc)
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            jobs = await queue_service.list_due_jobs(
+                session,
+                now=now,
+                limit=self._queue_batch_size,
+            )
+        for job in jobs:
+            await self._run_queue_job(job)
+
+    async def _drop_queue_job_if_max_attempts(
+        self,
+        job: queue_service.ContinuationQueueJob,
+    ) -> bool:
+        if job.attempt_count < self._continuation_max_attempts:
+            return False
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            dropped = await queue_service.mark_dropped(
+                session,
+                job_id=job.id,
+                reason="max_attempts_reached",
+            )
+            await session.commit()
+        if dropped is not None:
+            logger.warning(
+                "scheduler.queue_item_dropped_max_attempts",
+                extra={
+                    "event": "scheduler.queue_item_dropped_max_attempts",
+                    "queue_item_id": job.id,
+                    "user_id": job.user_id,
+                    "scholar_profile_id": job.scholar_profile_id,
+                    "attempt_count": job.attempt_count,
+                    "max_attempts": self._continuation_max_attempts,
+                },
+            )
+        return True
+
+    async def _mark_queue_job_retrying(
+        self,
+        job: queue_service.ContinuationQueueJob,
+    ) -> bool:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            queue_item = await queue_service.mark_retrying(session, job_id=job.id)
+            await session.commit()
+        if queue_item is None:
+            return False
+        if queue_item.status == QueueItemStatus.DROPPED.value:
+            return False
+        return True
+
+    async def _queue_job_has_available_scholar(
+        self,
+        job: queue_service.ContinuationQueueJob,
+    ) -> bool:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            scholar_result = await session.execute(
+                select(ScholarProfile.id).where(
+                    ScholarProfile.user_id == job.user_id,
+                    ScholarProfile.id == job.scholar_profile_id,
+                    ScholarProfile.is_enabled.is_(True),
+                )
+            )
+            scholar_id = scholar_result.scalar_one_or_none()
+            if scholar_id is not None:
+                return True
+            dropped = await queue_service.mark_dropped(
+                session,
+                job_id=job.id,
+                reason="scholar_unavailable",
+            )
+            await session.commit()
+        if dropped is not None:
+            logger.info(
+                "scheduler.queue_item_dropped_scholar_unavailable",
+                extra={
+                    "event": "scheduler.queue_item_dropped_scholar_unavailable",
+                    "queue_item_id": job.id,
+                    "user_id": job.user_id,
+                    "scholar_profile_id": job.scholar_profile_id,
+                },
+            )
+        return False
+
+    async def _reschedule_queue_job_lock_active(self, job: queue_service.ContinuationQueueJob) -> None:
+        session_factory = get_session_factory()
+        async with session_factory() as recovery_session:
+            await queue_service.reschedule_job(
+                recovery_session,
+                job_id=job.id,
+                delay_seconds=max(self._tick_seconds, 15),
+                reason="user_run_lock_active",
+                error="run_already_in_progress",
+            )
+            await recovery_session.commit()
+        logger.info(
+            "scheduler.queue_item_deferred_lock",
+            extra={"event": "scheduler.queue_item_deferred_lock", "queue_item_id": job.id, "user_id": job.user_id},
+        )
+
+    async def _reschedule_queue_job_safety_cooldown(
+        self,
+        job: queue_service.ContinuationQueueJob,
+        exc: RunBlockedBySafetyPolicyError,
+    ) -> None:
+        cooldown_remaining_seconds = max(
+            self._tick_seconds,
+            int(exc.safety_state.get("cooldown_remaining_seconds") or 0),
+        )
+        session_factory = get_session_factory()
+        async with session_factory() as recovery_session:
+            await queue_service.reschedule_job(
+                recovery_session,
+                job_id=job.id,
+                delay_seconds=max(self._tick_seconds, cooldown_remaining_seconds),
+                reason="scrape_safety_cooldown",
+                error=str(exc.message),
+            )
+            await recovery_session.commit()
+        logger.info(
+            "scheduler.queue_item_deferred_safety_cooldown",
+            extra={
+                "event": "scheduler.queue_item_deferred_safety_cooldown",
+                "queue_item_id": job.id,
+                "user_id": job.user_id,
+                "reason": exc.safety_state.get("cooldown_reason"),
+                "cooldown_remaining_seconds": cooldown_remaining_seconds,
+                "metric_name": "scheduler_queue_item_deferred_safety_cooldown_total",
+                "metric_value": 1,
+            },
+        )
+
+    async def _reschedule_queue_job_after_exception(
+        self,
+        job: queue_service.ContinuationQueueJob,
+        *,
+        exc: Exception,
+    ) -> None:
+        session_factory = get_session_factory()
+        async with session_factory() as recovery_session:
+            queue_item = await queue_service.increment_attempt_count(recovery_session, job_id=job.id)
+            if queue_item is None:
+                await recovery_session.commit()
+                return
+            if int(queue_item.attempt_count) >= self._continuation_max_attempts:
+                await queue_service.mark_dropped(
+                    recovery_session,
+                    job_id=job.id,
+                    reason="scheduler_exception_max_attempts",
+                    error=str(exc),
+                )
+                await recovery_session.commit()
+                logger.warning(
+                    "scheduler.queue_item_dropped_after_exception",
+                    extra={"event": "scheduler.queue_item_dropped_after_exception", "queue_item_id": job.id, "user_id": job.user_id, "attempt_count": queue_item.attempt_count},
+                )
+                return
+            delay_seconds = queue_service.compute_backoff_seconds(
+                base_seconds=self._continuation_base_delay_seconds,
+                attempt_count=int(queue_item.attempt_count),
+                max_seconds=self._continuation_max_delay_seconds,
+            )
+            await queue_service.reschedule_job(
+                recovery_session,
+                job_id=job.id,
+                delay_seconds=delay_seconds,
+                reason="scheduler_exception",
+                error=str(exc),
+            )
+            await recovery_session.commit()
+        logger.exception("scheduler.queue_item_run_failed", extra={"event": "scheduler.queue_item_run_failed", "queue_item_id": job.id, "user_id": job.user_id})
+
+    async def _run_ingestion_for_queue_job(
+        self,
+        job: queue_service.ContinuationQueueJob,
+    ):
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            request_delay_seconds = await self._load_request_delay_for_user(session, user_id=job.user_id)
+            ingestion = ScholarIngestionService(source=self._source)
+            try:
+                return await ingestion.run_for_user(
+                    session,
+                    user_id=job.user_id,
+                    trigger_type=RunTriggerType.SCHEDULED,
+                    request_delay_seconds=request_delay_seconds,
+                    network_error_retries=self._network_error_retries,
+                    retry_backoff_seconds=self._retry_backoff_seconds,
+                    rate_limit_retries=settings.ingestion_rate_limit_retries,
+                    rate_limit_backoff_seconds=settings.ingestion_rate_limit_backoff_seconds,
+                    max_pages_per_scholar=self._max_pages_per_scholar,
+                    page_size=self._page_size,
+                    scholar_profile_ids={job.scholar_profile_id},
+                    start_cstart_by_scholar_id={job.scholar_profile_id: job.resume_cstart},
+                    auto_queue_continuations=self._continuation_queue_enabled,
+                    queue_delay_seconds=self._continuation_base_delay_seconds,
+                    alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
+                    alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
+                    alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
+                )
+            except RunAlreadyInProgressError:
+                await session.rollback()
+                await self._reschedule_queue_job_lock_active(job)
+            except RunBlockedBySafetyPolicyError as exc:
+                await session.rollback()
+                await self._reschedule_queue_job_safety_cooldown(job, exc)
+            except Exception as exc:
+                await session.rollback()
+                await self._reschedule_queue_job_after_exception(job, exc=exc)
+        return None
+
+    @staticmethod
+    def _log_queue_item_resolved(
+        *,
+        event_name: str,
+        job: queue_service.ContinuationQueueJob,
+        run_summary,
+        attempt_count: int | None = None,
+        delay_seconds: int | None = None,
+    ) -> None:
+        payload = {
+            "event": event_name,
+            "queue_item_id": job.id,
+            "user_id": job.user_id,
+            "run_id": run_summary.crawl_run_id,
+            "status": run_summary.status.value,
+        }
+        if attempt_count is not None:
+            payload["attempt_count"] = attempt_count
+        if delay_seconds is not None:
+            payload["delay_seconds"] = delay_seconds
+        logger.info(event_name, extra=payload)
+
+    async def _finalize_queue_job_after_run(self, job: queue_service.ContinuationQueueJob, run_summary) -> None:
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            if int(run_summary.failed_count) <= 0:
+                queue_item = await queue_service.reset_attempt_count(session, job_id=job.id)
+                await session.commit()
+                if queue_item is None:
+                    self._log_queue_item_resolved(event_name="scheduler.queue_item_resolved", job=job, run_summary=run_summary)
+                    return
+                self._log_queue_item_resolved(
+                    event_name="scheduler.queue_item_progressed",
+                    job=job,
+                    run_summary=run_summary,
+                    attempt_count=int(queue_item.attempt_count),
+                )
+                return
+            queue_item = await queue_service.increment_attempt_count(session, job_id=job.id)
+            if queue_item is None:
+                await session.commit()
+                self._log_queue_item_resolved(event_name="scheduler.queue_item_resolved", job=job, run_summary=run_summary)
+                return
+            if int(queue_item.attempt_count) >= self._continuation_max_attempts:
+                await queue_service.mark_dropped(session, job_id=job.id, reason="max_attempts_after_run")
+                await session.commit()
+                logger.warning(
+                    "scheduler.queue_item_dropped_max_attempts_after_run",
+                    extra={"event": "scheduler.queue_item_dropped_max_attempts_after_run", "queue_item_id": job.id, "user_id": job.user_id, "attempt_count": queue_item.attempt_count, "run_id": run_summary.crawl_run_id, "status": run_summary.status.value},
+                )
+                return
+            delay_seconds = queue_service.compute_backoff_seconds(base_seconds=self._continuation_base_delay_seconds, attempt_count=int(queue_item.attempt_count), max_seconds=self._continuation_max_delay_seconds)
+            await queue_service.reschedule_job(session, job_id=job.id, delay_seconds=delay_seconds, reason=queue_item.reason, error=queue_item.last_error)
+            await session.commit()
+        self._log_queue_item_resolved(
+            event_name="scheduler.queue_item_rescheduled_failed",
+            job=job,
+            run_summary=run_summary,
+            attempt_count=int(queue_item.attempt_count),
+            delay_seconds=delay_seconds,
+        )
+
+    async def _run_queue_job(self, job: queue_service.ContinuationQueueJob) -> None:
+        if await self._drop_queue_job_if_max_attempts(job):
+            return
+        if not await self._mark_queue_job_retrying(job):
+            return
+        if not await self._queue_job_has_available_scholar(job):
+            return
+        run_summary = await self._run_ingestion_for_queue_job(job)
+        if run_summary is None:
+            return
+        await self._finalize_queue_job_after_run(job, run_summary)
+
+    async def _drain_pdf_queue(self) -> None:
+        from app.services.domains.publications.pdf_queue import drain_ready_jobs
+        
+        session_factory = get_session_factory()
+        async with session_factory() as session:
+            try:
+                processed = await drain_ready_jobs(
+                    session,
+                    limit=settings.scheduler_pdf_queue_batch_size,
+                    max_attempts=settings.pdf_auto_retry_max_attempts,
+                )
+                if processed > 0:
+                    logger.info("scheduler.pdf_queue_drain_completed", extra={
+                        "event": "scheduler.pdf_queue_drain_completed",
+                        "processed_count": processed,
+                    })
+            except Exception:
+                logger.exception("scheduler.pdf_queue_drain_failed", extra={
+                    "event": "scheduler.pdf_queue_drain_failed",
+                })
+
+    async def _load_request_delay_for_user(
+        self,
+        db_session: AsyncSession,
+        *,
+        user_id: int,
+    ) -> int:
+        result = await db_session.execute(
+            select(UserSetting.request_delay_seconds).where(UserSetting.user_id == user_id)
+        )
+        delay = result.scalar_one_or_none()
+        return _effective_request_delay_seconds(delay)

build/lib/app/services/domains/ingestion/types.py

@@ -0,0 +1,110 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from app.db.models import RunStatus
+from app.services.domains.scholar.parser import ParsedProfilePage, PublicationCandidate
+from app.services.domains.scholar.source import FetchResult
+
+
+@dataclass(frozen=True)
+class RunExecutionSummary:
+    crawl_run_id: int
+    status: RunStatus
+    scholar_count: int
+    succeeded_count: int
+    failed_count: int
+    partial_count: int
+    new_publication_count: int
+
+
+@dataclass(frozen=True)
+class PagedParseResult:
+    fetch_result: FetchResult
+    parsed_page: ParsedProfilePage
+    first_page_fetch_result: FetchResult
+    first_page_parsed_page: ParsedProfilePage
+    first_page_fingerprint_sha256: str | None
+    publications: list[PublicationCandidate]
+    attempt_log: list[dict[str, Any]]
+    page_logs: list[dict[str, Any]]
+    pages_fetched: int
+    pages_attempted: int
+    has_more_remaining: bool
+    pagination_truncated_reason: str | None
+    continuation_cstart: int | None
+    skipped_no_change: bool
+    discovered_publication_count: int
+
+
+@dataclass
+class RunProgress:
+    succeeded_count: int = 0
+    failed_count: int = 0
+    partial_count: int = 0
+    scholar_results: list[dict[str, Any]] = field(default_factory=list)
+
+
+@dataclass(frozen=True)
+class ScholarProcessingOutcome:
+    result_entry: dict[str, Any]
+    succeeded_count_delta: int
+    failed_count_delta: int
+    partial_count_delta: int
+    discovered_publication_count: int
+
+
+@dataclass(frozen=True)
+class RunFailureSummary:
+    failed_state_counts: dict[str, int]
+    failed_reason_counts: dict[str, int]
+    scrape_failure_counts: dict[str, int]
+    retries_scheduled_count: int
+    scholars_with_retries_count: int
+    retry_exhausted_count: int
+
+
+@dataclass(frozen=True)
+class RunAlertSummary:
+    blocked_failure_count: int
+    network_failure_count: int
+    blocked_failure_threshold: int
+    network_failure_threshold: int
+    retry_scheduled_threshold: int
+    alert_flags: dict[str, bool]
+
+
+@dataclass
+class PagedLoopState:
+    fetch_result: FetchResult
+    parsed_page: ParsedProfilePage
+    attempt_log: list[dict[str, Any]]
+    page_logs: list[dict[str, Any]]
+    publications: list[PublicationCandidate]
+    pages_fetched: int
+    pages_attempted: int
+    current_cstart: int
+    next_cstart: int
+    has_more_remaining: bool = False
+    pagination_truncated_reason: str | None = None
+    continuation_cstart: int | None = None
+    discovered_publication_count: int = 0
+
+
+class RunAlreadyInProgressError(RuntimeError):
+    """Raised when a run lock for a user is already held by another process."""
+
+
+class RunBlockedBySafetyPolicyError(RuntimeError):
+    def __init__(
+        self,
+        *,
+        code: str,
+        message: str,
+        safety_state: dict[str, Any],
+    ) -> None:
+        super().__init__(message)
+        self.code = code
+        self.message = message
+        self.safety_state = safety_state

build/lib/app/services/domains/openalex/__init__.py

@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+import logging
+
+logger = logging.getLogger(__name__)

build/lib/app/services/domains/openalex/client.py

@@ -0,0 +1,135 @@
+import asyncio
+import logging
+from typing import Any, Mapping
+
+import httpx
+from tenacity import (
+    retry,
+    retry_if_exception_type,
+    stop_after_attempt,
+    wait_exponential,
+)
+
+from app.services.domains.openalex.types import OpenAlexWork
+
+logger = logging.getLogger(__name__)
+
+OPENALEX_BASE_URL = "https://api.openalex.org"
+
+
+class OpenAlexClientError(Exception):
+    pass
+
+
+class OpenAlexRateLimitError(OpenAlexClientError):
+    pass
+
+
+class OpenAlexClient:
+    def __init__(
+        self,
+        api_key: str | None = None,
+        mailto: str | None = None,
+        timeout: float = 10.0,
+    ) -> None:
+        self.api_key = api_key
+        self.mailto = mailto
+        self.timeout = timeout
+
+    @property
+    def _base_params(self) -> dict[str, str]:
+        params = {}
+        if self.mailto:
+            params["mailto"] = self.mailto
+        if self.api_key:
+            params["api_key"] = self.api_key
+        return params
+
+    @retry(
+        retry=retry_if_exception_type((httpx.NetworkError, httpx.TimeoutException, OpenAlexRateLimitError)),
+        stop=stop_after_attempt(3),
+        wait=wait_exponential(multiplier=1, min=2, max=10),
+        reraise=True,
+    )
+    async def get_work_by_doi(self, doi: str) -> OpenAlexWork | None:
+        """Fetch a single work by DOI directly."""
+        clean_doi = doi.replace("https://doi.org/", "")
+        if not clean_doi:
+            return None
+
+        url = f"{OPENALEX_BASE_URL}/works/{clean_doi}"
+        
+        headers = {}
+        if self.mailto:
+            headers["User-Agent"] = f"scholar-scraper/1.0 (mailto:{self.mailto})"
+        else:
+            headers["User-Agent"] = "scholar-scraper/1.0"
+
+        async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True, headers=headers) as client:
+            response = await client.get(url, params=self._base_params)
+
+        if response.status_code == 404:
+            return None
+        if response.status_code == 429:
+            raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex work by DOI")
+        if response.status_code >= 400:
+            logger.warning("OpenAlex API error: %s %s", response.status_code, response.text)
+            raise OpenAlexClientError(f"API Error {response.status_code}")
+
+        data = response.json()
+        return OpenAlexWork.from_api_dict(data)
+
+    @retry(
+        retry=retry_if_exception_type((httpx.NetworkError, httpx.TimeoutException, OpenAlexRateLimitError)),
+        stop=stop_after_attempt(3),
+        wait=wait_exponential(multiplier=1, min=2, max=10),
+        reraise=True,
+    )
+    async def get_works_by_filter(
+        self,
+        filters: dict[str, str],
+        limit: int = 50,
+    ) -> list[OpenAlexWork]:
+        """
+        Fetch works using the ?filter= query parameter.
+        Supports fetching multiple records by joining filters with | (OR logic).
+        """
+        if not filters:
+            return []
+
+        # Example: {"doi": "10.foo|10.bar", "title.search": "query"}
+        filter_str = ",".join(f"{k}:{v}" for k, v in filters.items())
+        
+        params = self._base_params.copy()
+        params["filter"] = filter_str
+        params["per-page"] = str(limit)
+
+        url = f"{OPENALEX_BASE_URL}/works"
+        
+        headers = {}
+        if self.mailto:
+            headers["User-Agent"] = f"scholar-scraper/1.0 (mailto:{self.mailto})"
+        else:
+            headers["User-Agent"] = "scholar-scraper/1.0"
+
+        async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True, headers=headers) as client:
+            response = await client.get(url, params=params)
+
+        if response.status_code == 429:
+            raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex works list")
+        if response.status_code >= 400:
+            logger.warning("OpenAlex API error (filters=%s): %s %s", filters, response.status_code, response.text)
+            raise OpenAlexClientError(f"API Error {response.status_code}")
+
+        data = response.json()
+        results = data.get("results") or []
+        
+        parsed_works = []
+        for raw_work in results:
+            try:
+                parsed_works.append(OpenAlexWork.from_api_dict(raw_work))
+            except Exception as e:
+                logger.warning("Failed to parse OpenAlex raw dict: %s", e)
+                continue
+
+        return parsed_works

build/lib/app/services/domains/openalex/matching.py

@@ -0,0 +1,108 @@
+import logging
+import re
+
+from rapidfuzz import fuzz
+
+from app.services.domains.openalex.types import OpenAlexWork
+
+logger = logging.getLogger(__name__)
+
+# A minimum similarity score out of 100 for a title to be considered a match candidate.
+TITLE_MATCH_THRESHOLD = 90.0
+# The margin within the top score where a secondary tiebreaker (author/year) is necessary.
+TIEBREAKER_MARGIN = 5.0
+
+
+def _clean_string(s: str | None) -> str:
+    if not s:
+        return ""
+    # Strip non-alphanumeric (keep spaces), lowercase, and collapse whitespace
+    cleaned = re.sub(r"[^a-z0-9\s]", " ", s.lower())
+    return " ".join(cleaned.split())
+
+
+def _author_overlap_score(target_authors: str | None, candidate_authors: list[str]) -> bool:
+    if not target_authors or not candidate_authors:
+        return False
+    
+    target_clean = _clean_string(target_authors)
+    if not target_clean:
+        return False
+        
+    for candidate in candidate_authors:
+        cand_clean = _clean_string(candidate)
+        if cand_clean and (cand_clean in target_clean or target_clean in cand_clean):
+            return True
+        # Alternatively check rapidfuzz token_set_ratio
+        if cand_clean and fuzz.token_set_ratio(target_clean, cand_clean) > 80:
+            return True
+            
+    return False
+
+
+def find_best_match(
+    target_title: str,
+    target_year: int | None,
+    target_authors: str | None,
+    candidates: list[OpenAlexWork],
+) -> OpenAlexWork | None:
+    """
+    Finds the best matching OpenAlexWork from a list of candidates, prioritizing title similarity (>90%)
+    with year and author overlap as tiebreakers for close candidates.
+    """
+    if not target_title or not candidates:
+        return None
+
+    clean_target = _clean_string(target_title)
+    if not clean_target:
+        return None
+
+    scored_candidates: list[tuple[float, OpenAlexWork]] = []
+
+    for cand in candidates:
+        if not cand.title:
+            continue
+            
+        clean_cand = _clean_string(cand.title)
+        
+        # Primary sort: string similarity ratio
+        score = fuzz.ratio(clean_target, clean_cand)
+        
+        if score >= TITLE_MATCH_THRESHOLD:
+            scored_candidates.append((score, cand))
+
+    if not scored_candidates:
+        return None
+
+    # Sort descending by score
+    scored_candidates.sort(key=lambda x: x[0], reverse=True)
+    
+    best_score = scored_candidates[0][0]
+    
+    # Extract all candidates within the tiebreaker margin
+    top_scored_candidates = [
+        (score, cand) for score, cand in scored_candidates 
+        if best_score - score <= TIEBREAKER_MARGIN
+    ]
+
+    if len(top_scored_candidates) == 1:
+        return top_scored_candidates[0][1]
+
+    # We have a tie or near-tie. Use year and author overlap to break the tie.
+    # Score candidates: +1 for year match (within 1 year), +1 for author overlap
+    tiebreaker_scores: list[tuple[int, float, OpenAlexWork]] = []
+    
+    for original_score, cand in top_scored_candidates:
+        tb_score = 0
+        if target_year is not None and cand.publication_year is not None:
+            if abs(target_year - cand.publication_year) <= 1:
+                tb_score += 1
+                
+        candidate_author_names = [a.display_name for a in cand.authors if a.display_name]
+        if _author_overlap_score(target_authors, candidate_author_names):
+            tb_score += 1
+            
+        tiebreaker_scores.append((tb_score, original_score, cand))
+        
+    tiebreaker_scores.sort(key=lambda x: (x[0], x[1]), reverse=True)
+    return tiebreaker_scores[0][2]

build/lib/app/services/domains/openalex/types.py

@@ -0,0 +1,71 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Mapping
+
+
+@dataclass(frozen=True)
+class OpenAlexAuthor:
+    openalex_id: str | None
+    display_name: str | None
+
+
+@dataclass(frozen=True)
+class OpenAlexWork:
+    openalex_id: str
+    doi: str | None
+    pmid: str | None
+    pmcid: str | None
+    title: str | None
+    publication_year: int | None
+    cited_by_count: int
+    is_oa: bool
+    oa_url: str | None
+    authors: list[OpenAlexAuthor] = field(default_factory=list)
+    raw_data: Mapping[str, Any] = field(default_factory=dict, repr=False)
+
+    @classmethod
+    def from_api_dict(cls, data: Mapping[str, Any]) -> OpenAlexWork:
+        ids = data.get("ids") or {}
+        
+        # Extract DOI without the https://doi.org/ prefix
+        doi = ids.get("doi")
+        if doi and doi.startswith("https://doi.org/"):
+            doi = doi[16:]
+            
+        # Extract PMID without the url prefix
+        pmid = ids.get("pmid")
+        if pmid and pmid.startswith("https://pubmed.ncbi.nlm.nih.gov/"):
+            pmid = pmid[32:]
+            
+        # Extract PMCID without the url prefix
+        pmcid = ids.get("pmcid")
+        if pmcid and pmcid.startswith("https://www.ncbi.nlm.nih.gov/pmc/articles/"):
+            pmcid = pmcid[42:]
+
+        open_access = data.get("open_access") or {}
+        
+        authors = []
+        for authorship in data.get("authorships") or []:
+            author_data = authorship.get("author") or {}
+            authors.append(
+                OpenAlexAuthor(
+                    openalex_id=author_data.get("id"),
+                    display_name=author_data.get("display_name"),
+                )
+            )
+
+        return cls(
+            openalex_id=data.get("id", ""),
+            doi=doi,
+            pmid=pmid,
+            pmcid=pmcid,
+            title=data.get("title"),
+            publication_year=data.get("publication_year"),
+            cited_by_count=data.get("cited_by_count", 0),
+            is_oa=bool(open_access.get("is_oa")),
+            oa_url=open_access.get("oa_url"),
+            authors=authors,
+            raw_data=dict(data),
+        )

build/lib/app/services/domains/portability/__init__.py

@@ -0,0 +1 @@
+from app.services.domains.portability.application import *