justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

harden domain architecture and add lazy Crossref+Unpaywall PDF enrich… #14

Merged
JustinZeus merged 1 commit from develop into main 2026-02-20 22:57:12 +01:00
Conversation 0 Commits 1 Files changed 62 +2562 -688
62 changed files with 2562 additions and 688 deletions
Download patch file Download diff file Expand all files Collapse all files

10
.env.example
View file

@ -76,6 +76,16 @@ SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD=1
SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS=1800
SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD=2
SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD=3
UNPAYWALL_ENABLED=1
UNPAYWALL_EMAIL=
UNPAYWALL_TIMEOUT_SECONDS=4.0
UNPAYWALL_MAX_ITEMS_PER_REQUEST=20
UNPAYWALL_RETRY_COOLDOWN_SECONDS=1800
CROSSREF_ENABLED=1
CROSSREF_MAX_ROWS=10
CROSSREF_TIMEOUT_SECONDS=8.0
CROSSREF_MIN_INTERVAL_SECONDS=0.6
CROSSREF_MAX_LOOKUPS_PER_REQUEST=8
BOOTSTRAP_ADMIN_ON_START=0
BOOTSTRAP_ADMIN_EMAIL=

31
README.md
View file

@ -78,15 +78,18 @@ docker compose -f docker-compose.yml -f docker-compose.dev.yml down
- Scholar tracking is user-scoped: each account can track the same Scholar ID independently.
- Publications are shared/global records deduplicated by Scholar cluster ID and normalized fingerprint.
- Per-account visibility and read state is stored on scholar-publication links, not on the global publication row.
- Publication records include both canonical Scholar detail URLs (`pub_url`) and direct download targets (`pdf_url`) when available.
- Publication records include canonical Scholar detail URLs (`pub_url`), normalized DOI (`doi`), and resolved OA PDF links (`pdf_url`) when available.
## Backend Architecture (Refactored)
- Compatibility facades remain in `app/services/*.py`, but canonical service logic now lives in `app/services/domains/*`.
- Canonical service logic lives in `app/services/domains/*`; root-level `app/services/*.py` business-logic modules are removed.
- `app/services/domains/ingestion/*`: run orchestration (`application.py`), continuation queue (`queue.py`), scrape safety cooldown policy (`safety.py`), scheduler/queue draining (`scheduler.py`), plus shared constants/types/fingerprint helpers.
- `app/services/domains/scholar/*`: fail-fast Google Scholar parsing and source access. Parser flow is modularized across row extractors, state detection, constants, and parser types.
- `app/services/domains/scholars/*`: scholar CRUD, name-search safety/caching, profile-image validation/upload handling, and shared scholar validation helpers.
- `app/services/domains/publications/*`: publication listing/read-state query modules for dashboard/publication UIs (`pub_url` + `pdf_url`).
- `app/services/domains/publications/*`: publication listing/read-state query modules, lazy/background OA enrichment scheduling (non-blocking for list responses), and per-publication PDF retry support.
- `app/services/domains/doi/*`: DOI normalization helpers used by ingestion and enrichment.
- `app/services/domains/crossref/*`: DOI discovery fallback using Crossref metadata queries with bounded/paced request behavior.
- `app/services/domains/unpaywall/*`: OA resolution by DOI (best OA location + PDF URL extraction), using per-user email for API calls.
- `app/services/domains/runs/*`: run history summaries plus continuation queue status/retry/drop/clear operations.
- `app/services/domains/portability/*`: import/export of tracked scholars and publication-link state while preserving global publication deduplication.
- `app/services/domains/settings/*` and `app/services/domains/users/*`: user settings and user management services.
@ -225,6 +228,21 @@ Notes:
| `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD` | `2` | integer >= 1 | deploy, dev | Emit retry-threshold observability warning when name-search retry count reaches this value. |
| `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD` | `3` | integer >= 1 | deploy, dev | Emit cooldown-threshold observability alert after this many requests are rejected during active cooldown. |
### Open-Access Enrichment (Crossref + Unpaywall)
| Variable | Default | Options | Scope | Description |
| --- | --- | --- | --- | --- |
| `UNPAYWALL_ENABLED` | `1` | boolean | deploy, dev | Enable Unpaywall OA resolution. |
| `UNPAYWALL_EMAIL` | empty | valid email | deploy, dev | Fallback email if per-user email is unavailable. |
| `UNPAYWALL_TIMEOUT_SECONDS` | `4.0` | float >= 0.5 | deploy, dev | Timeout for Unpaywall requests. |
| `UNPAYWALL_MAX_ITEMS_PER_REQUEST` | `20` | integer >= 0 | deploy, dev | Max publications queued per lazy enrichment pass. |
| `UNPAYWALL_RETRY_COOLDOWN_SECONDS` | `1800` | integer >= 1 | deploy, dev | Per-user/per-publication cooldown before re-scheduling unresolved OA lookups. |
| `CROSSREF_ENABLED` | `1` | boolean | deploy, dev | Enable Crossref DOI discovery fallback when DOI is missing/insufficient. |
| `CROSSREF_MAX_ROWS` | `10` | integer >= 1 | deploy, dev | Max Crossref rows evaluated per lookup. |
| `CROSSREF_TIMEOUT_SECONDS` | `8.0` | float >= 0.5 | deploy, dev | Timeout budget per Crossref lookup task. |
| `CROSSREF_MIN_INTERVAL_SECONDS` | `0.6` | float >= 0 | deploy, dev | Minimum pacing interval between Crossref requests. |
| `CROSSREF_MAX_LOOKUPS_PER_REQUEST` | `8` | integer >= 0 | deploy, dev | Max Crossref DOI lookups performed during one lazy pass or manual retry. |
### Scrape Safety Operations
- Structured safety events are emitted for:
@ -290,7 +308,12 @@ Scheduled fixture probes run in GitHub Actions via `.github/workflows/scheduled-
- `GET /api/v1/publications` supports `mode=all|unread|latest` (plus temporary alias `mode=new`).
- `unread` = actionable read-state (`is_read=false`).
- `latest` = discovery-state (`first seen in the latest completed check`).
- Publications include `pub_url` and `pdf_url` in API responses. `pdf_url` is a direct-download target extracted from Scholar result-side links when present.
- Publications include `pub_url`, `doi`, and `pdf_url` in API responses.
- OA enrichment flow is DOI-first:
- existing DOI (or explicit DOI in source metadata) -> Unpaywall lookup,
- Crossref DOI discovery fallback (paced + bounded),
- Unpaywall DOI lookup for OA status and `pdf_url`.
- `POST /api/v1/publications/{publication_id}/retry-pdf` retries enrichment for a single publication in current user scope.
- Response counters:
- `unread_count`: unread publications in current scope.
- `latest_count`: publications discovered in latest completed check.

10
agents.md
View file

@ -27,10 +27,12 @@ These limits prevent IP bans and are not to be optimized away.
* **Frontend:** TypeScript, Vue 3, Vite.
* **Infrastructure:** Multi-stage Docker.
## 5. Refactored Service Boundaries (Current)
* **Compatibility rule:** `app/services/*.py` files are import façades only. Keep them thin and route all real logic to domain modules.
* **`app/services/domains/scholar/*`:** Parser contract is fail-fast. Layout drift must emit explicit `layout_*` reasons/warnings, never silent partial success.
## 5. Domain Service Boundaries
* **Strict Modularity:** Flat files in the `app/services/` root are strictly prohibited. All business logic and routing must reside exclusively within `app/services/domains/`.
* **`app/services/domains/scholar/*`:** Parser contract is fail-fast. Layout drift must emit explicit exceptions and `layout_*` reasons/warnings. Never allow silent partial success.
* **`app/services/domains/ingestion/application.py`:** Orchestrates ingestion runs; validate parser outputs before persistence; enforce publication candidate constraints before upsert.
* **`app/services/domains/publications/*`:** Publication list/read-state query layer; include both `pub_url` and `pdf_url` for UI consumption.
* **`app/services/domains/publications/*`:** Publication list/read-state query layer. Includes `doi` + `pdf_url` fields for UI consumption, enforces non-blocking lazy OA enrichment scheduling on list reads, and exposes per-publication PDF retry behavior.
* **`app/services/domains/crossref/*`:** DOI discovery fallback module. Must use bounded/paced lookups to avoid burst traffic and 429 responses.
* **`app/services/domains/unpaywall/*`:** OA resolver by DOI only (best OA location + PDF URL extraction). Do not use Google Scholar for PDF resolution to avoid N+1 scrape amplification.
* **`app/services/domains/portability/*`:** Handles JSON import/export for user-scoped scholars and scholar-publication link state while preserving global publication dedup rules.
* **`app/services/domains/ingestion/scheduler.py`:** Owns automatic runs and continuation queue retries/drops; do not bypass safety gate or cooldown logic.

50
alembic/versions/20260220_0011_add_publication_doi.py Normal file
View file

@ -0,0 +1,50 @@
"""Add DOI column to publications.
Revision ID: 20260220_0011
Revises: 20260219_0010
Create Date: 2026-02-20 13:35:00.000000
"""
from collections.abc import Sequence
from alembic import op
import sqlalchemy as sa
revision: str = "20260220_0011"
down_revision: str | Sequence[str] | None = "20260219_0010"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def _has_column(inspector: sa.Inspector, table_name: str, column_name: str) -> bool:
for column in inspector.get_columns(table_name):
if str(column.get("name")) == column_name:
return True
return False
def upgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
table_names = set(inspector.get_table_names())
if "publications" not in table_names:
return
if not _has_column(inspector, "publications", "doi"):
op.add_column("publications", sa.Column("doi", sa.String(length=255), nullable=True))
index_names = {index["name"] for index in inspector.get_indexes("publications")}
if "ix_publications_doi" not in index_names:
op.create_index("ix_publications_doi", "publications", ["doi"], unique=False)
def downgrade() -> None:
bind = op.get_bind()
inspector = sa.inspect(bind)
table_names = set(inspector.get_table_names())
if "publications" not in table_names:
return
index_names = {index["name"] for index in inspector.get_indexes("publications")}
if "ix_publications_doi" in index_names:
op.drop_index("ix_publications_doi", table_name="publications")
if _has_column(inspector, "publications", "doi"):
op.drop_column("publications", "doi")

2
app/api/routers/admin.py
View file

@ -20,7 +20,7 @@ from app.auth.deps import get_auth_service
from app.auth.service import AuthService
from app.db.models import User
from app.db.session import get_db_session
from app.services import users as user_service
from app.services.domains.users import application as user_service
logger = logging.getLogger(__name__)

89
app/api/routers/auth.py
View file

@ -24,13 +24,55 @@ from app.auth.session import set_session_user
from app.db.models import User
from app.db.session import get_db_session
from app.security.csrf import ensure_csrf_token
from app.services import users as user_service
from app.services.domains.users import application as user_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["api-auth"])
def _login_limiter_key_and_email(request: Request, payload: LoginRequest) -> tuple[str, str]:
return auth_runtime.login_rate_limit_key(request, payload.email), payload.email.strip().lower()
def _raise_rate_limited(normalized_email: str, retry_after_seconds: int) -> None:
logger.warning(
"api.auth.login_rate_limited",
extra={
"event": "api.auth.login_rate_limited",
"email": normalized_email,
"retry_after_seconds": retry_after_seconds,
},
)
raise ApiException(
status_code=429,
code="rate_limited",
message="Too many login attempts. Please try again later.",
details={"retry_after_seconds": retry_after_seconds},
)
def _serialize_user_payload(user: User) -> dict[str, object]:
return {
"id": int(user.id),
"email": user.email,
"is_admin": bool(user.is_admin),
"is_active": bool(user.is_active),
}
def _raise_invalid_credentials(*, normalized_email: str) -> None:
logger.info(
"api.auth.login_failed",
extra={"event": "api.auth.login_failed", "email": normalized_email},
)
raise ApiException(
status_code=401,
code="invalid_credentials",
message="Invalid email or password.",
)
@router.post(
"/login",
response_model=LoginEnvelope,
@ -42,24 +84,10 @@ async def login(
auth_service: AuthService = Depends(get_auth_service),
rate_limiter: SlidingWindowRateLimiter = Depends(get_login_rate_limiter),
):
limiter_key = auth_runtime.login_rate_limit_key(request, payload.email)
limiter_key, normalized_email = _login_limiter_key_and_email(request, payload)
decision = rate_limiter.check(limiter_key)
normalized_email = payload.email.strip().lower()
if not decision.allowed:
logger.warning(
"api.auth.login_rate_limited",
extra={
"event": "api.auth.login_rate_limited",
"email": normalized_email,
"retry_after_seconds": decision.retry_after_seconds,
},
)
raise ApiException(
status_code=429,
code="rate_limited",
message="Too many login attempts. Please try again later.",
details={"retry_after_seconds": decision.retry_after_seconds},
)
_raise_rate_limited(normalized_email, int(decision.retry_after_seconds))
user = await auth_service.authenticate_user(
db_session,
@ -68,18 +96,7 @@ async def login(
)
if user is None:
rate_limiter.record_failure(limiter_key)
logger.info(
"api.auth.login_failed",
extra={
"event": "api.auth.login_failed",
"email": normalized_email,
},
)
raise ApiException(
status_code=401,
code="invalid_credentials",
message="Invalid email or password.",
)
_raise_invalid_credentials(normalized_email=normalized_email)
rate_limiter.reset(limiter_key)
set_session_user(
@ -101,12 +118,7 @@ async def login(
data={
"authenticated": True,
"csrf_token": ensure_csrf_token(request),
"user": {
"id": int(user.id),
"email": user.email,
"is_admin": bool(user.is_admin),
"is_active": bool(user.is_active),
},
"user": _serialize_user_payload(user),
},
)
@ -124,12 +136,7 @@ async def get_current_session(
data={
"authenticated": True,
"csrf_token": ensure_csrf_token(request),
"user": {
"id": int(current_user.id),
"email": current_user.email,
"is_admin": bool(current_user.is_admin),
"is_active": bool(current_user.is_active),
},
"user": _serialize_user_payload(current_user),
},
)

207
app/api/routers/publications.py
View file

@ -3,7 +3,7 @@ from __future__ import annotations
import logging
from typing import Literal
from fastapi import APIRouter, Depends, Query, Request
from fastapi import APIRouter, Depends, Path, Query, Request
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.deps import get_api_current_user
@ -14,17 +14,104 @@ from app.api.schemas import (
MarkSelectedReadEnvelope,
MarkSelectedReadRequest,
PublicationsListEnvelope,
RetryPublicationPdfEnvelope,
RetryPublicationPdfRequest,
)
from app.db.models import User
from app.db.session import get_db_session
from app.services import publications as publication_service
from app.services import scholars as scholar_service
from app.services.domains.publications import application as publication_service
from app.services.domains.scholars import application as scholar_service
from app.settings import settings
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/publications", tags=["api-publications"])
async def _require_selected_profile(
db_session: AsyncSession,
*,
user_id: int,
selected_scholar_id: int | None,
) -> None:
if selected_scholar_id is None:
return
selected_profile = await scholar_service.get_user_scholar_by_id(
db_session,
user_id=user_id,
scholar_profile_id=selected_scholar_id,
)
if selected_profile is None:
raise ApiException(
status_code=404,
code="scholar_not_found",
message="Scholar filter not found.",
)
def _serialize_publication_item(item) -> dict[str, object]:
return {
"publication_id": item.publication_id,
"scholar_profile_id": item.scholar_profile_id,
"scholar_label": item.scholar_label,
"title": item.title,
"year": item.year,
"citation_count": item.citation_count,
"venue_text": item.venue_text,
"pub_url": item.pub_url,
"doi": item.doi,
"pdf_url": item.pdf_url,
"is_read": item.is_read,
"first_seen_at": item.first_seen_at,
"is_new_in_latest_run": item.is_new_in_latest_run,
}
async def _publication_counts(
db_session: AsyncSession,
*,
user_id: int,
selected_scholar_id: int | None,
) -> tuple[int, int, int]:
unread_count = await publication_service.count_unread_for_user(
db_session,
user_id=user_id,
scholar_profile_id=selected_scholar_id,
)
latest_count = await publication_service.count_latest_for_user(
db_session,
user_id=user_id,
scholar_profile_id=selected_scholar_id,
)
total_count = await publication_service.count_for_user(
db_session,
user_id=user_id,
mode=publication_service.MODE_ALL,
scholar_profile_id=selected_scholar_id,
)
return unread_count, latest_count, total_count
def _publications_list_data(
*,
mode: str,
selected_scholar_id: int | None,
unread_count: int,
latest_count: int,
total_count: int,
publications: list,
) -> dict[str, object]:
return {
"mode": mode,
"selected_scholar_profile_id": selected_scholar_id,
"unread_count": unread_count,
"latest_count": latest_count,
"new_count": latest_count,
"total_count": total_count,
"publications": [_serialize_publication_item(item) for item in publications],
}
@router.get(
"",
response_model=PublicationsListEnvelope,
@ -39,17 +126,10 @@ async def list_publications(
):
resolved_mode = publication_service.resolve_publication_view_mode(mode)
selected_scholar_id = scholar_profile_id
if selected_scholar_id is not None:
selected_profile = await scholar_service.get_user_scholar_by_id(
await _require_selected_profile(
db_session,
user_id=current_user.id,
scholar_profile_id=selected_scholar_id,
)
if selected_profile is None:
raise ApiException(
status_code=404,
code="scholar_not_found",
message="Scholar filter not found.",
selected_scholar_id=selected_scholar_id,
)
publications = await publication_service.list_for_user(
@ -59,50 +139,26 @@ async def list_publications(
scholar_profile_id=selected_scholar_id,
limit=limit,
)
unread_count = await publication_service.count_unread_for_user(
await publication_service.schedule_missing_pdf_enrichment_for_user(
user_id=current_user.id,
request_email=current_user.email,
items=publications,
max_items=settings.unpaywall_max_items_per_request,
)
unread_count, latest_count, total_count = await _publication_counts(
db_session,
user_id=current_user.id,
scholar_profile_id=selected_scholar_id,
selected_scholar_id=selected_scholar_id,
)
latest_count = await publication_service.count_latest_for_user(
db_session,
user_id=current_user.id,
scholar_profile_id=selected_scholar_id,
)
total_count = await publication_service.count_for_user(
db_session,
user_id=current_user.id,
mode=publication_service.MODE_ALL,
scholar_profile_id=selected_scholar_id,
)
return success_payload(
request,
data={
"mode": resolved_mode,
"selected_scholar_profile_id": selected_scholar_id,
"unread_count": unread_count,
"latest_count": latest_count,
"new_count": latest_count,
"total_count": total_count,
"publications": [
{
"publication_id": item.publication_id,
"scholar_profile_id": item.scholar_profile_id,
"scholar_label": item.scholar_label,
"title": item.title,
"year": item.year,
"citation_count": item.citation_count,
"venue_text": item.venue_text,
"pub_url": item.pub_url,
"pdf_url": item.pdf_url,
"is_read": item.is_read,
"first_seen_at": item.first_seen_at,
"is_new_in_latest_run": item.is_new_in_latest_run,
}
for item in publications
],
},
data = _publications_list_data(
mode=resolved_mode,
selected_scholar_id=selected_scholar_id,
unread_count=unread_count,
latest_count=latest_count,
total_count=total_count,
publications=publications,
)
return success_payload(request, data=data)
@router.post(
@ -173,3 +229,50 @@ async def mark_selected_publications_read(
"updated_count": updated_count,
},
)
@router.post(
"/{publication_id}/retry-pdf",
response_model=RetryPublicationPdfEnvelope,
)
async def retry_publication_pdf(
payload: RetryPublicationPdfRequest,
request: Request,
publication_id: int = Path(ge=1),
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
publication = await publication_service.retry_pdf_for_user(
db_session,
user_id=current_user.id,
scholar_profile_id=payload.scholar_profile_id,
publication_id=publication_id,
unpaywall_email=current_user.email,
)
if publication is None:
raise ApiException(
status_code=404,
code="publication_not_found",
message="Publication not found.",
)
resolved_pdf = bool(publication.pdf_url)
logger.info(
"api.publications.retry_pdf",
extra={
"event": "api.publications.retry_pdf",
"user_id": current_user.id,
"scholar_profile_id": payload.scholar_profile_id,
"publication_id": publication_id,
"resolved_pdf": resolved_pdf,
"has_doi": bool(publication.doi),
},
)
message = "Open-access PDF link resolved." if resolved_pdf else "No open-access PDF link found."
return success_payload(
request,
data={
"message": message,
"resolved_pdf": resolved_pdf,
"publication": _serialize_publication_item(publication),
},
)

399
app/api/routers/runs.py
View file

@ -21,10 +21,10 @@ from app.api.schemas import (
)
from app.db.models import RunStatus, RunTriggerType, User
from app.db.session import get_db_session
from app.services import ingestion as ingestion_service
from app.services import run_safety as run_safety_service
from app.services import runs as run_service
from app.services import user_settings as user_settings_service
from app.services.domains.ingestion import application as ingestion_service
from app.services.domains.ingestion import safety as run_safety_service
from app.services.domains.runs import application as run_service
from app.services.domains.settings import application as user_settings_service
from app.settings import settings
from app.api.runtime_deps import get_ingestion_service
@ -298,6 +298,224 @@ async def _load_safety_state(
return run_safety_service.get_safety_state_payload(user_settings)
def _raise_manual_runs_disabled(*, user_id: int, safety_state: dict[str, Any]) -> None:
logger.warning(
"api.runs.manual_blocked_policy",
extra={
"event": "api.runs.manual_blocked_policy",
"user_id": user_id,
"policy": {"manual_run_allowed": False},
"safety_state": safety_state,
"metric_name": "api_runs_manual_blocked_policy_total",
"metric_value": 1,
},
)
raise ApiException(
status_code=403,
code="manual_runs_disabled",
message="Manual checks are disabled by server policy.",
details={
"policy": {"manual_run_allowed": False},
"safety_state": safety_state,
},
)
async def _reused_manual_run_payload(
db_session: AsyncSession,
*,
request: Request,
user_id: int,
idempotency_key: str | None,
safety_state: dict[str, Any],
) -> dict[str, Any] | None:
if idempotency_key is None:
return None
previous_run = await run_service.get_manual_run_by_idempotency_key(
db_session,
user_id=user_id,
idempotency_key=idempotency_key,
)
if previous_run is None:
return None
if previous_run.status == RunStatus.RUNNING:
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run with this idempotency key is still in progress.",
details={"run_id": int(previous_run.id), "idempotency_key": idempotency_key},
)
return success_payload(
request,
data=_manual_run_payload_from_run(
previous_run,
idempotency_key=idempotency_key,
reused_existing_run=True,
safety_state=safety_state,
),
)
async def _run_ingestion_for_manual(
db_session: AsyncSession,
*,
ingest_service: ingestion_service.ScholarIngestionService,
user_id: int,
idempotency_key: str | None,
):
user_settings = await user_settings_service.get_or_create_settings(
db_session,
user_id=user_id,
)
return await ingest_service.run_for_user(
db_session,
user_id=user_id,
trigger_type=RunTriggerType.MANUAL,
request_delay_seconds=user_settings.request_delay_seconds,
network_error_retries=settings.ingestion_network_error_retries,
retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
page_size=settings.ingestion_page_size,
auto_queue_continuations=settings.ingestion_continuation_queue_enabled,
queue_delay_seconds=settings.ingestion_continuation_base_delay_seconds,
idempotency_key=idempotency_key,
alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
)
async def _recover_integrity_error(
db_session: AsyncSession,
*,
request: Request,
user_id: int,
idempotency_key: str | None,
original_exc: IntegrityError,
) -> dict[str, Any]:
if idempotency_key is None:
logger.exception(
"api.runs.manual_integrity_error",
extra={"event": "api.runs.manual_integrity_error", "user_id": user_id},
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
existing_run = await run_service.get_manual_run_by_idempotency_key(
db_session,
user_id=user_id,
idempotency_key=idempotency_key,
)
if existing_run is None:
logger.exception(
"api.runs.manual_integrity_error",
extra={"event": "api.runs.manual_integrity_error", "user_id": user_id},
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
if existing_run.status == RunStatus.RUNNING:
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run with this idempotency key is still in progress.",
details={"run_id": int(existing_run.id), "idempotency_key": idempotency_key},
) from original_exc
return success_payload(
request,
data=_manual_run_payload_from_run(
existing_run,
idempotency_key=idempotency_key,
reused_existing_run=True,
safety_state=await _load_safety_state(db_session, user_id=user_id),
),
)
async def _execute_manual_run(
db_session: AsyncSession,
*,
request: Request,
ingest_service: ingestion_service.ScholarIngestionService,
user_id: int,
idempotency_key: str | None,
):
try:
return await _run_ingestion_for_manual(
db_session,
ingest_service=ingest_service,
user_id=user_id,
idempotency_key=idempotency_key,
)
except ingestion_service.RunAlreadyInProgressError as exc:
await db_session.rollback()
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run is already in progress for this account.",
) from exc
except ingestion_service.RunBlockedBySafetyPolicyError as exc:
await db_session.rollback()
_raise_manual_blocked_safety(exc=exc, user_id=user_id)
except IntegrityError as exc:
await db_session.rollback()
return await _recover_integrity_error(
db_session,
request=request,
user_id=user_id,
idempotency_key=idempotency_key,
original_exc=exc,
)
except Exception as exc:
await db_session.rollback()
_raise_manual_failed(exc=exc, user_id=user_id)
def _raise_manual_blocked_safety(*, exc, user_id: int) -> None:
logger.info(
"api.runs.manual_blocked_safety",
extra={
"event": "api.runs.manual_blocked_safety",
"user_id": user_id,
"reason": exc.safety_state.get("cooldown_reason"),
"cooldown_until": exc.safety_state.get("cooldown_until"),
"cooldown_remaining_seconds": exc.safety_state.get("cooldown_remaining_seconds"),
"metric_name": "api_runs_manual_blocked_safety_total",
"metric_value": 1,
},
)
raise ApiException(
status_code=429,
code=exc.code,
message=exc.message,
details={"safety_state": exc.safety_state},
) from exc
def _raise_manual_failed(*, exc: Exception, user_id: int) -> None:
logger.exception(
"api.runs.manual_failed",
extra={"event": "api.runs.manual_failed", "user_id": user_id},
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from exc
def _manual_run_success_payload(
*,
run_summary,
idempotency_key: str | None,
safety_state: dict[str, Any],
) -> dict[str, Any]:
return {
"run_id": run_summary.crawl_run_id,
"status": run_summary.status.value,
"scholar_count": run_summary.scholar_count,
"succeeded_count": run_summary.succeeded_count,
"failed_count": run_summary.failed_count,
"partial_count": run_summary.partial_count,
"new_publication_count": run_summary.new_publication_count,
"reused_existing_run": False,
"idempotency_key": idempotency_key,
"safety_state": safety_state,
}
@router.get(
"",
response_model=RunsListEnvelope,
@ -383,164 +601,28 @@ async def run_manual(
user_id=current_user.id,
)
if not settings.ingestion_manual_run_allowed:
logger.warning(
"api.runs.manual_blocked_policy",
extra={
"event": "api.runs.manual_blocked_policy",
"user_id": current_user.id,
"policy": {"manual_run_allowed": False},
"safety_state": safety_state,
"metric_name": "api_runs_manual_blocked_policy_total",
"metric_value": 1,
},
)
raise ApiException(
status_code=403,
code="manual_runs_disabled",
message="Manual checks are disabled by server policy.",
details={
"policy": {
"manual_run_allowed": False,
},
"safety_state": safety_state,
},
)
_raise_manual_runs_disabled(user_id=current_user.id, safety_state=safety_state)
idempotency_key = _normalize_idempotency_key(request.headers.get(IDEMPOTENCY_HEADER))
if idempotency_key is not None:
previous_run = await run_service.get_manual_run_by_idempotency_key(
reused_payload = await _reused_manual_run_payload(
db_session,
request=request,
user_id=current_user.id,
idempotency_key=idempotency_key,
)
if previous_run is not None:
if previous_run.status == RunStatus.RUNNING:
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run with this idempotency key is still in progress.",
details={
"run_id": int(previous_run.id),
"idempotency_key": idempotency_key,
},
)
return success_payload(
request,
data=_manual_run_payload_from_run(
previous_run,
idempotency_key=idempotency_key,
reused_existing_run=True,
safety_state=safety_state,
),
)
if reused_payload is not None:
return reused_payload
user_settings = await user_settings_service.get_or_create_settings(
db_session,
user_id=current_user.id,
)
try:
run_summary = await ingest_service.run_for_user(
db_session,
user_id=current_user.id,
trigger_type=RunTriggerType.MANUAL,
request_delay_seconds=user_settings.request_delay_seconds,
network_error_retries=settings.ingestion_network_error_retries,
retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
max_pages_per_scholar=settings.ingestion_max_pages_per_scholar,
page_size=settings.ingestion_page_size,
auto_queue_continuations=settings.ingestion_continuation_queue_enabled,
queue_delay_seconds=settings.ingestion_continuation_base_delay_seconds,
idempotency_key=idempotency_key,
alert_blocked_failure_threshold=settings.ingestion_alert_blocked_failure_threshold,
alert_network_failure_threshold=settings.ingestion_alert_network_failure_threshold,
alert_retry_scheduled_threshold=settings.ingestion_alert_retry_scheduled_threshold,
)
except ingestion_service.RunAlreadyInProgressError as exc:
await db_session.rollback()
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run is already in progress for this account.",
) from exc
except ingestion_service.RunBlockedBySafetyPolicyError as exc:
await db_session.rollback()
logger.info(
"api.runs.manual_blocked_safety",
extra={
"event": "api.runs.manual_blocked_safety",
"user_id": current_user.id,
"reason": exc.safety_state.get("cooldown_reason"),
"cooldown_until": exc.safety_state.get("cooldown_until"),
"cooldown_remaining_seconds": exc.safety_state.get("cooldown_remaining_seconds"),
"metric_name": "api_runs_manual_blocked_safety_total",
"metric_value": 1,
},
)
raise ApiException(
status_code=429,
code=exc.code,
message=exc.message,
details={
"safety_state": exc.safety_state,
},
) from exc
except IntegrityError as exc:
await db_session.rollback()
if idempotency_key is not None:
existing_run = await run_service.get_manual_run_by_idempotency_key(
run_summary = await _execute_manual_run(
db_session,
request=request,
ingest_service=ingest_service,
user_id=current_user.id,
idempotency_key=idempotency_key,
)
if existing_run is not None:
if existing_run.status == RunStatus.RUNNING:
raise ApiException(
status_code=409,
code="run_in_progress",
message="A run with this idempotency key is still in progress.",
details={
"run_id": int(existing_run.id),
"idempotency_key": idempotency_key,
},
) from exc
return success_payload(
request,
data=_manual_run_payload_from_run(
existing_run,
idempotency_key=idempotency_key,
reused_existing_run=True,
safety_state=await _load_safety_state(
db_session,
user_id=current_user.id,
),
),
)
logger.exception(
"api.runs.manual_integrity_error",
extra={
"event": "api.runs.manual_integrity_error",
"user_id": current_user.id,
},
)
raise ApiException(
status_code=500,
code="manual_run_failed",
message="Manual run failed.",
) from exc
except Exception as exc:
await db_session.rollback()
logger.exception(
"api.runs.manual_failed",
extra={
"event": "api.runs.manual_failed",
"user_id": current_user.id,
},
)
raise ApiException(
status_code=500,
code="manual_run_failed",
message="Manual run failed.",
) from exc
if isinstance(run_summary, dict):
return run_summary
current_safety_state = await _load_safety_state(
db_session,
@ -548,18 +630,11 @@ async def run_manual(
)
return success_payload(
request,
data={
"run_id": run_summary.crawl_run_id,
"status": run_summary.status.value,
"scholar_count": run_summary.scholar_count,
"succeeded_count": run_summary.succeeded_count,
"failed_count": run_summary.failed_count,
"partial_count": run_summary.partial_count,
"new_publication_count": run_summary.new_publication_count,
"reused_existing_run": False,
"idempotency_key": idempotency_key,
"safety_state": current_safety_state,
},
data=_manual_run_success_payload(
run_summary=run_summary,
idempotency_key=idempotency_key,
safety_state=current_safety_state,
),
)

178
app/api/routers/scholars.py
View file

@ -25,9 +25,9 @@ from app.api.schemas import (
)
from app.db.models import User
from app.db.session import get_db_session
from app.services import import_export as import_export_service
from app.services import scholars as scholar_service
from app.services.scholar_source import ScholarSource
from app.services.domains.portability import application as import_export_service
from app.services.domains.scholars import application as scholar_service
from app.services.domains.scholar.source import ScholarSource
from app.settings import settings
logger = logging.getLogger(__name__)
@ -64,6 +64,107 @@ def _serialize_scholar(profile) -> dict[str, object]:
}
async def _hydrate_scholar_metadata_if_needed(
db_session: AsyncSession,
*,
profile,
source: ScholarSource,
user_id: int,
):
try:
if not profile.profile_image_url or not (profile.display_name or "").strip():
return await asyncio.wait_for(
scholar_service.hydrate_profile_metadata(
db_session,
profile=profile,
source=source,
),
timeout=5.0,
)
except Exception:
logger.warning(
"api.scholars.create_metadata_hydration_failed",
extra={
"event": "api.scholars.create_metadata_hydration_failed",
"user_id": user_id,
"scholar_profile_id": profile.id,
},
)
return profile
def _search_kwargs() -> dict[str, object]:
return {
"network_error_retries": settings.ingestion_network_error_retries,
"retry_backoff_seconds": settings.ingestion_retry_backoff_seconds,
"search_enabled": settings.scholar_name_search_enabled,
"cache_ttl_seconds": settings.scholar_name_search_cache_ttl_seconds,
"blocked_cache_ttl_seconds": settings.scholar_name_search_blocked_cache_ttl_seconds,
"cache_max_entries": settings.scholar_name_search_cache_max_entries,
"min_interval_seconds": settings.scholar_name_search_min_interval_seconds,
"interval_jitter_seconds": settings.scholar_name_search_interval_jitter_seconds,
"cooldown_block_threshold": settings.scholar_name_search_cooldown_block_threshold,
"cooldown_seconds": settings.scholar_name_search_cooldown_seconds,
"retry_alert_threshold": settings.scholar_name_search_alert_retry_count_threshold,
"cooldown_rejection_alert_threshold": (
settings.scholar_name_search_alert_cooldown_rejections_threshold
),
}
def _search_response_data(query: str, parsed) -> dict[str, object]:
return {
"query": query.strip(),
"state": parsed.state.value,
"state_reason": parsed.state_reason,
"action_hint": scholar_service.scrape_state_hint(
state=parsed.state,
state_reason=parsed.state_reason,
),
"candidates": [
{
"scholar_id": item.scholar_id,
"display_name": item.display_name,
"affiliation": item.affiliation,
"email_domain": item.email_domain,
"cited_by_count": item.cited_by_count,
"interests": item.interests,
"profile_url": item.profile_url,
"profile_image_url": item.profile_image_url,
}
for item in parsed.candidates
],
"warnings": parsed.warnings,
}
async def _read_uploaded_image(image: UploadFile) -> bytes:
try:
return await image.read()
finally:
await image.close()
async def _require_user_profile(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_id: int,
):
profile = await scholar_service.get_user_scholar_by_id(
db_session,
user_id=user_id,
scholar_profile_id=scholar_profile_id,
)
if profile is None:
raise ApiException(
status_code=404,
code="scholar_not_found",
message="Scholar not found.",
)
return profile
@router.get(
"",
response_model=ScholarsListEnvelope,
@ -181,24 +282,11 @@ async def create_scholar(
"scholar_profile_id": created.id,
},
)
try:
if not created.profile_image_url or not (created.display_name or "").strip():
created = await asyncio.wait_for(
scholar_service.hydrate_profile_metadata(
created = await _hydrate_scholar_metadata_if_needed(
db_session,
profile=created,
source=source,
),
timeout=5.0,
)
except Exception:
logger.warning(
"api.scholars.create_metadata_hydration_failed",
extra={
"event": "api.scholars.create_metadata_hydration_failed",
"user_id": current_user.id,
"scholar_profile_id": created.id,
},
user_id=current_user.id,
)
return success_payload(
@ -225,20 +313,7 @@ async def search_scholars(
db_session=db_session,
query=query,
limit=limit,
network_error_retries=settings.ingestion_network_error_retries,
retry_backoff_seconds=settings.ingestion_retry_backoff_seconds,
search_enabled=settings.scholar_name_search_enabled,
cache_ttl_seconds=settings.scholar_name_search_cache_ttl_seconds,
blocked_cache_ttl_seconds=settings.scholar_name_search_blocked_cache_ttl_seconds,
cache_max_entries=settings.scholar_name_search_cache_max_entries,
min_interval_seconds=settings.scholar_name_search_min_interval_seconds,
interval_jitter_seconds=settings.scholar_name_search_interval_jitter_seconds,
cooldown_block_threshold=settings.scholar_name_search_cooldown_block_threshold,
cooldown_seconds=settings.scholar_name_search_cooldown_seconds,
retry_alert_threshold=settings.scholar_name_search_alert_retry_count_threshold,
cooldown_rejection_alert_threshold=(
settings.scholar_name_search_alert_cooldown_rejections_threshold
),
**_search_kwargs(),
)
except scholar_service.ScholarServiceError as exc:
raise ApiException(
@ -259,29 +334,7 @@ async def search_scholars(
)
return success_payload(
request,
data={
"query": query.strip(),
"state": parsed.state.value,
"state_reason": parsed.state_reason,
"action_hint": scholar_service.scrape_state_hint(
state=parsed.state,
state_reason=parsed.state_reason,
),
"candidates": [
{
"scholar_id": item.scholar_id,
"display_name": item.display_name,
"affiliation": item.affiliation,
"email_domain": item.email_domain,
"cited_by_count": item.cited_by_count,
"interests": item.interests,
"profile_url": item.profile_url,
"profile_image_url": item.profile_image_url,
}
for item in parsed.candidates
],
"warnings": parsed.warnings,
},
data=_search_response_data(query, parsed),
)
@ -424,20 +477,14 @@ async def upload_scholar_image(
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
profile = await scholar_service.get_user_scholar_by_id(
profile = await _require_user_profile(
db_session,
user_id=current_user.id,
scholar_profile_id=scholar_profile_id,
)
if profile is None:
raise ApiException(
status_code=404,
code="scholar_not_found",
message="Scholar not found.",
)
image_bytes = await _read_uploaded_image(image)
try:
image_bytes = await image.read()
updated = await scholar_service.set_profile_image_upload(
db_session,
profile=profile,
@ -452,9 +499,8 @@ async def upload_scholar_image(
code="invalid_scholar_image",
message=str(exc),
) from exc
finally:
await image.close()
image_size = len(image_bytes)
logger.info(
"api.scholars.image_uploaded",
extra={
@ -462,7 +508,7 @@ async def upload_scholar_image(
"user_id": current_user.id,
"scholar_profile_id": updated.id,
"content_type": image.content_type,
"size_bytes": len(image_bytes),
"size_bytes": image_size,
},
)
return success_payload(

157
app/api/routers/settings.py
View file

@ -11,8 +11,8 @@ from app.api.responses import success_payload
from app.api.schemas import SettingsEnvelope, SettingsUpdateRequest
from app.db.models import User
from app.db.session import get_db_session
from app.services import run_safety as run_safety_service
from app.services import user_settings as user_settings_service
from app.services.domains.ingestion import safety as run_safety_service
from app.services.domains.settings import application as user_settings_service
from app.settings import settings as settings_module
logger = logging.getLogger(__name__)
@ -21,12 +21,7 @@ router = APIRouter(prefix="/settings", tags=["api-settings"])
def _serialize_settings(user_settings) -> dict[str, object]:
min_run_interval_minutes = user_settings_service.resolve_run_interval_minimum(
settings_module.ingestion_min_run_interval_minutes
)
min_request_delay_seconds = user_settings_service.resolve_request_delay_minimum(
settings_module.ingestion_min_request_delay_seconds
)
min_run_interval_minutes, min_request_delay_seconds = _minimum_policy()
return {
"auto_run_enabled": bool(user_settings.auto_run_enabled) and settings_module.ingestion_automation_allowed,
"run_interval_minutes": int(user_settings.run_interval_minutes),
@ -46,6 +41,76 @@ def _serialize_settings(user_settings) -> dict[str, object]:
}
def _minimum_policy() -> tuple[int, int]:
min_run_interval_minutes = user_settings_service.resolve_run_interval_minimum(
settings_module.ingestion_min_run_interval_minutes
)
min_request_delay_seconds = user_settings_service.resolve_request_delay_minimum(
settings_module.ingestion_min_request_delay_seconds
)
return min_run_interval_minutes, min_request_delay_seconds
def _parse_settings_payload(payload: SettingsUpdateRequest, user_settings) -> tuple[int, int, list[str]]:
min_run_interval_minutes, min_request_delay_seconds = _minimum_policy()
parsed_interval = user_settings_service.parse_run_interval_minutes(
str(payload.run_interval_minutes),
minimum=min_run_interval_minutes,
)
parsed_delay = user_settings_service.parse_request_delay_seconds(
str(payload.request_delay_seconds),
minimum=min_request_delay_seconds,
)
parsed_nav_visible_pages = user_settings_service.parse_nav_visible_pages(
payload.nav_visible_pages
if payload.nav_visible_pages is not None
else list(user_settings.nav_visible_pages or user_settings_service.DEFAULT_NAV_VISIBLE_PAGES)
)
return parsed_interval, parsed_delay, parsed_nav_visible_pages
async def _clear_expired_cooldown_with_log(
db_session: AsyncSession,
*,
user_id: int,
user_settings,
) -> None:
previous_safety_state = run_safety_service.get_safety_event_context(user_settings)
if not run_safety_service.clear_expired_cooldown(user_settings):
return
await db_session.commit()
await db_session.refresh(user_settings)
logger.info(
"api.settings.safety_cooldown_cleared",
extra={
"event": "api.settings.safety_cooldown_cleared",
"user_id": user_id,
"reason": previous_safety_state.get("cooldown_reason"),
"cooldown_until": previous_safety_state.get("cooldown_until"),
"metric_name": "api_settings_safety_cooldown_cleared_total",
"metric_value": 1,
},
)
def _effective_auto_run_enabled(payload: SettingsUpdateRequest) -> bool:
return bool(payload.auto_run_enabled) and settings_module.ingestion_automation_allowed
def _log_settings_update(*, user_id: int, updated) -> None:
logger.info(
"api.settings.updated",
extra={
"event": "api.settings.updated",
"user_id": user_id,
"auto_run_enabled": updated.auto_run_enabled,
"run_interval_minutes": updated.run_interval_minutes,
"request_delay_seconds": updated.request_delay_seconds,
"nav_visible_pages": updated.nav_visible_pages,
},
)
@router.get(
"",
response_model=SettingsEnvelope,
@ -59,20 +124,10 @@ async def get_settings(
db_session,
user_id=current_user.id,
)
previous_safety_state = run_safety_service.get_safety_event_context(user_settings)
if run_safety_service.clear_expired_cooldown(user_settings):
await db_session.commit()
await db_session.refresh(user_settings)
logger.info(
"api.settings.safety_cooldown_cleared",
extra={
"event": "api.settings.safety_cooldown_cleared",
"user_id": current_user.id,
"reason": previous_safety_state.get("cooldown_reason"),
"cooldown_until": previous_safety_state.get("cooldown_until"),
"metric_name": "api_settings_safety_cooldown_cleared_total",
"metric_value": 1,
},
await _clear_expired_cooldown_with_log(
db_session,
user_id=current_user.id,
user_settings=user_settings,
)
return success_payload(
request,
@ -95,26 +150,10 @@ async def update_settings(
user_id=current_user.id,
)
min_run_interval_minutes = user_settings_service.resolve_run_interval_minimum(
settings_module.ingestion_min_run_interval_minutes
)
min_request_delay_seconds = user_settings_service.resolve_request_delay_minimum(
settings_module.ingestion_min_request_delay_seconds
)
try:
parsed_interval = user_settings_service.parse_run_interval_minutes(
str(payload.run_interval_minutes),
minimum=min_run_interval_minutes,
)
parsed_delay = user_settings_service.parse_request_delay_seconds(
str(payload.request_delay_seconds),
minimum=min_request_delay_seconds,
)
parsed_nav_visible_pages = user_settings_service.parse_nav_visible_pages(
payload.nav_visible_pages
if payload.nav_visible_pages is not None
else list(user_settings.nav_visible_pages or user_settings_service.DEFAULT_NAV_VISIBLE_PAGES)
parsed_interval, parsed_delay, parsed_nav_visible_pages = _parse_settings_payload(
payload,
user_settings,
)
except user_settings_service.UserSettingsServiceError as exc:
raise ApiException(
@ -123,44 +162,20 @@ async def update_settings(
message=str(exc),
) from exc
effective_auto_run_enabled = (
bool(payload.auto_run_enabled) and settings_module.ingestion_automation_allowed
)
updated = await user_settings_service.update_settings(
db_session,
settings=user_settings,
auto_run_enabled=effective_auto_run_enabled,
auto_run_enabled=_effective_auto_run_enabled(payload),
run_interval_minutes=parsed_interval,
request_delay_seconds=parsed_delay,
nav_visible_pages=parsed_nav_visible_pages,
)
previous_safety_state = run_safety_service.get_safety_event_context(updated)
if run_safety_service.clear_expired_cooldown(updated):
await db_session.commit()
await db_session.refresh(updated)
logger.info(
"api.settings.safety_cooldown_cleared",
extra={
"event": "api.settings.safety_cooldown_cleared",
"user_id": current_user.id,
"reason": previous_safety_state.get("cooldown_reason"),
"cooldown_until": previous_safety_state.get("cooldown_until"),
"metric_name": "api_settings_safety_cooldown_cleared_total",
"metric_value": 1,
},
)
logger.info(
"api.settings.updated",
extra={
"event": "api.settings.updated",
"user_id": current_user.id,
"auto_run_enabled": updated.auto_run_enabled,
"run_interval_minutes": updated.run_interval_minutes,
"request_delay_seconds": updated.request_delay_seconds,
"nav_visible_pages": updated.nav_visible_pages,
},
await _clear_expired_cooldown_with_log(
db_session,
user_id=current_user.id,
user_settings=updated,
)
_log_settings_update(user_id=current_user.id, updated=updated)
return success_payload(
request,
data=_serialize_settings(updated),

23
app/api/schemas.py
View file

@ -205,6 +205,7 @@ class PublicationExportItemData(BaseModel):
author_text: str | None = None
venue_text: str | None = None
pub_url: str | None = None
doi: str | None = None
pdf_url: str | None = None
is_read: bool = False
@ -573,6 +574,7 @@ class PublicationItemData(BaseModel):
citation_count: int
venue_text: str | None
pub_url: str | None
doi: str | None
pdf_url: str | None
is_read: bool
first_seen_at: datetime
@ -640,3 +642,24 @@ class MarkSelectedReadEnvelope(BaseModel):
meta: ApiMeta
model_config = ConfigDict(extra="forbid")
class RetryPublicationPdfRequest(BaseModel):
scholar_profile_id: int = Field(ge=1)
model_config = ConfigDict(extra="forbid")
class RetryPublicationPdfData(BaseModel):
message: str
resolved_pdf: bool
publication: PublicationItemData
model_config = ConfigDict(extra="forbid")
class RetryPublicationPdfEnvelope(BaseModel):
data: RetryPublicationPdfData
meta: ApiMeta
model_config = ConfigDict(extra="forbid")

2
app/auth/runtime.py
View file

@ -8,7 +8,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.auth.session import clear_session_user, get_session_user, set_session_user
from app.db.models import User
from app.security.csrf import CSRF_SESSION_KEY
from app.services import users as user_service
from app.services.domains.users import application as user_service
logger = logging.getLogger(__name__)

1
app/db/models.py
View file

@ -227,6 +227,7 @@ class Publication(Base):
author_text: Mapped[str | None] = mapped_column(Text)
venue_text: Mapped[str | None] = mapped_column(Text)
pub_url: Mapped[str | None] = mapped_column(Text)
doi: Mapped[str | None] = mapped_column(String(255))
pdf_url: Mapped[str | None] = mapped_column(Text)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, server_default=func.now()

3
app/services/continuation_queue.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.ingestion.queue import *

5
app/services/domains/crossref/__init__.py Normal file
View file

@ -0,0 +1,5 @@
from __future__ import annotations
from app.services.domains.crossref.application import discover_doi_for_publication
__all__ = ["discover_doi_for_publication"]

259
app/services/domains/crossref/application.py Normal file
View file

@ -0,0 +1,259 @@
from __future__ import annotations
import asyncio
import logging
import re
import threading
import time
from typing import TYPE_CHECKING
from crossref.restful import Etiquette, Works
from app.services.domains.doi.normalize import normalize_doi
from app.settings import settings
if TYPE_CHECKING:
from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
TOKEN_RE = re.compile(r"[a-z0-9]+")
NON_ALNUM_RE = re.compile(r"[^a-z0-9\\s]+")
STOP_WORDS = {"the", "and", "for", "with", "from", "method", "study", "analysis"}
_RATE_LOCK = threading.Lock()
_LAST_REQUEST_AT = 0.0
logger = logging.getLogger(__name__)
def _rate_limit_wait(min_interval_seconds: float) -> None:
global _LAST_REQUEST_AT
interval = max(float(min_interval_seconds), 0.0)
with _RATE_LOCK:
elapsed = time.monotonic() - _LAST_REQUEST_AT
remaining = interval - elapsed
if remaining > 0:
time.sleep(remaining)
_LAST_REQUEST_AT = time.monotonic()
def _normalized_tokens(value: str) -> list[str]:
lowered = value.lower().replace("’", "'").replace("“", "\"").replace("”", "\"")
lowered = NON_ALNUM_RE.sub(" ", lowered)
return [token for token in TOKEN_RE.findall(lowered) if len(token) >= 3]
def _normalized_query(value: str) -> str:
tokens = [token for token in _normalized_tokens(value) if token not in STOP_WORDS]
if len(tokens) < 3:
tokens = _normalized_tokens(value)
if len(tokens) < 3:
return ""
return " ".join(tokens[:12]).strip()
def _query_author(value: str) -> str | None:
tokens = [token for token in value.strip().split() if token]
if len(tokens) < 2:
return None
return " ".join(tokens[:2])[:64]
def _author_surname(value: str) -> str | None:
tokens = [token for token in value.strip().split() if token]
if not tokens:
return None
return NON_ALNUM_RE.sub("", tokens[-1].lower()) or None
def _query_filters(year: int | None) -> list[tuple[str, str] | None]:
if year is None:
return [None]
return [
(f"{year - 1}-01-01", f"{year + 1}-12-31"),
(f"{year}-01-01", f"{year}-12-31"),
None,
]
def _candidate_title(item: dict) -> str:
titles = item.get("title")
if isinstance(titles, list) and titles:
return str(titles[0] or "")
return str(item.get("title") or "")
def _title_match_score(source: str, candidate: str) -> float:
source_tokens = {token for token in _normalized_tokens(source) if len(token) >= 3}
candidate_tokens = {token for token in _normalized_tokens(candidate) if len(token) >= 3}
if not source_tokens or not candidate_tokens:
return 0.0
return len(source_tokens & candidate_tokens) / float(len(source_tokens))
def _candidate_year(item: dict) -> int | None:
issued = item.get("issued")
if not isinstance(issued, dict):
return None
date_parts = issued.get("date-parts")
if not isinstance(date_parts, list) or not date_parts:
return None
first = date_parts[0]
if not isinstance(first, list) or not first:
return None
try:
return int(first[0])
except (TypeError, ValueError):
return None
def _candidate_author_match(item: dict, surname: str | None) -> bool:
if not surname:
return True
authors = item.get("author")
if not isinstance(authors, list):
return False
for author in authors:
if not isinstance(author, dict):
continue
family = NON_ALNUM_RE.sub("", str(author.get("family") or "").lower())
if family and family == surname:
return True
return False
def _candidate_rank(*, title: str, year: int | None, item: dict) -> tuple[float, str | None]:
doi = normalize_doi(str(item.get("DOI") or ""))
if doi is None:
return 0.0, None
score = _title_match_score(title, _candidate_title(item))
candidate_year = _candidate_year(item)
if year is not None and candidate_year is not None:
if abs(year - candidate_year) > 1:
return 0.0, None
score += 0.1
return score, doi
def _best_candidate_doi(
*,
title: str,
year: int | None,
items: list[dict],
author_surname: str | None,
) -> str | None:
best_score = 0.0
best_doi: str | None = None
best_year: int | None = None
for item in items:
if not isinstance(item, dict):
continue
if not _candidate_author_match(item, author_surname):
continue
score, doi = _candidate_rank(title=title, year=year, item=item)
candidate_year = _candidate_year(item)
if doi is None or score < 0.75:
continue
if score > best_score:
best_score = score
best_doi = doi
best_year = candidate_year
continue
if abs(score - best_score) > 0.02:
continue
if best_year is None or candidate_year is None:
continue
if candidate_year < best_year:
best_doi = doi
best_year = candidate_year
return best_doi
def _works_client(email: str | None) -> Works:
if email:
etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
return Works(etiquette=etiquette)
return Works()
def _fetch_items_sync(
*,
query: str,
author: str | None,
date_range: tuple[str, str] | None,
max_rows: int,
email: str | None,
min_interval_seconds: float,
) -> list[dict]:
_rate_limit_wait(min_interval_seconds)
works = _works_client(email)
params = {"bibliographic": query}
if author:
params["author"] = author
request = works.query(**params)
if date_range is not None:
from_date, until_date = date_range
request = request.filter(from_pub_date=from_date, until_pub_date=until_date)
request = request.select(["DOI", "title", "issued", "score", "author"])
items: list[dict] = []
for entry in request:
if isinstance(entry, dict):
items.append(entry)
if len(items) >= max(max_rows, 1):
break
return items
async def _fetch_items(
*,
query: str,
author: str | None,
date_range: tuple[str, str] | None,
max_rows: int,
email: str | None,
) -> list[dict]:
timeout = max(float(settings.crossref_timeout_seconds), 0.5)
try:
return await asyncio.wait_for(
asyncio.to_thread(
_fetch_items_sync,
query=query,
author=author,
date_range=date_range,
max_rows=max_rows,
email=email,
min_interval_seconds=settings.crossref_min_interval_seconds,
),
timeout=timeout,
)
except Exception:
return []
async def discover_doi_for_publication(
*,
item: PublicationListItem | UnreadPublicationItem,
max_rows: int = 10,
email: str | None = None,
) -> str | None:
title = (item.title or "").strip()
query = _normalized_query(title)
if not query:
return None
author = _query_author(item.scholar_label)
author_surname = _author_surname(item.scholar_label)
for date_range in _query_filters(item.year):
items = await _fetch_items(
query=query,
author=author,
date_range=date_range,
max_rows=max_rows,
email=email,
)
doi = _best_candidate_doi(
title=title,
year=item.year,
items=items,
author_surname=author_surname,
)
if doi:
logger.debug("crossref.doi_discovered", extra={"event": "crossref.doi_discovered"})
return doi
return None

23
app/services/domains/doi/normalize.py Normal file
View file

@ -0,0 +1,23 @@
from __future__ import annotations
import re
from urllib.parse import unquote
DOI_RE = re.compile(r"10\.\d{4,9}/[-._;()/:A-Z0-9]+", re.I)
def normalize_doi(value: str | None) -> str | None:
if not value:
return None
match = DOI_RE.search(unquote(value))
if not match:
return None
return match.group(0).rstrip(" .;,)").lower()
def first_doi_from_texts(*values: str | None) -> str | None:
for value in values:
doi = normalize_doi(value)
if doi:
return doi
return None

44
app/services/domains/ingestion/application.py
View file

@ -28,6 +28,7 @@ from app.services.domains.ingestion.constants import (
RESUMABLE_PARTIAL_REASONS,
RUN_LOCK_NAMESPACE,
)
from app.services.domains.doi.normalize import first_doi_from_texts
from app.services.domains.ingestion.fingerprints import (
_build_body_excerpt,
_dedupe_publication_candidates,
@ -55,6 +56,7 @@ from app.services.domains.scholar.parser import (
ParseState,
ParsedProfilePage,
PublicationCandidate,
ScholarParserError,
parse_profile_page,
)
from app.services.domains.scholar.source import FetchResult, ScholarSource
@ -1431,7 +1433,7 @@ class ScholarIngestionService:
cstart=cstart,
page_size=page_size,
)
parsed_page = parse_profile_page(fetch_result)
parsed_page = self._parse_profile_page_or_layout_error(fetch_result=fetch_result)
attempt_log.append(
self._attempt_log_entry(
attempt=attempt_index + 1,
@ -1458,6 +1460,38 @@ class ScholarIngestionService:
raise RuntimeError("Fetch-and-parse retry loop produced no result.")
return fetch_result, parsed_page, attempt_log
def _parse_profile_page_or_layout_error(
self,
*,
fetch_result: FetchResult,
) -> ParsedProfilePage:
try:
return parse_profile_page(fetch_result)
except ScholarParserError as exc:
return self._parsed_page_from_parser_error(
fetch_result=fetch_result,
code=exc.code,
)
@staticmethod
def _parsed_page_from_parser_error(
*,
fetch_result: FetchResult,
code: str,
) -> ParsedProfilePage:
return ParsedProfilePage(
state=ParseState.LAYOUT_CHANGED,
state_reason=code,
profile_name=None,
profile_image_url=None,
publications=[],
marker_counts={},
warnings=[code],
has_show_more_button=False,
has_operation_error_banner=False,
articles_range=None,
)
@staticmethod
def _page_log_entry(
*,
@ -1942,7 +1976,8 @@ class ScholarIngestionService:
author_text=candidate.authors_text,
venue_text=candidate.venue_text,
pub_url=build_publication_url(candidate.title_url),
pdf_url=build_publication_url(candidate.pdf_url),
doi=first_doi_from_texts(candidate.title_url, candidate.venue_text, candidate.title),
pdf_url=None,
)
db_session.add(publication)
await db_session.flush()
@ -1976,8 +2011,9 @@ class ScholarIngestionService:
publication.venue_text = candidate.venue_text
if candidate.title_url:
publication.pub_url = build_publication_url(candidate.title_url)
if candidate.pdf_url:
publication.pdf_url = build_publication_url(candidate.pdf_url)
local_doi = first_doi_from_texts(candidate.title_url, candidate.venue_text, candidate.title)
if local_doi and not publication.doi:
publication.doi = local_doi
async def _resolve_publication(
self,

3
app/services/domains/portability/exporting.py
View file

@ -34,6 +34,7 @@ def _serialize_export_publication(row: tuple[Any, ...]) -> dict[str, Any]:
author_text,
venue_text,
pub_url,
doi,
pdf_url,
is_read,
) = row
@ -47,6 +48,7 @@ def _serialize_export_publication(row: tuple[Any, ...]) -> dict[str, Any]:
"author_text": author_text,
"venue_text": venue_text,
"pub_url": pub_url,
"doi": doi,
"pdf_url": pdf_url,
"is_read": bool(is_read),
}
@ -73,6 +75,7 @@ async def export_user_data(
Publication.author_text,
Publication.venue_text,
Publication.pub_url,
Publication.doi,
Publication.pdf_url,
ScholarPublication.is_read,
)

10
app/services/domains/portability/publication_import.py
View file

@ -7,6 +7,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import Publication, ScholarProfile, ScholarPublication
from app.services.domains.ingestion.application import build_publication_url, normalize_title
from app.services.domains.doi.normalize import normalize_doi
from app.services.domains.portability.normalize import (
_normalize_citation_count,
_normalize_optional_text,
@ -70,6 +71,7 @@ def _apply_imported_publication_values(
author_text: str | None,
venue_text: str | None,
pub_url: str | None,
doi: str | None,
pdf_url: str | None,
cluster_id: str | None,
) -> bool:
@ -96,6 +98,9 @@ def _apply_imported_publication_values(
if pub_url and publication.pub_url != pub_url:
publication.pub_url = pub_url
updated = True
if doi and publication.doi != doi:
publication.doi = doi
updated = True
if pdf_url and publication.pdf_url != pdf_url:
publication.pdf_url = pdf_url
updated = True
@ -112,6 +117,7 @@ def _new_publication(
author_text: str | None,
venue_text: str | None,
pub_url: str | None,
doi: str | None,
pdf_url: str | None,
) -> Publication:
return Publication(
@ -124,6 +130,7 @@ def _new_publication(
author_text=author_text,
venue_text=venue_text,
pub_url=pub_url,
doi=doi,
pdf_url=pdf_url,
)
@ -226,6 +233,7 @@ def _build_imported_publication_input(
venue_text=venue_text,
cluster_id=_normalize_optional_text(item.get("cluster_id")),
pub_url=build_publication_url(_normalize_optional_text(item.get("pub_url"))),
doi=normalize_doi(_normalize_optional_text(item.get("doi"))),
pdf_url=build_publication_url(_normalize_optional_text(item.get("pdf_url"))),
fingerprint=_resolve_fingerprint(
title=title,
@ -277,6 +285,7 @@ async def _create_import_publication(
author_text=payload.author_text,
venue_text=payload.venue_text,
pub_url=payload.pub_url,
doi=payload.doi,
pdf_url=payload.pdf_url,
)
db_session.add(publication)
@ -297,6 +306,7 @@ def _update_import_publication(
author_text=payload.author_text,
venue_text=payload.venue_text,
pub_url=payload.pub_url,
doi=payload.doi,
pdf_url=payload.pdf_url,
cluster_id=payload.cluster_id,
)

1
app/services/domains/portability/types.py
View file

@ -19,6 +19,7 @@ class ImportedPublicationInput:
venue_text: str | None
cluster_id: str | None
pub_url: str | None
doi: str | None
pdf_url: str | None
fingerprint: str
is_read: bool

8
app/services/domains/publications/application.py
View file

@ -8,8 +8,12 @@ from app.services.domains.publications.counts import (
from app.services.domains.publications.listing import (
list_for_user,
list_new_for_latest_run_for_user,
retry_pdf_for_user,
list_unread_for_user,
)
from app.services.domains.publications.enrichment import (
schedule_missing_pdf_enrichment_for_user,
)
from app.services.domains.publications.modes import (
MODE_ALL,
MODE_LATEST,
@ -20,6 +24,7 @@ from app.services.domains.publications.modes import (
)
from app.services.domains.publications.queries import (
get_latest_completed_run_id_for_user,
get_publication_item_for_user,
publications_query,
)
from app.services.domains.publications.read_state import (
@ -39,9 +44,12 @@ __all__ = [
"resolve_mode",
"get_latest_completed_run_id_for_user",
"publications_query",
"get_publication_item_for_user",
"list_for_user",
"list_unread_for_user",
"list_new_for_latest_run_for_user",
"retry_pdf_for_user",
"schedule_missing_pdf_enrichment_for_user",
"count_for_user",
"count_unread_for_user",
"count_latest_for_user",

135
app/services/domains/publications/enrichment.py Normal file
View file

@ -0,0 +1,135 @@
from __future__ import annotations
import asyncio
import logging
import time
from app.db.session import get_session_factory
from app.services.domains.publications.listing import (
missing_pdf_items,
resolve_and_persist_oa_metadata,
)
from app.services.domains.publications.types import PublicationListItem
from app.settings import settings
logger = logging.getLogger(__name__)
_enrichment_lock = asyncio.Lock()
_inflight_publications: set[tuple[int, int]] = set()
_recent_attempt_seconds: dict[tuple[int, int], float] = {}
_scheduled_tasks: set[asyncio.Task[None]] = set()
def _cooldown_seconds() -> float:
return max(float(settings.unpaywall_retry_cooldown_seconds), 1.0)
def _prune_recent_attempts(now_seconds: float, *, cooldown_seconds: float) -> None:
expiry = cooldown_seconds * 3
stale_keys = [
key for key, attempted_seconds in _recent_attempt_seconds.items()
if now_seconds - attempted_seconds >= expiry
]
for key in stale_keys:
_recent_attempt_seconds.pop(key, None)
async def _claim_items(
*,
user_id: int,
items: list[PublicationListItem],
max_items: int,
) -> list[PublicationListItem]:
candidates = missing_pdf_items(items, limit=max_items)
if not candidates:
return []
now_seconds = time.monotonic()
cooldown_seconds = _cooldown_seconds()
claimed: list[PublicationListItem] = []
async with _enrichment_lock:
_prune_recent_attempts(now_seconds, cooldown_seconds=cooldown_seconds)
for item in candidates:
key = (user_id, item.publication_id)
attempted_seconds = _recent_attempt_seconds.get(key)
if key in _inflight_publications:
continue
if attempted_seconds is not None and now_seconds - attempted_seconds < cooldown_seconds:
continue
_inflight_publications.add(key)
_recent_attempt_seconds[key] = now_seconds
claimed.append(item)
return claimed
async def _release_claims(*, user_id: int, publication_ids: list[int]) -> None:
async with _enrichment_lock:
for publication_id in publication_ids:
_inflight_publications.discard((user_id, publication_id))
def _on_task_done(task: asyncio.Task[None]) -> None:
_scheduled_tasks.discard(task)
try:
task.result()
except Exception:
logger.exception(
"publications.enrichment.task_failed",
extra={"event": "publications.enrichment.task_failed"},
)
async def _run_enrichment(
*,
user_id: int,
request_email: str | None,
items: list[PublicationListItem],
) -> None:
publication_ids = [item.publication_id for item in items]
try:
session_factory = get_session_factory()
async with session_factory() as db_session:
await resolve_and_persist_oa_metadata(
db_session,
rows=items,
unpaywall_email=request_email,
)
finally:
await _release_claims(user_id=user_id, publication_ids=publication_ids)
logger.info(
"publications.enrichment.completed",
extra={
"event": "publications.enrichment.completed",
"user_id": user_id,
"publication_count": len(items),
},
)
async def schedule_missing_pdf_enrichment_for_user(
*,
user_id: int,
request_email: str | None,
items: list[PublicationListItem],
max_items: int,
) -> int:
claimed_items = await _claim_items(user_id=user_id, items=items, max_items=max_items)
if not claimed_items:
return 0
task = asyncio.create_task(
_run_enrichment(
user_id=user_id,
request_email=request_email,
items=claimed_items,
)
)
_scheduled_tasks.add(task)
task.add_done_callback(_on_task_done)
logger.info(
"publications.enrichment.scheduled",
extra={
"event": "publications.enrichment.scheduled",
"user_id": user_id,
"publication_count": len(claimed_items),
},
)
return len(claimed_items)

126
app/services/domains/publications/listing.py
View file

@ -10,11 +10,100 @@ from app.services.domains.publications.modes import (
)
from app.services.domains.publications.queries import (
get_latest_completed_run_id_for_user,
get_publication_item_for_user,
publication_list_item_from_row,
publications_query,
unread_item_from_row,
)
from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
from app.services.domains.unpaywall.application import resolve_publication_oa_metadata
from sqlalchemy import update
from app.db.models import Publication
def _with_oa_overrides(
rows: list[PublicationListItem],
oa_data: dict[int, tuple[str | None, str | None]],
) -> list[PublicationListItem]:
return [
PublicationListItem(
publication_id=row.publication_id,
scholar_profile_id=row.scholar_profile_id,
scholar_label=row.scholar_label,
title=row.title,
year=row.year,
citation_count=row.citation_count,
venue_text=row.venue_text,
pub_url=row.pub_url,
doi=(oa_data.get(row.publication_id) or (None, None))[0] or row.doi,
pdf_url=(oa_data.get(row.publication_id) or (None, None))[1] or row.pdf_url,
is_read=row.is_read,
first_seen_at=row.first_seen_at,
is_new_in_latest_run=row.is_new_in_latest_run,
)
for row in rows
]
def _resolved_fields(
*,
row: PublicationListItem,
resolved: tuple[str | None, str | None] | None,
) -> tuple[str | None, str | None]:
if resolved is None:
return row.doi, row.pdf_url
resolved_doi, resolved_pdf = resolved
return resolved_doi or row.doi, resolved_pdf or row.pdf_url
async def _persist_resolved_metadata(
db_session: AsyncSession,
*,
rows: list[PublicationListItem],
oa_data: dict[int, tuple[str | None, str | None]],
) -> None:
by_id = {row.publication_id: row for row in rows}
updates: list[tuple[int, str | None, str | None]] = []
for publication_id, resolved in oa_data.items():
existing = by_id.get(publication_id)
if existing is None:
continue
next_doi, next_pdf = _resolved_fields(row=existing, resolved=resolved)
if next_doi == existing.doi and next_pdf == existing.pdf_url:
continue
updates.append((publication_id, next_doi, next_pdf))
for publication_id, doi, pdf_url in updates:
await db_session.execute(
update(Publication)
.where(Publication.id == publication_id)
.values(doi=doi, pdf_url=pdf_url)
)
if updates:
await db_session.commit()
def missing_pdf_items(
rows: list[PublicationListItem],
*,
limit: int,
) -> list[PublicationListItem]:
bounded_limit = max(0, int(limit))
if bounded_limit == 0:
return []
return [row for row in rows if not row.pdf_url][:bounded_limit]
async def resolve_and_persist_oa_metadata(
db_session: AsyncSession,
*,
rows: list[PublicationListItem],
unpaywall_email: str | None = None,
) -> dict[int, tuple[str | None, str | None]]:
if not rows:
return {}
oa_data = await resolve_publication_oa_metadata(rows, request_email=unpaywall_email)
await _persist_resolved_metadata(db_session, rows=rows, oa_data=oa_data)
return oa_data
async def list_for_user(
@ -36,10 +125,35 @@ async def list_for_user(
limit=limit,
)
)
return [
rows = [
publication_list_item_from_row(row, latest_run_id=latest_run_id)
for row in result.all()
]
return rows
async def retry_pdf_for_user(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_id: int,
publication_id: int,
unpaywall_email: str | None = None,
) -> PublicationListItem | None:
item = await get_publication_item_for_user(
db_session,
user_id=user_id,
scholar_profile_id=scholar_profile_id,
publication_id=publication_id,
)
if item is None:
return None
oa_data = await resolve_and_persist_oa_metadata(
db_session,
rows=[item],
unpaywall_email=unpaywall_email,
)
return _with_oa_overrides([item], oa_data)[0]
async def list_unread_for_user(
@ -73,8 +187,11 @@ async def list_new_for_latest_run_for_user(
scholar_profile_id=None,
limit=limit,
)
return [
UnreadPublicationItem(
return [_to_unread_item(row) for row in rows]
def _to_unread_item(row: PublicationListItem) -> UnreadPublicationItem:
return UnreadPublicationItem(
publication_id=row.publication_id,
scholar_profile_id=row.scholar_profile_id,
scholar_label=row.scholar_label,
@ -83,7 +200,6 @@ async def list_new_for_latest_run_for_user(
citation_count=row.citation_count,
venue_text=row.venue_text,
pub_url=row.pub_url,
doi=row.doi,
pdf_url=row.pdf_url,
)
for row in rows
]

60
app/services/domains/publications/queries.py
View file

@ -50,6 +50,7 @@ def publications_query(
Publication.citation_count,
Publication.venue_text,
Publication.pub_url,
Publication.doi,
Publication.pdf_url,
ScholarPublication.is_read,
ScholarPublication.first_seen_run_id,
@ -72,6 +73,61 @@ def publications_query(
return stmt
def publication_query_for_user(
*,
user_id: int,
scholar_profile_id: int,
publication_id: int,
) -> Select[tuple]:
return (
select(
Publication.id,
ScholarProfile.id,
ScholarProfile.display_name,
ScholarProfile.scholar_id,
Publication.title_raw,
Publication.year,
Publication.citation_count,
Publication.venue_text,
Publication.pub_url,
Publication.doi,
Publication.pdf_url,
ScholarPublication.is_read,
ScholarPublication.first_seen_run_id,
ScholarPublication.created_at,
)
.join(ScholarPublication, ScholarPublication.publication_id == Publication.id)
.join(ScholarProfile, ScholarProfile.id == ScholarPublication.scholar_profile_id)
.where(
ScholarProfile.user_id == user_id,
ScholarProfile.id == scholar_profile_id,
Publication.id == publication_id,
)
.limit(1)
)
async def get_publication_item_for_user(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_id: int,
publication_id: int,
) -> PublicationListItem | None:
latest_run_id = await get_latest_completed_run_id_for_user(db_session, user_id=user_id)
result = await db_session.execute(
publication_query_for_user(
user_id=user_id,
scholar_profile_id=scholar_profile_id,
publication_id=publication_id,
)
)
row = result.one_or_none()
if row is None:
return None
return publication_list_item_from_row(row, latest_run_id=latest_run_id)
def publication_list_item_from_row(
row: tuple,
*,
@ -87,6 +143,7 @@ def publication_list_item_from_row(
citation_count,
venue_text,
pub_url,
doi,
pdf_url,
is_read,
first_seen_run_id,
@ -101,6 +158,7 @@ def publication_list_item_from_row(
citation_count=_normalized_citation_count(citation_count),
venue_text=venue_text,
pub_url=pub_url,
doi=doi,
pdf_url=pdf_url,
is_read=bool(is_read),
first_seen_at=created_at,
@ -121,6 +179,7 @@ def unread_item_from_row(row: tuple) -> UnreadPublicationItem:
citation_count,
venue_text,
pub_url,
doi,
pdf_url,
_is_read,
_first_seen_run_id,
@ -135,5 +194,6 @@ def unread_item_from_row(row: tuple) -> UnreadPublicationItem:
citation_count=_normalized_citation_count(citation_count),
venue_text=venue_text,
pub_url=pub_url,
doi=doi,
pdf_url=pdf_url,
)

2
app/services/domains/publications/types.py
View file

@ -14,6 +14,7 @@ class PublicationListItem:
citation_count: int
venue_text: str | None
pub_url: str | None
doi: str | None
pdf_url: str | None
is_read: bool
first_seen_at: datetime
@ -30,4 +31,5 @@ class UnreadPublicationItem:
citation_count: int
venue_text: str | None
pub_url: str | None
doi: str | None
pdf_url: str | None

65
app/services/domains/scholar/parser.py
View file

@ -11,14 +11,12 @@ from app.services.domains.scholar.parser_types import (
ParsedAuthorSearchPage,
ParsedProfilePage,
PublicationCandidate,
ScholarDomInvariantError,
ScholarMalformedDataError,
ScholarParserError,
ScholarSearchCandidate,
)
from app.services.domains.scholar.parser_utils import (
attr_class,
attr_href,
attr_src,
build_absolute_scholar_url,
normalize_space,
strip_tags,
)
from app.services.domains.scholar.profile_rows import (
@ -44,6 +42,55 @@ from app.services.domains.scholar.state_detection import (
)
def _raise_dom_error(code: str, message: str) -> None:
raise ScholarDomInvariantError(code=code, message=message)
def _assert_profile_dom_invariants(
*,
fetch_result: FetchResult,
marker_counts: dict[str, int],
publications: list[PublicationCandidate],
warnings: list[str],
has_show_more_button_flag: bool,
articles_range: str | None,
) -> None:
if fetch_result.status_code is None:
return
final_url = (fetch_result.final_url or "").lower()
if "accounts.google.com" in final_url or "sorry/index" in final_url:
return
if any(code.startswith("layout_") for code in warnings):
reason = next(code for code in warnings if code.startswith("layout_"))
_raise_dom_error(reason, f"Detected layout warning: {reason}")
if has_show_more_button_flag and not articles_range:
_raise_dom_error(
"layout_show_more_without_articles_range",
"Show-more control exists without an articles range marker.",
)
if marker_counts.get("gsc_a_tr", 0) > 0 and marker_counts.get("gsc_a_at", 0) <= 0:
_raise_dom_error(
"layout_missing_publication_title_anchor",
"Publication rows were present but title anchors were absent.",
)
if not publications:
has_profile_markers = marker_counts.get("gsc_prf_in", 0) > 0
has_table_markers = marker_counts.get("gsc_a_tr", 0) > 0 or marker_counts.get("gsc_a_at", 0) > 0
if not has_profile_markers and not has_table_markers:
_raise_dom_error("layout_markers_missing", "Expected scholar profile markers were absent.")
for publication in publications:
if not publication.title.strip():
raise ScholarMalformedDataError(
code="malformed_publication_title",
message="Encountered a publication candidate with an empty title.",
)
if publication.citation_count is not None and int(publication.citation_count) < 0:
raise ScholarMalformedDataError(
code="malformed_publication_negative_citations",
message="Encountered a publication candidate with negative citations.",
)
def parse_profile_page(fetch_result: FetchResult) -> ParsedProfilePage:
publications, warnings = parse_publications(fetch_result.body)
marker_counts = count_markers(fetch_result.body)
@ -59,6 +106,14 @@ def parse_profile_page(fetch_result: FetchResult) -> ParsedProfilePage:
warnings.append("operation_error_banner_present")
warnings = sorted(set(warnings))
_assert_profile_dom_invariants(
fetch_result=fetch_result,
marker_counts=marker_counts,
publications=publications,
warnings=warnings,
has_show_more_button_flag=show_more,
articles_range=articles_range,
)
state, state_reason = detect_state(
fetch_result,

16
app/services/domains/scholar/parser_types.py
View file

@ -12,6 +12,22 @@ class ParseState(StrEnum):
NETWORK_ERROR = "network_error"
class ScholarParserError(RuntimeError):
code: str
def __init__(self, *, code: str, message: str) -> None:
super().__init__(message)
self.code = code
class ScholarDomInvariantError(ScholarParserError):
pass
class ScholarMalformedDataError(ScholarParserError):
pass
@dataclass(frozen=True)
class PublicationCandidate:
title: str

56
app/services/domains/scholar/profile_rows.py
View file

@ -2,13 +2,10 @@ from __future__ import annotations
import re
from html.parser import HTMLParser
from typing import Any
from urllib.parse import parse_qs, urlparse
from app.services.domains.scholar.parser_constants import (
MARKER_KEYS,
PROFILE_ROW_DIRECT_LABEL_TOKENS,
PROFILE_ROW_PARSER_DIRECT_MARKERS,
SHOW_MORE_BUTTON_RE,
)
from app.services.domains.scholar.parser_types import PublicationCandidate
@ -26,7 +23,6 @@ class ScholarRowParser(HTMLParser):
def __init__(self) -> None:
super().__init__(convert_charrefs=True)
self.title_href: str | None = None
self.direct_download_href: str | None = None
self.title_parts: list[str] = []
self.citation_parts: list[str] = []
self.year_parts: list[str] = []
@ -35,14 +31,7 @@ class ScholarRowParser(HTMLParser):
self._title_depth = 0
self._citation_depth = 0
self._year_depth = 0
self._gray_stack: list[dict[str, Any]] = []
self._direct_marker_depth = 0
self._aux_link_stack: list[dict[str, Any]] = []
@staticmethod
def _contains_direct_marker(classes: str) -> bool:
lowered = classes.lower()
return any(marker in lowered for marker in PROFILE_ROW_PARSER_DIRECT_MARKERS)
self._gray_stack: list[dict[str, object]] = []
def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
if self._title_depth > 0:
@ -53,14 +42,8 @@ class ScholarRowParser(HTMLParser):
self._year_depth += 1
if self._gray_stack:
self._gray_stack[-1]["depth"] += 1
if self._direct_marker_depth > 0:
self._direct_marker_depth += 1
if self._aux_link_stack:
self._aux_link_stack[-1]["depth"] += 1
classes = attr_class(attrs)
if tag in {"div", "span"} and self._contains_direct_marker(classes):
self._direct_marker_depth = 1
if tag == "a" and "gsc_a_at" in classes:
self._title_depth = 1
@ -79,17 +62,6 @@ class ScholarRowParser(HTMLParser):
self._gray_stack.append({"depth": 1, "parts": []})
return
if tag == "a":
href = attr_href(attrs)
self._aux_link_stack.append(
{
"depth": 1,
"classes": classes,
"href": href,
"parts": [],
}
)
def handle_data(self, data: str) -> None:
if self._title_depth > 0:
self.title_parts.append(data)
@ -99,23 +71,6 @@ class ScholarRowParser(HTMLParser):
self.year_parts.append(data)
if self._gray_stack:
self._gray_stack[-1]["parts"].append(data)
if self._aux_link_stack:
self._aux_link_stack[-1]["parts"].append(data)
def _capture_direct_download_href(self, link_info: dict[str, Any]) -> None:
if self.direct_download_href:
return
href = link_info.get("href")
if not href:
return
classes = str(link_info.get("classes") or "")
label = normalize_space("".join(link_info.get("parts") or [])).lower()
if "gs_ggsd" in classes or "gs_ggs" in classes or "gs_ggsa" in classes:
self.direct_download_href = href
return
label_match = any(token in label for token in PROFILE_ROW_DIRECT_LABEL_TOKENS)
if self._direct_marker_depth > 0 and label_match:
self.direct_download_href = href
def handle_endtag(self, tag: str) -> None:
if self._title_depth > 0:
@ -131,13 +86,6 @@ class ScholarRowParser(HTMLParser):
if text_value:
self.gray_texts.append(text_value)
self._gray_stack.pop()
if self._aux_link_stack:
self._aux_link_stack[-1]["depth"] -= 1
if self._aux_link_stack[-1]["depth"] == 0:
link_info = self._aux_link_stack.pop()
self._capture_direct_download_href(link_info)
if self._direct_marker_depth > 0:
self._direct_marker_depth -= 1
def extract_rows(html: str) -> list[str]:
@ -223,7 +171,7 @@ def _parse_publication_row(row_html: str) -> tuple[PublicationCandidate | None,
citation_count=citation_count,
authors_text=authors_text,
venue_text=venue_text,
pdf_url=build_absolute_scholar_url(parser.direct_download_href),
pdf_url=None,
),
warnings,
)

13
app/services/domains/scholars/application.py
View file

@ -18,6 +18,7 @@ from app.db.models import AuthorSearchCacheEntry, AuthorSearchRuntimeState, Scho
from app.services.domains.scholar.parser import (
ParseState,
ParsedAuthorSearchPage,
ScholarParserError,
parse_author_search_page,
parse_profile_page,
)
@ -635,7 +636,16 @@ async def _fetch_author_search_with_retries(
retry_scheduled_count = 0
for attempt_index in range(max_attempts):
fetch_result = await source.fetch_author_search_html(normalized_query, start=0)
try:
parsed = parse_author_search_page(fetch_result)
except ScholarParserError as exc:
parsed = ParsedAuthorSearchPage(
state=ParseState.LAYOUT_CHANGED,
state_reason=exc.code,
candidates=[],
marker_counts={},
warnings=[exc.code],
)
if parsed.state != ParseState.NETWORK_ERROR or attempt_index >= max_attempts - 1:
break
retry_warnings.append("network_retry_scheduled_for_author_search")
@ -873,7 +883,10 @@ async def hydrate_profile_metadata(
source: ScholarSource,
) -> ScholarProfile:
fetch_result = await source.fetch_profile_html(profile.scholar_id)
try:
parsed_page = parse_profile_page(fetch_result)
except ScholarParserError:
return profile
if parsed_page.profile_name and not (profile.display_name or "").strip():
profile.display_name = parsed_page.profile_name

5
app/services/domains/unpaywall/__init__.py Normal file
View file

@ -0,0 +1,5 @@
from __future__ import annotations
from app.services.domains.unpaywall.application import resolve_publication_pdf_urls
__all__ = ["resolve_publication_pdf_urls"]

218
app/services/domains/unpaywall/application.py Normal file
View file

@ -0,0 +1,218 @@
from __future__ import annotations
import logging
import re
from urllib.parse import unquote
from app.services.domains.crossref.application import discover_doi_for_publication
from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
from app.services.domains.doi.normalize import normalize_doi
from app.settings import settings
DOI_PATTERN = re.compile(r"10\.\d{4,9}/[-._;()/:A-Z0-9]+", re.I)
DOI_PREFIX_RE = re.compile(r"\bdoi\s*[:=]\s*(10\.\d{4,9}/[-._;()/:A-Z0-9]+)", re.I)
DOI_URL_RE = re.compile(r"(?:https?://)?(?:dx\.)?doi\.org/(10\.\d{4,9}/[-._;()/:A-Z0-9]+)", re.I)
UNPAYWALL_URL_TEMPLATE = "https://api.unpaywall.org/v2/{doi}"
logger = logging.getLogger(__name__)
def _extract_doi_candidate(text: str | None) -> str | None:
if not text:
return None
decoded = unquote(text)
match = DOI_PATTERN.search(decoded)
if not match:
return None
return match.group(0).rstrip(" .;,)")
def _extract_explicit_doi(text: str | None) -> str | None:
if not text:
return None
decoded = unquote(text)
url_match = DOI_URL_RE.search(decoded)
if url_match:
return normalize_doi(url_match.group(1))
prefix_match = DOI_PREFIX_RE.search(decoded)
if prefix_match:
return normalize_doi(prefix_match.group(1))
return None
def _publication_doi(item: PublicationListItem | UnreadPublicationItem) -> str | None:
stored = normalize_doi(item.doi)
if stored:
in_metadata = any(
normalize_doi(_extract_explicit_doi(value)) == stored
for value in (item.pub_url, item.venue_text)
)
if in_metadata:
return stored
pub_url_doi = _extract_doi_candidate(item.pub_url)
if pub_url_doi:
return normalize_doi(pub_url_doi)
return (
_extract_explicit_doi(item.pub_url)
or _extract_explicit_doi(item.venue_text)
)
def _payload_pdf_url(payload: dict) -> str | None:
best = payload.get("best_oa_location")
if isinstance(best, dict):
pdf_url = best.get("url_for_pdf")
if isinstance(pdf_url, str) and pdf_url.strip():
return pdf_url.strip()
locations = payload.get("oa_locations")
if not isinstance(locations, list):
return None
for location in locations:
if not isinstance(location, dict):
continue
pdf_url = location.get("url_for_pdf")
if isinstance(pdf_url, str) and pdf_url.strip():
return pdf_url.strip()
return None
async def _fetch_unpaywall_payload_by_doi(
*,
client,
doi: str,
email: str,
) -> dict | None:
response = await client.get(
UNPAYWALL_URL_TEMPLATE.format(doi=doi),
params={"email": email},
)
if response.status_code != 200:
return None
payload = response.json()
if not isinstance(payload, dict):
return None
return payload
def _email_for_request(request_email: str | None) -> str | None:
email = (request_email or "").strip() or settings.unpaywall_email.strip()
return email or None
def _log_resolution_summary(
*,
publication_count: int,
doi_input_count: int,
search_attempt_count: int,
resolved_pdf_count: int,
email: str,
) -> None:
logger.info(
"unpaywall.resolve_completed",
extra={
"event": "unpaywall.resolve_completed",
"publication_count": publication_count,
"doi_input_count": doi_input_count,
"search_attempt_count": search_attempt_count,
"resolved_pdf_count": resolved_pdf_count,
"email_domain": email.split("@", 1)[-1] if "@" in email else None,
},
)
async def _resolve_item_payload(
*,
client,
item: PublicationListItem,
email: str,
allow_crossref: bool,
) -> tuple[dict | None, bool]:
doi = _publication_doi(item)
payload: dict | None = None
if doi:
payload = await _fetch_unpaywall_payload_by_doi(client=client, doi=doi, email=email)
if payload is not None and _payload_pdf_url(payload):
return payload, False
if not allow_crossref or not settings.crossref_enabled:
return payload, False
crossref_doi = await discover_doi_for_publication(
item=item,
max_rows=settings.crossref_max_rows,
email=email,
)
if crossref_doi is None or crossref_doi == doi:
return payload, crossref_doi is not None
crossref_payload = await _fetch_unpaywall_payload_by_doi(
client=client,
doi=crossref_doi,
email=email,
)
if crossref_payload is not None:
return crossref_payload, True
return payload, True
def _doi_and_pdf_from_payload(payload: dict) -> tuple[str | None, str | None]:
doi = normalize_doi(str(payload.get("doi") or ""))
return doi, _payload_pdf_url(payload)
def _resolution_targets(items: list[PublicationListItem]) -> list[PublicationListItem]:
return [item for item in items if not item.pdf_url]
def _crossref_budget_value() -> int:
return max(int(settings.crossref_max_lookups_per_request), 0)
async def resolve_publication_oa_metadata(
items: list[PublicationListItem],
*,
request_email: str | None = None,
) -> dict[int, tuple[str | None, str | None]]:
if not settings.unpaywall_enabled:
return {}
email = _email_for_request(request_email)
if email is None:
logger.debug("unpaywall.resolve_skipped_missing_email")
return {}
import httpx
timeout_seconds = max(float(settings.unpaywall_timeout_seconds), 0.5)
resolved: dict[int, tuple[str | None, str | None]] = {}
crossref_budget = _crossref_budget_value()
crossref_lookups = 0
targets = _resolution_targets(items)[: max(int(settings.unpaywall_max_items_per_request), 0)]
async with httpx.AsyncClient(timeout=timeout_seconds) as client:
for item in targets:
allow_crossref = crossref_budget > 0 and crossref_lookups < crossref_budget
payload, used_crossref = await _resolve_item_payload(
client=client,
item=item,
email=email,
allow_crossref=allow_crossref,
)
if used_crossref:
crossref_lookups += 1
if not isinstance(payload, dict):
continue
resolved[item.publication_id] = _doi_and_pdf_from_payload(payload)
resolved_count = sum(1 for _doi, pdf in resolved.values() if pdf)
doi_input_count = len([item for item in items if _publication_doi(item)])
target_doi_count = len([item for item in targets if _publication_doi(item)])
_log_resolution_summary(
publication_count=len(items),
doi_input_count=doi_input_count,
search_attempt_count=max(0, len(targets) - target_doi_count),
resolved_pdf_count=resolved_count,
email=email,
)
return resolved
async def resolve_publication_pdf_urls(
items: list[PublicationListItem],
*,
request_email: str | None = None,
) -> dict[int, str | None]:
resolved = await resolve_publication_oa_metadata(items, request_email=request_email)
return {publication_id: pdf for publication_id, (_doi, pdf) in resolved.items()}

3
app/services/import_export.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.portability.application import *

3
app/services/ingestion.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.ingestion.application import *

3
app/services/publications.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.publications.application import *

3
app/services/run_safety.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.ingestion.safety import *

3
app/services/runs.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.runs.application import *

3
app/services/scheduler.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.ingestion.scheduler import *

3
app/services/scholar_parser.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.scholar.parser import *

4
app/services/scholar_source.py
View file

@ -1,4 +0,0 @@
from __future__ import annotations
from app.services.domains.scholar.source import *
from app.services.domains.scholar.source import _build_profile_url

3
app/services/scholars.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.scholars.application import *

3
app/services/user_settings.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.settings.application import *

3
app/services/users.py
View file

@ -1,3 +0,0 @@
from __future__ import annotations
from app.services.domains.users.application import *

10
app/settings.py
View file

@ -229,6 +229,16 @@ class Settings:
"SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD",
3,
)
unpaywall_enabled: bool = _env_bool("UNPAYWALL_ENABLED", True)
unpaywall_email: str = _env_str("UNPAYWALL_EMAIL", "")
unpaywall_timeout_seconds: float = _env_float("UNPAYWALL_TIMEOUT_SECONDS", 4.0)
unpaywall_max_items_per_request: int = _env_int("UNPAYWALL_MAX_ITEMS_PER_REQUEST", 20)
unpaywall_retry_cooldown_seconds: int = _env_int("UNPAYWALL_RETRY_COOLDOWN_SECONDS", 1800)
crossref_enabled: bool = _env_bool("CROSSREF_ENABLED", True)
crossref_max_rows: int = _env_int("CROSSREF_MAX_ROWS", 10)
crossref_timeout_seconds: float = _env_float("CROSSREF_TIMEOUT_SECONDS", 8.0)
crossref_min_interval_seconds: float = _env_float("CROSSREF_MIN_INTERVAL_SECONDS", 0.6)
crossref_max_lookups_per_request: int = _env_int("CROSSREF_MAX_LOOKUPS_PER_REQUEST", 8)
settings = Settings()

2
frontend/src/components/layout/AppNav.vue
View file

@ -81,7 +81,7 @@ function onNavigate(): void {
class="min-h-0 min-w-0 overflow-x-hidden border-b border-stroke-subtle bg-surface-nav/70 px-4 py-4 lg:h-full lg:border-b-0 lg:border-r lg:px-5 lg:py-6"
>
<div class="flex h-full min-h-0 flex-col gap-4">
<nav class="grid grid-cols-2 content-start gap-2 sm:grid-cols-3 lg:grid-cols-1" aria-label="Primary">
<nav class="grid grid-cols-1 content-start gap-2" aria-label="Primary">
<RouterLink
v-for="link in links"
:key="link.to"

21
frontend/src/features/publications/index.ts
View file

@ -11,6 +11,7 @@ export interface PublicationItem {
citation_count: number;
venue_text: string | null;
pub_url: string | null;
doi: string | null;
pdf_url: string | null;
is_read: boolean;
first_seen_at: string;
@ -83,3 +84,23 @@ export async function markSelectedRead(selections: PublicationSelection[]): Prom
});
return response.data;
}
export interface RetryPublicationPdfResult {
message: string;
resolved_pdf: boolean;
publication: PublicationItem;
}
export async function retryPublicationPdf(
publicationId: number,
scholarProfileId: number,
): Promise<RetryPublicationPdfResult> {
const response = await apiRequest<RetryPublicationPdfResult>(
`/publications/${publicationId}/retry-pdf`,
{
method: "POST",
body: { scholar_profile_id: scholarProfileId },
},
);
return response.data;
}

1
frontend/src/features/scholars/index.ts
View file

@ -54,6 +54,7 @@ export interface PublicationExportItem {
author_text: string | null;
venue_text: string | null;
pub_url: string | null;
doi: string | null;
pdf_url: string | null;
is_read: boolean;
}

116
frontend/src/pages/PublicationsPage.vue
View file

@ -16,6 +16,7 @@ import {
listPublications,
markAllRead,
markSelectedRead,
retryPublicationPdf,
type PublicationItem,
type PublicationMode,
type PublicationsResult,
@ -37,6 +38,7 @@ const sortDirection = ref<"asc" | "desc">("desc");
const scholars = ref<ScholarProfile[]>([]);
const listState = ref<PublicationsResult | null>(null);
const selectedPublicationKeys = ref<Set<string>>(new Set());
const retryingPublicationKeys = ref<Set<string>>(new Set());
const errorMessage = ref<string | null>(null);
const errorRequestId = ref<string | null>(null);
@ -103,6 +105,13 @@ function publicationPrimaryUrl(item: PublicationItem): string | null {
return item.pub_url || item.pdf_url;
}
function publicationDoiUrl(item: PublicationItem): string | null {
if (!item.doi) {
return null;
}
return `https://doi.org/${item.doi}`;
}
const selectedScholarName = computed(() => {
const selectedId = Number(selectedScholarFilter.value);
if (!Number.isInteger(selectedId) || selectedId <= 0) {
@ -324,6 +333,49 @@ function onToggleRowSelection(item: PublicationItem, event: Event): void {
selectedPublicationKeys.value = next;
}
function isRetryingPublication(item: PublicationItem): boolean {
return retryingPublicationKeys.value.has(publicationKey(item));
}
function replacePublication(updated: PublicationItem): void {
if (!listState.value) {
return;
}
listState.value.publications = listState.value.publications.map((item) =>
publicationKey(item) === publicationKey(updated) ? updated : item,
);
}
async function onRetryPdf(item: PublicationItem): Promise<void> {
if (item.pdf_url) {
return;
}
const key = publicationKey(item);
const next = new Set(retryingPublicationKeys.value);
next.add(key);
retryingPublicationKeys.value = next;
errorMessage.value = null;
errorRequestId.value = null;
successMessage.value = null;
try {
const response = await retryPublicationPdf(item.publication_id, item.scholar_profile_id);
replacePublication(response.publication);
successMessage.value = response.message;
} catch (error) {
if (error instanceof ApiRequestError) {
errorMessage.value = error.message;
errorRequestId.value = error.requestId;
} else {
errorMessage.value = "Unable to retry PDF lookup for this publication.";
}
} finally {
const cleared = new Set(retryingPublicationKeys.value);
cleared.delete(key);
retryingPublicationKeys.value = cleared;
}
}
async function onMarkSelectedRead(): Promise<void> {
if (selectedPublicationKeys.value.size === 0 || !listState.value) {
return;
@ -404,19 +456,19 @@ watch(
<template>
<AppPage
title="Publications"
subtitle="Filter discovered publications, then mark what you have read so upcoming checks stay focused."
subtitle="Review discoveries, open PDFs, and keep read state current."
>
<AppCard class="space-y-4">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="space-y-1">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">What you can do here</h2>
<h2 class="text-lg font-semibold text-ink-primary">Publications workspace</h2>
<AppHelpHint
text="Publications are records discovered from tracked scholar profiles. Unread mode focuses only on items you have not marked as read."
text="Use this workspace to filter discoveries, open OA PDF links, and mark reviewed papers as read so future checks stay focused."
/>
</div>
<p class="text-sm text-secondary">
Select a scholar or scope, search within results, and mark items as read when you are done.
Choose scope, triage results, and update read state in one place.
</p>
</div>
@ -427,7 +479,10 @@ watch(
<div class="grid gap-3 xl:grid-cols-[minmax(0,15rem)_minmax(0,18rem)_minmax(0,1fr)] xl:items-end">
<div class="grid gap-1 text-xs text-secondary">
<div class="flex items-center gap-1">
<span>View mode</span>
<AppHelpHint text="Unread shows only unreviewed records; All records includes the full publication history." />
</div>
<div class="flex min-h-10 flex-wrap items-center gap-2">
<AppButton
:variant="mode === 'unread' ? 'primary' : 'secondary'"
@ -449,7 +504,10 @@ watch(
</div>
<label class="grid gap-1 text-xs text-secondary" for="publications-scholar-filter">
<span>Scholar</span>
<span class="inline-flex items-center gap-1">
Scholar
<AppHelpHint text="Filter results to one tracked scholar profile. The selected scholar is synced to the URL query." />
</span>
<AppSelect
id="publications-scholar-filter"
v-model="selectedScholarFilter"
@ -464,7 +522,10 @@ watch(
</label>
<label class="grid gap-1 text-xs text-secondary" for="publications-search-input">
<span>Search within current scope</span>
<span class="inline-flex items-center gap-1">
Search current results
<AppHelpHint text="Search matches title, scholar, venue, and year within the currently loaded scope." />
</span>
<div class="flex min-w-0 items-center gap-2">
<AppInput
id="publications-search-input"
@ -506,7 +567,6 @@ watch(
</AppButton>
</div>
</div>
</AppCard>
<RequestStateAlerts
:success-message="successMessage"
@ -517,12 +577,12 @@ watch(
@dismiss-success="successMessage = null"
/>
<AppCard class="space-y-4">
<section class="space-y-3 border-t border-stroke-default pt-3">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Publication List</h2>
<h3 class="text-base font-semibold text-ink-primary">Results</h3>
<AppHelpHint
text="Use sorting, search, and bulk actions here to move discovered records from unread into read history."
text="Sort columns to prioritize review. PDF opens a resolved open-access link, while retry triggers a new lookup for that row."
/>
</div>
<span class="text-xs text-secondary">Currently showing {{ selectedScholarName }}</span>
@ -559,6 +619,7 @@ watch(
Scholar <span aria-hidden="true">{{ sortMarker("scholar") }}</span>
</button>
</th>
<th scope="col">PDF</th>
<th scope="col">
<button type="button" class="table-sort" @click="toggleSort('year')">
Year <span aria-hidden="true">{{ sortMarker("year") }}</span>
@ -606,17 +667,37 @@ watch(
</a>
<span v-else>{{ item.title }}</span>
<a
v-if="item.pdf_url"
:href="item.pdf_url"
v-if="publicationDoiUrl(item)"
:href="publicationDoiUrl(item) || ''"
target="_blank"
rel="noreferrer"
class="link-inline text-xs"
>
Direct PDF
DOI: {{ item.doi }}
</a>
</div>
</td>
<td>{{ item.scholar_label }}</td>
<td>
<a
v-if="item.pdf_url"
:href="item.pdf_url"
target="_blank"
rel="noreferrer"
class="pdf-link-button"
>
PDF
</a>
<button
v-else
type="button"
class="pdf-retry-button"
:disabled="isRetryingPublication(item)"
@click="onRetryPdf(item)"
>
{{ isRetryingPublication(item) ? "..." : "retry" }}
</button>
</td>
<td>{{ item.year ?? "n/a" }}</td>
<td>{{ item.citation_count }}</td>
<td>
@ -634,6 +715,7 @@ watch(
</tbody>
</AppTable>
</AsyncStateGate>
</section>
</AppCard>
</AppPage>
</template>
@ -642,4 +724,12 @@ watch(
.table-sort {
@apply inline-flex items-center gap-1 text-left font-semibold text-ink-primary transition hover:text-ink-link focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset;
}
.pdf-link-button {
@apply inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium border-state-success-border bg-state-success-bg text-state-success-text shadow-sm transition hover:brightness-95 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset;
}
.pdf-retry-button {
@apply inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium border-state-warning-border bg-state-warning-bg text-state-warning-text transition hover:brightness-95 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset disabled:cursor-not-allowed disabled:opacity-50;
}
</style>

1
frontend/src/pages/ScholarsPage.vue
View file

@ -8,6 +8,7 @@ import AppAlert from "@/components/ui/AppAlert.vue";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppEmptyState from "@/components/ui/AppEmptyState.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
import AppModal from "@/components/ui/AppModal.vue";

3
pyproject.toml
View file

@ -12,7 +12,9 @@ dependencies = [
"alembic>=1.14,<2.0",
"argon2-cffi>=25.1,<26.0",
"asyncpg>=0.30,<0.31",
"crossrefapi>=1.6,<2.0",
"fastapi>=0.116,<0.117",
"httpx>=0.28,<0.29",
"itsdangerous>=2.2,<3.0",
"python-multipart>=0.0.9,<0.1",
"sqlalchemy>=2.0,<2.1",
@ -21,7 +23,6 @@ dependencies = [
[project.optional-dependencies]
dev = [
"httpx>=0.28,<0.29",
"pytest>=8.3,<9.0",
"pytest-asyncio>=0.25,<0.26",
]

180
tests/integration/test_api_v1.py
View file

@ -10,8 +10,8 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runtime_deps import get_scholar_source
from app.main import app
from app.services import user_settings as user_settings_service
from app.services.scholar_source import FetchResult
from app.services.domains.settings import application as user_settings_service
from app.services.domains.scholar.source import FetchResult
from app.settings import settings
from tests.integration.helpers import insert_user, login_user
@ -1465,6 +1465,182 @@ async def test_api_publications_list_and_mark_read(db_session: AsyncSession) ->
assert mark_response.json()["data"]["updated_count"] == 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_list_schedules_background_enrichment(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-bg@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "background111",
"display_name": "Background Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 3)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 17):064x}",
"title_raw": "Background Target",
"title_normalized": "background target",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
},
)
await db_session.commit()
async def _fail_resolver(*args, **kwargs):
raise AssertionError("list endpoint should not resolve OA metadata inline")
async def _fake_schedule(*, user_id: int, request_email: str | None, items, max_items: int):
assert user_id > 0
assert request_email == "api-pubs-bg@example.com"
assert int(max_items) > 0
assert any(int(item.publication_id) == publication_id for item in items)
return 1
monkeypatch.setattr(
"app.services.domains.publications.listing.resolve_publication_oa_metadata",
_fail_resolver,
)
monkeypatch.setattr(
"app.services.domains.publications.application.schedule_missing_pdf_enrichment_for_user",
_fake_schedule,
)
client = TestClient(app)
login_user(client, email="api-pubs-bg@example.com", password="api-password")
response = client.get("/api/v1/publications?mode=all")
assert response.status_code == 200
data = response.json()["data"]
assert len(data["publications"]) == 1
assert int(data["publications"][0]["publication_id"]) == publication_id
assert data["publications"][0]["pdf_url"] is None
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publication_retry_pdf_updates_publication(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-retry@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "retryScholar01",
"display_name": "Retry Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 7)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 9):064x}",
"title_raw": "Retry Target",
"title_normalized": "retry target",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
},
)
await db_session.commit()
async def _fake_resolver(items, *, request_email=None):
assert request_email == "api-pubs-retry@example.com"
return {int(items[0].publication_id): ("10.1000/retry-target", "https://oa.example/retry.pdf")}
monkeypatch.setattr(
"app.services.domains.publications.listing.resolve_publication_oa_metadata",
_fake_resolver,
)
client = TestClient(app)
login_user(client, email="api-pubs-retry@example.com", password="api-password")
headers = _api_csrf_headers(client)
response = client.post(
f"/api/v1/publications/{publication_id}/retry-pdf",
json={"scholar_profile_id": scholar_profile_id},
headers=headers,
)
assert response.status_code == 200
payload = response.json()["data"]
assert payload["resolved_pdf"] is True
assert payload["publication"]["publication_id"] == publication_id
assert payload["publication"]["pdf_url"] == "https://oa.example/retry.pdf"
assert payload["publication"]["doi"] == "10.1000/retry-target"
stored = await db_session.execute(
text("SELECT doi, pdf_url FROM publications WHERE id = :publication_id"),
{"publication_id": publication_id},
)
stored_doi, stored_pdf_url = stored.one()
assert stored_doi == "10.1000/retry-target"
assert stored_pdf_url == "https://oa.example/retry.pdf"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio

4
tests/integration/test_fixture_probe_runs.py
View file

@ -8,8 +8,8 @@ from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import RunStatus, RunTriggerType
from app.services.ingestion import ScholarIngestionService
from app.services.scholar_source import FetchResult
from app.services.domains.ingestion.application import ScholarIngestionService
from app.services.domains.scholar.source import FetchResult
from tests.integration.helpers import insert_user
REGRESSION_FIXTURE_DIR = Path("tests/fixtures/scholar/regression")

2
tests/integration/test_migrations.py
View file

@ -16,7 +16,7 @@ EXPECTED_TABLES = {
}
EXPECTED_ENUMS = {"run_status", "run_trigger_type"}
EXPECTED_REVISION = "20260219_0010"
EXPECTED_REVISION = "20260220_0011"
@pytest.mark.integration

75
tests/unit/test_crossref_lookup.py Normal file
View file

@ -0,0 +1,75 @@
from __future__ import annotations
from types import SimpleNamespace
import pytest
from app.services.domains.crossref import application as crossref_app
def _item(*, title: str, year: int | None, scholar_label: str = "Shinya Yamanaka"):
return SimpleNamespace(
publication_id=1,
scholar_label=scholar_label,
title=title,
year=year,
)
@pytest.mark.asyncio
async def test_crossref_discovers_doi_from_best_title_match(monkeypatch: pytest.MonkeyPatch) -> None:
responses = [
[],
[
{
"DOI": "10.1000/noisy",
"title": ["Completely unrelated paper"],
"issued": {"date-parts": [[2014]]},
"author": [{"family": "Other"}],
},
{
"DOI": "10.1016/j.cell.2007.11.019",
"title": ["Induction of Pluripotent Stem Cells from Adult Human Fibroblasts"],
"issued": {"date-parts": [[2007]]},
"author": [{"family": "Yamanaka"}],
},
],
]
async def _fake_fetch_items(**_kwargs):
return responses.pop(0) if responses else []
monkeypatch.setattr(crossref_app, "_fetch_items", _fake_fetch_items)
doi = await crossref_app.discover_doi_for_publication(
item=_item(
title="Induction of Pluripotent Stem Cells from Adult Human Fibroblasts",
year=2007,
),
max_rows=10,
email="user@example.com",
)
assert doi == "10.1016/j.cell.2007.11.019"
@pytest.mark.asyncio
async def test_crossref_rejects_large_year_mismatch(monkeypatch: pytest.MonkeyPatch) -> None:
async def _fake_fetch_items(**_kwargs):
return [
{
"DOI": "10.1000/wrong-year",
"title": ["Induction of Pluripotent Stem Cells from Adult Human Fibroblasts"],
"issued": {"date-parts": [[2014]]},
"author": [{"family": "Yamanaka"}],
}
]
monkeypatch.setattr(crossref_app, "_fetch_items", _fake_fetch_items)
doi = await crossref_app.discover_doi_for_publication(
item=_item(
title="Induction of Pluripotent Stem Cells from Adult Human Fibroblasts",
year=2007,
),
max_rows=10,
email=None,
)
assert doi is None

17
tests/unit/test_doi_normalize.py Normal file
View file

@ -0,0 +1,17 @@
from __future__ import annotations
from app.services.domains.doi.normalize import first_doi_from_texts, normalize_doi
def test_normalize_doi_extracts_and_lowercases() -> None:
value = "https://doi.org/10.48550/ARXIV.1412.6980"
assert normalize_doi(value) == "10.48550/arxiv.1412.6980"
def test_first_doi_from_texts_prefers_first_match() -> None:
result = first_doi_from_texts(
"no doi here",
"venue text 10.1000/ABC-123",
"title with 10.9999/ignored",
)
assert result == "10.1000/abc-123"

2
tests/unit/test_run_safety.py
View file

@ -3,7 +3,7 @@ from __future__ import annotations
from datetime import datetime, timedelta, timezone
from app.db.models import UserSetting
from app.services import run_safety
from app.services.domains.ingestion import safety as run_safety
def test_apply_run_safety_outcome_triggers_blocked_cooldown() -> None:

2
tests/unit/test_runs_summary.py
View file

@ -1,6 +1,6 @@
from __future__ import annotations
from app.services.runs import extract_run_summary
from app.services.domains.runs.application import extract_run_summary
def test_extract_run_summary_includes_extended_metrics() -> None:

26
tests/unit/test_scholar_parser.py
View file

@ -2,12 +2,15 @@ from __future__ import annotations
from pathlib import Path
from app.services.scholar_parser import (
import pytest
from app.services.domains.scholar.parser import (
ParseState,
ScholarDomInvariantError,
parse_author_search_page,
parse_profile_page,
)
from app.services.scholar_source import FetchResult
from app.services.domains.scholar.source import FetchResult
def _fixture(name: str) -> str:
@ -98,7 +101,7 @@ def test_parse_profile_page_handles_missing_optional_metadata() -> None:
assert publication.venue_text is None
def test_parse_profile_page_extracts_direct_pdf_link() -> None:
def test_parse_profile_page_ignores_direct_pdf_link_markup() -> None:
html = """
<html>
<div id="gsc_prf_in">Direct PDF Test</div>
@ -131,7 +134,7 @@ def test_parse_profile_page_extracts_direct_pdf_link() -> None:
assert parsed.state == ParseState.OK
assert len(parsed.publications) == 1
assert parsed.publications[0].pdf_url == "https://example.org/paper.pdf"
assert parsed.publications[0].pdf_url is None
def test_parse_profile_page_fails_fast_when_citation_markup_is_unparseable() -> None:
@ -160,10 +163,9 @@ def test_parse_profile_page_fails_fast_when_citation_markup_is_unparseable() ->
error=None,
)
parsed = parse_profile_page(fetch_result)
assert parsed.state == ParseState.LAYOUT_CHANGED
assert parsed.state_reason == "layout_row_citation_unparseable"
with pytest.raises(ScholarDomInvariantError) as exc:
parse_profile_page(fetch_result)
assert exc.value.code == "layout_row_citation_unparseable"
def test_parse_profile_page_detects_layout_change_when_markers_absent() -> None:
@ -175,11 +177,9 @@ def test_parse_profile_page_detects_layout_change_when_markers_absent() -> None:
error=None,
)
parsed = parse_profile_page(fetch_result)
assert parsed.state == ParseState.LAYOUT_CHANGED
assert parsed.state_reason == "layout_markers_missing"
assert "no_rows_detected" in parsed.warnings
with pytest.raises(ScholarDomInvariantError) as exc:
parse_profile_page(fetch_result)
assert exc.value.code == "layout_markers_missing"
def test_parse_profile_page_reports_network_reason_when_status_missing() -> None:

6
tests/unit/test_scholar_search_safety.py
View file

@ -3,9 +3,9 @@ from __future__ import annotations
import pytest
from sqlalchemy.ext.asyncio import AsyncSession
from app.services import scholars as scholar_service
from app.services.scholar_parser import ParseState
from app.services.scholar_source import FetchResult
from app.services.domains.scholars import application as scholar_service
from app.services.domains.scholar.parser import ParseState
from app.services.domains.scholar.source import FetchResult
pytestmark = [pytest.mark.integration, pytest.mark.db]

2
tests/unit/test_scholar_source.py
View file

@ -1,4 +1,4 @@
from app.services.scholar_source import _build_profile_url
from app.services.domains.scholar.source import _build_profile_url
def test_build_profile_url_includes_pagesize_for_initial_page() -> None:

2
tests/unit/test_user_settings.py
View file

@ -2,7 +2,7 @@ from __future__ import annotations
import pytest
from app.services.user_settings import (
from app.services.domains.settings.application import (
DEFAULT_NAV_VISIBLE_PAGES,
HARD_MIN_REQUEST_DELAY_SECONDS,
HARD_MIN_RUN_INTERVAL_MINUTES,

255
uv.lock generated
View file

@ -85,6 +85,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/42/b9/f8d6fa329ab25128b7e98fd83a3cb34d9db5b059a9847eddb840a0af45dd/argon2_cffi_bindings-25.1.0-cp39-abi3-win_arm64.whl", hash = "sha256:b0fdbcf513833809c882823f98dc2f931cf659d9a1429616ac3adebb49f5db94", size = 27149 },
]
[[package]]
name = "asttokens"
version = "3.0.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/be/a5/8e3f9b6771b0b408517c82d97aed8f2036509bc247d46114925e32fe33f0/asttokens-3.0.1.tar.gz", hash = "sha256:71a4ee5de0bde6a31d64f6b13f2293ac190344478f081c3d1bccfcf5eacb0cb7", size = 62308 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/d2/39/e7eaf1799466a4aef85b6a4fe7bd175ad2b1c6345066aa33f1f58d4b18d0/asttokens-3.0.1-py3-none-any.whl", hash = "sha256:15a3ebc0f43c2d0a50eeafea25e19046c68398e487b9f1f5b517f7c0f40f976a", size = 27047 },
]
[[package]]
name = "asyncpg"
version = "0.30.0"
@ -175,6 +184,63 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195 },
]
[[package]]
name = "charset-normalizer"
version = "3.4.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/13/69/33ddede1939fdd074bce5434295f38fae7136463422fe4fd3e0e89b98062/charset_normalizer-3.4.4.tar.gz", hash = "sha256:94537985111c35f28720e43603b8e7b43a6ecfb2ce1d3058bbe955b73404e21a", size = 129418 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f3/85/1637cd4af66fa687396e757dec650f28025f2a2f5a5531a3208dc0ec43f2/charset_normalizer-3.4.4-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:0a98e6759f854bd25a58a73fa88833fba3b7c491169f86ce1180c948ab3fd394", size = 208425 },
{ url = "https://files.pythonhosted.org/packages/9d/6a/04130023fef2a0d9c62d0bae2649b69f7b7d8d24ea5536feef50551029df/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5b290ccc2a263e8d185130284f8501e3e36c5e02750fc6b6bdeb2e9e96f1e25", size = 148162 },
{ url = "https://files.pythonhosted.org/packages/78/29/62328d79aa60da22c9e0b9a66539feae06ca0f5a4171ac4f7dc285b83688/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74bb723680f9f7a6234dcf67aea57e708ec1fbdf5699fb91dfd6f511b0a320ef", size = 144558 },
{ url = "https://files.pythonhosted.org/packages/86/bb/b32194a4bf15b88403537c2e120b817c61cd4ecffa9b6876e941c3ee38fe/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f1e34719c6ed0b92f418c7c780480b26b5d9c50349e9a9af7d76bf757530350d", size = 161497 },
{ url = "https://files.pythonhosted.org/packages/19/89/a54c82b253d5b9b111dc74aca196ba5ccfcca8242d0fb64146d4d3183ff1/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:2437418e20515acec67d86e12bf70056a33abdacb5cb1655042f6538d6b085a8", size = 159240 },
{ url = "https://files.pythonhosted.org/packages/c0/10/d20b513afe03acc89ec33948320a5544d31f21b05368436d580dec4e234d/charset_normalizer-3.4.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:11d694519d7f29d6cd09f6ac70028dba10f92f6cdd059096db198c283794ac86", size = 153471 },
{ url = "https://files.pythonhosted.org/packages/61/fa/fbf177b55bdd727010f9c0a3c49eefa1d10f960e5f09d1d887bf93c2e698/charset_normalizer-3.4.4-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ac1c4a689edcc530fc9d9aa11f5774b9e2f33f9a0c6a57864e90908f5208d30a", size = 150864 },
{ url = "https://files.pythonhosted.org/packages/05/12/9fbc6a4d39c0198adeebbde20b619790e9236557ca59fc40e0e3cebe6f40/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:21d142cc6c0ec30d2efee5068ca36c128a30b0f2c53c1c07bd78cb6bc1d3be5f", size = 150647 },
{ url = "https://files.pythonhosted.org/packages/ad/1f/6a9a593d52e3e8c5d2b167daf8c6b968808efb57ef4c210acb907c365bc4/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:5dbe56a36425d26d6cfb40ce79c314a2e4dd6211d51d6d2191c00bed34f354cc", size = 145110 },
{ url = "https://files.pythonhosted.org/packages/30/42/9a52c609e72471b0fc54386dc63c3781a387bb4fe61c20231a4ebcd58bdd/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:5bfbb1b9acf3334612667b61bd3002196fe2a1eb4dd74d247e0f2a4d50ec9bbf", size = 162839 },
{ url = "https://files.pythonhosted.org/packages/c4/5b/c0682bbf9f11597073052628ddd38344a3d673fda35a36773f7d19344b23/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:d055ec1e26e441f6187acf818b73564e6e6282709e9bcb5b63f5b23068356a15", size = 150667 },
{ url = "https://files.pythonhosted.org/packages/e4/24/a41afeab6f990cf2daf6cb8c67419b63b48cf518e4f56022230840c9bfb2/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:af2d8c67d8e573d6de5bc30cdb27e9b95e49115cd9baad5ddbd1a6207aaa82a9", size = 160535 },
{ url = "https://files.pythonhosted.org/packages/2a/e5/6a4ce77ed243c4a50a1fecca6aaaab419628c818a49434be428fe24c9957/charset_normalizer-3.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:780236ac706e66881f3b7f2f32dfe90507a09e67d1d454c762cf642e6e1586e0", size = 154816 },
{ url = "https://files.pythonhosted.org/packages/a8/ef/89297262b8092b312d29cdb2517cb1237e51db8ecef2e9af5edbe7b683b1/charset_normalizer-3.4.4-cp312-cp312-win32.whl", hash = "sha256:5833d2c39d8896e4e19b689ffc198f08ea58116bee26dea51e362ecc7cd3ed26", size = 99694 },
{ url = "https://files.pythonhosted.org/packages/3d/2d/1e5ed9dd3b3803994c155cd9aacb60c82c331bad84daf75bcb9c91b3295e/charset_normalizer-3.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:a79cfe37875f822425b89a82333404539ae63dbdddf97f84dcbc3d339aae9525", size = 107131 },
{ url = "https://files.pythonhosted.org/packages/d0/d9/0ed4c7098a861482a7b6a95603edce4c0d9db2311af23da1fb2b75ec26fc/charset_normalizer-3.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:376bec83a63b8021bb5c8ea75e21c4ccb86e7e45ca4eb81146091b56599b80c3", size = 100390 },
{ url = "https://files.pythonhosted.org/packages/97/45/4b3a1239bbacd321068ea6e7ac28875b03ab8bc0aa0966452db17cd36714/charset_normalizer-3.4.4-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:e1f185f86a6f3403aa2420e815904c67b2f9ebc443f045edd0de921108345794", size = 208091 },
{ url = "https://files.pythonhosted.org/packages/7d/62/73a6d7450829655a35bb88a88fca7d736f9882a27eacdca2c6d505b57e2e/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b39f987ae8ccdf0d2642338faf2abb1862340facc796048b604ef14919e55ed", size = 147936 },
{ url = "https://files.pythonhosted.org/packages/89/c5/adb8c8b3d6625bef6d88b251bbb0d95f8205831b987631ab0c8bb5d937c2/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3162d5d8ce1bb98dd51af660f2121c55d0fa541b46dff7bb9b9f86ea1d87de72", size = 144180 },
{ url = "https://files.pythonhosted.org/packages/91/ed/9706e4070682d1cc219050b6048bfd293ccf67b3d4f5a4f39207453d4b99/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:81d5eb2a312700f4ecaa977a8235b634ce853200e828fbadf3a9c50bab278328", size = 161346 },
{ url = "https://files.pythonhosted.org/packages/d5/0d/031f0d95e4972901a2f6f09ef055751805ff541511dc1252ba3ca1f80cf5/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5bd2293095d766545ec1a8f612559f6b40abc0eb18bb2f5d1171872d34036ede", size = 158874 },
{ url = "https://files.pythonhosted.org/packages/f5/83/6ab5883f57c9c801ce5e5677242328aa45592be8a00644310a008d04f922/charset_normalizer-3.4.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8a8b89589086a25749f471e6a900d3f662d1d3b6e2e59dcecf787b1cc3a1894", size = 153076 },
{ url = "https://files.pythonhosted.org/packages/75/1e/5ff781ddf5260e387d6419959ee89ef13878229732732ee73cdae01800f2/charset_normalizer-3.4.4-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:bc7637e2f80d8530ee4a78e878bce464f70087ce73cf7c1caf142416923b98f1", size = 150601 },
{ url = "https://files.pythonhosted.org/packages/d7/57/71be810965493d3510a6ca79b90c19e48696fb1ff964da319334b12677f0/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f8bf04158c6b607d747e93949aa60618b61312fe647a6369f88ce2ff16043490", size = 150376 },
{ url = "https://files.pythonhosted.org/packages/e5/d5/c3d057a78c181d007014feb7e9f2e65905a6c4ef182c0ddf0de2924edd65/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:554af85e960429cf30784dd47447d5125aaa3b99a6f0683589dbd27e2f45da44", size = 144825 },
{ url = "https://files.pythonhosted.org/packages/e6/8c/d0406294828d4976f275ffbe66f00266c4b3136b7506941d87c00cab5272/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:74018750915ee7ad843a774364e13a3db91682f26142baddf775342c3f5b1133", size = 162583 },
{ url = "https://files.pythonhosted.org/packages/d7/24/e2aa1f18c8f15c4c0e932d9287b8609dd30ad56dbe41d926bd846e22fb8d/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c0463276121fdee9c49b98908b3a89c39be45d86d1dbaa22957e38f6321d4ce3", size = 150366 },
{ url = "https://files.pythonhosted.org/packages/e4/5b/1e6160c7739aad1e2df054300cc618b06bf784a7a164b0f238360721ab86/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:362d61fd13843997c1c446760ef36f240cf81d3ebf74ac62652aebaf7838561e", size = 160300 },
{ url = "https://files.pythonhosted.org/packages/7a/10/f882167cd207fbdd743e55534d5d9620e095089d176d55cb22d5322f2afd/charset_normalizer-3.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9a26f18905b8dd5d685d6d07b0cdf98a79f3c7a918906af7cc143ea2e164c8bc", size = 154465 },
{ url = "https://files.pythonhosted.org/packages/89/66/c7a9e1b7429be72123441bfdbaf2bc13faab3f90b933f664db506dea5915/charset_normalizer-3.4.4-cp313-cp313-win32.whl", hash = "sha256:9b35f4c90079ff2e2edc5b26c0c77925e5d2d255c42c74fdb70fb49b172726ac", size = 99404 },
{ url = "https://files.pythonhosted.org/packages/c4/26/b9924fa27db384bdcd97ab83b4f0a8058d96ad9626ead570674d5e737d90/charset_normalizer-3.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:b435cba5f4f750aa6c0a0d92c541fb79f69a387c91e61f1795227e4ed9cece14", size = 107092 },
{ url = "https://files.pythonhosted.org/packages/af/8f/3ed4bfa0c0c72a7ca17f0380cd9e4dd842b09f664e780c13cff1dcf2ef1b/charset_normalizer-3.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:542d2cee80be6f80247095cc36c418f7bddd14f4a6de45af91dfad36d817bba2", size = 100408 },
{ url = "https://files.pythonhosted.org/packages/2a/35/7051599bd493e62411d6ede36fd5af83a38f37c4767b92884df7301db25d/charset_normalizer-3.4.4-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:da3326d9e65ef63a817ecbcc0df6e94463713b754fe293eaa03da99befb9a5bd", size = 207746 },
{ url = "https://files.pythonhosted.org/packages/10/9a/97c8d48ef10d6cd4fcead2415523221624bf58bcf68a802721a6bc807c8f/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8af65f14dc14a79b924524b1e7fffe304517b2bff5a58bf64f30b98bbc5079eb", size = 147889 },
{ url = "https://files.pythonhosted.org/packages/10/bf/979224a919a1b606c82bd2c5fa49b5c6d5727aa47b4312bb27b1734f53cd/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:74664978bb272435107de04e36db5a9735e78232b85b77d45cfb38f758efd33e", size = 143641 },
{ url = "https://files.pythonhosted.org/packages/ba/33/0ad65587441fc730dc7bd90e9716b30b4702dc7b617e6ba4997dc8651495/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:752944c7ffbfdd10c074dc58ec2d5a8a4cd9493b314d367c14d24c17684ddd14", size = 160779 },
{ url = "https://files.pythonhosted.org/packages/67/ed/331d6b249259ee71ddea93f6f2f0a56cfebd46938bde6fcc6f7b9a3d0e09/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d1f13550535ad8cff21b8d757a3257963e951d96e20ec82ab44bc64aeb62a191", size = 159035 },
{ url = "https://files.pythonhosted.org/packages/67/ff/f6b948ca32e4f2a4576aa129d8bed61f2e0543bf9f5f2b7fc3758ed005c9/charset_normalizer-3.4.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ecaae4149d99b1c9e7b88bb03e3221956f68fd6d50be2ef061b2381b61d20838", size = 152542 },
{ url = "https://files.pythonhosted.org/packages/16/85/276033dcbcc369eb176594de22728541a925b2632f9716428c851b149e83/charset_normalizer-3.4.4-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cb6254dc36b47a990e59e1068afacdcd02958bdcce30bb50cc1700a8b9d624a6", size = 149524 },
{ url = "https://files.pythonhosted.org/packages/9e/f2/6a2a1f722b6aba37050e626530a46a68f74e63683947a8acff92569f979a/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c8ae8a0f02f57a6e61203a31428fa1d677cbe50c93622b4149d5c0f319c1d19e", size = 150395 },
{ url = "https://files.pythonhosted.org/packages/60/bb/2186cb2f2bbaea6338cad15ce23a67f9b0672929744381e28b0592676824/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:47cc91b2f4dd2833fddaedd2893006b0106129d4b94fdb6af1f4ce5a9965577c", size = 143680 },
{ url = "https://files.pythonhosted.org/packages/7d/a5/bf6f13b772fbb2a90360eb620d52ed8f796f3c5caee8398c3b2eb7b1c60d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:82004af6c302b5d3ab2cfc4cc5f29db16123b1a8417f2e25f9066f91d4411090", size = 162045 },
{ url = "https://files.pythonhosted.org/packages/df/c5/d1be898bf0dc3ef9030c3825e5d3b83f2c528d207d246cbabe245966808d/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7d8f6c26245217bd2ad053761201e9f9680f8ce52f0fcd8d0755aeae5b2152", size = 149687 },
{ url = "https://files.pythonhosted.org/packages/a5/42/90c1f7b9341eef50c8a1cb3f098ac43b0508413f33affd762855f67a410e/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:799a7a5e4fb2d5898c60b640fd4981d6a25f1c11790935a44ce38c54e985f828", size = 160014 },
{ url = "https://files.pythonhosted.org/packages/76/be/4d3ee471e8145d12795ab655ece37baed0929462a86e72372fd25859047c/charset_normalizer-3.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:99ae2cffebb06e6c22bdc25801d7b30f503cc87dbd283479e7b606f70aff57ec", size = 154044 },
{ url = "https://files.pythonhosted.org/packages/b0/6f/8f7af07237c34a1defe7defc565a9bc1807762f672c0fde711a4b22bf9c0/charset_normalizer-3.4.4-cp314-cp314-win32.whl", hash = "sha256:f9d332f8c2a2fcbffe1378594431458ddbef721c1769d78e2cbc06280d8155f9", size = 99940 },
{ url = "https://files.pythonhosted.org/packages/4b/51/8ade005e5ca5b0d80fb4aff72a3775b325bdc3d27408c8113811a7cbe640/charset_normalizer-3.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:8a6562c3700cce886c5be75ade4a5db4214fda19fede41d9792d100288d8f94c", size = 107104 },
{ url = "https://files.pythonhosted.org/packages/da/5f/6b8f83a55bb8278772c5ae54a577f3099025f9ade59d0136ac24a0df4bde/charset_normalizer-3.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:de00632ca48df9daf77a2c65a484531649261ec9f25489917f09e455cb09ddb2", size = 100743 },
{ url = "https://files.pythonhosted.org/packages/0a/4c/925909008ed5a988ccbb72dcc897407e5d6d3bd72410d69e051fc0c14647/charset_normalizer-3.4.4-py3-none-any.whl", hash = "sha256:7a32c560861a02ff789ad905a2fe94e3f840803362c84fecf1851cb4cf3dc37f", size = 53402 },
]
[[package]]
name = "click"
version = "8.3.1"
@ -196,6 +262,38 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335 },
]
[[package]]
name = "crossrefapi"
version = "1.7.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "ipython" },
{ name = "requests" },
{ name = "urllib3" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f7/c7/bd020af76301bfede41869b3c18f3efb6fac7dd064371d2a9956bc3306c4/crossrefapi-1.7.0.tar.gz", hash = "sha256:598f27a3cc1bd8d29770de284b0b1315c820de1d70894084945d5265e6fe2dae", size = 13444 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ab/de/58f53b61330e74b8d1401abb448c27027fb66ca14556a120248fc3ff84b0/crossrefapi-1.7.0-py3-none-any.whl", hash = "sha256:577919e26727651a546dfb94781fc0f24b05fe0d05a62f8e49dba87c4bb7f87e", size = 14652 },
]
[[package]]
name = "decorator"
version = "5.2.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/43/fa/6d96a0978d19e17b68d634497769987b16c8f4cd0a7a05048bec693caa6b/decorator-5.2.1.tar.gz", hash = "sha256:65f266143752f734b0a7cc83c46f4618af75b8c5911b00ccb61d0ac9b6da0360", size = 56711 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/4e/8c/f3147f5c4b73e7550fe5f9352eaa956ae838d5c51eb58e7a25b9f3e2643b/decorator-5.2.1-py3-none-any.whl", hash = "sha256:d316bb415a2d9e2d2b3abcc4084c6502fc09240e292cd76a76afc106a1c8e04a", size = 9190 },
]
[[package]]
name = "executing"
version = "2.2.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/cc/28/c14e053b6762b1044f34a13aab6859bbf40456d37d23aa286ac24cfd9a5d/executing-2.2.1.tar.gz", hash = "sha256:3632cc370565f6648cc328b32435bd120a1e4ebb20c77e3fdde9a13cd1e533c4", size = 1129488 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c1/ea/53f2148663b321f21b5a606bd5f191517cf40b7072c0497d3c92c4a13b1e/executing-2.2.1-py2.py3-none-any.whl", hash = "sha256:760643d3452b4d777d295bb167ccc74c64a81df23fb5e08eff250c425a4b2017", size = 28317 },
]
[[package]]
name = "fastapi"
version = "0.116.2"
@ -337,6 +435,26 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484 },
]
[[package]]
name = "ipython"
version = "8.38.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "sys_platform == 'win32'" },
{ name = "decorator" },
{ name = "jedi" },
{ name = "matplotlib-inline" },
{ name = "pexpect", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
{ name = "prompt-toolkit" },
{ name = "pygments" },
{ name = "stack-data" },
{ name = "traitlets" },
]
sdist = { url = "https://files.pythonhosted.org/packages/e5/61/1810830e8b93c72dcd3c0f150c80a00c3deb229562d9423807ec92c3a539/ipython-8.38.0.tar.gz", hash = "sha256:9cfea8c903ce0867cc2f23199ed8545eb741f3a69420bfcf3743ad1cec856d39", size = 5513996 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/9f/df/db59624f4c71b39717c423409950ac3f2c8b2ce4b0aac843112c7fb3f721/ipython-8.38.0-py3-none-any.whl", hash = "sha256:750162629d800ac65bb3b543a14e7a74b0e88063eac9b92124d4b2aa3f6d8e86", size = 831813 },
]
[[package]]
name = "itsdangerous"
version = "2.2.0"
@ -346,6 +464,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl", hash = "sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef", size = 16234 },
]
[[package]]
name = "jedi"
version = "0.19.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "parso" },
]
sdist = { url = "https://files.pythonhosted.org/packages/72/3a/79a912fbd4d8dd6fbb02bf69afd3bb72cf0c729bb3063c6f4498603db17a/jedi-0.19.2.tar.gz", hash = "sha256:4770dc3de41bde3966b02eb84fbcf557fb33cce26ad23da12c742fb50ecb11f0", size = 1231287 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/c0/5a/9cac0c82afec3d09ccd97c8b6502d48f165f9124db81b4bcb90b4af974ee/jedi-0.19.2-py2.py3-none-any.whl", hash = "sha256:a8ef22bde8490f57fe5c7681a3c83cb58874daf72b4784de3cce5b6ef6edb5b9", size = 1572278 },
]
[[package]]
name = "mako"
version = "1.3.10"
@ -421,6 +551,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146 },
]
[[package]]
name = "matplotlib-inline"
version = "0.2.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "traitlets" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c7/74/97e72a36efd4ae2bccb3463284300f8953f199b5ffbc04cbbb0ec78f74b1/matplotlib_inline-0.2.1.tar.gz", hash = "sha256:e1ee949c340d771fc39e241ea75683deb94762c8fa5f2927ec57c83c4dffa9fe", size = 8110 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/af/33/ee4519fa02ed11a94aef9559552f3b17bb863f2ecfe1a35dc7f548cde231/matplotlib_inline-0.2.1-py3-none-any.whl", hash = "sha256:d56ce5156ba6085e00a9d54fead6ed29a9c47e215cd1bba2e976ef39f5710a76", size = 9516 },
]
[[package]]
name = "packaging"
version = "26.0"
@ -430,6 +572,27 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/b7/b9/c538f279a4e237a006a2c98387d081e9eb060d203d8ed34467cc0f0b9b53/packaging-26.0-py3-none-any.whl", hash = "sha256:b36f1fef9334a5588b4166f8bcd26a14e521f2b55e6b9de3aaa80d3ff7a37529", size = 74366 },
]
[[package]]
name = "parso"
version = "0.8.6"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/81/76/a1e769043c0c0c9fe391b702539d594731a4362334cdf4dc25d0c09761e7/parso-0.8.6.tar.gz", hash = "sha256:2b9a0332696df97d454fa67b81618fd69c35a7b90327cbe6ba5c92d2c68a7bfd", size = 401621 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b6/61/fae042894f4296ec49e3f193aff5d7c18440da9e48102c3315e1bc4519a7/parso-0.8.6-py2.py3-none-any.whl", hash = "sha256:2c549f800b70a5c4952197248825584cb00f033b29c692671d3bf08bf380baff", size = 106894 },
]
[[package]]
name = "pexpect"
version = "4.9.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "ptyprocess" },
]
sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523", size = 63772 },
]
[[package]]
name = "pluggy"
version = "1.6.0"
@ -439,6 +602,36 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538 },
]
[[package]]
name = "prompt-toolkit"
version = "3.0.52"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "wcwidth" },
]
sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431 },
]
[[package]]
name = "ptyprocess"
version = "0.7.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/20/e5/16ff212c1e452235a90aeb09066144d0c5a6a8c0834397e03f5224495c4e/ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220", size = 70762 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35", size = 13993 },
]
[[package]]
name = "pure-eval"
version = "0.2.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/cd/05/0a34433a064256a578f1783a10da6df098ceaa4a57bbeaa96a6c0352786b/pure_eval-0.2.3.tar.gz", hash = "sha256:5f4e983f40564c576c7c8635ae88db5956bb2229d7e9237d03b3c0b0190eaf42", size = 19752 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842 },
]
[[package]]
name = "pycparser"
version = "3.0"
@ -631,6 +824,21 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341 },
]
[[package]]
name = "requests"
version = "2.32.5"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "certifi" },
{ name = "charset-normalizer" },
{ name = "idna" },
{ name = "urllib3" },
]
sdist = { url = "https://files.pythonhosted.org/packages/c9/74/b3ff8e6c8446842c3f5c837e9c3dfcfe2018ea6ecef224c710c85ef728f4/requests-2.32.5.tar.gz", hash = "sha256:dbba0bac56e100853db0ea71b82b4dfd5fe2bf6d3754a8893c3af500cec7d7cf", size = 134517 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/1e/db/4254e3eabe8020b458f1a747140d32277ec7a271daf1d235b70dc0b4e6e3/requests-2.32.5-py3-none-any.whl", hash = "sha256:2462f94637a34fd532264295e186976db0f5d453d1cdd31473c85a6a161affb6", size = 64738 },
]
[[package]]
name = "scholarr"
version = "0.1.0"
@ -639,7 +847,9 @@ dependencies = [
{ name = "alembic" },
{ name = "argon2-cffi" },
{ name = "asyncpg" },
{ name = "crossrefapi" },
{ name = "fastapi" },
{ name = "httpx" },
{ name = "itsdangerous" },
{ name = "python-multipart" },
{ name = "sqlalchemy" },
@ -648,7 +858,6 @@ dependencies = [
[package.optional-dependencies]
dev = [
{ name = "httpx" },
{ name = "pytest" },
{ name = "pytest-asyncio" },
]
@ -658,8 +867,9 @@ requires-dist = [
{ name = "alembic", specifier = ">=1.14,<2.0" },
{ name = "argon2-cffi", specifier = ">=25.1,<26.0" },
{ name = "asyncpg", specifier = ">=0.30,<0.31" },
{ name = "crossrefapi", specifier = ">=1.6,<2.0" },
{ name = "fastapi", specifier = ">=0.116,<0.117" },
{ name = "httpx", marker = "extra == 'dev'", specifier = ">=0.28,<0.29" },
{ name = "httpx", specifier = ">=0.28,<0.29" },
{ name = "itsdangerous", specifier = ">=2.2,<3.0" },
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8.3,<9.0" },
{ name = "pytest-asyncio", marker = "extra == 'dev'", specifier = ">=0.25,<0.26" },
@ -711,6 +921,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fc/a1/9c4efa03300926601c19c18582531b45aededfb961ab3c3585f1e24f120b/sqlalchemy-2.0.46-py3-none-any.whl", hash = "sha256:f9c11766e7e7c0a2767dda5acb006a118640c9fc0a4104214b96269bfb78399e", size = 1937882 },
]
[[package]]
name = "stack-data"
version = "0.6.3"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "asttokens" },
{ name = "executing" },
{ name = "pure-eval" },
]
sdist = { url = "https://files.pythonhosted.org/packages/28/e3/55dcc2cfbc3ca9c29519eb6884dd1415ecb53b0e934862d3559ddcb7e20b/stack_data-0.6.3.tar.gz", hash = "sha256:836a778de4fec4dcd1dcd89ed8abff8a221f58308462e1c4aa2a3cf30148f0b9", size = 44707 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl", hash = "sha256:d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695", size = 24521 },
]
[[package]]
name = "starlette"
version = "0.48.0"
@ -724,6 +948,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/be/72/2db2f49247d0a18b4f1bb9a5a39a0162869acf235f3a96418363947b3d46/starlette-0.48.0-py3-none-any.whl", hash = "sha256:0764ca97b097582558ecb498132ed0c7d942f233f365b86ba37770e026510659", size = 73736 },
]
[[package]]
name = "traitlets"
version = "5.14.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/eb/79/72064e6a701c2183016abbbfedaba506d81e30e232a68c9f0d6f6fcd1574/traitlets-5.14.3.tar.gz", hash = "sha256:9ed0579d3502c94b4b3732ac120375cda96f923114522847de4b3bb98b96b6b7", size = 161621 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/00/c0/8f5d070730d7836adc9c9b6408dec68c6ced86b304a9b26a14df072a6e8c/traitlets-5.14.3-py3-none-any.whl", hash = "sha256:b74e89e397b1ed28cc831db7aea759ba6640cb3de13090ca145426688ff1ac4f", size = 85359 },
]
[[package]]
name = "typing-extensions"
version = "4.15.0"
@ -745,6 +978,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611 },
]
[[package]]
name = "urllib3"
version = "2.6.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/c7/24/5f1b3bdffd70275f6661c76461e25f024d5a38a46f04aaca912426a2b1d3/urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed", size = 435556 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/39/08/aaaad47bc4e9dc8c725e68f9d04865dbcb2052843ff09c97b08904852d84/urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4", size = 131584 },
]
[[package]]
name = "uvicorn"
version = "0.34.3"
@ -871,6 +1113,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/e3/bd/fa9bb053192491b3867ba07d2343d9f2252e00811567d30ae8d0f78136fe/watchfiles-1.1.1-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:a916a2932da8f8ab582f242c065f5c81bed3462849ca79ee357dd9551b0e9b01", size = 622112 },
]
[[package]]
name = "wcwidth"
version = "0.6.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/35/a2/8e3becb46433538a38726c948d3399905a4c7cabd0df578ede5dc51f0ec2/wcwidth-0.6.0.tar.gz", hash = "sha256:cdc4e4262d6ef9a1a57e018384cbeb1208d8abbc64176027e2c2455c81313159", size = 159684 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/68/5a/199c59e0a824a3db2b89c5d2dade7ab5f9624dbf6448dc291b46d5ec94d3/wcwidth-0.6.0-py3-none-any.whl", hash = "sha256:1a3a1e510b553315f8e146c54764f4fb6264ffad731b3d78088cdb1478ffbdad", size = 94189 },
]
[[package]]
name = "websockets"
version = "16.0"