Compare commits

...
Sign in to create a new pull request.

40 commits

Author SHA1 Message Date
f501ea4f94 fix(frontend): confirm modal, settings display, race guard, and test coverage
- S1: Replace window.confirm() with AppConfirmModal for delete actions
- S4: Add formatHours helper to avoid ugly decimals in settings hints
- W5: Add generation counter to SettingsAdminPanel to prevent stale
  async results when section changes rapidly
- S7: Remove unused ApiRequestError import from useScholarBulkActions
- S2: Add error-path tests for bulk delete/toggle network failures
- S3: Add CSRF boundary tests for bulk-delete and bulk-toggle endpoints
2026-03-07 18:07:29 +01:00
f8e3098fc3 fix(backend): harden bulk endpoints and redact pdf-queue email for non-admins
- W1: _parse_ids_param now returns 400 on non-integer input instead of 500
- W2: bulk_delete_scholars wraps commit in IntegrityError guard; router
  catches ScholarServiceError → 409
- W3: Move `from typing import Any` to top-level import in application.py
- W4: Redact requested_by_email in PDF queue response for non-admin users
2026-03-07 17:57:37 +01:00
c85fa08b7b fix(frontend): add cooldown tooltips and fix settings tab initial load
Add AppHelpHint tooltips to ScrapeSafetyBadge and ScrapeSafetyBanner
showing rate-limit explanation, cooldown reason, and recommended action.
When no cooldown is active, show a simple "ready" message.

Fix settings tabs not loading on initial navigation by deferring
onMounted load via nextTick, and using flush:'post' on the section
watcher so template refs are attached before load() runs.
2026-03-07 16:50:35 +01:00
39c7eb686c fix(mypy): use CursorResult type annotation for bulk_toggle_scholars 2026-03-07 16:16:55 +01:00
3e933998a0 fix(ci): fix mypy error in bulk_toggle_scholars
Use int(cursor.rowcount) instead of type: ignore comment.
2026-03-07 16:11:25 +01:00
6f3e0eec39 fix(ci): fix ruff format in scholars router
Put each structured_log argument on its own line.
2026-03-07 16:11:25 +01:00
76c4811405 fix(ci): fix import sorting, API contract, and pdf-queue test
- Add missing blank line after imports in application.py (ruff I001)
- Use full path string instead of template literal for export API call
  to avoid false positive in API contract drift check
- Update pdf-queue test to expect 200 for non-admin listing
2026-03-07 16:11:25 +01:00
6742eff773 feat(scholars): add multiselect and bulk actions (delete, toggle, export)
Add Set<number>-based selection state to ScholarsPage with checkboxes
in both table and card views, select-all toggle, and stale-key pruning.

Backend: POST /scholars/bulk-delete, POST /scholars/bulk-toggle, and
GET /scholars/export?ids= filter. Service functions use user-scoped
queries. Structured logging for bulk operations.

Frontend: useScholarBulkActions composable extracts selection and bulk
action logic. AppSelect-based action dropdown with dynamic options.
Delete prompts window.confirm; enable/disable applies immediately.
Export downloads filtered JSON.

Includes integration tests for all bulk endpoints and frontend unit
tests for selection toggle, select-all, and stale pruning.
2026-03-07 16:11:25 +01:00
61ac6f206f fix(test): add Pinia setup and admin user to AdminPdfQueueSection tests
The component uses useAuthStore() which requires an active Pinia
instance, and the "Queue all" button is admin-only (v-if="auth.isAdmin").
2026-03-07 15:51:48 +01:00
afbb25d217 fix(ci): use theme tokens for badges, update pdf-queue test
- Replace raw dark: utility variants with state-* theme tokens
  in ScholarStatusBadges and ScholarSettingsModal
- Update integration test to expect 200 for pdf-queue listing
  (endpoint now uses get_api_current_user, not admin-only)
2026-03-07 15:32:42 +01:00
813da91608 fix(scholars): improve validation, fix delete, add status badges
- Extract scholar ID parsing to dedicated module with robust URL
  handling (trailing slashes, fragments, extra query params, encoded
  characters) and per-token error feedback showing why each invalid
  entry was skipped
- Wrap delete_scholar DB commit in IntegrityError handler so cascade
  failures return a proper API error instead of 500
- Add ScholarStatusBadges component showing Pending/Failed/Partial/OK
  badges on scholar rows based on baseline_completed and last_run_status
- Show Pending badge in ScholarSettingsModal for unscraped scholars
- Surface failed_count and partial_count in dashboard latest run summary
  and recent run list items
- Extend backend validator tests with edge cases (empty, whitespace,
  unicode, URL-as-ID)
- Add integration tests for DELETE 404, DELETE with cascade, and POST
  with corrupted scholar_id
- Add comprehensive frontend tests for parsing logic and component
  behavior
2026-03-07 14:05:39 +01:00
0d955c31dc feat(settings): hours interval, pdf queue for all users, fix users tab loading
- Convert check interval setting from minutes to hours in the UI
- Show PDF queue tab to all authenticated users (not admin-only); requeue
  actions remain admin-only in both backend and frontend
- Fix users tab not loading on first visit by removing the AsyncStateGate
  wrapper that prevented child component refs from being populated before
  onMounted fired

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:53:00 +01:00
a72151cfdc feat(frontend): show run and cancel buttons to all users
Remove the auth.isAdmin guard from the Start check and Cancel check
buttons in the Activity Monitor. Admin-only route links (check history,
diagnostics) are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:53:00 +01:00
JustinZeus
c6c89879c9
Merge pull request #47 from JustinZeus/feat/live-scholar-progress-counter
fix(ingestion,frontend): show live counter immediately on run start
2026-03-03 18:38:36 +01:00
2c591a0ca3 fix(ingestion,frontend): show live counter immediately on run start
Two issues with the initial implementation:
- Counter only appeared after the first SSE event arrived (scholarProgress null
  kept the v-else static template showing until a scholar completed)
- First events were often dropped because the EventSource hadn't subscribed yet

Fix:
- Backend: emit an initial scholar_progress kickoff event (0/0/N) at the
  start of run_scholar_iteration so the frontend receives total early
- Frontend: gate on isLikelyRunning alone (not scholarProgress); use optional
  chaining with fallbacks so "0 / … visited · 0 finished" shows immediately
  when a run starts, before any SSE events arrive

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 18:37:53 +01:00
JustinZeus
5754bffaa0
Merge pull request #46 from JustinZeus/feat/live-scholar-progress-counter
feat(ingestion,frontend): add live scholar progress counter to dashboard
2026-03-03 17:52:46 +01:00
4620978dce feat(ingestion,frontend): add live scholar progress counter to dashboard
Emit scholar_progress SSE events during two-pass ingestion so the
Activity Monitor shows "X / N visited · Y finished · Z new publications"
in real time instead of static post-run counts.

- scholar_processing: add on_progress callback to _run_first_pass and
  _run_depth_pass; build _emit closure in run_scholar_iteration that
  publishes visited/finished/total via run_events; batch-emit scholars
  finished in first pass before depth pass; skip emission on abort/cancel
- run_status store: add scholarProgress state, reset on stream open and
  reset(); subscribe to scholar_progress SSE events with safe parsing
- DashboardPage: show live counter during run, fall back to static text
  when run is not active or no progress received yet

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 17:36:43 +01:00
JustinZeus
cfb1033110
Merge pull request #45 from JustinZeus/bugfix/broken-import
openalex cooldown
2026-03-03 16:17:39 +01:00
f50b609a36 fix(db,frontend): reserve DB connections for API and reduce frontend polling
Background tasks (ingestion, PDF resolution, scheduler) now acquire an
asyncio.Semaphore before checking out a DB connection, guaranteeing at
least N connections remain available for API request handlers. Removes
duplicate polling loop from PublicationsPage, debounces publication
reload on run-status changes, and increases poll interval from 5s to 15s
since SSE handles live discovery.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 19:09:06 +01:00
a490e14126 openalex cooldown 2026-03-02 18:01:42 +01:00
JustinZeus
8ba3b3d41a
Merge pull request #44 from JustinZeus/bugfix/broken-import
added test for new publication detection
2026-03-02 13:59:07 +01:00
4c82fe4b8e added test for new publication detection 2026-03-02 13:58:50 +01:00
JustinZeus
92ad054b97
Merge pull request #43 from JustinZeus/bugfix/broken-import
fix: resolve mypy errors, fix arxiv test session isolation, and add p…
2026-03-02 12:46:35 +01:00
87aa89b58c fix: resolve mypy errors, fix arxiv test session isolation, and add premerge script
- Remove unused type:ignore comment in scholars.py
- Fix 51 mypy errors across test files (Any return types, cast(), protocol stubs)
- Fix arxiv unit test failures caused by session isolation (patch_session_factory fixture)
- Fix integration test rate-limit bleed between search and create
- Add scripts/premerge.sh for local CI verification
- Add docs/website/.docusaurus/ to .gitignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:46:10 +01:00
JustinZeus
acf23a7d6c
Merge pull request #42 from JustinZeus/bugfix/broken-import
fixed import
2026-03-02 00:03:55 +01:00
49cf7034b2 2026-03-02 00:03:27 +01:00
JustinZeus
4d949e8657
Merge pull request #41 from JustinZeus/openalex-size-reduction-scholar-hydration
fix: URL-length-aware OpenAlex batching and eager scholar metadata hy…
2026-03-01 23:44:12 +01:00
d9be57a6c1 fix: URL-length-aware OpenAlex batching and eager scholar metadata hydration
- Replace fixed batch_size=25 with URL-byte-length-aware chunking for
  OpenAlex title.search queries. Long/Unicode titles caused URLs to
  exceed server limits, resulting in 500 errors.
- Always hydrate scholar profile metadata (name, image) on creation,
  even when an initial scrape job is queued, so the UI shows profile
  info immediately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:42:15 +01:00
JustinZeus
e2fe34946c
Merge pull request #40 from JustinZeus/mypi-fix
fix: align test assertion with rollback behavior
2026-03-01 23:18:05 +01:00
e4e9b94cfe fix: align test assertion with rollback behavior
The upsert rolls back the entire transaction on partial failure,
so new_pub_count and link count should be consistent (both 0),
not assuming the first publication survived.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:16:07 +01:00
JustinZeus
76469e2d1e
Merge pull request #39 from JustinZeus/mypi-fix
" "
2026-03-01 23:12:26 +01:00
83e0407491 " " 2026-03-01 23:11:52 +01:00
JustinZeus
9401aa9b69
Merge pull request #38 from JustinZeus/mypi-fix
db tets fix
2026-03-01 23:08:15 +01:00
c7d170a202 db tets fix 2026-03-01 23:07:49 +01:00
JustinZeus
6e17a65b2f
Merge pull request #37 from JustinZeus/mypi-fix
Mypi fix
2026-03-01 23:04:55 +01:00
6fe4814442 " " 2026-03-01 23:04:23 +01:00
0ee89136ff fix: handle optional cookie field for mypy type check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:04:02 +01:00
JustinZeus
84faaad5eb
Merge pull request #36 from JustinZeus/ci-fixes
Ci fixes
2026-03-01 23:02:03 +01:00
f649c07e10 ci fixed 2026-03-01 23:01:06 +01:00
79f3e41471 fix: resolve CI lint and docs build failures
Apply ruff formatting to 3 files and add missing package-lock.json for docs/website.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:00:31 +01:00
60 changed files with 20699 additions and 326 deletions

View file

@ -8,7 +8,23 @@
"Bash(done)",
"Bash(grep -c 'def test_' tests/unit/*.py tests/unit/**/*.py tests/integration/*.py)",
"Bash(sort -t: -k2 -rn)",
"Bash(wc -l frontend/src/pages/*.vue)"
"Bash(wc -l frontend/src/pages/*.vue)",
"Bash(uv run ruff format app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py)",
"Bash(ruff format app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py)",
"Bash(npm install --package-lock-only)",
"Bash(python -m ruff format app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py)",
"Bash(pip show ruff)",
"Bash(pip install ruff)",
"Bash(uvx ruff format app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py)",
"Bash(git add app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py docs/website/package-lock.json)",
"Bash(git push)",
"Bash(uvx ruff format --check app/api/routers/runs.py app/services/arxiv/rate_limit.py tests/integration/test_run_lifecycle_consistency.py)",
"Bash(test -f docs/website/package-lock.json)",
"Bash(gh run list --branch openalex-size-reduction-scholar-hydration --limit 3 --json databaseId,status,conclusion,name,event)",
"Bash(gh run list --limit 10 --json databaseId,status,conclusion,name,event,headBranch)",
"Bash(gh run view 22554585841 --log-failed)",
"Bash(gh run rerun 22554585841 --failed)",
"Bash(bash scripts/premerge.sh)"
],
"additionalDirectories": [
"/home/jv/src/personal/playground/scholar_scraper/scholar-scraper/frontend/src"

View file

@ -28,6 +28,7 @@ DATABASE_POOL_MODE=auto
DATABASE_POOL_SIZE=5
DATABASE_POOL_MAX_OVERFLOW=10
DATABASE_POOL_TIMEOUT_SECONDS=30
DATABASE_RESERVED_API_CONNECTIONS=3
# ------------------------------
# Frontend Dev Overrides

2
.gitignore vendored
View file

@ -18,3 +18,5 @@ frontend/dist/
frontend/.vite/
docs/website/node_modules/
docs/website/build/
docs/website/.docusaurus/
.claude/

View file

@ -5,7 +5,7 @@ import logging
from fastapi import APIRouter, Depends, Path, Query, Request
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.deps import get_api_admin_user
from app.api.deps import get_api_admin_user, get_api_current_user
from app.api.errors import ApiException
from app.api.responses import success_payload
from app.api.schemas import (
@ -57,7 +57,7 @@ def _requested_by_value(*, payload, admin_user: User) -> str:
return from_payload or admin_user.email
def _serialize_pdf_queue_item(item) -> dict[str, object]:
def _serialize_pdf_queue_item(item, *, is_admin: bool = False) -> dict[str, object]:
return {
"publication_id": item.publication_id,
"title": item.title,
@ -69,7 +69,7 @@ def _serialize_pdf_queue_item(item) -> dict[str, object]:
"last_failure_detail": item.last_failure_detail,
"last_source": item.last_source,
"requested_by_user_id": item.requested_by_user_id,
"requested_by_email": item.requested_by_email,
"requested_by_email": item.requested_by_email if is_admin else None,
"queued_at": item.queued_at,
"last_attempt_at": item.last_attempt_at,
"resolved_at": item.resolved_at,
@ -185,7 +185,7 @@ async def get_pdf_queue(
offset: int | None = Query(default=None, ge=0),
status: str | None = Query(default=None),
db_session: AsyncSession = Depends(get_db_session),
admin_user: User = Depends(get_api_admin_user),
current_user: User = Depends(get_api_current_user),
):
normalized_status = (status or "").strip().lower() or None
resolved_page, resolved_limit, resolved_offset = _resolve_pdf_queue_paging(
@ -204,7 +204,7 @@ async def get_pdf_queue(
logger,
"info",
"api.admin.db.pdf_queue_listed",
admin_user_id=int(admin_user.id),
user_id=int(current_user.id),
page=int(resolved_page),
page_size=int(resolved_limit),
offset=int(resolved_offset),
@ -215,7 +215,7 @@ async def get_pdf_queue(
return success_payload(
request,
data={
"items": [_serialize_pdf_queue_item(item) for item in queue_page.items],
"items": [_serialize_pdf_queue_item(item, is_admin=current_user.is_admin) for item in queue_page.items],
**_pdf_queue_page_data(
total_count=queue_page.total_count,
page=resolved_page,

View file

@ -55,7 +55,7 @@ def _apply_scholar_http_settings(payload: AdminScholarHttpSettingsUpdateRequest)
max_length=_MAX_ACCEPT_LANGUAGE_LENGTH,
)
cookie = _normalize_header_value(
payload.cookie,
payload.cookie or "",
field_name="cookie",
max_length=_MAX_COOKIE_LENGTH,
)

View file

@ -56,6 +56,7 @@ def _drop_finished_task(task: asyncio.Task[Any]) -> None:
except Exception:
structured_log(logger, "exception", "runs.background_task_failed")
router = APIRouter(prefix="/runs", tags=["api-runs"])
ACTIVE_RUN_STATUSES = {RunStatus.RUNNING, RunStatus.RESOLVING}
@ -220,11 +221,11 @@ def _spawn_background_execution(
user_settings: Any,
idempotency_key: str | None,
) -> None:
from app.db.session import get_session_factory
from app.db.background_session import background_session
task = asyncio.create_task(
ingest_service.execute_run(
session_factory=get_session_factory(),
session_factory=background_session,
run_id=run.id,
user_id=current_user.id,
scholars=scholars,

View file

@ -23,6 +23,9 @@ from app.api.schemas import (
DataImportEnvelope,
DataImportRequest,
MessageEnvelope,
ScholarBulkCountEnvelope,
ScholarBulkIdsRequest,
ScholarBulkToggleRequest,
ScholarCreateRequest,
ScholarEnvelope,
ScholarImageUrlUpdateRequest,
@ -42,6 +45,22 @@ logger = logging.getLogger(__name__)
router = APIRouter(prefix="/scholars", tags=["api-scholars"])
def _parse_ids_param(ids: str | None) -> list[int] | None:
if not ids:
return None
parts = [p.strip() for p in ids.split(",") if p.strip()]
if not parts:
return None
try:
return [int(p) for p in parts]
except ValueError as exc:
raise ApiException(
status_code=400,
code="invalid_ids_param",
message="The 'ids' parameter must be a comma-separated list of integers.",
) from exc
@router.get(
"",
response_model=ScholarsListEnvelope,
@ -69,16 +88,81 @@ async def list_scholars(
)
async def export_scholars_and_publications(
request: Request,
ids: str | None = Query(None, description="Comma-separated scholar profile IDs to export"),
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
scholar_profile_ids = _parse_ids_param(ids)
data = await import_export_service.export_user_data(
db_session,
user_id=current_user.id,
scholar_profile_ids=scholar_profile_ids,
)
return success_payload(request, data=data)
@router.post(
"/bulk-delete",
response_model=ScholarBulkCountEnvelope,
)
async def bulk_delete_scholars(
payload: ScholarBulkIdsRequest,
request: Request,
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
try:
deleted_count = await scholar_service.bulk_delete_scholars(
db_session,
user_id=current_user.id,
scholar_profile_ids=payload.scholar_profile_ids,
upload_dir=settings.scholar_image_upload_dir,
)
except scholar_service.ScholarServiceError as exc:
raise ApiException(
status_code=409,
code="scholar_bulk_delete_failed",
message=str(exc),
) from exc
structured_log(
logger,
"info",
"scholars.bulk_delete",
user_id=current_user.id,
requested_ids=payload.scholar_profile_ids,
deleted_count=deleted_count,
)
return success_payload(request, data={"deleted_count": deleted_count, "updated_count": 0})
@router.post(
"/bulk-toggle",
response_model=ScholarBulkCountEnvelope,
)
async def bulk_toggle_scholars(
payload: ScholarBulkToggleRequest,
request: Request,
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
updated_count = await scholar_service.bulk_toggle_scholars(
db_session,
user_id=current_user.id,
scholar_profile_ids=payload.scholar_profile_ids,
is_enabled=payload.is_enabled,
)
structured_log(
logger,
"info",
"scholars.bulk_toggle",
user_id=current_user.id,
requested_ids=payload.scholar_profile_ids,
is_enabled=payload.is_enabled,
updated_count=updated_count,
)
return success_payload(request, data={"deleted_count": 0, "updated_count": updated_count})
@router.post(
"/import",
response_model=DataImportEnvelope,
@ -144,18 +228,17 @@ async def create_scholar(
message=str(exc),
) from exc
structured_log(logger, "info", "api.scholars.created", user_id=current_user.id, scholar_profile_id=created.id)
did_queue_initial_scrape = await enqueue_initial_scrape_job_for_scholar(
await enqueue_initial_scrape_job_for_scholar(
db_session,
profile=created,
user_id=current_user.id,
)
if not did_queue_initial_scrape:
created = await hydrate_scholar_metadata_if_needed(
db_session,
profile=created,
source=source,
user_id=current_user.id,
)
created = await hydrate_scholar_metadata_if_needed(
db_session,
profile=created,
source=source,
user_id=current_user.id,
)
return success_payload(
request,
@ -262,11 +345,18 @@ async def delete_scholar(
code="scholar_not_found",
message="Scholar not found.",
)
await scholar_service.delete_scholar(
db_session,
profile=profile,
upload_dir=settings.scholar_image_upload_dir,
)
try:
await scholar_service.delete_scholar(
db_session,
profile=profile,
upload_dir=settings.scholar_image_upload_dir,
)
except scholar_service.ScholarServiceError as exc:
raise ApiException(
status_code=409,
code="scholar_delete_failed",
message=str(exc),
) from exc
structured_log(
logger, "info", "api.scholars.deleted", user_id=current_user.id, scholar_profile_id=scholar_profile_id
)

View file

@ -2,7 +2,7 @@ from __future__ import annotations
from datetime import datetime
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, model_validator
from app.api.schemas.common import ApiMeta
@ -126,13 +126,49 @@ class DataExportEnvelope(BaseModel):
model_config = ConfigDict(extra="forbid")
class ScholarBulkIdsRequest(BaseModel):
scholar_profile_ids: list[int] = Field(..., min_length=1, max_length=500)
model_config = ConfigDict(extra="forbid")
class ScholarBulkToggleRequest(BaseModel):
scholar_profile_ids: list[int] = Field(..., min_length=1, max_length=500)
is_enabled: bool
model_config = ConfigDict(extra="forbid")
class ScholarBulkCountData(BaseModel):
deleted_count: int = 0
updated_count: int = 0
model_config = ConfigDict(extra="forbid")
class ScholarBulkCountEnvelope(BaseModel):
data: ScholarBulkCountData
meta: ApiMeta
model_config = ConfigDict(extra="forbid")
class DataImportRequest(BaseModel):
schema_version: int | None = None
exported_at: str | None = None
scholars: list[ScholarExportItemData] = Field(default_factory=list)
publications: list[PublicationExportItemData] = Field(default_factory=list)
model_config = ConfigDict(extra="forbid")
@model_validator(mode="before")
@classmethod
def unwrap_export_envelope(cls, values: dict) -> dict:
"""Accept the full export envelope format (with data/meta wrapper)."""
if isinstance(values, dict) and "data" in values and isinstance(values["data"], dict):
return values["data"]
return values
class DataImportResultData(BaseModel):
scholars_created: int

View file

@ -0,0 +1,57 @@
"""Semaphore-gated DB sessions for background tasks.
Background tasks (ingestion, PDF resolution, scheduler) acquire a semaphore
permit before checking out a database connection. The semaphore cap is
``pool_size + max_overflow - reserved_for_api``, which guarantees that at
least *reserved_for_api* connections remain available for API request
handlers at all times.
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.session import get_session_factory
from app.logging_utils import structured_log
from app.settings import settings
logger = logging.getLogger(__name__)
_semaphore: asyncio.Semaphore | None = None
def _build_semaphore() -> asyncio.Semaphore:
pool_capacity = max(1, settings.database_pool_size) + max(0, settings.database_pool_max_overflow)
reserved = max(0, settings.database_reserved_api_connections)
limit = max(1, pool_capacity - reserved)
structured_log(
logger,
"info",
"db.background_semaphore_initialized",
pool_capacity=pool_capacity,
reserved_for_api=reserved,
background_limit=limit,
)
return asyncio.Semaphore(limit)
def get_background_semaphore() -> asyncio.Semaphore:
global _semaphore
if _semaphore is None:
_semaphore = _build_semaphore()
return _semaphore
@asynccontextmanager
async def background_session() -> AsyncIterator[AsyncSession]:
"""Yield a DB session after acquiring the background semaphore."""
semaphore = get_background_semaphore()
async with semaphore:
session_factory = get_session_factory()
async with session_factory() as session:
yield session

View file

@ -99,9 +99,7 @@ async def _run_serialized_fetch(
"arxiv.request_completed",
status_code=int(response.status_code),
wait_seconds=wait_seconds,
cooldown_remaining_seconds=_cooldown_remaining_seconds(
cooldown_until_value, now_utc=datetime.now(UTC)
),
cooldown_remaining_seconds=_cooldown_remaining_seconds(cooldown_until_value, now_utc=datetime.now(UTC)),
source_path=source_path,
)
return response, hit_rate_limit

View file

@ -37,6 +37,40 @@ def _sanitize_titles(publications: list) -> list[str]:
return titles
# OpenAlex (and most HTTP servers) struggle with very long query strings.
# Non-ASCII characters expand significantly when percent-encoded in URLs,
# so a fixed batch count is unreliable. Instead we cap the combined
# byte-length of the pipe-joined title filter value to stay well within
# safe URL limits (~4 KiB of filter text ≈ ~6 KiB when percent-encoded).
_MAX_TITLE_FILTER_BYTES = 4_000
def _chunk_titles_by_url_length(
titles: list[str],
max_bytes: int = _MAX_TITLE_FILTER_BYTES,
) -> list[list[str]]:
"""Split *titles* into chunks whose pipe-joined UTF-8 size stays under *max_bytes*."""
chunks: list[list[str]] = []
current_chunk: list[str] = []
current_bytes = 0
for title in titles:
title_bytes = len(title.encode("utf-8"))
# Account for the pipe separator between titles
separator = 3 if current_chunk else 0 # len("|".encode) but url-encoded as %7C = 3
if current_chunk and current_bytes + separator + title_bytes > max_bytes:
chunks.append(current_chunk)
current_chunk = [title]
current_bytes = title_bytes
else:
current_chunk.append(title)
current_bytes += separator + title_bytes
if current_chunk:
chunks.append(current_chunk)
return chunks
class EnrichmentRunner:
"""Post-run OpenAlex enrichment logic.
@ -156,21 +190,34 @@ class EnrichmentRunner:
resolved_key = openalex_api_key or settings.openalex_api_key
client = OpenAlexClient(api_key=resolved_key, mailto=settings.crossref_api_mailto)
batch_size = 25
now = datetime.now(UTC)
arxiv_lookup_allowed = True
for i in range(0, len(publications), batch_size):
all_titles = _sanitize_titles(publications)
title_chunks = _chunk_titles_by_url_length(all_titles)
# Build a mapping from sanitized title → publications for batch association
title_to_pubs: dict[str, list[Publication]] = {}
for p in publications:
raw = getattr(p, "title_raw", None) or getattr(p, "title", None)
if not raw or not raw.strip():
continue
safe = " ".join(re.sub(r"[^\w\s]", " ", raw).split())
if safe:
title_to_pubs.setdefault(safe, []).append(p)
for title_chunk in title_chunks:
if await self.run_is_canceled(db_session, run_id=run_id):
structured_log(logger, "info", "ingestion.enrichment_aborted", run_id=run_id)
return
batch = publications[i : i + batch_size]
titles = _sanitize_titles(batch)
if not titles:
batch = []
for t in title_chunk:
batch.extend(title_to_pubs.get(t, []))
if not batch:
continue
try:
openalex_works = await client.get_works_by_filter(
{"title.search": "|".join(titles)}, limit=batch_size * 3
{"title.search": "|".join(title_chunk)}, limit=len(title_chunk) * 3
)
except OpenAlexBudgetExhaustedError:
structured_log(logger, "warning", "ingestion.openalex_budget_exhausted", run_id=run_id)
@ -289,19 +336,37 @@ class EnrichmentRunner:
client = OpenAlexClient(api_key=settings.openalex_api_key, mailto=settings.crossref_api_mailto)
batch_size = 25
all_titles = _sanitize_titles(publications)
title_chunks = _chunk_titles_by_url_length(all_titles)
# Build title → publication mapping for batch association
title_to_pubs: dict[str, list[PublicationCandidate]] = {}
for p in publications:
raw = getattr(p, "title", None)
if not raw or not raw.strip():
continue
safe = " ".join(re.sub(r"[^\w\s]", " ", raw).split())
if safe:
title_to_pubs.setdefault(safe, []).append(p)
# Collect publications that had no sanitizable title — pass through as-is
pubs_with_titles = set()
for titles in title_to_pubs.values():
for p in titles:
pubs_with_titles.add(id(p))
enriched: list[PublicationCandidate] = []
for i in range(0, len(publications), batch_size):
batch = publications[i : i + batch_size]
titles = _sanitize_titles(batch)
if not titles:
enriched.extend(batch)
for title_chunk in title_chunks:
batch = []
for t in title_chunk:
batch.extend(title_to_pubs.get(t, []))
if not batch:
continue
try:
openalex_works = await client.get_works_by_filter(
{"title.search": "|".join(titles)}, limit=batch_size * 3
{"title.search": "|".join(title_chunk)}, limit=len(title_chunk) * 3
)
except Exception as e:
structured_log(
@ -335,4 +400,10 @@ class EnrichmentRunner:
enriched.append(new_p)
else:
enriched.append(p)
# Append publications that had no sanitizable title (pass-through)
for p in publications:
if id(p) not in pubs_with_titles:
enriched.append(p)
return enriched

View file

@ -5,8 +5,8 @@ from datetime import UTC, datetime
from sqlalchemy import select
from app.db.background_session import background_session
from app.db.models import QueueItemStatus, RunTriggerType, ScholarProfile, UserSetting
from app.db.session import get_session_factory
from app.logging_utils import structured_log
from app.services.ingestion import queue as queue_service
from app.services.ingestion.application import (
@ -57,8 +57,7 @@ class QueueJobRunner:
async def drain_continuation_queue(self) -> None:
now = datetime.now(UTC)
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
jobs = await queue_service.list_due_jobs(
session,
now=now,
@ -73,8 +72,7 @@ class QueueJobRunner:
) -> bool:
if job.attempt_count < self._continuation_max_attempts:
return False
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
dropped = await queue_service.mark_dropped(
session,
job_id=job.id,
@ -98,8 +96,7 @@ class QueueJobRunner:
self,
job: queue_service.ContinuationQueueJob,
) -> bool:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
queue_item = await queue_service.mark_retrying(session, job_id=job.id)
await session.commit()
if queue_item is None:
@ -110,8 +107,7 @@ class QueueJobRunner:
self,
job: queue_service.ContinuationQueueJob,
) -> bool:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
scholar_result = await session.execute(
select(ScholarProfile.id).where(
ScholarProfile.user_id == job.user_id,
@ -140,8 +136,7 @@ class QueueJobRunner:
return False
async def _reschedule_queue_job_lock_active(self, job: queue_service.ContinuationQueueJob) -> None:
session_factory = get_session_factory()
async with session_factory() as recovery_session:
async with background_session() as recovery_session:
await queue_service.reschedule_job(
recovery_session,
job_id=job.id,
@ -167,8 +162,7 @@ class QueueJobRunner:
self._tick_seconds,
int(exc.safety_state.get("cooldown_remaining_seconds") or 0),
)
session_factory = get_session_factory()
async with session_factory() as recovery_session:
async with background_session() as recovery_session:
await queue_service.reschedule_job(
recovery_session,
job_id=job.id,
@ -193,8 +187,7 @@ class QueueJobRunner:
*,
exc: Exception,
) -> None:
session_factory = get_session_factory()
async with session_factory() as recovery_session:
async with background_session() as recovery_session:
queue_item = await queue_service.increment_attempt_count(recovery_session, job_id=job.id)
if queue_item is None:
await recovery_session.commit()
@ -238,8 +231,7 @@ class QueueJobRunner:
)
async def _load_request_delay_for_user(self, user_id: int, *, floor: int) -> int:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
result = await session.execute(
select(UserSetting.request_delay_seconds).where(UserSetting.user_id == user_id)
)
@ -253,8 +245,7 @@ class QueueJobRunner:
request_delay_floor: int,
):
request_delay_seconds = await self._load_request_delay_for_user(job.user_id, floor=request_delay_floor)
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
ingestion = ScholarIngestionService(source=self._source)
try:
return await ingestion.run_for_user(
@ -366,8 +357,7 @@ class QueueJobRunner:
)
async def _finalize_queue_job_after_run(self, job: queue_service.ContinuationQueueJob, run_summary) -> None:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
if int(run_summary.failed_count) <= 0:
await self._finalize_successful_queue_job(session, job, run_summary)
else:

View file

@ -8,13 +8,13 @@ from typing import Any
from sqlalchemy import select
from app.db.background_session import background_session
from app.db.models import (
CrawlRun,
RunTriggerType,
User,
UserSetting,
)
from app.db.session import get_session_factory
from app.logging_utils import structured_log
from app.services.ingestion.application import (
RunAlreadyInProgressError,
@ -148,8 +148,7 @@ class SchedulerService:
await self._run_candidate(candidate)
async def _load_candidate_rows(self) -> list[Any]:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
result = await session.execute(
select(
UserSetting.user_id,
@ -204,8 +203,7 @@ class SchedulerService:
return candidates
async def _is_due(self, candidate: _AutoRunCandidate, *, now: datetime) -> bool:
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
result = await session.execute(
select(CrawlRun.start_dt)
.where(
@ -227,8 +225,7 @@ class SchedulerService:
*,
candidate: _AutoRunCandidate,
):
session_factory = get_session_factory()
async with session_factory() as session:
async with background_session() as session:
ingestion = ScholarIngestionService(source=self._source)
try:
return await ingestion.run_for_user(
@ -289,9 +286,12 @@ class SchedulerService:
async def _drain_pdf_queue(self) -> None:
from app.services.publications.pdf_queue import drain_ready_jobs
from app.services.publications.pdf_queue_resolution import is_budget_cooldown_active
session_factory = get_session_factory()
async with session_factory() as session:
if is_budget_cooldown_active():
return
async with background_session() as session:
try:
processed = await drain_ready_jobs(
session,

View file

@ -3,6 +3,7 @@ from __future__ import annotations
import asyncio
import logging
import random
from collections.abc import Callable, Coroutine
from datetime import UTC, datetime
from typing import Any
@ -274,6 +275,7 @@ async def _run_first_pass(
request_delay_seconds: int,
queue_delay_seconds: int,
progress: RunProgress,
on_progress: Callable[[int, int], Coroutine[Any, Any, None]] | None = None,
) -> dict[int, int]:
first_pass_cstarts: dict[int, int] = {}
for index, scholar in enumerate(scholars):
@ -310,6 +312,8 @@ async def _run_first_pass(
scholars_remaining=len(scholars) - index - 1,
)
return first_pass_cstarts
if on_progress is not None:
await on_progress(1, 0)
resume_cstart = outcome.result_entry.get("continuation_cstart")
if resume_cstart is not None and int(resume_cstart) > start_cstart:
first_pass_cstarts[int(scholar.id)] = int(resume_cstart)
@ -330,6 +334,7 @@ async def _run_depth_pass(
auto_queue_continuations: bool,
queue_delay_seconds: int,
progress: RunProgress,
on_progress: Callable[[int, int], Coroutine[Any, Any, None]] | None = None,
) -> None:
for index, scholar in enumerate(scholars):
resume_cstart = first_pass_cstarts.get(int(scholar.id))
@ -367,6 +372,8 @@ async def _run_depth_pass(
scholars_remaining=len(scholars) - index - 1,
)
break
if on_progress is not None:
await on_progress(0, 1)
async def run_scholar_iteration(
@ -387,6 +394,8 @@ async def run_scholar_iteration(
auto_queue_continuations: bool,
queue_delay_seconds: int,
) -> RunProgress:
from app.services.runs.events import run_events
progress = RunProgress()
scholar_kwargs: dict[str, Any] = {
"request_delay_seconds": request_delay_seconds,
@ -396,6 +405,22 @@ async def run_scholar_iteration(
"rate_limit_backoff_seconds": rate_limit_backoff_seconds,
"page_size": page_size,
}
visited = 0
finished = 0
total = len(scholars)
async def _emit(v: int = 0, f: int = 0) -> None:
nonlocal visited, finished
visited += v
finished += f
await run_events.publish(
run.id,
"scholar_progress",
{"visited": visited, "finished": finished, "total": total},
)
await _emit()
first_pass_cstarts = await _run_first_pass(
db_session,
scholars=scholars,
@ -407,8 +432,12 @@ async def run_scholar_iteration(
request_delay_seconds=request_delay_seconds,
queue_delay_seconds=queue_delay_seconds,
progress=progress,
on_progress=_emit,
)
remaining_max = max(max_pages_per_scholar - 1, 0)
scholars_finished_in_first_pass = len(scholars) - len(first_pass_cstarts)
if scholars_finished_in_first_pass > 0:
await _emit(f=scholars_finished_in_first_pass)
if remaining_max <= 0:
return progress
await _run_depth_pass(
@ -424,5 +453,6 @@ async def run_scholar_iteration(
auto_queue_continuations=auto_queue_continuations,
queue_delay_seconds=queue_delay_seconds,
progress=progress,
on_progress=_emit,
)
return progress

View file

@ -56,11 +56,14 @@ async def export_user_data(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_ids: list[int] | None = None,
) -> dict[str, Any]:
scholars_result = await db_session.execute(
select(ScholarProfile).where(ScholarProfile.user_id == user_id).order_by(ScholarProfile.id.asc())
)
publication_result = await db_session.execute(
scholar_query = select(ScholarProfile).where(ScholarProfile.user_id == user_id)
if scholar_profile_ids:
scholar_query = scholar_query.where(ScholarProfile.id.in_(scholar_profile_ids))
scholars_result = await db_session.execute(scholar_query.order_by(ScholarProfile.id.asc()))
pub_query = (
select(
ScholarProfile.scholar_id,
Publication.cluster_id,
@ -77,8 +80,13 @@ async def export_user_data(
.join(ScholarPublication, ScholarPublication.scholar_profile_id == ScholarProfile.id)
.join(Publication, Publication.id == ScholarPublication.publication_id)
.where(ScholarProfile.user_id == user_id)
.order_by(ScholarPublication.created_at.desc(), Publication.id.desc())
)
if scholar_profile_ids:
pub_query = pub_query.where(ScholarProfile.id.in_(scholar_profile_ids))
publication_result = await db_session.execute(
pub_query.order_by(ScholarPublication.created_at.desc(), Publication.id.desc())
)
scholars = [_serialize_export_scholar(profile) for profile in scholars_result.scalars().all()]
publications = [_serialize_export_publication(row) for row in publication_result.all()]
return {

View file

@ -2,9 +2,10 @@ from __future__ import annotations
import asyncio
import logging
from datetime import UTC, datetime, timedelta
from app.db.background_session import background_session
from app.db.models import Publication, PublicationPdfJob
from app.db.session import get_session_factory
from app.logging_utils import structured_log
from app.services.publication_identifiers import application as identifier_service
from app.services.publications.pdf_queue_common import (
@ -29,8 +30,20 @@ PDF_EVENT_ATTEMPT_STARTED = "attempt_started"
PDF_EVENT_RESOLVED = "resolved"
PDF_EVENT_FAILED = "failed"
_BUDGET_COOLDOWN_MINUTES = 15
logger = logging.getLogger(__name__)
_scheduled_tasks: set[asyncio.Task[None]] = set()
_budget_cooldown_until: datetime | None = None
def is_budget_cooldown_active() -> bool:
return _budget_cooldown_until is not None and datetime.now(UTC) < _budget_cooldown_until
def _enter_budget_cooldown() -> None:
global _budget_cooldown_until
_budget_cooldown_until = datetime.now(UTC) + timedelta(minutes=_BUDGET_COOLDOWN_MINUTES)
async def _mark_attempt_started(
@ -38,8 +51,7 @@ async def _mark_attempt_started(
publication_id: int,
user_id: int,
) -> None:
session_factory = get_session_factory()
async with session_factory() as db_session:
async with background_session() as db_session:
job = await db_session.get(PublicationPdfJob, publication_id)
if job is None:
job = queued_job(publication_id=publication_id, user_id=user_id)
@ -125,8 +137,7 @@ async def _persist_outcome(
user_id: int,
outcome: OaResolutionOutcome,
) -> None:
session_factory = get_session_factory()
async with session_factory() as db_session:
async with background_session() as db_session:
publication = await db_session.get(Publication, publication_id)
job = await db_session.get(PublicationPdfJob, publication_id)
if publication is None or job is None:
@ -207,8 +218,7 @@ async def _run_resolution_task(
openalex_api_key: str | None = None
try:
session_factory = get_session_factory()
async with session_factory() as key_session:
async with background_session() as key_session:
user_settings = await user_settings_service.get_or_create_settings(key_session, user_id=user_id)
openalex_api_key = getattr(user_settings, "openalex_api_key", None) or settings.openalex_api_key
except Exception:
@ -233,11 +243,13 @@ async def _run_resolution_task(
detail="arXiv temporarily disabled for remaining batch after rate limit",
)
except OpenAlexBudgetExhaustedError:
_enter_budget_cooldown()
structured_log(
logger,
"warning",
"pdf_queue.budget_exhausted",
detail="Stopping PDF resolution batch — OpenAlex daily budget exhausted",
cooldown_minutes=_BUDGET_COOLDOWN_MINUTES,
)
break
except Exception:

View file

@ -1,6 +1,7 @@
from __future__ import annotations
import os
from typing import Any
from uuid import uuid4
from sqlalchemy.exc import IntegrityError
@ -10,7 +11,11 @@ from app.db.models import ScholarProfile
from app.services.scholar.parser import ScholarParserError, parse_profile_page
from app.services.scholar.source import ScholarSource
from app.services.scholars.author_search import search_author_candidates
from app.services.scholars.constants import ALLOWED_IMAGE_UPLOAD_CONTENT_TYPES
from app.services.scholars.constants import (
ALLOWED_IMAGE_UPLOAD_CONTENT_TYPES,
SEARCH_COOLDOWN_REASON,
SEARCH_DISABLED_REASON,
)
from app.services.scholars.exceptions import ScholarServiceError
from app.services.scholars.uploads import (
_ensure_upload_root,
@ -23,8 +28,66 @@ from app.services.scholars.validators import (
validate_scholar_id,
)
async def bulk_delete_scholars(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_ids: list[int],
upload_dir: str | None = None,
) -> int:
from sqlalchemy import select
result = await db_session.execute(
select(ScholarProfile).where(
ScholarProfile.id.in_(scholar_profile_ids),
ScholarProfile.user_id == user_id,
)
)
profiles = list(result.scalars().all())
if not profiles:
return 0
if upload_dir:
upload_root = _ensure_upload_root(upload_dir, create=True)
for profile in profiles:
_safe_remove_upload(upload_root, profile.profile_image_upload_path)
for profile in profiles:
await db_session.delete(profile)
try:
await db_session.commit()
except IntegrityError as exc:
await db_session.rollback()
raise ScholarServiceError("Unable to bulk-delete scholars due to a database constraint.") from exc
return len(profiles)
async def bulk_toggle_scholars(
db_session: AsyncSession,
*,
user_id: int,
scholar_profile_ids: list[int],
is_enabled: bool,
) -> int:
from sqlalchemy import CursorResult, update
cursor: CursorResult[Any] = await db_session.execute( # type: ignore[assignment]
update(ScholarProfile)
.where(
ScholarProfile.id.in_(scholar_profile_ids),
ScholarProfile.user_id == user_id,
)
.values(is_enabled=is_enabled)
)
await db_session.commit()
return int(cursor.rowcount or 0)
__all__ = [
"SEARCH_COOLDOWN_REASON",
"SEARCH_DISABLED_REASON",
"ScholarServiceError",
"bulk_delete_scholars",
"bulk_toggle_scholars",
"clear_profile_image_customization",
"create_scholar_for_user",
"delete_scholar",
@ -119,7 +182,11 @@ async def delete_scholar(
_safe_remove_upload(upload_root, profile.profile_image_upload_path)
await db_session.delete(profile)
await db_session.commit()
try:
await db_session.commit()
except IntegrityError as exc:
await db_session.rollback()
raise ScholarServiceError("Unable to delete scholar due to a database constraint.") from exc
async def hydrate_profile_metadata(

View file

@ -275,6 +275,8 @@ class Settings:
crossref_max_lookups_per_request: int = _env_int("CROSSREF_MAX_LOOKUPS_PER_REQUEST", 8)
openalex_api_key: str | None = os.getenv("OPENALEX_API_KEY")
database_reserved_api_connections: int = _env_int("DATABASE_RESERVED_API_CONNECTIONS", 3)
crossref_api_token: str | None = os.getenv("CROSSREF_API_TOKEN")
crossref_api_mailto: str | None = os.getenv("CROSSREF_API_MAILTO")

18461
docs/website/package-lock.json generated Normal file

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,61 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import ScrapeSafetyBadge from "./ScrapeSafetyBadge.vue";
import { createDefaultSafetyState, type ScrapeSafetyState } from "@/features/safety";
function buildState(overrides: Partial<ScrapeSafetyState> = {}): ScrapeSafetyState {
return { ...createDefaultSafetyState(), ...overrides };
}
describe("ScrapeSafetyBadge", () => {
it("shows ready tooltip when cooldown is inactive", () => {
const wrapper = mount(ScrapeSafetyBadge, {
props: { state: buildState({ cooldown_active: false }) },
});
expect(wrapper.text()).toContain("Safety ready");
const hint = wrapper.findComponent({ name: "AppHelpHint" });
expect(hint.exists()).toBe(true);
expect(hint.props("text")).toContain("No active cooldown");
});
it("shows cooldown tooltip with reason and action when active", () => {
const wrapper = mount(ScrapeSafetyBadge, {
props: {
state: buildState({
cooldown_active: true,
cooldown_reason: "blocked_failure_threshold_exceeded",
cooldown_reason_label: "Too many blocked requests",
recommended_action: "Wait for cooldown to expire",
cooldown_remaining_seconds: 120,
}),
},
});
expect(wrapper.text()).toContain("Safety cooldown");
const hint = wrapper.findComponent({ name: "AppHelpHint" });
expect(hint.exists()).toBe(true);
const text = hint.props("text") as string;
expect(text).toContain("Google Scholar rate-limits");
expect(text).toContain("Too many blocked requests");
expect(text).toContain("Wait for cooldown to expire");
});
it("shows cooldown tooltip without optional fields", () => {
const wrapper = mount(ScrapeSafetyBadge, {
props: {
state: buildState({
cooldown_active: true,
cooldown_reason: "some_reason",
cooldown_reason_label: null,
recommended_action: null,
cooldown_remaining_seconds: 60,
}),
},
});
const hint = wrapper.findComponent({ name: "AppHelpHint" });
const text = hint.props("text") as string;
expect(text).toContain("Google Scholar rate-limits");
expect(text).not.toContain("Why:");
expect(text).not.toContain("Action:");
});
});

View file

@ -1,6 +1,7 @@
<script setup lang="ts">
import { computed } from "vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import { type ScrapeSafetyState } from "@/features/safety";
const props = defineProps<{
@ -18,10 +19,30 @@ const toneClass = computed(() => {
}
return "border-state-warning-border bg-state-warning-bg text-state-warning-text";
});
const READY_TOOLTIP = "No active cooldown. Scraping can proceed normally.";
const RATE_LIMIT_INTRO =
"Google Scholar rate-limits automated requests. The cooldown pauses scraping to avoid your IP being blocked.";
const tooltipText = computed(() => {
if (!props.state.cooldown_active) return READY_TOOLTIP;
const parts = [RATE_LIMIT_INTRO];
if (props.state.cooldown_reason_label) {
parts.push(`Why: ${props.state.cooldown_reason_label}`);
}
if (props.state.recommended_action) {
parts.push(`Action: ${props.state.recommended_action}`);
}
return parts.join(" \u2014 ");
});
</script>
<template>
<span class="inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-semibold" :class="toneClass">
{{ label }}
<span class="inline-flex items-center gap-1">
<span class="inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-semibold" :class="toneClass">
{{ label }}
</span>
<AppHelpHint :text="tooltipText" />
</span>
</template>

View file

@ -2,6 +2,7 @@
import { computed, onBeforeUnmount, onMounted, ref } from "vue";
import AppAlert from "@/components/ui/AppAlert.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import {
formatCooldownCountdown,
type ScrapeSafetyState,
@ -79,6 +80,10 @@ const actionText = computed(() => {
return props.safetyState.recommended_action;
});
const bannerHintText = computed(() => {
return "Google Scholar rate-limits automated requests. The cooldown pauses scraping to avoid your IP being blocked.";
});
onMounted(() => {
timer = setInterval(() => {
now.value = Date.now();
@ -95,7 +100,12 @@ onBeforeUnmount(() => {
<template>
<AppAlert v-if="isVisible" :tone="tone">
<template #title>{{ title }}</template>
<template #title>
<span class="inline-flex items-center gap-1">
{{ title }}
<AppHelpHint :text="bannerHintText" />
</span>
</template>
<p>{{ detailText }}</p>
<p v-if="actionText" class="text-secondary">{{ actionText }}</p>
</AppAlert>

View file

@ -0,0 +1,28 @@
<script setup lang="ts">
import AppButton from "@/components/ui/AppButton.vue";
import AppModal from "@/components/ui/AppModal.vue";
withDefaults(
defineProps<{
open: boolean;
title: string;
message: string;
confirmLabel?: string;
cancelLabel?: string;
variant?: "danger" | "default";
}>(),
{ confirmLabel: "Confirm", cancelLabel: "Cancel", variant: "default" },
);
const emit = defineEmits<{ confirm: []; cancel: [] }>();
</script>
<template>
<AppModal :open="open" :title="title" @close="emit('cancel')">
<p class="mb-6 text-sm text-secondary">{{ message }}</p>
<div class="flex justify-end gap-2">
<AppButton variant="secondary" @click="emit('cancel')">{{ cancelLabel }}</AppButton>
<AppButton :variant="variant === 'danger' ? 'danger' : 'primary'" @click="emit('confirm')">{{ confirmLabel }}</AppButton>
</div>
</AppModal>
</template>

View file

@ -277,12 +277,16 @@ export function usePublicationData() {
// --- Run-triggered refresh watcher ---
let previousRunStatusKey: string | null = null;
watch(
() => runStatus.latestRun,
async (nextRun, previousRun) => {
const nextRunId = nextRun && (nextRun.status === "running" || nextRun.status === "resolving") ? nextRun.id : null;
const previousRunId = previousRun && (previousRun.status === "running" || previousRun.status === "resolving") ? previousRun.id : null;
if (nextRunId === null || nextRunId === previousRunId) return;
async (nextRun) => {
const nextStatus = nextRun ? `${nextRun.id}:${nextRun.status}` : null;
if (nextStatus === previousRunStatusKey) return;
previousRunStatusKey = nextStatus;
const isActive = nextRun && (nextRun.status === "running" || nextRun.status === "resolving");
if (!isActive) return;
resetPageAndSnapshot();
await loadPublications();
},

View file

@ -2,10 +2,144 @@
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import ScholarBatchAdd from "./ScholarBatchAdd.vue";
import {
parseScholarIds,
parseScholarTokens,
extractScholarIdFromUrl,
validateTokenAsId,
} from "./scholar-batch-parsing";
const defaultProps = { saving: false, loading: false };
describe("ScholarBatchAdd", () => {
describe("parseScholarIds (unit)", () => {
it("parses a single bare ID", () => {
expect(parseScholarIds("A-UbBTPM15wL")).toEqual(["A-UbBTPM15wL"]);
});
it("parses multiple IDs separated by commas", () => {
expect(parseScholarIds("A-UbBTPM15wL, B-UbBTPM15wL")).toEqual(["A-UbBTPM15wL", "B-UbBTPM15wL"]);
});
it("deduplicates IDs", () => {
expect(parseScholarIds("A-UbBTPM15wL\nA-UbBTPM15wL")).toEqual(["A-UbBTPM15wL"]);
});
it("extracts ID from standard Google Scholar URL", () => {
expect(parseScholarIds("https://scholar.google.com/citations?hl=en&user=A-UbBTPM15wL")).toEqual(["A-UbBTPM15wL"]);
});
it("extracts ID from URL with trailing slash", () => {
expect(parseScholarIds("https://scholar.google.com/citations?user=A-UbBTPM15wL/")).toEqual(["A-UbBTPM15wL"]);
});
it("extracts ID from URL with fragment", () => {
expect(parseScholarIds("https://scholar.google.com/citations?user=A-UbBTPM15wL#section")).toEqual(["A-UbBTPM15wL"]);
});
it("extracts ID from URL with extra query params", () => {
expect(parseScholarIds("https://scholar.google.com/citations?hl=en&user=A-UbBTPM15wL&view_op=list_works&sortby=pubdate")).toEqual(["A-UbBTPM15wL"]);
});
it("handles URL-encoded characters in non-ID parts", () => {
expect(parseScholarIds("https://scholar.google.com/citations?hl=en&user=A-UbBTPM15wL&label=%E4%B8%AD%E6%96%87")).toEqual(["A-UbBTPM15wL"]);
});
it("rejects malformed IDs (too short)", () => {
expect(parseScholarIds("ABC123")).toEqual([]);
});
it("rejects malformed IDs (too long)", () => {
expect(parseScholarIds("ABCDEF1234567")).toEqual([]);
});
it("rejects IDs with special characters", () => {
expect(parseScholarIds("ABCDEF12345!")).toEqual([]);
});
it("rejects IDs with embedded whitespace", () => {
expect(parseScholarIds("ABC DEF12345")).toEqual([]);
});
it("handles mixed valid and invalid tokens", () => {
expect(parseScholarIds("A-UbBTPM15wL, invalid, B-UbBTPM15wL")).toEqual(["A-UbBTPM15wL", "B-UbBTPM15wL"]);
});
it("returns empty array for empty string", () => {
expect(parseScholarIds("")).toEqual([]);
});
it("returns empty array for whitespace only", () => {
expect(parseScholarIds(" ")).toEqual([]);
});
});
describe("extractScholarIdFromUrl", () => {
it("returns null for non-URL strings", () => {
expect(extractScholarIdFromUrl("not-a-url")).toBeNull();
});
it("returns null for URL without user param", () => {
expect(extractScholarIdFromUrl("https://scholar.google.com/citations?hl=en")).toBeNull();
});
it("extracts from URL with trailing slashes", () => {
expect(extractScholarIdFromUrl("https://scholar.google.com/citations?user=A-UbBTPM15wL///")).toBe("A-UbBTPM15wL");
});
it("strips fragment before parsing", () => {
expect(extractScholarIdFromUrl("https://scholar.google.com/citations?user=A-UbBTPM15wL#foo")).toBe("A-UbBTPM15wL");
});
it("returns null when user param is invalid length", () => {
expect(extractScholarIdFromUrl("https://scholar.google.com/citations?user=short")).toBeNull();
});
});
describe("validateTokenAsId", () => {
it("returns null for valid 12-char ID", () => {
expect(validateTokenAsId("ABCDEF123456")).toBeNull();
});
it("detects wrong length", () => {
expect(validateTokenAsId("ABC")).toContain("12 characters");
});
it("detects invalid characters", () => {
expect(validateTokenAsId("ABCDEF12345!")).toContain("invalid characters");
});
it("detects whitespace", () => {
expect(validateTokenAsId("ABC DEF12345")).toContain("whitespace");
});
it("detects empty input", () => {
expect(validateTokenAsId("")).toContain("empty");
});
});
describe("parseScholarTokens (per-token errors)", () => {
it("marks invalid tokens with error messages", () => {
const tokens = parseScholarTokens("A-UbBTPM15wL, short, B-UbBTPM15wL");
expect(tokens).toHaveLength(3);
expect(tokens[0].id).toBe("A-UbBTPM15wL");
expect(tokens[0].error).toBeNull();
expect(tokens[1].id).toBeNull();
expect(tokens[1].error).toContain("12 characters");
expect(tokens[2].id).toBe("B-UbBTPM15wL");
});
it("marks duplicate tokens", () => {
const tokens = parseScholarTokens("A-UbBTPM15wL, A-UbBTPM15wL");
expect(tokens[1].error).toBe("duplicate");
});
it("marks bad URL tokens", () => {
const tokens = parseScholarTokens("https://scholar.google.com/citations?hl=en");
expect(tokens[0].error).toContain("could not extract");
});
});
describe("ScholarBatchAdd (component)", () => {
it("renders the heading and input", () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
expect(wrapper.text()).toContain("Add Scholar Profiles");
@ -73,4 +207,18 @@ describe("ScholarBatchAdd", () => {
const wrapper = mount(ScholarBatchAdd, { props: { ...defaultProps, saving: true } });
expect(wrapper.text()).toContain("Adding...");
});
it("shows validation errors for invalid tokens", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("A-UbBTPM15wL, bad!");
const errors = wrapper.find("[data-testid='validation-errors']");
expect(errors.exists()).toBe(true);
});
it("shows skipped count alongside valid count", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("A-UbBTPM15wL, tooshort");
expect(wrapper.text()).toContain("1 valid");
expect(wrapper.text()).toContain("1 skipped");
});
});

View file

@ -3,6 +3,7 @@ import { computed, ref } from "vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import { parseScholarTokens } from "./scholar-batch-parsing";
defineProps<{
saving: boolean;
@ -15,38 +16,13 @@ const emit = defineEmits<{
const scholarBatchInput = ref("");
const SCHOLAR_ID_PATTERN = /^[a-zA-Z0-9_-]{12}$/;
const URL_USER_PARAM_PATTERN = /(?:\?|&)user=([a-zA-Z0-9_-]{12})(?:&|#|$)/i;
function parseScholarIds(raw: string): string[] {
const ordered: string[] = [];
const seen = new Set<string>();
const tokens = raw.split(/[\s,;]+/).map((v) => v.trim()).filter((v) => v.length > 0);
for (const token of tokens) {
let candidate: string | null = null;
if (SCHOLAR_ID_PATTERN.test(token)) candidate = token;
if (!candidate) {
const directParamMatch = token.match(URL_USER_PARAM_PATTERN);
if (directParamMatch) candidate = directParamMatch[1];
}
if (!candidate && token.includes("scholar.google")) {
try {
const parsed = new URL(token);
const userParam = parsed.searchParams.get("user");
if (userParam && SCHOLAR_ID_PATTERN.test(userParam)) candidate = userParam;
} catch (_error) { /* Ignore non-URL tokens. */ }
}
if (!candidate || seen.has(candidate)) continue;
seen.add(candidate);
ordered.push(candidate);
}
return ordered;
}
const parsedBatchCount = computed(() => parseScholarIds(scholarBatchInput.value).length);
const parsedTokens = computed(() => parseScholarTokens(scholarBatchInput.value));
const validIds = computed(() => parsedTokens.value.filter((t) => t.id !== null));
const invalidTokens = computed(() => parsedTokens.value.filter((t) => t.id === null));
const parsedBatchCount = computed(() => validIds.value.length);
function onSubmit(): void {
const ids = parseScholarIds(scholarBatchInput.value);
const ids = validIds.value.map((t) => t.id!);
if (ids.length > 0) {
emit("add-scholars", ids);
scholarBatchInput.value = "";
@ -76,11 +52,33 @@ function onSubmit(): void {
/>
</label>
<div class="flex flex-wrap items-center justify-between gap-2">
<div v-if="parsedTokens.length > 0" class="space-y-1">
<p class="text-xs text-secondary">
Parsed IDs: <strong class="text-ink-primary">{{ parsedBatchCount }}</strong>
<strong class="text-ink-primary">{{ parsedBatchCount }}</strong> valid ID{{ parsedBatchCount === 1 ? "" : "s" }}
<template v-if="invalidTokens.length > 0">
· <span class="text-state-warning-text">{{ invalidTokens.length }} skipped</span>
</template>
</p>
<AppButton type="submit" :disabled="saving || loading">
<ul v-if="invalidTokens.length > 0" class="space-y-0.5" data-testid="validation-errors">
<li
v-for="item in invalidTokens.slice(0, 5)"
:key="item.index"
class="text-xs text-state-warning-text"
>
#{{ item.index }}: {{ item.error }}
<code class="ml-1 break-all text-ink-muted">{{ item.raw.length > 60 ? item.raw.slice(0, 57) + "..." : item.raw }}</code>
</li>
<li v-if="invalidTokens.length > 5" class="text-xs text-state-warning-text">
+{{ invalidTokens.length - 5 }} more skipped
</li>
</ul>
</div>
<div v-else class="text-xs text-secondary">
Parsed IDs: <strong class="text-ink-primary">0</strong>
</div>
<div class="flex flex-wrap items-center justify-end gap-2">
<AppButton type="submit" :disabled="saving || loading || parsedBatchCount === 0">
{{ saving ? "Adding..." : "Add scholars" }}
</AppButton>
</div>

View file

@ -47,7 +47,14 @@ function scholarPublicationsRoute(profile: ScholarProfile): { name: string; quer
:image-url="scholar.profile_image_url"
/>
<div class="min-w-0 space-y-1">
<p class="truncate text-sm font-semibold text-ink-primary">{{ scholarLabel(scholar) }}</p>
<p class="truncate text-sm font-semibold text-ink-primary">
{{ scholarLabel(scholar) }}
<span
v-if="!scholar.baseline_completed"
class="ml-1.5 inline-flex items-center rounded-full border border-state-warning-border bg-state-warning-bg px-1.5 py-0.5 text-[10px] font-medium text-state-warning-text"
title="This scholar hasn't been scraped yet. Publications will appear after the next run."
>Pending</span>
</p>
<p class="text-xs text-secondary">ID: <code>{{ scholar.scholar_id }}</code></p>
<div class="flex flex-wrap items-center gap-3">
<RouterLink :to="scholarPublicationsRoute(scholar)" class="link-inline text-xs">

View file

@ -0,0 +1,29 @@
<script setup lang="ts">
defineProps<{
baselineCompleted: boolean;
lastRunStatus: string | null;
}>();
</script>
<template>
<span
v-if="!baselineCompleted"
class="ml-1.5 inline-flex items-center rounded-full border border-state-warning-border bg-state-warning-bg px-1.5 py-0.5 text-[10px] font-medium text-state-warning-text"
title="This scholar hasn't been scraped yet. Publications will appear after the next run."
>Pending</span>
<span
v-if="lastRunStatus === 'failed'"
class="ml-1.5 inline-flex items-center rounded-full border border-state-danger-border bg-state-danger-bg px-1.5 py-0.5 text-[10px] font-medium text-state-danger-text"
title="The last scrape run for this scholar failed."
>Failed</span>
<span
v-else-if="lastRunStatus === 'partial'"
class="ml-1.5 inline-flex items-center rounded-full border border-state-warning-border bg-state-warning-bg px-1.5 py-0.5 text-[10px] font-medium text-state-warning-text"
title="The last scrape run for this scholar completed with partial results."
>Partial</span>
<span
v-else-if="lastRunStatus === 'success'"
class="ml-1.5 inline-flex items-center rounded-full border border-state-success-border bg-state-success-bg px-1.5 py-0.5 text-[10px] font-medium text-state-success-text"
title="The last scrape run for this scholar completed successfully."
>OK</span>
</template>

View file

@ -0,0 +1,76 @@
const SCHOLAR_ID_PATTERN = /^[a-zA-Z0-9_-]{12}$/;
export interface ParsedToken {
index: number;
raw: string;
id: string | null;
error: string | null;
}
export function extractScholarIdFromUrl(token: string): string | null {
const cleaned = token.replace(/\/+$/, "").replace(/#.*$/, "");
try {
const parsed = new URL(cleaned);
const userParam = parsed.searchParams.get("user");
if (!userParam) return null;
const decoded = decodeURIComponent(userParam).trim();
if (SCHOLAR_ID_PATTERN.test(decoded)) return decoded;
return null;
} catch {
return null;
}
}
export function validateTokenAsId(token: string): string | null {
const trimmed = token.trim();
if (!trimmed) return "empty input";
if (/\s/.test(trimmed)) return "contains whitespace";
if (trimmed.length !== 12) return `must be 12 characters (got ${trimmed.length})`;
if (!/^[a-zA-Z0-9_-]+$/.test(trimmed)) {
return "contains invalid characters (only a-z, A-Z, 0-9, _ and - allowed)";
}
return null;
}
export function parseScholarTokens(raw: string): ParsedToken[] {
const tokens = raw.split(/[\s,;]+/).map((v) => v.trim()).filter((v) => v.length > 0);
const results: ParsedToken[] = [];
const seen = new Set<string>();
for (let i = 0; i < tokens.length; i++) {
const token = tokens[i];
if (SCHOLAR_ID_PATTERN.test(token)) {
if (seen.has(token)) {
results.push({ index: i + 1, raw: token, id: null, error: "duplicate" });
continue;
}
seen.add(token);
results.push({ index: i + 1, raw: token, id: token, error: null });
continue;
}
if (token.includes("scholar.google") || token.startsWith("http")) {
const extracted = extractScholarIdFromUrl(token);
if (extracted) {
if (seen.has(extracted)) {
results.push({ index: i + 1, raw: token, id: null, error: "duplicate" });
continue;
}
seen.add(extracted);
results.push({ index: i + 1, raw: token, id: extracted, error: null });
continue;
}
results.push({ index: i + 1, raw: token, id: null, error: "could not extract scholar ID from URL" });
continue;
}
const reason = validateTokenAsId(token);
results.push({ index: i + 1, raw: token, id: null, error: reason ?? "invalid scholar ID" });
}
return results;
}
export function parseScholarIds(raw: string): string[] {
return parseScholarTokens(raw).filter((t) => t.id !== null).map((t) => t.id!);
}

View file

@ -0,0 +1,148 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi } from "vitest";
import { nextTick, ref } from "vue";
import { useScholarBulkActions, type ScholarBulkAction } from "./useScholarBulkActions";
import type { ScholarProfile } from "@/features/scholars";
vi.mock("@/features/scholars", () => ({
bulkDeleteScholars: vi.fn(),
bulkToggleScholars: vi.fn(),
exportScholarData: vi.fn(),
}));
function makeProfile(id: number, name: string): ScholarProfile {
return {
id,
scholar_id: `scholar_${id}`,
display_name: name,
profile_image_url: null,
profile_image_source: "none",
is_enabled: true,
baseline_completed: false,
last_run_dt: null,
last_run_status: null,
};
}
function setup(profiles: ScholarProfile[] = []) {
const visibleScholars = ref(profiles);
const callbacks = {
clearMessages: vi.fn(),
assignError: vi.fn(),
setSuccess: vi.fn(),
reloadScholars: vi.fn(async () => {}),
};
const bulk = useScholarBulkActions(visibleScholars, callbacks);
return { visibleScholars, callbacks, bulk };
}
describe("useScholarBulkActions", () => {
it("starts with empty selection", () => {
const { bulk } = setup([makeProfile(1, "Alice")]);
expect(bulk.selectedIds.value.size).toBe(0);
expect(bulk.hasSelection.value).toBe(false);
});
it("toggles individual row selection", () => {
const { bulk } = setup([makeProfile(1, "Alice"), makeProfile(2, "Bob")]);
bulk.onToggleRow(1, { target: { checked: true } } as unknown as Event);
expect(bulk.selectedIds.value.has(1)).toBe(true);
expect(bulk.selectedCount.value).toBe(1);
bulk.onToggleRow(1, { target: { checked: false } } as unknown as Event);
expect(bulk.selectedIds.value.has(1)).toBe(false);
expect(bulk.selectedCount.value).toBe(0);
});
it("toggles all visible scholars", () => {
const { bulk } = setup([makeProfile(1, "Alice"), makeProfile(2, "Bob")]);
bulk.onToggleAll({ target: { checked: true } } as unknown as Event);
expect(bulk.selectedIds.value.size).toBe(2);
expect(bulk.allVisibleSelected.value).toBe(true);
bulk.onToggleAll({ target: { checked: false } } as unknown as Event);
expect(bulk.selectedIds.value.size).toBe(0);
});
it("prunes stale selections when list changes", async () => {
const { visibleScholars, bulk } = setup([makeProfile(1, "Alice"), makeProfile(2, "Bob")]);
bulk.onToggleAll({ target: { checked: true } } as unknown as Event);
expect(bulk.selectedIds.value.size).toBe(2);
// Remove Bob from the visible list
visibleScholars.value = [makeProfile(1, "Alice")];
await nextTick();
expect(bulk.selectedIds.value.size).toBe(1);
expect(bulk.selectedIds.value.has(1)).toBe(true);
expect(bulk.selectedIds.value.has(2)).toBe(false);
});
it("shows correct bulk action options with and without selection", () => {
const { bulk } = setup([makeProfile(1, "Alice")]);
// Without selection
expect(bulk.bulkActionOptions.value.length).toBe(1);
expect(bulk.bulkActionOptions.value[0].value).toBe("select_all");
// With selection
bulk.onToggleRow(1, { target: { checked: true } } as unknown as Event);
expect(bulk.bulkActionOptions.value.length).toBe(5);
const values = bulk.bulkActionOptions.value.map((o) => o.value);
expect(values).toContain("delete_selected");
expect(values).toContain("enable_selected");
expect(values).toContain("disable_selected");
expect(values).toContain("export_selected");
expect(values).toContain("clear_selection");
});
it("select all action selects all visible", async () => {
const { bulk } = setup([makeProfile(1, "Alice"), makeProfile(2, "Bob")]);
bulk.bulkAction.value = "select_all" as ScholarBulkAction;
await bulk.onApplyBulkAction();
expect(bulk.selectedIds.value.size).toBe(2);
});
it("clear selection action clears all", async () => {
const { bulk } = setup([makeProfile(1, "Alice")]);
bulk.onToggleRow(1, { target: { checked: true } } as unknown as Event);
bulk.bulkAction.value = "clear_selection" as ScholarBulkAction;
await bulk.onApplyBulkAction();
expect(bulk.selectedIds.value.size).toBe(0);
});
it("bulk delete calls assignError on network failure", async () => {
const { bulkDeleteScholars } = await import("@/features/scholars");
vi.mocked(bulkDeleteScholars).mockRejectedValueOnce(new Error("Network error"));
const { bulk, callbacks } = setup([makeProfile(1, "Alice")]);
bulk.onToggleRow(1, { target: { checked: true } } as unknown as Event);
bulk.bulkAction.value = "delete_selected" as ScholarBulkAction;
await bulk.onApplyBulkAction();
// Trigger the confirm callback
expect(bulk.confirmState.value.open).toBe(true);
await bulk.confirmState.value.onConfirm();
expect(callbacks.assignError).toHaveBeenCalledWith(
expect.any(Error),
"Unable to bulk delete scholars.",
);
expect(bulk.bulkBusy.value).toBe(false);
});
it("bulk toggle calls assignError on network failure", async () => {
const { bulkToggleScholars } = await import("@/features/scholars");
vi.mocked(bulkToggleScholars).mockRejectedValueOnce(new Error("Network error"));
const { bulk, callbacks } = setup([makeProfile(1, "Alice")]);
bulk.onToggleRow(1, { target: { checked: true } } as unknown as Event);
bulk.bulkAction.value = "enable_selected" as ScholarBulkAction;
await bulk.onApplyBulkAction();
expect(callbacks.assignError).toHaveBeenCalledWith(
expect.any(Error),
"Unable to bulk enable scholars.",
);
expect(bulk.bulkBusy.value).toBe(false);
});
});

View file

@ -0,0 +1,221 @@
import { computed, ref, watch, type Ref } from "vue";
import {
bulkDeleteScholars,
bulkToggleScholars,
exportScholarData,
type ScholarProfile,
} from "@/features/scholars";
export type ScholarBulkAction =
| "delete_selected"
| "enable_selected"
| "disable_selected"
| "export_selected"
| "clear_selection"
| "select_all";
export interface ScholarBulkActionOption {
value: ScholarBulkAction;
label: string;
}
export interface ConfirmState {
open: boolean;
title: string;
message: string;
variant: "danger" | "default";
onConfirm: () => void;
}
export interface BulkActionCallbacks {
clearMessages: () => void;
assignError: (error: unknown, fallback: string) => void;
setSuccess: (msg: string) => void;
reloadScholars: () => Promise<void>;
}
export function useScholarBulkActions(
visibleScholars: Ref<ScholarProfile[]>,
callbacks: BulkActionCallbacks,
) {
const selectedIds = ref<Set<number>>(new Set());
const bulkAction = ref<ScholarBulkAction>("select_all");
const bulkBusy = ref(false);
const confirmState = ref<ConfirmState>({ open: false, title: "", message: "", variant: "default", onConfirm: () => {} });
const selectedCount = computed(() => selectedIds.value.size);
const hasSelection = computed(() => selectedCount.value > 0);
const allVisibleSelected = computed(() => {
if (visibleScholars.value.length === 0) return false;
for (const item of visibleScholars.value) {
if (!selectedIds.value.has(item.id)) return false;
}
return true;
});
const bulkActionOptions = computed<ScholarBulkActionOption[]>(() => {
if (!hasSelection.value) return [{ value: "select_all", label: "Select all" }];
const n = selectedCount.value;
return [
{ value: "delete_selected", label: `Delete selected (${n})` },
{ value: "enable_selected", label: `Enable selected (${n})` },
{ value: "disable_selected", label: `Disable selected (${n})` },
{ value: "export_selected", label: `Export selected (${n})` },
{ value: "clear_selection", label: "Clear selection" },
];
});
const bulkApplyLabel = computed(() => {
if (bulkBusy.value) return "Applying...";
if (bulkAction.value === "select_all") return "Select";
if (bulkAction.value === "clear_selection") return "Clear";
return "Apply";
});
const bulkApplyDisabled = computed(() => {
if (bulkBusy.value) return true;
if (bulkAction.value === "select_all") return visibleScholars.value.length === 0;
return selectedCount.value === 0;
});
// Prune stale selections when visible list changes
watch(visibleScholars, (items) => {
const validIds = new Set(items.map((item) => item.id));
const next = new Set<number>();
for (const id of selectedIds.value) {
if (validIds.has(id)) next.add(id);
}
if (next.size !== selectedIds.value.size) selectedIds.value = next;
});
// Reset bulk action dropdown when selection state changes
watch(hasSelection, (has) => {
const validValues = new Set(bulkActionOptions.value.map((o) => o.value));
if (validValues.has(bulkAction.value)) return;
bulkAction.value = has ? "delete_selected" : "select_all";
});
function onToggleAll(event: Event): void {
const checked = (event.target as HTMLInputElement).checked;
const next = new Set(selectedIds.value);
for (const item of visibleScholars.value) {
if (checked) { next.add(item.id); } else { next.delete(item.id); }
}
selectedIds.value = next;
}
function onToggleRow(id: number, event: Event): void {
const checked = (event.target as HTMLInputElement).checked;
const next = new Set(selectedIds.value);
if (checked) { next.add(id); } else { next.delete(id); }
selectedIds.value = next;
}
function downloadJsonFile(filename: string, payload: unknown): void {
const blob = new Blob([JSON.stringify(payload, null, 2)], { type: "application/json" });
const url = URL.createObjectURL(blob);
const anchor = document.createElement("a");
anchor.href = url;
anchor.download = filename;
anchor.click();
URL.revokeObjectURL(url);
}
async function onApplyBulkAction(): Promise<void> {
if (bulkApplyDisabled.value) return;
if (bulkAction.value === "select_all") {
selectedIds.value = new Set(visibleScholars.value.map((item) => item.id));
return;
}
if (bulkAction.value === "clear_selection") { selectedIds.value = new Set(); return; }
if (bulkAction.value === "delete_selected") { await onBulkDelete(); return; }
if (bulkAction.value === "enable_selected") { await onBulkToggle(true); return; }
if (bulkAction.value === "disable_selected") { await onBulkToggle(false); return; }
if (bulkAction.value === "export_selected") { await onBulkExport(); return; }
}
function dismissConfirm(): void {
confirmState.value = { ...confirmState.value, open: false };
}
function requestConfirm(title: string, message: string, variant: "danger" | "default", onConfirm: () => void): void {
confirmState.value = { open: true, title, message, variant, onConfirm };
}
function onBulkDelete(): void {
const ids = [...selectedIds.value];
requestConfirm(
`Delete ${ids.length} scholar(s)?`,
"This removes all linked publications and queue data. This action cannot be undone.",
"danger",
async () => {
dismissConfirm();
bulkBusy.value = true;
callbacks.clearMessages();
try {
const result = await bulkDeleteScholars(ids);
callbacks.setSuccess(`${result.deleted_count} scholar(s) deleted.`);
selectedIds.value = new Set();
await callbacks.reloadScholars();
} catch (error) {
callbacks.assignError(error, "Unable to bulk delete scholars.");
} finally {
bulkBusy.value = false;
}
},
);
}
async function onBulkToggle(isEnabled: boolean): Promise<void> {
bulkBusy.value = true;
callbacks.clearMessages();
try {
const ids = [...selectedIds.value];
const result = await bulkToggleScholars(ids, isEnabled);
const verb = isEnabled ? "enabled" : "disabled";
callbacks.setSuccess(`${result.updated_count} scholar(s) ${verb}.`);
selectedIds.value = new Set();
await callbacks.reloadScholars();
} catch (error) {
callbacks.assignError(error, `Unable to bulk ${isEnabled ? "enable" : "disable"} scholars.`);
} finally {
bulkBusy.value = false;
}
}
async function onBulkExport(): Promise<void> {
bulkBusy.value = true;
callbacks.clearMessages();
try {
const ids = [...selectedIds.value];
const payload = await exportScholarData(ids);
const dateSlug = payload.exported_at.slice(0, 10) || "unknown-date";
downloadJsonFile(`scholarr-export-${dateSlug}.json`, payload);
callbacks.setSuccess("Export complete.");
} catch (error) {
callbacks.assignError(error, "Unable to export selected scholars.");
} finally {
bulkBusy.value = false;
}
}
return {
selectedIds,
selectedCount,
hasSelection,
allVisibleSelected,
bulkAction,
bulkBusy,
bulkActionOptions,
bulkApplyLabel,
bulkApplyDisabled,
confirmState,
dismissConfirm,
requestConfirm,
onToggleAll,
onToggleRow,
onApplyBulkAction,
};
}

View file

@ -160,8 +160,30 @@ export async function clearScholarImage(
return response.data;
}
export async function exportScholarData(): Promise<DataExportPayload> {
const response = await apiRequest<DataExportPayload>("/scholars/export", {
export interface BulkCountResult {
deleted_count: number;
updated_count: number;
}
export async function bulkDeleteScholars(scholarProfileIds: number[]): Promise<BulkCountResult> {
const response = await apiRequest<BulkCountResult>("/scholars/bulk-delete", {
method: "POST",
body: { scholar_profile_ids: scholarProfileIds },
});
return response.data;
}
export async function bulkToggleScholars(scholarProfileIds: number[], isEnabled: boolean): Promise<BulkCountResult> {
const response = await apiRequest<BulkCountResult>("/scholars/bulk-toggle", {
method: "POST",
body: { scholar_profile_ids: scholarProfileIds, is_enabled: isEnabled },
});
return response.data;
}
export async function exportScholarData(ids?: number[]): Promise<DataExportPayload> {
const path = ids && ids.length > 0 ? `/scholars/export?ids=${ids.join(",")}` : "/scholars/export";
const response = await apiRequest<DataExportPayload>(path, {
method: "GET",
});
return response.data;

View file

@ -0,0 +1,94 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import { nextTick } from "vue";
import SettingsAdminPanel from "./SettingsAdminPanel.vue";
vi.mock("@/features/admin_users", () => ({
listAdminUsers: vi.fn().mockResolvedValue([]),
createAdminUser: vi.fn(),
setAdminUserActive: vi.fn(),
resetAdminUserPassword: vi.fn(),
}));
vi.mock("@/features/admin_dbops", () => ({
getAdminDbIntegrityReport: vi.fn().mockResolvedValue({
status: "ok",
warnings: [],
failures: [],
checked_at: null,
checks: [],
}),
listAdminPdfQueue: vi.fn().mockResolvedValue({
items: [],
total_count: 0,
has_next: false,
has_prev: false,
page: 1,
page_size: 50,
}),
requeueAdminPdfLookup: vi.fn(),
requeueAllAdminPdfLookups: vi.fn(),
}));
vi.mock("@/stores/auth", () => ({
useAuthStore: () => ({ isAdmin: true }),
}));
vi.mock("@/features/admin_repairs", () => ({
listAdminRepairTasks: vi.fn().mockResolvedValue([]),
runAdminRepairTask: vi.fn(),
}));
import { listAdminUsers } from "@/features/admin_users";
import { getAdminDbIntegrityReport, listAdminPdfQueue } from "@/features/admin_dbops";
const mockedListUsers = vi.mocked(listAdminUsers);
const mockedGetReport = vi.mocked(getAdminDbIntegrityReport);
const mockedListPdfQueue = vi.mocked(listAdminPdfQueue);
describe("SettingsAdminPanel", () => {
beforeEach(() => {
mockedListUsers.mockClear();
mockedGetReport.mockClear();
mockedListPdfQueue.mockClear();
});
it("calls load on users section after nextTick when section=users", async () => {
mount(SettingsAdminPanel, { props: { section: "users" } });
await nextTick();
await nextTick();
await flushPromises();
expect(mockedListUsers).toHaveBeenCalled();
});
it("calls load on integrity section after nextTick when section=integrity", async () => {
mount(SettingsAdminPanel, { props: { section: "integrity" } });
await nextTick();
await nextTick();
await flushPromises();
expect(mockedGetReport).toHaveBeenCalled();
});
it("calls load on new section when section prop changes", async () => {
const wrapper = mount(SettingsAdminPanel, { props: { section: "users" } });
await nextTick();
await nextTick();
await flushPromises();
mockedListUsers.mockClear();
mockedGetReport.mockClear();
await wrapper.setProps({ section: "integrity" });
await nextTick();
await flushPromises();
expect(mockedGetReport).toHaveBeenCalled();
});
it("calls load on pdf section when section=pdf", async () => {
mount(SettingsAdminPanel, { props: { section: "pdf" } });
await nextTick();
await nextTick();
await flushPromises();
expect(mockedListPdfQueue).toHaveBeenCalled();
});
});

View file

@ -1,7 +1,7 @@
<script setup lang="ts">
import { onMounted, ref, watch } from "vue";
import { nextTick, onMounted, watch } from "vue";
import { ref } from "vue";
import AsyncStateGate from "@/components/patterns/AsyncStateGate.vue";
import RequestStateAlerts from "@/components/patterns/RequestStateAlerts.vue";
import AdminIntegritySection from "@/features/settings/components/AdminIntegritySection.vue";
import AdminPdfQueueSection from "@/features/settings/components/AdminPdfQueueSection.vue";
@ -18,7 +18,6 @@ const props = defineProps<{
section: "users" | "integrity" | "repairs" | "pdf";
}>();
const loading = ref(true);
const { errorMessage, errorRequestId, successMessage, clearAlerts, assignError } = useRequestState();
const usersRef = ref<InstanceType<typeof AdminUsersSection> | null>(null);
@ -26,39 +25,33 @@ const integrityRef = ref<InstanceType<typeof AdminIntegritySection> | null>(null
const repairsRef = ref<InstanceType<typeof AdminRepairsSection> | null>(null);
const pdfQueueRef = ref<InstanceType<typeof AdminPdfQueueSection> | null>(null);
async function refreshForSection(): Promise<void> {
if (props.section === SECTION_USERS && usersRef.value) {
await usersRef.value.load();
return;
}
if (props.section === SECTION_INTEGRITY && integrityRef.value) {
await integrityRef.value.load();
return;
}
if (props.section === SECTION_REPAIRS) {
await usersRef.value?.load();
await repairsRef.value?.load();
return;
}
if (props.section === SECTION_PDF && pdfQueueRef.value) {
await pdfQueueRef.value.load();
}
}
let loadGeneration = 0;
async function loadSection(): Promise<void> {
loading.value = true;
const gen = ++loadGeneration;
clearAlerts();
try {
await refreshForSection();
if (props.section === SECTION_USERS && usersRef.value) {
await usersRef.value.load();
} else if (props.section === SECTION_INTEGRITY && integrityRef.value) {
await integrityRef.value.load();
} else if (props.section === SECTION_REPAIRS) {
await usersRef.value?.load();
if (gen !== loadGeneration) return;
await repairsRef.value?.load();
} else if (props.section === SECTION_PDF && pdfQueueRef.value) {
await pdfQueueRef.value.load();
}
} catch (error) {
if (gen !== loadGeneration) return;
assignError(error, "Unable to load admin data.");
} finally {
loading.value = false;
}
}
onMounted(loadSection);
watch(() => props.section, loadSection);
onMounted(() => {
nextTick(loadSection);
});
watch(() => props.section, loadSection, { flush: "post" });
</script>
<template>
@ -72,11 +65,9 @@ watch(() => props.section, loadSection);
@dismiss-success="successMessage = null"
/>
<AsyncStateGate :loading="loading" :loading-lines="10" :show-empty="false">
<AdminUsersSection v-if="props.section === SECTION_USERS || props.section === SECTION_REPAIRS" v-show="props.section === SECTION_USERS" ref="usersRef" />
<AdminIntegritySection v-if="props.section === SECTION_INTEGRITY" ref="integrityRef" />
<AdminRepairsSection v-if="props.section === SECTION_REPAIRS" ref="repairsRef" :users="usersRef?.users ?? []" />
<AdminPdfQueueSection v-if="props.section === SECTION_PDF" ref="pdfQueueRef" />
</AsyncStateGate>
<AdminUsersSection v-if="props.section === SECTION_USERS || props.section === SECTION_REPAIRS" v-show="props.section === SECTION_USERS" ref="usersRef" />
<AdminIntegritySection v-if="props.section === SECTION_INTEGRITY" ref="integrityRef" />
<AdminRepairsSection v-if="props.section === SECTION_REPAIRS" ref="repairsRef" :users="usersRef?.users ?? []" />
<AdminPdfQueueSection v-if="props.section === SECTION_PDF" ref="pdfQueueRef" />
</section>
</template>

View file

@ -1,6 +1,8 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import { setActivePinia, createPinia } from "pinia";
import { useAuthStore } from "@/stores/auth";
import AdminPdfQueueSection from "./AdminPdfQueueSection.vue";
vi.mock("@/features/admin_dbops", () => ({
@ -36,6 +38,9 @@ function buildQueueItem(overrides: Record<string, unknown> = {}) {
describe("AdminPdfQueueSection", () => {
beforeEach(() => {
setActivePinia(createPinia());
const auth = useAuthStore();
auth.$patch({ state: "authenticated", user: { id: 1, email: "admin@test.com", is_admin: true, is_active: true } });
mockedListQueue.mockReset();
});

View file

@ -1,5 +1,6 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import { useAuthStore } from "@/stores/auth";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
@ -17,6 +18,7 @@ import {
import { useRequestState } from "@/composables/useRequestState";
const { errorMessage, errorRequestId, successMessage, clearAlerts, assignError, setSuccess } = useRequestState();
const auth = useAuthStore();
const refreshingPdfQueue = ref(false);
const requeueingPublicationId = ref<number | null>(null);
@ -162,7 +164,7 @@ defineExpose({ load });
<option value="50">50 / page</option>
<option value="100">100 / page</option>
</AppSelect>
<AppButton variant="secondary" class="!min-h-8 whitespace-nowrap !px-2.5 !py-1 !text-xs" :disabled="requeueingAllPdfs" title="Queue all missing PDFs" @click="onRequeueAllPdfs">
<AppButton v-if="auth.isAdmin" variant="secondary" class="!min-h-8 whitespace-nowrap !px-2.5 !py-1 !text-xs" :disabled="requeueingAllPdfs" title="Queue all missing PDFs" @click="onRequeueAllPdfs">
{{ requeueingAllPdfs ? "Queueing..." : "Queue all" }}
</AppButton>
<AppRefreshButton variant="secondary" size="sm" :loading="refreshingPdfQueue" title="Refresh PDF queue" loading-title="Refreshing PDF queue" @click="refreshPdfQueue" />
@ -194,7 +196,7 @@ defineExpose({ load });
<td>{{ formatTimestamp(item.last_attempt_at) }}</td>
<td>{{ formatTimestamp(item.resolved_at) }}</td>
<td>
<AppButton variant="ghost" :disabled="requeueingPublicationId === item.publication_id || !canRequeuePdf(item)" @click="onRequeuePdf(item)">
<AppButton v-if="auth.isAdmin" variant="ghost" :disabled="requeueingPublicationId === item.publication_id || !canRequeuePdf(item)" @click="onRequeuePdf(item)">
{{ requeueingPublicationId === item.publication_id ? "Requeueing..." : "Requeue" }}
</AppButton>
</td>

View file

@ -404,7 +404,7 @@ watch(
</div>
<div class="flex items-center gap-2">
<AppButton
v-if="auth.isAdmin && runStatus.isLikelyRunning"
v-if="runStatus.isLikelyRunning"
variant="danger"
:disabled="!activeRunId || isCancelAnimating"
@click="onCancelRun"
@ -420,7 +420,6 @@ watch(
</span>
</AppButton>
<AppButton
v-if="auth.isAdmin"
:disabled="isStartBlocked"
:title="startCheckDisabledReason || undefined"
:class="isStartCheckAnimating ? 'shadow-[0_0_0_1px_var(--color-state-info-border)]' : ''"
@ -486,9 +485,21 @@ watch(
</RouterLink>
</div>
<p class="mt-2 text-sm text-secondary">
Started {{ formatDate(displayedLatestRun.start_dt) }}. Processed
{{ displayedLatestRun.scholar_count }} scholars and discovered
{{ displayedLatestRun.new_publication_count }} new publications.
Started {{ formatDate(displayedLatestRun.start_dt) }}.
<template v-if="runStatus.isLikelyRunning">
{{ runStatus.scholarProgress?.visited ?? 0 }} / {{ runStatus.scholarProgress?.total ?? '…' }} visited
· {{ runStatus.scholarProgress?.finished ?? 0 }} finished
· {{ displayedLatestRun.new_publication_count }} new publications.
</template>
<template v-else>
Processed {{ displayedLatestRun.scholar_count }} scholars and discovered
{{ displayedLatestRun.new_publication_count }} new publications.
<template v-if="displayedLatestRun.failed_count > 0 || displayedLatestRun.partial_count > 0">
<span class="text-state-warning-text">
{{ displayedLatestRun.failed_count > 0 ? `${displayedLatestRun.failed_count} failed` : "" }}{{ displayedLatestRun.failed_count > 0 && displayedLatestRun.partial_count > 0 ? ", " : "" }}{{ displayedLatestRun.partial_count > 0 ? `${displayedLatestRun.partial_count} partial` : "" }}.
</span>
</template>
</template>
</p>
</div>
<AppEmptyState
@ -521,6 +532,9 @@ watch(
</div>
<span class="text-xs text-secondary">
{{ formatDate(run.start_dt) }} {{ run.new_publication_count }} new
<template v-if="run.failed_count > 0">
<span class="text-state-warning-text">{{ run.failed_count }} failed</span>
</template>
</span>
</li>
</ul>

View file

@ -1,5 +1,5 @@
<script setup lang="ts">
import { computed, onMounted, onUnmounted, ref, watch } from "vue";
import { computed, onMounted, ref, watch } from "vue";
import AppPage from "@/components/layout/AppPage.vue";
import AsyncStateGate from "@/components/patterns/AsyncStateGate.vue";
@ -37,23 +37,6 @@ const selectedPublicationKeys = ref<Set<string>>(new Set());
const retryingPublicationKeys = ref<Set<string>>(new Set());
const favoriteUpdatingKeys = ref<Set<string>>(new Set());
const PUBLICATIONS_RUN_STATUS_SYNC_INTERVAL_MS = 5000;
let runStatusSyncTimer: ReturnType<typeof setInterval> | null = null;
function startRunStatusSyncLoop(): void {
if (runStatusSyncTimer !== null) return;
runStatusSyncTimer = setInterval(() => {
if (runStatus.isRunActive) return;
void runStatus.syncLatest();
}, PUBLICATIONS_RUN_STATUS_SYNC_INTERVAL_MS);
}
function stopRunStatusSyncLoop(): void {
if (runStatusSyncTimer === null) return;
clearInterval(runStatusSyncTimer);
runStatusSyncTimer = null;
}
// --- Computed ---
const selectedCount = computed(() => selectedPublicationKeys.value.size);
@ -358,13 +341,8 @@ async function onStartRun(): Promise<void> {
onMounted(() => {
pub.syncFiltersFromRoute();
startRunStatusSyncLoop();
void Promise.all([pub.loadScholarFilters(), pub.loadPublications(), runStatus.syncLatest()]);
});
onUnmounted(() => {
stopRunStatusSyncLoop();
});
</script>
<template>

View file

@ -6,6 +6,7 @@ import AsyncStateGate from "@/components/patterns/AsyncStateGate.vue";
import RequestStateAlerts from "@/components/patterns/RequestStateAlerts.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppConfirmModal from "@/components/ui/AppConfirmModal.vue";
import AppEmptyState from "@/components/ui/AppEmptyState.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
@ -27,10 +28,12 @@ import {
type ScholarProfile,
type ScholarSearchCandidate,
} from "@/features/scholars";
import { useScholarBulkActions } from "@/features/scholars/composables/useScholarBulkActions";
import ScholarAvatar from "@/features/scholars/components/ScholarAvatar.vue";
import ScholarBatchAdd from "@/features/scholars/components/ScholarBatchAdd.vue";
import ScholarNameSearch from "@/features/scholars/components/ScholarNameSearch.vue";
import ScholarSettingsModal from "@/features/scholars/components/ScholarSettingsModal.vue";
import ScholarStatusBadges from "@/features/scholars/components/ScholarStatusBadges.vue";
import { ApiRequestError } from "@/lib/api/errors";
import { useRunStatusStore } from "@/stores/run_status";
@ -270,102 +273,71 @@ async function onToggleScholar(): Promise<void> {
}
}
async function onDeleteScholar(): Promise<void> {
function onDeleteScholar(): void {
const profile = activeScholarSettings.value;
if (!profile) return;
const label = scholarLabel(profile);
if (!window.confirm(`Delete scholar ${label}? This removes all linked publications and queue data.`)) return;
activeScholarId.value = profile.id;
clearMessages();
try {
await deleteScholar(profile.id);
successMessage.value = `${label} deleted.`;
activeScholarSettingsId.value = null;
await loadScholars();
} catch (error) {
assignError(error, "Unable to delete scholar.");
} finally {
activeScholarId.value = null;
}
bulk.requestConfirm(
`Delete ${label}?`,
"This removes all linked publications and queue data. This action cannot be undone.",
"danger",
async () => {
bulk.dismissConfirm();
activeScholarId.value = profile.id;
clearMessages();
try {
await deleteScholar(profile.id);
successMessage.value = `${label} deleted.`;
activeScholarSettingsId.value = null;
await loadScholars();
} catch (error) {
assignError(error, "Unable to delete scholar.");
} finally {
activeScholarId.value = null;
}
},
);
}
async function onSaveImageUrl(): Promise<void> {
const profile = activeScholarSettings.value;
if (!profile) return;
const candidate = (imageUrlDraftByScholarId.value[profile.id] || "").trim();
if (!candidate) { errorMessage.value = "Enter an image URL before saving, or use Reset image."; return; }
imageSavingScholarId.value = profile.id;
const p = activeScholarSettings.value;
if (!p) return;
const url = (imageUrlDraftByScholarId.value[p.id] || "").trim();
if (!url) { errorMessage.value = "Enter an image URL before saving, or use Reset image."; return; }
imageSavingScholarId.value = p.id;
clearMessages();
try {
await setScholarImageUrl(profile.id, candidate);
successMessage.value = `Image URL updated for ${scholarLabel(profile)}.`;
await loadScholars();
} catch (error) {
assignError(error, "Unable to update scholar image URL.");
} finally {
imageSavingScholarId.value = null;
}
try { await setScholarImageUrl(p.id, url); successMessage.value = `Image URL updated for ${scholarLabel(p)}.`; await loadScholars(); }
catch (e) { assignError(e, "Unable to update scholar image URL."); }
finally { imageSavingScholarId.value = null; }
}
async function onUploadImage(event: Event): Promise<void> {
const profile = activeScholarSettings.value;
if (!profile) return;
const p = activeScholarSettings.value;
if (!p) return;
const input = event.target as HTMLInputElement | null;
const file = input?.files?.[0] ?? null;
if (!file) return;
imageUploadingScholarId.value = profile.id;
imageUploadingScholarId.value = p.id;
clearMessages();
try {
await uploadScholarImage(profile.id, file);
successMessage.value = `Uploaded image for ${scholarLabel(profile)}.`;
await loadScholars();
} catch (error) {
assignError(error, "Unable to upload scholar image.");
} finally {
imageUploadingScholarId.value = null;
if (input) input.value = "";
}
try { await uploadScholarImage(p.id, file); successMessage.value = `Uploaded image for ${scholarLabel(p)}.`; await loadScholars(); }
catch (e) { assignError(e, "Unable to upload scholar image."); }
finally { imageUploadingScholarId.value = null; if (input) input.value = ""; }
}
async function onResetImage(): Promise<void> {
const profile = activeScholarSettings.value;
if (!profile) return;
imageSavingScholarId.value = profile.id;
const p = activeScholarSettings.value;
if (!p) return;
imageSavingScholarId.value = p.id;
clearMessages();
try {
await clearScholarImage(profile.id);
successMessage.value = `Image reset for ${scholarLabel(profile)}.`;
await loadScholars();
} catch (error) {
assignError(error, "Unable to reset scholar image.");
} finally {
imageSavingScholarId.value = null;
}
try { await clearScholarImage(p.id); successMessage.value = `Image reset for ${scholarLabel(p)}.`; await loadScholars(); }
catch (e) { assignError(e, "Unable to reset scholar image."); }
finally { imageSavingScholarId.value = null; }
}
// --- Import/export ---
function suggestExportFilename(exportedAt: string): string {
return `scholarr-export-${exportedAt.slice(0, 10) || "unknown-date"}.json`;
}
function downloadJsonFile(filename: string, payload: unknown): void {
const blob = new Blob([JSON.stringify(payload, null, 2)], { type: "application/json" });
const url = URL.createObjectURL(blob);
const anchor = document.createElement("a");
anchor.href = url;
anchor.download = filename;
anchor.click();
URL.revokeObjectURL(url);
}
function importSummary(result: DataImportResult): string {
return (
`Import complete. Scholars +${result.scholars_created}` +
` / updated ${result.scholars_updated}; publications +${result.publications_created}` +
` / updated ${result.publications_updated}; links +${result.links_created}` +
` / updated ${result.links_updated}; skipped ${result.skipped_records}.`
);
function importSummary(r: DataImportResult): string {
return `Import complete. Scholars +${r.scholars_created} / updated ${r.scholars_updated}; publications +${r.publications_created} / updated ${r.publications_updated}; links +${r.links_created} / updated ${r.links_updated}; skipped ${r.skipped_records}.`;
}
async function onExportData(): Promise<void> {
@ -373,7 +345,13 @@ async function onExportData(): Promise<void> {
clearMessages();
try {
const payload = await exportScholarData();
downloadJsonFile(suggestExportFilename(payload.exported_at), payload);
const blob = new Blob([JSON.stringify(payload, null, 2)], { type: "application/json" });
const url = URL.createObjectURL(blob);
const a = document.createElement("a");
a.href = url;
a.download = `scholarr-export-${payload.exported_at.slice(0, 10) || "unknown-date"}.json`;
a.click();
URL.revokeObjectURL(url);
successMessage.value = "Export complete.";
} catch (error) {
assignError(error, "Unable to export scholars and publications.");
@ -382,10 +360,6 @@ async function onExportData(): Promise<void> {
}
}
function onOpenImportPicker(): void {
importFileInput.value?.click();
}
async function onImportFileSelected(event: Event): Promise<void> {
const input = event.target as HTMLInputElement | null;
const file = input?.files?.[0] ?? null;
@ -394,12 +368,13 @@ async function onImportFileSelected(event: Event): Promise<void> {
clearMessages();
try {
const raw = await file.text();
const parsed = JSON.parse(raw) as DataImportPayload;
if (!parsed || !Array.isArray(parsed.scholars) || !Array.isArray(parsed.publications)) {
let parsed = JSON.parse(raw);
if (parsed?.data && Array.isArray(parsed.data.scholars)) parsed = parsed.data;
const payload = parsed as DataImportPayload;
if (!payload || !Array.isArray(payload.scholars) || !Array.isArray(payload.publications)) {
throw new Error("Invalid import file: expected scholars[] and publications[] arrays.");
}
const result = await importScholarData(parsed);
successMessage.value = importSummary(result);
successMessage.value = importSummary(await importScholarData(payload));
await loadScholars();
} catch (error) {
assignError(error, "Unable to import scholars and publications.");
@ -409,6 +384,15 @@ async function onImportFileSelected(event: Event): Promise<void> {
}
}
// --- Bulk actions ---
const bulk = useScholarBulkActions(visibleScholars, {
clearMessages,
assignError,
setSuccess: (msg: string) => { successMessage.value = msg; },
reloadScholars: loadScholars,
});
// --- Lifecycle ---
onMounted(() => { void loadScholars(); });
@ -466,7 +450,7 @@ watch(
<AppButton variant="secondary" :disabled="loading || exportingData" @click="onExportData">
{{ exportingData ? "Exporting..." : "Export" }}
</AppButton>
<AppButton variant="secondary" :disabled="loading || importingData" @click="onOpenImportPicker">
<AppButton variant="secondary" :disabled="loading || importingData" @click="importFileInput?.click()">
{{ importingData ? "Importing..." : "Import" }}
</AppButton>
<AppRefreshButton variant="secondary" :disabled="saving" :loading="loading" title="Refresh scholars" loading-title="Refreshing scholars" @click="loadScholars" />
@ -491,6 +475,34 @@ watch(
</div>
</div>
<div class="flex flex-wrap items-center justify-between gap-2 border-t border-stroke-default pt-2">
<span class="text-xs text-secondary">
{{ trackedCountLabel }}
<template v-if="bulk.hasSelection.value"> · {{ bulk.selectedCount.value }} selected</template>
</span>
<div class="flex items-center gap-1">
<label for="scholars-bulk-action" class="sr-only">Bulk action</label>
<AppSelect
id="scholars-bulk-action"
v-model="bulk.bulkAction.value"
:disabled="bulk.bulkBusy.value || loading"
class="max-w-[14rem] !py-1.5 !text-xs"
>
<option v-for="option in bulk.bulkActionOptions.value" :key="option.value" :value="option.value">
{{ option.label }}
</option>
</AppSelect>
<AppButton
variant="secondary"
class="h-8 min-h-8 shrink-0 px-2 text-xs"
:disabled="bulk.bulkApplyDisabled.value"
@click="bulk.onApplyBulkAction"
>
{{ bulk.bulkApplyLabel.value }}
</AppButton>
</div>
</div>
<div class="min-h-0 flex-1 xl:overflow-hidden">
<AsyncStateGate :loading="loading" :loading-lines="6" :empty="!hasTrackedScholars" :show-empty="!errorMessage" empty-title="No scholars tracked" empty-body="Add a Scholar ID or URL to start ingestion tracking.">
<AppEmptyState v-if="!hasVisibleScholars" title="No scholars match this filter" body="Clear or adjust the filter to see tracked scholars." />
@ -498,9 +510,19 @@ watch(
<ul class="flex gap-3 overflow-x-auto p-1 lg:hidden">
<li v-for="item in visibleScholars" :key="item.id" class="rounded-xl border border-stroke-default bg-surface-card-muted/70 p-3">
<div class="flex items-start gap-3">
<input
type="checkbox"
class="bulk-check mt-1 shrink-0"
:checked="bulk.selectedIds.value.has(item.id)"
:aria-label="`Select ${scholarLabel(item)}`"
@change="bulk.onToggleRow(item.id, $event)"
/>
<ScholarAvatar :label="item.display_name" :scholar-id="item.scholar_id" :image-url="item.profile_image_url" />
<div class="min-w-0 flex-1 space-y-1">
<p class="truncate text-sm font-semibold text-ink-primary">{{ scholarLabel(item) }}</p>
<p class="truncate text-sm font-semibold text-ink-primary">
{{ scholarLabel(item) }}
<ScholarStatusBadges :baseline-completed="item.baseline_completed" :last-run-status="item.last_run_status" />
</p>
<div class="flex flex-wrap items-center gap-3">
<RouterLink :to="scholarPublicationsRoute(item)" class="link-inline text-xs">Publications</RouterLink>
<a :href="scholarProfileUrl(item.scholar_id)" target="_blank" rel="noreferrer" class="link-inline text-xs">Open profile</a>
@ -514,17 +536,39 @@ watch(
<AppTable class="h-full overflow-y-scroll overscroll-contain" label="Tracked scholars table">
<thead>
<tr>
<th scope="col" class="w-10">
<input
type="checkbox"
class="bulk-check"
:checked="bulk.allVisibleSelected.value"
:disabled="visibleScholars.length === 0"
aria-label="Select all visible scholars"
@change="bulk.onToggleAll"
/>
</th>
<th scope="col">Scholar</th>
<th scope="col" class="w-[11rem]">Manage</th>
</tr>
</thead>
<tbody>
<tr v-for="item in visibleScholars" :key="item.id">
<td>
<input
type="checkbox"
class="bulk-check"
:checked="bulk.selectedIds.value.has(item.id)"
:aria-label="`Select ${scholarLabel(item)}`"
@change="bulk.onToggleRow(item.id, $event)"
/>
</td>
<td>
<div class="flex items-start gap-3">
<ScholarAvatar :label="item.display_name" :scholar-id="item.scholar_id" :image-url="item.profile_image_url" />
<div class="grid min-w-0 gap-1">
<strong class="truncate text-ink-primary">{{ scholarLabel(item) }}</strong>
<strong class="truncate text-ink-primary">
{{ scholarLabel(item) }}
<ScholarStatusBadges :baseline-completed="item.baseline_completed" :last-run-status="item.last_run_status" />
</strong>
<div class="flex flex-wrap items-center gap-3">
<RouterLink :to="scholarPublicationsRoute(item)" class="link-inline text-xs">Publications</RouterLink>
<a :href="scholarProfileUrl(item.scholar_id)" target="_blank" rel="noreferrer" class="link-inline text-xs">Open profile</a>
@ -561,5 +605,21 @@ watch(
@toggle="onToggleScholar"
@delete="onDeleteScholar"
/>
<AppConfirmModal
:open="bulk.confirmState.value.open"
:title="bulk.confirmState.value.title"
:message="bulk.confirmState.value.message"
:variant="bulk.confirmState.value.variant"
confirm-label="Delete"
@confirm="bulk.confirmState.value.onConfirm()"
@cancel="bulk.dismissConfirm()"
/>
</AppPage>
</template>
<style scoped>
.bulk-check {
@apply h-4 w-4 rounded border-stroke-interactive bg-surface-input text-brand-600 focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset;
}
</style>

View file

@ -47,7 +47,7 @@ const savingScholarHttp = ref(false);
const updatingPassword = ref(false);
const autoRunEnabled = ref(false);
const runIntervalMinutes = ref("60");
const runIntervalHours = ref("1");
const requestDelaySeconds = ref("2");
const navVisiblePages = ref<string[]>([]);
const openalexApiKey = ref("");
@ -77,13 +77,13 @@ const tabItems = computed<AppTabItem[]>(() => {
const tabs: AppTabItem[] = [
{ id: TAB_CHECKING, label: "Checking" },
{ id: TAB_ACCOUNT, label: "Account" },
{ id: TAB_ADMIN_PDF, label: "PDF Queue" },
];
if (auth.isAdmin) {
tabs.push(
{ id: TAB_ADMIN_USERS, label: "Users" },
{ id: TAB_ADMIN_INTEGRITY, label: "Integrity" },
{ id: TAB_ADMIN_REPAIRS, label: "Repairs" },
{ id: TAB_ADMIN_PDF, label: "PDF Queue" },
{ id: TAB_ADMIN_INTEGRATIONS, label: "Integrations" },
);
}
@ -138,7 +138,7 @@ function hydrateSettings(settings: UserSettings): void {
manualRunAllowed.value = Boolean(settings.policy?.manual_run_allowed ?? true);
autoRunEnabled.value = Boolean(settings.auto_run_enabled) && automationAllowed.value;
runIntervalMinutes.value = String(settings.run_interval_minutes);
runIntervalHours.value = String(Number(settings.run_interval_minutes) / 60);
requestDelaySeconds.value = String(settings.request_delay_seconds);
navVisiblePages.value = normalizeUserNavVisiblePages(settings.nav_visible_pages);
@ -168,6 +168,23 @@ function parseBoundedInteger(value: string, label: string, minimum: number): num
return parsed;
}
function formatHours(minutes: number): string {
const h = minutes / 60;
return Number.isInteger(h) ? String(h) : h.toFixed(2).replace(/\.?0+$/, "");
}
function parseHoursToMinutes(value: string, minMinutes: number): number {
const hours = Number(value);
if (!Number.isFinite(hours) || hours <= 0) {
throw new Error("Check interval must be a positive number of hours.");
}
const minutes = Math.round(hours * 60);
if (minutes < minMinutes) {
throw new Error(`Check interval must be at least ${formatHours(minMinutes)} hours.`);
}
return minutes;
}
async function loadSettings(): Promise<void> {
loading.value = true;
errorMessage.value = null;
@ -225,9 +242,8 @@ async function onSaveSettings(): Promise<void> {
try {
const payload: UserSettingsUpdate = {
auto_run_enabled: autoRunEnabled.value,
run_interval_minutes: parseBoundedInteger(
runIntervalMinutes.value,
"Check interval (minutes)",
run_interval_minutes: parseHoursToMinutes(
runIntervalHours.value,
minCheckIntervalMinutes.value,
),
request_delay_seconds: parseBoundedInteger(
@ -357,11 +373,11 @@ onMounted(async () => {
<label class="grid gap-2 text-sm font-medium text-ink-secondary">
<span class="inline-flex items-center gap-1">
Check interval (minutes)
Check interval (hours)
<AppHelpHint text="Minimum is controlled by server policy." />
</span>
<AppInput v-model="runIntervalMinutes" inputmode="numeric" />
<span class="text-xs text-secondary">Minimum: {{ minCheckIntervalMinutes }}</span>
<AppInput v-model="runIntervalHours" inputmode="decimal" />
<span class="text-xs text-secondary">Minimum: {{ formatHours(minCheckIntervalMinutes) }} hours</span>
</label>
<label class="grid gap-2 text-sm font-medium text-ink-secondary">

View file

@ -9,7 +9,7 @@ import {
import { type PublicationItem } from "@/features/publications";
import { ApiRequestError } from "@/lib/api/errors";
export const RUN_STATUS_POLL_INTERVAL_MS = 5000;
export const RUN_STATUS_POLL_INTERVAL_MS = 15000;
export const RUN_STATUS_STARTING_PHASE_MS = 1500;
export type StartManualCheckResult =
@ -168,6 +168,7 @@ export const useRunStatusStore = defineStore("runStatus", {
lastErrorRequestId: null as string | null,
lastSyncAt: null as number | null,
livePublications: [] as Array<PublicationItem>,
scholarProgress: null as { visited: number; finished: number; total: number } | null,
}),
getters: {
isRunActive(state): boolean {
@ -257,6 +258,7 @@ export const useRunStatusStore = defineStore("runStatus", {
}
activeStreamRunId = targetRunId;
this.livePublications = [];
this.scholarProgress = null;
eventSource = new EventSource(`/api/v1/runs/${targetRunId}/stream`);
eventSource.addEventListener("publication_discovered", (e) => {
try {
@ -312,6 +314,19 @@ export const useRunStatusStore = defineStore("runStatus", {
console.error("Failed to parse SSE event", err);
}
});
eventSource.addEventListener("scholar_progress", (e) => {
try {
const data = JSON.parse(e.data);
const visited = typeof data.visited === "number" ? Math.max(0, Math.trunc(data.visited)) : null;
const finished = typeof data.finished === "number" ? Math.max(0, Math.trunc(data.finished)) : null;
const total = typeof data.total === "number" ? Math.max(1, Math.trunc(data.total)) : null;
if (visited !== null && finished !== null && total !== null) {
this.scholarProgress = { visited, finished, total };
}
} catch (err) {
console.error("Failed to parse SSE event", err);
}
});
eventSource.onerror = () => {
// Reconnecting is handled automatically by EventSource,
// but if it's permanently closed, we could do something here.
@ -491,6 +506,7 @@ export const useRunStatusStore = defineStore("runStatus", {
this.lastSyncAt = null;
this.safetyState = createDefaultSafetyState();
this.livePublications = [];
this.scholarProgress = null;
},
},
});

58
scripts/premerge.sh Executable file
View file

@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
DC="docker compose -f docker-compose.yml -f docker-compose.dev.yml"
passed=0
failed=0
failures=()
step() {
local label="$1"
shift
printf '\n\033[1;34m── %s\033[0m\n' "$label"
if "$@"; then
printf '\033[1;32m PASS\033[0m %s\n' "$label"
((passed++)) || true
else
printf '\033[1;31m FAIL\033[0m %s\n' "$label"
((failed++)) || true
failures+=("$label")
fi
}
# ── repo hygiene (host-side, no runtime deps) ────────────────────────────────
step "No generated artifacts" ./scripts/check_no_generated_artifacts.sh
step "Env contract parity" python3 scripts/check_env_contract.py
step "API contract drift" python3 scripts/check_frontend_api_contract.py
# ── backend lint + typecheck ─────────────────────────────────────────────────
step "Ruff check" $DC run --rm app ruff check .
step "Ruff format" $DC run --rm app ruff format --check .
step "Mypy" $DC run --rm app mypy app/ --ignore-missing-imports
# ── backend tests ────────────────────────────────────────────────────────────
step "Unit tests" $DC run --rm app python -m pytest tests/unit
step "Integration tests" $DC run --rm app python -m pytest -m integration
# ── frontend ─────────────────────────────────────────────────────────────────
step "Frontend theme tokens" $DC run --rm frontend npm run check:theme-tokens
step "Frontend typecheck" $DC run --rm frontend npm run typecheck
step "Frontend unit tests" $DC run --rm frontend npm run test:run
step "Frontend build" $DC run --rm frontend npm run build
# ── docs ─────────────────────────────────────────────────────────────────────
step "Docs build" npm --prefix docs/website run build
# ── summary ──────────────────────────────────────────────────────────────────
printf '\n\033[1;34m── Summary ─────────────────────────────────────────────\033[0m\n'
printf ' %s passed, %s failed\n' "$passed" "$failed"
if ((failed > 0)); then
printf '\n\033[1;31m Failed steps:\033[0m\n'
for f in "${failures[@]}"; do
printf ' - %s\n' "$f"
done
exit 1
else
printf '\n\033[1;32m All checks passed.\033[0m\n'
fi

View file

@ -340,10 +340,11 @@ async def test_api_admin_dbops_forbidden_for_non_admin_and_validates_scope(db_se
assert forbidden_integrity.status_code == 403
assert forbidden_integrity.json()["error"]["code"] == "forbidden"
forbidden_pdf_queue = client.get("/api/v1/admin/db/pdf-queue")
assert forbidden_pdf_queue.status_code == 403
assert forbidden_pdf_queue.json()["error"]["code"] == "forbidden"
# PDF queue listing is accessible to all authenticated users (not admin-only).
allowed_pdf_queue = client.get("/api/v1/admin/db/pdf-queue")
assert allowed_pdf_queue.status_code == 200
# But requeue actions still require admin.
forbidden_requeue = client.post(
"/api/v1/admin/db/pdf-queue/1/requeue",
headers=headers,

View file

@ -9,6 +9,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runtime_deps import get_scholar_source
from app.main import app
from app.services.scholar.rate_limit import reset_scholar_rate_limit_state_for_tests
from app.services.scholar.source import FetchResult
from app.settings import settings
from tests.integration.helpers import (
@ -69,6 +70,110 @@ async def test_api_scholars_crud_flow(db_session: AsyncSession) -> None:
assert delete_response.json()["data"]["message"] == "Scholar deleted."
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_delete_scholar_returns_404_for_nonexistent(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-delete-404@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-delete-404@example.com", password="api-password")
headers = api_csrf_headers(client)
response = client.delete("/api/v1/scholars/999999", headers=headers)
assert response.status_code == 404
assert response.json()["error"]["code"] == "scholar_not_found"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_delete_scholar_cascades_linked_publications(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-delete-cascade@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{"user_id": user_id, "scholar_id": "delCASCADE01", "display_name": "Cascade Test"},
)
scholar_profile_id = int(scholar_result.scalar_one())
pub_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 0)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 7000):064x}",
"title_raw": "Cascade Publication",
"title_normalized": "cascadepublication",
},
)
publication_id = int(pub_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{"scholar_profile_id": scholar_profile_id, "publication_id": publication_id},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-delete-cascade@example.com", password="api-password")
headers = api_csrf_headers(client)
delete_response = client.delete(f"/api/v1/scholars/{scholar_profile_id}", headers=headers)
assert delete_response.status_code == 200
assert delete_response.json()["data"]["message"] == "Scholar deleted."
link_result = await db_session.execute(
text("SELECT count(*) FROM scholar_publications WHERE scholar_profile_id = :id"),
{"id": scholar_profile_id},
)
assert int(link_result.scalar_one()) == 0
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_create_scholar_rejects_corrupted_id(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-bad-id@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-bad-id@example.com", password="api-password")
headers = api_csrf_headers(client)
for bad_id in ["short", "ABCDEF12345!", "", " ", "AB CD EF1234"]:
response = client.post(
"/api/v1/scholars",
json={"scholar_id": bad_id},
headers=headers,
)
assert response.status_code == 400, f"Expected 400 for scholar_id={bad_id!r}"
assert response.json()["error"]["code"] == "invalid_scholar"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
@ -141,6 +246,7 @@ async def test_api_scholars_search_and_profile_image_management(
assert candidate["scholar_id"] == "abcDEF123456"
assert candidate["profile_image_url"] == "https://scholar.google.com/citations/images/avatar_scholar_256.png"
reset_scholar_rate_limit_state_for_tests()
create_response = client.post(
"/api/v1/scholars",
json={

View file

@ -0,0 +1,170 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from tests.integration.helpers import (
api_csrf_headers,
insert_user,
login_user,
)
def _create_scholar(client: TestClient, headers: dict, scholar_id: str) -> int:
resp = client.post("/api/v1/scholars", json={"scholar_id": scholar_id}, headers=headers)
assert resp.status_code == 201
return int(resp.json()["data"]["id"])
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_bulk_delete_with_valid_ids(db_session: AsyncSession) -> None:
await insert_user(db_session, email="bulk@example.com", password="pw123456")
client = TestClient(app)
login_user(client, email="bulk@example.com", password="pw123456")
headers = api_csrf_headers(client)
id1 = _create_scholar(client, headers, "aaaBBB111222")
id2 = _create_scholar(client, headers, "cccDDD333444")
resp = client.post(
"/api/v1/scholars/bulk-delete",
json={"scholar_profile_ids": [id1, id2]},
headers=headers,
)
assert resp.status_code == 200
assert resp.json()["data"]["deleted_count"] == 2
list_resp = client.get("/api/v1/scholars")
assert len(list_resp.json()["data"]["scholars"]) == 0
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_bulk_delete_only_deletes_own_scholars(db_session: AsyncSession) -> None:
await insert_user(db_session, email="user1@example.com", password="pw123456")
await insert_user(db_session, email="user2@example.com", password="pw123456")
client1 = TestClient(app)
login_user(client1, email="user1@example.com", password="pw123456")
headers1 = api_csrf_headers(client1)
client2 = TestClient(app)
login_user(client2, email="user2@example.com", password="pw123456")
headers2 = api_csrf_headers(client2)
id_user1 = _create_scholar(client1, headers1, "aaaBBB111222")
id_user2 = _create_scholar(client2, headers2, "cccDDD333444")
# User1 tries to delete both — should only delete own
resp = client1.post(
"/api/v1/scholars/bulk-delete",
json={"scholar_profile_ids": [id_user1, id_user2]},
headers=headers1,
)
assert resp.status_code == 200
assert resp.json()["data"]["deleted_count"] == 1
# User2's scholar still exists
list_resp = client2.get("/api/v1/scholars")
scholars = list_resp.json()["data"]["scholars"]
assert len(scholars) == 1
assert int(scholars[0]["id"]) == id_user2
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_bulk_toggle_enables_and_disables(db_session: AsyncSession) -> None:
await insert_user(db_session, email="toggle@example.com", password="pw123456")
client = TestClient(app)
login_user(client, email="toggle@example.com", password="pw123456")
headers = api_csrf_headers(client)
id1 = _create_scholar(client, headers, "aaaBBB111222")
id2 = _create_scholar(client, headers, "cccDDD333444")
# Disable both
resp = client.post(
"/api/v1/scholars/bulk-toggle",
json={"scholar_profile_ids": [id1, id2], "is_enabled": False},
headers=headers,
)
assert resp.status_code == 200
assert resp.json()["data"]["updated_count"] == 2
scholars = client.get("/api/v1/scholars").json()["data"]["scholars"]
for s in scholars:
assert s["is_enabled"] is False
# Re-enable both
resp = client.post(
"/api/v1/scholars/bulk-toggle",
json={"scholar_profile_ids": [id1, id2], "is_enabled": True},
headers=headers,
)
assert resp.status_code == 200
assert resp.json()["data"]["updated_count"] == 2
scholars = client.get("/api/v1/scholars").json()["data"]["scholars"]
for s in scholars:
assert s["is_enabled"] is True
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_export_with_ids_filter(db_session: AsyncSession) -> None:
await insert_user(db_session, email="export@example.com", password="pw123456")
client = TestClient(app)
login_user(client, email="export@example.com", password="pw123456")
headers = api_csrf_headers(client)
id1 = _create_scholar(client, headers, "aaaBBB111222")
_create_scholar(client, headers, "cccDDD333444")
# Export only id1
resp = client.get(f"/api/v1/scholars/export?ids={id1}")
assert resp.status_code == 200
data = resp.json()["data"]
assert len(data["scholars"]) == 1
assert data["scholars"][0]["scholar_id"] == "aaaBBB111222"
# Export all (no filter)
resp_all = client.get("/api/v1/scholars/export")
assert resp_all.status_code == 200
assert len(resp_all.json()["data"]["scholars"]) == 2
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_bulk_delete_rejects_without_csrf(db_session: AsyncSession) -> None:
await insert_user(db_session, email="csrf-bulk-del@example.com", password="pw123456")
client = TestClient(app)
login_user(client, email="csrf-bulk-del@example.com", password="pw123456")
response = client.post(
"/api/v1/scholars/bulk-delete",
json={"scholar_profile_ids": [1]},
)
assert response.status_code == 403
assert response.json()["error"]["code"] == "csrf_invalid"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_bulk_toggle_rejects_without_csrf(db_session: AsyncSession) -> None:
await insert_user(db_session, email="csrf-bulk-tog@example.com", password="pw123456")
client = TestClient(app)
login_user(client, email="csrf-bulk-tog@example.com", password="pw123456")
response = client.post(
"/api/v1/scholars/bulk-toggle",
json={"scholar_profile_ids": [1], "is_enabled": False},
)
assert response.status_code == 403
assert response.json()["error"]["code"] == "csrf_invalid"

View file

@ -76,6 +76,9 @@ class FixtureScholarSource:
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
return await self.fetch_profile_page_html(scholar_id, cstart=0, pagesize=100)
async def fetch_author_search_html(self, query: str, *, start: int = 0) -> FetchResult:
raise NotImplementedError
@pytest.mark.integration
@pytest.mark.db

View file

@ -0,0 +1,176 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runtime_deps import get_scholar_source
from app.main import app
from app.services.scholar.source import FetchResult
from tests.integration.helpers import (
api_csrf_headers,
insert_user,
login_user,
wait_for_run_complete,
)
SCHOLAR_ID = "newPubDetc01"
def _build_profile_html(publications: list[dict[str, str]]) -> str:
rows = []
for pub in publications:
rows.append(
f"""
<tr class="gsc_a_tr">
<td class="gsc_a_t">
<a class="gsc_a_at"
href="/citations?view_op=view_citation&amp;citation_for_view={SCHOLAR_ID}:{pub["cluster"]}"
>{pub["title"]}</a>
<div class="gs_gray">{pub.get("authors", "A Author")}</div>
<div class="gs_gray">{pub.get("venue", "Some Venue")}</div>
</td>
<td class="gsc_a_c"><a class="gsc_a_ac">{pub.get("citations", "0")}</a></td>
<td class="gsc_a_y"><span class="gsc_a_h">{pub.get("year", "2024")}</span></td>
</tr>"""
)
count = len(publications)
return f"""
<html>
<head>
<meta property="og:image" content="https://images.example.com/detect.png" />
</head>
<body>
<div id="gsc_prf_in">New Pub Detector</div>
<span id="gsc_a_nn">Articles 1-{count}</span>
<table>
<tbody id="gsc_a_b">{"".join(rows)}
</tbody>
</table>
</body>
</html>
"""
INITIAL_PUBLICATIONS = [
{"cluster": "aaa111", "title": "Existing Paper One", "year": "2023", "citations": "10"},
{"cluster": "bbb222", "title": "Existing Paper Two", "year": "2022", "citations": "5"},
]
UPDATED_PUBLICATIONS = [
*INITIAL_PUBLICATIONS,
{"cluster": "ccc333", "title": "Brand New Paper Three", "year": "2025", "citations": "0"},
]
class _MutableScholarSource:
"""Source that serves different HTML per phase to simulate page changes."""
def __init__(self, initial_html: str) -> None:
self.html = initial_html
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
return self._result(scholar_id)
async def fetch_profile_page_html(
self,
scholar_id: str,
*,
cstart: int,
pagesize: int,
) -> FetchResult:
return self._result(scholar_id)
def _result(self, scholar_id: str) -> FetchResult:
return FetchResult(
requested_url=f"https://scholar.google.com/citations?hl=en&user={scholar_id}",
status_code=200,
final_url=f"https://scholar.google.com/citations?hl=en&user={scholar_id}",
body=self.html,
error=None,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_new_publication_detected_after_page_change(
db_session: AsyncSession,
) -> None:
"""Verify the fingerprint short-circuit lets new publications through.
Phase 1: Initial scrape discovers 2 publications.
Phase 2: Same page skipped via no_change fingerprint.
Phase 3: Page gains a 3rd publication fingerprint differs,
scrape runs and discovers the new entry.
"""
await insert_user(db_session, email="new-pub-detect@example.com", password="api-password")
source = _MutableScholarSource(_build_profile_html(INITIAL_PUBLICATIONS))
app.dependency_overrides[get_scholar_source] = lambda: source
try:
with TestClient(app) as client:
login_user(client, email="new-pub-detect@example.com", password="api-password")
headers = api_csrf_headers(client)
create_resp = client.post("/api/v1/scholars", json={"scholar_id": SCHOLAR_ID}, headers=headers)
assert create_resp.status_code == 201
# ── Phase 1: initial scrape ─────────────────────────────
run1 = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "detect-run-001"},
)
assert run1.status_code == 200
run1_id = int(run1.json()["data"]["run_id"])
run1_data = await wait_for_run_complete(client, run1_id)
assert run1_data["scholar_results"][0]["outcome"] == "success"
assert run1_data["scholar_results"][0]["publication_count"] == 2
assert run1_data["scholar_results"][0].get("state_reason") != "no_change_initial_page_signature"
pubs1 = client.get("/api/v1/publications?mode=latest").json()["data"]
assert pubs1["total_count"] == 2
assert all(p["is_new_in_latest_run"] for p in pubs1["publications"])
# ── Phase 2: unchanged page → skipped ──────────────────
run2 = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "detect-run-002"},
)
assert run2.status_code == 200
run2_id = int(run2.json()["data"]["run_id"])
run2_data = await wait_for_run_complete(client, run2_id)
assert run2_data["scholar_results"][0]["state_reason"] == "no_change_initial_page_signature"
assert run2_data["scholar_results"][0]["publication_count"] == 0
# ── Phase 3: new publication appears ────────────────────
source.html = _build_profile_html(UPDATED_PUBLICATIONS)
run3 = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "detect-run-003"},
)
assert run3.status_code == 200
run3_id = int(run3.json()["data"]["run_id"])
run3_data = await wait_for_run_complete(client, run3_id)
assert run3_data["scholar_results"][0]["outcome"] == "success"
assert run3_data["scholar_results"][0].get("state_reason") != "no_change_initial_page_signature"
assert run3_data["scholar_results"][0]["publication_count"] == 3
pubs3 = client.get("/api/v1/publications?mode=latest").json()["data"]
assert pubs3["total_count"] == 3
titles = {p["title"] for p in pubs3["publications"]}
assert "Brand New Paper Three" in titles
new_pub = next(p for p in pubs3["publications"] if p["title"] == "Brand New Paper Three")
assert new_pub["is_new_in_latest_run"] is True
old_pubs = [p for p in pubs3["publications"] if p["title"] != "Brand New Paper Three"]
for p in old_pubs:
assert p["is_new_in_latest_run"] is False
finally:
app.dependency_overrides.pop(get_scholar_source, None)

View file

@ -30,11 +30,12 @@ class _PassthroughSource:
error=None,
)
async def fetch_profile_page_html(
self, scholar_id: str, *, cstart: int, pagesize: int
) -> FetchResult:
async def fetch_profile_page_html(self, scholar_id: str, *, cstart: int, pagesize: int) -> FetchResult:
return await self.fetch_profile_html(scholar_id)
async def fetch_author_search_html(self, query: str, *, start: int = 0) -> FetchResult:
raise NotImplementedError
def _csrf_headers(client: TestClient) -> dict[str, str]:
response = client.get("/api/v1/auth/me")
@ -348,6 +349,7 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
)
db_session.add_all([run, publication])
await db_session.commit()
run_id = int(run.id)
call_count = 0
@ -399,9 +401,8 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
publications=publications,
)
refreshed_run = await db_session.get(CrawlRun, int(run.id))
refreshed_run = await db_session.get(CrawlRun, run_id)
assert refreshed_run is not None
assert int(refreshed_run.new_pub_count) == 1
link_count_result = await db_session.execute(
text(
"""
@ -411,9 +412,10 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
AND first_seen_run_id = :run_id
"""
),
{"scholar_profile_id": scholar_profile_id, "run_id": int(run.id)},
{"scholar_profile_id": scholar_profile_id, "run_id": run_id},
)
assert int(link_count_result.scalar_one()) == 1
link_count = int(link_count_result.scalar_one())
assert int(refreshed_run.new_pub_count) == link_count
@pytest.mark.integration

View file

@ -0,0 +1,17 @@
from __future__ import annotations
import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@pytest.fixture
def patch_session_factory(db_session: AsyncSession, monkeypatch: pytest.MonkeyPatch):
"""Patch get_session_factory in arxiv modules to use the test's DB engine.
Without this, the implementation creates its own engine via get_session_factory()
which may not share transaction visibility with the test's db_session fixture.
"""
factory = async_sessionmaker(db_session.bind, expire_on_commit=False)
monkeypatch.setattr("app.services.arxiv.rate_limit.get_session_factory", lambda: factory)
monkeypatch.setattr("app.services.arxiv.cache.get_session_factory", lambda: factory)
return factory

View file

@ -147,8 +147,8 @@ async def test_client_coalesces_concurrent_identical_search_requests() -> None:
async def test_client_logs_cache_hit_and_miss(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
patch_session_factory,
) -> None:
_ = db_session
calls = {"count": 0}
logged: list[dict[str, object]] = []

View file

@ -1,6 +1,6 @@
from __future__ import annotations
from types import SimpleNamespace
from typing import Any
import pytest
@ -10,7 +10,9 @@ from app.services.arxiv.types import ArxivEntry, ArxivFeed, ArxivOpenSearchMeta
from app.settings import settings
def _item(*, title: str = "A Test Paper", scholar_label: str = "Ada Lovelace") -> SimpleNamespace:
def _item(*, title: str = "A Test Paper", scholar_label: str = "Ada Lovelace") -> Any:
from types import SimpleNamespace
return SimpleNamespace(title=title, scholar_label=scholar_label)
@ -71,7 +73,7 @@ async def test_http_gateway_returns_none_when_disabled(monkeypatch: pytest.Monke
fake_client = FakeClient()
object.__setattr__(settings, "arxiv_enabled", False)
try:
gateway = arxiv_gateway.HttpArxivGateway(client=fake_client)
gateway = arxiv_gateway.HttpArxivGateway(client=fake_client) # type: ignore[arg-type]
result = await gateway.discover_arxiv_id_for_publication(item=_item())
assert result is None
assert fake_client.calls == 0
@ -102,7 +104,7 @@ async def test_http_gateway_uses_client_search_for_discovery() -> None:
)
fake_client = FakeClient()
gateway = arxiv_gateway.HttpArxivGateway(client=fake_client)
gateway = arxiv_gateway.HttpArxivGateway(client=fake_client) # type: ignore[arg-type]
result = await gateway.discover_arxiv_id_for_publication(
item=_item(title="My Paper", scholar_label="Ada Lovelace"),
request_email="user@example.com",

View file

@ -16,7 +16,7 @@ from app.settings import settings
@pytest.mark.asyncio
async def test_arxiv_rate_limit_respects_cooldown(db_session: AsyncSession) -> None:
async def test_arxiv_rate_limit_respects_cooldown(db_session: AsyncSession, patch_session_factory) -> None:
db_session.add(
ArxivRuntimeState(
state_key=ARXIV_RUNTIME_STATE_KEY,
@ -37,7 +37,7 @@ async def test_arxiv_rate_limit_respects_cooldown(db_session: AsyncSession) -> N
@pytest.mark.asyncio
async def test_arxiv_rate_limit_persists_cooldown_after_429(db_session: AsyncSession) -> None:
async def test_arxiv_rate_limit_persists_cooldown_after_429(db_session: AsyncSession, patch_session_factory) -> None:
previous_interval = settings.arxiv_min_interval_seconds
previous_cooldown = settings.arxiv_rate_limit_cooldown_seconds
object.__setattr__(settings, "arxiv_min_interval_seconds", 0.0)
@ -62,7 +62,7 @@ async def test_arxiv_rate_limit_persists_cooldown_after_429(db_session: AsyncSes
@pytest.mark.asyncio
async def test_arxiv_rate_limit_serializes_concurrent_calls(db_session: AsyncSession) -> None:
async def test_arxiv_rate_limit_serializes_concurrent_calls(db_session: AsyncSession, patch_session_factory) -> None:
previous_interval = settings.arxiv_min_interval_seconds
previous_cooldown = settings.arxiv_rate_limit_cooldown_seconds
object.__setattr__(settings, "arxiv_min_interval_seconds", 0.2)
@ -88,7 +88,9 @@ async def test_arxiv_rate_limit_serializes_concurrent_calls(db_session: AsyncSes
@pytest.mark.asyncio
async def test_arxiv_rate_limit_logs_request_scheduled_and_completed(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
patch_session_factory,
) -> None:
logged: list[dict[str, object]] = []
@ -106,14 +108,16 @@ async def test_arxiv_rate_limit_logs_request_scheduled_and_completed(
assert scheduled
assert completed
assert float(scheduled[0]["wait_seconds"]) >= 0.0
assert int(completed[0]["status_code"]) == 200
assert float(str(scheduled[0]["wait_seconds"])) >= 0.0
assert int(str(completed[0]["status_code"])) == 200
assert completed[0]["source_path"] == "search"
@pytest.mark.asyncio
async def test_arxiv_rate_limit_logs_cooldown_activation(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
patch_session_factory,
) -> None:
logged_warning: list[dict[str, object]] = []
previous_interval = settings.arxiv_min_interval_seconds
@ -138,11 +142,11 @@ async def test_arxiv_rate_limit_logs_cooldown_activation(
cooldown_events = [entry for entry in logged_warning if entry.get("event") == "arxiv.cooldown_activated"]
assert cooldown_events
assert cooldown_events[0]["source_path"] == "lookup_ids"
assert float(cooldown_events[0]["cooldown_remaining_seconds"]) > 0.0
assert float(str(cooldown_events[0]["cooldown_remaining_seconds"])) > 0.0
@pytest.mark.asyncio
async def test_get_arxiv_cooldown_status_reads_active_cooldown(db_session: AsyncSession) -> None:
async def test_get_arxiv_cooldown_status_reads_active_cooldown(db_session: AsyncSession, patch_session_factory) -> None:
now_utc = datetime(2026, 2, 26, 13, 0, tzinfo=UTC)
existing = await db_session.get(ArxivRuntimeState, ARXIV_RUNTIME_STATE_KEY)
if existing is None:

View file

@ -1,6 +1,7 @@
from __future__ import annotations
from types import SimpleNamespace
from typing import Any, cast
import pytest
@ -38,8 +39,8 @@ async def test_discover_identifiers_for_enrichment_disables_arxiv_on_rate_limit(
monkeypatch.setattr(runner, "_publish_identifier_update_event", _publish_noop)
result = await runner._discover_identifiers_for_enrichment(
object(),
publication=publication,
cast(Any, object()),
publication=cast(Any, publication),
run_id=321,
allow_arxiv_lookup=True,
)

View file

@ -1,5 +1,6 @@
from __future__ import annotations
from typing import Any
from unittest.mock import MagicMock
from app.services.portability.publication_import import (
@ -9,14 +10,14 @@ from app.services.portability.publication_import import (
)
def _mock_profile(scholar_id: str = "ABC123DEF456") -> MagicMock:
def _mock_profile(scholar_id: str = "ABC123DEF456") -> Any:
profile = MagicMock()
profile.id = 1
profile.scholar_id = scholar_id
return profile
def _scholar_map(scholar_id: str = "ABC123DEF456") -> dict[str, MagicMock]:
def _scholar_map(scholar_id: str = "ABC123DEF456") -> dict[str, Any]:
return {scholar_id: _mock_profile(scholar_id)}

View file

@ -111,8 +111,9 @@ class TestBuildFingerprint:
assert all(c in "0123456789abcdef" for c in fp)
def test_deterministic(self) -> None:
kwargs = {"title": "Test Title", "year": 2024, "author_text": "Smith", "venue_text": "ICML"}
assert _build_fingerprint(**kwargs) == _build_fingerprint(**kwargs)
fp_a = _build_fingerprint(title="Test Title", year=2024, author_text="Smith", venue_text="ICML")
fp_b = _build_fingerprint(title="Test Title", year=2024, author_text="Smith", venue_text="ICML")
assert fp_a == fp_b
def test_different_titles_produce_different_fingerprints(self) -> None:
fp1 = _build_fingerprint(title="Title A", year=2024, author_text=None, venue_text=None)

View file

@ -1,7 +1,9 @@
from __future__ import annotations
import contextlib
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from typing import Any
import pytest
@ -27,7 +29,7 @@ def _job(
def _row(
*, pub_url: str | None = "https://scholar.google.com/citations?view_op=view_citation&citation_for_view=abc:xyz"
) -> SimpleNamespace:
) -> Any:
return SimpleNamespace(
publication_id=1,
scholar_profile_id=1,
@ -235,10 +237,12 @@ async def test_run_resolution_task_disables_arxiv_for_remaining_batch(
) -> None:
calls: list[tuple[int, bool]] = []
first = _row()
second = SimpleNamespace(**{**first.__dict__, "publication_id": 2})
second: Any = SimpleNamespace(**{**first.__dict__, "publication_id": 2})
def _raise_session_factory_error():
@contextlib.asynccontextmanager
async def _raise_background_session_error():
raise RuntimeError("skip user settings lookup in test")
yield # pragma: no cover
async def _fake_resolve_publication_row(
*,
@ -252,7 +256,7 @@ async def test_run_resolution_task_disables_arxiv_for_remaining_batch(
calls.append((int(row.publication_id), bool(allow_arxiv_lookup)))
return row.publication_id == 1
monkeypatch.setattr(pdf_queue_resolution, "get_session_factory", _raise_session_factory_error)
monkeypatch.setattr(pdf_queue_resolution, "background_session", _raise_background_session_error)
monkeypatch.setattr(pdf_queue_resolution, "_resolve_publication_row", _fake_resolve_publication_row)
await pdf_queue_resolution._run_resolution_task(

View file

@ -2,6 +2,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from types import SimpleNamespace
from typing import Any
import pytest
@ -11,7 +12,7 @@ from app.services.publications import pdf_resolution_pipeline as pipeline
from app.services.unpaywall.application import OaResolutionOutcome
def _row(*, display_identifier: DisplayIdentifier | None = None) -> SimpleNamespace:
def _row(*, display_identifier: DisplayIdentifier | None = None) -> Any:
return SimpleNamespace(
publication_id=1,
scholar_profile_id=1,

View file

@ -23,6 +23,12 @@ class StubScholarSource:
index = min(self.calls - 1, len(self._fetch_results) - 1)
return self._fetch_results[index]
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
raise NotImplementedError
async def fetch_profile_page_html(self, scholar_id: str, *, cstart: int, pagesize: int) -> FetchResult:
raise NotImplementedError
def _ok_author_search_fetch() -> FetchResult:
body = (

View file

@ -42,6 +42,34 @@ class TestValidateScholarId:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF12345!")
def test_rejects_empty_string(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("")
def test_rejects_whitespace_only(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id(" ")
def test_rejects_embedded_spaces(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF 12345")
def test_rejects_tab_characters(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF\t12345")
def test_rejects_url_as_id(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("https://scholar.google.com/citations?user=ABCDEF123456")
def test_rejects_newline_in_id(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF\n12345")
def test_rejects_unicode_characters(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF12345\u00e9")
class TestNormalizeDisplayName:
def test_returns_stripped_name(self) -> None:

View file

@ -1,4 +1,5 @@
from types import SimpleNamespace
from typing import Any, cast
import pytest
@ -42,9 +43,9 @@ async def test_create_hydration_skips_when_global_throttle_active(
)
result = await scholar_helpers.hydrate_scholar_metadata_if_needed(
db_session=None,
db_session=cast(Any, None),
profile=profile,
source=object(),
source=cast(Any, object()),
user_id=7,
)
@ -77,9 +78,9 @@ async def test_create_hydration_runs_when_throttle_is_clear(
)
result = await scholar_helpers.hydrate_scholar_metadata_if_needed(
db_session=None,
db_session=cast(Any, None),
profile=profile,
source=object(),
source=cast(Any, object()),
user_id=8,
)
@ -109,7 +110,7 @@ async def test_initial_scrape_job_enqueued_on_create(
)
queued = await scholar_helpers.enqueue_initial_scrape_job_for_scholar(
session,
cast(Any, session),
profile=profile,
user_id=9,
)
@ -134,7 +135,7 @@ async def test_initial_scrape_job_not_enqueued_when_disabled(
)
queued = await scholar_helpers.enqueue_initial_scrape_job_for_scholar(
session,
cast(Any, session),
profile=profile,
user_id=11,
)