justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

Feat/decomposition #19

Merged
JustinZeus merged 38 commits from feat/decomposition into main 2026-03-01 22:55:33 +01:00
Conversation 1 Commits 38 Files changed 327 +8489 -7302
80 changed files with 8489 additions and 7302 deletions
Download patch file Download diff file Expand all files Collapse all files
Showing only changes of commit a5165dc3ba - Show all commits

17
.claude/settings.json Normal file
View file

@ -0,0 +1,17 @@
{
"permissions": {
"allow": [
"Bash(wc -l /home/jv/src/personal/playground/scholar_scraper/scholar-scraper/frontend/src/features/settings/SettingsAdminPanel.vue /home/jv/src/personal/playground/scholar_scraper/scholar-scraper/frontend/src/features/settings/components/Admin*.vue)",
"Read(//home/jv/src/personal/playground/scholar_scraper/scholar-scraper/**)",
"Bash(while read line rest)",
"Bash(do echo \"$line $rest\")",
"Bash(done)",
"Bash(grep -c 'def test_' tests/unit/*.py tests/unit/**/*.py tests/integration/*.py)",
"Bash(sort -t: -k2 -rn)",
"Bash(wc -l frontend/src/pages/*.vue)"
],
"additionalDirectories": [
"/home/jv/src/personal/playground/scholar_scraper/scholar-scraper/frontend/src"
]
}
}

5
.github/workflows/release.yml vendored
View file

@ -19,9 +19,10 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install python-semantic-release
- uses: astral-sh/setup-uv@v4
- run: uv sync --extra dev
- name: Semantic Release
id: release
run: semantic-release publish
run: uv run semantic-release publish
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

193
DECOMPOSITION_CLEANUP.md
View file

@ -1,193 +0,0 @@
# Decomposition Cleanup — 3-Pass Fix Prompt
You are working on the scholarr repository (`feat/decomposition` branch). A decomposition refactor split oversized files into focused modules. An audit found residual issues. Fix them in 3 sequential passes.
**Read `agents.md` before starting.** Key rules:
- Function length: 50 lines max
- File length: 400 target, 600 hard ceiling (kwargs-heavy signatures are acceptable overage)
- No dead code, no duplicate boilerplate, no backward-compatibility shims
- All business logic in `app/services/<domain>/`
- Use `structured_log()` for domain logging
**Constraints for every pass:**
- Pure refactoring — no behavioral changes
- All tests must pass after each pass: `docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app python -m pytest`
- Commit after each pass using conventional commit format
---
## Pass 1: Naming, DRY, and Dead Code
**1a. Rename `_int_or_default` → `int_or_default`**
This function in `app/services/ingestion/run_completion.py:38` is imported externally by `app/services/ingestion/application.py:25`, making it public API with a private-convention name.
- Rename the definition in `run_completion.py` (line 38)
- Update all internal call sites in `run_completion.py` (lines 71, 350, 375)
- Update the import in `application.py` (line 25) and call sites (lines 138, 336, 342)
**1b. Consolidate duplicate functions between `pdf_queue.py` and `pdf_queue_resolution.py`**
Three functions exist with identical signatures in both files:
| Function | `pdf_queue.py` | `pdf_queue_resolution.py` |
|---|---|---|
| `_utcnow()` | line 46 | line 33 |
| `_event_row()` | line 150 | line 39 |
| `_queued_job()` | line 165 | line 60 |
Fix: Keep these in `pdf_queue.py` (the parent module). Have `pdf_queue_resolution.py` import them from `pdf_queue.py`. Since `pdf_queue.py` already imports from `pdf_queue_resolution.py`, check for circular imports — if importing `_utcnow`, `_event_row`, `_queued_job` from `pdf_queue` into `pdf_queue_resolution` would create a cycle, extract them to a shared `pdf_queue_common.py` instead.
Also deduplicate the PDF status constants (`PDF_STATUS_QUEUED`, `PDF_STATUS_RUNNING`, `PDF_STATUS_RESOLVED`, `PDF_STATUS_FAILED`) that are duplicated between `pdf_queue.py` (lines 23-27) and `pdf_queue_resolution.py` (lines 20-23). Keep them in `pdf_queue.py` only.
**1c. Consolidate `_effective_request_delay_seconds` duplication**
`scheduler.py` (lines 32-42) and `queue_runner.py` (line 23-28) both define `_effective_request_delay_seconds` with slightly different signatures. The `queue_runner.py` version takes an explicit `floor` parameter (cleaner). Keep the `queue_runner.py` version. In `scheduler.py`, delete `_request_delay_floor_seconds()` and `_effective_request_delay_seconds()`, and instead import from `queue_runner`:
```python
from app.services.ingestion.queue_runner import _effective_request_delay_seconds
```
At the call site in `scheduler.py`, pass `floor=_request_delay_floor_seconds_value` where needed (inline the floor computation or keep it as a one-liner).
Actually — since `_effective_request_delay_seconds` would now be imported externally, rename it to `effective_request_delay_seconds` (drop the underscore) in `queue_runner.py`.
Commit message: `refactor: fix naming, remove duplicate code from decomposition`
---
## Pass 2: Function length — ingestion core (6 worst offenders)
All 6 functions are >75 lines. Extract helpers to bring each under 50 lines. Below is specific guidance for each.
**2a. `EnrichmentRunner.enrich_pending_publications` — 133 lines → ≤50** (`enrichment.py:47-179`)
This function does 4 things: load publications, set up client, batch-loop with API calls, run dedup sweep. Extract:
1. `_load_unenriched_publications(db_session, *, run_id) → list[Publication]` — lines 66-87 (query + return)
2. `_enrich_batch(self, db_session, *, batch, run_id, client, openalex_works, now) → bool` — lines 142-166 (per-publication enrichment loop; return False if canceled). This inner loop iterates `batch`, calls `_discover_identifiers_for_enrichment`, and applies the match. The caller handles the outer batch loop and API call.
3. `_flush_and_sweep_duplicates(db_session, *, run_id)` — lines 167-179 (flush + dedup sweep)
The main function becomes: load pubs → early return → create client → batch loop (API call + `_enrich_batch`) → `_flush_and_sweep_duplicates`.
**2b. `EnrichmentRunner.enrich_publications_with_openalex` — 64 lines → ≤50** (`enrichment.py:260-323`)
Extract:
1. `_sanitize_titles(publications) → list[str]` — lines 279-286 (title cleaning loop)
The outer batch loop + match logic stays in the main method, which should now fit in ~45 lines.
**2c. `ScholarIngestionService.run_for_user` — 99 lines → ≤50** (`application.py:525-623`)
This function calls `initialize_run`, builds paging/threshold kwargs, calls `run_scholar_iteration`, `complete_run_for_user`, handles enrichment, commits, and logs. Extract:
1. `_run_iteration_and_complete(self, db_session, *, run, scholars, user_id, start_cstart_map, paging, thresholds, auto_queue_continuations, queue_delay_seconds, idempotency_key) → tuple[RunProgress, RunFailureSummary, RunAlertSummary]` — lines 579-599 (the iteration + completion block). This calls `run_scholar_iteration` and `complete_run_for_user` and returns their results.
2. `_inline_enrich_and_finalize(self, db_session, *, run, user_settings, intended_final_status) → None` — lines 604-614 (enrichment + status fixup + commit). This calls `enrich_pending_publications`, handles the exception, fixes status, and commits.
**2d. `ScholarIngestionService.execute_run` — 88 lines → ≤50** (`application.py:374-461`)
Extract:
1. `_execute_run_body(self, db_session, session_factory, *, run_id, user_id, scholars, start_cstart_map, paging, thresholds, auto_queue_continuations, queue_delay_seconds, idempotency_key) → None` — lines 411-457 (the try-body: prepare, iterate, complete, commit, log, spawn enrichment task). The outer `execute_run` handles kwargs resolution and the `async with session_factory()` + `except`.
Or alternatively, the `_resolve_paging_kwargs` / `_threshold_kwargs` calls can be lifted into the caller or inlined since they're trivial dict builders — this alone may shave enough lines. Consider whether the kwargs signature itself is the bulk (it is — 22 lines of signature). If so, this function may be acceptable as-is per the kwargs-exception rule. Use your judgment: if the function body (excluding signature and kwargs resolution) is ≤50 lines, document it as acceptable and move on.
**2e. `run_scholar_iteration` — 84 lines → ≤50** (`scholar_processing.py:531-614`)
This has two clear phases: pass 1 (breadth-first, lines 560-584) and pass 2 (depth, lines 586-614). Extract:
1. `_run_first_pass(db_session, *, scholars, pagination, run, user_id, start_cstart_map, scholar_kwargs, request_delay_seconds, queue_delay_seconds, progress) → dict[int, int]` — the breadth-first loop (lines 560-584), returns `first_pass_cstarts`
2. `_run_depth_pass(db_session, *, scholars, first_pass_cstarts, pagination, run, user_id, scholar_kwargs, request_delay_seconds, remaining_max, auto_queue_continuations, queue_delay_seconds, progress) → None` — the depth loop (lines 591-613)
The main function becomes: build kwargs → call pass 1 → compute remaining → call pass 2 → return progress.
**2f. `QueueJobRunner._finalize_queue_job_after_run` — 79 lines → ≤50** (`queue_runner.py:284-362`)
Two branches: success path (lines 287-311) and failure path (lines 312-362). Extract:
1. `_finalize_successful_queue_job(self, session, job, run_summary) → None` — success path
2. `_finalize_failed_queue_job(self, session, job, run_summary) → None` — failure path
The main function opens a session, branches on `failed_count`, and delegates.
Commit message: `refactor: extract helpers from long functions in ingestion core`
---
## Pass 3: Function length — remaining violations (10 functions, 58-75 lines)
**3a. `run_manual` — 105 lines → ≤50** (`app/api/routers/runs.py:183-287`)
This route handler has a large try block. Extract:
1. `_start_manual_run(db_session, *, current_user, ingest_service, idempotency_key) → tuple[CrawlRun, list, dict]` — lines 205-221 (initialize_run call with all the settings kwargs)
2. `_spawn_background_execution(ingest_service, *, run, current_user, scholars, start_cstart_map, user_settings, idempotency_key) → None` — lines 226-249 (create_task + background_tasks management)
3. `_manual_run_success_response(request, *, run, idempotency_key, db_session, current_user) → dict` — lines 251-265 (build success payload)
The route handler becomes: load safety → check disabled → check idempotency → try: start → spawn → respond; except: handle errors.
**3b. `PaginationEngine._paginate_loop` — 75 lines → ≤50** (`pagination.py:276-350`)
Extract:
1. `_upsert_page_publications(self, db_session, *, run, scholar, publications, seen_canonical, state, upsert_publications_fn)` — the dedup + upsert block (appears twice: lines 294-302 and 339-347). Deduplicate by extracting a single helper called from both places.
This should bring the loop body under 50 lines.
**3c. `PaginationEngine.fetch_and_parse_all_pages` — 72 lines → ≤50** (`pagination.py:415-486`)
Most of the length is the kwargs-heavy calls. Check if the function body (excluding the 17-line signature) is ≤50 lines — it should be ~55. Extract:
1. Move the `_short_circuit_initial_page` + `_build_loop_state` + `_paginate_loop` + `_result_from_pagination_state` sequence into a helper `_paginate_from_initial_page(self, *, scholar, run, db_session, state, ...) → PagedParseResult` if needed. Or accept this as kwargs-acceptable overage and document why.
**3d. `PageFetcher.fetch_and_parse_with_retry` — 70 lines → ≤50** (`page_fetch.py:148-217`)
Extract:
1. `_classify_attempt(parsed_page, *, network_attempts, rate_limit_attempts) → tuple[int, int, int]` — lines 173-183 (attempt counter logic, returns updated network_attempts, rate_limit_attempts, total_attempts)
This plus the existing `_should_retry` and `_sleep_backoff` helpers should bring the loop under 50 lines.
**3e. `complete_run_for_user` — 66 lines → ≤50** (`run_completion.py:265-330`)
Extract:
1. `_log_alert_threshold_warnings(*, user_id, run, failure_summary, alert_summary) → None` — lines 284-313 (the 3 if-blocks that log threshold warnings)
The main function becomes: summarize → build alerts → log warnings → apply safety → resolve status → finalize → return. ~35 lines.
**3f. `process_scholar` — 61 lines → ≤50** (`scholar_processing.py:343-403`)
The function body is mostly kwargs pass-through. Check if the actual body (excluding the 18-line signature) is ≤50 lines — it's ~43 lines of body. If so, this is acceptable under the kwargs exception. Document and move on.
**3g. `_fetch_and_prepare_scholar_result` — 59 lines → ≤50** (`scholar_processing.py:406-464`)
Same analysis: 16-line signature, 43-line body. If the body alone is ≤50, this is kwargs-acceptable. Document and move on.
**3h. `_perform_live_author_search` — 60 lines → ≤50** (`author_search.py:459-518`)
18-line signature, 42-line body. Kwargs-acceptable. Document and move on.
**3i. `search_author_candidates` — 60 lines → ≤50** (`author_search.py:521-580`)
19-line signature, 41-line body. Kwargs-acceptable. Document and move on.
**3j. `ScholarIngestionService.initialize_run` — 58 lines → ≤50** (`application.py:315-372`)
20-line signature, 38-line body. Kwargs-acceptable. Document and move on.
Commit message: `refactor: extract helpers from remaining long functions`
---
## Verification (after all 3 passes)
```bash
# No function exceeds 50 lines (body, excluding kwargs signatures that exceed the limit)
# Run the AST checker from the audit to verify
# All tests pass
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app python -m pytest
# Linting
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff check .
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff format --check .
# Type checking
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app mypy app/
```

242
DECOMPOSITION_SLICES.md
View file

@ -1,242 +0,0 @@
# Scholarr Decomposition Slices
## How to use this file
This file contains self-contained decomposition prompts ("slices") to bring the codebase into compliance with `agents.md` file size rules. Each slice is an independent task for an LLM executor.
**Rules from `agents.md`:**
- File length: 400 lines target, 600 lines hard ceiling
- Slightly exceeding the 600-line ceiling is acceptable when the excess is driven by verbose keyword-argument signatures (named parameters), not by business logic density
- Function length: 50 lines max
- All business logic in `app/services/<domain>/`
- Read `agents.md` fully before starting any slice
**Execution protocol:**
1. Pick the next incomplete slice in order (slices depend on prior ones)
2. Read the prompt section — it contains everything you need
3. Read all source files mentioned before making changes
4. Make the changes described — no business logic changes, only file reorganization
5. Verify: all imports resolve, all tests pass
6. Commit with the specified message
---
## Progress
| # | Slice | Status |
|---|-------|--------|
| 1 | Schema split | **DONE** |
| 2 | Ingestion: pagination + publication upsert | **DONE** |
| 3 | Ingestion: enrichment + scholar processing + run completion | **DONE** |
| 4 | Scholars service + PDF queue | **DONE** |
| 5 | Routers + scheduler | **DONE** |
---
## Slice 1: Split `app/api/schemas.py` into domain packages — DONE
Completed. The 945-line `app/api/schemas.py` was split into `app/api/schemas/` package:
| File | Lines | Contents |
|---|---|---|
| `common.py` | 39 | `ApiMeta`, `ApiErrorData`, `ApiErrorEnvelope`, `MessageData`, `MessageEnvelope` |
| `auth.py` | 73 | `SessionUserData`, `AuthMe*`, `CsrfBootstrap*`, `Login*`, `ChangePasswordRequest` |
| `scholars.py` | 153 | `ScholarItem*`, `ScholarSearch*`, `ScholarExport*`, `DataExport*`, `DataImport*` |
| `publications.py` | 151 | `DisplayIdentifierData`, `PublicationItem*`, `MarkAll*`, `MarkSelected*`, `RetryPublication*`, `TogglePublication*` |
| `runs.py` | 226 | `RunListItem*`, `RunSummary*`, `RunDebug*`, `RunScholarResult*`, `RunDetail*`, `ManualRun*`, `Queue*`, `ScrapeSafety*` |
| `admin.py` | 289 | All `Admin*` models including PDF queue, repair, near-duplicate schemas |
| `settings.py` | 54 | `SettingsPolicy*`, `Settings*`, `SettingsUpdateRequest` |
| `__init__.py` | 7 | Wildcard re-exports — all `from app.api.schemas import X` imports unchanged |
---
## Slice 2: Decompose `app/services/ingestion/application.py` — pagination + publication upsert — DONE
Completed. Extracted three modules from `application.py` (2,955 → 2,084 lines):
| File | Lines | Contents |
|---|---|---|
| `app/services/ingestion/page_fetch.py` | 219 | `PageFetcher` class — single-page fetch, parse-or-layout-error, retry with backoff |
| `app/services/ingestion/pagination.py` | 486 | `PaginationEngine` class — multi-page orchestration, loop state, short-circuit, fingerprint checks |
| `app/services/ingestion/publication_upsert.py` | 239 | Module-level functions — `resolve_publication`, `upsert_profile_publications`, find/create/update helpers |
Also fixed two external import paths (`portability/normalize.py`, `portability/publication_import.py`) that imported `normalize_title`/`build_publication_url` from `application.py` — now correctly sourced from `fingerprints.py`. Updated one integration test to monkeypatch the new module-level function instead of the old instance method.
---
## Slice 3: Decompose `app/services/ingestion/application.py` — enrichment, scholar processing, run completion — DONE
Completed. Extracted four logical chunks from `application.py` (2,085 → 635 lines):
| File | Lines | Contents |
|---|---|---|
| `app/services/ingestion/enrichment.py` | 323 | `EnrichmentRunner` class — OpenAlex enrichment, identifier discovery, dedup sweep |
| `app/services/ingestion/scholar_processing.py` | 614 | `process_scholar`, `run_scholar_iteration`, continuation queue, outcome resolution |
| `app/services/ingestion/run_completion.py` | 417 | `complete_run_for_user`, failure summary, alert summary, safety outcome, progress tracking |
Also updated two tests (`test_ingestion_arxiv_rate_limit.py`, `test_ingestion_progress_reporting.py`) and one integration test (`test_deferred_enrichment.py`) to import from new modules.
**Why:** Three remaining logical chunks. Extracting all three brings `application.py` down to ~500 lines (the orchestration spine).
**Files to create:** `app/services/ingestion/enrichment.py`, `app/services/ingestion/scholar_processing.py`, `app/services/ingestion/run_completion.py`
**Files to modify:** `app/services/ingestion/application.py`
**Prompt:**
You are working on the scholarr repository. Read `agents.md` for project conventions. Continue decomposing `app/services/ingestion/application.py` (slice 2 is complete — `page_fetch.py`, `pagination.py`, and `publication_upsert.py` already extracted). Extract three remaining chunks.
**A. `app/services/ingestion/enrichment.py` (~300 lines)**
Move these methods (post-run OpenAlex enrichment):
- `_run_is_canceled`, `_enrich_pending_publications`, `_discover_identifiers_for_enrichment`
- `_publish_identifier_update_event`, `_enrich_publications_with_openalex`
Create an `EnrichmentRunner` class that receives service dependencies (OpenAlex client, identifier service, dedup service, DB session factory) in its constructor.
**B. `app/services/ingestion/scholar_processing.py` (~400 lines)**
Move these methods (per-scholar outcome resolution):
- `_assert_valid_paged_parse_result`, `_apply_first_page_profile_metadata`, `_build_result_entry`
- `_skipped_no_change_outcome`, `_upsert_publications_outcome`, `_upsert_success_or_exception`
- `_upsert_success`, `_upsert_exception_outcome`, `_parse_failure_outcome`
- `_sync_continuation_queue`, `_process_scholar`, `_process_scholar_inner`
- `_fetch_and_prepare_scholar_result`, `_resolve_scholar_outcome`, `_unexpected_scholar_exception_outcome`
These become methods on a `ScholarProcessor` class or module-level functions.
**C. `app/services/ingestion/run_completion.py` (~300 lines)**
Move these methods (run finalization + alerting):
- `_classify_failure_bucket` (standalone function)
- `_summarize_failures`, `_build_alert_summary`, `_apply_safety_outcome`, `_finalize_run_record`
- `_resolve_run_status`, `_resolve_continuation_queue_target`, `_build_failure_debug_context`
- `_complete_run_for_user`
These become module-level functions accepting explicit parameters.
**After all extractions, `application.py` should contain only:**
- `ScholarIngestionService.__init__` and config helpers
- Run lifecycle: `initialize_run`, `execute_run`, `run_for_user`
- Scholar iteration orchestration: `_run_scholar_iteration`
- Progress tracking: `_result_counters`, `_adjust_progress_counts`
- Safety gate integration, background task management
Target: `application.py` ≤ 500 lines. No new file exceeds 400 lines. All tests must pass unchanged.
Commit message: `refactor: extract enrichment, scholar processing, and run completion from ingestion service`
---
## Slice 4: Decompose `app/services/scholars/application.py` (996 lines) and `app/services/publications/pdf_queue.py` (969 lines) — DONE
Completed. Extracted four modules from two oversized service files:
| File | Lines | Contents |
|---|---|---|
| `app/services/scholars/author_search_cache.py` | 239 | Serialize/deserialize cache entries, cache get/set/prune |
| `app/services/scholars/author_search.py` | 580 | Cooldown, throttle, retry, circuit breaker, `search_author_candidates` |
| `app/services/scholars/application.py` | 221 | Scholar CRUD only (list, create, get, toggle, delete, image ops) |
| `app/services/publications/pdf_queue_queries.py` | 395 | SQL builders, row hydration, listing/counting/pagination, retry item builders |
| `app/services/publications/pdf_queue_resolution.py` | 311 | Task execution, outcome persistence, scheduling (`schedule_rows`) |
| `app/services/publications/pdf_queue.py` | 364 | Dataclasses, constants, enqueueing logic, public API |
Also updated one test file (`test_publication_pdf_queue_policy.py`) to monkeypatch `pdf_queue_resolution` instead of `pdf_queue` for the 4 resolution-related tests, and updated `publications/application.py` and `publication_identifiers/application.py` imports.
**Prompt:**
You are working on the scholarr repository. Read `agents.md` for project conventions. Two service files exceed the 600-line ceiling. Decompose both.
**A. `app/services/scholars/application.py` (996 lines)**
This file mixes scholar CRUD (~350 lines) with the entire author search pipeline (~650 lines). Split them:
Create `app/services/scholars/author_search_cache.py` (~150 lines): Move all `_serialize_*` and `_deserialize_*` functions. Move `_cache_get_author_search_result`, `_cache_set_author_search_result`, `_prune_author_search_cache`. These are pure functions operating on the DB cache table.
Create `app/services/scholars/author_search.py` (~500 lines): Move remaining author search functions — cooldown management, throttle logic, retry wrappers, circuit breaker, lock acquisition, and `search_author_candidates`. Import cache functions from `author_search_cache.py`. Accept DB session and settings as explicit parameters.
Update `application.py`: Keep only scholar CRUD methods. Import and delegate to `search_author_candidates` from `author_search.py`. Target: ≤ 350 lines.
**B. `app/services/publications/pdf_queue.py` (969 lines)**
Three concerns in one file. Split by concern:
Create `app/services/publications/pdf_queue_queries.py` (~330 lines): Move SQL query builders (`_tracked_queue_select_base`, `_tracked_queue_select`, etc.), result hydration (`_queue_item_from_row`, `_hydrated_queue_items`), listing/counting/pagination (`list_pdf_queue_items`, `count_pdf_queue_items`, `list_pdf_queue_page`), retry item builders and missing-PDF candidate queries.
Create `app/services/publications/pdf_queue_resolution.py` (~300 lines): Move task execution (`_mark_attempt_started`, `_failed_outcome`, `_fetch_outcome_for_row`), outcome persistence (`_apply_publication_update`, `_apply_job_outcome`, `_persist_outcome`), scheduling (`_resolve_publication_row`, `_run_resolution_task`, `_register_task`, `_drop_finished_task`, `_schedule_rows`).
Keep in `pdf_queue.py` (~340 lines): dataclasses, constants, enqueueing logic, public API.
No file should exceed 500 lines. All tests must pass unchanged.
Commit message: `refactor: decompose scholars service and pdf_queue into focused modules`
---
## Slice 5: Decompose router files and scheduler — DONE
Completed. Extracted helpers from three oversized files:
| File | Lines | Contents |
|---|---|---|
| `app/api/routers/run_serializers.py` | 238 | `serialize_run`, `serialize_queue_item`, `normalize_scholar_result`, `manual_run_payload_from_run`, `manual_run_success_payload`, value coercion helpers, idempotency key validation |
| `app/api/routers/run_manual.py` | 237 | `load_safety_state`, `raise_manual_runs_disabled`, `reused_manual_run_payload`, `run_ingestion_for_manual`, `recover_integrity_error`, `execute_manual_run`, safety/failure raise helpers |
| `app/api/routers/runs.py` | 440 | Route handlers only + imports |
| `app/api/routers/scholar_helpers.py` | 234 | `serialize_scholar`, `hydrate_scholar_metadata_if_needed`, `enqueue_initial_scrape_job_for_scholar`, `search_kwargs`, `search_response_data`, `require_user_profile`, `read_uploaded_image` |
| `app/api/routers/scholars.py` | 406 | Route handlers only + imports |
| `app/services/ingestion/queue_runner.py` | 379 | `QueueJobRunner` class — drain continuation queue, job lifecycle (drop/retry/reschedule), ingestion dispatch, finalization |
| `app/services/ingestion/scheduler.py` | 312 | `SchedulerService` — auto-run scheduling, `_run_loop`, `_tick_once`, candidate loading, PDF queue drain |
Also updated one test file (`test_scholars_create_hydration.py`) to monkeypatch `scholar_helpers` instead of the `scholars` router for the 4 helper-related tests.
**Files to create:** `app/api/routers/run_serializers.py`, `app/api/routers/run_manual.py`, `app/api/routers/scholar_helpers.py`, `app/services/ingestion/queue_runner.py`
**Files to modify:** `app/api/routers/runs.py` (875 → ~405), `app/api/routers/scholars.py` (616 → ~396), `app/services/ingestion/scheduler.py` (622 → ~280)
**Prompt:**
You are working on the scholarr repository. Read `agents.md` for project conventions. Three files are slightly over the 600-line ceiling. Extract helpers from each.
**A. `app/api/routers/runs.py` (875 lines) → 3 files**
Create `app/api/routers/run_serializers.py` (~250 lines): Move all `_serialize_*`, `_normalize_*`, `_int_value`, `_str_value`, `_bool_value` helper functions. Move `_manual_run_payload_from_run` and `_manual_run_success_payload`.
Create `app/api/routers/run_manual.py` (~220 lines): Move `_load_safety_state`, `_raise_manual_runs_disabled`, `_reused_manual_run_payload`, `_run_ingestion_for_manual`, `_recover_integrity_error`, `_execute_manual_run`, `_raise_manual_blocked_safety`, `_raise_manual_failed`.
Keep in `runs.py` (~405 lines): only route handler functions + imports.
**B. `app/api/routers/scholars.py` (616 lines) → 2 files**
Create `app/api/routers/scholar_helpers.py` (~220 lines): Move `_needs_metadata_hydration`, `_is_create_hydration_rate_limited`, `_auto_enqueue_new_scholar_enabled`, `_enqueue_initial_scrape_job_for_scholar`, `_uploaded_image_media_path`, `_serialize_scholar`, `_hydrate_scholar_metadata_if_needed`, `_search_kwargs`, `_search_response_data`, `_read_uploaded_image`, `_require_user_profile`.
Keep in `scholars.py` (~396 lines): only route handlers.
**C. `app/services/ingestion/scheduler.py` (622 lines) → 2 files**
Create `app/services/ingestion/queue_runner.py` (~350 lines): Move all continuation-queue job processing — `_drain_continuation_queue`, `_drop_queue_job_if_max_attempts`, `_mark_queue_job_retrying`, `_queue_job_has_available_scholar`, all `_reschedule_*` methods, `_run_ingestion_for_queue_job`, `_finalize_queue_job_after_run`, `_run_queue_job`. Create a `QueueJobRunner` class receiving config params in constructor.
Keep in `scheduler.py` (~280 lines): auto-run scheduling, `_run_loop`, `_tick_once`, candidate loading.
All tests must pass unchanged after each extraction.
Commit message: `refactor: extract helpers from oversized router and scheduler files`
---
## Verification (after all slices)
```bash
# No Python file exceeds 600 lines
find app/ -name "*.py" -exec wc -l {} + | awk '$1 > 600 && !/total/ {print "FAIL:", $0}'
# All tests pass
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app python -m pytest
# Linting
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff check .
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff format --check .
```

628
MODERNIZATION_PLAN.md
View file

@ -1,628 +0,0 @@
# Scholarr Modernization Plan — Slices 2-10
> **Slice 1 is complete** (commit `7736cab`): removed 114 stale `build/` files, fixed `.gitignore`, committed pending deletions, replaced `npm install` with `npm ci` in CI.
>
> Each slice below is a self-contained prompt designed for an LLM executor. Execute in order — each slice depends on the prior ones being complete.
---
## Slice 2: Create `structured_log()` Utility
**Why:** The codebase has 19 `_log_*` helper functions (422 lines) and 138 instances of duplicated `"event"` keys in logging calls. A single utility eliminates all of it.
**Files to create:**
- `app/logging_utils.py`
**Files to modify:**
- `tests/unit/test_logging.py`
**Context files (read but don't modify):**
- `app/logging_config.py` — line 88 has `"event": getattr(record, "event", record.getMessage())` which means when no `event` key is in `extra`, it falls back to `getMessage()`. This is the mechanism that makes `structured_log()` work without duplicating the event name.
- `app/logging_context.py`
**Prompt:**
```
You are working on the scholarr repository. The current logging pattern has a DRY problem:
1. Every log call duplicates the event name as both the message and in extra["event"]:
`logger.info("event.name", extra={"event": "event.name", "key": value})`
2. There are 19 `_log_*` helper functions (422 lines total) that just wrap logger calls with typed signatures to build extra dicts.
3. Metrics data (metric_name/metric_value) is mixed into log extra dicts but has no consumer.
Create `app/logging_utils.py`:
```python
"""Structured logging utility — eliminates boilerplate across domain services."""
from __future__ import annotations
import logging
from typing import Any
def structured_log(
logger: logging.Logger,
level: str,
event: str,
/,
**fields: Any,
) -> None:
"""Emit a structured log entry.
The event name is passed as the log message. The JsonLogFormatter in
logging_config.py extracts it via record.getMessage() when no explicit
'event' key exists in extra — so we do NOT duplicate it.
Usage:
structured_log(logger, "info", "ingestion.run_started", user_id=1, scholar_count=5)
"""
fields.pop("metric_name", None)
fields.pop("metric_value", None)
log_method = getattr(logger, level.lower())
log_method(event, extra=fields)
```
Verify that `app/logging_config.py` line 88 already handles this correctly — when no `event` key is in extra, it falls back to getMessage() which returns the event string passed as the first argument. No changes to logging_config.py should be needed.
Add tests in `tests/unit/test_logging.py` (append to existing file):
- Test that `structured_log()` produces a log record where the JsonLogFormatter outputs the event name correctly
- Test that `structured_log()` works with ConsoleLogFormatter
- Test that metric_name/metric_value fields are stripped from output
- Test that extra fields (user_id, scholar_id, etc.) appear in the formatted output
Commit message: "refactor: add structured_log utility to eliminate logging boilerplate"
```
---
## Slice 3: Migrate Ingestion Logging to `structured_log()`
**Why:** `app/services/ingestion/application.py` is 3,089 lines. 8 `_log_*` helpers consume ~239 lines. 34 inline logger calls duplicate event names and include dead metric fields. This slice removes ~300 lines.
**Files to modify:**
- `app/services/ingestion/application.py`
**Context files:**
- `app/logging_utils.py` (created in Slice 2)
**Prompt:**
```
You are working on the scholarr repository. The file `app/services/ingestion/application.py` (3,089 lines) has 8 `_log_*` helper functions consuming ~239 lines, plus 34 inline logger calls with duplicated `"event":` keys and mixed `metric_name`/`metric_value` fields.
Migrate ALL logging in this file to use `structured_log()` from `app.logging_utils`.
1. Add `from app.logging_utils import structured_log` at the top.
2. Delete these 8 helper methods entirely:
- `_log_request_delay_coercion` (line ~123, 21 lines)
- `_log_run_started` (line ~310, 36 lines)
- `_log_scholar_parsed` (line ~383, 28 lines)
- `_log_alert_thresholds` (line ~1013, 47 lines)
- `_log_safety_transition` (line ~1093, 42 lines)
- `_log_run_completed` (line ~1178, 28 lines)
- `_attempt_log_entry` (line ~1837, 16 lines)
- `_page_log_entry` (line ~1999, 21 lines)
3. Replace every call to these deleted helpers with an inline `structured_log()` call. Example:
Before:
```python
self._log_run_started(
user_id=user_id,
trigger_type=trigger_type,
scholar_count=scholar_count,
...
)
```
After:
```python
structured_log(
logger, "info", "ingestion.run_started",
user_id=user_id,
trigger_type=trigger_type.value,
scholar_count=scholar_count,
...
)
```
4. For ALL remaining inline `logger.info/warning/debug/exception()` calls:
- Replace with `structured_log()` where the call uses `extra={"event": ..., ...}` pattern
- Do NOT convert `logger.exception()` calls — keep them but remove the duplicated `"event"` key from their extra dict
- Remove `metric_name` and `metric_value` from all calls
5. Do NOT change any business logic. Only logging call sites.
Expected outcome: ~300 lines removed. All existing tests must pass unchanged.
Commit message: "refactor: migrate ingestion service logging to structured_log"
```
---
## Slice 4: Migrate All Remaining Services to `structured_log()`
**Why:** 11 more `_log_*` helpers remain across 8 files (183 lines total), plus inline logger calls with the same duplication pattern in ~20 files.
**Files to modify (all have `_log_*` helpers to delete):**
- `app/services/ingestion/scheduler.py` — `_log_queue_item_resolved` (line ~518, 21 lines)
- `app/services/arxiv/rate_limit.py` — `_log_request_scheduled` (199), `_log_request_completed` (216), `_log_cooldown_activated` (235)
- `app/services/arxiv/client.py` — `_log_cache_event` (283), `_log_request_skipped_for_cooldown` (299)
- `app/services/publications/pdf_resolution_pipeline.py` — `_log_arxiv_skip` (148)
- `app/services/unpaywall/application.py` — `_log_resolution_summary` (194)
- `app/api/routers/publications.py` — `_log_retry_pdf_result` (343)
- `app/api/routers/settings.py` — `_log_settings_update` (103)
- `app/main.py` — `_log_startup_build_marker` (53)
**Files to modify (inline `extra={"event":...}` pattern only — no helpers to delete):**
- `app/http/middleware.py`
- `app/db/session.py`
- `app/auth/runtime.py`
- `app/security/csrf.py`
- `app/api/routers/scholars.py`
- `app/api/routers/runs.py`
- `app/api/routers/admin_dbops.py`
- `app/api/routers/admin.py`
- `app/api/routers/auth.py`
- `app/api/routers/publications.py`
- `app/services/scholars/application.py`
- `app/services/runs/events.py`
- `app/services/openalex/client.py`
- `app/services/openalex/matching.py`
- `app/services/crossref/application.py`
- `app/services/publications/dedup.py`
- `app/services/publications/enrichment.py`
- `app/services/publications/pdf_queue.py`
- `app/services/scholar/source.py`
- `app/services/arxiv/gateway.py`
**Prompt:**
```
You are working on the scholarr repository. Slice 3 migrated `ingestion/application.py` to `structured_log()`. Now do the same for ALL remaining files.
1. Delete these `_log_*` helper functions and replace their call sites with inline `structured_log()`:
- `app/services/ingestion/scheduler.py:~518` — `_log_queue_item_resolved` (21 lines)
- `app/services/arxiv/rate_limit.py:~199` — `_log_request_scheduled` (17 lines)
- `app/services/arxiv/rate_limit.py:~216` — `_log_request_completed` (19 lines)
- `app/services/arxiv/rate_limit.py:~235` — `_log_cooldown_activated` (13 lines)
- `app/services/arxiv/client.py:~283` — `_log_cache_event` (16 lines)
- `app/services/arxiv/client.py:~299` — `_log_request_skipped_for_cooldown` (13 lines)
- `app/services/publications/pdf_resolution_pipeline.py:~148` — `_log_arxiv_skip` (11 lines)
- `app/services/unpaywall/application.py:~194` — `_log_resolution_summary` (21 lines)
- `app/api/routers/publications.py:~343` — `_log_retry_pdf_result` (24 lines)
- `app/api/routers/settings.py:~103` — `_log_settings_update` (15 lines)
- `app/main.py:~53` — `_log_startup_build_marker` (13 lines)
2. In ALL files under `app/` that have inline `logger.info/warning/debug("event.name", extra={"event": "event.name", ...})` calls:
- Replace with `structured_log(logger, "level", "event.name", key=value, ...)`
- Add `from app.logging_utils import structured_log` import
- Remove `metric_name`/`metric_value` from all calls
- Keep `logger.exception()` calls but remove the duplicated `"event"` key from their extra dicts
3. In `app/services/openalex/client.py`, the lines that log `response.text` (lines ~87, ~137):
- Truncate: `response.text[:500]`
4. Do NOT change any business logic. Only logging call sites.
Verify: `grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__` should return 0 matches.
Verify: `grep -rn 'def _log_' app/ --include="*.py"` should return 0 matches.
Commit message: "refactor: migrate all remaining logging to structured_log, remove boilerplate helpers"
```
---
## Slice 5: Logging Relevance Audit — Reduce Noise, Improve Readability
**Why:** Self-hosted users need clear, actionable logs. Current logging has verbose debug noise (per-HTTP-request, per-publication), overly long event names, and no console display optimization.
**Files to modify:**
- `app/services/scholar/source.py`
- `app/services/ingestion/application.py`
- `app/logging_config.py`
- Any files with event names longer than ~40 chars
**Prompt:**
```
You are working on the scholarr repository. The logging has been migrated to `structured_log()` (slices 2-4). Now audit log RELEVANCE and READABILITY for self-hosted users running this in Docker.
1. **Remove redundant debug logs:**
- `app/services/scholar/source.py`: Remove the 3 debug-level HTTP fetch events (fetch_started, search_fetch_started, publication_fetch_started). Keep only `fetch_succeeded` at DEBUG. The HTTP middleware already logs request.started/completed.
- `app/services/ingestion/application.py`: Remove per-publication `publication.discovered` and `publication.created` DEBUG logs. The run_completed summary already reports totals.
2. **Shorten overly long event names** (search all structured_log calls in app/):
- `ingestion.request_delay_coerced_to_policy_floor` → `ingestion.delay_coerced`
- `ingestion.safety_cooldown_cleared` → `ingestion.cooldown_cleared`
- Any other names longer than ~40 chars — shorten while preserving meaning
3. **Improve console readability** in `app/logging_config.py` `ConsoleLogFormatter.format()`:
- Add a short-name mapping for common context fields in console output ONLY (JSON keeps full names):
```python
_CONSOLE_SHORT_KEYS = {
"user_id": "user",
"scholar_id": "scholar",
"crawl_run_id": "run",
"run_id": "run",
}
```
- Apply in the `for key in sorted(payload.keys())` loop
4. Do NOT remove any WARNING or ERROR level logs. Only remove/demote DEBUG/INFO that are redundant.
5. Do NOT change business logic.
Commit message: "refactor: reduce log noise, improve event naming and console readability"
```
---
## Slice 6: Add Ruff + Mypy to CI
**Why:** No Python linting or type checking in CI. Code quality regressions go undetected.
**Files to modify:**
- `pyproject.toml`
- `.github/workflows/ci.yml`
**Prompt:**
```
You are working on the scholarr repository. Add Python linting and type checking to CI.
1. In `pyproject.toml`, add ruff configuration:
```toml
[tool.ruff]
target-version = "py312"
line-length = 120
[tool.ruff.lint]
select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
ignore = ["E501"]
[tool.ruff.lint.isort]
known-first-party = ["app"]
```
2. In `pyproject.toml`, add mypy configuration:
```toml
[tool.mypy]
python_version = "3.12"
ignore_missing_imports = true
check_untyped_defs = false
warn_unused_ignores = true
```
3. In `.github/workflows/ci.yml`, add a `lint` job after `repo-hygiene` and before `test`:
```yaml
lint:
runs-on: ubuntu-latest
needs: [repo-hygiene]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- uses: astral-sh/setup-uv@v4
- run: uv sync --extra dev
- name: Ruff check
run: uv run ruff check .
- name: Ruff format check
run: uv run ruff format --check .
- name: Mypy
run: uv run mypy app/ --ignore-missing-imports
```
4. Update the `test` job's `needs` to include `lint`.
5. Add `ruff` and `mypy` to dev dependencies in `pyproject.toml` under `[project.optional-dependencies]`.
6. Run `ruff check .` locally and fix auto-fixable issues with `ruff check --fix .`. For unfixable issues, add targeted `noqa` comments only if fixing would change behavior.
Commit message: "ci: add ruff linting and mypy type checking"
```
---
## Slice 7: Add CodeQL + Dependabot
**Why:** No security scanning. Missed vulnerabilities, no automated dependency updates.
**Files to create:**
- `.github/workflows/codeql.yml`
- `.github/dependabot.yml`
**Prompt:**
```
You are working on the scholarr repository. Add security scanning.
1. Create `.github/workflows/codeql.yml`:
```yaml
name: CodeQL
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: "30 5 * * 1"
jobs:
analyze:
runs-on: ubuntu-latest
permissions:
security-events: write
contents: read
strategy:
matrix:
language: [python, javascript]
steps:
- uses: actions/checkout@v4
- uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
- uses: github/codeql-action/autobuild@v3
- uses: github/codeql-action/analyze@v3
```
2. Create `.github/dependabot.yml`:
```yaml
version: 2
updates:
- package-ecosystem: pip
directory: /
schedule:
interval: weekly
- package-ecosystem: npm
directory: /frontend
schedule:
interval: weekly
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
```
Commit message: "ci: add CodeQL security scanning and Dependabot"
```
---
## Slice 8: Fix Version Remnants + Adopt Semantic Release
**Why:** Version `0.1.0` is hardcoded in 5 places (including `crossref/application.py:294`). `0.0.1` exists in `docs/website/package.json`. No CHANGELOG, no GitHub Releases.
**Files to modify:**
- `pyproject.toml`
- `app/services/crossref/application.py`
**Files to create:**
- `.github/workflows/release.yml`
**Prompt:**
```
You are working on the scholarr repository. Set up automated versioning with python-semantic-release.
1. Fix hardcoded version in `app/services/crossref/application.py` (line ~294):
```python
# Before:
etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
# After:
from importlib.metadata import version as pkg_version
_APP_VERSION = pkg_version("scholarr")
# then in function:
etiquette = Etiquette(settings.app_name, _APP_VERSION, "https://scholarr.local", email)
```
2. In `pyproject.toml`, add semantic-release config:
```toml
[tool.semantic_release]
version_toml = ["pyproject.toml:project.version"]
version_variables = ["frontend/package.json:version"]
branch = "main"
build_command = ""
commit_message = "chore(release): v{version}"
```
3. Create `.github/workflows/release.yml`:
```yaml
name: Release
on:
push:
branches: [main]
jobs:
release:
runs-on: ubuntu-latest
if: github.repository == 'JustinZeus/scholarr'
permissions:
contents: write
id-token: write
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install python-semantic-release
- name: Semantic Release
id: release
run: semantic-release publish
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```
4. Add `python-semantic-release` to dev dependencies in `pyproject.toml`.
Note: `docs/website/package.json` version (`0.0.1`) is addressed in Slice 9 (docs rebuild). `frontend/package-lock.json` and `uv.lock` versions auto-update on next `npm install`/`uv sync`.
Commit message: "ci: adopt python-semantic-release, fix hardcoded version strings"
```
---
## Slice 9: Docs — Delete and Rebuild from Scratch
**Why:** Current docs have split authority (two api-contract.md files), orphaned pages, stale version (`0.0.1`), and missing critical docs (testing, deployment, configuration reference). Clean slate is faster than migration.
**Files to delete:**
- `docs/` (entire directory)
**Files to create:**
- Complete new `docs/` tree
**Prompt:**
```
You are working on the scholarr repository. Delete the entire `docs/` directory and rebuild from scratch.
Target IA:
docs/
├── index.md # Landing page: what is scholarr, quick links
├── user/
│ ├── overview.md # What scholarr does, key concepts
│ ├── getting-started.md # Install (docker compose), first run, add first scholar
│ └── configuration.md # ALL env vars from .env.example, organized by category
├── developer/
│ ├── overview.md # Dev quickstart
│ ├── architecture.md # FastAPI + SQLAlchemy + Vue 3, domain service boundaries
│ ├── local-development.md # docker-compose.dev.yml, hot reload, running tests
│ ├── contributing.md # PR process, conventional commits, code standards from agents.md
│ ├── ingestion.md # Ingestion pipeline: parsing, rate limiting, safety gates
│ ├── frontend-theme-inventory.md # Theme tokens, Tailwind integration
│ └── testing.md # Test tiers (unit/integration/smoke), markers, fixtures, how to run
├── operations/
│ ├── overview.md # Ops quickstart
│ ├── deployment.md # Production Docker, scaling, health checks
│ ├── database-runbook.md # Backup, restore, integrity checks, migration procedures
│ ├── scrape-safety-runbook.md # Rate limiting, cooldowns, CAPTCHA handling
│ └── arxiv-runbook.md # ArXiv rate limits, cache, query patterns
├── reference/
│ ├── overview.md # Reference index
│ ├── api.md # CANONICAL: envelope spec + endpoints + DTO contract
│ ├── environment.md # Env var quick-reference (links to user/configuration.md)
│ └── changelog.md # Placeholder: "auto-generated by semantic-release"
└── website/
├── docusaurus.config.js
├── sidebars.js # ALL pages listed, no orphans
├── package.json # version: "0.1.0"
└── src/css/custom.css
Content guidelines:
- Self-contained, scannable docs (headers, bullets, tables)
- Frontmatter: `title`, `sidebar_position`
- `configuration.md`: read `.env.example`, organize every variable with description, type, default, example
- `testing.md`: pytest markers from pyproject.toml (`integration`, `db`, `migrations`, `schema`, `smoke`), container-based runner from agents.md, fixture org under tests/fixtures/
- `api.md`: merge old `developer/api-contract.md` (DTO structure) and `reference/api-contract.md` (envelope spec). Include exact envelope shapes:
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
- Publication semantics (modes, pagination, identifiers)
- Scholar portability endpoints
- `deployment.md`: practical Docker commands from docker-compose.yml
- `database-runbook.md`: reference scripts/db/ (backup_full.sh, restore_dump.sh, check_integrity.py, repair_publication_links.py)
Docusaurus config:
- url: "https://justinzeus.github.io"
- baseUrl: "/scholarr/"
- organizationName: "JustinZeus"
- projectName: "scholarr"
- onBrokenLinks: "throw"
- docs path: ".." (reads from docs/ parent)
- Exclude: "website/**", "README.md"
Commit message: "docs: rebuild documentation from scratch with clean IA"
```
---
## Slice 10: Begin `ingestion/application.py` Decomposition
**Why:** After logging cleanup (slices 3+5), the file is ~2,700-2,800 lines. Still the largest source file by 3x. This begins decomposition with the safest extract: safety gate logic.
**Files to modify:**
- `app/services/ingestion/application.py`
**Files to create:**
- `app/services/ingestion/safety.py`
**Prompt:**
```
You are working on the scholarr repository. `app/services/ingestion/application.py` is ~2,700-2,800 lines (after logging cleanup). It contains the entire ingestion orchestration in a single class.
Extract the FIRST logical chunk: safety gate logic.
1. Read `app/services/ingestion/application.py` fully.
2. Identify safety/cooldown methods:
- `_enforce_safety_gate`
- `_raise_safety_blocked_start`
- Cooldown activation/clearing methods
- Alert threshold evaluation methods
3. Create `app/services/ingestion/safety.py`:
- Move identified methods to standalone functions or a small class
- Accept explicit parameters (no implicit self.xxx state)
- Keep same function signatures where possible
4. Update `application.py` to import from `safety.py`.
5. Run full test suite: `docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/`
All tests must pass unchanged.
Constraints from `agents.md`:
- Max 50 lines per function
- No flat files in `app/services/` root — all within `app/services/ingestion/`
- Fail fast, early returns, guard clauses
Commit message: "refactor: extract ingestion safety gate logic to dedicated module"
```
---
## Verification (after all slices)
Run these checks to confirm everything is clean:
```bash
# 1. No tracked build artifacts
git ls-files build/ | wc -l # expect: 0
# 2. No duplicated event keys in logging
grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__ | wc -l # expect: 0
# 3. No _log_ helper functions remain
grep -rn 'def _log_' app/ --include="*.py" | wc -l # expect: 0
# 4. No stale version strings
grep -rn '0\.0\.1' . --include="*.py" --include="*.json" --include="*.toml" \
| grep -v node_modules | grep -v .venv | grep -v uv.lock | wc -l # expect: 0
# 5. Docs build
npm --prefix docs/website run build # expect: success, no broken links
# 6. All tests pass
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/
```
---
## Slice Summary
| # | Slice | Files Changed | Estimated Impact |
|---|-------|---------------|------------------|
| ~~1~~ | ~~Repo hygiene~~ | ~~120 files~~ | ~~-18,380 lines~~ **DONE** |
| 2 | `structured_log()` utility | 2 files | +50 lines |
| 3 | Migrate ingestion logging | 1 file | -300 lines |
| 4 | Migrate remaining logging | ~20 files | -180 lines |
| 5 | Logging noise/readability | ~5 files | -30 lines, better output |
| 6 | Ruff + mypy in CI | 2 files | +50 lines config |
| 7 | CodeQL + Dependabot | 2 files | +40 lines config |
| 8 | Version fix + semantic-release | 3 files | +40 lines config |
| 9 | Docs rebuild | full replacement | ~20 new files |
| 10 | Ingestion decomposition | 2 files | net zero (move) |

2
agents.md
View file

@ -77,7 +77,7 @@ Types: `feat`, `fix`, `docs`, `ci`, `refactor`, `test`, `chore`, `perf`.
## 7. Testing & Quality Gates
All commands run inside containers. Do not run bare `npm`, `pytest`, or `ruff` on the host.
All **local development** commands run inside containers. Do not run bare `npm`, `pytest`, or `ruff` on the host. CI workflows (`.github/workflows/`) use bare runners with `uv` and `npm` directly — this is intentional since CI has no Docker-in-Docker dependency.
### Backend

18
app/api/routers/run_manual.py
View file

@ -138,9 +138,11 @@ async def recover_integrity_error(
original_exc: IntegrityError,
) -> dict[str, Any]:
if idempotency_key is None:
logger.exception(
structured_log(
logger,
"exception",
"api.runs.manual_integrity_error",
extra={"user_id": user_id},
user_id=user_id,
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
existing_run = await run_service.get_manual_run_by_idempotency_key(
@ -149,9 +151,11 @@ async def recover_integrity_error(
idempotency_key=idempotency_key,
)
if existing_run is None:
logger.exception(
structured_log(
logger,
"exception",
"api.runs.manual_integrity_error",
extra={"user_id": user_id},
user_id=user_id,
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from original_exc
if existing_run.status in (RunStatus.RUNNING, RunStatus.RESOLVING):
@ -230,8 +234,10 @@ def raise_manual_blocked_safety(*, exc, user_id: int) -> None:
def raise_manual_failed(*, exc: Exception, user_id: int) -> None:
logger.exception(
structured_log(
logger,
"exception",
"api.runs.manual_failed",
extra={"user_id": user_id},
user_id=user_id,
)
raise ApiException(status_code=500, code="manual_run_failed", message="Manual run failed.") from exc

31
app/api/routers/runs.py
View file

@ -37,7 +37,8 @@ from app.api.schemas import (
RunsListEnvelope,
)
from app.db.models import RunStatus, RunTriggerType, User
from app.db.session import get_db_session
from app.db.session import get_db_session, get_session_factory
from app.logging_utils import structured_log
from app.services.ingestion import application as ingestion_service
from app.services.runs import application as run_service
from app.services.runs.events import event_generator
@ -47,6 +48,14 @@ from app.settings import settings
logger = logging.getLogger(__name__)
_background_tasks: set[asyncio.Task[Any]] = set()
def _drop_finished_task(task: asyncio.Task[Any]) -> None:
_background_tasks.discard(task)
try:
task.result()
except Exception:
structured_log(logger, "exception", "runs.background_task_failed")
router = APIRouter(prefix="/runs", tags=["api-runs"])
ACTIVE_RUN_STATUSES = {RunStatus.RUNNING, RunStatus.RESOLVING}
@ -234,7 +243,7 @@ def _spawn_background_execution(
)
)
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
task.add_done_callback(_drop_finished_task)
@router.post(
@ -247,15 +256,16 @@ async def run_manual(
current_user: User = Depends(get_api_current_user),
ingest_service: ingestion_service.ScholarIngestionService = Depends(get_ingestion_service),
):
safety_state = await load_safety_state(db_session, user_id=current_user.id)
user_id = int(current_user.id)
safety_state = await load_safety_state(db_session, user_id=user_id)
if not settings.ingestion_manual_run_allowed:
raise_manual_runs_disabled(user_id=current_user.id, safety_state=safety_state)
raise_manual_runs_disabled(user_id=user_id, safety_state=safety_state)
idempotency_key = normalize_idempotency_key(request.headers.get(IDEMPOTENCY_HEADER))
reused_payload = await reused_manual_run_payload(
db_session,
request=request,
user_id=current_user.id,
user_id=user_id,
idempotency_key=idempotency_key,
safety_state=safety_state,
)
@ -291,12 +301,12 @@ async def run_manual(
"new_publication_count": 0,
"reused_existing_run": False,
"idempotency_key": idempotency_key,
"safety_state": await load_safety_state(db_session, user_id=current_user.id),
"safety_state": await load_safety_state(db_session, user_id=user_id),
},
)
except ingestion_service.RunBlockedBySafetyPolicyError as exc:
await db_session.rollback()
raise_manual_blocked_safety(exc=exc, user_id=current_user.id)
raise_manual_blocked_safety(exc=exc, user_id=user_id)
except ingestion_service.RunAlreadyInProgressError as exc:
await db_session.rollback()
raise ApiException(
@ -309,13 +319,13 @@ async def run_manual(
return await recover_integrity_error(
db_session,
request=request,
user_id=current_user.id,
user_id=user_id,
idempotency_key=idempotency_key,
original_exc=exc,
)
except Exception as exc:
await db_session.rollback()
raise_manual_failed(exc=exc, user_id=current_user.id)
raise_manual_failed(exc=exc, user_id=user_id)
@router.get(
@ -454,9 +464,10 @@ async def clear_queue_item(
@router.get("/{run_id}/stream")
async def stream_run_events(
run_id: int,
db_session: AsyncSession = Depends(get_db_session),
current_user: User = Depends(get_api_current_user),
):
session_factory = get_session_factory()
async with session_factory() as db_session:
run = await run_service.get_run_for_user(
db_session,
user_id=current_user.id,

6
app/api/schemas/admin.py
View file

@ -34,8 +34,8 @@ class AdminUsersListEnvelope(BaseModel):
class AdminUserCreateRequest(BaseModel):
email: str
password: str
email: str = Field(max_length=254)
password: str = Field(max_length=128)
is_admin: bool = False
model_config = ConfigDict(extra="forbid")
@ -55,7 +55,7 @@ class AdminUserEnvelope(BaseModel):
class AdminResetPasswordRequest(BaseModel):
new_password: str
new_password: str = Field(max_length=128)
model_config = ConfigDict(extra="forbid")

12
app/api/schemas/auth.py
View file

@ -1,6 +1,6 @@
from __future__ import annotations
from pydantic import BaseModel, ConfigDict
from pydantic import BaseModel, ConfigDict, Field
from app.api.schemas.common import ApiMeta
@ -44,8 +44,8 @@ class CsrfBootstrapEnvelope(BaseModel):
class LoginRequest(BaseModel):
email: str
password: str
email: str = Field(max_length=254)
password: str = Field(max_length=128)
model_config = ConfigDict(extra="forbid")
@ -66,8 +66,8 @@ class LoginEnvelope(BaseModel):
class ChangePasswordRequest(BaseModel):
current_password: str
new_password: str
confirm_password: str
current_password: str = Field(max_length=128)
new_password: str = Field(max_length=128)
confirm_password: str = Field(max_length=128)
model_config = ConfigDict(extra="forbid")

2
app/db/session.py
View file

@ -85,7 +85,7 @@ async def check_database() -> bool:
result = await conn.execute(text("SELECT 1"))
return result.scalar_one() == 1
except Exception:
logger.exception("db.healthcheck_failed")
structured_log(logger, "exception", "db.healthcheck_failed")
return False

12
app/http/middleware.py
View file

@ -74,13 +74,13 @@ class RequestLoggingMiddleware(BaseHTTPMiddleware):
response = await call_next(request)
except Exception:
duration_ms = int((time.perf_counter() - start) * 1000)
logger.exception(
structured_log(
logger,
"exception",
"request.failed",
extra={
"method": request.method,
"path": request.url.path,
"duration_ms": duration_ms,
},
method=request.method,
path=request.url.path,
duration_ms=duration_ms,
)
raise
else:

9
app/main.py
View file

@ -73,8 +73,13 @@ async def lifespan(_: FastAPI):
)
await session.commit()
structured_log(logger, "info", "app.startup_orphaned_runs_cleaned")
except Exception as e:
logger.error(f"Failed to clean orphaned runs at startup: {e}")
except Exception as exc:
structured_log(
logger,
"error",
"app.startup_orphaned_runs_cleanup_failed",
error=str(exc),
)
await scheduler_service.start()
yield

24
app/security/csrf.py
View file

@ -6,9 +6,10 @@ from urllib.parse import parse_qs
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse, PlainTextResponse, Response
from starlette.responses import PlainTextResponse, Response
from starlette.types import Message
from app.api.responses import error_response
from app.logging_utils import structured_log
CSRF_SESSION_KEY = "csrf_token"
@ -88,7 +89,10 @@ class CSRFMiddleware(BaseHTTPMiddleware):
content_type = request.headers.get("content-type", "")
if not content_type.startswith("application/x-www-form-urlencoded"):
return None
try:
parsed = parse_qs(body.decode("utf-8"), keep_blank_values=True)
except (UnicodeDecodeError, ValueError):
return None
values = parsed.get(CSRF_FORM_FIELD)
if not values:
return None
@ -114,19 +118,11 @@ class CSRFMiddleware(BaseHTTPMiddleware):
message: str,
) -> Response:
if request.url.path.startswith("/api/"):
request_state = getattr(request, "state", None)
request_id = getattr(request_state, "request_id", None) if request_state else None
return JSONResponse(
{
"error": {
"code": code,
"message": message,
"details": None,
},
"meta": {
"request_id": request_id,
},
},
return error_response(
request,
status_code=403,
code=code,
message=message,
details=None,
)
return PlainTextResponse(message, status_code=403)

9
app/services/arxiv/rate_limit.py
View file

@ -75,7 +75,14 @@ async def _run_serialized_fetch(
runtime_state,
source_path=source_path,
)
runtime_state.next_allowed_at = datetime.now(UTC) + timedelta(seconds=_min_interval_seconds())
if wait_seconds > 0:
await asyncio.sleep(wait_seconds)
response = await fetch()
async with session_factory() as db_session, db_session.begin():
runtime_state = await _load_runtime_state_for_update(db_session)
hit_rate_limit = _record_post_response_state(
runtime_state,
response_status=int(response.status_code),
@ -144,8 +151,6 @@ async def _wait_for_allowed_slot_or_raise(
source_path=source_path,
cooldown_remaining_seconds=0.0,
)
if wait_seconds > 0:
await asyncio.sleep(wait_seconds)
return wait_seconds

3
app/services/crossref/application.py
View file

@ -350,7 +350,8 @@ async def _fetch_items(
),
timeout=timeout,
)
except Exception:
except Exception as exc:
structured_log(logger, "warning", "crossref.fetch_failed", error=str(exc))
return []

10
app/services/dbops/near_duplicate_repair.py
View file

@ -6,12 +6,14 @@ from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import DataRepairJob
from app.services.dbops.application import (
REPAIR_STATUS_COMPLETED,
REPAIR_STATUS_FAILED,
REPAIR_STATUS_PLANNED,
REPAIR_STATUS_RUNNING,
)
from app.services.publications import dedup as dedup_service
REPAIR_STATUS_PLANNED = "planned"
REPAIR_STATUS_RUNNING = "running"
REPAIR_STATUS_COMPLETED = "completed"
REPAIR_STATUS_FAILED = "failed"
NEAR_DUP_JOB_NAME = "repair_publication_near_duplicates"
NEAR_DUP_DEFAULT_MAX_CLUSTERS = 25

255
app/services/ingestion/application.py
View file

@ -1,6 +1,5 @@
from __future__ import annotations
import asyncio
import logging
from datetime import UTC, datetime
from typing import Any
@ -17,21 +16,28 @@ from app.db.models import (
)
from app.logging_utils import structured_log
from app.services.ingestion import queue as queue_service
from app.services.ingestion import safety as run_safety_service
from app.services.ingestion.constants import RUN_LOCK_NAMESPACE
from app.services.ingestion.enrichment import EnrichmentRunner
from app.services.ingestion.pagination import PaginationEngine
from app.services.ingestion.preflight import check_scholar_reachable
from app.services.ingestion.run_completion import (
complete_run_for_user,
int_or_default,
log_run_completed,
run_execution_summary,
)
from app.services.ingestion.run_enrichment import (
inline_enrich_and_finalize,
spawn_background_enrichment_task,
)
from app.services.ingestion.run_guards import (
load_user_settings_for_run,
run_preflight_guard,
)
from app.services.ingestion.scholar_processing import run_scholar_iteration
from app.services.ingestion.types import (
RunAlertSummary,
RunAlreadyInProgressError,
RunBlockedBySafetyPolicyError,
RunBlockedBySafetyPolicyError, # noqa: F401 (re-exported)
RunFailureSummary,
RunProgress,
)
@ -42,8 +48,6 @@ from app.settings import settings
logger = logging.getLogger(__name__)
ACTIVE_RUN_INDEX_NAME = "uq_crawl_runs_user_active"
_background_tasks: set[asyncio.Task[Any]] = set()
def _is_active_run_integrity_error(exc: IntegrityError) -> bool:
original_error = getattr(exc, "orig", None)
@ -97,34 +101,6 @@ def _threshold_kwargs(
}
def _log_run_completed(
*,
run: CrawlRun,
user_id: int,
scholars: list[ScholarProfile],
progress: RunProgress,
failure_summary: RunFailureSummary,
alert_summary: RunAlertSummary,
) -> None:
structured_log(
logger,
"info",
"ingestion.run_completed",
user_id=user_id,
crawl_run_id=run.id,
status=run.status.value,
scholar_count=len(scholars),
succeeded_count=progress.succeeded_count,
failed_count=progress.failed_count,
partial_count=progress.partial_count,
new_publication_count=run.new_pub_count,
blocked_failure_count=alert_summary.blocked_failure_count,
network_failure_count=alert_summary.network_failure_count,
retries_scheduled_count=failure_summary.retries_scheduled_count,
alert_flags=alert_summary.alert_flags,
)
class ScholarIngestionService:
def __init__(self, *, source: ScholarSource) -> None:
self._source = source
@ -138,78 +114,6 @@ class ScholarIngestionService:
)
return max(policy_minimum, int_or_default(value, policy_minimum))
async def _load_user_settings_for_run(
self,
db_session: AsyncSession,
*,
user_id: int,
trigger_type: RunTriggerType,
):
user_settings = await user_settings_service.get_or_create_settings(db_session, user_id=user_id)
await self._enforce_safety_gate(
db_session, user_settings=user_settings, user_id=user_id, trigger_type=trigger_type
)
return user_settings
async def _enforce_safety_gate(
self,
db_session: AsyncSession,
*,
user_settings,
user_id: int,
trigger_type: RunTriggerType,
) -> None:
now_utc = datetime.now(UTC)
previous = run_safety_service.get_safety_event_context(user_settings, now_utc=now_utc)
if run_safety_service.clear_expired_cooldown(user_settings, now_utc=now_utc):
await db_session.commit()
await db_session.refresh(user_settings)
structured_log(
logger,
"info",
"ingestion.cooldown_cleared",
user_id=user_id,
reason=previous.get("cooldown_reason"),
cooldown_until=previous.get("cooldown_until"),
)
now_utc = datetime.now(UTC)
if run_safety_service.is_cooldown_active(user_settings, now_utc=now_utc):
await self._raise_safety_blocked_start(
db_session,
user_settings=user_settings,
user_id=user_id,
trigger_type=trigger_type,
now_utc=now_utc,
)
async def _raise_safety_blocked_start(
self,
db_session: AsyncSession,
*,
user_settings,
user_id: int,
trigger_type: RunTriggerType,
now_utc: datetime,
) -> None:
safety_state = run_safety_service.register_cooldown_blocked_start(user_settings, now_utc=now_utc)
await db_session.commit()
structured_log(
logger,
"warning",
"ingestion.safety_policy_blocked_run_start",
user_id=user_id,
trigger_type=trigger_type.value,
reason=safety_state.get("cooldown_reason"),
cooldown_until=safety_state.get("cooldown_until"),
cooldown_remaining_seconds=safety_state.get("cooldown_remaining_seconds"),
blocked_start_count=((safety_state.get("counters") or {}).get("blocked_start_count")),
)
raise RunBlockedBySafetyPolicyError(
code="scrape_cooldown_active",
message="Scrape safety cooldown is active; run start is temporarily blocked.",
safety_state=safety_state,
)
async def _load_target_scholars(
self,
db_session: AsyncSession,
@ -232,49 +136,6 @@ class ScholarIngestionService:
await queue_service.clear_job_for_scholar(db_session, user_id=user_id, scholar_profile_id=sid)
return scholars
async def _run_preflight_guard(
self,
db_session: AsyncSession,
*,
user_settings: Any,
user_id: int,
scholars: list[ScholarProfile],
) -> None:
if not scholars:
return
result = await check_scholar_reachable(
self._source,
scholar_id=scholars[0].scholar_id,
)
if result.passed:
return
now_utc = datetime.now(UTC)
safety_state, _ = run_safety_service.apply_run_safety_outcome(
user_settings,
run_id=0,
blocked_failure_count=1,
network_failure_count=0,
blocked_failure_threshold=1,
network_failure_threshold=1,
blocked_cooldown_seconds=settings.ingestion_safety_cooldown_blocked_seconds,
network_cooldown_seconds=settings.ingestion_safety_cooldown_network_seconds,
now_utc=now_utc,
)
await db_session.commit()
structured_log(
logger,
"warning",
"ingestion.cooldown_entered_preflight",
user_id=user_id,
block_reason=result.block_reason,
cooldown_until=safety_state.get("cooldown_until"),
)
raise RunBlockedBySafetyPolicyError(
code="scrape_cooldown_active",
message=f"Preflight detected Scholar block ({result.block_reason}). Cooldown activated.",
safety_state=safety_state,
)
async def _initialize_run_for_user(
self,
db_session: AsyncSession,
@ -287,7 +148,7 @@ class ScholarIngestionService:
threshold_kwargs: dict[str, Any],
idempotency_key: str | None,
) -> tuple[Any, CrawlRun, list[ScholarProfile], dict[int, int]]:
user_settings = await self._load_user_settings_for_run(
user_settings = await load_user_settings_for_run(
db_session,
user_id=user_id,
trigger_type=trigger_type,
@ -301,8 +162,9 @@ class ScholarIngestionService:
user_id=user_id,
filtered_scholar_ids=filtered_scholar_ids,
)
await self._run_preflight_guard(
await run_preflight_guard(
db_session,
self._source,
user_settings=user_settings,
user_id=user_id,
scholars=scholars,
@ -486,7 +348,7 @@ class ScholarIngestionService:
if intended_final_status not in (RunStatus.CANCELED,):
run.status = RunStatus.RESOLVING
await db_session.commit()
_log_run_completed(
log_run_completed(
run=run,
user_id=user_id,
scholars=attached_scholars,
@ -495,19 +357,22 @@ class ScholarIngestionService:
alert_summary=alert_summary,
)
if intended_final_status not in (RunStatus.CANCELED,):
task = asyncio.create_task(
self._background_enrich(
spawn_background_enrichment_task(
session_factory,
self._enrichment,
run_id=run.id,
intended_final_status=intended_final_status,
openalex_api_key=getattr(user_settings, "openalex_api_key", None),
)
)
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)
except Exception as exc:
await db_session.rollback()
logger.exception("ingestion.background_run_failed", extra={"run_id": run_id, "user_id": user_id})
structured_log(
logger,
"exception",
"ingestion.background_run_failed",
run_id=run_id,
user_id=user_id,
)
await self._fail_run_in_background(session_factory, run_id, exc)
async def _prepare_execute_run(
@ -530,6 +395,7 @@ class ScholarIngestionService:
return run, user_settings, list(scholars_result.scalars().all())
async def _fail_run_in_background(self, session_factory: Any, run_id: int, exc: Exception) -> None:
try:
async with session_factory() as cleanup_session:
run_to_fail = await cleanup_session.get(CrawlRun, run_id)
if run_to_fail:
@ -537,40 +403,13 @@ class ScholarIngestionService:
run_to_fail.end_dt = datetime.now(UTC)
run_to_fail.error_log["terminal_exception"] = str(exc)
await cleanup_session.commit()
async def _background_enrich(
self,
session_factory: Any,
*,
run_id: int,
intended_final_status: RunStatus,
openalex_api_key: str | None = None,
) -> None:
try:
async with session_factory() as session:
await self._enrichment.enrich_pending_publications(
session,
except Exception:
structured_log(
logger,
"exception",
"ingestion.fail_run_cleanup_failed",
run_id=run_id,
openalex_api_key=openalex_api_key,
)
run = await session.get(CrawlRun, run_id)
if run is not None and run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await session.commit()
logger.info(
"ingestion.background_enrichment_complete",
extra={"run_id": run_id, "final_status": str(intended_final_status)},
)
except Exception:
logger.exception("ingestion.background_enrichment_failed", extra={"run_id": run_id})
try:
async with session_factory() as fallback_session:
run = await fallback_session.get(CrawlRun, run_id)
if run is not None and run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await fallback_session.commit()
except Exception:
logger.exception("ingestion.background_enrichment_fallback_failed", extra={"run_id": run_id})
async def run_for_user(
self,
@ -626,7 +465,7 @@ class ScholarIngestionService:
alert_network_failure_threshold=alert_network_failure_threshold,
alert_retry_scheduled_threshold=alert_retry_scheduled_threshold,
)
progress, failure_summary, alert_summary = await self._run_iteration_and_complete(
progress, failure_summary, alert_summary, intended_final_status = await self._run_iteration_and_complete(
db_session,
run=run,
scholars=scholars,
@ -639,10 +478,14 @@ class ScholarIngestionService:
idempotency_key=idempotency_key,
)
user_settings = await user_settings_service.get_or_create_settings(db_session, user_id=user_id)
await self._inline_enrich_and_finalize(
db_session, run=run, user_settings=user_settings, intended_final_status=run.status
await inline_enrich_and_finalize(
db_session,
self._enrichment,
run=run,
user_settings=user_settings,
intended_final_status=intended_final_status,
)
_log_run_completed(
log_run_completed(
run=run,
user_id=user_id,
scholars=scholars,
@ -665,7 +508,7 @@ class ScholarIngestionService:
auto_queue_continuations: bool,
queue_delay_seconds: int,
idempotency_key: str | None,
) -> tuple[RunProgress, RunFailureSummary, RunAlertSummary]:
) -> tuple[RunProgress, RunFailureSummary, RunAlertSummary, RunStatus]:
progress = await run_scholar_iteration(
db_session,
pagination=self._pagination,
@ -691,27 +534,7 @@ class ScholarIngestionService:
if intended_final_status not in (RunStatus.CANCELED,):
run.status = RunStatus.RESOLVING
await db_session.commit()
return progress, failure_summary, alert_summary
async def _inline_enrich_and_finalize(
self,
db_session: AsyncSession,
*,
run: CrawlRun,
user_settings: Any,
intended_final_status: RunStatus,
) -> None:
try:
await self._enrichment.enrich_pending_publications(
db_session,
run_id=run.id,
openalex_api_key=getattr(user_settings, "openalex_api_key", None),
)
except Exception:
logger.exception("ingestion.enrichment_failed", extra={"run_id": run.id})
if run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await db_session.commit()
return progress, failure_summary, alert_summary, intended_final_status
async def _try_acquire_user_lock(
self,

26
app/services/ingestion/enrichment.py
View file

@ -96,7 +96,7 @@ class EnrichmentRunner:
for p in batch:
if await self.run_is_canceled(db_session, run_id=run_id):
logger.info("ingestion.enrichment_aborted", extra={"run_id": run_id})
structured_log(logger, "info", "ingestion.enrichment_aborted", run_id=run_id)
return False, arxiv_lookup_allowed
p.openalex_last_attempt_at = now
arxiv_lookup_allowed = await self._discover_identifiers_for_enrichment(
@ -112,9 +112,9 @@ class EnrichmentRunner:
candidates=openalex_works,
)
if match:
p.year = match.publication_year or p.year
p.citation_count = match.cited_by_count or p.citation_count
p.pdf_url = match.oa_url or p.pdf_url
p.year = match.publication_year if match.publication_year is not None else p.year
p.citation_count = match.cited_by_count if match.cited_by_count is not None else p.citation_count
p.pdf_url = match.oa_url if match.oa_url is not None else p.pdf_url
p.openalex_enriched = True
return True, arxiv_lookup_allowed
@ -162,7 +162,7 @@ class EnrichmentRunner:
for i in range(0, len(publications), batch_size):
if await self.run_is_canceled(db_session, run_id=run_id):
logger.info("ingestion.enrichment_aborted", extra={"run_id": run_id})
structured_log(logger, "info", "ingestion.enrichment_aborted", run_id=run_id)
return
batch = publications[i : i + batch_size]
titles = _sanitize_titles(batch)
@ -181,6 +181,8 @@ class EnrichmentRunner:
continue
except Exception as e:
structured_log(logger, "warning", "ingestion.openalex_enrichment_failed", error=str(e), run_id=run_id)
for p in batch:
p.openalex_last_attempt_at = now
continue
should_continue, arxiv_lookup_allowed = await self._enrich_batch(
db_session,
@ -302,9 +304,13 @@ class EnrichmentRunner:
{"title.search": "|".join(titles)}, limit=batch_size * 3
)
except Exception as e:
logger.warning(
structured_log(
logger,
"warning",
"ingestion.openalex_enrichment_failed",
extra={"error": str(e), "batch_size": len(batch), "scholar_id": scholar.id},
error=str(e),
batch_size=len(batch),
scholar_id=scholar.id,
)
openalex_works = []
@ -320,11 +326,11 @@ class EnrichmentRunner:
title=p.title,
title_url=p.title_url,
cluster_id=p.cluster_id,
year=match.publication_year or p.year,
citation_count=match.cited_by_count,
year=match.publication_year if match.publication_year is not None else p.year,
citation_count=match.cited_by_count if match.cited_by_count is not None else p.citation_count,
authors_text=p.authors_text,
venue_text=p.venue_text,
pdf_url=match.oa_url or p.pdf_url,
pdf_url=match.oa_url if match.oa_url is not None else p.pdf_url,
)
enriched.append(new_p)
else:

12
app/services/ingestion/page_fetch.py
View file

@ -49,13 +49,13 @@ class PageFetcher:
error="source_does_not_support_pagination",
)
except Exception as exc:
logger.exception(
structured_log(
logger,
"exception",
"ingestion.fetch_unexpected_error",
extra={
"scholar_id": scholar_id,
"cstart": cstart,
"page_size": page_size,
},
scholar_id=scholar_id,
cstart=cstart,
page_size=page_size,
)
return FetchResult(
requested_url=(

6
app/services/ingestion/pagination.py
View file

@ -353,6 +353,9 @@ class PaginationEngine:
page_attempt_log=next_attempt_log,
)
if self._handle_page_state_transition(state=state):
return
if next_parsed_page.publications:
await self._upsert_page_publications(
db_session,
@ -364,9 +367,6 @@ class PaginationEngine:
upsert_publications_fn=upsert_publications_fn,
)
if self._handle_page_state_transition(state=state):
return
@staticmethod
def _result_from_pagination_state(
*,

3
app/services/ingestion/publication_upsert.py
View file

@ -8,7 +8,6 @@ from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import CrawlRun, Publication, ScholarProfile, ScholarPublication
from app.services.doi.normalize import first_doi_from_texts
from app.services.ingestion.fingerprints import (
build_publication_fingerprint,
build_publication_url,
@ -106,6 +105,7 @@ def update_existing_publication(
) -> None:
if candidate.cluster_id and publication.cluster_id is None:
publication.cluster_id = candidate.cluster_id
if not publication.title_raw:
publication.title_raw = candidate.title
publication.title_normalized = normalize_title(candidate.title)
if candidate.year is not None:
@ -118,7 +118,6 @@ def update_existing_publication(
publication.venue_text = candidate.venue_text
if candidate.title_url:
publication.pub_url = build_publication_url(candidate.title_url)
first_doi_from_texts(candidate.title_url, candidate.venue_text, candidate.title)
async def resolve_publication(

8
app/services/ingestion/queue_runner.py
View file

@ -229,7 +229,13 @@ class QueueJobRunner:
error=str(exc),
)
await recovery_session.commit()
logger.exception("scheduler.queue_item_run_failed", extra={"queue_item_id": job.id, "user_id": job.user_id})
structured_log(
logger,
"exception",
"scheduler.queue_item_run_failed",
queue_item_id=job.id,
user_id=job.user_id,
)
async def _load_request_delay_for_user(self, user_id: int, *, floor: int) -> int:
session_factory = get_session_factory()

28
app/services/ingestion/run_completion.py
View file

@ -427,3 +427,31 @@ def run_execution_summary(
partial_count=progress.partial_count,
new_publication_count=run.new_pub_count,
)
def log_run_completed(
*,
run: CrawlRun,
user_id: int,
scholars: list[ScholarProfile],
progress: RunProgress,
failure_summary: RunFailureSummary,
alert_summary: RunAlertSummary,
) -> None:
structured_log(
logger,
"info",
"ingestion.run_completed",
user_id=user_id,
crawl_run_id=run.id,
status=run.status.value,
scholar_count=len(scholars),
succeeded_count=progress.succeeded_count,
failed_count=progress.failed_count,
partial_count=progress.partial_count,
new_publication_count=run.new_pub_count,
blocked_failure_count=alert_summary.blocked_failure_count,
network_failure_count=alert_summary.network_failure_count,
retries_scheduled_count=failure_summary.retries_scheduled_count,
alert_flags=alert_summary.alert_flags,
)

110
app/services/ingestion/run_enrichment.py Normal file
View file

@ -0,0 +1,110 @@
from __future__ import annotations
import asyncio
import logging
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import CrawlRun, RunStatus
from app.logging_utils import structured_log
from app.services.ingestion.enrichment import EnrichmentRunner
logger = logging.getLogger(__name__)
_background_tasks: set[asyncio.Task[Any]] = set()
async def background_enrich(
session_factory: Any,
enrichment_runner: EnrichmentRunner,
*,
run_id: int,
intended_final_status: RunStatus,
openalex_api_key: str | None = None,
) -> None:
try:
async with session_factory() as session:
await enrichment_runner.enrich_pending_publications(
session,
run_id=run_id,
openalex_api_key=openalex_api_key,
)
run = await session.get(CrawlRun, run_id)
if run is not None and run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await session.commit()
structured_log(
logger,
"info",
"ingestion.background_enrichment_complete",
run_id=run_id,
final_status=str(intended_final_status),
)
except Exception:
structured_log(
logger,
"exception",
"ingestion.background_enrichment_failed",
run_id=run_id,
)
try:
async with session_factory() as fallback_session:
run = await fallback_session.get(CrawlRun, run_id)
if run is not None and run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await fallback_session.commit()
except Exception:
structured_log(
logger,
"exception",
"ingestion.background_enrichment_fallback_failed",
run_id=run_id,
)
async def inline_enrich_and_finalize(
db_session: AsyncSession,
enrichment_runner: EnrichmentRunner,
*,
run: CrawlRun,
user_settings: Any,
intended_final_status: RunStatus,
) -> None:
try:
await enrichment_runner.enrich_pending_publications(
db_session,
run_id=run.id,
openalex_api_key=getattr(user_settings, "openalex_api_key", None),
)
except Exception:
structured_log(
logger,
"exception",
"ingestion.enrichment_failed",
run_id=run.id,
)
if run.status == RunStatus.RESOLVING:
run.status = intended_final_status
await db_session.commit()
def spawn_background_enrichment_task(
session_factory: Any,
enrichment_runner: EnrichmentRunner,
*,
run_id: int,
intended_final_status: RunStatus,
openalex_api_key: str | None,
) -> None:
task = asyncio.create_task(
background_enrich(
session_factory,
enrichment_runner,
run_id=run_id,
intended_final_status=intended_final_status,
openalex_api_key=openalex_api_key,
)
)
_background_tasks.add(task)
task.add_done_callback(_background_tasks.discard)

135
app/services/ingestion/run_guards.py Normal file
View file

@ -0,0 +1,135 @@
from __future__ import annotations
import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import (
RunTriggerType,
ScholarProfile,
)
from app.logging_utils import structured_log
from app.services.ingestion import safety as run_safety_service
from app.services.ingestion.preflight import check_scholar_reachable
from app.services.ingestion.types import RunBlockedBySafetyPolicyError
from app.services.scholar.source import ScholarSource
from app.services.settings import application as user_settings_service
from app.settings import settings
logger = logging.getLogger(__name__)
async def load_user_settings_for_run(
db_session: AsyncSession,
*,
user_id: int,
trigger_type: RunTriggerType,
):
user_settings = await user_settings_service.get_or_create_settings(db_session, user_id=user_id)
await enforce_safety_gate(db_session, user_settings=user_settings, user_id=user_id, trigger_type=trigger_type)
return user_settings
async def enforce_safety_gate(
db_session: AsyncSession,
*,
user_settings,
user_id: int,
trigger_type: RunTriggerType,
) -> None:
now_utc = datetime.now(UTC)
previous = run_safety_service.get_safety_event_context(user_settings, now_utc=now_utc)
if run_safety_service.clear_expired_cooldown(user_settings, now_utc=now_utc):
await db_session.commit()
await db_session.refresh(user_settings)
structured_log(
logger,
"info",
"ingestion.cooldown_cleared",
user_id=user_id,
reason=previous.get("cooldown_reason"),
cooldown_until=previous.get("cooldown_until"),
)
now_utc = datetime.now(UTC)
if run_safety_service.is_cooldown_active(user_settings, now_utc=now_utc):
await raise_safety_blocked_start(
db_session,
user_settings=user_settings,
user_id=user_id,
trigger_type=trigger_type,
now_utc=now_utc,
)
async def raise_safety_blocked_start(
db_session: AsyncSession,
*,
user_settings,
user_id: int,
trigger_type: RunTriggerType,
now_utc: datetime,
) -> None:
safety_state = run_safety_service.register_cooldown_blocked_start(user_settings, now_utc=now_utc)
await db_session.commit()
structured_log(
logger,
"warning",
"ingestion.safety_policy_blocked_run_start",
user_id=user_id,
trigger_type=trigger_type.value,
reason=safety_state.get("cooldown_reason"),
cooldown_until=safety_state.get("cooldown_until"),
cooldown_remaining_seconds=safety_state.get("cooldown_remaining_seconds"),
blocked_start_count=((safety_state.get("counters") or {}).get("blocked_start_count")),
)
raise RunBlockedBySafetyPolicyError(
code="scrape_cooldown_active",
message="Scrape safety cooldown is active; run start is temporarily blocked.",
safety_state=safety_state,
)
async def run_preflight_guard(
db_session: AsyncSession,
source: ScholarSource,
*,
user_settings: Any,
user_id: int,
scholars: list[ScholarProfile],
) -> None:
if not scholars:
return
result = await check_scholar_reachable(
source,
scholar_id=scholars[0].scholar_id,
)
if result.passed:
return
now_utc = datetime.now(UTC)
safety_state, _ = run_safety_service.apply_run_safety_outcome(
user_settings,
run_id=0,
blocked_failure_count=1,
network_failure_count=0,
blocked_failure_threshold=1,
network_failure_threshold=1,
blocked_cooldown_seconds=settings.ingestion_safety_cooldown_blocked_seconds,
network_cooldown_seconds=settings.ingestion_safety_cooldown_network_seconds,
now_utc=now_utc,
)
await db_session.commit()
structured_log(
logger,
"warning",
"ingestion.cooldown_entered_preflight",
user_id=user_id,
block_reason=result.block_reason,
cooldown_until=safety_state.get("cooldown_until"),
)
raise RunBlockedBySafetyPolicyError(
code="scrape_cooldown_active",
message=f"Preflight detected Scholar block ({result.block_reason}). Cooldown activated.",
safety_state=safety_state,
)

18
app/services/ingestion/scheduler.py
View file

@ -125,9 +125,10 @@ class SchedulerService:
except asyncio.CancelledError:
raise
except Exception:
logger.exception(
structured_log(
logger,
"exception",
"scheduler.tick_failed",
extra={},
)
await asyncio.sleep(float(self._tick_seconds))
@ -263,7 +264,12 @@ class SchedulerService:
return None
except Exception:
await session.rollback()
logger.exception("scheduler.run_failed", extra={"user_id": candidate.user_id})
structured_log(
logger,
"exception",
"scheduler.run_failed",
user_id=candidate.user_id,
)
return None
async def _run_candidate(self, candidate: _AutoRunCandidate) -> None:
@ -300,4 +306,8 @@ class SchedulerService:
processed_count=processed,
)
except Exception:
logger.exception("scheduler.pdf_queue_drain_failed", extra={})
structured_log(
logger,
"exception",
"scheduler.pdf_queue_drain_failed",
)

308
app/services/ingestion/scholar_outcomes.py Normal file
View file

@ -0,0 +1,308 @@
from __future__ import annotations
import logging
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import (
CrawlRun,
RunStatus,
ScholarProfile,
)
from app.logging_utils import structured_log
from app.services.ingestion.run_completion import build_failure_debug_context
from app.services.ingestion.types import (
PagedParseResult,
ScholarProcessingOutcome,
)
from app.services.scholar.parser import ParseState
logger = logging.getLogger(__name__)
def assert_valid_paged_parse_result(
*,
scholar_id: str,
paged_parse_result: PagedParseResult,
) -> None:
parsed_page = paged_parse_result.parsed_page
if parsed_page.state in {ParseState.OK, ParseState.NO_RESULTS} and any(
code.startswith("layout_") for code in parsed_page.warnings
):
raise RuntimeError(f"Layout warning marked as terminal for scholar_id={scholar_id}.")
for publication in paged_parse_result.publications:
if not publication.title.strip():
raise RuntimeError(f"Malformed publication title for scholar_id={scholar_id}.")
if publication.citation_count is not None and int(publication.citation_count) < 0:
raise RuntimeError(f"Negative citation count for scholar_id={scholar_id}.")
def apply_first_page_profile_metadata(
*,
scholar: ScholarProfile,
paged_parse_result: PagedParseResult,
run_dt: datetime,
) -> None:
first_page = paged_parse_result.first_page_parsed_page
if first_page.profile_name and not (scholar.display_name or "").strip():
scholar.display_name = first_page.profile_name
if first_page.profile_image_url:
scholar.profile_image_url = first_page.profile_image_url
scholar.last_initial_page_checked_at = run_dt
def build_result_entry(
*,
scholar: ScholarProfile,
start_cstart: int,
paged_parse_result: PagedParseResult,
) -> dict[str, Any]:
parsed_page = paged_parse_result.parsed_page
return {
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
"state": parsed_page.state.value,
"state_reason": parsed_page.state_reason,
"outcome": "failed",
"attempt_count": len(paged_parse_result.attempt_log),
"publication_count": len(paged_parse_result.publications),
"start_cstart": start_cstart,
"articles_range": parsed_page.articles_range,
"warnings": parsed_page.warnings,
"has_show_more_button": parsed_page.has_show_more_button,
"pages_fetched": paged_parse_result.pages_fetched,
"pages_attempted": paged_parse_result.pages_attempted,
"has_more_remaining": paged_parse_result.has_more_remaining,
"pagination_truncated_reason": paged_parse_result.pagination_truncated_reason,
"continuation_cstart": paged_parse_result.continuation_cstart,
"skipped_no_change": paged_parse_result.skipped_no_change,
"initial_page_fingerprint_sha256": paged_parse_result.first_page_fingerprint_sha256,
}
def skipped_no_change_outcome(
*,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
first_page = paged_parse_result.first_page_parsed_page
scholar.last_run_status = RunStatus.SUCCESS
scholar.last_run_dt = run_dt
result_entry["state"] = first_page.state.value
result_entry["state_reason"] = "no_change_initial_page_signature"
result_entry["outcome"] = "success"
result_entry["publication_count"] = 0
result_entry["warnings"] = first_page.warnings
result_entry["debug"] = {
"state_reason": "no_change_initial_page_signature",
"first_page_fingerprint_sha256": paged_parse_result.first_page_fingerprint_sha256,
"attempt_log": paged_parse_result.attempt_log,
"page_logs": paged_parse_result.page_logs,
}
return ScholarProcessingOutcome(
result_entry=result_entry,
succeeded_count_delta=1,
failed_count_delta=0,
partial_count_delta=0,
discovered_publication_count=0,
)
async def upsert_publications_outcome(
db_session: AsyncSession,
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
parsed_page = paged_parse_result.parsed_page
publications = paged_parse_result.publications
had_page_failure = parsed_page.state not in {ParseState.OK, ParseState.NO_RESULTS}
has_partial_set = len(publications) > 0 and had_page_failure
if (not had_page_failure) or has_partial_set:
return await _upsert_success_or_exception(
db_session,
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
has_partial_publication_set=has_partial_set,
)
return _parse_failure_outcome(
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
)
async def _upsert_success_or_exception(
db_session: AsyncSession,
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
has_partial_publication_set: bool,
) -> ScholarProcessingOutcome:
try:
return _upsert_success(
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
has_partial_publication_set=has_partial_publication_set,
)
except Exception as exc:
return _upsert_exception_outcome(
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
exc=exc,
)
def _upsert_success(
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
has_partial_publication_set: bool,
) -> ScholarProcessingOutcome:
discovered_count = paged_parse_result.discovered_publication_count
is_partial = (
paged_parse_result.has_more_remaining
or paged_parse_result.pagination_truncated_reason is not None
or has_partial_publication_set
)
scholar.last_run_status = RunStatus.PARTIAL_FAILURE if is_partial else RunStatus.SUCCESS
scholar.last_run_dt = run_dt
if not is_partial and paged_parse_result.first_page_fingerprint_sha256:
scholar.last_initial_page_fingerprint_sha256 = paged_parse_result.first_page_fingerprint_sha256
result_entry["outcome"] = "partial" if is_partial else "success"
if is_partial:
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
)
return ScholarProcessingOutcome(
result_entry=result_entry,
succeeded_count_delta=1,
failed_count_delta=0,
partial_count_delta=1 if is_partial else 0,
discovered_publication_count=discovered_count,
)
def _upsert_exception_outcome(
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
exc: Exception,
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = run_dt
result_entry["state"] = "ingestion_error"
result_entry["state_reason"] = "publication_upsert_exception"
result_entry["outcome"] = "failed"
result_entry["error"] = str(exc)
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
exception=exc,
)
structured_log(
logger,
"exception",
"ingestion.scholar_failed",
crawl_run_id=run.id,
scholar_profile_id=scholar.id,
scholar_id=scholar.scholar_id,
)
return ScholarProcessingOutcome(result_entry, 0, 1, 0, 0)
def _parse_failure_outcome(
*,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = run_dt
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
)
structured_log(
logger,
"warning",
"ingestion.scholar_parse_failed",
scholar_profile_id=scholar.id,
scholar_id=scholar.scholar_id,
state=paged_parse_result.parsed_page.state.value,
state_reason=paged_parse_result.parsed_page.state_reason,
status_code=paged_parse_result.fetch_result.status_code,
)
return ScholarProcessingOutcome(result_entry, 0, 1, 0, 0)
def unexpected_scholar_exception_outcome(
*,
run: CrawlRun,
scholar: ScholarProfile,
start_cstart: int,
exc: Exception,
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = datetime.now(UTC)
structured_log(
logger,
"exception",
"ingestion.scholar_unexpected_failure",
crawl_run_id=run.id,
scholar_profile_id=scholar.id,
scholar_id=scholar.scholar_id,
)
return ScholarProcessingOutcome(
result_entry={
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
"state": "ingestion_error",
"state_reason": "scholar_processing_exception",
"outcome": "failed",
"attempt_count": 0,
"publication_count": 0,
"start_cstart": start_cstart,
"warnings": [],
"error": str(exc),
"debug": {"exception_type": type(exc).__name__, "exception_message": str(exc)},
},
succeeded_count_delta=0,
failed_count_delta=1,
partial_count_delta=0,
discovered_publication_count=0,
)

299
app/services/ingestion/scholar_processing.py
View file

@ -6,6 +6,7 @@ import random
from datetime import UTC, datetime
from typing import Any
from sqlalchemy.exc import InterfaceError, OperationalError
from sqlalchemy.ext.asyncio import AsyncSession
from app.db.models import (
@ -21,7 +22,15 @@ from app.services.ingestion.constants import (
)
from app.services.ingestion.pagination import PaginationEngine
from app.services.ingestion.publication_upsert import upsert_profile_publications
from app.services.ingestion.run_completion import apply_outcome_to_progress, build_failure_debug_context
from app.services.ingestion.run_completion import apply_outcome_to_progress
from app.services.ingestion.scholar_outcomes import (
apply_first_page_profile_metadata,
assert_valid_paged_parse_result,
build_result_entry,
skipped_no_change_outcome,
unexpected_scholar_exception_outcome,
upsert_publications_outcome,
)
from app.services.ingestion.types import (
PagedParseResult,
RunProgress,
@ -33,254 +42,6 @@ from app.services.scholar.state_detection import is_hard_challenge_reason
logger = logging.getLogger(__name__)
def assert_valid_paged_parse_result(
*,
scholar_id: str,
paged_parse_result: PagedParseResult,
) -> None:
parsed_page = paged_parse_result.parsed_page
if parsed_page.state in {ParseState.OK, ParseState.NO_RESULTS} and any(
code.startswith("layout_") for code in parsed_page.warnings
):
raise RuntimeError(f"Layout warning marked as terminal for scholar_id={scholar_id}.")
for publication in paged_parse_result.publications:
if not publication.title.strip():
raise RuntimeError(f"Malformed publication title for scholar_id={scholar_id}.")
if publication.citation_count is not None and int(publication.citation_count) < 0:
raise RuntimeError(f"Negative citation count for scholar_id={scholar_id}.")
def apply_first_page_profile_metadata(
*,
scholar: ScholarProfile,
paged_parse_result: PagedParseResult,
run_dt: datetime,
) -> None:
first_page = paged_parse_result.first_page_parsed_page
if first_page.profile_name and not (scholar.display_name or "").strip():
scholar.display_name = first_page.profile_name
if first_page.profile_image_url:
scholar.profile_image_url = first_page.profile_image_url
scholar.last_initial_page_checked_at = run_dt
def build_result_entry(
*,
scholar: ScholarProfile,
start_cstart: int,
paged_parse_result: PagedParseResult,
) -> dict[str, Any]:
parsed_page = paged_parse_result.parsed_page
return {
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
"state": parsed_page.state.value,
"state_reason": parsed_page.state_reason,
"outcome": "failed",
"attempt_count": len(paged_parse_result.attempt_log),
"publication_count": len(paged_parse_result.publications),
"start_cstart": start_cstart,
"articles_range": parsed_page.articles_range,
"warnings": parsed_page.warnings,
"has_show_more_button": parsed_page.has_show_more_button,
"pages_fetched": paged_parse_result.pages_fetched,
"pages_attempted": paged_parse_result.pages_attempted,
"has_more_remaining": paged_parse_result.has_more_remaining,
"pagination_truncated_reason": paged_parse_result.pagination_truncated_reason,
"continuation_cstart": paged_parse_result.continuation_cstart,
"skipped_no_change": paged_parse_result.skipped_no_change,
"initial_page_fingerprint_sha256": paged_parse_result.first_page_fingerprint_sha256,
}
def skipped_no_change_outcome(
*,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
first_page = paged_parse_result.first_page_parsed_page
scholar.last_run_status = RunStatus.SUCCESS
scholar.last_run_dt = run_dt
result_entry["state"] = first_page.state.value
result_entry["state_reason"] = "no_change_initial_page_signature"
result_entry["outcome"] = "success"
result_entry["publication_count"] = 0
result_entry["warnings"] = first_page.warnings
result_entry["debug"] = {
"state_reason": "no_change_initial_page_signature",
"first_page_fingerprint_sha256": paged_parse_result.first_page_fingerprint_sha256,
"attempt_log": paged_parse_result.attempt_log,
"page_logs": paged_parse_result.page_logs,
}
return ScholarProcessingOutcome(
result_entry=result_entry,
succeeded_count_delta=1,
failed_count_delta=0,
partial_count_delta=0,
discovered_publication_count=0,
)
async def upsert_publications_outcome(
db_session: AsyncSession,
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
parsed_page = paged_parse_result.parsed_page
publications = paged_parse_result.publications
had_page_failure = parsed_page.state not in {ParseState.OK, ParseState.NO_RESULTS}
has_partial_set = len(publications) > 0 and had_page_failure
if (not had_page_failure) or has_partial_set:
return await _upsert_success_or_exception(
db_session,
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
has_partial_publication_set=has_partial_set,
)
return _parse_failure_outcome(
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
)
async def _upsert_success_or_exception(
db_session: AsyncSession,
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
has_partial_publication_set: bool,
) -> ScholarProcessingOutcome:
try:
return _upsert_success(
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
has_partial_publication_set=has_partial_publication_set,
)
except Exception as exc:
return _upsert_exception_outcome(
run=run,
scholar=scholar,
run_dt=run_dt,
paged_parse_result=paged_parse_result,
result_entry=result_entry,
exc=exc,
)
def _upsert_success(
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
has_partial_publication_set: bool,
) -> ScholarProcessingOutcome:
discovered_count = paged_parse_result.discovered_publication_count
is_partial = (
paged_parse_result.has_more_remaining
or paged_parse_result.pagination_truncated_reason is not None
or has_partial_publication_set
)
scholar.last_run_status = RunStatus.PARTIAL_FAILURE if is_partial else RunStatus.SUCCESS
scholar.last_run_dt = run_dt
if not is_partial and paged_parse_result.first_page_fingerprint_sha256:
scholar.last_initial_page_fingerprint_sha256 = paged_parse_result.first_page_fingerprint_sha256
result_entry["outcome"] = "partial" if is_partial else "success"
if is_partial:
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
)
return ScholarProcessingOutcome(
result_entry=result_entry,
succeeded_count_delta=1,
failed_count_delta=0,
partial_count_delta=1 if is_partial else 0,
discovered_publication_count=discovered_count,
)
def _upsert_exception_outcome(
*,
run: CrawlRun,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
exc: Exception,
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = run_dt
result_entry["state"] = "ingestion_error"
result_entry["state_reason"] = "publication_upsert_exception"
result_entry["outcome"] = "failed"
result_entry["error"] = str(exc)
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
exception=exc,
)
logger.exception(
"ingestion.scholar_failed",
extra={
"crawl_run_id": run.id,
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
},
)
return ScholarProcessingOutcome(result_entry, 0, 1, 0, 0)
def _parse_failure_outcome(
*,
scholar: ScholarProfile,
run_dt: datetime,
paged_parse_result: PagedParseResult,
result_entry: dict[str, Any],
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = run_dt
result_entry["debug"] = build_failure_debug_context(
fetch_result=paged_parse_result.fetch_result,
parsed_page=paged_parse_result.parsed_page,
attempt_log=paged_parse_result.attempt_log,
page_logs=paged_parse_result.page_logs,
)
structured_log(
logger,
"warning",
"ingestion.scholar_parse_failed",
scholar_profile_id=scholar.id,
scholar_id=scholar.scholar_id,
state=paged_parse_result.parsed_page.state.value,
state_reason=paged_parse_result.parsed_page.state_reason,
status_code=paged_parse_result.fetch_result.status_code,
)
return ScholarProcessingOutcome(result_entry, 0, 1, 0, 0)
async def sync_continuation_queue(
db_session: AsyncSession,
*,
@ -396,6 +157,8 @@ async def process_scholar(
queue_delay_seconds=queue_delay_seconds,
)
return outcome
except (OperationalError, InterfaceError):
raise
except Exception as exc:
return unexpected_scholar_exception_outcome(
run=run,
@ -492,44 +255,6 @@ async def _resolve_scholar_outcome(
)
def unexpected_scholar_exception_outcome(
*,
run: CrawlRun,
scholar: ScholarProfile,
start_cstart: int,
exc: Exception,
) -> ScholarProcessingOutcome:
scholar.last_run_status = RunStatus.FAILED
scholar.last_run_dt = datetime.now(UTC)
logger.exception(
"ingestion.scholar_unexpected_failure",
extra={
"crawl_run_id": run.id,
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
},
)
return ScholarProcessingOutcome(
result_entry={
"scholar_profile_id": scholar.id,
"scholar_id": scholar.scholar_id,
"state": "ingestion_error",
"state_reason": "scholar_processing_exception",
"outcome": "failed",
"attempt_count": 0,
"publication_count": 0,
"start_cstart": start_cstart,
"warnings": [],
"error": str(exc),
"debug": {"exception_type": type(exc).__name__, "exception_message": str(exc)},
},
succeeded_count_delta=0,
failed_count_delta=1,
partial_count_delta=0,
discovered_publication_count=0,
)
def _is_hard_challenge_outcome(outcome: ScholarProcessingOutcome) -> bool:
entry = outcome.result_entry
return str(entry.get("state", "")) == ParseState.BLOCKED_OR_CAPTCHA.value and is_hard_challenge_reason(

27
app/services/openalex/client.py
View file

@ -8,6 +8,7 @@ from tenacity import (
wait_exponential,
)
from app.logging_utils import structured_log
from app.services.openalex.types import OpenAlexWork
logger = logging.getLogger(__name__)
@ -82,7 +83,13 @@ class OpenAlexClient:
raise OpenAlexBudgetExhaustedError("Daily API budget exhausted; retrying won't help until midnight UTC")
raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex work by DOI")
if response.status_code >= 400:
logger.warning("OpenAlex API error: %s %s", response.status_code, response.text[:500])
structured_log(
logger,
"warning",
"openalex.api_error",
status_code=response.status_code,
response_preview=response.text[:500],
)
raise OpenAlexClientError(f"API Error {response.status_code}")
data = response.json()
@ -130,7 +137,14 @@ class OpenAlexClient:
raise OpenAlexBudgetExhaustedError("Daily API budget exhausted; retrying won't help until midnight UTC")
raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex works list")
if response.status_code >= 400:
logger.warning("OpenAlex API error (filters=%s): %s %s", filters, response.status_code, response.text[:500])
structured_log(
logger,
"warning",
"openalex.api_error_with_filters",
filters=filters,
status_code=response.status_code,
response_preview=response.text[:500],
)
raise OpenAlexClientError(f"API Error {response.status_code}")
data = response.json()
@ -140,8 +154,13 @@ class OpenAlexClient:
for raw_work in results:
try:
parsed_works.append(OpenAlexWork.from_api_dict(raw_work))
except Exception as e:
logger.warning("Failed to parse OpenAlex raw dict: %s", e)
except Exception as exc:
structured_log(
logger,
"warning",
"openalex.parse_failed",
error=str(exc),
)
continue
return parsed_works

1
app/services/publications/pdf_queue_common.py
View file

@ -4,6 +4,7 @@ from datetime import UTC, datetime
from app.db.models import PublicationPdfJob, PublicationPdfJobEvent
PDF_STATUS_UNTRACKED = "untracked"
PDF_STATUS_QUEUED = "queued"
PDF_STATUS_RUNNING = "running"
PDF_STATUS_RESOLVED = "resolved"

11
app/services/publications/pdf_queue_queries.py
View file

@ -16,14 +16,13 @@ from app.db.models import (
)
from app.services.publication_identifiers import application as identifier_service
from app.services.publication_identifiers.types import DisplayIdentifier
from app.services.publications.pdf_queue_common import (
PDF_STATUS_QUEUED,
PDF_STATUS_RUNNING,
PDF_STATUS_UNTRACKED,
)
from app.services.publications.types import PublicationListItem
PDF_STATUS_UNTRACKED = "untracked"
PDF_STATUS_QUEUED = "queued"
PDF_STATUS_RUNNING = "running"
PDF_STATUS_RESOLVED = "resolved"
PDF_STATUS_FAILED = "failed"
def _bounded_limit(limit: int, *, max_value: int = 500) -> int:
return max(1, min(int(limit), max_value))

2
app/services/publications/pdf_queue_resolution.py
View file

@ -251,7 +251,7 @@ def _drop_finished_task(task: asyncio.Task[None]) -> None:
try:
task.result()
except Exception:
logger.exception("publications.pdf_queue.task_failed")
structured_log(logger, "exception", "publications.pdf_queue.task_failed")
def schedule_rows(

14
app/services/publications/queries.py
View file

@ -16,6 +16,12 @@ from app.db.models import (
ScholarPublication,
)
from app.services.publications.modes import MODE_LATEST, MODE_UNREAD
from app.services.publications.pdf_queue_common import (
PDF_STATUS_FAILED,
PDF_STATUS_QUEUED,
PDF_STATUS_RESOLVED,
PDF_STATUS_RUNNING,
)
from app.services.publications.types import PublicationListItem, UnreadPublicationItem
@ -29,10 +35,10 @@ def _normalized_citation_count(value: object) -> int:
def _pdf_status_sort_rank():
return case(
(Publication.pdf_url.is_not(None), 4),
(PublicationPdfJob.status == "resolved", 4),
(PublicationPdfJob.status == "running", 3),
(PublicationPdfJob.status == "queued", 2),
(PublicationPdfJob.status == "failed", 0),
(PublicationPdfJob.status == PDF_STATUS_RESOLVED, 4),
(PublicationPdfJob.status == PDF_STATUS_RUNNING, 3),
(PublicationPdfJob.status == PDF_STATUS_QUEUED, 2),
(PublicationPdfJob.status == PDF_STATUS_FAILED, 0),
else_=1,
)

24
app/services/runs/events.py
View file

@ -4,6 +4,8 @@ import logging
from collections.abc import AsyncGenerator
from typing import Any
from app.logging_utils import structured_log
logger = logging.getLogger(__name__)
@ -17,7 +19,13 @@ class RunEventPublisher:
self._subscribers[run_id] = set()
queue: asyncio.Queue[Any] = asyncio.Queue()
self._subscribers[run_id].add(queue)
logger.debug(f"New subscriber for run {run_id}. Total: {len(self._subscribers[run_id])}")
structured_log(
logger,
"debug",
"runs.event_subscriber_added",
run_id=run_id,
subscriber_count=len(self._subscribers[run_id]),
)
return queue
def unsubscribe(self, run_id: int, queue: asyncio.Queue) -> None:
@ -37,7 +45,12 @@ class RunEventPublisher:
try:
queue.put_nowait(message)
except asyncio.QueueFull:
logger.warning(f"Subscriber queue full for run {run_id}, dropping message")
structured_log(
logger,
"warning",
"runs.event_subscriber_queue_full",
run_id=run_id,
)
run_events = RunEventPublisher()
@ -54,7 +67,12 @@ async def event_generator(run_id: int) -> AsyncGenerator[str, None]:
data_str = json.dumps(message["data"])
yield f"event: {event_type}\ndata: {data_str}\n\n"
except asyncio.CancelledError:
logger.debug(f"Client disconnected from SSE stream for run {run_id}")
structured_log(
logger,
"debug",
"runs.event_stream_disconnected",
run_id=run_id,
)
raise
finally:
run_events.unsubscribe(run_id, queue)

5
app/services/scholar/profile_rows.py
View file

@ -131,7 +131,10 @@ def parse_citation_count(parts: list[str]) -> int | None:
text = normalize_space(" ".join(parts))
if not text:
return 0
digits = re.sub(r"\D+", "", text)
match = re.search(r"[\d,]+", text)
if not match:
return None
digits = match.group(0).replace(",", "")
if not digits:
return None
return int(digits)

2
app/services/unpaywall/application.py
View file

@ -393,7 +393,7 @@ async def resolve_publication_oa_outcomes(
return {}
email = _email_for_request(request_email)
if email is None:
logger.debug("unpaywall.resolve_skipped_missing_email")
structured_log(logger, "debug", "unpaywall.resolve_skipped_missing_email")
return {}
import httpx

700
frontend/package-lock.json generated
View file

@ -15,7 +15,9 @@
"devDependencies": {
"@types/node": "^22.10.2",
"@vitejs/plugin-vue": "^5.2.1",
"@vue/test-utils": "^2.4.6",
"autoprefixer": "^10.4.21",
"happy-dom": "^20.7.0",
"postcss": "^8.5.3",
"tailwindcss": "^3.4.17",
"typescript": "^5.7.2",
@ -474,6 +476,24 @@
"node": ">=12"
}
},
"node_modules/@isaacs/cliui": {
"version": "8.0.2",
"resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-8.0.2.tgz",
"integrity": "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==",
"dev": true,
"license": "ISC",
"dependencies": {
"string-width": "^5.1.2",
"string-width-cjs": "npm:string-width@^4.2.0",
"strip-ansi": "^7.0.1",
"strip-ansi-cjs": "npm:strip-ansi@^6.0.1",
"wrap-ansi": "^8.1.0",
"wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/@jridgewell/gen-mapping": {
"version": "0.3.13",
"resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz",
@ -550,6 +570,24 @@
"node": ">= 8"
}
},
"node_modules/@one-ini/wasm": {
"version": "0.1.1",
"resolved": "https://registry.npmjs.org/@one-ini/wasm/-/wasm-0.1.1.tgz",
"integrity": "sha512-XuySG1E38YScSJoMlqovLru4KTUNSjgVTIjyh7qMX6aNN5HY5Ct5LhRJdxO79JtTzKfzV/bnWpz+zquYrISsvw==",
"dev": true,
"license": "MIT"
},
"node_modules/@pkgjs/parseargs": {
"version": "0.11.0",
"resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz",
"integrity": "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==",
"dev": true,
"license": "MIT",
"optional": true,
"engines": {
"node": ">=14"
}
},
"node_modules/@rollup/rollup-android-arm-eabi": {
"version": "4.57.1",
"resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.57.1.tgz",
@ -917,6 +955,23 @@
"undici-types": "~6.21.0"
}
},
"node_modules/@types/whatwg-mimetype": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/@types/whatwg-mimetype/-/whatwg-mimetype-3.0.2.tgz",
"integrity": "sha512-c2AKvDT8ToxLIOUlN51gTiHXflsfIFisS4pO7pDPoKouJCESkhZnEy623gwP9laCy5lnLDAw1vAzu2vM2YLOrA==",
"dev": true,
"license": "MIT"
},
"node_modules/@types/ws": {
"version": "8.18.1",
"resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.1.tgz",
"integrity": "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*"
}
},
"node_modules/@vitejs/plugin-vue": {
"version": "5.2.4",
"resolved": "https://registry.npmjs.org/@vitejs/plugin-vue/-/plugin-vue-5.2.4.tgz",
@ -1227,6 +1282,27 @@
"integrity": "sha512-cfWa1fCGBxrvaHRhvV3Is0MgmrbSCxYTXCSCau2I0a1Xw1N1pHAvkWCiXPRAqjvToILvguNyEwjevUqAuBQWvQ==",
"license": "MIT"
},
"node_modules/@vue/test-utils": {
"version": "2.4.6",
"resolved": "https://registry.npmjs.org/@vue/test-utils/-/test-utils-2.4.6.tgz",
"integrity": "sha512-FMxEjOpYNYiFe0GkaHsnJPXFHxQ6m4t8vI/ElPGpMWxZKpmRvQ33OIrvRXemy6yha03RxhOlQuy+gZMC3CQSow==",
"dev": true,
"license": "MIT",
"dependencies": {
"js-beautify": "^1.14.9",
"vue-component-type-helpers": "^2.0.0"
}
},
"node_modules/abbrev": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/abbrev/-/abbrev-2.0.0.tgz",
"integrity": "sha512-6/mh1E2u2YgEsCHdY0Yx5oW+61gZU+1vXaoiHHrpKeuRNNgFvS+/jrwHiQhB5apAf5oB7UB7E19ol2R2LKH8hQ==",
"dev": true,
"license": "ISC",
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
}
},
"node_modules/alien-signals": {
"version": "1.0.13",
"resolved": "https://registry.npmjs.org/alien-signals/-/alien-signals-1.0.13.tgz",
@ -1234,6 +1310,32 @@
"dev": true,
"license": "MIT"
},
"node_modules/ansi-regex": {
"version": "6.2.2",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.2.2.tgz",
"integrity": "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-regex?sponsor=1"
}
},
"node_modules/ansi-styles": {
"version": "6.2.3",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.3.tgz",
"integrity": "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/any-promise": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/any-promise/-/any-promise-1.3.0.tgz",
@ -1502,6 +1604,26 @@
"node": ">= 6"
}
},
"node_modules/color-convert": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz",
"integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"color-name": "~1.1.4"
},
"engines": {
"node": ">=7.0.0"
}
},
"node_modules/color-name": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz",
"integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==",
"dev": true,
"license": "MIT"
},
"node_modules/commander": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/commander/-/commander-4.1.1.tgz",
@ -1512,6 +1634,32 @@
"node": ">= 6"
}
},
"node_modules/config-chain": {
"version": "1.1.13",
"resolved": "https://registry.npmjs.org/config-chain/-/config-chain-1.1.13.tgz",
"integrity": "sha512-qj+f8APARXHrM0hraqXYb2/bOVSV4PvJQlNZ/DVj0QrmNM2q2euizkeuVckQ57J+W0mRH6Hvi+k50M4Jul2VRQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"ini": "^1.3.4",
"proto-list": "~1.2.1"
}
},
"node_modules/cross-spawn": {
"version": "7.0.6",
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
"integrity": "sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==",
"dev": true,
"license": "MIT",
"dependencies": {
"path-key": "^3.1.0",
"shebang-command": "^2.0.0",
"which": "^2.0.1"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/cssesc": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/cssesc/-/cssesc-3.0.0.tgz",
@ -1580,6 +1728,42 @@
"dev": true,
"license": "MIT"
},
"node_modules/eastasianwidth": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz",
"integrity": "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==",
"dev": true,
"license": "MIT"
},
"node_modules/editorconfig": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/editorconfig/-/editorconfig-1.0.7.tgz",
"integrity": "sha512-e0GOtq/aTQhVdNyDU9e02+wz9oDDM+SIOQxWME2QRjzRX5yyLAuHDE+0aE8vHb9XRC8XD37eO2u57+F09JqFhw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@one-ini/wasm": "0.1.1",
"commander": "^10.0.0",
"minimatch": "^9.0.1",
"semver": "^7.5.3"
},
"bin": {
"editorconfig": "bin/editorconfig"
},
"engines": {
"node": ">=14"
}
},
"node_modules/editorconfig/node_modules/commander": {
"version": "10.0.1",
"resolved": "https://registry.npmjs.org/commander/-/commander-10.0.1.tgz",
"integrity": "sha512-y4Mg2tXshplEbSGzx7amzPwKKOCGuoSRP/CjEdwwk0FOGlUbq6lKuoyDZTNZkmxHdJtp54hdfY/JUrdL7Xfdug==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14"
}
},
"node_modules/electron-to-chromium": {
"version": "1.5.286",
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.286.tgz",
@ -1587,6 +1771,13 @@
"dev": true,
"license": "ISC"
},
"node_modules/emoji-regex": {
"version": "9.2.2",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-9.2.2.tgz",
"integrity": "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg==",
"dev": true,
"license": "MIT"
},
"node_modules/entities": {
"version": "7.0.1",
"resolved": "https://registry.npmjs.org/entities/-/entities-7.0.1.tgz",
@ -1728,6 +1919,23 @@
"node": ">=8"
}
},
"node_modules/foreground-child": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.3.1.tgz",
"integrity": "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw==",
"dev": true,
"license": "ISC",
"dependencies": {
"cross-spawn": "^7.0.6",
"signal-exit": "^4.0.1"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/fraction.js": {
"version": "5.3.4",
"resolved": "https://registry.npmjs.org/fraction.js/-/fraction.js-5.3.4.tgz",
@ -1767,6 +1975,28 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/glob": {
"version": "10.5.0",
"resolved": "https://registry.npmjs.org/glob/-/glob-10.5.0.tgz",
"integrity": "sha512-DfXN8DfhJ7NH3Oe7cFmu3NCu1wKbkReJ8TorzSAFbSKrlNaQSKfIzqYqVY8zlbs2NLBbWpRiU52GX2PbaBVNkg==",
"deprecated": "Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me",
"dev": true,
"license": "ISC",
"dependencies": {
"foreground-child": "^3.1.0",
"jackspeak": "^3.1.2",
"minimatch": "^9.0.4",
"minipass": "^7.1.2",
"package-json-from-dist": "^1.0.0",
"path-scurry": "^1.11.1"
},
"bin": {
"glob": "dist/esm/bin.mjs"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/glob-parent": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-6.0.2.tgz",
@ -1780,6 +2010,24 @@
"node": ">=10.13.0"
}
},
"node_modules/happy-dom": {
"version": "20.7.0",
"resolved": "https://registry.npmjs.org/happy-dom/-/happy-dom-20.7.0.tgz",
"integrity": "sha512-hR/uLYQdngTyEfxnOoa+e6KTcfBFyc1hgFj/Cc144A5JJUuHFYqIEBDcD4FeGqUeKLRZqJ9eN9u7/GDjYEgS1g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": ">=20.0.0",
"@types/whatwg-mimetype": "^3.0.2",
"@types/ws": "^8.18.1",
"entities": "^7.0.1",
"whatwg-mimetype": "^3.0.0",
"ws": "^8.18.3"
},
"engines": {
"node": ">=20.0.0"
}
},
"node_modules/hasown": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
@ -1803,6 +2051,13 @@
"he": "bin/he"
}
},
"node_modules/ini": {
"version": "1.3.8",
"resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"dev": true,
"license": "ISC"
},
"node_modules/is-binary-path": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/is-binary-path/-/is-binary-path-2.1.0.tgz",
@ -1842,6 +2097,16 @@
"node": ">=0.10.0"
}
},
"node_modules/is-fullwidth-code-point": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz",
"integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/is-glob": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz",
@ -1865,6 +2130,29 @@
"node": ">=0.12.0"
}
},
"node_modules/isexe": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
"dev": true,
"license": "ISC"
},
"node_modules/jackspeak": {
"version": "3.4.3",
"resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-3.4.3.tgz",
"integrity": "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"@isaacs/cliui": "^8.0.2"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
},
"optionalDependencies": {
"@pkgjs/parseargs": "^0.11.0"
}
},
"node_modules/jiti": {
"version": "1.21.7",
"resolved": "https://registry.npmjs.org/jiti/-/jiti-1.21.7.tgz",
@ -1875,6 +2163,38 @@
"jiti": "bin/jiti.js"
}
},
"node_modules/js-beautify": {
"version": "1.15.4",
"resolved": "https://registry.npmjs.org/js-beautify/-/js-beautify-1.15.4.tgz",
"integrity": "sha512-9/KXeZUKKJwqCXUdBxFJ3vPh467OCckSBmYDwSK/EtV090K+iMJ7zx2S3HLVDIWFQdqMIsZWbnaGiba18aWhaA==",
"dev": true,
"license": "MIT",
"dependencies": {
"config-chain": "^1.1.13",
"editorconfig": "^1.0.4",
"glob": "^10.4.2",
"js-cookie": "^3.0.5",
"nopt": "^7.2.1"
},
"bin": {
"css-beautify": "js/bin/css-beautify.js",
"html-beautify": "js/bin/html-beautify.js",
"js-beautify": "js/bin/js-beautify.js"
},
"engines": {
"node": ">=14"
}
},
"node_modules/js-cookie": {
"version": "3.0.5",
"resolved": "https://registry.npmjs.org/js-cookie/-/js-cookie-3.0.5.tgz",
"integrity": "sha512-cEiJEAEoIbWfCZYKWhVwFuvPX1gETRYPw6LlaTKoxD3s2AkXzkCjnp6h0V77ozyqj0jakteJ4YqDJT830+lVGw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=14"
}
},
"node_modules/lilconfig": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/lilconfig/-/lilconfig-3.1.3.tgz",
@ -1902,6 +2222,13 @@
"dev": true,
"license": "MIT"
},
"node_modules/lru-cache": {
"version": "10.4.3",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.4.3.tgz",
"integrity": "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ==",
"dev": true,
"license": "ISC"
},
"node_modules/magic-string": {
"version": "0.30.21",
"resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz",
@ -1951,6 +2278,16 @@
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/minipass": {
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz",
"integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==",
"dev": true,
"license": "BlueOak-1.0.0",
"engines": {
"node": ">=16 || 14 >=14.17"
}
},
"node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
@ -2002,6 +2339,22 @@
"dev": true,
"license": "MIT"
},
"node_modules/nopt": {
"version": "7.2.1",
"resolved": "https://registry.npmjs.org/nopt/-/nopt-7.2.1.tgz",
"integrity": "sha512-taM24ViiimT/XntxbPyJQzCG+p4EKOpgD3mxFwW38mGjVUrfERQOeY4EDHjdnptttfHuHQXFx+lTP08Q+mLa/w==",
"dev": true,
"license": "ISC",
"dependencies": {
"abbrev": "^2.0.0"
},
"bin": {
"nopt": "bin/nopt.js"
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
}
},
"node_modules/normalize-path": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/normalize-path/-/normalize-path-3.0.0.tgz",
@ -2032,6 +2385,13 @@
"node": ">= 6"
}
},
"node_modules/package-json-from-dist": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/package-json-from-dist/-/package-json-from-dist-1.0.1.tgz",
"integrity": "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw==",
"dev": true,
"license": "BlueOak-1.0.0"
},
"node_modules/path-browserify": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/path-browserify/-/path-browserify-1.0.1.tgz",
@ -2039,6 +2399,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/path-key": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/path-key/-/path-key-3.1.1.tgz",
"integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/path-parse": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
@ -2046,6 +2416,23 @@
"dev": true,
"license": "MIT"
},
"node_modules/path-scurry": {
"version": "1.11.1",
"resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.11.1.tgz",
"integrity": "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"lru-cache": "^10.2.0",
"minipass": "^5.0.0 || ^6.0.2 || ^7.0.0"
},
"engines": {
"node": ">=16 || 14 >=14.18"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/pathe": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/pathe/-/pathe-1.1.2.tgz",
@ -2286,6 +2673,13 @@
"dev": true,
"license": "MIT"
},
"node_modules/proto-list": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/proto-list/-/proto-list-1.2.4.tgz",
"integrity": "sha512-vtK/94akxsTMhe0/cbfpR+syPuszcuwhqVjJq26CuNDgFGj682oRBXOP5MJpv2r7JtE8MsiepGIqvvOTBwn2vA==",
"dev": true,
"license": "ISC"
},
"node_modules/queue-microtask": {
"version": "1.2.3",
"resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz",
@ -2431,6 +2825,42 @@
"queue-microtask": "^1.2.2"
}
},
"node_modules/semver": {
"version": "7.7.4",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz",
"integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==",
"dev": true,
"license": "ISC",
"bin": {
"semver": "bin/semver.js"
},
"engines": {
"node": ">=10"
}
},
"node_modules/shebang-command": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/shebang-command/-/shebang-command-2.0.0.tgz",
"integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==",
"dev": true,
"license": "MIT",
"dependencies": {
"shebang-regex": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/shebang-regex": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/shebang-regex/-/shebang-regex-3.0.0.tgz",
"integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/siginfo": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz",
@ -2438,6 +2868,19 @@
"dev": true,
"license": "ISC"
},
"node_modules/signal-exit": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz",
"integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==",
"dev": true,
"license": "ISC",
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/source-map-js": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
@ -2461,6 +2904,110 @@
"dev": true,
"license": "MIT"
},
"node_modules/string-width": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz",
"integrity": "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==",
"dev": true,
"license": "MIT",
"dependencies": {
"eastasianwidth": "^0.2.0",
"emoji-regex": "^9.2.2",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/string-width-cjs": {
"name": "string-width",
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dev": true,
"license": "MIT",
"dependencies": {
"emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0",
"strip-ansi": "^6.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/string-width-cjs/node_modules/ansi-regex": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/string-width-cjs/node_modules/emoji-regex": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==",
"dev": true,
"license": "MIT"
},
"node_modules/string-width-cjs/node_modules/strip-ansi": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/strip-ansi": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.2.0.tgz",
"integrity": "sha512-yDPMNjp4WyfYBkHnjIRLfca1i6KMyGCtsVgoKe/z1+6vukgaENdgGBZt+ZmKPc4gavvEZ5OgHfHdrazhgNyG7w==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-regex": "^6.2.2"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/strip-ansi?sponsor=1"
}
},
"node_modules/strip-ansi-cjs": {
"name": "strip-ansi",
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/strip-ansi-cjs/node_modules/ansi-regex": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/sucrase": {
"version": "3.35.1",
"resolved": "https://registry.npmjs.org/sucrase/-/sucrase-3.35.1.tgz",
@ -2906,6 +3453,13 @@
}
}
},
"node_modules/vue-component-type-helpers": {
"version": "2.2.12",
"resolved": "https://registry.npmjs.org/vue-component-type-helpers/-/vue-component-type-helpers-2.2.12.tgz",
"integrity": "sha512-YbGqHZ5/eW4SnkPNR44mKVc6ZKQoRs/Rux1sxC6rdwXb4qpbOSYfDr9DsTHolOTGmIKgM9j141mZbBeg05R1pw==",
"dev": true,
"license": "MIT"
},
"node_modules/vue-demi": {
"version": "0.14.10",
"resolved": "https://registry.npmjs.org/vue-demi/-/vue-demi-0.14.10.tgz",
@ -2964,6 +3518,32 @@
"typescript": ">=5.0.0"
}
},
"node_modules/whatwg-mimetype": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-3.0.0.tgz",
"integrity": "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=12"
}
},
"node_modules/which": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/which/-/which-2.0.2.tgz",
"integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==",
"dev": true,
"license": "ISC",
"dependencies": {
"isexe": "^2.0.0"
},
"bin": {
"node-which": "bin/node-which"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/why-is-node-running": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz",
@ -2980,6 +3560,126 @@
"engines": {
"node": ">=8"
}
},
"node_modules/wrap-ansi": {
"version": "8.1.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz",
"integrity": "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-styles": "^6.1.0",
"string-width": "^5.0.1",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrap-ansi-cjs": {
"name": "wrap-ansi",
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-styles": "^4.0.0",
"string-width": "^4.1.0",
"strip-ansi": "^6.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrap-ansi-cjs/node_modules/ansi-regex": {
"version": "5.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz",
"integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=8"
}
},
"node_modules/wrap-ansi-cjs/node_modules/ansi-styles": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz",
"integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==",
"dev": true,
"license": "MIT",
"dependencies": {
"color-convert": "^2.0.1"
},
"engines": {
"node": ">=8"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/wrap-ansi-cjs/node_modules/emoji-regex": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==",
"dev": true,
"license": "MIT"
},
"node_modules/wrap-ansi-cjs/node_modules/string-width": {
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dev": true,
"license": "MIT",
"dependencies": {
"emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0",
"strip-ansi": "^6.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/wrap-ansi-cjs/node_modules/strip-ansi": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dev": true,
"license": "MIT",
"dependencies": {
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/ws": {
"version": "8.19.0",
"resolved": "https://registry.npmjs.org/ws/-/ws-8.19.0.tgz",
"integrity": "sha512-blAT2mjOEIi0ZzruJfIhb3nps74PRWTCz1IjglWEEpQl5XS/UNama6u2/rjFkDDouqr4L67ry+1aGIALViWjDg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=10.0.0"
},
"peerDependencies": {
"bufferutil": "^4.0.1",
"utf-8-validate": ">=5.0.2"
},
"peerDependenciesMeta": {
"bufferutil": {
"optional": true
},
"utf-8-validate": {
"optional": true
}
}
}
}
}

2
frontend/package.json
View file

@ -21,7 +21,9 @@
"devDependencies": {
"@types/node": "^22.10.2",
"@vitejs/plugin-vue": "^5.2.1",
"@vue/test-utils": "^2.4.6",
"autoprefixer": "^10.4.21",
"happy-dom": "^20.7.0",
"postcss": "^8.5.3",
"tailwindcss": "^3.4.17",
"typescript": "^5.7.2",

51
frontend/src/composables/useRequestState.test.ts Normal file
View file

@ -0,0 +1,51 @@
import { describe, expect, it } from "vitest";
import { ApiRequestError } from "@/lib/api/errors";
import { useRequestState } from "@/composables/useRequestState";
describe("useRequestState", () => {
it("initializes with all values null", () => {
const { errorMessage, errorRequestId, successMessage } = useRequestState();
expect(errorMessage.value).toBeNull();
expect(errorRequestId.value).toBeNull();
expect(successMessage.value).toBeNull();
});
it("assignError extracts message and requestId from ApiRequestError", () => {
const { errorMessage, errorRequestId, assignError } = useRequestState();
assignError(
new ApiRequestError({ status: 409, code: "conflict", message: "Conflict occurred", requestId: "req-42" }),
"fallback",
);
expect(errorMessage.value).toBe("Conflict occurred");
expect(errorRequestId.value).toBe("req-42");
});
it("assignError uses Error.message for generic errors", () => {
const { errorMessage, errorRequestId, assignError } = useRequestState();
assignError(new Error("something broke"), "fallback");
expect(errorMessage.value).toBe("something broke");
expect(errorRequestId.value).toBeNull();
});
it("assignError uses fallback for non-Error values", () => {
const { errorMessage, assignError } = useRequestState();
assignError("not an error object", "fallback text");
expect(errorMessage.value).toBe("fallback text");
});
it("setSuccess sets the success message", () => {
const { successMessage, setSuccess } = useRequestState();
setSuccess("Operation completed");
expect(successMessage.value).toBe("Operation completed");
});
it("clearAlerts resets all messages", () => {
const { errorMessage, errorRequestId, successMessage, assignError, setSuccess, clearAlerts } = useRequestState();
assignError(new ApiRequestError({ status: 500, code: "err", message: "fail", requestId: "r1" }), "fb");
setSuccess("ok");
clearAlerts();
expect(errorMessage.value).toBeNull();
expect(errorRequestId.value).toBeNull();
expect(successMessage.value).toBeNull();
});
});

40
frontend/src/composables/useRequestState.ts Normal file
View file

@ -0,0 +1,40 @@
import { ref } from "vue";
import { ApiRequestError } from "@/lib/api/errors";
export function useRequestState() {
const errorMessage = ref<string | null>(null);
const errorRequestId = ref<string | null>(null);
const successMessage = ref<string | null>(null);
function clearAlerts(): void {
errorMessage.value = null;
errorRequestId.value = null;
successMessage.value = null;
}
function assignError(error: unknown, fallback: string): void {
if (error instanceof ApiRequestError) {
errorMessage.value = error.message;
errorRequestId.value = error.requestId;
return;
}
if (error instanceof Error && error.message) {
errorMessage.value = error.message;
return;
}
errorMessage.value = fallback;
}
function setSuccess(message: string): void {
successMessage.value = message;
}
return {
errorMessage,
errorRequestId,
successMessage,
clearAlerts,
assignError,
setSuccess,
};
}

139
frontend/src/features/publications/components/PublicationTableRow.test.ts Normal file
View file

@ -0,0 +1,139 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import PublicationTableRow from "./PublicationTableRow.vue";
import type { PublicationItem } from "@/features/publications";
function buildItem(overrides: Partial<PublicationItem> = {}): PublicationItem {
return {
publication_id: 1,
scholar_profile_id: 10,
title: "Test Publication",
year: 2025,
citation_count: 42,
pub_url: "https://example.com/pub",
pdf_url: null,
pdf_status: null,
is_read: false,
is_favorite: false,
is_new_in_latest_run: true,
scholar_label: "Dr. Test",
first_seen_at: "2025-01-15T00:00:00Z",
display_identifier: null,
...overrides,
} as PublicationItem;
}
const defaultProps = {
item: buildItem(),
itemKey: "pub-1",
selected: false,
favoriteUpdating: false,
retrying: false,
canRetry: false,
};
describe("PublicationTableRow", () => {
it("renders the publication title", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.text()).toContain("Test Publication");
});
it("renders scholar label", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.text()).toContain("Dr. Test");
});
it("renders year and citation count", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.text()).toContain("2025");
expect(wrapper.text()).toContain("42");
});
it("shows New badge when is_new_in_latest_run is true", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.text()).toContain("New");
});
it("shows Seen badge when is_new_in_latest_run is false", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, item: buildItem({ is_new_in_latest_run: false }) },
});
expect(wrapper.text()).toContain("Seen");
});
it("shows Unread badge for unread items", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.text()).toContain("Unread");
});
it("shows Read badge for read items", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, item: buildItem({ is_read: true }) },
});
expect(wrapper.text()).toContain("Read");
});
it("disables checkbox when item is read", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, item: buildItem({ is_read: true }) },
});
const checkbox = wrapper.find('input[type="checkbox"]');
expect((checkbox.element as HTMLInputElement).disabled).toBe(true);
});
it("emits toggle-selection when checkbox is changed", async () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
await wrapper.find('input[type="checkbox"]').trigger("change");
expect(wrapper.emitted("toggle-selection")).toHaveLength(1);
});
it("emits toggle-favorite when star button is clicked", async () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
await wrapper.find(".favorite-star-button").trigger("click");
expect(wrapper.emitted("toggle-favorite")).toHaveLength(1);
});
it("shows filled star when item is favorite", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, item: buildItem({ is_favorite: true }) },
});
expect(wrapper.find(".favorite-star-on").exists()).toBe(true);
expect(wrapper.text()).toContain("★");
});
it("shows empty star when item is not favorite", () => {
const wrapper = mount(PublicationTableRow, { props: defaultProps });
expect(wrapper.find(".favorite-star-off").exists()).toBe(true);
expect(wrapper.text()).toContain("☆");
});
it("shows Available link when pdf_url is set", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, item: buildItem({ pdf_url: "https://example.com/paper.pdf" }) },
});
expect(wrapper.text()).toContain("Available");
});
it("shows retry button when canRetry is true and no pdf_url", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, canRetry: true },
});
expect(wrapper.text()).toContain("Missing (Retry)");
});
it("emits retry-pdf when retry button is clicked", async () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, canRetry: true },
});
await wrapper.find(".pdf-retry-button").trigger("click");
expect(wrapper.emitted("retry-pdf")).toHaveLength(1);
});
it("shows Retrying... label when retrying", () => {
const wrapper = mount(PublicationTableRow, {
props: { ...defaultProps, canRetry: true, retrying: true },
});
expect(wrapper.text()).toContain("Retrying...");
});
});

178
frontend/src/features/publications/components/PublicationTableRow.vue Normal file
View file

@ -0,0 +1,178 @@
<script setup lang="ts">
import AppBadge from "@/components/ui/AppBadge.vue";
import type { PublicationItem } from "@/features/publications";
const props = defineProps<{
item: PublicationItem;
itemKey: string;
selected: boolean;
favoriteUpdating: boolean;
retrying: boolean;
canRetry: boolean;
}>();
const emit = defineEmits<{
(e: "toggle-selection", event: Event): void;
(e: "toggle-favorite"): void;
(e: "retry-pdf"): void;
}>();
function primaryUrl(): string | null {
return props.item.pub_url || props.item.pdf_url;
}
function identifierUrl(): string | null {
return props.item.display_identifier?.url ?? null;
}
function identifierLabel(): string | null {
return props.item.display_identifier?.label ?? null;
}
function pdfPendingLabel(): string {
if (props.item.pdf_status === "queued") return "Queued";
if (props.item.pdf_status === "running") return "Resolving...";
if (props.item.pdf_status === "failed") return "Missing";
return "Untracked";
}
function formatDate(value: string): string {
const asDate = new Date(value);
if (Number.isNaN(asDate.getTime())) return value;
return asDate.toLocaleDateString();
}
</script>
<template>
<tr>
<td>
<input
type="checkbox"
class="h-4 w-4 rounded border-stroke-interactive bg-surface-input text-brand-600 focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset"
:checked="selected"
:disabled="item.is_read"
:aria-label="`Select publication ${item.title}`"
@change="emit('toggle-selection', $event)"
/>
</td>
<td>
<button
type="button"
class="favorite-star-button"
:class="item.is_favorite ? 'favorite-star-on' : 'favorite-star-off'"
:aria-label="item.is_favorite ? `Remove ${item.title} from favorites` : `Add ${item.title} to favorites`"
:aria-pressed="item.is_favorite"
:disabled="favoriteUpdating"
@click="emit('toggle-favorite')"
>
{{ item.is_favorite ? "★" : "☆" }}
</button>
</td>
<td class="max-w-0">
<div class="grid min-w-0 gap-1">
<a
v-if="primaryUrl()"
:href="primaryUrl() || ''"
target="_blank"
rel="noreferrer"
class="link-inline block truncate font-medium"
:title="item.title"
>
{{ item.title }}
</a>
<span v-else class="block truncate font-medium" :title="item.title">{{ item.title }}</span>
<a
v-if="identifierUrl()"
:href="identifierUrl() || ''"
target="_blank"
rel="noreferrer"
class="link-inline block truncate text-xs"
:title="identifierLabel() || ''"
>
{{ identifierLabel() }}
</a>
</div>
</td>
<td>
<span class="block truncate" :title="item.scholar_label">{{ item.scholar_label }}</span>
</td>
<td class="whitespace-nowrap">
<a
v-if="item.pdf_url"
:href="item.pdf_url"
target="_blank"
rel="noreferrer"
class="pdf-link-button"
title="Open PDF"
>
<svg class="mr-1 h-3 w-3" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 6H6a2 2 0 00-2 2v10a2 2 0 002 2h10a2 2 0 002-2v-4M14 4h6m0 0v6m0-6L10 14" />
</svg>
Available
</a>
<button
v-else-if="canRetry"
type="button"
class="pdf-retry-button"
:disabled="retrying"
@click="emit('retry-pdf')"
>
<svg v-if="retrying" class="mr-1 h-3 w-3 animate-spin" viewBox="0 0 24 24" fill="none">
<circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>
<path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
</svg>
{{ retrying ? "Retrying..." : "Missing (Retry)" }}
</button>
<span v-else class="pdf-state-label" :class="{ 'bg-surface-accent-muted border-accent-300 text-accent-700': item.pdf_status === 'running' || item.pdf_status === 'queued' }">
<svg v-if="item.pdf_status === 'running'" class="mr-1 h-3 w-3 animate-spin text-accent-600" viewBox="0 0 24 24" fill="none">
<circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4"></circle>
<path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
</svg>
{{ pdfPendingLabel() }}
</span>
</td>
<td class="whitespace-nowrap">{{ item.year ?? "n/a" }}</td>
<td class="whitespace-nowrap">{{ item.citation_count }}</td>
<td>
<div class="status-badges-row">
<AppBadge :tone="item.is_new_in_latest_run ? 'info' : 'neutral'">
{{ item.is_new_in_latest_run ? "New" : "Seen" }}
</AppBadge>
<AppBadge :tone="item.is_read ? 'success' : 'warning'">
{{ item.is_read ? "Read" : "Unread" }}
</AppBadge>
</div>
</td>
<td class="whitespace-nowrap">{{ formatDate(item.first_seen_at) }}</td>
</tr>
</template>
<style scoped>
.favorite-star-button {
@apply inline-flex h-7 w-7 items-center justify-center rounded-full border text-sm leading-none transition focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset disabled:cursor-not-allowed disabled:opacity-50;
}
.favorite-star-on {
@apply border-warning-300 bg-warning-100 text-warning-700 hover:bg-warning-200;
}
.favorite-star-off {
@apply border-stroke-default bg-surface-card-muted text-ink-muted hover:border-stroke-interactive hover:text-ink-secondary;
}
.pdf-link-button {
@apply inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium border-state-success-border bg-state-success-bg text-state-success-text shadow-sm transition hover:brightness-95 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset;
}
.pdf-retry-button {
@apply inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium border-state-warning-border bg-state-warning-bg text-state-warning-text transition hover:brightness-95 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset disabled:cursor-not-allowed disabled:opacity-50;
}
.pdf-state-label {
@apply inline-flex items-center rounded-full border px-2 py-0.5 text-xs font-medium border-stroke-default bg-surface-card-muted text-secondary;
}
.status-badges-row {
@apply inline-flex items-center gap-1 whitespace-nowrap;
}
</style>

89
frontend/src/features/publications/components/PublicationToolbar.test.ts Normal file
View file

@ -0,0 +1,89 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import PublicationToolbar from "./PublicationToolbar.vue";
const defaultProps = {
mode: "all" as const,
selectedScholarFilter: "",
searchQuery: "",
favoriteOnly: false,
actionBusy: false,
loading: false,
scholars: [],
startRunDisabled: false,
startRunDisabledReason: null,
startRunButtonLabel: "Check now",
};
describe("PublicationToolbar", () => {
it("renders mode, scholar filter, and search inputs", () => {
const wrapper = mount(PublicationToolbar, { props: defaultProps });
expect(wrapper.text()).toContain("Status");
expect(wrapper.text()).toContain("Scholar");
expect(wrapper.text()).toContain("Search");
});
it("renders the start run button with provided label", () => {
const wrapper = mount(PublicationToolbar, { props: defaultProps });
expect(wrapper.text()).toContain("Check now");
});
it("disables start run button when startRunDisabled is true", () => {
const wrapper = mount(PublicationToolbar, {
props: { ...defaultProps, startRunDisabled: true, startRunDisabledReason: "Cooldown active" },
});
const buttons = wrapper.findAll("button");
const runButton = buttons.find((b) => b.text().includes("Check now"));
expect(runButton?.attributes("disabled")).toBeDefined();
});
it("emits start-run when start button is clicked", async () => {
const wrapper = mount(PublicationToolbar, { props: defaultProps });
const buttons = wrapper.findAll("button");
const runButton = buttons.find((b) => b.text().includes("Check now"));
await runButton?.trigger("click");
expect(wrapper.emitted("start-run")).toHaveLength(1);
});
it("emits favorite-only-changed when star filter is clicked", async () => {
const wrapper = mount(PublicationToolbar, { props: defaultProps });
await wrapper.find(".favorite-filter-button").trigger("click");
expect(wrapper.emitted("favorite-only-changed")).toHaveLength(1);
});
it("shows active star filter style when favoriteOnly is true", () => {
const wrapper = mount(PublicationToolbar, { props: { ...defaultProps, favoriteOnly: true } });
expect(wrapper.find(".favorite-filter-on").exists()).toBe(true);
});
it("shows inactive star filter style when favoriteOnly is false", () => {
const wrapper = mount(PublicationToolbar, { props: defaultProps });
expect(wrapper.find(".favorite-filter-off").exists()).toBe(true);
});
it("renders scholar options from props", () => {
const scholars = [
{ id: 1, scholar_id: "abc123", display_name: "Alice", profile_image_url: null, profile_image_source: "none" as const, is_enabled: true, baseline_completed: true, last_run_dt: null, last_run_status: null },
{ id: 2, scholar_id: "def456", display_name: "", profile_image_url: null, profile_image_source: "none" as const, is_enabled: true, baseline_completed: true, last_run_dt: null, last_run_status: null },
];
const wrapper = mount(PublicationToolbar, { props: { ...defaultProps, scholars } });
expect(wrapper.text()).toContain("Alice");
expect(wrapper.text()).toContain("def456");
});
it("shows clear button only when search query is non-empty", () => {
const empty = mount(PublicationToolbar, { props: defaultProps });
expect(empty.text()).not.toContain("Clear");
const withQuery = mount(PublicationToolbar, { props: { ...defaultProps, searchQuery: "test" } });
expect(withQuery.text()).toContain("Clear");
});
it("emits reset-search when clear button is clicked", async () => {
const wrapper = mount(PublicationToolbar, { props: { ...defaultProps, searchQuery: "test" } });
const clearButton = wrapper.findAll("button").find((b) => b.text().includes("Clear"));
await clearButton?.trigger("click");
expect(wrapper.emitted("reset-search")).toHaveLength(1);
});
});

148
frontend/src/features/publications/components/PublicationToolbar.vue Normal file
View file

@ -0,0 +1,148 @@
<script setup lang="ts">
import AppButton from "@/components/ui/AppButton.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
import AppRefreshButton from "@/components/ui/AppRefreshButton.vue";
import AppSelect from "@/components/ui/AppSelect.vue";
import type { PublicationMode } from "@/features/publications";
import type { ScholarProfile } from "@/features/scholars";
defineProps<{
mode: PublicationMode;
selectedScholarFilter: string;
searchQuery: string;
favoriteOnly: boolean;
actionBusy: boolean;
loading: boolean;
scholars: ScholarProfile[];
startRunDisabled: boolean;
startRunDisabledReason: string | null;
startRunButtonLabel: string;
}>();
const emit = defineEmits<{
(e: "update:mode", value: PublicationMode): void;
(e: "update:selectedScholarFilter", value: string): void;
(e: "update:searchQuery", value: string): void;
(e: "mode-changed"): void;
(e: "scholar-filter-changed"): void;
(e: "favorite-only-changed"): void;
(e: "reset-search"): void;
(e: "start-run"): void;
(e: "refresh"): void;
}>();
function scholarLabel(item: ScholarProfile): string {
return item.display_name || item.scholar_id;
}
</script>
<template>
<div class="grid gap-3 xl:grid-cols-[minmax(0,13rem)_minmax(0,18rem)_minmax(0,1fr)_auto] xl:items-end">
<div class="grid gap-1 text-xs text-secondary">
<div class="flex items-center gap-1">
<span>Status</span>
<AppHelpHint text="All shows full history. Unread and New narrow the dataset before search." />
</div>
<AppSelect
:model-value="mode"
:disabled="actionBusy"
@update:model-value="emit('update:mode', $event as PublicationMode)"
@change="emit('mode-changed')"
>
<option value="all">All records</option>
<option value="unread">Unread</option>
<option value="latest">New (latest run)</option>
</AppSelect>
</div>
<label class="grid gap-1 text-xs text-secondary" for="publications-scholar-filter">
<span class="inline-flex items-center gap-1">
Scholar
<AppHelpHint text="Filter to one tracked scholar profile. Filter is synced to URL query." />
</span>
<AppSelect
id="publications-scholar-filter"
:model-value="selectedScholarFilter"
:disabled="actionBusy"
@update:model-value="emit('update:selectedScholarFilter', $event as string)"
@change="emit('scholar-filter-changed')"
>
<option value="">All scholars</option>
<option v-for="scholar in scholars" :key="scholar.id" :value="String(scholar.id)">
{{ scholarLabel(scholar) }}
</option>
</AppSelect>
</label>
<label class="grid gap-1 text-xs text-secondary" for="publications-search-input">
<span class="inline-flex items-center gap-1">
Search
<AppHelpHint text="Searches title, scholar name, and venue." />
</span>
<div class="flex min-w-0 items-center gap-2">
<AppInput
id="publications-search-input"
:model-value="searchQuery"
placeholder="Search title, scholar, venue, year"
:disabled="loading"
@update:model-value="emit('update:searchQuery', $event as string)"
/>
<AppButton
v-if="searchQuery.trim().length > 0"
variant="secondary"
class="shrink-0"
:disabled="loading"
@click="emit('reset-search')"
>
Clear
</AppButton>
</div>
</label>
<div class="flex flex-wrap items-end justify-end gap-2">
<button
type="button"
class="favorite-filter-button"
:class="favoriteOnly ? 'favorite-filter-on' : 'favorite-filter-off'"
:disabled="actionBusy"
:title="favoriteOnly ? 'Favorites-only filter is active' : 'Show only favorites'"
:aria-pressed="favoriteOnly"
:aria-label="favoriteOnly ? 'Disable favorites-only filter' : 'Enable favorites-only filter'"
@click="emit('favorite-only-changed')"
>
<span aria-hidden="true">{{ favoriteOnly ? "★" : "☆" }}</span>
</button>
<AppButton
variant="secondary"
:disabled="startRunDisabled"
:title="startRunDisabledReason || undefined"
@click="emit('start-run')"
>
{{ startRunButtonLabel }}
</AppButton>
<AppRefreshButton
variant="ghost"
:disabled="loading"
:loading="loading"
title="Refresh publications"
loading-title="Refreshing publications"
@click="emit('refresh')"
/>
</div>
</div>
</template>
<style scoped>
.favorite-filter-button {
@apply inline-flex min-h-10 h-10 w-10 items-center justify-center rounded-full border text-base leading-none transition focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset disabled:cursor-not-allowed disabled:opacity-50;
}
.favorite-filter-on {
@apply border-warning-300 bg-warning-100 text-warning-700 hover:bg-warning-200;
}
.favorite-filter-off {
@apply border-stroke-default bg-surface-card-muted text-ink-muted hover:border-stroke-interactive hover:text-ink-secondary;
}
</style>

118
frontend/src/features/publications/composables/usePublicationData.test.ts Normal file
View file

@ -0,0 +1,118 @@
import { describe, expect, it, vi, beforeEach } from "vitest";
import { createPinia, setActivePinia } from "pinia";
vi.mock("vue-router", () => ({
useRoute: vi.fn(() => ({ query: {} })),
useRouter: vi.fn(() => ({ replace: vi.fn() })),
}));
vi.mock("@/features/publications", async (importOriginal) => {
const original = (await importOriginal()) as Record<string, unknown>;
return {
...original,
listPublications: vi.fn(),
};
});
vi.mock("@/features/scholars", () => ({
listScholars: vi.fn(),
}));
import { listPublications } from "@/features/publications";
import { listScholars } from "@/features/scholars";
import { usePublicationData } from "./usePublicationData";
const mockedListPublications = vi.mocked(listPublications);
const mockedListScholars = vi.mocked(listScholars);
describe("usePublicationData", () => {
beforeEach(() => {
setActivePinia(createPinia());
mockedListPublications.mockReset();
mockedListScholars.mockReset();
});
it("initializes with default filter values", () => {
const data = usePublicationData();
expect(data.mode.value).toBe("all");
expect(data.favoriteOnly.value).toBe(false);
expect(data.searchQuery.value).toBe("");
expect(data.selectedScholarFilter.value).toBe("");
expect(data.loading.value).toBe(true);
});
it("loadScholarFilters calls listScholars", async () => {
mockedListScholars.mockResolvedValueOnce([
{ id: 1, scholar_id: "abc", display_name: "Test", profile_image_url: null, profile_image_source: "none" as const, is_enabled: true, baseline_completed: true, last_run_dt: null, last_run_status: null },
]);
const data = usePublicationData();
await data.loadScholarFilters();
expect(mockedListScholars).toHaveBeenCalled();
expect(data.scholars.value).toHaveLength(1);
});
it("loadPublications calls listPublications with current filters", async () => {
mockedListPublications.mockResolvedValueOnce({
publications: [],
mode: "all" as const,
favorite_only: false,
selected_scholar_profile_id: null,
unread_count: 0,
favorites_count: 0,
latest_count: 0,
new_count: 0,
total_count: 0,
page: 1,
page_size: 25,
snapshot: "",
has_next: false,
has_prev: false,
});
const data = usePublicationData();
await data.loadPublications();
expect(mockedListPublications).toHaveBeenCalled();
});
it("resetPageAndSnapshot resets page to 1", () => {
const data = usePublicationData();
data.currentPage.value = 5;
data.resetPageAndSnapshot();
expect(data.currentPage.value).toBe(1);
});
it("toggleSort reverses direction for same key", async () => {
mockedListPublications.mockResolvedValue({
publications: [],
mode: "all" as const,
favorite_only: false,
selected_scholar_profile_id: null,
unread_count: 0,
favorites_count: 0,
latest_count: 0,
new_count: 0,
total_count: 0,
page: 1,
page_size: 25,
snapshot: "",
has_next: false,
has_prev: false,
});
const data = usePublicationData();
const initialDirection = data.sortDirection.value;
await data.toggleSort(data.sortKey.value);
expect(data.sortDirection.value).toBe(initialDirection === "asc" ? "desc" : "asc");
});
it("sortMarker returns arrow indicators", () => {
const data = usePublicationData();
expect(data.sortMarker(data.sortKey.value)).toMatch(/[↑↓]/);
expect(data.sortMarker("year" as never)).toBe("↕");
});
it("computed pagination properties work correctly", () => {
const data = usePublicationData();
expect(data.hasPrevPage.value).toBe(false);
expect(data.totalPages.value).toBe(1);
expect(data.totalCount.value).toBe(0);
});
});

338
frontend/src/features/publications/composables/usePublicationData.ts Normal file
View file

@ -0,0 +1,338 @@
import { computed, ref, watch } from "vue";
import { useRoute, useRouter } from "vue-router";
import {
listPublications,
type PublicationItem,
type PublicationMode,
type PublicationsResult,
} from "@/features/publications";
import { listScholars, type ScholarProfile } from "@/features/scholars";
import { ApiRequestError } from "@/lib/api/errors";
import { useRunStatusStore } from "@/stores/run_status";
export type PublicationSortKey =
| "title"
| "scholar"
| "year"
| "citations"
| "first_seen"
| "pdf_status";
export function publicationKey(item: PublicationItem): string {
return `${item.scholar_profile_id}:${item.publication_id}`;
}
export function usePublicationData() {
const loading = ref(true);
const mode = ref<PublicationMode>("all");
const favoriteOnly = ref(false);
const selectedScholarFilter = ref("");
const searchQuery = ref("");
const sortKey = ref<PublicationSortKey>("first_seen");
const sortDirection = ref<"asc" | "desc">("desc");
const currentPage = ref(1);
const pageSize = ref("50");
const publicationSnapshot = ref<string | null>(null);
const scholars = ref<ScholarProfile[]>([]);
const listState = ref<PublicationsResult | null>(null);
const errorMessage = ref<string | null>(null);
const errorRequestId = ref<string | null>(null);
const successMessage = ref<string | null>(null);
const route = useRoute();
const router = useRouter();
const textCollator = new Intl.Collator(undefined, { sensitivity: "base", numeric: true });
const runStatus = useRunStatusStore();
// --- Route sync ---
function normalizeScholarFilterQuery(value: unknown): string {
if (Array.isArray(value)) return normalizeScholarFilterQuery(value[0]);
if (typeof value !== "string") return "";
const parsed = Number(value);
return Number.isInteger(parsed) && parsed > 0 ? String(parsed) : "";
}
function normalizeFavoriteOnlyQuery(value: unknown): boolean {
if (Array.isArray(value)) return normalizeFavoriteOnlyQuery(value[0]);
if (typeof value !== "string") return false;
const normalized = value.trim().toLowerCase();
return normalized === "1" || normalized === "true" || normalized === "yes";
}
function normalizePageQuery(value: unknown): number {
if (Array.isArray(value)) return normalizePageQuery(value[0]);
if (typeof value !== "string") return 1;
const parsed = Number(value);
return Number.isInteger(parsed) && parsed > 0 ? parsed : 1;
}
function syncFiltersFromRoute(): boolean {
let changed = false;
const nextScholar = normalizeScholarFilterQuery(route.query.scholar);
const nextFavoriteOnly = normalizeFavoriteOnlyQuery(route.query.favorite);
const nextPage = normalizePageQuery(route.query.page);
if (selectedScholarFilter.value !== nextScholar) { selectedScholarFilter.value = nextScholar; changed = true; }
if (favoriteOnly.value !== nextFavoriteOnly) { favoriteOnly.value = nextFavoriteOnly; changed = true; }
if (currentPage.value !== nextPage) { currentPage.value = nextPage; changed = true; }
return changed;
}
async function syncFiltersToRoute(): Promise<void> {
const nextScholar = selectedScholarFilter.value.trim();
const currentScholar = normalizeScholarFilterQuery(route.query.scholar);
const currentFavoriteOnly = normalizeFavoriteOnlyQuery(route.query.favorite);
const currentPageQuery = normalizePageQuery(route.query.page);
if (
nextScholar === currentScholar
&& favoriteOnly.value === currentFavoriteOnly
&& currentPage.value === currentPageQuery
) {
return;
}
await router.replace({
query: {
...route.query,
scholar: nextScholar || undefined,
favorite: favoriteOnly.value ? "1" : undefined,
page: currentPage.value > 1 ? String(currentPage.value) : undefined,
},
});
}
// --- Sorting helpers ---
function publicationSortValue(item: PublicationItem, key: PublicationSortKey): number | string {
if (key === "title") return item.title;
if (key === "scholar") return item.scholar_label;
if (key === "year") return item.year ?? -1;
if (key === "citations") return item.citation_count;
if (key === "pdf_status") {
if (item.pdf_url || item.pdf_status === "resolved") return 4;
if (item.pdf_status === "running") return 3;
if (item.pdf_status === "queued") return 2;
if (item.pdf_status === "failed") return 0;
return 1;
}
const timestamp = Date.parse(item.first_seen_at);
return Number.isNaN(timestamp) ? 0 : timestamp;
}
// --- Computed ---
const selectedScholarName = computed(() => {
const selectedId = Number(selectedScholarFilter.value);
if (!Number.isInteger(selectedId) || selectedId <= 0) return "all scholars";
const profile = scholars.value.find((item) => item.id === selectedId);
return profile ? (profile.display_name || profile.scholar_id) : "the selected scholar";
});
const filteredPublications = computed(() => {
let stream = [...runStatus.livePublications];
if (favoriteOnly.value) stream = stream.filter((p) => p.is_favorite);
if (mode.value === "unread") stream = stream.filter((p) => !p.is_read);
const selectedScholarId = Number(selectedScholarFilter.value);
if (Number.isInteger(selectedScholarId) && selectedScholarId > 0) {
stream = stream.filter((p) => p.scholar_profile_id === selectedScholarId);
}
const base = listState.value?.publications ?? [];
const merged = [...stream, ...base];
const seenKeys = new Set<string>();
const deduped: typeof base = [];
for (const item of merged) {
const key = publicationKey(item);
if (!seenKeys.has(key)) { seenKeys.add(key); deduped.push(item); }
}
return deduped;
});
const sortedPublications = computed(() => {
const direction = sortDirection.value === "asc" ? 1 : -1;
return [...filteredPublications.value].sort((left, right) => {
const leftValue = publicationSortValue(left, sortKey.value);
const rightValue = publicationSortValue(right, sortKey.value);
let comparison = 0;
if (typeof leftValue === "string" && typeof rightValue === "string") {
comparison = textCollator.compare(leftValue, rightValue);
} else {
comparison = Number(leftValue) - Number(rightValue);
}
if (comparison !== 0) return comparison * direction;
return right.publication_id - left.publication_id;
});
});
const visibleUnreadKeys = computed(() => {
const keys = new Set<string>();
for (const item of sortedPublications.value) {
if (!item.is_read) keys.add(publicationKey(item));
}
return keys;
});
const pageSizeValue = computed(() => {
const parsed = Number(pageSize.value);
if (!Number.isInteger(parsed)) return 50;
return Math.max(10, Math.min(200, parsed));
});
const hasNextPage = computed(() => Boolean(listState.value?.has_next));
const hasPrevPage = computed(() => Boolean(listState.value?.has_prev));
const totalPages = computed(() => {
if (!listState.value) return 1;
return Math.max(1, Math.ceil(listState.value.total_count / Math.max(listState.value.page_size, 1)));
});
const totalCount = computed(() => listState.value?.total_count ?? 0);
const visibleCount = computed(() => sortedPublications.value.length);
const visibleUnreadCount = computed(() => visibleUnreadKeys.value.size);
const visibleFavoriteCount = computed(
() => sortedPublications.value.filter((item) => item.is_favorite).length,
);
// --- Data loading ---
async function loadScholarFilters(): Promise<void> {
try { scholars.value = await listScholars(); } catch { scholars.value = []; }
}
function selectedScholarId(): number | undefined {
const parsed = Number(selectedScholarFilter.value);
return Number.isInteger(parsed) && parsed > 0 ? parsed : undefined;
}
async function loadPublications(): Promise<void> {
loading.value = true;
errorMessage.value = null;
errorRequestId.value = null;
try {
listState.value = await listPublications({
mode: mode.value,
favoriteOnly: favoriteOnly.value,
scholarProfileId: selectedScholarId(),
search: searchQuery.value.trim() || undefined,
sortBy: sortKey.value,
sortDir: sortDirection.value,
page: currentPage.value,
pageSize: pageSizeValue.value,
snapshot: publicationSnapshot.value ?? undefined,
});
publicationSnapshot.value = listState.value.snapshot;
currentPage.value = listState.value.page;
pageSize.value = String(listState.value.page_size);
} catch (error) {
listState.value = null;
if (error instanceof ApiRequestError) {
errorMessage.value = error.message;
errorRequestId.value = error.requestId;
} else {
errorMessage.value = "Unable to load publications.";
}
} finally {
loading.value = false;
}
}
function resetPageAndSnapshot(): void {
currentPage.value = 1;
publicationSnapshot.value = null;
}
async function toggleSort(nextKey: PublicationSortKey): Promise<void> {
if (sortKey.value === nextKey) {
sortDirection.value = sortDirection.value === "asc" ? "desc" : "asc";
} else {
sortKey.value = nextKey;
sortDirection.value = nextKey === "first_seen" || nextKey === "pdf_status" ? "desc" : "asc";
}
resetPageAndSnapshot();
await loadPublications();
}
function sortMarker(key: PublicationSortKey): string {
if (sortKey.value !== key) return "↕";
return sortDirection.value === "asc" ? "↑" : "↓";
}
function replacePublication(updated: PublicationItem): void {
if (!listState.value) return;
listState.value.publications = listState.value.publications.map((item) =>
publicationKey(item) !== publicationKey(updated) ? item : updated,
);
}
// --- Search debounce watcher setup ---
let searchDebounceTimer: ReturnType<typeof setTimeout> | null = null;
watch(searchQuery, () => {
if (searchDebounceTimer !== null) clearTimeout(searchDebounceTimer);
searchDebounceTimer = setTimeout(() => {
resetPageAndSnapshot();
void loadPublications();
}, 300);
});
// --- Run-triggered refresh watcher ---
watch(
() => runStatus.latestRun,
async (nextRun, previousRun) => {
const nextRunId = nextRun && (nextRun.status === "running" || nextRun.status === "resolving") ? nextRun.id : null;
const previousRunId = previousRun && (previousRun.status === "running" || previousRun.status === "resolving") ? previousRun.id : null;
if (nextRunId === null || nextRunId === previousRunId) return;
resetPageAndSnapshot();
await loadPublications();
},
);
// --- Route watcher ---
watch(
() => [route.query.scholar, route.query.favorite, route.query.page],
async () => {
const previousScholar = selectedScholarFilter.value;
const previousFavorite = favoriteOnly.value;
const changed = syncFiltersFromRoute();
if (!changed) return;
if (selectedScholarFilter.value !== previousScholar || favoriteOnly.value !== previousFavorite) {
publicationSnapshot.value = null;
}
await loadPublications();
},
);
return {
loading,
mode,
favoriteOnly,
selectedScholarFilter,
searchQuery,
sortKey,
sortDirection,
currentPage,
pageSize,
scholars,
listState,
errorMessage,
errorRequestId,
successMessage,
selectedScholarName,
sortedPublications,
visibleUnreadKeys,
pageSizeValue,
hasNextPage,
hasPrevPage,
totalPages,
totalCount,
visibleCount,
visibleUnreadCount,
visibleFavoriteCount,
loadScholarFilters,
loadPublications,
resetPageAndSnapshot,
toggleSort,
sortMarker,
replacePublication,
syncFiltersFromRoute,
syncFiltersToRoute,
};
}

76
frontend/src/features/scholars/components/ScholarBatchAdd.test.ts Normal file
View file

@ -0,0 +1,76 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import ScholarBatchAdd from "./ScholarBatchAdd.vue";
const defaultProps = { saving: false, loading: false };
describe("ScholarBatchAdd", () => {
it("renders the heading and input", () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
expect(wrapper.text()).toContain("Add Scholar Profiles");
expect(wrapper.find("textarea").exists()).toBe(true);
});
it("shows parsed ID count as 0 for empty input", () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
expect(wrapper.text()).toContain("Parsed IDs:");
});
it("parses bare scholar IDs from textarea input", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("A-UbBTPM15wL");
expect(wrapper.text()).toContain("1");
});
it("parses scholar IDs from Google Scholar URLs", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue(
"https://scholar.google.com/citations?hl=en&user=A-UbBTPM15wL",
);
expect(wrapper.text()).toContain("1");
});
it("deduplicates IDs from mixed input", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue(
"A-UbBTPM15wL\nhttps://scholar.google.com/citations?user=A-UbBTPM15wL",
);
expect(wrapper.text()).toContain("1");
});
it("parses multiple IDs separated by commas", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("A-UbBTPM15wL, B-UbBTPM15wL");
expect(wrapper.text()).toContain("2");
});
it("emits add-scholars with parsed IDs on submit", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("A-UbBTPM15wL");
await wrapper.find("form").trigger("submit");
const emitted = wrapper.emitted("add-scholars");
expect(emitted).toHaveLength(1);
expect(emitted![0][0]).toEqual(["A-UbBTPM15wL"]);
});
it("clears textarea after successful submit", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
const textarea = wrapper.find("textarea");
await textarea.setValue("A-UbBTPM15wL");
await wrapper.find("form").trigger("submit");
expect((textarea.element as HTMLTextAreaElement).value).toBe("");
});
it("does not emit when input contains no valid IDs", async () => {
const wrapper = mount(ScholarBatchAdd, { props: defaultProps });
await wrapper.find("textarea").setValue("not-a-valid-id");
await wrapper.find("form").trigger("submit");
expect(wrapper.emitted("add-scholars")).toBeUndefined();
});
it("shows Adding... label when saving prop is true", () => {
const wrapper = mount(ScholarBatchAdd, { props: { ...defaultProps, saving: true } });
expect(wrapper.text()).toContain("Adding...");
});
});

89
frontend/src/features/scholars/components/ScholarBatchAdd.vue Normal file
View file

@ -0,0 +1,89 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
defineProps<{
saving: boolean;
loading: boolean;
}>();
const emit = defineEmits<{
(e: "add-scholars", ids: string[]): void;
}>();
const scholarBatchInput = ref("");
const SCHOLAR_ID_PATTERN = /^[a-zA-Z0-9_-]{12}$/;
const URL_USER_PARAM_PATTERN = /(?:\?|&)user=([a-zA-Z0-9_-]{12})(?:&|#|$)/i;
function parseScholarIds(raw: string): string[] {
const ordered: string[] = [];
const seen = new Set<string>();
const tokens = raw.split(/[\s,;]+/).map((v) => v.trim()).filter((v) => v.length > 0);
for (const token of tokens) {
let candidate: string | null = null;
if (SCHOLAR_ID_PATTERN.test(token)) candidate = token;
if (!candidate) {
const directParamMatch = token.match(URL_USER_PARAM_PATTERN);
if (directParamMatch) candidate = directParamMatch[1];
}
if (!candidate && token.includes("scholar.google")) {
try {
const parsed = new URL(token);
const userParam = parsed.searchParams.get("user");
if (userParam && SCHOLAR_ID_PATTERN.test(userParam)) candidate = userParam;
} catch (_error) { /* Ignore non-URL tokens. */ }
}
if (!candidate || seen.has(candidate)) continue;
seen.add(candidate);
ordered.push(candidate);
}
return ordered;
}
const parsedBatchCount = computed(() => parseScholarIds(scholarBatchInput.value).length);
function onSubmit(): void {
const ids = parseScholarIds(scholarBatchInput.value);
if (ids.length > 0) {
emit("add-scholars", ids);
scholarBatchInput.value = "";
}
}
</script>
<template>
<AppCard class="space-y-4 xl:flex xl:min-h-0 xl:flex-col">
<div class="space-y-1">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Add Scholar Profiles</h2>
<AppHelpHint text="A scholar profile is a Google Scholar author page that Scholarr will monitor for publication changes." />
</div>
<p class="text-sm text-secondary">Paste one or more Scholar IDs or profile URLs and add them in one action.</p>
</div>
<form class="grid gap-3" @submit.prevent="onSubmit">
<label class="grid gap-2 text-sm font-medium text-ink-secondary" for="scholar-batch-input">
<span>Scholar IDs or profile URLs</span>
<textarea
id="scholar-batch-input"
v-model="scholarBatchInput"
rows="5"
placeholder="A-UbBTPM15wL&#10;https://scholar.google.com/citations?hl=en&user=A-UbBTPM15wL"
class="w-full rounded-lg border border-stroke-interactive bg-surface-input px-3 py-2 text-sm text-ink-primary outline-none ring-focus-ring transition placeholder:text-ink-muted focus-visible:ring-2 focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset disabled:cursor-not-allowed disabled:opacity-60"
/>
</label>
<div class="flex flex-wrap items-center justify-between gap-2">
<p class="text-xs text-secondary">
Parsed IDs: <strong class="text-ink-primary">{{ parsedBatchCount }}</strong>
</p>
<AppButton type="submit" :disabled="saving || loading">
{{ saving ? "Adding..." : "Add scholars" }}
</AppButton>
</div>
</form>
</AppCard>
</template>

43
frontend/src/features/scholars/components/ScholarNameSearch.test.ts Normal file
View file

@ -0,0 +1,43 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import ScholarNameSearch from "./ScholarNameSearch.vue";
const defaultProps = {
trackedScholarIds: new Set<string>(),
addingCandidateScholarId: null,
};
describe("ScholarNameSearch", () => {
it("renders the search heading", () => {
const wrapper = mount(ScholarNameSearch, {
props: defaultProps,
global: { stubs: { ScholarAvatar: true } },
});
expect(wrapper.text()).toContain("Search by Name");
});
it("shows WIP badge indicating the feature is incomplete", () => {
const wrapper = mount(ScholarNameSearch, {
props: defaultProps,
global: { stubs: { ScholarAvatar: true } },
});
expect(wrapper.text()).toContain("WIP");
});
it("renders the search input", () => {
const wrapper = mount(ScholarNameSearch, {
props: defaultProps,
global: { stubs: { ScholarAvatar: true } },
});
expect(wrapper.find("form").exists()).toBe(true);
});
it("shows warning about Google Scholar login challenges", () => {
const wrapper = mount(ScholarNameSearch, {
props: defaultProps,
global: { stubs: { ScholarAvatar: true } },
});
expect(wrapper.text()).toContain("login challenge");
});
});

167
frontend/src/features/scholars/components/ScholarNameSearch.vue Normal file
View file

@ -0,0 +1,167 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import AsyncStateGate from "@/components/patterns/AsyncStateGate.vue";
import RequestStateAlerts from "@/components/patterns/RequestStateAlerts.vue";
import AppAlert from "@/components/ui/AppAlert.vue";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
import ScholarAvatar from "@/features/scholars/components/ScholarAvatar.vue";
import type { ScholarSearchCandidate, ScholarSearchResult } from "@/features/scholars";
defineProps<{
trackedScholarIds: Set<string>;
addingCandidateScholarId: string | null;
}>();
const emit = defineEmits<{
(e: "add-candidate", candidate: ScholarSearchCandidate): void;
}>();
const nameSearchWip = true;
const searchingByName = ref(false);
const searchQuery = ref("");
const searchResult = ref<ScholarSearchResult | null>(null);
const searchErrorMessage = ref<string | null>(null);
const searchErrorRequestId = ref<string | null>(null);
const searchHasRun = computed(() => searchResult.value !== null);
const searchIsDegraded = computed(() => {
if (!searchResult.value) return false;
return searchResult.value.state === "blocked_or_captcha" || searchResult.value.state === "network_error";
});
const searchStateTone = computed(() => {
const result = searchResult.value;
if (!result) return "neutral" as const;
if (result.state === "ok") return "success" as const;
if (result.state === "blocked_or_captcha" || result.state === "network_error") return "warning" as const;
return "neutral" as const;
});
const emptySearchCandidates = computed(() => {
const result = searchResult.value;
if (!result) return false;
return result.candidates.length === 0;
});
</script>
<template>
<AppCard class="relative space-y-4 select-none xl:flex xl:min-h-0 xl:flex-col xl:overflow-hidden">
<div class="flex flex-wrap items-start justify-between gap-2">
<div class="space-y-1">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Search by Name</h2>
<AppHelpHint text="Google Scholar now challenges this endpoint with account login and anti-bot checks, so this workflow stays disabled." />
</div>
<p class="text-sm text-secondary">
This helper remains paused because Google Scholar currently requires account login for reliable name search access.
</p>
</div>
<AppHelpHint text="Name search is kept as WIP. Use direct Scholar ID or profile URL adds for production tracking.">
<template #trigger>
<AppBadge tone="warning">WIP</AppBadge>
</template>
</AppHelpHint>
</div>
<form class="pointer-events-none flex flex-wrap items-center gap-2 opacity-60">
<div class="min-w-0 flex-1">
<AppInput
id="scholar-search-name"
v-model="searchQuery"
placeholder="e.g. Geoffrey Hinton"
:disabled="nameSearchWip"
/>
</div>
<AppButton type="button" :disabled="nameSearchWip">
Search
</AppButton>
</form>
<p class="text-xs text-secondary">
Direct Scholar ID/URL adds above are the supported path until name search can run without login challenges.
</p>
<div class="min-h-0 xl:flex-1 xl:overflow-y-auto xl:pr-1">
<RequestStateAlerts
:error-message="searchErrorMessage"
:error-request-id="searchErrorRequestId"
error-title="Search request failed"
/>
<AsyncStateGate :loading="searchingByName" :loading-lines="4" :show-empty="false">
<template v-if="searchResult">
<div class="flex flex-wrap items-center justify-between gap-2">
<p class="text-sm text-secondary">
{{ searchResult.candidates.length }} candidate{{ searchResult.candidates.length === 1 ? "" : "s" }}
for <strong class="text-ink-primary">{{ searchResult.query }}</strong>
</p>
<AppBadge :tone="searchStateTone">{{ searchResult.state }}</AppBadge>
</div>
<p v-if="searchResult.state !== 'ok' || searchResult.warnings.length > 0" class="text-xs text-secondary">
<span>State reason: <code>{{ searchResult.state_reason }}</code></span>
<span v-if="searchResult.action_hint">. {{ searchResult.action_hint }}</span>
<span v-if="searchResult.warnings.length > 0">. Warnings: {{ searchResult.warnings.join(", ") }}</span>
</p>
<AppAlert v-if="searchIsDegraded" tone="warning">
<template #title>Name search is degraded</template>
<p>This endpoint throttles aggressively to avoid blocks. Use Scholar URL/ID adds for dependable tracking.</p>
</AppAlert>
<AsyncStateGate
:loading="false"
:show-empty="true"
:empty="emptySearchCandidates"
empty-title="No scholar matches returned"
empty-body="Try another query later or add directly by Scholar URL/ID."
>
<ul class="grid gap-3">
<li
v-for="candidate in searchResult.candidates"
:key="candidate.scholar_id"
class="rounded-xl border border-stroke-default bg-surface-card-muted/70 p-3"
>
<div class="flex items-start gap-3">
<ScholarAvatar
size="sm"
:label="candidate.display_name"
:scholar-id="candidate.scholar_id"
:image-url="candidate.profile_image_url"
/>
<div class="min-w-0 flex-1 space-y-1">
<div class="flex flex-wrap items-center gap-2">
<strong class="text-sm text-ink-primary">{{ candidate.display_name }}</strong>
<code class="text-xs text-secondary">{{ candidate.scholar_id }}</code>
<AppBadge v-if="trackedScholarIds.has(candidate.scholar_id)" tone="success">Tracked</AppBadge>
</div>
<p class="truncate text-xs text-secondary">{{ candidate.affiliation || "No affiliation provided" }}</p>
<div class="flex flex-wrap items-center gap-2 text-xs text-secondary">
<span v-if="candidate.email_domain">Email: {{ candidate.email_domain }}</span>
<span v-if="candidate.cited_by_count !== null">Cited by: {{ candidate.cited_by_count }}</span>
</div>
</div>
<div class="grid shrink-0 gap-2">
<AppButton
:disabled="trackedScholarIds.has(candidate.scholar_id) || addingCandidateScholarId === candidate.scholar_id"
@click="emit('add-candidate', candidate)"
>
{{ addingCandidateScholarId === candidate.scholar_id ? "Adding..." : "Add" }}
</AppButton>
<a :href="candidate.profile_url" target="_blank" rel="noreferrer" class="link-inline text-xs text-center">
Open profile
</a>
</div>
</div>
</li>
</ul>
</AsyncStateGate>
</template>
<p v-else-if="!searchErrorMessage && !searchHasRun" class="text-sm text-secondary">
Name search remains disabled while login-gated responses are unresolved.
</p>
</AsyncStateGate>
</div>
</AppCard>
</template>

92
frontend/src/features/scholars/components/ScholarSettingsModal.test.ts Normal file
View file

@ -0,0 +1,92 @@
// @vitest-environment happy-dom
import { describe, expect, it } from "vitest";
import { mount } from "@vue/test-utils";
import ScholarSettingsModal from "./ScholarSettingsModal.vue";
import type { ScholarProfile } from "@/features/scholars";
function buildScholar(overrides: Partial<ScholarProfile> = {}): ScholarProfile {
return {
id: 1,
scholar_id: "abcDEF123456",
display_name: "Dr. Test Scholar",
profile_image_url: null,
profile_image_source: "none" as const,
is_enabled: true,
baseline_completed: true,
last_run_dt: null,
last_run_status: null,
...overrides,
};
}
const defaultProps = {
scholar: buildScholar(),
imageUrlDraft: "",
imageBusy: false,
imageSaving: false,
imageUploading: false,
saving: false,
};
describe("ScholarSettingsModal", () => {
it("renders scholar name and ID when scholar is provided", () => {
const wrapper = mount(ScholarSettingsModal, {
props: defaultProps,
global: { stubs: { AppModal: false, RouterLink: true, ScholarAvatar: true } },
});
expect(wrapper.text()).toContain("Dr. Test Scholar");
expect(wrapper.text()).toContain("abcDEF123456");
});
it("renders nothing when scholar is null", () => {
const wrapper = mount(ScholarSettingsModal, {
props: { ...defaultProps, scholar: null },
global: { stubs: { AppModal: false, RouterLink: true, ScholarAvatar: true } },
});
expect(wrapper.text()).not.toContain("Dr. Test Scholar");
});
it("emits close when close event fires", async () => {
const wrapper = mount(ScholarSettingsModal, {
props: defaultProps,
global: { stubs: { RouterLink: true, ScholarAvatar: true } },
});
const modal = wrapper.findComponent({ name: "AppModal" });
if (modal.exists()) {
modal.vm.$emit("close");
expect(wrapper.emitted("close")).toHaveLength(1);
}
});
it("emits delete when delete button is clicked", async () => {
const wrapper = mount(ScholarSettingsModal, {
props: defaultProps,
global: { stubs: { RouterLink: true, ScholarAvatar: true } },
});
const deleteButton = wrapper.findAll("button").find((b) => b.text().includes("Delete"));
if (deleteButton) {
await deleteButton.trigger("click");
expect(wrapper.emitted("delete")).toHaveLength(1);
}
});
it("emits toggle when enable/disable button is clicked", async () => {
const wrapper = mount(ScholarSettingsModal, {
props: defaultProps,
global: { stubs: { RouterLink: true, ScholarAvatar: true } },
});
const toggleButton = wrapper.findAll("button").find((b) => b.text().includes("Disable"));
if (toggleButton) {
await toggleButton.trigger("click");
expect(wrapper.emitted("toggle")).toHaveLength(1);
}
});
it("shows Enable button for disabled scholars", () => {
const wrapper = mount(ScholarSettingsModal, {
props: { ...defaultProps, scholar: buildScholar({ is_enabled: false }) },
global: { stubs: { RouterLink: true, ScholarAvatar: true } },
});
expect(wrapper.text()).toContain("Enable");
});
});

113
frontend/src/features/scholars/components/ScholarSettingsModal.vue Normal file
View file

@ -0,0 +1,113 @@
<script setup lang="ts">
import AppButton from "@/components/ui/AppButton.vue";
import AppInput from "@/components/ui/AppInput.vue";
import AppModal from "@/components/ui/AppModal.vue";
import ScholarAvatar from "@/features/scholars/components/ScholarAvatar.vue";
import type { ScholarProfile } from "@/features/scholars";
const props = defineProps<{
scholar: ScholarProfile | null;
imageUrlDraft: string;
imageBusy: boolean;
imageSaving: boolean;
imageUploading: boolean;
saving: boolean;
}>();
const emit = defineEmits<{
(e: "close"): void;
(e: "update:image-url-draft", value: string): void;
(e: "save-image-url"): void;
(e: "upload-image", event: Event): void;
(e: "reset-image"): void;
(e: "toggle"): void;
(e: "delete"): void;
}>();
function scholarLabel(profile: ScholarProfile): string {
return profile.display_name || profile.scholar_id;
}
function scholarProfileUrl(scholarId: string): string {
return `https://scholar.google.com/citations?hl=en&user=${encodeURIComponent(scholarId)}`;
}
function scholarPublicationsRoute(profile: ScholarProfile): { name: string; query: { scholar: string } } {
return { name: "publications", query: { scholar: String(profile.id) } };
}
</script>
<template>
<AppModal :open="scholar !== null" title="Scholar settings" @close="emit('close')">
<div v-if="scholar" class="grid gap-4">
<div class="flex items-start gap-3">
<ScholarAvatar
:label="scholar.display_name"
:scholar-id="scholar.scholar_id"
:image-url="scholar.profile_image_url"
/>
<div class="min-w-0 space-y-1">
<p class="truncate text-sm font-semibold text-ink-primary">{{ scholarLabel(scholar) }}</p>
<p class="text-xs text-secondary">ID: <code>{{ scholar.scholar_id }}</code></p>
<div class="flex flex-wrap items-center gap-3">
<RouterLink :to="scholarPublicationsRoute(scholar)" class="link-inline text-xs">
View publications
</RouterLink>
<a :href="scholarProfileUrl(scholar.scholar_id)" target="_blank" rel="noreferrer" class="link-inline text-xs">
Open Google Scholar profile
</a>
</div>
</div>
</div>
<div class="grid gap-2">
<label class="text-sm font-medium text-ink-secondary" :for="`scholar-image-url-${scholar.id}`">
Profile image URL override
</label>
<div class="flex flex-wrap items-center gap-2">
<div class="min-w-0 flex-1">
<AppInput
:id="`scholar-image-url-${scholar.id}`"
:model-value="imageUrlDraft"
placeholder="https://example.com/avatar.jpg"
:disabled="imageBusy"
@update:model-value="emit('update:image-url-draft', $event as string)"
/>
</div>
<AppButton variant="secondary" :disabled="imageBusy" @click="emit('save-image-url')">
{{ imageSaving ? "Saving..." : "Save URL" }}
</AppButton>
</div>
<div class="flex flex-wrap items-center gap-2">
<label
:for="`scholar-image-upload-${scholar.id}`"
class="inline-flex min-h-10 cursor-pointer items-center justify-center rounded-lg border border-stroke-strong bg-action-secondary-bg px-3 py-2 text-sm font-semibold text-action-secondary-text transition hover:bg-action-secondary-hover-bg focus-within:outline-none focus-within:ring-2 focus-within:ring-focus-ring focus-within:ring-offset-2 focus-within:ring-offset-focus-offset"
:class="{ 'pointer-events-none opacity-60': imageBusy }"
>
{{ imageUploading ? "Uploading..." : "Upload image" }}
</label>
<input
:id="`scholar-image-upload-${scholar.id}`"
type="file"
class="sr-only"
accept="image/jpeg,image/png,image/webp,image/gif"
:disabled="imageBusy"
@change="emit('upload-image', $event)"
/>
<AppButton variant="ghost" :disabled="imageBusy" @click="emit('reset-image')">
Reset image
</AppButton>
</div>
</div>
<div class="flex flex-wrap items-center justify-between gap-2 border-t border-stroke-default pt-3">
<AppButton variant="secondary" :disabled="imageBusy || saving" @click="emit('toggle')">
{{ scholar.is_enabled ? "Disable scholar" : "Enable scholar" }}
</AppButton>
<AppButton variant="danger" :disabled="imageBusy || saving" @click="emit('delete')">
Delete scholar
</AppButton>
</div>
</div>
</AppModal>
</template>

1097
frontend/src/features/settings/SettingsAdminPanel.vue
View file

File diff suppressed because it is too large Load diff

72
frontend/src/features/settings/components/AdminIntegritySection.test.ts Normal file
View file

@ -0,0 +1,72 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import AdminIntegritySection from "./AdminIntegritySection.vue";
vi.mock("@/features/admin_dbops", () => ({
getAdminDbIntegrityReport: vi.fn(),
}));
import { getAdminDbIntegrityReport, type AdminDbIntegrityReport } from "@/features/admin_dbops";
const mockedGetReport = vi.mocked(getAdminDbIntegrityReport);
function buildReport(overrides: Partial<AdminDbIntegrityReport> = {}): AdminDbIntegrityReport {
return {
status: "ok",
warnings: [],
failures: [],
checked_at: "2025-01-15T12:00:00Z",
checks: [
{ name: "orphaned_publications", count: 0, severity: "metric", message: "No orphaned publications found." },
],
...overrides,
};
}
describe("AdminIntegritySection", () => {
beforeEach(() => {
mockedGetReport.mockReset();
});
it("renders the section heading", () => {
const wrapper = mount(AdminIntegritySection);
expect(wrapper.text()).toContain("Integrity Report");
});
it("exposes load function that fetches integrity report", async () => {
mockedGetReport.mockResolvedValueOnce(buildReport());
const wrapper = mount(AdminIntegritySection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(mockedGetReport).toHaveBeenCalled();
});
it("renders check results after loading", async () => {
mockedGetReport.mockResolvedValueOnce(buildReport());
const wrapper = mount(AdminIntegritySection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("orphaned_publications");
});
it("displays warning count when present", async () => {
mockedGetReport.mockResolvedValueOnce(buildReport({ warnings: ["w1", "w2", "w3"] }));
const wrapper = mount(AdminIntegritySection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("3");
});
it("renders status badge", async () => {
mockedGetReport.mockResolvedValueOnce(buildReport({ status: "failed" }));
const wrapper = mount(AdminIntegritySection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("failed");
});
});

88
frontend/src/features/settings/components/AdminIntegritySection.vue Normal file
View file

@ -0,0 +1,88 @@
<script setup lang="ts">
import { ref } from "vue";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppRefreshButton from "@/components/ui/AppRefreshButton.vue";
import AppTable from "@/components/ui/AppTable.vue";
import {
getAdminDbIntegrityReport,
type AdminDbIntegrityCheck,
type AdminDbIntegrityReport,
} from "@/features/admin_dbops";
const refreshingIntegrity = ref(false);
const integrityReport = ref<AdminDbIntegrityReport | null>(null);
function formatTimestamp(value: string | null): string {
if (!value) return "n/a";
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? value : parsed.toLocaleString();
}
function statusTone(status: string): "success" | "warning" | "danger" | "info" | "neutral" {
if (status === "ok" || status === "completed" || status === "resolved") return "success";
if (status === "warning" || status === "running" || status === "queued") return "warning";
if (status === "failed") return "danger";
return "info";
}
function checkTone(check: AdminDbIntegrityCheck): "warning" | "danger" | "neutral" | "info" {
if (check.severity === "metric") return "info";
if (check.count <= 0) return "neutral";
return check.severity === "failure" ? "danger" : "warning";
}
async function refreshIntegrity(): Promise<void> {
refreshingIntegrity.value = true;
try {
integrityReport.value = await getAdminDbIntegrityReport();
} finally {
refreshingIntegrity.value = false;
}
}
async function load(): Promise<void> {
await refreshIntegrity();
}
defineExpose({ load });
</script>
<template>
<AppCard class="space-y-3">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Integrity Report</h2>
<AppHelpHint text="Read-only checks for known corruption patterns and data drift." />
</div>
<AppRefreshButton variant="secondary" :loading="refreshingIntegrity" title="Refresh integrity report" loading-title="Refreshing integrity report" @click="refreshIntegrity" />
</div>
<div v-if="integrityReport" class="flex flex-wrap items-center gap-2 text-xs">
<AppBadge :tone="statusTone(integrityReport.status)">Status: {{ integrityReport.status }}</AppBadge>
<AppBadge tone="warning">Warnings: {{ integrityReport.warnings.length }}</AppBadge>
<AppBadge tone="danger">Failures: {{ integrityReport.failures.length }}</AppBadge>
<span class="text-secondary">Checked: {{ formatTimestamp(integrityReport.checked_at) }}</span>
</div>
<AppTable v-if="integrityReport" label="Integrity checks">
<thead>
<tr>
<th scope="col">Check</th>
<th scope="col">Count</th>
<th scope="col">Severity</th>
<th scope="col">Message</th>
</tr>
</thead>
<tbody>
<tr v-for="check in integrityReport.checks" :key="check.name">
<td>{{ check.name }}</td>
<td>{{ check.count }}</td>
<td><AppBadge :tone="checkTone(check)">{{ check.severity }}</AppBadge></td>
<td>{{ check.message }}</td>
</tr>
</tbody>
</AppTable>
</AppCard>
</template>

104
frontend/src/features/settings/components/AdminPdfQueueSection.test.ts Normal file
View file

@ -0,0 +1,104 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import AdminPdfQueueSection from "./AdminPdfQueueSection.vue";
vi.mock("@/features/admin_dbops", () => ({
listAdminPdfQueue: vi.fn(),
requeueAdminPdfLookup: vi.fn(),
requeueAllAdminPdfLookups: vi.fn(),
}));
import { listAdminPdfQueue } from "@/features/admin_dbops";
const mockedListQueue = vi.mocked(listAdminPdfQueue);
function buildQueueItem(overrides: Record<string, unknown> = {}) {
return {
publication_id: 1,
title: "Test Paper",
display_identifier: null,
pdf_url: null,
status: "queued",
attempt_count: 0,
last_failure_reason: null,
last_failure_detail: null,
last_source: "unpaywall",
requested_by_user_id: null,
requested_by_email: null,
queued_at: "2025-01-15T12:00:00Z",
last_attempt_at: null,
resolved_at: null,
updated_at: "2025-01-15T12:00:00Z",
...overrides,
};
}
describe("AdminPdfQueueSection", () => {
beforeEach(() => {
mockedListQueue.mockReset();
});
it("renders the PDF queue heading", () => {
const wrapper = mount(AdminPdfQueueSection);
expect(wrapper.text()).toContain("PDF");
});
it("exposes load function that fetches PDF queue", async () => {
mockedListQueue.mockResolvedValueOnce({
items: [buildQueueItem()],
total_count: 1,
has_next: false,
has_prev: false,
page: 1,
page_size: 50,
});
const wrapper = mount(AdminPdfQueueSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(mockedListQueue).toHaveBeenCalled();
});
it("renders queue items after loading", async () => {
mockedListQueue.mockResolvedValueOnce({
items: [buildQueueItem({ title: "My Paper" })],
total_count: 1,
has_next: false,
has_prev: false,
page: 1,
page_size: 50,
});
const wrapper = mount(AdminPdfQueueSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("My Paper");
});
it("renders status filter dropdown", () => {
const wrapper = mount(AdminPdfQueueSection);
expect(wrapper.text()).toContain("Status");
});
it("renders queue all button", () => {
const wrapper = mount(AdminPdfQueueSection);
expect(wrapper.text()).toContain("Queue all");
});
it("shows pagination summary after loading", async () => {
mockedListQueue.mockResolvedValueOnce({
items: Array.from({ length: 3 }, (_, i) => buildQueueItem({ publication_id: i + 1 })),
total_count: 3,
has_next: false,
has_prev: false,
page: 1,
page_size: 50,
});
const wrapper = mount(AdminPdfQueueSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("3");
});
});

203
frontend/src/features/settings/components/AdminPdfQueueSection.vue Normal file
View file

@ -0,0 +1,203 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppRefreshButton from "@/components/ui/AppRefreshButton.vue";
import AppSelect from "@/components/ui/AppSelect.vue";
import AppTable from "@/components/ui/AppTable.vue";
import {
listAdminPdfQueue,
requeueAdminPdfLookup,
requeueAllAdminPdfLookups,
type AdminPdfQueueItem,
} from "@/features/admin_dbops";
import { useRequestState } from "@/composables/useRequestState";
const { clearAlerts, assignError, setSuccess } = useRequestState();
const refreshingPdfQueue = ref(false);
const requeueingPublicationId = ref<number | null>(null);
const requeueingAllPdfs = ref(false);
const pdfQueueItems = ref<AdminPdfQueueItem[]>([]);
const pdfQueueStatusFilter = ref("");
const pdfQueuePage = ref(1);
const pdfQueuePageSize = ref("50");
const pdfQueueTotalCount = ref(0);
const pdfQueueHasNext = ref(false);
const pdfQueueHasPrev = ref(false);
const pdfQueuePageSizeValue = computed(() => {
const parsed = Number(pdfQueuePageSize.value);
if (!Number.isFinite(parsed)) return 50;
return Math.max(1, Math.min(500, Math.trunc(parsed)));
});
const pdfQueueTotalPages = computed(() =>
Math.max(1, Math.ceil(pdfQueueTotalCount.value / Math.max(pdfQueuePageSizeValue.value, 1))),
);
const pdfQueueSummary = computed(() =>
`${pdfQueueTotalCount.value} item${pdfQueueTotalCount.value === 1 ? "" : "s"} total`,
);
function statusTone(status: string): "success" | "warning" | "danger" | "info" | "neutral" {
if (status === "ok" || status === "completed" || status === "resolved") return "success";
if (status === "warning" || status === "running" || status === "queued") return "warning";
if (status === "failed") return "danger";
return "info";
}
function formatTimestamp(value: string | null): string {
if (!value) return "n/a";
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? value : parsed.toLocaleString();
}
function canRequeuePdf(item: AdminPdfQueueItem): boolean {
return item.status !== "queued" && item.status !== "running";
}
async function refreshPdfQueue(): Promise<void> {
refreshingPdfQueue.value = true;
try {
const status = pdfQueueStatusFilter.value.trim() || null;
const page = await listAdminPdfQueue(pdfQueuePage.value, pdfQueuePageSizeValue.value, status);
pdfQueueItems.value = page.items;
pdfQueueTotalCount.value = page.total_count;
pdfQueueHasNext.value = page.has_next;
pdfQueueHasPrev.value = page.has_prev;
pdfQueuePage.value = page.page;
pdfQueuePageSize.value = String(page.page_size);
} finally {
refreshingPdfQueue.value = false;
}
}
async function onPdfQueueFilterChanged(): Promise<void> {
pdfQueuePage.value = 1;
await refreshPdfQueue();
}
async function onPdfQueuePageSizeChanged(): Promise<void> {
pdfQueuePage.value = 1;
await refreshPdfQueue();
}
async function onPdfQueuePrevPage(): Promise<void> {
if (!pdfQueueHasPrev.value || pdfQueuePage.value <= 1) return;
pdfQueuePage.value = Math.max(pdfQueuePage.value - 1, 1);
await refreshPdfQueue();
}
async function onPdfQueueNextPage(): Promise<void> {
if (!pdfQueueHasNext.value) return;
pdfQueuePage.value += 1;
await refreshPdfQueue();
}
async function onRequeuePdf(item: AdminPdfQueueItem): Promise<void> {
requeueingPublicationId.value = item.publication_id;
clearAlerts();
try {
const result = await requeueAdminPdfLookup(item.publication_id);
setSuccess(result.message);
await refreshPdfQueue();
} catch (error) {
assignError(error, "Unable to requeue PDF lookup.");
} finally {
requeueingPublicationId.value = null;
}
}
async function onRequeueAllPdfs(): Promise<void> {
requeueingAllPdfs.value = true;
clearAlerts();
try {
const result = await requeueAllAdminPdfLookups(5000);
setSuccess(result.message);
await refreshPdfQueue();
} catch (error) {
assignError(error, "Unable to queue missing PDF lookups.");
} finally {
requeueingAllPdfs.value = false;
}
}
async function load(): Promise<void> {
await refreshPdfQueue();
}
defineExpose({ load });
</script>
<template>
<AppCard class="space-y-3">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">PDF Gathering Queue</h2>
<AppHelpHint text="Live queue and outcome history for PDF acquisition across all publications." />
</div>
<div class="flex items-center gap-2">
<AppSelect v-model="pdfQueueStatusFilter" class="min-w-[12rem] !py-1.5 !text-sm" @change="onPdfQueueFilterChanged">
<option value="">All statuses</option>
<option value="untracked">untracked</option>
<option value="queued">queued</option>
<option value="running">running</option>
<option value="failed">failed</option>
<option value="resolved">resolved</option>
</AppSelect>
<AppSelect v-model="pdfQueuePageSize" class="min-w-[10rem] !py-1.5 !text-sm" @change="onPdfQueuePageSizeChanged">
<option value="25">25 / page</option>
<option value="50">50 / page</option>
<option value="100">100 / page</option>
</AppSelect>
<AppButton variant="secondary" class="!min-h-8 whitespace-nowrap !px-2.5 !py-1 !text-xs" :disabled="requeueingAllPdfs" title="Queue all missing PDFs" @click="onRequeueAllPdfs">
{{ requeueingAllPdfs ? "Queueing..." : "Queue all" }}
</AppButton>
<AppRefreshButton variant="secondary" size="sm" :loading="refreshingPdfQueue" title="Refresh PDF queue" loading-title="Refreshing PDF queue" @click="refreshPdfQueue" />
</div>
</div>
<AppTable label="PDF queue table">
<thead>
<tr>
<th scope="col">Publication</th><th scope="col">Status</th><th scope="col">Attempts</th>
<th scope="col">Failure reason</th><th scope="col">Source</th><th scope="col">Requested by</th>
<th scope="col">Queued</th><th scope="col">Last attempt</th><th scope="col">Resolved</th><th scope="col">Actions</th>
</tr>
</thead>
<tbody>
<tr v-for="item in pdfQueueItems" :key="item.publication_id">
<td>
<div class="grid gap-1">
<span class="font-medium text-ink-primary">{{ item.title }}</span>
<a v-if="item.display_identifier?.url" :href="item.display_identifier.url" target="_blank" rel="noreferrer" class="link-inline text-xs">{{ item.display_identifier.label }}</a>
</div>
</td>
<td><AppBadge :tone="statusTone(item.status)">{{ item.status }}</AppBadge></td>
<td>{{ item.attempt_count }}</td>
<td>{{ item.last_failure_reason || "n/a" }}</td>
<td>{{ item.last_source || "n/a" }}</td>
<td>{{ item.requested_by_email || "n/a" }}</td>
<td>{{ formatTimestamp(item.queued_at) }}</td>
<td>{{ formatTimestamp(item.last_attempt_at) }}</td>
<td>{{ formatTimestamp(item.resolved_at) }}</td>
<td>
<AppButton variant="ghost" :disabled="requeueingPublicationId === item.publication_id || !canRequeuePdf(item)" @click="onRequeuePdf(item)">
{{ requeueingPublicationId === item.publication_id ? "Requeueing..." : "Requeue" }}
</AppButton>
</td>
</tr>
</tbody>
</AppTable>
<div class="flex flex-wrap items-center justify-between gap-2 text-xs text-secondary">
<span>{{ pdfQueueSummary }}</span>
<div class="flex items-center gap-2">
<span>Page {{ pdfQueuePage }} / {{ pdfQueueTotalPages }}</span>
<AppButton variant="ghost" :disabled="!pdfQueueHasPrev" @click="onPdfQueuePrevPage">Prev</AppButton>
<AppButton variant="ghost" :disabled="!pdfQueueHasNext" @click="onPdfQueueNextPage">Next</AppButton>
</div>
</div>
</AppCard>
</template>

91
frontend/src/features/settings/components/AdminRepairsSection.test.ts Normal file
View file

@ -0,0 +1,91 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import AdminRepairsSection from "./AdminRepairsSection.vue";
vi.mock("@/features/admin_dbops", () => ({
listAdminDbRepairJobs: vi.fn(),
triggerPublicationLinkRepair: vi.fn(),
triggerPublicationNearDuplicateRepair: vi.fn(),
dropAllPublications: vi.fn(),
}));
import { listAdminDbRepairJobs, dropAllPublications } from "@/features/admin_dbops";
const mockedListJobs = vi.mocked(listAdminDbRepairJobs);
const mockedDrop = vi.mocked(dropAllPublications);
function buildUser(overrides: Record<string, unknown> = {}) {
return {
id: 1,
email: "admin@test.com",
is_admin: true,
is_active: true,
created_at: "2025-01-01T00:00:00Z",
updated_at: "2025-01-01T00:00:00Z",
...overrides,
};
}
describe("AdminRepairsSection", () => {
beforeEach(() => {
mockedListJobs.mockReset();
mockedDrop.mockReset();
});
it("renders the repair sections", () => {
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
expect(wrapper.text()).toContain("Publication Link Repair");
expect(wrapper.text()).toContain("Near-Duplicate");
expect(wrapper.text()).toContain("Drop All Publications");
});
it("exposes load function that fetches repair jobs", async () => {
mockedListJobs.mockResolvedValueOnce([]);
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(mockedListJobs).toHaveBeenCalled();
});
it("renders repair job rows after loading", async () => {
mockedListJobs.mockResolvedValueOnce([
{
id: 1,
job_name: "publication_link_repair",
status: "completed",
dry_run: true,
requested_by: "admin@test.com",
scope: {},
summary: { links_in_scope: 100, links_deleted: 5 },
error_text: null,
started_at: "2025-01-15T12:00:00Z",
finished_at: "2025-01-15T12:01:00Z",
created_at: "2025-01-15T12:00:00Z",
updated_at: "2025-01-15T12:01:00Z",
},
]);
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("publication_link_repair");
});
it("has typed confirmation guard for drop all publications", () => {
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
const dropButton = wrapper.findAll("button").find((b) => b.text().includes("Drop all publications"));
expect(dropButton?.attributes("disabled")).toBeDefined();
});
it("renders dry-run toggle in repair form", () => {
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
expect(wrapper.text()).toContain("Dry-run");
});
it("renders scope mode selector", () => {
const wrapper = mount(AdminRepairsSection, { props: { users: [buildUser()] } });
expect(wrapper.text()).toContain("Scope");
});
});

372
frontend/src/features/settings/components/AdminRepairsSection.vue Normal file
View file

@ -0,0 +1,372 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import AppBadge from "@/components/ui/AppBadge.vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppCheckbox from "@/components/ui/AppCheckbox.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
import AppRefreshButton from "@/components/ui/AppRefreshButton.vue";
import AppSelect from "@/components/ui/AppSelect.vue";
import AppTable from "@/components/ui/AppTable.vue";
import {
dropAllPublications,
listAdminDbRepairJobs,
triggerPublicationLinkRepair,
triggerPublicationNearDuplicateRepair,
type AdminDbRepairJob,
type NearDuplicateCluster,
type TriggerPublicationLinkRepairResult,
type TriggerPublicationNearDuplicateRepairResult,
} from "@/features/admin_dbops";
import type { AdminUser } from "@/features/admin_users";
import { useRequestState } from "@/composables/useRequestState";
const props = defineProps<{
users: AdminUser[];
}>();
const SCOPE_SINGLE_USER = "single_user";
const SCOPE_ALL_USERS = "all_users";
const APPLY_ALL_USERS_CONFIRM_TEXT = "REPAIR ALL USERS";
const APPLY_NEAR_DUPLICATES_CONFIRM_TEXT = "MERGE SELECTED DUPLICATES";
const DROP_PUBLICATIONS_CONFIRM_TEXT = "DROP ALL PUBLICATIONS";
type RepairScopeMode = typeof SCOPE_SINGLE_USER | typeof SCOPE_ALL_USERS;
const { clearAlerts, assignError, setSuccess } = useRequestState();
const refreshingJobs = ref(false);
const runningRepair = ref(false);
const repairJobs = ref<AdminDbRepairJob[]>([]);
const lastRepairResult = ref<TriggerPublicationLinkRepairResult | null>(null);
const repairScopeMode = ref<RepairScopeMode>(SCOPE_SINGLE_USER);
const repairUserId = ref("");
const repairScholarIds = ref("");
const repairRequestedBy = ref("");
const repairDryRun = ref(true);
const repairGcOrphans = ref(false);
const repairConfirmationText = ref("");
const runningNearDuplicateScan = ref(false);
const applyingNearDuplicateRepair = ref(false);
const nearDuplicateRequestedBy = ref("");
const nearDuplicateSimilarityThreshold = ref("0.78");
const nearDuplicateMinSharedTokens = ref("3");
const nearDuplicateMaxYearDelta = ref("1");
const nearDuplicateMaxClusters = ref("25");
const nearDuplicateConfirmationText = ref("");
const nearDuplicateSelectedClusterKeys = ref<Set<string>>(new Set());
const nearDuplicateClusters = ref<NearDuplicateCluster[]>([]);
const lastNearDuplicateResult = ref<TriggerPublicationNearDuplicateRepairResult | null>(null);
const droppingPublications = ref(false);
const dropConfirmationText = ref("");
const dropResult = ref<{ deleted_count: number; message: string } | null>(null);
const dropConfirmationValid = computed(() => dropConfirmationText.value.trim() === DROP_PUBLICATIONS_CONFIRM_TEXT);
const typedConfirmationRequired = computed(
() => repairScopeMode.value === SCOPE_ALL_USERS && !repairDryRun.value,
);
const nearDuplicateApplyEnabled = computed(() => nearDuplicateSelectedClusterKeys.value.size > 0);
function statusTone(status: string): "success" | "warning" | "danger" | "info" | "neutral" {
if (status === "ok" || status === "completed" || status === "resolved") return "success";
if (status === "warning" || status === "running" || status === "queued") return "warning";
if (status === "failed") return "danger";
return "info";
}
function formatTimestamp(value: string | null): string {
if (!value) return "n/a";
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? value : parsed.toLocaleString();
}
function parseScholarIds(raw: string): number[] {
const tokens = raw.split(/[,\s]+/).map((t) => t.trim()).filter((t) => t.length > 0);
const deduped = new Set<number>();
for (const token of tokens) {
const parsed = Number(token);
if (!Number.isInteger(parsed) || parsed < 1) throw new Error("Scholar profile IDs must be positive integers.");
deduped.add(parsed);
}
return [...deduped];
}
function parseRepairUserIdOrThrow(raw: string): number {
const parsed = Number((raw || "").trim());
if (!Number.isInteger(parsed) || parsed < 1) throw new Error("Select a valid target user.");
return parsed;
}
function validateTypedConfirmation(): string {
const normalized = repairConfirmationText.value.trim();
if (typedConfirmationRequired.value && normalized !== APPLY_ALL_USERS_CONFIRM_TEXT) {
throw new Error(`Type '${APPLY_ALL_USERS_CONFIRM_TEXT}' to apply repair for all users.`);
}
return normalized;
}
function parseBoundedNumber(raw: string, opts: { minimum: number; maximum: number; fallback: number }): number {
const parsed = Number(raw.trim());
if (!Number.isFinite(parsed)) return opts.fallback;
return Math.max(opts.minimum, Math.min(opts.maximum, parsed));
}
function nearDuplicatePayloadBase() {
return {
similarity_threshold: parseBoundedNumber(nearDuplicateSimilarityThreshold.value, { minimum: 0.5, maximum: 1.0, fallback: 0.78 }),
min_shared_tokens: Math.trunc(parseBoundedNumber(nearDuplicateMinSharedTokens.value, { minimum: 1, maximum: 8, fallback: 3 })),
max_year_delta: Math.trunc(parseBoundedNumber(nearDuplicateMaxYearDelta.value, { minimum: 0, maximum: 5, fallback: 1 })),
max_clusters: Math.trunc(parseBoundedNumber(nearDuplicateMaxClusters.value, { minimum: 1, maximum: 200, fallback: 25 })),
requested_by: nearDuplicateRequestedBy.value.trim() || undefined,
};
}
function checkboxEventChecked(event: Event): boolean {
return event.target instanceof HTMLInputElement ? event.target.checked : false;
}
function summaryCount(job: AdminDbRepairJob, key: string): string {
const value = job.summary[key];
return typeof value === "number" ? String(value) : "n/a";
}
function ensureRepairUserSelected(): void {
if (repairScopeMode.value !== SCOPE_SINGLE_USER || repairUserId.value || props.users.length === 0) return;
repairUserId.value = String(props.users[0].id);
}
async function refreshRepairJobs(): Promise<void> {
refreshingJobs.value = true;
try { repairJobs.value = await listAdminDbRepairJobs(30); } finally { refreshingJobs.value = false; }
}
async function onRunRepair(): Promise<void> {
runningRepair.value = true;
clearAlerts();
try {
const payload = {
scope_mode: repairScopeMode.value,
scholar_profile_ids: parseScholarIds(repairScholarIds.value),
dry_run: repairDryRun.value,
gc_orphan_publications: repairGcOrphans.value,
requested_by: repairRequestedBy.value.trim() || undefined,
confirmation_text: validateTypedConfirmation() || undefined,
user_id: repairScopeMode.value === SCOPE_SINGLE_USER ? parseRepairUserIdOrThrow(repairUserId.value) : undefined,
};
const result = await triggerPublicationLinkRepair(payload);
lastRepairResult.value = result;
setSuccess(`Repair job #${result.job_id} completed (${result.status}).`);
repairConfirmationText.value = "";
await refreshRepairJobs();
} catch (error) {
assignError(error, "Unable to run publication link repair.");
} finally {
runningRepair.value = false;
}
}
function onToggleNearDuplicateClusterSelection(clusterKey: string, checked: boolean): void {
const next = new Set(nearDuplicateSelectedClusterKeys.value);
if (checked) { next.add(clusterKey); } else { next.delete(clusterKey); }
nearDuplicateSelectedClusterKeys.value = next;
}
async function onRunNearDuplicateScan(): Promise<void> {
runningNearDuplicateScan.value = true;
clearAlerts();
try {
const result = await triggerPublicationNearDuplicateRepair({ dry_run: true, ...nearDuplicatePayloadBase() });
nearDuplicateClusters.value = result.clusters;
nearDuplicateSelectedClusterKeys.value = new Set();
nearDuplicateConfirmationText.value = "";
lastNearDuplicateResult.value = result;
setSuccess(`Near-duplicate scan completed (job #${result.job_id}).`);
await refreshRepairJobs();
} catch (error) {
assignError(error, "Unable to scan for near-duplicate publications.");
} finally {
runningNearDuplicateScan.value = false;
}
}
async function onApplyNearDuplicateRepair(): Promise<void> {
applyingNearDuplicateRepair.value = true;
clearAlerts();
try {
const selectedKeys = [...nearDuplicateSelectedClusterKeys.value].sort((a, b) => a.localeCompare(b));
if (selectedKeys.length === 0) throw new Error("Select at least one near-duplicate cluster before applying.");
if (nearDuplicateConfirmationText.value.trim() !== APPLY_NEAR_DUPLICATES_CONFIRM_TEXT) {
throw new Error(`Type '${APPLY_NEAR_DUPLICATES_CONFIRM_TEXT}' to confirm merge.`);
}
const result = await triggerPublicationNearDuplicateRepair({
dry_run: false,
selected_cluster_keys: selectedKeys,
confirmation_text: nearDuplicateConfirmationText.value.trim(),
...nearDuplicatePayloadBase(),
});
nearDuplicateClusters.value = result.clusters;
nearDuplicateSelectedClusterKeys.value = new Set();
nearDuplicateConfirmationText.value = "";
lastNearDuplicateResult.value = result;
setSuccess(`Merged selected near-duplicate clusters (job #${result.job_id}).`);
await refreshRepairJobs();
} catch (error) {
assignError(error, "Unable to apply near-duplicate merge.");
} finally {
applyingNearDuplicateRepair.value = false;
}
}
async function onDropAllPublications(): Promise<void> {
droppingPublications.value = true;
clearAlerts();
dropResult.value = null;
try {
const result = await dropAllPublications(dropConfirmationText.value.trim());
dropResult.value = result;
setSuccess(result.message);
dropConfirmationText.value = "";
} catch (error) {
assignError(error, "Unable to drop publications.");
} finally {
droppingPublications.value = false;
}
}
async function load(): Promise<void> {
await refreshRepairJobs();
ensureRepairUserSelected();
}
defineExpose({ load });
</script>
<template>
<section class="grid gap-4">
<AppCard class="space-y-3">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Publication Link Repair</h2>
<AppHelpHint text="Dry-run first. For all-users apply mode, typed confirmation is required." />
</div>
<form class="grid gap-3 md:grid-cols-2" @submit.prevent="onRunRepair">
<label class="grid gap-1 text-sm font-medium text-ink-secondary">
<span>Scope</span>
<AppSelect v-model="repairScopeMode" @change="ensureRepairUserSelected">
<option :value="SCOPE_SINGLE_USER">Single user</option>
<option :value="SCOPE_ALL_USERS">All users</option>
</AppSelect>
</label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary">
<span>Target user</span>
<AppSelect v-model="repairUserId" :disabled="repairScopeMode === SCOPE_ALL_USERS || users.length === 0">
<option value="" disabled>Select user</option>
<option v-for="user in users" :key="user.id" :value="String(user.id)">{{ user.email }} (ID {{ user.id }})</option>
</AppSelect>
</label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary md:col-span-2">
<span>Scholar profile IDs (optional)</span>
<AppInput v-model="repairScholarIds" placeholder="e.g. 12,13,14" />
</label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary md:col-span-2">
<span>Requested by (optional)</span>
<AppInput v-model="repairRequestedBy" placeholder="email/name/ticket id" />
</label>
<div class="flex flex-wrap items-center gap-4 md:col-span-2">
<AppCheckbox id="repair-dry-run" v-model="repairDryRun" label="Dry-run (no writes)" />
<AppCheckbox id="repair-gc-orphans" v-model="repairGcOrphans" label="Delete orphan publications" />
</div>
<label v-if="typedConfirmationRequired" class="grid gap-1 text-sm font-medium text-ink-secondary md:col-span-2">
<span>Type '{{ APPLY_ALL_USERS_CONFIRM_TEXT }}' to confirm</span>
<AppInput v-model="repairConfirmationText" :placeholder="APPLY_ALL_USERS_CONFIRM_TEXT" />
</label>
<p v-if="repairScopeMode === SCOPE_ALL_USERS" class="text-xs text-secondary md:col-span-2">All-users scope includes every scholar profile across all accounts.</p>
<div class="md:col-span-2">
<AppButton type="submit" :disabled="runningRepair">{{ runningRepair ? "Running..." : "Run repair job" }}</AppButton>
</div>
</form>
<div v-if="lastRepairResult" class="rounded-lg border border-stroke-default bg-surface-card-muted p-3 text-xs">
<div class="mb-2 flex flex-wrap items-center gap-2">
<AppBadge :tone="statusTone(lastRepairResult.status)">Job #{{ lastRepairResult.job_id }}</AppBadge>
<span class="text-secondary">Status: {{ lastRepairResult.status }}</span>
</div>
<pre class="overflow-x-auto text-secondary">{{ JSON.stringify(lastRepairResult.summary, null, 2) }}</pre>
</div>
</AppCard>
<AppCard class="space-y-3">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Near-Duplicate Publication Repair</h2>
<AppHelpHint text="Run a dry-run scan first, verify candidate clusters, then merge only selected clusters." />
</div>
<form class="grid gap-3 md:grid-cols-2" @submit.prevent="onRunNearDuplicateScan">
<label class="grid gap-1 text-sm font-medium text-ink-secondary"><span>Similarity threshold</span><AppInput v-model="nearDuplicateSimilarityThreshold" placeholder="0.78" /></label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary"><span>Min shared tokens</span><AppInput v-model="nearDuplicateMinSharedTokens" placeholder="3" /></label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary"><span>Max year delta</span><AppInput v-model="nearDuplicateMaxYearDelta" placeholder="1" /></label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary"><span>Max preview clusters</span><AppInput v-model="nearDuplicateMaxClusters" placeholder="25" /></label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary md:col-span-2"><span>Requested by (optional)</span><AppInput v-model="nearDuplicateRequestedBy" placeholder="email/name/ticket id" /></label>
<div class="md:col-span-2">
<AppButton type="submit" :disabled="runningNearDuplicateScan">{{ runningNearDuplicateScan ? "Scanning..." : "Scan near-duplicate clusters" }}</AppButton>
</div>
</form>
<div v-if="nearDuplicateClusters.length > 0" class="space-y-3">
<AppTable label="Near duplicate clusters table">
<thead><tr><th scope="col">Select</th><th scope="col">Cluster</th><th scope="col">Winner</th><th scope="col">Members</th><th scope="col">Similarity</th></tr></thead>
<tbody>
<tr v-for="cluster in nearDuplicateClusters" :key="cluster.cluster_key">
<td><input :id="`near-dup-${cluster.cluster_key}`" class="h-4 w-4 rounded border-stroke-default bg-surface-card" type="checkbox" :checked="nearDuplicateSelectedClusterKeys.has(cluster.cluster_key)" @change="onToggleNearDuplicateClusterSelection(cluster.cluster_key, checkboxEventChecked($event))" /></td>
<td class="font-mono text-xs">{{ cluster.cluster_key }}</td>
<td>#{{ cluster.winner_publication_id }}</td>
<td><div class="grid gap-1"><span v-for="member in cluster.members" :key="member.publication_id" class="text-xs text-secondary">#{{ member.publication_id }} · {{ member.title }}</span></div></td>
<td>{{ cluster.similarity_score.toFixed(2) }}</td>
</tr>
</tbody>
</AppTable>
<form class="grid gap-3 md:grid-cols-2" @submit.prevent="onApplyNearDuplicateRepair">
<label class="grid gap-1 text-sm font-medium text-ink-secondary md:col-span-2"><span>Type '{{ APPLY_NEAR_DUPLICATES_CONFIRM_TEXT }}' to merge selected clusters</span><AppInput v-model="nearDuplicateConfirmationText" :placeholder="APPLY_NEAR_DUPLICATES_CONFIRM_TEXT" /></label>
<div class="md:col-span-2"><AppButton type="submit" :disabled="applyingNearDuplicateRepair || !nearDuplicateApplyEnabled">{{ applyingNearDuplicateRepair ? "Merging..." : "Merge selected clusters" }}</AppButton></div>
</form>
</div>
<div v-if="lastNearDuplicateResult" class="rounded-lg border border-stroke-default bg-surface-card-muted p-3 text-xs">
<div class="mb-2 flex flex-wrap items-center gap-2"><AppBadge :tone="statusTone(lastNearDuplicateResult.status)">Job #{{ lastNearDuplicateResult.job_id }}</AppBadge><span class="text-secondary">Status: {{ lastNearDuplicateResult.status }}</span></div>
<pre class="overflow-x-auto text-secondary">{{ JSON.stringify(lastNearDuplicateResult.summary, null, 2) }}</pre>
</div>
</AppCard>
<AppCard class="space-y-3">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="flex items-center gap-1"><h2 class="text-lg font-semibold text-ink-primary">Recent Repair Jobs</h2><AppHelpHint text="Audit history and summary counters for each repair job." /></div>
<AppRefreshButton variant="secondary" :loading="refreshingJobs" title="Refresh repair jobs" loading-title="Refreshing repair jobs" @click="refreshRepairJobs" />
</div>
<AppTable label="Repair jobs table">
<thead><tr><th scope="col">Job</th><th scope="col">Status</th><th scope="col">Mode</th><th scope="col">Requested by</th><th scope="col">Created</th><th scope="col">Links in scope</th><th scope="col">Links deleted</th></tr></thead>
<tbody>
<tr v-for="job in repairJobs" :key="job.id">
<td>#{{ job.id }} · {{ job.job_name }}</td>
<td><AppBadge :tone="statusTone(job.status)">{{ job.status }}</AppBadge></td>
<td>{{ job.dry_run ? "dry-run" : "apply" }}</td>
<td>{{ job.requested_by || "n/a" }}</td>
<td>{{ formatTimestamp(job.created_at) }}</td>
<td>{{ summaryCount(job, "links_in_scope") }}</td>
<td>{{ summaryCount(job, "links_deleted") }}</td>
</tr>
</tbody>
</AppTable>
</AppCard>
<AppCard class="space-y-3 border-state-danger-border">
<div class="flex items-center gap-1"><h2 class="text-lg font-semibold text-state-danger-text">Drop All Publications</h2><AppHelpHint text="Permanently delete ALL publications, links, identifiers, and PDF jobs. Scholar baselines will be reset so the next run re-discovers everything." /></div>
<p class="text-sm text-secondary">This action is <strong>irreversible</strong>. It deletes every publication record across all users. The next ingestion run will re-populate all data from scratch.</p>
<form class="grid gap-3" @submit.prevent="onDropAllPublications">
<label class="grid gap-1 text-sm font-medium text-ink-secondary"><span>Type '{{ DROP_PUBLICATIONS_CONFIRM_TEXT }}' to confirm</span><AppInput v-model="dropConfirmationText" :placeholder="DROP_PUBLICATIONS_CONFIRM_TEXT" autocomplete="off" /></label>
<div><AppButton type="submit" variant="danger" :disabled="droppingPublications || !dropConfirmationValid">{{ droppingPublications ? "Dropping..." : "Drop all publications" }}</AppButton></div>
</form>
<div v-if="dropResult" class="rounded-lg border border-stroke-default bg-surface-card-muted p-3 text-xs">
<div class="mb-1 flex items-center gap-2"><AppBadge tone="danger">Deleted: {{ dropResult.deleted_count }}</AppBadge></div>
<p class="text-secondary">{{ dropResult.message }}</p>
</div>
</AppCard>
</section>
</template>

84
frontend/src/features/settings/components/AdminUsersSection.test.ts Normal file
View file

@ -0,0 +1,84 @@
// @vitest-environment happy-dom
import { describe, expect, it, vi, beforeEach } from "vitest";
import { mount, flushPromises } from "@vue/test-utils";
import AdminUsersSection from "./AdminUsersSection.vue";
vi.mock("@/features/admin_users", () => ({
listAdminUsers: vi.fn(),
createAdminUser: vi.fn(),
setAdminUserActive: vi.fn(),
resetAdminUserPassword: vi.fn(),
}));
import { listAdminUsers, createAdminUser } from "@/features/admin_users";
const mockedListUsers = vi.mocked(listAdminUsers);
const mockedCreateUser = vi.mocked(createAdminUser);
function buildUser(overrides: Record<string, unknown> = {}) {
return {
id: 1,
email: "user@example.com",
is_admin: false,
is_active: true,
created_at: "2025-01-01T00:00:00Z",
updated_at: "2025-01-01T00:00:00Z",
...overrides,
};
}
describe("AdminUsersSection", () => {
beforeEach(() => {
mockedListUsers.mockReset();
mockedCreateUser.mockReset();
});
it("renders the create user form", () => {
const wrapper = mount(AdminUsersSection);
expect(wrapper.text()).toContain("Create user");
});
it("exposes load function that calls listAdminUsers", async () => {
mockedListUsers.mockResolvedValueOnce([buildUser()]);
const wrapper = mount(AdminUsersSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void>; users: unknown[] };
await vm.load();
await flushPromises();
expect(mockedListUsers).toHaveBeenCalled();
expect(vm.users).toHaveLength(1);
});
it("renders user table after loading", async () => {
mockedListUsers.mockResolvedValueOnce([buildUser({ email: "alice@test.com" })]);
const wrapper = mount(AdminUsersSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await vm.load();
await flushPromises();
expect(wrapper.text()).toContain("alice@test.com");
});
it("propagates error when load fails", async () => {
mockedListUsers.mockRejectedValueOnce(new Error("Network error"));
const wrapper = mount(AdminUsersSection);
const vm = wrapper.vm as unknown as { load: () => Promise<void> };
await expect(vm.load()).rejects.toThrow("Network error");
});
it("emits user creation via form submit", async () => {
mockedCreateUser.mockResolvedValueOnce(buildUser({ email: "new@test.com" }));
mockedListUsers.mockResolvedValue([]);
const wrapper = mount(AdminUsersSection);
const inputs = wrapper.findAll("input");
const emailInput = inputs.find((i) => (i.element as HTMLInputElement).type === "email");
const passwordInput = inputs.find((i) => (i.element as HTMLInputElement).type === "password");
if (emailInput && passwordInput) {
await emailInput.setValue("new@test.com");
await passwordInput.setValue("securepassword123");
await wrapper.find("form").trigger("submit");
await flushPromises();
expect(mockedCreateUser).toHaveBeenCalled();
}
});
});

211
frontend/src/features/settings/components/AdminUsersSection.vue Normal file
View file

@ -0,0 +1,211 @@
<script setup lang="ts">
import { computed, ref } from "vue";
import AppButton from "@/components/ui/AppButton.vue";
import AppCard from "@/components/ui/AppCard.vue";
import AppCheckbox from "@/components/ui/AppCheckbox.vue";
import AppHelpHint from "@/components/ui/AppHelpHint.vue";
import AppInput from "@/components/ui/AppInput.vue";
import AppModal from "@/components/ui/AppModal.vue";
import AppRefreshButton from "@/components/ui/AppRefreshButton.vue";
import AppTable from "@/components/ui/AppTable.vue";
import {
createAdminUser,
listAdminUsers,
resetAdminUserPassword,
setAdminUserActive,
type AdminUser,
} from "@/features/admin_users";
import { useRequestState } from "@/composables/useRequestState";
const { errorMessage, errorRequestId, successMessage, clearAlerts, assignError, setSuccess } = useRequestState();
const refreshingUsers = ref(false);
const creating = ref(false);
const togglingUserId = ref<number | null>(null);
const resettingPassword = ref(false);
const users = ref<AdminUser[]>([]);
const email = ref("");
const password = ref("");
const createIsAdmin = ref(false);
const activeUserId = ref<number | null>(null);
const resetPassword = ref("");
const activeUser = computed(() => users.value.find((u) => u.id === activeUserId.value) ?? null);
function userRoleLabel(user: AdminUser): string {
return user.is_admin ? "Admin" : "User";
}
function statusDotClass(user: AdminUser): string {
return user.is_active ? "bg-success-500 ring-success-200" : "bg-ink-muted/70 ring-stroke-default";
}
function formatTimestamp(value: string | null): string {
if (!value) return "n/a";
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? value : parsed.toLocaleString();
}
function openUserModal(user: AdminUser): void {
activeUserId.value = user.id;
resetPassword.value = "";
}
function closeUserModal(): void {
activeUserId.value = null;
resetPassword.value = "";
}
async function refreshUsers(): Promise<void> {
refreshingUsers.value = true;
try {
users.value = await listAdminUsers();
if (activeUserId.value !== null && !users.value.some((item) => item.id === activeUserId.value)) {
closeUserModal();
}
} finally {
refreshingUsers.value = false;
}
}
async function onCreateUser(): Promise<void> {
creating.value = true;
clearAlerts();
try {
if (!email.value.trim() || !password.value) throw new Error("Email and password are required.");
const created = await createAdminUser({ email: email.value.trim(), password: password.value, is_admin: createIsAdmin.value });
email.value = "";
password.value = "";
createIsAdmin.value = false;
setSuccess(`User created: ${created.email}`);
await refreshUsers();
} catch (error) {
assignError(error, "Unable to create user.");
} finally {
creating.value = false;
}
}
async function onToggleUser(user: AdminUser): Promise<void> {
togglingUserId.value = user.id;
clearAlerts();
try {
const updated = await setAdminUserActive(user.id, !user.is_active);
setSuccess(`${updated.email} is now ${updated.is_active ? "active" : "inactive"}.`);
await refreshUsers();
} catch (error) {
assignError(error, "Unable to update user.");
} finally {
togglingUserId.value = null;
}
}
async function onResetPassword(): Promise<void> {
const user = activeUser.value;
if (!user) return;
resettingPassword.value = true;
clearAlerts();
try {
const candidate = resetPassword.value.trim();
if (candidate.length < 12) throw new Error("New password must be at least 12 characters.");
const result = await resetAdminUserPassword(user.id, candidate);
resetPassword.value = "";
setSuccess(result.message || `Password reset for ${user.email}.`);
} catch (error) {
assignError(error, "Unable to reset password.");
} finally {
resettingPassword.value = false;
}
}
async function load(): Promise<void> {
await refreshUsers();
}
defineExpose({ load, errorMessage, errorRequestId, successMessage, users });
</script>
<template>
<AppCard class="space-y-4">
<div class="flex flex-wrap items-center justify-between gap-2">
<div class="flex items-center gap-1">
<h2 class="text-lg font-semibold text-ink-primary">Users</h2>
<AppHelpHint text="Create accounts, toggle active status, and reset passwords." />
</div>
<AppRefreshButton variant="secondary" :loading="refreshingUsers" title="Refresh users" loading-title="Refreshing users" @click="refreshUsers" />
</div>
<form class="grid gap-3 md:grid-cols-3" @submit.prevent="onCreateUser">
<label class="grid gap-1 text-sm font-medium text-ink-secondary">
<span>Email</span>
<AppInput v-model="email" type="email" autocomplete="off" />
</label>
<label class="grid gap-1 text-sm font-medium text-ink-secondary">
<span>Password</span>
<AppInput v-model="password" type="password" autocomplete="new-password" />
</label>
<div class="grid gap-2 text-sm font-medium text-ink-secondary">
<span>Role</span>
<div class="flex items-center gap-3">
<AppCheckbox id="admin-create-user-is-admin" v-model="createIsAdmin" label="Grant admin" />
<AppButton type="submit" :disabled="creating">{{ creating ? "Creating..." : "Create user" }}</AppButton>
</div>
</div>
</form>
<AppTable label="Users table">
<thead>
<tr>
<th scope="col">Email</th>
<th scope="col">Role</th>
<th scope="col">Status</th>
<th scope="col">Actions</th>
</tr>
</thead>
<tbody>
<tr v-for="user in users" :key="user.id">
<td class="align-middle">
<button type="button" class="group inline-flex items-center gap-2 rounded-md px-1 py-0.5 text-left text-ink-primary transition hover:bg-surface-card-muted focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-focus-ring focus-visible:ring-offset-2 focus-visible:ring-offset-focus-offset" @click="openUserModal(user)">
<span :class="statusDotClass(user)" class="h-2.5 w-2.5 rounded-full ring-2" :aria-label="user.is_active ? 'Active user' : 'Inactive user'" />
<span class="underline-offset-2 group-hover:underline">{{ user.email }}</span>
</button>
</td>
<td>{{ userRoleLabel(user) }}</td>
<td>{{ user.is_active ? "active" : "inactive" }}</td>
<td class="flex flex-wrap items-center gap-2">
<AppButton variant="secondary" :disabled="togglingUserId === user.id" @click="onToggleUser(user)">
{{ togglingUserId === user.id ? "Updating..." : user.is_active ? "Deactivate" : "Activate" }}
</AppButton>
<AppButton variant="ghost" @click="openUserModal(user)">Manage</AppButton>
</td>
</tr>
</tbody>
</AppTable>
</AppCard>
<AppModal :open="activeUser !== null" title="User settings" @close="closeUserModal">
<div v-if="activeUser" class="grid gap-4">
<div class="space-y-1">
<div class="flex items-center gap-2">
<span :class="statusDotClass(activeUser)" class="h-2.5 w-2.5 rounded-full ring-2" :aria-label="activeUser.is_active ? 'Active user' : 'Inactive user'" />
<p class="truncate text-sm font-semibold text-ink-primary">{{ activeUser.email }}</p>
</div>
<p class="text-sm text-secondary">Role: {{ userRoleLabel(activeUser) }}</p>
<p class="text-xs text-secondary">Last updated: {{ formatTimestamp(activeUser.updated_at) }}</p>
</div>
<div class="grid gap-2">
<label class="grid gap-1 text-sm font-medium text-ink-secondary">
<span>New password</span>
<AppInput id="admin-reset-password" v-model="resetPassword" type="password" autocomplete="new-password" placeholder="At least 12 characters" />
</label>
<div class="flex flex-wrap gap-2">
<AppButton :disabled="resettingPassword" @click="onResetPassword">{{ resettingPassword ? "Resetting..." : "Reset password" }}</AppButton>
<AppButton variant="secondary" :disabled="togglingUserId === activeUser.id" @click="onToggleUser(activeUser)">
{{ togglingUserId === activeUser.id ? "Updating..." : activeUser.is_active ? "Deactivate user" : "Activate user" }}
</AppButton>
</div>
</div>
</div>
</AppModal>
</template>

1163
frontend/src/pages/PublicationsPage.vue
View file

File diff suppressed because it is too large Load diff

1157
frontend/src/pages/ScholarsPage.vue
View file

File diff suppressed because it is too large Load diff

114
tests/integration/helpers.py
View file

@ -1,11 +1,35 @@
from __future__ import annotations
import asyncio
from pathlib import Path
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.auth.security import PasswordService
REGRESSION_FIXTURE_DIR = Path("tests/fixtures/scholar/regression")
SAFETY_STATE_KEYS = {
"cooldown_active",
"cooldown_reason",
"cooldown_reason_label",
"cooldown_until",
"cooldown_remaining_seconds",
"recommended_action",
"counters",
}
SAFETY_COUNTER_KEYS = {
"consecutive_blocked_runs",
"consecutive_network_runs",
"cooldown_entry_count",
"blocked_start_count",
"last_blocked_failure_count",
"last_network_failure_count",
"last_evaluated_run_id",
}
ACTIVE_RUN_STATUSES = {"running", "resolving"}
def login_user(client: TestClient, *, email: str, password: str) -> None:
bootstrap_response = client.get("/api/v1/auth/csrf")
@ -54,3 +78,93 @@ async def insert_user(
user_id = int(result.scalar_one())
await db_session.commit()
return user_id
def api_bootstrap_csrf_headers(client: TestClient) -> dict[str, str]:
bootstrap_response = client.get("/api/v1/auth/csrf")
assert bootstrap_response.status_code == 200
payload = bootstrap_response.json()["data"]
token = payload["csrf_token"]
assert isinstance(token, str) and token
return {"X-CSRF-Token": token}
def api_csrf_headers(client: TestClient) -> dict[str, str]:
me_response = client.get("/api/v1/auth/me")
assert me_response.status_code == 200
body = me_response.json()
token = body["data"]["csrf_token"]
assert isinstance(token, str) and token
return {"X-CSRF-Token": token}
def regression_fixture(name: str) -> str:
return (REGRESSION_FIXTURE_DIR / name).read_text(encoding="utf-8")
def assert_safety_state_contract(payload: dict[str, object]) -> None:
assert set(payload.keys()) == SAFETY_STATE_KEYS
counters = payload["counters"]
assert isinstance(counters, dict)
assert set(counters.keys()) == SAFETY_COUNTER_KEYS
async def wait_for_run_complete(
client: TestClient,
run_id: int,
*,
max_retries: int = 300,
poll_interval: float = 0.2,
) -> dict:
for _ in range(max_retries):
await asyncio.sleep(poll_interval)
r = client.get(f"/api/v1/runs/{run_id}")
assert r.status_code == 200
data = r.json()["data"]
if data["run"]["status"] not in ACTIVE_RUN_STATUSES:
return data
r = client.get(f"/api/v1/runs/{run_id}")
return r.json()["data"]
async def seed_publication_link_for_user(
db_session: AsyncSession,
*,
user_id: int,
scholar_id: str,
title: str,
fingerprint: str,
) -> tuple[int, int]:
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{"user_id": user_id, "scholar_id": scholar_id, "display_name": scholar_id},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 4)
RETURNING id
"""
),
{"fingerprint": fingerprint, "title_raw": title, "title_normalized": title.lower()},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{"scholar_profile_id": scholar_profile_id, "publication_id": publication_id},
)
await db_session.commit()
return scholar_profile_id, publication_id

576
tests/integration/test_api_admin.py Normal file
View file

@ -0,0 +1,576 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from app.settings import settings
from tests.integration.helpers import (
api_bootstrap_csrf_headers,
api_csrf_headers,
insert_user,
login_user,
seed_publication_link_for_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_user_management_endpoints(db_session: AsyncSession) -> None:
admin_user_id = await insert_user(
db_session,
email="api-admin@example.com",
password="admin-password",
is_admin=True,
)
target_user_id = await insert_user(
db_session,
email="api-target@example.com",
password="target-password",
is_admin=False,
)
await insert_user(
db_session,
email="api-member@example.com",
password="member-password",
is_admin=False,
)
client = TestClient(app)
login_user(client, email="api-admin@example.com", password="admin-password")
headers = api_csrf_headers(client)
list_response = client.get("/api/v1/admin/users")
assert list_response.status_code == 200
users = list_response.json()["data"]["users"]
assert any(item["email"] == "api-target@example.com" for item in users)
create_response = client.post(
"/api/v1/admin/users",
json={
"email": "api-created@example.com",
"password": "created-password",
"is_admin": True,
},
headers=headers,
)
assert create_response.status_code == 201
created_user = create_response.json()["data"]
assert created_user["email"] == "api-created@example.com"
created_user_id = int(created_user["id"])
deactivate_response = client.patch(
f"/api/v1/admin/users/{target_user_id}/active",
json={"is_active": False},
headers=headers,
)
assert deactivate_response.status_code == 200
assert deactivate_response.json()["data"]["is_active"] is False
reactivate_response = client.patch(
f"/api/v1/admin/users/{target_user_id}/active",
json={"is_active": True},
headers=headers,
)
assert reactivate_response.status_code == 200
assert reactivate_response.json()["data"]["is_active"] is True
reset_response = client.post(
f"/api/v1/admin/users/{target_user_id}/reset-password",
json={"new_password": "target-password-updated"},
headers=headers,
)
assert reset_response.status_code == 200
assert "Password reset" in reset_response.json()["data"]["message"]
self_deactivate = client.patch(
f"/api/v1/admin/users/{admin_user_id}/active",
json={"is_active": False},
headers=headers,
)
assert self_deactivate.status_code == 400
assert self_deactivate.json()["error"]["code"] == "cannot_deactivate_self"
logout_response = client.post("/api/v1/auth/logout", headers=headers)
assert logout_response.status_code == 200
target_headers = api_bootstrap_csrf_headers(client)
target_login = client.post(
"/api/v1/auth/login",
json={"email": "api-target@example.com", "password": "target-password-updated"},
headers=target_headers,
)
assert target_login.status_code == 200
non_admin_headers = {"X-CSRF-Token": target_login.json()["data"]["csrf_token"]}
forbidden_response = client.get("/api/v1/admin/users")
assert forbidden_response.status_code == 403
assert forbidden_response.json()["error"]["code"] == "forbidden"
forbidden_create = client.post(
"/api/v1/admin/users",
json={
"email": "should-not-work@example.com",
"password": "password-123",
"is_admin": False,
},
headers=non_admin_headers,
)
assert forbidden_create.status_code == 403
created_exists = await db_session.execute(
text("SELECT COUNT(*) FROM users WHERE id = :user_id"),
{"user_id": created_user_id},
)
assert created_exists.scalar_one() == 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_scholar_http_settings_endpoints(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-admin-http@example.com",
password="admin-password",
is_admin=True,
)
await insert_user(
db_session,
email="api-member-http@example.com",
password="member-password",
is_admin=False,
)
previous_user_agent = settings.scholar_http_user_agent
previous_rotate = settings.scholar_http_rotate_user_agent
previous_accept_language = settings.scholar_http_accept_language
previous_cookie = settings.scholar_http_cookie
try:
client = TestClient(app)
login_user(client, email="api-admin-http@example.com", password="admin-password")
headers = api_csrf_headers(client)
read_response = client.get("/api/v1/admin/settings/scholar-http")
assert read_response.status_code == 200
update_response = client.put(
"/api/v1/admin/settings/scholar-http",
json={
"user_agent": "Mozilla/5.0 Test Runner",
"rotate_user_agent": False,
"accept_language": "en-US,en;q=0.8",
"cookie": "SID=test-cookie",
},
headers=headers,
)
assert update_response.status_code == 200
payload = update_response.json()["data"]
assert payload["user_agent"] == "Mozilla/5.0 Test Runner"
assert payload["cookie"] == "SID=test-cookie"
assert settings.scholar_http_user_agent == "Mozilla/5.0 Test Runner"
client.post("/api/v1/auth/logout", headers=headers)
login_user(client, email="api-member-http@example.com", password="member-password")
forbidden_response = client.get("/api/v1/admin/settings/scholar-http")
assert forbidden_response.status_code == 403
finally:
object.__setattr__(settings, "scholar_http_user_agent", previous_user_agent)
object.__setattr__(settings, "scholar_http_rotate_user_agent", previous_rotate)
object.__setattr__(settings, "scholar_http_accept_language", previous_accept_language)
object.__setattr__(settings, "scholar_http_cookie", previous_cookie)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_integrity_and_repair_flow(db_session: AsyncSession) -> None:
await insert_user(db_session, email="api-admin-dbops@example.com", password="admin-password", is_admin=True)
target_user_id = await insert_user(db_session, email="api-dbops-target@example.com", password="target-password")
_scholar_id, publication_id = await seed_publication_link_for_user(
db_session,
user_id=target_user_id,
scholar_id="dbopsTarget01",
title="DB Ops Target Paper",
fingerprint=f"{(target_user_id + 31):064x}",
)
client = TestClient(app)
login_user(client, email="api-admin-dbops@example.com", password="admin-password")
headers = api_csrf_headers(client)
integrity_response = client.get("/api/v1/admin/db/integrity")
assert integrity_response.status_code == 200
integrity_payload = integrity_response.json()["data"]
assert integrity_payload["status"] in {"ok", "warning", "failed"}
assert any(check["name"] == "missing_pdf_url" for check in integrity_payload["checks"])
repair_response = client.post(
"/api/v1/admin/db/repairs/publication-links",
json={"user_id": target_user_id, "dry_run": True},
headers=headers,
)
assert repair_response.status_code == 200
repair_data = repair_response.json()["data"]
assert repair_data["status"] == "completed"
assert bool(repair_data["summary"]["dry_run"]) is True
job_id = int(repair_data["job_id"])
jobs_response = client.get("/api/v1/admin/db/repair-jobs?limit=10")
assert jobs_response.status_code == 200
jobs = jobs_response.json()["data"]["jobs"]
assert any(int(job["id"]) == job_id for job in jobs)
pdf_queue_response = client.get("/api/v1/admin/db/pdf-queue?limit=20")
assert pdf_queue_response.status_code == 200
pdf_queue_payload = pdf_queue_response.json()["data"]
assert isinstance(pdf_queue_payload["items"], list)
assert int(pdf_queue_payload["page"]) == 1
assert int(pdf_queue_payload["page_size"]) == 20
assert int(pdf_queue_payload["total_count"]) >= 0
assert isinstance(pdf_queue_payload["has_next"], bool)
assert isinstance(pdf_queue_payload["has_prev"], bool)
page_two_response = client.get("/api/v1/admin/db/pdf-queue?page=2&page_size=1")
assert page_two_response.status_code == 200
page_two_payload = page_two_response.json()["data"]
assert int(page_two_payload["page"]) == 2
assert int(page_two_payload["page_size"]) == 1
assert page_two_payload["has_prev"] is True
untracked_response = client.get("/api/v1/admin/db/pdf-queue?limit=20&status=untracked")
assert untracked_response.status_code == 200
assert any(item["status"] == "untracked" for item in untracked_response.json()["data"]["items"])
link_count_result = await db_session.execute(
text("SELECT count(*) FROM scholar_publications WHERE publication_id = :publication_id"),
{"publication_id": publication_id},
)
assert int(link_count_result.scalar_one()) == 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_pdf_queue_requeue_endpoint(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
await insert_user(db_session, email="api-admin-pdf@example.com", password="admin-password", is_admin=True)
target_user_id = await insert_user(db_session, email="api-pdf-target@example.com", password="target-password")
_scholar_id, publication_id = await seed_publication_link_for_user(
db_session,
user_id=target_user_id,
scholar_id="dbopsPdf01",
title="PDF Queue Target",
fingerprint=f"{(target_user_id + 73):064x}",
)
monkeypatch.setattr(
"app.services.publications.pdf_queue.schedule_rows",
lambda **_kwargs: None,
)
client = TestClient(app)
login_user(client, email="api-admin-pdf@example.com", password="admin-password")
headers = api_csrf_headers(client)
first_response = client.post(
f"/api/v1/admin/db/pdf-queue/{publication_id}/requeue",
headers=headers,
)
assert first_response.status_code == 200
first_data = first_response.json()["data"]
assert first_data["publication_id"] == publication_id
assert first_data["queued"] is True
assert first_data["status"] == "queued"
second_response = client.post(
f"/api/v1/admin/db/pdf-queue/{publication_id}/requeue",
headers=headers,
)
assert second_response.status_code == 200
second_data = second_response.json()["data"]
assert second_data["queued"] is False
assert second_data["status"] == "blocked"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_pdf_queue_requeue_all_endpoint(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
await insert_user(db_session, email="api-admin-pdf-all@example.com", password="admin-password", is_admin=True)
target_user_id = await insert_user(db_session, email="api-pdf-all-target@example.com", password="target-password")
_scholar_id, _publication_id = await seed_publication_link_for_user(
db_session,
user_id=target_user_id,
scholar_id="dbopsPdfAll01",
title="PDF Queue All Target",
fingerprint=f"{(target_user_id + 83):064x}",
)
monkeypatch.setattr(
"app.services.publications.pdf_queue.schedule_rows",
lambda **_kwargs: None,
)
client = TestClient(app)
login_user(client, email="api-admin-pdf-all@example.com", password="admin-password")
headers = api_csrf_headers(client)
response = client.post(
"/api/v1/admin/db/pdf-queue/requeue-all?limit=500",
headers=headers,
)
assert response.status_code == 200
payload = response.json()["data"]
assert int(payload["requested_count"]) >= 1
assert int(payload["queued_count"]) >= 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_forbidden_for_non_admin_and_validates_scope(db_session: AsyncSession) -> None:
await insert_user(db_session, email="api-admin-dbops2@example.com", password="admin-password", is_admin=True)
member_user_id = await insert_user(db_session, email="api-dbops-member@example.com", password="member-password")
client = TestClient(app)
login_user(client, email="api-dbops-member@example.com", password="member-password")
headers = api_csrf_headers(client)
forbidden_integrity = client.get("/api/v1/admin/db/integrity")
assert forbidden_integrity.status_code == 403
assert forbidden_integrity.json()["error"]["code"] == "forbidden"
forbidden_pdf_queue = client.get("/api/v1/admin/db/pdf-queue")
assert forbidden_pdf_queue.status_code == 403
assert forbidden_pdf_queue.json()["error"]["code"] == "forbidden"
forbidden_requeue = client.post(
"/api/v1/admin/db/pdf-queue/1/requeue",
headers=headers,
)
assert forbidden_requeue.status_code == 403
assert forbidden_requeue.json()["error"]["code"] == "forbidden"
forbidden_requeue_all = client.post(
"/api/v1/admin/db/pdf-queue/requeue-all?limit=100",
headers=headers,
)
assert forbidden_requeue_all.status_code == 403
assert forbidden_requeue_all.json()["error"]["code"] == "forbidden"
forbidden_repair = client.post(
"/api/v1/admin/db/repairs/publication-links",
json={"user_id": member_user_id, "dry_run": True},
headers=headers,
)
assert forbidden_repair.status_code == 403
assert forbidden_repair.json()["error"]["code"] == "forbidden"
forbidden_near_duplicate = client.post(
"/api/v1/admin/db/repairs/publication-near-duplicates",
json={"dry_run": True},
headers=headers,
)
assert forbidden_near_duplicate.status_code == 403
assert forbidden_near_duplicate.json()["error"]["code"] == "forbidden"
admin_headers = api_bootstrap_csrf_headers(client)
admin_login = client.post(
"/api/v1/auth/login",
json={"email": "api-admin-dbops2@example.com", "password": "admin-password"},
headers=admin_headers,
)
assert admin_login.status_code == 200
post_login_headers = {"X-CSRF-Token": admin_login.json()["data"]["csrf_token"]}
invalid_scope = client.post(
"/api/v1/admin/db/repairs/publication-links",
json={"user_id": member_user_id, "dry_run": True},
headers=post_login_headers,
)
assert invalid_scope.status_code == 400
assert invalid_scope.json()["error"]["code"] == "invalid_repair_scope"
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_all_users_apply_requires_confirmation(db_session: AsyncSession) -> None:
await insert_user(db_session, email="api-admin-dbops3@example.com", password="admin-password", is_admin=True)
first_user_id = await insert_user(db_session, email="api-dbops-a@example.com", password="user-password")
second_user_id = await insert_user(db_session, email="api-dbops-b@example.com", password="user-password")
await seed_publication_link_for_user(
db_session,
user_id=first_user_id,
scholar_id="dbopsAll01",
title="DB Ops All User Paper One",
fingerprint=f"{(first_user_id + 61):064x}",
)
await seed_publication_link_for_user(
db_session,
user_id=second_user_id,
scholar_id="dbopsAll02",
title="DB Ops All User Paper Two",
fingerprint=f"{(second_user_id + 71):064x}",
)
client = TestClient(app)
login_user(client, email="api-admin-dbops3@example.com", password="admin-password")
headers = api_csrf_headers(client)
missing_confirmation = client.post(
"/api/v1/admin/db/repairs/publication-links",
json={"scope_mode": "all_users", "dry_run": False},
headers=headers,
)
assert missing_confirmation.status_code == 422
assert "confirmation_text" in str(missing_confirmation.json())
apply_response = client.post(
"/api/v1/admin/db/repairs/publication-links",
json={
"scope_mode": "all_users",
"dry_run": False,
"confirmation_text": "REPAIR ALL USERS",
},
headers=headers,
)
assert apply_response.status_code == 200
apply_data = apply_response.json()["data"]
assert apply_data["status"] == "completed"
assert apply_data["scope"]["scope_mode"] == "all_users"
assert int(apply_data["summary"]["links_deleted"]) >= 2
remaining_links = await db_session.execute(text("SELECT count(*) FROM scholar_publications"))
assert int(remaining_links.scalar_one()) == 0
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_admin_dbops_near_duplicate_repair_scan_and_apply(
db_session: AsyncSession,
) -> None:
await insert_user(
db_session,
email="api-admin-near-dup@example.com",
password="admin-password",
is_admin=True,
)
user_id = await insert_user(
db_session,
email="api-near-dup-target@example.com",
password="user-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "nearDupScholar01",
"display_name": "Near Dup Target",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
first_pub = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, year, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 2014, 100)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1201):064x}",
"title_raw": "Adam: A method for stochastic optimization",
"title_normalized": "adam a method for stochastic optimization",
},
)
first_publication_id = int(first_pub.scalar_one())
second_pub = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, year, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 2015, 10)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1202):064x}",
"title_raw": "†œAdam: A method for stochastic optimization, †3rd Int. Conf. Learn. Represent.",
"title_normalized": "adam method for stochastic optimization",
},
)
second_publication_id = int(second_pub.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :first_publication_id, false),
(:scholar_profile_id, :second_publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"first_publication_id": first_publication_id,
"second_publication_id": second_publication_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-admin-near-dup@example.com", password="admin-password")
headers = api_csrf_headers(client)
scan_response = client.post(
"/api/v1/admin/db/repairs/publication-near-duplicates",
json={"dry_run": True, "max_clusters": 20},
headers=headers,
)
assert scan_response.status_code == 200
scan_data = scan_response.json()["data"]
assert scan_data["status"] == "completed"
assert int(scan_data["summary"]["candidate_cluster_count"]) >= 1
assert len(scan_data["clusters"]) >= 1
selected_key = str(scan_data["clusters"][0]["cluster_key"])
missing_confirmation = client.post(
"/api/v1/admin/db/repairs/publication-near-duplicates",
json={"dry_run": False, "selected_cluster_keys": [selected_key]},
headers=headers,
)
assert missing_confirmation.status_code == 422
assert "confirmation_text" in str(missing_confirmation.json())
apply_response = client.post(
"/api/v1/admin/db/repairs/publication-near-duplicates",
json={
"dry_run": False,
"selected_cluster_keys": [selected_key],
"confirmation_text": "MERGE SELECTED DUPLICATES",
},
headers=headers,
)
assert apply_response.status_code == 200
apply_data = apply_response.json()["data"]
assert apply_data["status"] == "completed"
assert int(apply_data["summary"]["merged_publications"]) >= 1
remaining = await db_session.execute(
text(
"""
SELECT count(*)
FROM publications
WHERE id IN (:first_publication_id, :second_publication_id)
"""
),
{
"first_publication_id": first_publication_id,
"second_publication_id": second_publication_id,
},
)
assert int(remaining.scalar_one()) == 1

182
tests/integration/test_api_auth.py Normal file
View file

@ -0,0 +1,182 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from tests.integration.helpers import (
api_bootstrap_csrf_headers,
insert_user,
login_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_me_requires_authentication() -> None:
client = TestClient(app)
response = client.get("/api/v1/auth/me")
assert response.status_code == 401
payload = response.json()
assert payload["error"]["code"] == "auth_required"
assert payload["error"]["message"] == "Authentication required."
assert "request_id" in payload["meta"]
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_envelope_contract_for_success_and_csrf_error() -> None:
client = TestClient(app)
success_response = client.get("/api/v1/auth/csrf")
assert success_response.status_code == 200
success_payload = success_response.json()
assert set(success_payload.keys()) == {"data", "meta"}
assert isinstance(success_payload["data"]["csrf_token"], str)
assert set(success_payload["meta"].keys()) == {"request_id"}
error_response = client.post(
"/api/v1/auth/login",
json={"email": "missing-csrf@example.com", "password": "irrelevant"},
)
assert error_response.status_code == 403
error_payload = error_response.json()
assert set(error_payload.keys()) == {"error", "meta"}
assert set(error_payload["error"].keys()) == {"code", "message", "details"}
assert error_payload["error"]["code"] == "csrf_invalid"
assert error_payload["error"]["details"] is None
assert set(error_payload["meta"].keys()) == {"request_id"}
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_csrf_bootstrap_returns_token_for_anonymous_and_authenticated(
db_session: AsyncSession,
) -> None:
await insert_user(
db_session,
email="api-bootstrap@example.com",
password="api-password",
)
client = TestClient(app)
anonymous_response = client.get("/api/v1/auth/csrf")
assert anonymous_response.status_code == 200
anonymous_payload = anonymous_response.json()["data"]
assert isinstance(anonymous_payload["csrf_token"], str)
assert anonymous_payload["authenticated"] is False
login_user(client, email="api-bootstrap@example.com", password="api-password")
authenticated_response = client.get("/api/v1/auth/csrf")
assert authenticated_response.status_code == 200
authenticated_payload = authenticated_response.json()["data"]
assert isinstance(authenticated_payload["csrf_token"], str)
assert authenticated_payload["authenticated"] is True
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_me_returns_user_and_csrf_token(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-me@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-me@example.com", password="api-password")
response = client.get("/api/v1/auth/me")
assert response.status_code == 200
payload = response.json()
assert payload["data"]["authenticated"] is True
assert payload["data"]["user"]["email"] == "api-me@example.com"
assert isinstance(payload["data"]["csrf_token"], str)
assert payload["meta"]["request_id"]
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_login_and_change_password_flow(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-login@example.com",
password="old-password",
)
client = TestClient(app)
missing_csrf = client.post(
"/api/v1/auth/login",
json={"email": "api-login@example.com", "password": "old-password"},
)
assert missing_csrf.status_code == 403
assert missing_csrf.json()["error"]["code"] == "csrf_missing"
login_headers = api_bootstrap_csrf_headers(client)
login_response = client.post(
"/api/v1/auth/login",
json={"email": "api-login@example.com", "password": "old-password"},
headers=login_headers,
)
assert login_response.status_code == 200
login_payload = login_response.json()["data"]
assert login_payload["authenticated"] is True
assert login_payload["user"]["email"] == "api-login@example.com"
assert isinstance(login_payload["csrf_token"], str)
bad_change_response = client.post(
"/api/v1/auth/change-password",
json={
"current_password": "not-correct",
"new_password": "new-password",
"confirm_password": "new-password",
},
headers={"X-CSRF-Token": login_payload["csrf_token"]},
)
assert bad_change_response.status_code == 400
assert bad_change_response.json()["error"]["code"] == "invalid_current_password"
change_response = client.post(
"/api/v1/auth/change-password",
json={
"current_password": "old-password",
"new_password": "new-password",
"confirm_password": "new-password",
},
headers={"X-CSRF-Token": login_payload["csrf_token"]},
)
assert change_response.status_code == 200
assert change_response.json()["data"]["message"] == "Password updated successfully."
logout_response = client.post(
"/api/v1/auth/logout",
headers={"X-CSRF-Token": login_payload["csrf_token"]},
)
assert logout_response.status_code == 200
relogin_headers = api_bootstrap_csrf_headers(client)
old_password_login = client.post(
"/api/v1/auth/login",
json={"email": "api-login@example.com", "password": "old-password"},
headers=relogin_headers,
)
assert old_password_login.status_code == 401
assert old_password_login.json()["error"]["code"] == "invalid_credentials"
fresh_headers = api_bootstrap_csrf_headers(client)
new_password_login = client.post(
"/api/v1/auth/login",
json={"email": "api-login@example.com", "password": "new-password"},
headers=fresh_headers,
)
assert new_password_login.status_code == 200
assert new_password_login.json()["data"]["authenticated"] is True

573
tests/integration/test_api_publications.py Normal file
View file

@ -0,0 +1,573 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from tests.integration.helpers import (
api_csrf_headers,
insert_user,
login_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_list_and_mark_read(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-pubs@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "abcDEF123456",
"display_name": "Publication Owner",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_a = await db_session.execute(
text(
"""
INSERT INTO publications (
fingerprint_sha256,
title_raw,
title_normalized,
citation_count,
pdf_url
)
VALUES (:fingerprint, :title_raw, :title_normalized, 10, :pdf_url)
RETURNING id
"""
),
{
"fingerprint": f"{user_id:064x}",
"title_raw": "Paper A",
"title_normalized": "paper a",
"pdf_url": "https://example.org/paper-a.pdf",
},
)
publication_a_id = int(publication_a.scalar_one())
publication_b = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 4)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1):064x}",
"title_raw": "Paper B",
"title_normalized": "paper b",
},
)
publication_b_id = int(publication_b.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES
(:scholar_profile_id, :publication_a_id, false),
(:scholar_profile_id, :publication_b_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_a_id": publication_a_id,
"publication_b_id": publication_b_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs@example.com", password="api-password")
headers = api_csrf_headers(client)
list_response = client.get("/api/v1/publications?mode=all")
assert list_response.status_code == 200
data = list_response.json()["data"]
assert data["mode"] == "all"
assert data["favorite_only"] is False
assert data["total_count"] == 2
assert data["unread_count"] == 2
assert data["favorites_count"] == 0
assert data["latest_count"] == 0
assert data["new_count"] == data["latest_count"]
assert data["page"] == 1
assert data["page_size"] == 100
assert data["has_prev"] is False
assert data["has_next"] is False
assert isinstance(data["publications"], list)
assert len(data["publications"]) == 2
assert all(bool(item["is_favorite"]) is False for item in data["publications"])
pdf_urls = {item["title"]: item["pdf_url"] for item in data["publications"]}
assert pdf_urls["Paper A"] == "https://example.org/paper-a.pdf"
assert pdf_urls["Paper B"] is None
latest_response = client.get("/api/v1/publications?mode=latest")
assert latest_response.status_code == 200
latest_data = latest_response.json()["data"]
assert latest_data["mode"] == "latest"
assert latest_data["latest_count"] == 0
unread_response = client.get("/api/v1/publications?mode=unread")
assert unread_response.status_code == 200
unread_data = unread_response.json()["data"]
assert unread_data["mode"] == "unread"
assert unread_data["unread_count"] == 2
alias_response = client.get("/api/v1/publications?mode=new")
assert alias_response.status_code == 200
alias_data = alias_response.json()["data"]
assert alias_data["mode"] == "latest"
assert alias_data["latest_count"] == latest_data["latest_count"]
assert alias_data["publications"] == latest_data["publications"]
mark_selected_response = client.post(
"/api/v1/publications/mark-read",
json={
"selections": [
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_a_id,
}
]
},
headers=headers,
)
assert mark_selected_response.status_code == 200
assert mark_selected_response.json()["data"]["requested_count"] == 1
assert mark_selected_response.json()["data"]["updated_count"] == 1
read_state = await db_session.execute(
text(
"""
SELECT publication_id, is_read
FROM scholar_publications
WHERE scholar_profile_id = :scholar_profile_id
ORDER BY publication_id
"""
),
{"scholar_profile_id": scholar_profile_id},
)
states = {int(row[0]): bool(row[1]) for row in read_state.all()}
assert states[publication_a_id] is True
assert states[publication_b_id] is False
mark_response = client.post("/api/v1/publications/mark-all-read", headers=headers)
assert mark_response.status_code == 200
assert mark_response.json()["data"]["updated_count"] == 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_list_supports_pagination(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-paging@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "pagingScholar01",
"display_name": "Paging Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_ids: list[int] = []
for index in range(3):
created = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 1)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 500 + index):064x}",
"title_raw": f"Paged Paper {index}",
"title_normalized": f"paged paper {index}",
},
)
publication_ids.append(int(created.scalar_one()))
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read, is_favorite)
VALUES
(:scholar_profile_id, :publication_1, false, false),
(:scholar_profile_id, :publication_2, false, false),
(:scholar_profile_id, :publication_3, false, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_1": publication_ids[0],
"publication_2": publication_ids[1],
"publication_3": publication_ids[2],
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs-paging@example.com", password="api-password")
first_page = client.get("/api/v1/publications?mode=all&page=1&page_size=2")
assert first_page.status_code == 200
first_data = first_page.json()["data"]
assert first_data["total_count"] == 3
assert first_data["page"] == 1
assert first_data["page_size"] == 2
assert first_data["has_prev"] is False
assert first_data["has_next"] is True
assert len(first_data["publications"]) == 2
second_page = client.get("/api/v1/publications?mode=all&page=2&page_size=2")
assert second_page.status_code == 200
second_data = second_page.json()["data"]
assert second_data["total_count"] == 3
assert second_data["page"] == 2
assert second_data["page_size"] == 2
assert second_data["has_prev"] is True
assert second_data["has_next"] is False
assert len(second_data["publications"]) == 1
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_search_pagination_uses_filtered_total_count(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-search-page@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "searchPagingScholar01",
"display_name": "Search Paging Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
titles = ["Alpha Optimization", "Alpha Learning", "Beta Methods"]
publication_ids: list[int] = []
for index, title in enumerate(titles):
created = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 1)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 900 + index):064x}",
"title_raw": title,
"title_normalized": title.lower(),
},
)
publication_ids.append(int(created.scalar_one()))
for publication_id in publication_ids:
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read, is_favorite)
VALUES (:scholar_profile_id, :publication_id, false, false)
"""
),
{"scholar_profile_id": scholar_profile_id, "publication_id": publication_id},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs-search-page@example.com", password="api-password")
response = client.get("/api/v1/publications?mode=all&page=1&page_size=2&search=alpha")
assert response.status_code == 200
data = response.json()["data"]
assert int(data["total_count"]) == 2
assert data["has_next"] is False
assert len(data["publications"]) == 2
assert all("alpha" in str(item["title"]).lower() for item in data["publications"])
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_supports_sort_by_pdf_status(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-pdf-sort@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "pdfSortScholar01",
"display_name": "PDF Sort Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
resolved_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count, pdf_url)
VALUES (:fingerprint, :title_raw, :title_normalized, 1, :pdf_url)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1300):064x}",
"title_raw": "Resolved PDF",
"title_normalized": "resolved pdf",
"pdf_url": "https://example.org/resolved.pdf",
},
)
resolved_publication_id = int(resolved_result.scalar_one())
queued_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 1)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1301):064x}",
"title_raw": "Queued PDF",
"title_normalized": "queued pdf",
},
)
queued_publication_id = int(queued_result.scalar_one())
failed_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 1)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 1302):064x}",
"title_raw": "Failed PDF",
"title_normalized": "failed pdf",
},
)
failed_publication_id = int(failed_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read, is_favorite)
VALUES
(:scholar_profile_id, :resolved_publication_id, false, false),
(:scholar_profile_id, :queued_publication_id, false, false),
(:scholar_profile_id, :failed_publication_id, false, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"resolved_publication_id": resolved_publication_id,
"queued_publication_id": queued_publication_id,
"failed_publication_id": failed_publication_id,
},
)
await db_session.execute(
text(
"""
INSERT INTO publication_pdf_jobs (publication_id, status, attempt_count)
VALUES (:queued_publication_id, 'queued', 1),
(:failed_publication_id, 'failed', 2)
"""
),
{
"queued_publication_id": queued_publication_id,
"failed_publication_id": failed_publication_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs-pdf-sort@example.com", password="api-password")
response = client.get("/api/v1/publications?mode=all&sort_by=pdf_status&sort_dir=desc")
assert response.status_code == 200
publications = response.json()["data"]["publications"]
publication_ids = [int(item["publication_id"]) for item in publications]
assert publication_ids[0] == resolved_publication_id
assert publication_ids[-1] == failed_publication_id
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_favorite_toggle_and_filter(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-favorites@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "favoriteScholar01",
"display_name": "Favorite Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_a_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 9)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 101):064x}",
"title_raw": "Favorite Target",
"title_normalized": "favorite target",
},
)
publication_a_id = int(publication_a_result.scalar_one())
publication_b_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 2)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 102):064x}",
"title_raw": "Not Favorite",
"title_normalized": "not favorite",
},
)
publication_b_id = int(publication_b_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read, is_favorite)
VALUES
(:scholar_profile_id, :publication_a_id, false, false),
(:scholar_profile_id, :publication_b_id, false, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_a_id": publication_a_id,
"publication_b_id": publication_b_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs-favorites@example.com", password="api-password")
headers = api_csrf_headers(client)
set_response = client.post(
f"/api/v1/publications/{publication_a_id}/favorite",
json={"scholar_profile_id": scholar_profile_id, "is_favorite": True},
headers=headers,
)
assert set_response.status_code == 200
set_data = set_response.json()["data"]
assert set_data["publication"]["publication_id"] == publication_a_id
assert set_data["publication"]["is_favorite"] is True
favorite_only_response = client.get("/api/v1/publications?mode=all&favorite_only=true")
assert favorite_only_response.status_code == 200
favorite_only_data = favorite_only_response.json()["data"]
assert favorite_only_data["favorite_only"] is True
assert favorite_only_data["favorites_count"] == 1
assert favorite_only_data["total_count"] == 1
assert len(favorite_only_data["publications"]) == 1
assert int(favorite_only_data["publications"][0]["publication_id"]) == publication_a_id
clear_response = client.post(
f"/api/v1/publications/{publication_a_id}/favorite",
json={"scholar_profile_id": scholar_profile_id, "is_favorite": False},
headers=headers,
)
assert clear_response.status_code == 200
clear_data = clear_response.json()["data"]
assert clear_data["publication"]["publication_id"] == publication_a_id
assert clear_data["publication"]["is_favorite"] is False
favorite_only_after_clear_response = client.get("/api/v1/publications?mode=all&favorite_only=true")
assert favorite_only_after_clear_response.status_code == 200
favorite_only_after_clear_data = favorite_only_after_clear_response.json()["data"]
assert favorite_only_after_clear_data["favorites_count"] == 0
assert favorite_only_after_clear_data["total_count"] == 0
assert favorite_only_after_clear_data["publications"] == []
favorite_state_result = await db_session.execute(
text(
"""
SELECT is_favorite
FROM scholar_publications
WHERE scholar_profile_id = :scholar_profile_id
AND publication_id = :publication_id
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_a_id,
},
)
assert bool(favorite_state_result.scalar_one()) is False

337
tests/integration/test_api_publications_advanced.py Normal file
View file

@ -0,0 +1,337 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from app.services.publications.types import PublicationListItem
from tests.integration.helpers import (
api_csrf_headers,
insert_user,
login_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_list_schedules_background_enrichment(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-bg@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "background111",
"display_name": "Background Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 3)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 17):064x}",
"title_raw": "Background Target",
"title_normalized": "background target",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
},
)
await db_session.commit()
async def _fake_schedule(db_session, *, user_id: int, request_email: str | None, items, max_items: int):
assert db_session is not None
assert user_id > 0
assert request_email == "api-pubs-bg@example.com"
assert int(max_items) > 0
assert any(int(item.publication_id) == publication_id for item in items)
return 1
monkeypatch.setattr(
"app.services.publications.application.schedule_missing_pdf_enrichment_for_user",
_fake_schedule,
)
client = TestClient(app)
login_user(client, email="api-pubs-bg@example.com", password="api-password")
response = client.get("/api/v1/publications?mode=all")
assert response.status_code == 200
data = response.json()["data"]
assert len(data["publications"]) == 1
assert int(data["publications"][0]["publication_id"]) == publication_id
assert data["publications"][0]["pdf_url"] is None
assert data["publications"][0]["pdf_status"] in {"untracked", "queued", "running", "failed"}
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publication_retry_pdf_queues_resolution_job(
db_session: AsyncSession,
monkeypatch: pytest.MonkeyPatch,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-retry@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "retryScholar01",
"display_name": "Retry Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 7)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 9):064x}",
"title_raw": "Retry Target",
"title_normalized": "retry target",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
},
)
await db_session.commit()
async def _fake_retry_scheduler(db_session, *, user_id: int, request_email: str | None, item: PublicationListItem):
assert db_session is not None
assert user_id > 0
assert request_email == "api-pubs-retry@example.com"
assert int(item.publication_id) == publication_id
return True
monkeypatch.setattr(
"app.services.publications.application.schedule_retry_pdf_enrichment_for_row",
_fake_retry_scheduler,
)
async def _fake_hydrate(db_session, *, items: list[PublicationListItem]):
assert db_session is not None
assert len(items) == 1
item = items[0]
return [
PublicationListItem(
publication_id=item.publication_id,
scholar_profile_id=item.scholar_profile_id,
scholar_label=item.scholar_label,
title=item.title,
year=item.year,
citation_count=item.citation_count,
venue_text=item.venue_text,
pub_url=item.pub_url,
pdf_url=item.pdf_url,
is_read=item.is_read,
first_seen_at=item.first_seen_at,
is_new_in_latest_run=item.is_new_in_latest_run,
pdf_status="queued",
pdf_attempt_count=1,
pdf_failure_reason="no_pdf_found",
pdf_failure_detail="no_pdf_found",
)
]
monkeypatch.setattr(
"app.services.publications.application.hydrate_pdf_enrichment_state",
_fake_hydrate,
)
client = TestClient(app)
login_user(client, email="api-pubs-retry@example.com", password="api-password")
headers = api_csrf_headers(client)
response = client.post(
f"/api/v1/publications/{publication_id}/retry-pdf",
json={"scholar_profile_id": scholar_profile_id},
headers=headers,
)
assert response.status_code == 200
payload = response.json()["data"]
assert payload["queued"] is True
assert payload["resolved_pdf"] is False
assert payload["publication"]["publication_id"] == publication_id
assert payload["publication"]["pdf_url"] is None
assert payload["publication"]["pdf_status"] == "queued"
assert payload["publication"]["pdf_attempt_count"] == 1
assert payload["publication"]["pdf_failure_reason"] == "no_pdf_found"
stored = await db_session.execute(
text("SELECT pdf_url FROM publications WHERE id = :publication_id"),
{"publication_id": publication_id},
)
stored_pdf_url = stored.scalar_one()
assert stored_pdf_url is None
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_publications_unread_and_latest_modes_can_diverge(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-pubs-mismatch@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "mismatchScholar01",
"display_name": "Mismatch Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
older_run_result = await db_session.execute(
text(
"""
INSERT INTO crawl_runs (user_id, trigger_type, status, scholar_count, new_pub_count)
VALUES (:user_id, 'manual', 'success', 1, 1)
RETURNING id
"""
),
{"user_id": user_id},
)
older_run_id = int(older_run_result.scalar_one())
latest_run_result = await db_session.execute(
text(
"""
INSERT INTO crawl_runs (user_id, trigger_type, status, scholar_count, new_pub_count)
VALUES (:user_id, 'scheduled', 'success', 1, 0)
RETURNING id
"""
),
{"user_id": user_id},
)
latest_run_id = int(latest_run_result.scalar_one())
assert latest_run_id > older_run_id
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 12)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 99):064x}",
"title_raw": "Unread But Not Latest",
"title_normalized": "unread but not latest",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (
scholar_profile_id,
publication_id,
is_read,
first_seen_run_id
)
VALUES (:scholar_profile_id, :publication_id, false, :first_seen_run_id)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
"first_seen_run_id": older_run_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-pubs-mismatch@example.com", password="api-password")
unread_response = client.get("/api/v1/publications?mode=unread")
assert unread_response.status_code == 200
unread_data = unread_response.json()["data"]
assert unread_data["mode"] == "unread"
assert unread_data["unread_count"] == 1
assert unread_data["latest_count"] == 0
assert unread_data["new_count"] == 0
assert len(unread_data["publications"]) == 1
assert int(unread_data["publications"][0]["publication_id"]) == publication_id
assert unread_data["publications"][0]["is_new_in_latest_run"] is False
latest_response = client.get("/api/v1/publications?mode=latest")
assert latest_response.status_code == 200
latest_data = latest_response.json()["data"]
assert latest_data["mode"] == "latest"
assert latest_data["latest_count"] == 0
assert len(latest_data["publications"]) == 0
alias_response = client.get("/api/v1/publications?mode=new")
assert alias_response.status_code == 200
alias_data = alias_response.json()["data"]
assert alias_data["mode"] == "latest"
assert alias_data["latest_count"] == latest_data["latest_count"]
assert alias_data["publications"] == latest_data["publications"]

499
tests/integration/test_api_runs.py Normal file
View file

@ -0,0 +1,499 @@
from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runtime_deps import get_scholar_source
from app.main import app
from app.services.scholar.source import FetchResult
from app.settings import settings
from tests.integration.helpers import (
api_csrf_headers,
assert_safety_state_contract,
insert_user,
login_user,
regression_fixture,
wait_for_run_complete,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_manual_run_skips_unchanged_initial_page_for_scholar(
db_session: AsyncSession,
) -> None:
await insert_user(
db_session,
email="api-skip-unchanged@example.com",
password="api-password",
)
profile_html = """
<html>
<head>
<meta property="og:image" content="https://images.example.com/skip.png" />
</head>
<body>
<div id="gsc_prf_in">Skip Candidate</div>
<span id="gsc_a_nn">Articles 1-1</span>
<table>
<tbody id="gsc_a_b">
<tr class="gsc_a_tr">
<td class="gsc_a_t">
<a class="gsc_a_at" href="/citations?view_op=view_citation&citation_for_view=abcDEF123456:xyz123">Stable Paper</a>
<div class="gs_gray">A Author</div>
<div class="gs_gray">Stable Venue</div>
</td>
<td class="gsc_a_c"><a class="gsc_a_ac">5</a></td>
<td class="gsc_a_y"><span class="gsc_a_h">2024</span></td>
</tr>
</tbody>
</table>
</body>
</html>
"""
class StubScholarSource:
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
assert scholar_id == "abcDEF123456"
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
status_code=200,
final_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
body=profile_html,
error=None,
)
async def fetch_profile_page_html(
self,
scholar_id: str,
*,
cstart: int,
pagesize: int,
) -> FetchResult:
assert scholar_id == "abcDEF123456"
assert cstart == 0
assert pagesize == settings.ingestion_page_size
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
status_code=200,
final_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
body=profile_html,
error=None,
)
app.dependency_overrides[get_scholar_source] = lambda: StubScholarSource()
try:
with TestClient(app) as client:
login_user(client, email="api-skip-unchanged@example.com", password="api-password")
headers = api_csrf_headers(client)
create_response = client.post(
"/api/v1/scholars",
json={"scholar_id": "abcDEF123456"},
headers=headers,
)
assert create_response.status_code == 201
first_run_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "skip-unchanged-run-001"},
)
assert first_run_response.status_code == 200
first_run_id = int(first_run_response.json()["data"]["run_id"])
first_detail_data = await wait_for_run_complete(client, first_run_id)
first_results = first_detail_data["scholar_results"]
assert len(first_results) == 1
assert first_results[0]["state_reason"] != "no_change_initial_page_signature"
second_run_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "skip-unchanged-run-002"},
)
assert second_run_response.status_code == 200
second_run_id = int(second_run_response.json()["data"]["run_id"])
second_detail_data = await wait_for_run_complete(client, second_run_id)
second_results = second_detail_data["scholar_results"]
assert len(second_results) == 1
assert second_results[0]["state_reason"] == "no_change_initial_page_signature"
assert second_results[0]["publication_count"] == 0
assert second_results[0]["outcome"] == "success"
finally:
app.dependency_overrides.pop(get_scholar_source, None)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_runs_manual_and_queue_actions(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-runs@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-runs@example.com", password="api-password")
headers = api_csrf_headers(client)
run_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "manual-run-0001"},
)
assert run_response.status_code == 200
run_payload = run_response.json()["data"]
assert "run_id" in run_payload
assert run_payload["status"] in {"running", "resolving", "success", "partial_failure", "failed"}
assert run_payload["reused_existing_run"] is False
assert run_payload["idempotency_key"] == "manual-run-0001"
assert_safety_state_contract(run_payload["safety_state"])
run_id = int(run_payload["run_id"])
stored_key = await db_session.execute(
text("SELECT idempotency_key FROM crawl_runs WHERE id = :run_id"),
{"run_id": run_id},
)
assert stored_key.scalar_one() == "manual-run-0001"
replay_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "manual-run-0001"},
)
assert replay_response.status_code in {200, 409}
if replay_response.status_code == 200:
replay_payload = replay_response.json()["data"]
assert replay_payload["run_id"] == run_payload["run_id"]
assert replay_payload["reused_existing_run"] is True
assert_safety_state_contract(replay_payload["safety_state"])
else:
replay_error = replay_response.json()["error"]
assert replay_error["code"] == "run_in_progress"
runs_response = client.get("/api/v1/runs")
assert runs_response.status_code == 200
assert len(runs_response.json()["data"]["runs"]) >= 1
assert_safety_state_contract(runs_response.json()["data"]["safety_state"])
run_detail_response = client.get(f"/api/v1/runs/{run_id}")
assert run_detail_response.status_code == 200
detail_payload = run_detail_response.json()["data"]
assert "summary" in detail_payload
assert isinstance(detail_payload["scholar_results"], list)
assert_safety_state_contract(detail_payload["safety_state"])
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "abcDEF123456",
"display_name": "Queue Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
queue_result = await db_session.execute(
text(
"""
INSERT INTO ingestion_queue_items (
user_id,
scholar_profile_id,
resume_cstart,
reason,
status,
attempt_count,
next_attempt_dt,
dropped_reason,
dropped_at
)
VALUES (
:user_id,
:scholar_profile_id,
7,
'dropped',
'dropped',
2,
NOW(),
'manual_drop',
NOW()
)
RETURNING id
"""
),
{"user_id": user_id, "scholar_profile_id": scholar_profile_id},
)
queue_item_id = int(queue_result.scalar_one())
await db_session.commit()
queue_list_response = client.get("/api/v1/runs/queue/items")
assert queue_list_response.status_code == 200
assert any(int(item["id"]) == queue_item_id for item in queue_list_response.json()["data"]["queue_items"])
retry_response = client.post(f"/api/v1/runs/queue/{queue_item_id}/retry", headers=headers)
assert retry_response.status_code == 200
assert retry_response.json()["data"]["status"] == "queued"
retry_again_response = client.post(
f"/api/v1/runs/queue/{queue_item_id}/retry",
headers=headers,
)
assert retry_again_response.status_code == 409
assert retry_again_response.json()["error"]["code"] == "queue_item_already_queued"
drop_response = client.post(f"/api/v1/runs/queue/{queue_item_id}/drop", headers=headers)
assert drop_response.status_code == 200
assert drop_response.json()["data"]["status"] == "dropped"
clear_response = client.request(
"DELETE",
f"/api/v1/runs/queue/{queue_item_id}",
headers=headers,
)
assert clear_response.status_code == 200
assert clear_response.json()["data"]["status"] == "cleared"
assert clear_response.json()["data"]["message"] == "Queue item cleared."
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_manual_run_can_be_disabled_by_policy(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-runs-policy@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-runs-policy@example.com", password="api-password")
headers = api_csrf_headers(client)
previous_manual_allowed = settings.ingestion_manual_run_allowed
object.__setattr__(settings, "ingestion_manual_run_allowed", False)
try:
response = client.post("/api/v1/runs/manual", headers=headers)
assert response.status_code == 403
payload = response.json()
assert payload["error"]["code"] == "manual_runs_disabled"
assert payload["error"]["details"]["policy"]["manual_run_allowed"] is False
assert payload["error"]["details"]["safety_state"]["cooldown_active"] is False
finally:
object.__setattr__(settings, "ingestion_manual_run_allowed", previous_manual_allowed)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_manual_run_enforces_scrape_safety_cooldown(db_session: AsyncSession) -> None:
user_id = await insert_user(
db_session,
email="api-runs-safety@example.com",
password="api-password",
)
await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
"""
),
{
"user_id": user_id,
"scholar_id": "A1B2C3D4E5F6",
"display_name": "Safety Probe",
},
)
await db_session.commit()
blocked_fixture = regression_fixture("profile_AAAAAAAAAAAA.html")
class BlockedScholarSource:
async def fetch_profile_page_html(
self,
scholar_id: str,
*,
cstart: int,
pagesize: int,
) -> FetchResult:
_ = (scholar_id, cstart, pagesize)
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&user=A1B2C3D4E5F6",
status_code=200,
final_url=(
"https://accounts.google.com/v3/signin/identifier"
"?continue=https%3A%2F%2Fscholar.google.com%2Fcitations"
),
body=blocked_fixture,
error=None,
)
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
_ = scholar_id
return await self.fetch_profile_page_html(
"A1B2C3D4E5F6",
cstart=0,
pagesize=settings.ingestion_page_size,
)
app.dependency_overrides[get_scholar_source] = lambda: BlockedScholarSource()
previous_blocked_cooldown_seconds = settings.ingestion_safety_cooldown_blocked_seconds
object.__setattr__(settings, "ingestion_safety_cooldown_blocked_seconds", 600)
try:
with TestClient(app) as client:
login_user(client, email="api-runs-safety@example.com", password="api-password")
headers = api_csrf_headers(client)
preflight_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "safety-cooldown-run-1"},
)
assert preflight_response.status_code == 429
preflight_payload = preflight_response.json()
assert preflight_payload["error"]["code"] == "scrape_cooldown_active"
preflight_state = preflight_payload["error"]["details"]["safety_state"]
assert_safety_state_contract(preflight_state)
assert preflight_state["cooldown_active"] is True
settings_response = client.get("/api/v1/settings")
assert settings_response.status_code == 200
safety_state = settings_response.json()["data"]["safety_state"]
assert_safety_state_contract(safety_state)
assert safety_state["cooldown_active"] is True
assert safety_state["cooldown_reason"] == "blocked_failure_threshold_exceeded"
assert int(safety_state["cooldown_remaining_seconds"]) > 0
assert int(safety_state["counters"]["cooldown_entry_count"]) >= 1
blocked_start_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "safety-cooldown-run-2"},
)
assert blocked_start_response.status_code == 429
blocked_payload = blocked_start_response.json()
assert blocked_payload["error"]["code"] == "scrape_cooldown_active"
blocked_state = blocked_payload["error"]["details"]["safety_state"]
assert_safety_state_contract(blocked_state)
assert blocked_state["cooldown_active"] is True
assert blocked_state["cooldown_reason"] == "blocked_failure_threshold_exceeded"
assert int(blocked_state["counters"]["blocked_start_count"]) >= 1
runs_response = client.get("/api/v1/runs")
assert runs_response.status_code == 200
assert_safety_state_contract(runs_response.json()["data"]["safety_state"])
assert runs_response.json()["data"]["safety_state"]["cooldown_active"] is True
finally:
object.__setattr__(settings, "ingestion_safety_cooldown_blocked_seconds", previous_blocked_cooldown_seconds)
app.dependency_overrides.pop(get_scholar_source, None)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_manual_run_enforces_network_failure_safety_cooldown(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-runs-safety-network@example.com",
password="api-password",
)
await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
"""
),
{
"user_id": user_id,
"scholar_id": "NNN111NNN111",
"display_name": "Network Safety Probe",
},
)
await db_session.commit()
class NetworkFailureScholarSource:
async def fetch_profile_page_html(
self,
scholar_id: str,
*,
cstart: int,
pagesize: int,
) -> FetchResult:
_ = (scholar_id, cstart, pagesize)
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&user=NNN111NNN111",
status_code=None,
final_url=None,
body="",
error="timed out",
)
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
_ = scholar_id
return await self.fetch_profile_page_html(
"NNN111NNN111",
cstart=0,
pagesize=settings.ingestion_page_size,
)
app.dependency_overrides[get_scholar_source] = lambda: NetworkFailureScholarSource()
previous_network_threshold = settings.ingestion_alert_network_failure_threshold
previous_network_cooldown_seconds = settings.ingestion_safety_cooldown_network_seconds
previous_network_retries = settings.ingestion_network_error_retries
previous_retry_backoff = settings.ingestion_retry_backoff_seconds
object.__setattr__(settings, "ingestion_alert_network_failure_threshold", 1)
object.__setattr__(settings, "ingestion_safety_cooldown_network_seconds", 600)
object.__setattr__(settings, "ingestion_network_error_retries", 0)
object.__setattr__(settings, "ingestion_retry_backoff_seconds", 0.0)
try:
with TestClient(app) as client:
login_user(client, email="api-runs-safety-network@example.com", password="api-password")
headers = api_csrf_headers(client)
first_run_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "safety-network-cooldown-run-1"},
)
assert first_run_response.status_code == 200
first_run_id = int(first_run_response.json()["data"]["run_id"])
await wait_for_run_complete(client, first_run_id)
settings_response = client.get("/api/v1/settings")
assert settings_response.status_code == 200
safety_state = settings_response.json()["data"]["safety_state"]
assert_safety_state_contract(safety_state)
assert safety_state["cooldown_active"] is True
assert safety_state["cooldown_reason"] == "network_failure_threshold_exceeded"
assert int(safety_state["cooldown_remaining_seconds"]) > 0
assert int(safety_state["counters"]["last_network_failure_count"]) >= 1
blocked_start_response = client.post(
"/api/v1/runs/manual",
headers={**headers, "Idempotency-Key": "safety-network-cooldown-run-2"},
)
assert blocked_start_response.status_code == 429
blocked_payload = blocked_start_response.json()
assert blocked_payload["error"]["code"] == "scrape_cooldown_active"
blocked_state = blocked_payload["error"]["details"]["safety_state"]
assert_safety_state_contract(blocked_state)
assert blocked_state["cooldown_active"] is True
assert blocked_state["cooldown_reason"] == "network_failure_threshold_exceeded"
assert int(blocked_state["counters"]["blocked_start_count"]) >= 1
finally:
object.__setattr__(settings, "ingestion_alert_network_failure_threshold", previous_network_threshold)
object.__setattr__(settings, "ingestion_safety_cooldown_network_seconds", previous_network_cooldown_seconds)
object.__setattr__(settings, "ingestion_network_error_retries", previous_network_retries)
object.__setattr__(settings, "ingestion_retry_backoff_seconds", previous_retry_backoff)
app.dependency_overrides.pop(get_scholar_source, None)

370
tests/integration/test_api_scholars.py Normal file
View file

@ -0,0 +1,370 @@
from __future__ import annotations
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.api.runtime_deps import get_scholar_source
from app.main import app
from app.services.scholar.source import FetchResult
from app.settings import settings
from tests.integration.helpers import (
api_csrf_headers,
insert_user,
login_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_scholars_crud_flow(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-scholars@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-scholars@example.com", password="api-password")
headers = api_csrf_headers(client)
missing_csrf = client.post(
"/api/v1/scholars",
json={"scholar_id": "abcDEF123456"},
)
assert missing_csrf.status_code == 403
assert missing_csrf.json()["error"]["code"] == "csrf_invalid"
create_response = client.post(
"/api/v1/scholars",
json={"scholar_id": "abcDEF123456"},
headers=headers,
)
assert create_response.status_code == 201
created = create_response.json()["data"]
assert created["scholar_id"] == "abcDEF123456"
scholar_profile_id = int(created["id"])
list_response = client.get("/api/v1/scholars")
assert list_response.status_code == 200
scholars = list_response.json()["data"]["scholars"]
assert any(int(item["id"]) == scholar_profile_id for item in scholars)
toggle_response = client.patch(
f"/api/v1/scholars/{scholar_profile_id}/toggle",
headers=headers,
)
assert toggle_response.status_code == 200
assert toggle_response.json()["data"]["is_enabled"] is False
delete_response = client.delete(
f"/api/v1/scholars/{scholar_profile_id}",
headers=headers,
)
assert delete_response.status_code == 200
assert delete_response.json()["data"]["message"] == "Scholar deleted."
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_scholars_search_and_profile_image_management(
db_session: AsyncSession,
tmp_path: Path,
) -> None:
await insert_user(
db_session,
email="api-scholar-images@example.com",
password="api-password",
)
class StubScholarSource:
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
assert scholar_id == "abcDEF123456"
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
status_code=200,
final_url="https://scholar.google.com/citations?hl=en&user=abcDEF123456",
body=(
"<html><head>"
'<meta property="og:image" content="https://images.example.com/ada.png" />'
"</head><body>"
'<div id="gsc_prf_in">Ada Lovelace</div>'
"</body></html>"
),
error=None,
)
async def fetch_author_search_html(self, query: str, *, start: int) -> FetchResult:
assert query == "Ada Lovelace"
assert start == 0
return FetchResult(
requested_url="https://scholar.google.com/citations?hl=en&view_op=search_authors&mauthors=ada",
status_code=200,
final_url="https://scholar.google.com/citations?hl=en&view_op=search_authors&mauthors=ada",
body=(
'<div class="gsc_1usr">'
'<img src="/citations/images/avatar_scholar_256.png" />'
'<a class="gs_ai_name" href="/citations?hl=en&user=abcDEF123456">Ada Lovelace</a>'
'<div class="gs_ai_aff">Analytical Engine</div>'
'<div class="gs_ai_eml">Verified email at computing.example</div>'
'<div class="gs_ai_cby">Cited by 42</div>'
'<a class="gs_ai_one_int">Mathematics</a>'
"</div>"
),
error=None,
)
previous_upload_dir = settings.scholar_image_upload_dir
previous_upload_max_bytes = settings.scholar_image_upload_max_bytes
previous_queue_enabled = settings.ingestion_continuation_queue_enabled
app.dependency_overrides[get_scholar_source] = lambda: StubScholarSource()
object.__setattr__(settings, "scholar_image_upload_dir", str(tmp_path / "scholar_images"))
object.__setattr__(settings, "scholar_image_upload_max_bytes", 1_000_000)
object.__setattr__(settings, "ingestion_continuation_queue_enabled", False)
try:
client = TestClient(app)
login_user(client, email="api-scholar-images@example.com", password="api-password")
headers = api_csrf_headers(client)
search_response = client.get("/api/v1/scholars/search", params={"query": "Ada Lovelace", "limit": 5})
assert search_response.status_code == 200
search_payload = search_response.json()["data"]
assert search_payload["state"] == "ok"
assert len(search_payload["candidates"]) == 1
candidate = search_payload["candidates"][0]
assert candidate["scholar_id"] == "abcDEF123456"
assert candidate["profile_image_url"] == "https://scholar.google.com/citations/images/avatar_scholar_256.png"
create_response = client.post(
"/api/v1/scholars",
json={
"scholar_id": candidate["scholar_id"],
},
headers=headers,
)
assert create_response.status_code == 201
created = create_response.json()["data"]
scholar_profile_id = int(created["id"])
assert created["profile_image_source"] == "scraped"
assert created["profile_image_url"] == "https://images.example.com/ada.png"
set_url_response = client.put(
f"/api/v1/scholars/{scholar_profile_id}/image/url",
json={"image_url": "https://cdn.example.com/custom-avatar.png"},
headers=headers,
)
assert set_url_response.status_code == 200
set_url_data = set_url_response.json()["data"]
assert set_url_data["profile_image_source"] == "override"
assert set_url_data["profile_image_url"] == "https://cdn.example.com/custom-avatar.png"
uploaded_bytes = b"\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x01"
upload_response = client.post(
f"/api/v1/scholars/{scholar_profile_id}/image/upload",
files={"image": ("avatar.png", uploaded_bytes, "image/png")},
headers=headers,
)
assert upload_response.status_code == 200
upload_data = upload_response.json()["data"]
assert upload_data["profile_image_source"] == "upload"
assert upload_data["profile_image_url"] == f"/scholar-images/{scholar_profile_id}/upload"
uploaded_image_response = client.get(f"/scholar-images/{scholar_profile_id}/upload")
assert uploaded_image_response.status_code == 200
assert uploaded_image_response.headers["content-type"].startswith("image/png")
assert uploaded_image_response.content == uploaded_bytes
clear_response = client.delete(
f"/api/v1/scholars/{scholar_profile_id}/image",
headers=headers,
)
assert clear_response.status_code == 200
clear_data = clear_response.json()["data"]
assert clear_data["profile_image_source"] == "scraped"
assert clear_data["profile_image_url"] == "https://images.example.com/ada.png"
finally:
app.dependency_overrides.pop(get_scholar_source, None)
object.__setattr__(settings, "scholar_image_upload_dir", previous_upload_dir)
object.__setattr__(
settings,
"scholar_image_upload_max_bytes",
previous_upload_max_bytes,
)
object.__setattr__(settings, "ingestion_continuation_queue_enabled", previous_queue_enabled)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_scholar_import_export_round_trip(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-import-export@example.com",
password="api-password",
)
scholar_result = await db_session.execute(
text(
"""
INSERT INTO scholar_profiles (user_id, scholar_id, display_name, is_enabled)
VALUES (:user_id, :scholar_id, :display_name, true)
RETURNING id
"""
),
{
"user_id": user_id,
"scholar_id": "abcDEF123456",
"display_name": "Existing Scholar",
},
)
scholar_profile_id = int(scholar_result.scalar_one())
publication_result = await db_session.execute(
text(
"""
INSERT INTO publications (fingerprint_sha256, title_raw, title_normalized, citation_count)
VALUES (:fingerprint, :title_raw, :title_normalized, 1)
RETURNING id
"""
),
{
"fingerprint": f"{(user_id + 500):064x}",
"title_raw": "Existing Publication",
"title_normalized": "existingpublication",
},
)
publication_id = int(publication_result.scalar_one())
await db_session.execute(
text(
"""
INSERT INTO scholar_publications (scholar_profile_id, publication_id, is_read)
VALUES (:scholar_profile_id, :publication_id, false)
"""
),
{
"scholar_profile_id": scholar_profile_id,
"publication_id": publication_id,
},
)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-import-export@example.com", password="api-password")
headers = api_csrf_headers(client)
export_response = client.get("/api/v1/scholars/export")
assert export_response.status_code == 200
export_payload = export_response.json()["data"]
assert export_payload["schema_version"] == 1
assert len(export_payload["scholars"]) == 1
assert len(export_payload["publications"]) == 1
import_response = client.post(
"/api/v1/scholars/import",
json={
"schema_version": 1,
"scholars": [
{
"scholar_id": "abcDEF123456",
"display_name": "Updated Scholar",
"is_enabled": False,
"profile_image_override_url": "https://cdn.example.com/avatar.png",
},
{
"scholar_id": "zzzYYY111222",
"display_name": "Imported Scholar",
"is_enabled": True,
"profile_image_override_url": None,
},
],
"publications": [
{
"scholar_id": "abcDEF123456",
"title": "Existing Publication",
"year": 2024,
"citation_count": 8,
"author_text": "A. Author",
"venue_text": "Test Venue",
"pub_url": "https://example.org/existing",
"pdf_url": "https://example.org/existing.pdf",
"is_read": True,
},
{
"scholar_id": "zzzYYY111222",
"title": "Imported Publication",
"year": 2025,
"citation_count": 2,
"author_text": "B. Author",
"venue_text": "Another Venue",
"pub_url": "https://example.org/imported",
"pdf_url": "https://example.org/imported.pdf",
"is_read": False,
},
],
},
headers=headers,
)
assert import_response.status_code == 200
import_data = import_response.json()["data"]
assert int(import_data["scholars_created"]) == 1
assert int(import_data["scholars_updated"]) >= 1
assert int(import_data["publications_created"]) == 1
assert int(import_data["links_created"]) == 1
updated_scholar_result = await db_session.execute(
text(
"""
SELECT display_name, is_enabled, profile_image_override_url
FROM scholar_profiles
WHERE user_id = :user_id AND scholar_id = :scholar_id
"""
),
{
"user_id": user_id,
"scholar_id": "abcDEF123456",
},
)
updated_scholar = updated_scholar_result.one()
assert updated_scholar[0] == "Updated Scholar"
assert updated_scholar[1] is False
assert updated_scholar[2] == "https://cdn.example.com/avatar.png"
imported_pub_result = await db_session.execute(
text(
"""
SELECT p.title_raw, p.pdf_url
FROM publications p
WHERE p.title_raw = :title
"""
),
{"title": "Imported Publication"},
)
imported_pub = imported_pub_result.one()
assert imported_pub[0] == "Imported Publication"
assert imported_pub[1] == "https://example.org/imported.pdf"
updated_link_result = await db_session.execute(
text(
"""
SELECT sp.is_read
FROM scholar_publications sp
JOIN scholar_profiles s ON s.id = sp.scholar_profile_id
JOIN publications p ON p.id = sp.publication_id
WHERE s.user_id = :user_id
AND s.scholar_id = :scholar_id
AND p.title_raw = :title
"""
),
{
"user_id": user_id,
"scholar_id": "abcDEF123456",
"title": "Existing Publication",
},
)
assert bool(updated_link_result.scalar_one()) is True

183
tests/integration/test_api_settings.py Normal file
View file

@ -0,0 +1,183 @@
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import pytest
from fastapi.testclient import TestClient
from sqlalchemy.ext.asyncio import AsyncSession
from app.main import app
from app.services.settings import application as user_settings_service
from app.settings import settings
from tests.integration.helpers import (
api_csrf_headers,
assert_safety_state_contract,
insert_user,
login_user,
)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_settings_get_and_update(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-settings@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-settings@example.com", password="api-password")
headers = api_csrf_headers(client)
get_response = client.get("/api/v1/settings")
assert get_response.status_code == 200
settings_payload = get_response.json()["data"]
assert "request_delay_seconds" in settings_payload
assert "nav_visible_pages" in settings_payload
assert settings_payload["policy"]["min_run_interval_minutes"] == user_settings_service.resolve_run_interval_minimum(
settings.ingestion_min_run_interval_minutes
)
assert settings_payload["policy"][
"min_request_delay_seconds"
] == user_settings_service.resolve_request_delay_minimum(settings.ingestion_min_request_delay_seconds)
assert settings_payload["policy"]["automation_allowed"] is settings.ingestion_automation_allowed
assert settings_payload["policy"]["manual_run_allowed"] is settings.ingestion_manual_run_allowed
assert settings_payload["policy"]["blocked_failure_threshold"] == max(
1,
int(settings.ingestion_alert_blocked_failure_threshold),
)
assert settings_payload["policy"]["network_failure_threshold"] == max(
1,
int(settings.ingestion_alert_network_failure_threshold),
)
assert settings_payload["policy"]["cooldown_blocked_seconds"] == max(
60,
int(settings.ingestion_safety_cooldown_blocked_seconds),
)
assert settings_payload["policy"]["cooldown_network_seconds"] == max(
60,
int(settings.ingestion_safety_cooldown_network_seconds),
)
assert_safety_state_contract(settings_payload["safety_state"])
assert settings_payload["safety_state"]["cooldown_active"] is False
policy = settings_payload["policy"]
run_interval_minutes = max(45, int(policy["min_run_interval_minutes"]))
request_delay_seconds = max(6, int(policy["min_request_delay_seconds"]))
update_response = client.put(
"/api/v1/settings",
json={
"auto_run_enabled": True,
"run_interval_minutes": run_interval_minutes,
"request_delay_seconds": request_delay_seconds,
"nav_visible_pages": ["dashboard", "scholars", "publications", "settings", "runs"],
},
headers=headers,
)
assert update_response.status_code == 200
updated = update_response.json()["data"]
assert updated["auto_run_enabled"] is True
assert updated["run_interval_minutes"] == run_interval_minutes
assert updated["request_delay_seconds"] == request_delay_seconds
assert updated["nav_visible_pages"] == [
"dashboard",
"scholars",
"publications",
"settings",
"runs",
]
assert "policy" in updated
assert_safety_state_contract(updated["safety_state"])
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_settings_enforce_env_minimums(db_session: AsyncSession) -> None:
await insert_user(
db_session,
email="api-settings-policy@example.com",
password="api-password",
)
client = TestClient(app)
login_user(client, email="api-settings-policy@example.com", password="api-password")
headers = api_csrf_headers(client)
previous_min_interval = settings.ingestion_min_run_interval_minutes
previous_min_delay = settings.ingestion_min_request_delay_seconds
object.__setattr__(settings, "ingestion_min_run_interval_minutes", 30)
object.__setattr__(settings, "ingestion_min_request_delay_seconds", 8)
try:
interval_response = client.put(
"/api/v1/settings",
json={
"auto_run_enabled": True,
"run_interval_minutes": 29,
"request_delay_seconds": 9,
"nav_visible_pages": ["dashboard", "scholars", "settings"],
},
headers=headers,
)
assert interval_response.status_code == 400
assert interval_response.json()["error"]["code"] == "invalid_settings"
assert interval_response.json()["error"]["message"] == "Check interval must be at least 30 minutes."
delay_response = client.put(
"/api/v1/settings",
json={
"auto_run_enabled": True,
"run_interval_minutes": 30,
"request_delay_seconds": 7,
"nav_visible_pages": ["dashboard", "scholars", "settings"],
},
headers=headers,
)
assert delay_response.status_code == 400
assert delay_response.json()["error"]["code"] == "invalid_settings"
assert delay_response.json()["error"]["message"] == "Request delay must be at least 8 seconds."
finally:
object.__setattr__(settings, "ingestion_min_run_interval_minutes", previous_min_interval)
object.__setattr__(settings, "ingestion_min_request_delay_seconds", previous_min_delay)
@pytest.mark.integration
@pytest.mark.db
@pytest.mark.asyncio
async def test_api_settings_clears_expired_scrape_safety_cooldown(
db_session: AsyncSession,
) -> None:
user_id = await insert_user(
db_session,
email="api-settings-safety-expired@example.com",
password="api-password",
)
user_settings = await user_settings_service.get_or_create_settings(
db_session,
user_id=user_id,
)
user_settings.scrape_safety_state = {"blocked_start_count": 2}
user_settings.scrape_cooldown_reason = "blocked_failure_threshold_exceeded"
user_settings.scrape_cooldown_until = datetime.now(UTC) - timedelta(seconds=15)
await db_session.commit()
client = TestClient(app)
login_user(client, email="api-settings-safety-expired@example.com", password="api-password")
settings_response = client.get("/api/v1/settings")
assert settings_response.status_code == 200
settings_safety_state = settings_response.json()["data"]["safety_state"]
assert_safety_state_contract(settings_safety_state)
assert settings_safety_state["cooldown_active"] is False
assert settings_safety_state["cooldown_reason"] is None
assert int(settings_safety_state["counters"]["blocked_start_count"]) == 2
runs_response = client.get("/api/v1/runs")
assert runs_response.status_code == 200
runs_safety_state = runs_response.json()["data"]["safety_state"]
assert_safety_state_contract(runs_safety_state)
assert runs_safety_state["cooldown_active"] is False
assert runs_safety_state["cooldown_reason"] is None

2721
tests/integration/test_api_v1.py
View file

File diff suppressed because it is too large Load diff

2
tests/integration/test_fixture_probe_runs.py
View file

@ -95,7 +95,7 @@ async def test_fixture_probe_run_emits_failure_and_retry_summary_metrics(
"retry": "RSTUVWX12345",
}
for label in ("ok", "blocked", "retry"):
for label in ("ok", "retry", "blocked"):
await db_session.execute(
text(
"""

29
tests/integration/test_run_lifecycle_consistency.py
View file

@ -13,9 +13,29 @@ from app.api.runtime_deps import get_ingestion_service
from app.db.models import CrawlRun, Publication, RunStatus, RunTriggerType
from app.main import app
from app.services.ingestion.application import ScholarIngestionService
from app.services.ingestion.run_enrichment import background_enrich
from app.services.scholar.source import FetchResult
from tests.integration.helpers import insert_user, login_user
class _PassthroughSource:
"""Minimal source that passes preflight without making real requests."""
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
return FetchResult(
requested_url=f"https://scholar.google.com/citations?user={scholar_id}",
status_code=200,
final_url=f"https://scholar.google.com/citations?user={scholar_id}",
body="<html></html>",
error=None,
)
async def fetch_profile_page_html(
self, scholar_id: str, *, cstart: int, pagesize: int
) -> FetchResult:
return await self.fetch_profile_html(scholar_id)
def _csrf_headers(client: TestClient) -> dict[str, str]:
response = client.get("/api/v1/auth/me")
assert response.status_code == 200
@ -81,7 +101,7 @@ async def test_api_manual_run_conflicts_when_an_active_run_exists(
)
await db_session.commit()
service = ScholarIngestionService(source=object())
service = ScholarIngestionService(source=_PassthroughSource())
async def _stall_execute_run(**_kwargs: Any) -> None:
await asyncio.sleep(0.4)
@ -218,9 +238,12 @@ async def test_background_enrichment_preserves_canceled_status(
_OpenAlexClientStub,
)
service = ScholarIngestionService(source=object())
await service._background_enrich(
from app.services.ingestion.enrichment import EnrichmentRunner
enrichment_runner = EnrichmentRunner()
await background_enrich(
session_factory,
enrichment_runner,
run_id=int(run.id),
intended_final_status=RunStatus.SUCCESS,
)

40
tests/unit/test_logging_policy.py Normal file
View file

@ -0,0 +1,40 @@
from __future__ import annotations
import ast
from pathlib import Path
RAW_LOG_METHODS = {
"debug",
"info",
"warning",
"error",
"exception",
"critical",
}
APP_DIR = Path(__file__).resolve().parents[2] / "app"
def _raw_logger_calls(path: Path) -> list[tuple[int, str]]:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
violations: list[tuple[int, str]] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call):
continue
func = node.func
if not isinstance(func, ast.Attribute):
continue
if not isinstance(func.value, ast.Name) or func.value.id != "logger":
continue
if func.attr not in RAW_LOG_METHODS:
continue
violations.append((int(node.lineno), func.attr))
return violations
def test_app_runtime_uses_structured_log_only() -> None:
violations: list[str] = []
for path in sorted(APP_DIR.rglob("*.py")):
for line_number, method in _raw_logger_calls(path):
relative_path = path.relative_to(APP_DIR.parent)
violations.append(f"{relative_path}:{line_number} logger.{method}()")
assert not violations, "raw logger calls found:\n" + "\n".join(violations)

121
tests/unit/test_portability_import.py Normal file
View file

@ -0,0 +1,121 @@
from __future__ import annotations
from unittest.mock import MagicMock
from app.services.portability.publication_import import (
_build_imported_publication_input,
_initialize_import_counters,
_update_link_counters,
)
def _mock_profile(scholar_id: str = "ABC123DEF456") -> MagicMock:
profile = MagicMock()
profile.id = 1
profile.scholar_id = scholar_id
return profile
def _scholar_map(scholar_id: str = "ABC123DEF456") -> dict[str, MagicMock]:
return {scholar_id: _mock_profile(scholar_id)}
class TestBuildImportedPublicationInput:
def test_returns_parsed_input_for_valid_item(self) -> None:
item = {
"scholar_id": "ABC123DEF456",
"title": "Deep Learning",
"year": 2024,
"author_text": "Smith, J",
"venue_text": "ICML",
"citation_count": 10,
}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.title == "Deep Learning"
assert result.year == 2024
assert result.citation_count == 10
assert result.author_text == "Smith, J"
def test_returns_none_when_title_missing(self) -> None:
item = {"scholar_id": "ABC123DEF456"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is None
def test_returns_none_when_scholar_id_missing(self) -> None:
item = {"title": "Test Title"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is None
def test_returns_none_when_scholar_not_in_map(self) -> None:
item = {"scholar_id": "UNKNOWN12345", "title": "Test Title"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is None
def test_defaults_is_read_to_false(self) -> None:
item = {"scholar_id": "ABC123DEF456", "title": "Test Title"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.is_read is False
def test_respects_is_read_true(self) -> None:
item = {"scholar_id": "ABC123DEF456", "title": "Test Title", "is_read": True}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.is_read is True
def test_normalizes_year(self) -> None:
item = {"scholar_id": "ABC123DEF456", "title": "Test Title", "year": "invalid"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.year is None
def test_normalizes_citation_count(self) -> None:
item = {"scholar_id": "ABC123DEF456", "title": "Test Title", "citation_count": "not_a_number"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.citation_count == 0
def test_computes_fingerprint_when_not_provided(self) -> None:
item = {"scholar_id": "ABC123DEF456", "title": "Test Title"}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert len(result.fingerprint) == 64
def test_uses_provided_valid_fingerprint(self) -> None:
valid_sha = "b" * 64
item = {"scholar_id": "ABC123DEF456", "title": "Test Title", "fingerprint_sha256": valid_sha}
result = _build_imported_publication_input(item=item, scholar_map=_scholar_map())
assert result is not None
assert result.fingerprint == valid_sha
class TestInitializeImportCounters:
def test_adds_publication_and_link_keys(self) -> None:
counters: dict[str, int] = {"existing_key": 5}
_initialize_import_counters(counters)
assert counters["publications_created"] == 0
assert counters["publications_updated"] == 0
assert counters["links_created"] == 0
assert counters["links_updated"] == 0
assert counters["existing_key"] == 5
class TestUpdateLinkCounters:
def test_increments_created(self) -> None:
counters = {"links_created": 0, "links_updated": 0}
_update_link_counters(counters=counters, link_created=True, link_updated=False)
assert counters["links_created"] == 1
assert counters["links_updated"] == 0
def test_increments_updated(self) -> None:
counters = {"links_created": 0, "links_updated": 0}
_update_link_counters(counters=counters, link_created=False, link_updated=True)
assert counters["links_created"] == 0
assert counters["links_updated"] == 1
def test_no_change_when_both_false(self) -> None:
counters = {"links_created": 0, "links_updated": 0}
_update_link_counters(counters=counters, link_created=False, link_updated=False)
assert counters["links_created"] == 0
assert counters["links_updated"] == 0

193
tests/unit/test_portability_normalize.py Normal file
View file

@ -0,0 +1,193 @@
from __future__ import annotations
import pytest
from app.services.portability.constants import MAX_IMPORT_PUBLICATIONS, MAX_IMPORT_SCHOLARS
from app.services.portability.normalize import (
_build_fingerprint,
_first_author_last_name,
_first_venue_word,
_normalize_citation_count,
_normalize_optional_text,
_normalize_optional_year,
_resolve_fingerprint,
_validate_import_sizes,
)
from app.services.portability.types import ImportExportError
class TestNormalizeOptionalText:
def test_returns_stripped_string(self) -> None:
assert _normalize_optional_text(" hello ") == "hello"
def test_returns_none_for_none(self) -> None:
assert _normalize_optional_text(None) is None
def test_returns_none_for_empty_string(self) -> None:
assert _normalize_optional_text("") is None
def test_returns_none_for_whitespace_only(self) -> None:
assert _normalize_optional_text(" ") is None
def test_coerces_non_string_to_string(self) -> None:
assert _normalize_optional_text(42) == "42"
class TestNormalizeOptionalYear:
def test_valid_year(self) -> None:
assert _normalize_optional_year(2024) == 2024
def test_string_year(self) -> None:
assert _normalize_optional_year("1999") == 1999
def test_returns_none_for_none(self) -> None:
assert _normalize_optional_year(None) is None
def test_returns_none_for_non_numeric(self) -> None:
assert _normalize_optional_year("abc") is None
def test_returns_none_for_year_below_1500(self) -> None:
assert _normalize_optional_year(1499) is None
def test_returns_none_for_year_above_3000(self) -> None:
assert _normalize_optional_year(3001) is None
def test_boundary_1500_accepted(self) -> None:
assert _normalize_optional_year(1500) == 1500
def test_boundary_3000_accepted(self) -> None:
assert _normalize_optional_year(3000) == 3000
class TestNormalizeCitationCount:
def test_valid_positive(self) -> None:
assert _normalize_citation_count(10) == 10
def test_zero(self) -> None:
assert _normalize_citation_count(0) == 0
def test_negative_clamped_to_zero(self) -> None:
assert _normalize_citation_count(-5) == 0
def test_string_number(self) -> None:
assert _normalize_citation_count("42") == 42
def test_none_returns_zero(self) -> None:
assert _normalize_citation_count(None) == 0
def test_invalid_string_returns_zero(self) -> None:
assert _normalize_citation_count("abc") == 0
class TestFirstAuthorLastName:
def test_single_author(self) -> None:
assert _first_author_last_name("John Smith") == "smith"
def test_multiple_authors(self) -> None:
assert _first_author_last_name("Jane Doe, John Smith") == "doe"
def test_empty_string(self) -> None:
assert _first_author_last_name("") == ""
def test_none(self) -> None:
assert _first_author_last_name(None) == ""
class TestFirstVenueWord:
def test_returns_first_word(self) -> None:
assert _first_venue_word("Nature Communications") == "nature"
def test_empty_string(self) -> None:
assert _first_venue_word("") == ""
def test_none(self) -> None:
assert _first_venue_word(None) == ""
class TestBuildFingerprint:
def test_produces_64_hex_digest(self) -> None:
fp = _build_fingerprint(title="Test Title", year=2024, author_text="Smith", venue_text="ICML")
assert len(fp) == 64
assert all(c in "0123456789abcdef" for c in fp)
def test_deterministic(self) -> None:
kwargs = {"title": "Test Title", "year": 2024, "author_text": "Smith", "venue_text": "ICML"}
assert _build_fingerprint(**kwargs) == _build_fingerprint(**kwargs)
def test_different_titles_produce_different_fingerprints(self) -> None:
fp1 = _build_fingerprint(title="Title A", year=2024, author_text=None, venue_text=None)
fp2 = _build_fingerprint(title="Title B", year=2024, author_text=None, venue_text=None)
assert fp1 != fp2
def test_none_year_handled(self) -> None:
fp = _build_fingerprint(title="Test", year=None, author_text=None, venue_text=None)
assert len(fp) == 64
class TestResolveFingerprint:
def test_uses_provided_valid_sha256(self) -> None:
valid_sha = "a" * 64
result = _resolve_fingerprint(
title="Test",
year=2024,
author_text=None,
venue_text=None,
provided_fingerprint=valid_sha,
)
assert result == valid_sha
def test_computes_fingerprint_when_provided_is_none(self) -> None:
result = _resolve_fingerprint(
title="Test",
year=2024,
author_text=None,
venue_text=None,
provided_fingerprint=None,
)
assert len(result) == 64
def test_computes_fingerprint_when_provided_is_invalid(self) -> None:
result = _resolve_fingerprint(
title="Test",
year=2024,
author_text=None,
venue_text=None,
provided_fingerprint="not-a-sha",
)
assert len(result) == 64
def test_normalizes_provided_to_lowercase(self) -> None:
valid_sha = "A" * 64
result = _resolve_fingerprint(
title="Test",
year=2024,
author_text=None,
venue_text=None,
provided_fingerprint=valid_sha,
)
assert result == "a" * 64
class TestValidateImportSizes:
def test_accepts_within_limits(self) -> None:
_validate_import_sizes(scholars=[{}], publications=[{}])
def test_rejects_too_many_scholars(self) -> None:
with pytest.raises(ImportExportError, match="max scholars"):
_validate_import_sizes(
scholars=[{}] * (MAX_IMPORT_SCHOLARS + 1),
publications=[],
)
def test_rejects_too_many_publications(self) -> None:
with pytest.raises(ImportExportError, match="max publications"):
_validate_import_sizes(
scholars=[],
publications=[{}] * (MAX_IMPORT_PUBLICATIONS + 1),
)
def test_accepts_exact_limit(self) -> None:
_validate_import_sizes(
scholars=[{}] * MAX_IMPORT_SCHOLARS,
publications=[{}] * MAX_IMPORT_PUBLICATIONS,
)

142
tests/unit/test_scholar_validators.py Normal file
View file

@ -0,0 +1,142 @@
from __future__ import annotations
import os
import tempfile
import pytest
from app.services.scholars.constants import MAX_IMAGE_URL_LENGTH
from app.services.scholars.exceptions import ScholarServiceError
from app.services.scholars.uploads import (
_ensure_upload_root,
_resolve_upload_path,
_safe_remove_upload,
resolve_upload_file_path,
)
from app.services.scholars.validators import (
normalize_display_name,
normalize_profile_image_url,
validate_scholar_id,
)
class TestValidateScholarId:
def test_valid_12_char_id(self) -> None:
assert validate_scholar_id("ABCDEF123456") == "ABCDEF123456"
def test_valid_with_hyphens_and_underscores(self) -> None:
assert validate_scholar_id("AB-CD_EF1234") == "AB-CD_EF1234"
def test_strips_whitespace(self) -> None:
assert validate_scholar_id(" ABCDEF123456 ") == "ABCDEF123456"
def test_rejects_too_short(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABC123")
def test_rejects_too_long(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF1234567")
def test_rejects_special_characters(self) -> None:
with pytest.raises(ScholarServiceError, match="12"):
validate_scholar_id("ABCDEF12345!")
class TestNormalizeDisplayName:
def test_returns_stripped_name(self) -> None:
assert normalize_display_name(" John Doe ") == "John Doe"
def test_returns_none_for_empty(self) -> None:
assert normalize_display_name("") is None
def test_returns_none_for_whitespace_only(self) -> None:
assert normalize_display_name(" ") is None
def test_preserves_non_empty(self) -> None:
assert normalize_display_name("A") == "A"
class TestNormalizeProfileImageUrl:
def test_valid_https_url(self) -> None:
assert normalize_profile_image_url("https://example.com/img.png") == "https://example.com/img.png"
def test_valid_http_url(self) -> None:
assert normalize_profile_image_url("http://example.com/img.png") == "http://example.com/img.png"
def test_returns_none_for_none(self) -> None:
assert normalize_profile_image_url(None) is None
def test_returns_none_for_empty(self) -> None:
assert normalize_profile_image_url("") is None
def test_returns_none_for_whitespace(self) -> None:
assert normalize_profile_image_url(" ") is None
def test_rejects_ftp_scheme(self) -> None:
with pytest.raises(ScholarServiceError, match="http"):
normalize_profile_image_url("ftp://example.com/img.png")
def test_rejects_no_scheme(self) -> None:
with pytest.raises(ScholarServiceError, match="http"):
normalize_profile_image_url("example.com/img.png")
def test_rejects_url_too_long(self) -> None:
long_url = "https://example.com/" + "a" * MAX_IMAGE_URL_LENGTH
with pytest.raises(ScholarServiceError, match="characters or fewer"):
normalize_profile_image_url(long_url)
def test_accepts_url_at_max_length(self) -> None:
url = "https://example.com/" + "a" * (MAX_IMAGE_URL_LENGTH - len("https://example.com/"))
assert len(url) == MAX_IMAGE_URL_LENGTH
assert normalize_profile_image_url(url) == url
class TestUploadPathSafety:
def test_resolve_upload_path_rejects_traversal(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
with pytest.raises(ScholarServiceError, match="Invalid"):
_resolve_upload_path(root, "../../../etc/passwd")
def test_resolve_upload_path_accepts_valid_relative(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
result = _resolve_upload_path(root, "scholar_img.png")
assert result.name == "scholar_img.png"
assert root in result.parents or root == result.parent
def test_ensure_upload_root_creates_directory(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
new_dir = os.path.join(tmpdir, "uploads", "images")
root = _ensure_upload_root(new_dir, create=True)
assert root.exists()
def test_safe_remove_upload_removes_existing_file(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
test_file = root / "test.png"
test_file.write_bytes(b"fake image")
assert test_file.exists()
_safe_remove_upload(root, "test.png")
assert not test_file.exists()
def test_safe_remove_upload_ignores_missing_file(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
_safe_remove_upload(root, "nonexistent.png")
def test_safe_remove_upload_ignores_none(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
_safe_remove_upload(root, None)
def test_safe_remove_upload_ignores_traversal_path(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
root = _ensure_upload_root(tmpdir, create=False)
_safe_remove_upload(root, "../../../etc/passwd")
def test_resolve_upload_file_path(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
result = resolve_upload_file_path(upload_dir=tmpdir, relative_path="img.png")
assert result.name == "img.png"

79
tests/unit/test_user_validation.py Normal file
View file

@ -0,0 +1,79 @@
from __future__ import annotations
import pytest
from app.services.users.application import (
UserServiceError,
normalize_email,
validate_email,
validate_password,
)
class TestNormalizeEmail:
def test_lowercases_and_strips(self) -> None:
assert normalize_email(" Alice@Example.COM ") == "alice@example.com"
def test_already_normalized(self) -> None:
assert normalize_email("user@example.com") == "user@example.com"
class TestValidateEmail:
def test_valid_email(self) -> None:
assert validate_email("user@example.com") == "user@example.com"
def test_strips_and_lowercases(self) -> None:
assert validate_email(" User@Example.COM ") == "user@example.com"
def test_rejects_missing_at(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email("userexample.com")
def test_rejects_missing_domain(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email("user@")
def test_rejects_missing_tld(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email("user@example")
def test_rejects_empty_string(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email("")
def test_rejects_whitespace_only(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email(" ")
def test_rejects_spaces_in_local_part(self) -> None:
with pytest.raises(UserServiceError, match="valid email"):
validate_email("us er@example.com")
def test_accepts_plus_addressing(self) -> None:
assert validate_email("user+tag@example.com") == "user+tag@example.com"
def test_accepts_dots_in_local(self) -> None:
assert validate_email("first.last@example.com") == "first.last@example.com"
class TestValidatePassword:
def test_valid_password(self) -> None:
assert validate_password("securepass") == "securepass"
def test_exactly_8_characters(self) -> None:
assert validate_password("12345678") == "12345678"
def test_rejects_7_characters(self) -> None:
with pytest.raises(UserServiceError, match="at least 8"):
validate_password("1234567")
def test_rejects_empty(self) -> None:
with pytest.raises(UserServiceError, match="at least 8"):
validate_password("")
def test_strips_whitespace(self) -> None:
assert validate_password(" securepass ") == "securepass"
def test_whitespace_only_too_short(self) -> None:
with pytest.raises(UserServiceError, match="at least 8"):
validate_password(" ")