Commit graph

122 commits

Author SHA1 Message Date
dependabot[bot]
40b72ae420
Merge e583a363ea into f501ea4f94 2026-03-08 00:11:59 +00:00
f501ea4f94 fix(frontend): confirm modal, settings display, race guard, and test coverage
- S1: Replace window.confirm() with AppConfirmModal for delete actions
- S4: Add formatHours helper to avoid ugly decimals in settings hints
- W5: Add generation counter to SettingsAdminPanel to prevent stale
  async results when section changes rapidly
- S7: Remove unused ApiRequestError import from useScholarBulkActions
- S2: Add error-path tests for bulk delete/toggle network failures
- S3: Add CSRF boundary tests for bulk-delete and bulk-toggle endpoints
2026-03-07 18:07:29 +01:00
f8e3098fc3 fix(backend): harden bulk endpoints and redact pdf-queue email for non-admins
- W1: _parse_ids_param now returns 400 on non-integer input instead of 500
- W2: bulk_delete_scholars wraps commit in IntegrityError guard; router
  catches ScholarServiceError → 409
- W3: Move `from typing import Any` to top-level import in application.py
- W4: Redact requested_by_email in PDF queue response for non-admin users
2026-03-07 17:57:37 +01:00
c85fa08b7b fix(frontend): add cooldown tooltips and fix settings tab initial load
Add AppHelpHint tooltips to ScrapeSafetyBadge and ScrapeSafetyBanner
showing rate-limit explanation, cooldown reason, and recommended action.
When no cooldown is active, show a simple "ready" message.

Fix settings tabs not loading on initial navigation by deferring
onMounted load via nextTick, and using flush:'post' on the section
watcher so template refs are attached before load() runs.
2026-03-07 16:50:35 +01:00
39c7eb686c fix(mypy): use CursorResult type annotation for bulk_toggle_scholars 2026-03-07 16:16:55 +01:00
3e933998a0 fix(ci): fix mypy error in bulk_toggle_scholars
Use int(cursor.rowcount) instead of type: ignore comment.
2026-03-07 16:11:25 +01:00
6f3e0eec39 fix(ci): fix ruff format in scholars router
Put each structured_log argument on its own line.
2026-03-07 16:11:25 +01:00
76c4811405 fix(ci): fix import sorting, API contract, and pdf-queue test
- Add missing blank line after imports in application.py (ruff I001)
- Use full path string instead of template literal for export API call
  to avoid false positive in API contract drift check
- Update pdf-queue test to expect 200 for non-admin listing
2026-03-07 16:11:25 +01:00
6742eff773 feat(scholars): add multiselect and bulk actions (delete, toggle, export)
Add Set<number>-based selection state to ScholarsPage with checkboxes
in both table and card views, select-all toggle, and stale-key pruning.

Backend: POST /scholars/bulk-delete, POST /scholars/bulk-toggle, and
GET /scholars/export?ids= filter. Service functions use user-scoped
queries. Structured logging for bulk operations.

Frontend: useScholarBulkActions composable extracts selection and bulk
action logic. AppSelect-based action dropdown with dynamic options.
Delete prompts window.confirm; enable/disable applies immediately.
Export downloads filtered JSON.

Includes integration tests for all bulk endpoints and frontend unit
tests for selection toggle, select-all, and stale pruning.
2026-03-07 16:11:25 +01:00
61ac6f206f fix(test): add Pinia setup and admin user to AdminPdfQueueSection tests
The component uses useAuthStore() which requires an active Pinia
instance, and the "Queue all" button is admin-only (v-if="auth.isAdmin").
2026-03-07 15:51:48 +01:00
afbb25d217 fix(ci): use theme tokens for badges, update pdf-queue test
- Replace raw dark: utility variants with state-* theme tokens
  in ScholarStatusBadges and ScholarSettingsModal
- Update integration test to expect 200 for pdf-queue listing
  (endpoint now uses get_api_current_user, not admin-only)
2026-03-07 15:32:42 +01:00
813da91608 fix(scholars): improve validation, fix delete, add status badges
- Extract scholar ID parsing to dedicated module with robust URL
  handling (trailing slashes, fragments, extra query params, encoded
  characters) and per-token error feedback showing why each invalid
  entry was skipped
- Wrap delete_scholar DB commit in IntegrityError handler so cascade
  failures return a proper API error instead of 500
- Add ScholarStatusBadges component showing Pending/Failed/Partial/OK
  badges on scholar rows based on baseline_completed and last_run_status
- Show Pending badge in ScholarSettingsModal for unscraped scholars
- Surface failed_count and partial_count in dashboard latest run summary
  and recent run list items
- Extend backend validator tests with edge cases (empty, whitespace,
  unicode, URL-as-ID)
- Add integration tests for DELETE 404, DELETE with cascade, and POST
  with corrupted scholar_id
- Add comprehensive frontend tests for parsing logic and component
  behavior
2026-03-07 14:05:39 +01:00
0d955c31dc feat(settings): hours interval, pdf queue for all users, fix users tab loading
- Convert check interval setting from minutes to hours in the UI
- Show PDF queue tab to all authenticated users (not admin-only); requeue
  actions remain admin-only in both backend and frontend
- Fix users tab not loading on first visit by removing the AsyncStateGate
  wrapper that prevented child component refs from being populated before
  onMounted fired

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:53:00 +01:00
a72151cfdc feat(frontend): show run and cancel buttons to all users
Remove the auth.isAdmin guard from the Start check and Cancel check
buttons in the Activity Monitor. Admin-only route links (check history,
diagnostics) are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-07 13:53:00 +01:00
JustinZeus
c6c89879c9
Merge pull request #47 from JustinZeus/feat/live-scholar-progress-counter
fix(ingestion,frontend): show live counter immediately on run start
2026-03-03 18:38:36 +01:00
2c591a0ca3 fix(ingestion,frontend): show live counter immediately on run start
Two issues with the initial implementation:
- Counter only appeared after the first SSE event arrived (scholarProgress null
  kept the v-else static template showing until a scholar completed)
- First events were often dropped because the EventSource hadn't subscribed yet

Fix:
- Backend: emit an initial scholar_progress kickoff event (0/0/N) at the
  start of run_scholar_iteration so the frontend receives total early
- Frontend: gate on isLikelyRunning alone (not scholarProgress); use optional
  chaining with fallbacks so "0 / … visited · 0 finished" shows immediately
  when a run starts, before any SSE events arrive

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 18:37:53 +01:00
JustinZeus
5754bffaa0
Merge pull request #46 from JustinZeus/feat/live-scholar-progress-counter
feat(ingestion,frontend): add live scholar progress counter to dashboard
2026-03-03 17:52:46 +01:00
4620978dce feat(ingestion,frontend): add live scholar progress counter to dashboard
Emit scholar_progress SSE events during two-pass ingestion so the
Activity Monitor shows "X / N visited · Y finished · Z new publications"
in real time instead of static post-run counts.

- scholar_processing: add on_progress callback to _run_first_pass and
  _run_depth_pass; build _emit closure in run_scholar_iteration that
  publishes visited/finished/total via run_events; batch-emit scholars
  finished in first pass before depth pass; skip emission on abort/cancel
- run_status store: add scholarProgress state, reset on stream open and
  reset(); subscribe to scholar_progress SSE events with safe parsing
- DashboardPage: show live counter during run, fall back to static text
  when run is not active or no progress received yet

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 17:36:43 +01:00
JustinZeus
cfb1033110
Merge pull request #45 from JustinZeus/bugfix/broken-import
openalex cooldown
2026-03-03 16:17:39 +01:00
f50b609a36 fix(db,frontend): reserve DB connections for API and reduce frontend polling
Background tasks (ingestion, PDF resolution, scheduler) now acquire an
asyncio.Semaphore before checking out a DB connection, guaranteeing at
least N connections remain available for API request handlers. Removes
duplicate polling loop from PublicationsPage, debounces publication
reload on run-status changes, and increases poll interval from 5s to 15s
since SSE handles live discovery.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 19:09:06 +01:00
a490e14126 openalex cooldown 2026-03-02 18:01:42 +01:00
JustinZeus
8ba3b3d41a
Merge pull request #44 from JustinZeus/bugfix/broken-import
added test for new publication detection
2026-03-02 13:59:07 +01:00
4c82fe4b8e added test for new publication detection 2026-03-02 13:58:50 +01:00
JustinZeus
92ad054b97
Merge pull request #43 from JustinZeus/bugfix/broken-import
fix: resolve mypy errors, fix arxiv test session isolation, and add p…
2026-03-02 12:46:35 +01:00
87aa89b58c fix: resolve mypy errors, fix arxiv test session isolation, and add premerge script
- Remove unused type:ignore comment in scholars.py
- Fix 51 mypy errors across test files (Any return types, cast(), protocol stubs)
- Fix arxiv unit test failures caused by session isolation (patch_session_factory fixture)
- Fix integration test rate-limit bleed between search and create
- Add scripts/premerge.sh for local CI verification
- Add docs/website/.docusaurus/ to .gitignore

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:46:10 +01:00
JustinZeus
acf23a7d6c
Merge pull request #42 from JustinZeus/bugfix/broken-import
fixed import
2026-03-02 00:03:55 +01:00
49cf7034b2 2026-03-02 00:03:27 +01:00
JustinZeus
4d949e8657
Merge pull request #41 from JustinZeus/openalex-size-reduction-scholar-hydration
fix: URL-length-aware OpenAlex batching and eager scholar metadata hy…
2026-03-01 23:44:12 +01:00
d9be57a6c1 fix: URL-length-aware OpenAlex batching and eager scholar metadata hydration
- Replace fixed batch_size=25 with URL-byte-length-aware chunking for
  OpenAlex title.search queries. Long/Unicode titles caused URLs to
  exceed server limits, resulting in 500 errors.
- Always hydrate scholar profile metadata (name, image) on creation,
  even when an initial scrape job is queued, so the UI shows profile
  info immediately.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:42:15 +01:00
JustinZeus
e2fe34946c
Merge pull request #40 from JustinZeus/mypi-fix
fix: align test assertion with rollback behavior
2026-03-01 23:18:05 +01:00
e4e9b94cfe fix: align test assertion with rollback behavior
The upsert rolls back the entire transaction on partial failure,
so new_pub_count and link count should be consistent (both 0),
not assuming the first publication survived.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:16:07 +01:00
JustinZeus
76469e2d1e
Merge pull request #39 from JustinZeus/mypi-fix
" "
2026-03-01 23:12:26 +01:00
83e0407491 " " 2026-03-01 23:11:52 +01:00
JustinZeus
9401aa9b69
Merge pull request #38 from JustinZeus/mypi-fix
db tets fix
2026-03-01 23:08:15 +01:00
c7d170a202 db tets fix 2026-03-01 23:07:49 +01:00
JustinZeus
6e17a65b2f
Merge pull request #37 from JustinZeus/mypi-fix
Mypi fix
2026-03-01 23:04:55 +01:00
6fe4814442 " " 2026-03-01 23:04:23 +01:00
0ee89136ff fix: handle optional cookie field for mypy type check
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:04:02 +01:00
JustinZeus
84faaad5eb
Merge pull request #36 from JustinZeus/ci-fixes
Ci fixes
2026-03-01 23:02:03 +01:00
f649c07e10 ci fixed 2026-03-01 23:01:06 +01:00
79f3e41471 fix: resolve CI lint and docs build failures
Apply ruff formatting to 3 files and add missing package-lock.json for docs/website.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:00:31 +01:00
dependabot[bot]
e583a363ea
chore(deps): bump actions/setup-python from 5 to 6
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 5 to 6.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](https://github.com/actions/setup-python/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-01 21:56:35 +00:00
JustinZeus
33f9579169
Merge pull request #19 from JustinZeus/feat/decomposition
Feat/decomposition
2026-03-01 22:55:32 +01:00
4403c9ebde Bugfixeds 2026-03-01 22:54:23 +01:00
a5165dc3ba after audit 2026-03-01 18:14:22 +01:00
e8e20637e6 Better crawling safety with preflight checks and cooldowns 2026-02-27 16:44:54 +01:00
839593e1b3 2026-02-27 16:25:51 +01:00
c0e66c509f style: fix ruff lint and format issues across service modules
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:24:36 +01:00
72bc046215 refactor: extract helpers from remaining long functions
- Extract _start_manual_run and _spawn_background_execution from run_manual route
- Extract _upsert_page_publications to deduplicate pagination loop
- Extract _classify_attempt from fetch_and_parse_with_retry
- Extract _log_alert_threshold_warnings from complete_run_for_user
- Functions 3f-3j accepted as kwargs-exception (body ≤50 lines excluding signatures)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:21:49 +01:00
2a6e573a52 refactor: extract helpers from long functions in ingestion core
- Extract _load_unenriched_publications, _enrich_batch, _flush_and_sweep_duplicates from enrich_pending_publications
- Extract _sanitize_titles shared helper for title cleaning
- Extract _run_iteration_and_complete and _inline_enrich_and_finalize from run_for_user
- Extract _run_first_pass and _run_depth_pass from run_scholar_iteration
- Extract _finalize_successful_queue_job and _finalize_failed_queue_job from _finalize_queue_job_after_run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:18:21 +01:00