Commit graph

7 commits

Author SHA1 Message Date
2c591a0ca3 fix(ingestion,frontend): show live counter immediately on run start
Two issues with the initial implementation:
- Counter only appeared after the first SSE event arrived (scholarProgress null
  kept the v-else static template showing until a scholar completed)
- First events were often dropped because the EventSource hadn't subscribed yet

Fix:
- Backend: emit an initial scholar_progress kickoff event (0/0/N) at the
  start of run_scholar_iteration so the frontend receives total early
- Frontend: gate on isLikelyRunning alone (not scholarProgress); use optional
  chaining with fallbacks so "0 / … visited · 0 finished" shows immediately
  when a run starts, before any SSE events arrive

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 18:37:53 +01:00
4620978dce feat(ingestion,frontend): add live scholar progress counter to dashboard
Emit scholar_progress SSE events during two-pass ingestion so the
Activity Monitor shows "X / N visited · Y finished · Z new publications"
in real time instead of static post-run counts.

- scholar_processing: add on_progress callback to _run_first_pass and
  _run_depth_pass; build _emit closure in run_scholar_iteration that
  publishes visited/finished/total via run_events; batch-emit scholars
  finished in first pass before depth pass; skip emission on abort/cancel
- run_status store: add scholarProgress state, reset on stream open and
  reset(); subscribe to scholar_progress SSE events with safe parsing
- DashboardPage: show live counter during run, fall back to static text
  when run is not active or no progress received yet

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 17:36:43 +01:00
a5165dc3ba after audit 2026-03-01 18:14:22 +01:00
e8e20637e6 Better crawling safety with preflight checks and cooldowns 2026-02-27 16:44:54 +01:00
2a6e573a52 refactor: extract helpers from long functions in ingestion core
- Extract _load_unenriched_publications, _enrich_batch, _flush_and_sweep_duplicates from enrich_pending_publications
- Extract _sanitize_titles shared helper for title cleaning
- Extract _run_iteration_and_complete and _inline_enrich_and_finalize from run_for_user
- Extract _run_first_pass and _run_depth_pass from run_scholar_iteration
- Extract _finalize_successful_queue_job and _finalize_failed_queue_job from _finalize_queue_job_after_run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 16:18:21 +01:00
74249fa24c fix: resolve all mypy type errors across service modules
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 15:49:27 +01:00
6379c97e90 decompose application.py into enrichment, scholar processing, run completion 2026-02-27 14:19:39 +01:00