scholarr/app/services/ingestion
Justin Visser 4620978dce feat(ingestion,frontend): add live scholar progress counter to dashboard
Emit scholar_progress SSE events during two-pass ingestion so the
Activity Monitor shows "X / N visited · Y finished · Z new publications"
in real time instead of static post-run counts.

- scholar_processing: add on_progress callback to _run_first_pass and
  _run_depth_pass; build _emit closure in run_scholar_iteration that
  publishes visited/finished/total via run_events; batch-emit scholars
  finished in first pass before depth pass; skip emission on abort/cancel
- run_status store: add scholarProgress state, reset on stream open and
  reset(); subscribe to scholar_progress SSE events with safe parsing
- DashboardPage: show live counter during run, fall back to static text
  when run is not active or no progress received yet

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 17:36:43 +01:00
..
__init__.py ci: add CodeQL security scanning and Dependabot 2026-02-27 00:05:17 +01:00
application.py Bugfixeds 2026-03-01 22:54:23 +01:00
constants.py Initial release prep 2026-02-27 10:46:05 +01:00
enrichment.py fix: URL-length-aware OpenAlex batching and eager scholar metadata hydration 2026-03-01 23:42:15 +01:00
fingerprints.py Initial release prep 2026-02-27 10:46:05 +01:00
page_fetch.py after audit 2026-03-01 18:14:22 +01:00
pagination.py after audit 2026-03-01 18:14:22 +01:00
preflight.py Better crawling safety with preflight checks and cooldowns 2026-02-27 16:44:54 +01:00
publication_upsert.py Bugfixeds 2026-03-01 22:54:23 +01:00
queue.py Bugfixeds 2026-03-01 22:54:23 +01:00
queue_runner.py fix(db,frontend): reserve DB connections for API and reduce frontend polling 2026-03-02 19:09:06 +01:00
run_completion.py after audit 2026-03-01 18:14:22 +01:00
run_enrichment.py after audit 2026-03-01 18:14:22 +01:00
run_guards.py after audit 2026-03-01 18:14:22 +01:00
safety.py ci: add CodeQL security scanning and Dependabot 2026-02-27 00:05:17 +01:00
scheduler.py fix(db,frontend): reserve DB connections for API and reduce frontend polling 2026-03-02 19:09:06 +01:00
scholar_outcomes.py after audit 2026-03-01 18:14:22 +01:00
scholar_processing.py feat(ingestion,frontend): add live scholar progress counter to dashboard 2026-03-03 17:36:43 +01:00
types.py Initial release prep 2026-02-27 10:46:05 +01:00