scholarr/DECOMPOSITION_SLICES.md

# Scholarr Decomposition Slices

## How to use this file

This file contains self-contained decomposition prompts ("slices") to bring the codebase into compliance with `agents.md` file size rules. Each slice is an independent task for an LLM executor.

**Rules from `agents.md`:**
- File length: 400 lines target, 600 lines hard ceiling
- Slightly exceeding the 600-line ceiling is acceptable when the excess is driven by verbose keyword-argument signatures (named parameters), not by business logic density
- Function length: 50 lines max
- All business logic in `app/services/<domain>/`
- Read `agents.md` fully before starting any slice

**Execution protocol:**
1. Pick the next incomplete slice in order (slices depend on prior ones)
2. Read the prompt section — it contains everything you need
3. Read all source files mentioned before making changes
4. Make the changes described — no business logic changes, only file reorganization
5. Verify: all imports resolve, all tests pass
6. Commit with the specified message

---

## Progress

| # | Slice | Status |
|---|-------|--------|
| 1 | Schema split | **DONE** |
| 2 | Ingestion: pagination + publication upsert | **DONE** |
| 3 | Ingestion: enrichment + scholar processing + run completion | **DONE** |
| 4 | Scholars service + PDF queue | **DONE** |
| 5 | Routers + scheduler | **DONE** |

---

## Slice 1: Split `app/api/schemas.py` into domain packages — DONE

Completed. The 945-line `app/api/schemas.py` was split into `app/api/schemas/` package:

| File | Lines | Contents |
|---|---|---|
| `common.py` | 39 | `ApiMeta`, `ApiErrorData`, `ApiErrorEnvelope`, `MessageData`, `MessageEnvelope` |
| `auth.py` | 73 | `SessionUserData`, `AuthMe*`, `CsrfBootstrap*`, `Login*`, `ChangePasswordRequest` |
| `scholars.py` | 153 | `ScholarItem*`, `ScholarSearch*`, `ScholarExport*`, `DataExport*`, `DataImport*` |
| `publications.py` | 151 | `DisplayIdentifierData`, `PublicationItem*`, `MarkAll*`, `MarkSelected*`, `RetryPublication*`, `TogglePublication*` |
| `runs.py` | 226 | `RunListItem*`, `RunSummary*`, `RunDebug*`, `RunScholarResult*`, `RunDetail*`, `ManualRun*`, `Queue*`, `ScrapeSafety*` |
| `admin.py` | 289 | All `Admin*` models including PDF queue, repair, near-duplicate schemas |
| `settings.py` | 54 | `SettingsPolicy*`, `Settings*`, `SettingsUpdateRequest` |
| `__init__.py` | 7 | Wildcard re-exports — all `from app.api.schemas import X` imports unchanged |

---

## Slice 2: Decompose `app/services/ingestion/application.py` — pagination + publication upsert — DONE

Completed. Extracted three modules from `application.py` (2,955 → 2,084 lines):

| File | Lines | Contents |
|---|---|---|
| `app/services/ingestion/page_fetch.py` | 219 | `PageFetcher` class — single-page fetch, parse-or-layout-error, retry with backoff |
| `app/services/ingestion/pagination.py` | 486 | `PaginationEngine` class — multi-page orchestration, loop state, short-circuit, fingerprint checks |
| `app/services/ingestion/publication_upsert.py` | 239 | Module-level functions — `resolve_publication`, `upsert_profile_publications`, find/create/update helpers |

Also fixed two external import paths (`portability/normalize.py`, `portability/publication_import.py`) that imported `normalize_title`/`build_publication_url` from `application.py` — now correctly sourced from `fingerprints.py`. Updated one integration test to monkeypatch the new module-level function instead of the old instance method.

---

## Slice 3: Decompose `app/services/ingestion/application.py` — enrichment, scholar processing, run completion — DONE

Completed. Extracted four logical chunks from `application.py` (2,085 → 635 lines):

| File | Lines | Contents |
|---|---|---|
| `app/services/ingestion/enrichment.py` | 323 | `EnrichmentRunner` class — OpenAlex enrichment, identifier discovery, dedup sweep |
| `app/services/ingestion/scholar_processing.py` | 614 | `process_scholar`, `run_scholar_iteration`, continuation queue, outcome resolution |
| `app/services/ingestion/run_completion.py` | 417 | `complete_run_for_user`, failure summary, alert summary, safety outcome, progress tracking |

Also updated two tests (`test_ingestion_arxiv_rate_limit.py`, `test_ingestion_progress_reporting.py`) and one integration test (`test_deferred_enrichment.py`) to import from new modules.

**Why:** Three remaining logical chunks. Extracting all three brings `application.py` down to ~500 lines (the orchestration spine).

**Files to create:** `app/services/ingestion/enrichment.py`, `app/services/ingestion/scholar_processing.py`, `app/services/ingestion/run_completion.py`

**Files to modify:** `app/services/ingestion/application.py`

**Prompt:**

You are working on the scholarr repository. Read `agents.md` for project conventions. Continue decomposing `app/services/ingestion/application.py` (slice 2 is complete — `page_fetch.py`, `pagination.py`, and `publication_upsert.py` already extracted). Extract three remaining chunks.

**A. `app/services/ingestion/enrichment.py` (~300 lines)**

Move these methods (post-run OpenAlex enrichment):

- `_run_is_canceled`, `_enrich_pending_publications`, `_discover_identifiers_for_enrichment`
- `_publish_identifier_update_event`, `_enrich_publications_with_openalex`

Create an `EnrichmentRunner` class that receives service dependencies (OpenAlex client, identifier service, dedup service, DB session factory) in its constructor.

**B. `app/services/ingestion/scholar_processing.py` (~400 lines)**

Move these methods (per-scholar outcome resolution):

- `_assert_valid_paged_parse_result`, `_apply_first_page_profile_metadata`, `_build_result_entry`
- `_skipped_no_change_outcome`, `_upsert_publications_outcome`, `_upsert_success_or_exception`
- `_upsert_success`, `_upsert_exception_outcome`, `_parse_failure_outcome`
- `_sync_continuation_queue`, `_process_scholar`, `_process_scholar_inner`
- `_fetch_and_prepare_scholar_result`, `_resolve_scholar_outcome`, `_unexpected_scholar_exception_outcome`

These become methods on a `ScholarProcessor` class or module-level functions.

**C. `app/services/ingestion/run_completion.py` (~300 lines)**

Move these methods (run finalization + alerting):

- `_classify_failure_bucket` (standalone function)
- `_summarize_failures`, `_build_alert_summary`, `_apply_safety_outcome`, `_finalize_run_record`
- `_resolve_run_status`, `_resolve_continuation_queue_target`, `_build_failure_debug_context`
- `_complete_run_for_user`

These become module-level functions accepting explicit parameters.

**After all extractions, `application.py` should contain only:**

- `ScholarIngestionService.__init__` and config helpers
- Run lifecycle: `initialize_run`, `execute_run`, `run_for_user`
- Scholar iteration orchestration: `_run_scholar_iteration`
- Progress tracking: `_result_counters`, `_adjust_progress_counts`
- Safety gate integration, background task management

Target: `application.py` ≤ 500 lines. No new file exceeds 400 lines. All tests must pass unchanged.

Commit message: `refactor: extract enrichment, scholar processing, and run completion from ingestion service`

---

## Slice 4: Decompose `app/services/scholars/application.py` (996 lines) and `app/services/publications/pdf_queue.py` (969 lines) — DONE

Completed. Extracted four modules from two oversized service files:

| File | Lines | Contents |
|---|---|---|
| `app/services/scholars/author_search_cache.py` | 239 | Serialize/deserialize cache entries, cache get/set/prune |
| `app/services/scholars/author_search.py` | 580 | Cooldown, throttle, retry, circuit breaker, `search_author_candidates` |
| `app/services/scholars/application.py` | 221 | Scholar CRUD only (list, create, get, toggle, delete, image ops) |
| `app/services/publications/pdf_queue_queries.py` | 395 | SQL builders, row hydration, listing/counting/pagination, retry item builders |
| `app/services/publications/pdf_queue_resolution.py` | 311 | Task execution, outcome persistence, scheduling (`schedule_rows`) |
| `app/services/publications/pdf_queue.py` | 364 | Dataclasses, constants, enqueueing logic, public API |

Also updated one test file (`test_publication_pdf_queue_policy.py`) to monkeypatch `pdf_queue_resolution` instead of `pdf_queue` for the 4 resolution-related tests, and updated `publications/application.py` and `publication_identifiers/application.py` imports.

**Prompt:**

You are working on the scholarr repository. Read `agents.md` for project conventions. Two service files exceed the 600-line ceiling. Decompose both.

**A. `app/services/scholars/application.py` (996 lines)**

This file mixes scholar CRUD (~350 lines) with the entire author search pipeline (~650 lines). Split them:

Create `app/services/scholars/author_search_cache.py` (~150 lines): Move all `_serialize_*` and `_deserialize_*` functions. Move `_cache_get_author_search_result`, `_cache_set_author_search_result`, `_prune_author_search_cache`. These are pure functions operating on the DB cache table.

Create `app/services/scholars/author_search.py` (~500 lines): Move remaining author search functions — cooldown management, throttle logic, retry wrappers, circuit breaker, lock acquisition, and `search_author_candidates`. Import cache functions from `author_search_cache.py`. Accept DB session and settings as explicit parameters.

Update `application.py`: Keep only scholar CRUD methods. Import and delegate to `search_author_candidates` from `author_search.py`. Target: ≤ 350 lines.

**B. `app/services/publications/pdf_queue.py` (969 lines)**

Three concerns in one file. Split by concern:

Create `app/services/publications/pdf_queue_queries.py` (~330 lines): Move SQL query builders (`_tracked_queue_select_base`, `_tracked_queue_select`, etc.), result hydration (`_queue_item_from_row`, `_hydrated_queue_items`), listing/counting/pagination (`list_pdf_queue_items`, `count_pdf_queue_items`, `list_pdf_queue_page`), retry item builders and missing-PDF candidate queries.

Create `app/services/publications/pdf_queue_resolution.py` (~300 lines): Move task execution (`_mark_attempt_started`, `_failed_outcome`, `_fetch_outcome_for_row`), outcome persistence (`_apply_publication_update`, `_apply_job_outcome`, `_persist_outcome`), scheduling (`_resolve_publication_row`, `_run_resolution_task`, `_register_task`, `_drop_finished_task`, `_schedule_rows`).

Keep in `pdf_queue.py` (~340 lines): dataclasses, constants, enqueueing logic, public API.

No file should exceed 500 lines. All tests must pass unchanged.

Commit message: `refactor: decompose scholars service and pdf_queue into focused modules`

---

## Slice 5: Decompose router files and scheduler — DONE

Completed. Extracted helpers from three oversized files:

| File | Lines | Contents |
|---|---|---|
| `app/api/routers/run_serializers.py` | 238 | `serialize_run`, `serialize_queue_item`, `normalize_scholar_result`, `manual_run_payload_from_run`, `manual_run_success_payload`, value coercion helpers, idempotency key validation |
| `app/api/routers/run_manual.py` | 237 | `load_safety_state`, `raise_manual_runs_disabled`, `reused_manual_run_payload`, `run_ingestion_for_manual`, `recover_integrity_error`, `execute_manual_run`, safety/failure raise helpers |
| `app/api/routers/runs.py` | 440 | Route handlers only + imports |
| `app/api/routers/scholar_helpers.py` | 234 | `serialize_scholar`, `hydrate_scholar_metadata_if_needed`, `enqueue_initial_scrape_job_for_scholar`, `search_kwargs`, `search_response_data`, `require_user_profile`, `read_uploaded_image` |
| `app/api/routers/scholars.py` | 406 | Route handlers only + imports |
| `app/services/ingestion/queue_runner.py` | 379 | `QueueJobRunner` class — drain continuation queue, job lifecycle (drop/retry/reschedule), ingestion dispatch, finalization |
| `app/services/ingestion/scheduler.py` | 312 | `SchedulerService` — auto-run scheduling, `_run_loop`, `_tick_once`, candidate loading, PDF queue drain |

Also updated one test file (`test_scholars_create_hydration.py`) to monkeypatch `scholar_helpers` instead of the `scholars` router for the 4 helper-related tests.

**Files to create:** `app/api/routers/run_serializers.py`, `app/api/routers/run_manual.py`, `app/api/routers/scholar_helpers.py`, `app/services/ingestion/queue_runner.py`

**Files to modify:** `app/api/routers/runs.py` (875 → ~405), `app/api/routers/scholars.py` (616 → ~396), `app/services/ingestion/scheduler.py` (622 → ~280)

**Prompt:**

You are working on the scholarr repository. Read `agents.md` for project conventions. Three files are slightly over the 600-line ceiling. Extract helpers from each.

**A. `app/api/routers/runs.py` (875 lines) → 3 files**

Create `app/api/routers/run_serializers.py` (~250 lines): Move all `_serialize_*`, `_normalize_*`, `_int_value`, `_str_value`, `_bool_value` helper functions. Move `_manual_run_payload_from_run` and `_manual_run_success_payload`.

Create `app/api/routers/run_manual.py` (~220 lines): Move `_load_safety_state`, `_raise_manual_runs_disabled`, `_reused_manual_run_payload`, `_run_ingestion_for_manual`, `_recover_integrity_error`, `_execute_manual_run`, `_raise_manual_blocked_safety`, `_raise_manual_failed`.

Keep in `runs.py` (~405 lines): only route handler functions + imports.

**B. `app/api/routers/scholars.py` (616 lines) → 2 files**

Create `app/api/routers/scholar_helpers.py` (~220 lines): Move `_needs_metadata_hydration`, `_is_create_hydration_rate_limited`, `_auto_enqueue_new_scholar_enabled`, `_enqueue_initial_scrape_job_for_scholar`, `_uploaded_image_media_path`, `_serialize_scholar`, `_hydrate_scholar_metadata_if_needed`, `_search_kwargs`, `_search_response_data`, `_read_uploaded_image`, `_require_user_profile`.

Keep in `scholars.py` (~396 lines): only route handlers.

**C. `app/services/ingestion/scheduler.py` (622 lines) → 2 files**

Create `app/services/ingestion/queue_runner.py` (~350 lines): Move all continuation-queue job processing — `_drain_continuation_queue`, `_drop_queue_job_if_max_attempts`, `_mark_queue_job_retrying`, `_queue_job_has_available_scholar`, all `_reschedule_*` methods, `_run_ingestion_for_queue_job`, `_finalize_queue_job_after_run`, `_run_queue_job`. Create a `QueueJobRunner` class receiving config params in constructor.

Keep in `scheduler.py` (~280 lines): auto-run scheduling, `_run_loop`, `_tick_once`, candidate loading.

All tests must pass unchanged after each extraction.

Commit message: `refactor: extract helpers from oversized router and scheduler files`

---

## Verification (after all slices)

```bash
# No Python file exceeds 600 lines
find app/ -name "*.py" -exec wc -l {} + | awk '$1 > 600 && !/total/ {print "FAIL:", $0}'

# All tests pass
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app python -m pytest

# Linting
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff check .
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app ruff format --check .
```