scholarr/MODERNIZATION_PLAN.md

Scholarr Modernization Plan — Slices 2-10

Slice 1 is complete (commit 7736cab): removed 114 stale build/ files, fixed .gitignore, committed pending deletions, replaced npm install with npm ci in CI.

Each slice below is a self-contained prompt designed for an LLM executor. Execute in order — each slice depends on the prior ones being complete.

---

Slice 2: Create structured_log() Utility

Why: The codebase has 19 _log_* helper functions (422 lines) and 138 instances of duplicated "event" keys in logging calls. A single utility eliminates all of it.

Files to create:

• app/logging_utils.py

Files to modify:

• tests/unit/test_logging.py

Context files (read but don't modify):

• app/logging_config.py — line 88 has "event": getattr(record, "event", record.getMessage()) which means when no event key is in extra, it falls back to getMessage(). This is the mechanism that makes structured_log() work without duplicating the event name.
• app/logging_context.py

Prompt:

You are working on the scholarr repository. The current logging pattern has a DRY problem:

1. Every log call duplicates the event name as both the message and in extra["event"]:
   `logger.info("event.name", extra={"event": "event.name", "key": value})`

2. There are 19 `_log_*` helper functions (422 lines total) that just wrap logger calls with typed signatures to build extra dicts.

3. Metrics data (metric_name/metric_value) is mixed into log extra dicts but has no consumer.

Create `app/logging_utils.py`:

```python
"""Structured logging utility — eliminates boilerplate across domain services."""

from __future__ import annotations

import logging
from typing import Any


def structured_log(
    logger: logging.Logger,
    level: str,
    event: str,
    /,
    **fields: Any,
) -> None:
    """Emit a structured log entry.

    The event name is passed as the log message. The JsonLogFormatter in
    logging_config.py extracts it via record.getMessage() when no explicit
    'event' key exists in extra — so we do NOT duplicate it.

    Usage:
        structured_log(logger, "info", "ingestion.run_started", user_id=1, scholar_count=5)
    """
    fields.pop("metric_name", None)
    fields.pop("metric_value", None)

    log_method = getattr(logger, level.lower())
    log_method(event, extra=fields)

Verify that app/logging_config.py line 88 already handles this correctly — when no event key is in extra, it falls back to getMessage() which returns the event string passed as the first argument. No changes to logging_config.py should be needed.

Add tests in tests/unit/test_logging.py (append to existing file):

• Test that structured_log() produces a log record where the JsonLogFormatter outputs the event name correctly
• Test that structured_log() works with ConsoleLogFormatter
• Test that metric_name/metric_value fields are stripped from output
• Test that extra fields (user_id, scholar_id, etc.) appear in the formatted output

Commit message: "refactor: add structured_log utility to eliminate logging boilerplate"

---

## Slice 3: Migrate Ingestion Logging to `structured_log()`

**Why:** `app/services/ingestion/application.py` is 3,089 lines. 8 `_log_*` helpers consume ~239 lines. 34 inline logger calls duplicate event names and include dead metric fields. This slice removes ~300 lines.

**Files to modify:**
- `app/services/ingestion/application.py`

**Context files:**
- `app/logging_utils.py` (created in Slice 2)

**Prompt:**

You are working on the scholarr repository. The file app/services/ingestion/application.py (3,089 lines) has 8 _log_* helper functions consuming ~239 lines, plus 34 inline logger calls with duplicated "event": keys and mixed metric_name/metric_value fields.

Migrate ALL logging in this file to use structured_log() from app.logging_utils.

1. Add from app.logging_utils import structured_log at the top.

2. Delete these 8 helper methods entirely:

   • _log_request_delay_coercion (line ~123, 21 lines)
   • _log_run_started (line ~310, 36 lines)
   • _log_scholar_parsed (line ~383, 28 lines)
   • _log_alert_thresholds (line ~1013, 47 lines)
   • _log_safety_transition (line ~1093, 42 lines)
   • _log_run_completed (line ~1178, 28 lines)
   • _attempt_log_entry (line ~1837, 16 lines)
   • _page_log_entry (line ~1999, 21 lines)

3. Replace every call to these deleted helpers with an inline structured_log() call. Example:

   Before:

   self._log_run_started(
       user_id=user_id,
       trigger_type=trigger_type,
       scholar_count=scholar_count,
       ...
   )
   

   After:

   structured_log(
       logger, "info", "ingestion.run_started",
       user_id=user_id,
       trigger_type=trigger_type.value,
       scholar_count=scholar_count,
       ...
   )
   

4. For ALL remaining inline logger.info/warning/debug/exception() calls:

   • Replace with structured_log() where the call uses extra={"event": ..., ...} pattern
   • Do NOT convert logger.exception() calls — keep them but remove the duplicated "event" key from their extra dict
   • Remove metric_name and metric_value from all calls

5. Do NOT change any business logic. Only logging call sites.

Expected outcome: ~300 lines removed. All existing tests must pass unchanged.

Commit message: "refactor: migrate ingestion service logging to structured_log"

---

## Slice 4: Migrate All Remaining Services to `structured_log()`

**Why:** 11 more `_log_*` helpers remain across 8 files (183 lines total), plus inline logger calls with the same duplication pattern in ~20 files.

**Files to modify (all have `_log_*` helpers to delete):**
- `app/services/ingestion/scheduler.py` — `_log_queue_item_resolved` (line ~518, 21 lines)
- `app/services/arxiv/rate_limit.py` — `_log_request_scheduled` (199), `_log_request_completed` (216), `_log_cooldown_activated` (235)
- `app/services/arxiv/client.py` — `_log_cache_event` (283), `_log_request_skipped_for_cooldown` (299)
- `app/services/publications/pdf_resolution_pipeline.py` — `_log_arxiv_skip` (148)
- `app/services/unpaywall/application.py` — `_log_resolution_summary` (194)
- `app/api/routers/publications.py` — `_log_retry_pdf_result` (343)
- `app/api/routers/settings.py` — `_log_settings_update` (103)
- `app/main.py` — `_log_startup_build_marker` (53)

**Files to modify (inline `extra={"event":...}` pattern only — no helpers to delete):**
- `app/http/middleware.py`
- `app/db/session.py`
- `app/auth/runtime.py`
- `app/security/csrf.py`
- `app/api/routers/scholars.py`
- `app/api/routers/runs.py`
- `app/api/routers/admin_dbops.py`
- `app/api/routers/admin.py`
- `app/api/routers/auth.py`
- `app/api/routers/publications.py`
- `app/services/scholars/application.py`
- `app/services/runs/events.py`
- `app/services/openalex/client.py`
- `app/services/openalex/matching.py`
- `app/services/crossref/application.py`
- `app/services/publications/dedup.py`
- `app/services/publications/enrichment.py`
- `app/services/publications/pdf_queue.py`
- `app/services/scholar/source.py`
- `app/services/arxiv/gateway.py`

**Prompt:**

You are working on the scholarr repository. Slice 3 migrated ingestion/application.py to structured_log(). Now do the same for ALL remaining files.

1. Delete these _log_* helper functions and replace their call sites with inline structured_log():

   • app/services/ingestion/scheduler.py:~518 — _log_queue_item_resolved (21 lines)
   • app/services/arxiv/rate_limit.py:~199 — _log_request_scheduled (17 lines)
   • app/services/arxiv/rate_limit.py:~216 — _log_request_completed (19 lines)
   • app/services/arxiv/rate_limit.py:~235 — _log_cooldown_activated (13 lines)
   • app/services/arxiv/client.py:~283 — _log_cache_event (16 lines)
   • app/services/arxiv/client.py:~299 — _log_request_skipped_for_cooldown (13 lines)
   • app/services/publications/pdf_resolution_pipeline.py:~148 — _log_arxiv_skip (11 lines)
   • app/services/unpaywall/application.py:~194 — _log_resolution_summary (21 lines)
   • app/api/routers/publications.py:~343 — _log_retry_pdf_result (24 lines)
   • app/api/routers/settings.py:~103 — _log_settings_update (15 lines)
   • app/main.py:~53 — _log_startup_build_marker (13 lines)

2. In ALL files under app/ that have inline logger.info/warning/debug("event.name", extra={"event": "event.name", ...}) calls:

   • Replace with structured_log(logger, "level", "event.name", key=value, ...)
   • Add from app.logging_utils import structured_log import
   • Remove metric_name/metric_value from all calls
   • Keep logger.exception() calls but remove the duplicated "event" key from their extra dicts

3. In app/services/openalex/client.py, the lines that log response.text (lines ~87, ~137):

   • Truncate: response.text[:500]

4. Do NOT change any business logic. Only logging call sites.

Verify: grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__ should return 0 matches. Verify: grep -rn 'def _log_' app/ --include="*.py" should return 0 matches.

Commit message: "refactor: migrate all remaining logging to structured_log, remove boilerplate helpers"

---

## Slice 5: Logging Relevance Audit — Reduce Noise, Improve Readability

**Why:** Self-hosted users need clear, actionable logs. Current logging has verbose debug noise (per-HTTP-request, per-publication), overly long event names, and no console display optimization.

**Files to modify:**
- `app/services/scholar/source.py`
- `app/services/ingestion/application.py`
- `app/logging_config.py`
- Any files with event names longer than ~40 chars

**Prompt:**

You are working on the scholarr repository. The logging has been migrated to structured_log() (slices 2-4). Now audit log RELEVANCE and READABILITY for self-hosted users running this in Docker.

1. Remove redundant debug logs:

   • app/services/scholar/source.py: Remove the 3 debug-level HTTP fetch events (fetch_started, search_fetch_started, publication_fetch_started). Keep only fetch_succeeded at DEBUG. The HTTP middleware already logs request.started/completed.
   • app/services/ingestion/application.py: Remove per-publication publication.discovered and publication.created DEBUG logs. The run_completed summary already reports totals.

2. Shorten overly long event names (search all structured_log calls in app/):

   • ingestion.request_delay_coerced_to_policy_floor → ingestion.delay_coerced
   • ingestion.safety_cooldown_cleared → ingestion.cooldown_cleared
   • Any other names longer than ~40 chars — shorten while preserving meaning

3. Improve console readability in app/logging_config.py ConsoleLogFormatter.format():

   • Add a short-name mapping for common context fields in console output ONLY (JSON keeps full names):

     _CONSOLE_SHORT_KEYS = {
         "user_id": "user",
         "scholar_id": "scholar",
         "crawl_run_id": "run",
         "run_id": "run",
     }
     

   • Apply in the for key in sorted(payload.keys()) loop

4. Do NOT remove any WARNING or ERROR level logs. Only remove/demote DEBUG/INFO that are redundant.

5. Do NOT change business logic.

Commit message: "refactor: reduce log noise, improve event naming and console readability"

---

## Slice 6: Add Ruff + Mypy to CI

**Why:** No Python linting or type checking in CI. Code quality regressions go undetected.

**Files to modify:**
- `pyproject.toml`
- `.github/workflows/ci.yml`

**Prompt:**

You are working on the scholarr repository. Add Python linting and type checking to CI.

1. In pyproject.toml, add ruff configuration:

   [tool.ruff]
   target-version = "py312"
   line-length = 120
   
   [tool.ruff.lint]
   select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
   ignore = ["E501"]
   
   [tool.ruff.lint.isort]
   known-first-party = ["app"]
   

2. In pyproject.toml, add mypy configuration:

   [tool.mypy]
   python_version = "3.12"
   ignore_missing_imports = true
   check_untyped_defs = false
   warn_unused_ignores = true
   

3. In .github/workflows/ci.yml, add a lint job after repo-hygiene and before test:

   lint:
     runs-on: ubuntu-latest
     needs: [repo-hygiene]
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
       - uses: astral-sh/setup-uv@v4
       - run: uv sync --extra dev
       - name: Ruff check
         run: uv run ruff check .
       - name: Ruff format check
         run: uv run ruff format --check .
       - name: Mypy
         run: uv run mypy app/ --ignore-missing-imports
   

4. Update the test job's needs to include lint.

5. Add ruff and mypy to dev dependencies in pyproject.toml under [project.optional-dependencies].

6. Run ruff check . locally and fix auto-fixable issues with ruff check --fix .. For unfixable issues, add targeted noqa comments only if fixing would change behavior.

Commit message: "ci: add ruff linting and mypy type checking"

---

## Slice 7: Add CodeQL + Dependabot

**Why:** No security scanning. Missed vulnerabilities, no automated dependency updates.

**Files to create:**
- `.github/workflows/codeql.yml`
- `.github/dependabot.yml`

**Prompt:**

You are working on the scholarr repository. Add security scanning.

1. Create .github/workflows/codeql.yml:

   name: CodeQL
   
   on:
     push:
       branches: [main]
     pull_request:
       branches: [main]
     schedule:
       - cron: "30 5 * * 1"
   
   jobs:
     analyze:
       runs-on: ubuntu-latest
       permissions:
         security-events: write
         contents: read
       strategy:
         matrix:
           language: [python, javascript]
       steps:
         - uses: actions/checkout@v4
         - uses: github/codeql-action/init@v3
           with:
             languages: ${{ matrix.language }}
         - uses: github/codeql-action/autobuild@v3
         - uses: github/codeql-action/analyze@v3
   

2. Create .github/dependabot.yml:

   version: 2
   updates:
     - package-ecosystem: pip
       directory: /
       schedule:
         interval: weekly
     - package-ecosystem: npm
       directory: /frontend
       schedule:
         interval: weekly
     - package-ecosystem: github-actions
       directory: /
       schedule:
         interval: weekly
   

Commit message: "ci: add CodeQL security scanning and Dependabot"

---

## Slice 8: Fix Version Remnants + Adopt Semantic Release

**Why:** Version `0.1.0` is hardcoded in 5 places (including `crossref/application.py:294`). `0.0.1` exists in `docs/website/package.json`. No CHANGELOG, no GitHub Releases.

**Files to modify:**
- `pyproject.toml`
- `app/services/crossref/application.py`

**Files to create:**
- `.github/workflows/release.yml`

**Prompt:**

You are working on the scholarr repository. Set up automated versioning with python-semantic-release.

1. Fix hardcoded version in app/services/crossref/application.py (line ~294):

   # Before:
   etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
   
   # After:
   from importlib.metadata import version as pkg_version
   _APP_VERSION = pkg_version("scholarr")
   # then in function:
   etiquette = Etiquette(settings.app_name, _APP_VERSION, "https://scholarr.local", email)
   

2. In pyproject.toml, add semantic-release config:

   [tool.semantic_release]
   version_toml = ["pyproject.toml:project.version"]
   version_variables = ["frontend/package.json:version"]
   branch = "main"
   build_command = ""
   commit_message = "chore(release): v{version}"
   

3. Create .github/workflows/release.yml:

   name: Release
   
   on:
     push:
       branches: [main]
   
   jobs:
     release:
       runs-on: ubuntu-latest
       if: github.repository == 'JustinZeus/scholarr'
       permissions:
         contents: write
         id-token: write
       steps:
         - uses: actions/checkout@v4
           with:
             fetch-depth: 0
             token: ${{ secrets.GITHUB_TOKEN }}
         - uses: actions/setup-python@v5
           with:
             python-version: "3.12"
         - run: pip install python-semantic-release
         - name: Semantic Release
           id: release
           run: semantic-release publish
           env:
             GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
   

4. Add python-semantic-release to dev dependencies in pyproject.toml.

Note: docs/website/package.json version (0.0.1) is addressed in Slice 9 (docs rebuild). frontend/package-lock.json and uv.lock versions auto-update on next npm install/uv sync.

Commit message: "ci: adopt python-semantic-release, fix hardcoded version strings"

---

## Slice 9: Docs — Delete and Rebuild from Scratch

**Why:** Current docs have split authority (two api-contract.md files), orphaned pages, stale version (`0.0.1`), and missing critical docs (testing, deployment, configuration reference). Clean slate is faster than migration.

**Files to delete:**
- `docs/` (entire directory)

**Files to create:**
- Complete new `docs/` tree

**Prompt:**

You are working on the scholarr repository. Delete the entire docs/ directory and rebuild from scratch.

Target IA:

docs/ ├── index.md # Landing page: what is scholarr, quick links ├── user/ │ ├── overview.md # What scholarr does, key concepts │ ├── getting-started.md # Install (docker compose), first run, add first scholar │ └── configuration.md # ALL env vars from .env.example, organized by category ├── developer/ │ ├── overview.md # Dev quickstart │ ├── architecture.md # FastAPI + SQLAlchemy + Vue 3, domain service boundaries │ ├── local-development.md # docker-compose.dev.yml, hot reload, running tests │ ├── contributing.md # PR process, conventional commits, code standards from agents.md │ ├── ingestion.md # Ingestion pipeline: parsing, rate limiting, safety gates │ ├── frontend-theme-inventory.md # Theme tokens, Tailwind integration │ └── testing.md # Test tiers (unit/integration/smoke), markers, fixtures, how to run ├── operations/ │ ├── overview.md # Ops quickstart │ ├── deployment.md # Production Docker, scaling, health checks │ ├── database-runbook.md # Backup, restore, integrity checks, migration procedures │ ├── scrape-safety-runbook.md # Rate limiting, cooldowns, CAPTCHA handling │ └── arxiv-runbook.md # ArXiv rate limits, cache, query patterns ├── reference/ │ ├── overview.md # Reference index │ ├── api.md # CANONICAL: envelope spec + endpoints + DTO contract │ ├── environment.md # Env var quick-reference (links to user/configuration.md) │ └── changelog.md # Placeholder: "auto-generated by semantic-release" └── website/ ├── docusaurus.config.js ├── sidebars.js # ALL pages listed, no orphans ├── package.json # version: "0.1.0" └── src/css/custom.css

Content guidelines:

• Self-contained, scannable docs (headers, bullets, tables)
• Frontmatter: title, sidebar_position
• configuration.md: read .env.example, organize every variable with description, type, default, example
• testing.md: pytest markers from pyproject.toml (integration, db, migrations, schema, smoke), container-based runner from agents.md, fixture org under tests/fixtures/
• api.md: merge old developer/api-contract.md (DTO structure) and reference/api-contract.md (envelope spec). Include exact envelope shapes:
  • Success: {"data": ..., "meta": {"request_id": "..."}}
  • Error: {"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}
  • Publication semantics (modes, pagination, identifiers)
  • Scholar portability endpoints
• deployment.md: practical Docker commands from docker-compose.yml
• database-runbook.md: reference scripts/db/ (backup_full.sh, restore_dump.sh, check_integrity.py, repair_publication_links.py)

Docusaurus config:

• url: "https://justinzeus.github.io"
• baseUrl: "/scholarr/"
• organizationName: "JustinZeus"
• projectName: "scholarr"
• onBrokenLinks: "throw"
• docs path: ".." (reads from docs/ parent)
• Exclude: "website/**", "README.md"

Commit message: "docs: rebuild documentation from scratch with clean IA"

---

## Slice 10: Begin `ingestion/application.py` Decomposition

**Why:** After logging cleanup (slices 3+5), the file is ~2,700-2,800 lines. Still the largest source file by 3x. This begins decomposition with the safest extract: safety gate logic.

**Files to modify:**
- `app/services/ingestion/application.py`

**Files to create:**
- `app/services/ingestion/safety.py`

**Prompt:**

You are working on the scholarr repository. app/services/ingestion/application.py is ~2,700-2,800 lines (after logging cleanup). It contains the entire ingestion orchestration in a single class.

Extract the FIRST logical chunk: safety gate logic.

1. Read app/services/ingestion/application.py fully.

2. Identify safety/cooldown methods:

   • _enforce_safety_gate
   • _raise_safety_blocked_start
   • Cooldown activation/clearing methods
   • Alert threshold evaluation methods

3. Create app/services/ingestion/safety.py:

   • Move identified methods to standalone functions or a small class
   • Accept explicit parameters (no implicit self.xxx state)
   • Keep same function signatures where possible

4. Update application.py to import from safety.py.

5. Run full test suite: docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/ All tests must pass unchanged.

Constraints from agents.md:

• Max 50 lines per function
• No flat files in app/services/ root — all within app/services/ingestion/
• Fail fast, early returns, guard clauses

Commit message: "refactor: extract ingestion safety gate logic to dedicated module"

---

## Verification (after all slices)

Run these checks to confirm everything is clean:

```bash
# 1. No tracked build artifacts
git ls-files build/ | wc -l  # expect: 0

# 2. No duplicated event keys in logging
grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__ | wc -l  # expect: 0

# 3. No _log_ helper functions remain
grep -rn 'def _log_' app/ --include="*.py" | wc -l  # expect: 0

# 4. No stale version strings
grep -rn '0\.0\.1' . --include="*.py" --include="*.json" --include="*.toml" \
  | grep -v node_modules | grep -v .venv | grep -v uv.lock | wc -l  # expect: 0

# 5. Docs build
npm --prefix docs/website run build  # expect: success, no broken links

# 6. All tests pass
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/

---

Slice Summary

#	Slice	Files Changed	Estimated Impact
1	Repo hygiene	120 files	-18,380 lines DONE
2	structured_log() utility	2 files	+50 lines
3	Migrate ingestion logging	1 file	-300 lines
4	Migrate remaining logging	~20 files	-180 lines
5	Logging noise/readability	~5 files	-30 lines, better output
6	Ruff + mypy in CI	2 files	+50 lines config
7	CodeQL + Dependabot	2 files	+40 lines config
8	Version fix + semantic-release	3 files	+40 lines config
9	Docs rebuild	full replacement	~20 new files
10	Ingestion decomposition	2 files	net zero (move)