628 lines
24 KiB
Markdown
628 lines
24 KiB
Markdown
# Scholarr Modernization Plan — Slices 2-10
|
|
|
|
> **Slice 1 is complete** (commit `7736cab`): removed 114 stale `build/` files, fixed `.gitignore`, committed pending deletions, replaced `npm install` with `npm ci` in CI.
|
|
>
|
|
> Each slice below is a self-contained prompt designed for an LLM executor. Execute in order — each slice depends on the prior ones being complete.
|
|
|
|
---
|
|
|
|
## Slice 2: Create `structured_log()` Utility
|
|
|
|
**Why:** The codebase has 19 `_log_*` helper functions (422 lines) and 138 instances of duplicated `"event"` keys in logging calls. A single utility eliminates all of it.
|
|
|
|
**Files to create:**
|
|
- `app/logging_utils.py`
|
|
|
|
**Files to modify:**
|
|
- `tests/unit/test_logging.py`
|
|
|
|
**Context files (read but don't modify):**
|
|
- `app/logging_config.py` — line 88 has `"event": getattr(record, "event", record.getMessage())` which means when no `event` key is in `extra`, it falls back to `getMessage()`. This is the mechanism that makes `structured_log()` work without duplicating the event name.
|
|
- `app/logging_context.py`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. The current logging pattern has a DRY problem:
|
|
|
|
1. Every log call duplicates the event name as both the message and in extra["event"]:
|
|
`logger.info("event.name", extra={"event": "event.name", "key": value})`
|
|
|
|
2. There are 19 `_log_*` helper functions (422 lines total) that just wrap logger calls with typed signatures to build extra dicts.
|
|
|
|
3. Metrics data (metric_name/metric_value) is mixed into log extra dicts but has no consumer.
|
|
|
|
Create `app/logging_utils.py`:
|
|
|
|
```python
|
|
"""Structured logging utility — eliminates boilerplate across domain services."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
from typing import Any
|
|
|
|
|
|
def structured_log(
|
|
logger: logging.Logger,
|
|
level: str,
|
|
event: str,
|
|
/,
|
|
**fields: Any,
|
|
) -> None:
|
|
"""Emit a structured log entry.
|
|
|
|
The event name is passed as the log message. The JsonLogFormatter in
|
|
logging_config.py extracts it via record.getMessage() when no explicit
|
|
'event' key exists in extra — so we do NOT duplicate it.
|
|
|
|
Usage:
|
|
structured_log(logger, "info", "ingestion.run_started", user_id=1, scholar_count=5)
|
|
"""
|
|
fields.pop("metric_name", None)
|
|
fields.pop("metric_value", None)
|
|
|
|
log_method = getattr(logger, level.lower())
|
|
log_method(event, extra=fields)
|
|
```
|
|
|
|
Verify that `app/logging_config.py` line 88 already handles this correctly — when no `event` key is in extra, it falls back to getMessage() which returns the event string passed as the first argument. No changes to logging_config.py should be needed.
|
|
|
|
Add tests in `tests/unit/test_logging.py` (append to existing file):
|
|
- Test that `structured_log()` produces a log record where the JsonLogFormatter outputs the event name correctly
|
|
- Test that `structured_log()` works with ConsoleLogFormatter
|
|
- Test that metric_name/metric_value fields are stripped from output
|
|
- Test that extra fields (user_id, scholar_id, etc.) appear in the formatted output
|
|
|
|
Commit message: "refactor: add structured_log utility to eliminate logging boilerplate"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 3: Migrate Ingestion Logging to `structured_log()`
|
|
|
|
**Why:** `app/services/ingestion/application.py` is 3,089 lines. 8 `_log_*` helpers consume ~239 lines. 34 inline logger calls duplicate event names and include dead metric fields. This slice removes ~300 lines.
|
|
|
|
**Files to modify:**
|
|
- `app/services/ingestion/application.py`
|
|
|
|
**Context files:**
|
|
- `app/logging_utils.py` (created in Slice 2)
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. The file `app/services/ingestion/application.py` (3,089 lines) has 8 `_log_*` helper functions consuming ~239 lines, plus 34 inline logger calls with duplicated `"event":` keys and mixed `metric_name`/`metric_value` fields.
|
|
|
|
Migrate ALL logging in this file to use `structured_log()` from `app.logging_utils`.
|
|
|
|
1. Add `from app.logging_utils import structured_log` at the top.
|
|
|
|
2. Delete these 8 helper methods entirely:
|
|
- `_log_request_delay_coercion` (line ~123, 21 lines)
|
|
- `_log_run_started` (line ~310, 36 lines)
|
|
- `_log_scholar_parsed` (line ~383, 28 lines)
|
|
- `_log_alert_thresholds` (line ~1013, 47 lines)
|
|
- `_log_safety_transition` (line ~1093, 42 lines)
|
|
- `_log_run_completed` (line ~1178, 28 lines)
|
|
- `_attempt_log_entry` (line ~1837, 16 lines)
|
|
- `_page_log_entry` (line ~1999, 21 lines)
|
|
|
|
3. Replace every call to these deleted helpers with an inline `structured_log()` call. Example:
|
|
|
|
Before:
|
|
```python
|
|
self._log_run_started(
|
|
user_id=user_id,
|
|
trigger_type=trigger_type,
|
|
scholar_count=scholar_count,
|
|
...
|
|
)
|
|
```
|
|
|
|
After:
|
|
```python
|
|
structured_log(
|
|
logger, "info", "ingestion.run_started",
|
|
user_id=user_id,
|
|
trigger_type=trigger_type.value,
|
|
scholar_count=scholar_count,
|
|
...
|
|
)
|
|
```
|
|
|
|
4. For ALL remaining inline `logger.info/warning/debug/exception()` calls:
|
|
- Replace with `structured_log()` where the call uses `extra={"event": ..., ...}` pattern
|
|
- Do NOT convert `logger.exception()` calls — keep them but remove the duplicated `"event"` key from their extra dict
|
|
- Remove `metric_name` and `metric_value` from all calls
|
|
|
|
5. Do NOT change any business logic. Only logging call sites.
|
|
|
|
Expected outcome: ~300 lines removed. All existing tests must pass unchanged.
|
|
|
|
Commit message: "refactor: migrate ingestion service logging to structured_log"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 4: Migrate All Remaining Services to `structured_log()`
|
|
|
|
**Why:** 11 more `_log_*` helpers remain across 8 files (183 lines total), plus inline logger calls with the same duplication pattern in ~20 files.
|
|
|
|
**Files to modify (all have `_log_*` helpers to delete):**
|
|
- `app/services/ingestion/scheduler.py` — `_log_queue_item_resolved` (line ~518, 21 lines)
|
|
- `app/services/arxiv/rate_limit.py` — `_log_request_scheduled` (199), `_log_request_completed` (216), `_log_cooldown_activated` (235)
|
|
- `app/services/arxiv/client.py` — `_log_cache_event` (283), `_log_request_skipped_for_cooldown` (299)
|
|
- `app/services/publications/pdf_resolution_pipeline.py` — `_log_arxiv_skip` (148)
|
|
- `app/services/unpaywall/application.py` — `_log_resolution_summary` (194)
|
|
- `app/api/routers/publications.py` — `_log_retry_pdf_result` (343)
|
|
- `app/api/routers/settings.py` — `_log_settings_update` (103)
|
|
- `app/main.py` — `_log_startup_build_marker` (53)
|
|
|
|
**Files to modify (inline `extra={"event":...}` pattern only — no helpers to delete):**
|
|
- `app/http/middleware.py`
|
|
- `app/db/session.py`
|
|
- `app/auth/runtime.py`
|
|
- `app/security/csrf.py`
|
|
- `app/api/routers/scholars.py`
|
|
- `app/api/routers/runs.py`
|
|
- `app/api/routers/admin_dbops.py`
|
|
- `app/api/routers/admin.py`
|
|
- `app/api/routers/auth.py`
|
|
- `app/api/routers/publications.py`
|
|
- `app/services/scholars/application.py`
|
|
- `app/services/runs/events.py`
|
|
- `app/services/openalex/client.py`
|
|
- `app/services/openalex/matching.py`
|
|
- `app/services/crossref/application.py`
|
|
- `app/services/publications/dedup.py`
|
|
- `app/services/publications/enrichment.py`
|
|
- `app/services/publications/pdf_queue.py`
|
|
- `app/services/scholar/source.py`
|
|
- `app/services/arxiv/gateway.py`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. Slice 3 migrated `ingestion/application.py` to `structured_log()`. Now do the same for ALL remaining files.
|
|
|
|
1. Delete these `_log_*` helper functions and replace their call sites with inline `structured_log()`:
|
|
|
|
- `app/services/ingestion/scheduler.py:~518` — `_log_queue_item_resolved` (21 lines)
|
|
- `app/services/arxiv/rate_limit.py:~199` — `_log_request_scheduled` (17 lines)
|
|
- `app/services/arxiv/rate_limit.py:~216` — `_log_request_completed` (19 lines)
|
|
- `app/services/arxiv/rate_limit.py:~235` — `_log_cooldown_activated` (13 lines)
|
|
- `app/services/arxiv/client.py:~283` — `_log_cache_event` (16 lines)
|
|
- `app/services/arxiv/client.py:~299` — `_log_request_skipped_for_cooldown` (13 lines)
|
|
- `app/services/publications/pdf_resolution_pipeline.py:~148` — `_log_arxiv_skip` (11 lines)
|
|
- `app/services/unpaywall/application.py:~194` — `_log_resolution_summary` (21 lines)
|
|
- `app/api/routers/publications.py:~343` — `_log_retry_pdf_result` (24 lines)
|
|
- `app/api/routers/settings.py:~103` — `_log_settings_update` (15 lines)
|
|
- `app/main.py:~53` — `_log_startup_build_marker` (13 lines)
|
|
|
|
2. In ALL files under `app/` that have inline `logger.info/warning/debug("event.name", extra={"event": "event.name", ...})` calls:
|
|
- Replace with `structured_log(logger, "level", "event.name", key=value, ...)`
|
|
- Add `from app.logging_utils import structured_log` import
|
|
- Remove `metric_name`/`metric_value` from all calls
|
|
- Keep `logger.exception()` calls but remove the duplicated `"event"` key from their extra dicts
|
|
|
|
3. In `app/services/openalex/client.py`, the lines that log `response.text` (lines ~87, ~137):
|
|
- Truncate: `response.text[:500]`
|
|
|
|
4. Do NOT change any business logic. Only logging call sites.
|
|
|
|
Verify: `grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__` should return 0 matches.
|
|
Verify: `grep -rn 'def _log_' app/ --include="*.py"` should return 0 matches.
|
|
|
|
Commit message: "refactor: migrate all remaining logging to structured_log, remove boilerplate helpers"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 5: Logging Relevance Audit — Reduce Noise, Improve Readability
|
|
|
|
**Why:** Self-hosted users need clear, actionable logs. Current logging has verbose debug noise (per-HTTP-request, per-publication), overly long event names, and no console display optimization.
|
|
|
|
**Files to modify:**
|
|
- `app/services/scholar/source.py`
|
|
- `app/services/ingestion/application.py`
|
|
- `app/logging_config.py`
|
|
- Any files with event names longer than ~40 chars
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. The logging has been migrated to `structured_log()` (slices 2-4). Now audit log RELEVANCE and READABILITY for self-hosted users running this in Docker.
|
|
|
|
1. **Remove redundant debug logs:**
|
|
- `app/services/scholar/source.py`: Remove the 3 debug-level HTTP fetch events (fetch_started, search_fetch_started, publication_fetch_started). Keep only `fetch_succeeded` at DEBUG. The HTTP middleware already logs request.started/completed.
|
|
- `app/services/ingestion/application.py`: Remove per-publication `publication.discovered` and `publication.created` DEBUG logs. The run_completed summary already reports totals.
|
|
|
|
2. **Shorten overly long event names** (search all structured_log calls in app/):
|
|
- `ingestion.request_delay_coerced_to_policy_floor` → `ingestion.delay_coerced`
|
|
- `ingestion.safety_cooldown_cleared` → `ingestion.cooldown_cleared`
|
|
- Any other names longer than ~40 chars — shorten while preserving meaning
|
|
|
|
3. **Improve console readability** in `app/logging_config.py` `ConsoleLogFormatter.format()`:
|
|
- Add a short-name mapping for common context fields in console output ONLY (JSON keeps full names):
|
|
```python
|
|
_CONSOLE_SHORT_KEYS = {
|
|
"user_id": "user",
|
|
"scholar_id": "scholar",
|
|
"crawl_run_id": "run",
|
|
"run_id": "run",
|
|
}
|
|
```
|
|
- Apply in the `for key in sorted(payload.keys())` loop
|
|
|
|
4. Do NOT remove any WARNING or ERROR level logs. Only remove/demote DEBUG/INFO that are redundant.
|
|
5. Do NOT change business logic.
|
|
|
|
Commit message: "refactor: reduce log noise, improve event naming and console readability"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 6: Add Ruff + Mypy to CI
|
|
|
|
**Why:** No Python linting or type checking in CI. Code quality regressions go undetected.
|
|
|
|
**Files to modify:**
|
|
- `pyproject.toml`
|
|
- `.github/workflows/ci.yml`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. Add Python linting and type checking to CI.
|
|
|
|
1. In `pyproject.toml`, add ruff configuration:
|
|
```toml
|
|
[tool.ruff]
|
|
target-version = "py312"
|
|
line-length = 120
|
|
|
|
[tool.ruff.lint]
|
|
select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
|
|
ignore = ["E501"]
|
|
|
|
[tool.ruff.lint.isort]
|
|
known-first-party = ["app"]
|
|
```
|
|
|
|
2. In `pyproject.toml`, add mypy configuration:
|
|
```toml
|
|
[tool.mypy]
|
|
python_version = "3.12"
|
|
ignore_missing_imports = true
|
|
check_untyped_defs = false
|
|
warn_unused_ignores = true
|
|
```
|
|
|
|
3. In `.github/workflows/ci.yml`, add a `lint` job after `repo-hygiene` and before `test`:
|
|
```yaml
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
needs: [repo-hygiene]
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
- uses: astral-sh/setup-uv@v4
|
|
- run: uv sync --extra dev
|
|
- name: Ruff check
|
|
run: uv run ruff check .
|
|
- name: Ruff format check
|
|
run: uv run ruff format --check .
|
|
- name: Mypy
|
|
run: uv run mypy app/ --ignore-missing-imports
|
|
```
|
|
|
|
4. Update the `test` job's `needs` to include `lint`.
|
|
|
|
5. Add `ruff` and `mypy` to dev dependencies in `pyproject.toml` under `[project.optional-dependencies]`.
|
|
|
|
6. Run `ruff check .` locally and fix auto-fixable issues with `ruff check --fix .`. For unfixable issues, add targeted `noqa` comments only if fixing would change behavior.
|
|
|
|
Commit message: "ci: add ruff linting and mypy type checking"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 7: Add CodeQL + Dependabot
|
|
|
|
**Why:** No security scanning. Missed vulnerabilities, no automated dependency updates.
|
|
|
|
**Files to create:**
|
|
- `.github/workflows/codeql.yml`
|
|
- `.github/dependabot.yml`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. Add security scanning.
|
|
|
|
1. Create `.github/workflows/codeql.yml`:
|
|
```yaml
|
|
name: CodeQL
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches: [main]
|
|
schedule:
|
|
- cron: "30 5 * * 1"
|
|
|
|
jobs:
|
|
analyze:
|
|
runs-on: ubuntu-latest
|
|
permissions:
|
|
security-events: write
|
|
contents: read
|
|
strategy:
|
|
matrix:
|
|
language: [python, javascript]
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: github/codeql-action/init@v3
|
|
with:
|
|
languages: ${{ matrix.language }}
|
|
- uses: github/codeql-action/autobuild@v3
|
|
- uses: github/codeql-action/analyze@v3
|
|
```
|
|
|
|
2. Create `.github/dependabot.yml`:
|
|
```yaml
|
|
version: 2
|
|
updates:
|
|
- package-ecosystem: pip
|
|
directory: /
|
|
schedule:
|
|
interval: weekly
|
|
- package-ecosystem: npm
|
|
directory: /frontend
|
|
schedule:
|
|
interval: weekly
|
|
- package-ecosystem: github-actions
|
|
directory: /
|
|
schedule:
|
|
interval: weekly
|
|
```
|
|
|
|
Commit message: "ci: add CodeQL security scanning and Dependabot"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 8: Fix Version Remnants + Adopt Semantic Release
|
|
|
|
**Why:** Version `0.1.0` is hardcoded in 5 places (including `crossref/application.py:294`). `0.0.1` exists in `docs/website/package.json`. No CHANGELOG, no GitHub Releases.
|
|
|
|
**Files to modify:**
|
|
- `pyproject.toml`
|
|
- `app/services/crossref/application.py`
|
|
|
|
**Files to create:**
|
|
- `.github/workflows/release.yml`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. Set up automated versioning with python-semantic-release.
|
|
|
|
1. Fix hardcoded version in `app/services/crossref/application.py` (line ~294):
|
|
```python
|
|
# Before:
|
|
etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
|
|
|
|
# After:
|
|
from importlib.metadata import version as pkg_version
|
|
_APP_VERSION = pkg_version("scholarr")
|
|
# then in function:
|
|
etiquette = Etiquette(settings.app_name, _APP_VERSION, "https://scholarr.local", email)
|
|
```
|
|
|
|
2. In `pyproject.toml`, add semantic-release config:
|
|
```toml
|
|
[tool.semantic_release]
|
|
version_toml = ["pyproject.toml:project.version"]
|
|
version_variables = ["frontend/package.json:version"]
|
|
branch = "main"
|
|
build_command = ""
|
|
commit_message = "chore(release): v{version}"
|
|
```
|
|
|
|
3. Create `.github/workflows/release.yml`:
|
|
```yaml
|
|
name: Release
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
|
|
jobs:
|
|
release:
|
|
runs-on: ubuntu-latest
|
|
if: github.repository == 'JustinZeus/scholarr'
|
|
permissions:
|
|
contents: write
|
|
id-token: write
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
token: ${{ secrets.GITHUB_TOKEN }}
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
- run: pip install python-semantic-release
|
|
- name: Semantic Release
|
|
id: release
|
|
run: semantic-release publish
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
```
|
|
|
|
4. Add `python-semantic-release` to dev dependencies in `pyproject.toml`.
|
|
|
|
Note: `docs/website/package.json` version (`0.0.1`) is addressed in Slice 9 (docs rebuild). `frontend/package-lock.json` and `uv.lock` versions auto-update on next `npm install`/`uv sync`.
|
|
|
|
Commit message: "ci: adopt python-semantic-release, fix hardcoded version strings"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 9: Docs — Delete and Rebuild from Scratch
|
|
|
|
**Why:** Current docs have split authority (two api-contract.md files), orphaned pages, stale version (`0.0.1`), and missing critical docs (testing, deployment, configuration reference). Clean slate is faster than migration.
|
|
|
|
**Files to delete:**
|
|
- `docs/` (entire directory)
|
|
|
|
**Files to create:**
|
|
- Complete new `docs/` tree
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. Delete the entire `docs/` directory and rebuild from scratch.
|
|
|
|
Target IA:
|
|
|
|
docs/
|
|
├── index.md # Landing page: what is scholarr, quick links
|
|
├── user/
|
|
│ ├── overview.md # What scholarr does, key concepts
|
|
│ ├── getting-started.md # Install (docker compose), first run, add first scholar
|
|
│ └── configuration.md # ALL env vars from .env.example, organized by category
|
|
├── developer/
|
|
│ ├── overview.md # Dev quickstart
|
|
│ ├── architecture.md # FastAPI + SQLAlchemy + Vue 3, domain service boundaries
|
|
│ ├── local-development.md # docker-compose.dev.yml, hot reload, running tests
|
|
│ ├── contributing.md # PR process, conventional commits, code standards from agents.md
|
|
│ ├── ingestion.md # Ingestion pipeline: parsing, rate limiting, safety gates
|
|
│ ├── frontend-theme-inventory.md # Theme tokens, Tailwind integration
|
|
│ └── testing.md # Test tiers (unit/integration/smoke), markers, fixtures, how to run
|
|
├── operations/
|
|
│ ├── overview.md # Ops quickstart
|
|
│ ├── deployment.md # Production Docker, scaling, health checks
|
|
│ ├── database-runbook.md # Backup, restore, integrity checks, migration procedures
|
|
│ ├── scrape-safety-runbook.md # Rate limiting, cooldowns, CAPTCHA handling
|
|
│ └── arxiv-runbook.md # ArXiv rate limits, cache, query patterns
|
|
├── reference/
|
|
│ ├── overview.md # Reference index
|
|
│ ├── api.md # CANONICAL: envelope spec + endpoints + DTO contract
|
|
│ ├── environment.md # Env var quick-reference (links to user/configuration.md)
|
|
│ └── changelog.md # Placeholder: "auto-generated by semantic-release"
|
|
└── website/
|
|
├── docusaurus.config.js
|
|
├── sidebars.js # ALL pages listed, no orphans
|
|
├── package.json # version: "0.1.0"
|
|
└── src/css/custom.css
|
|
|
|
Content guidelines:
|
|
- Self-contained, scannable docs (headers, bullets, tables)
|
|
- Frontmatter: `title`, `sidebar_position`
|
|
- `configuration.md`: read `.env.example`, organize every variable with description, type, default, example
|
|
- `testing.md`: pytest markers from pyproject.toml (`integration`, `db`, `migrations`, `schema`, `smoke`), container-based runner from agents.md, fixture org under tests/fixtures/
|
|
- `api.md`: merge old `developer/api-contract.md` (DTO structure) and `reference/api-contract.md` (envelope spec). Include exact envelope shapes:
|
|
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
|
|
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
|
|
- Publication semantics (modes, pagination, identifiers)
|
|
- Scholar portability endpoints
|
|
- `deployment.md`: practical Docker commands from docker-compose.yml
|
|
- `database-runbook.md`: reference scripts/db/ (backup_full.sh, restore_dump.sh, check_integrity.py, repair_publication_links.py)
|
|
|
|
Docusaurus config:
|
|
- url: "https://justinzeus.github.io"
|
|
- baseUrl: "/scholarr/"
|
|
- organizationName: "JustinZeus"
|
|
- projectName: "scholarr"
|
|
- onBrokenLinks: "throw"
|
|
- docs path: ".." (reads from docs/ parent)
|
|
- Exclude: "website/**", "README.md"
|
|
|
|
Commit message: "docs: rebuild documentation from scratch with clean IA"
|
|
```
|
|
|
|
---
|
|
|
|
## Slice 10: Begin `ingestion/application.py` Decomposition
|
|
|
|
**Why:** After logging cleanup (slices 3+5), the file is ~2,700-2,800 lines. Still the largest source file by 3x. This begins decomposition with the safest extract: safety gate logic.
|
|
|
|
**Files to modify:**
|
|
- `app/services/ingestion/application.py`
|
|
|
|
**Files to create:**
|
|
- `app/services/ingestion/safety.py`
|
|
|
|
**Prompt:**
|
|
```
|
|
You are working on the scholarr repository. `app/services/ingestion/application.py` is ~2,700-2,800 lines (after logging cleanup). It contains the entire ingestion orchestration in a single class.
|
|
|
|
Extract the FIRST logical chunk: safety gate logic.
|
|
|
|
1. Read `app/services/ingestion/application.py` fully.
|
|
|
|
2. Identify safety/cooldown methods:
|
|
- `_enforce_safety_gate`
|
|
- `_raise_safety_blocked_start`
|
|
- Cooldown activation/clearing methods
|
|
- Alert threshold evaluation methods
|
|
|
|
3. Create `app/services/ingestion/safety.py`:
|
|
- Move identified methods to standalone functions or a small class
|
|
- Accept explicit parameters (no implicit self.xxx state)
|
|
- Keep same function signatures where possible
|
|
|
|
4. Update `application.py` to import from `safety.py`.
|
|
|
|
5. Run full test suite: `docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/`
|
|
All tests must pass unchanged.
|
|
|
|
Constraints from `agents.md`:
|
|
- Max 50 lines per function
|
|
- No flat files in `app/services/` root — all within `app/services/ingestion/`
|
|
- Fail fast, early returns, guard clauses
|
|
|
|
Commit message: "refactor: extract ingestion safety gate logic to dedicated module"
|
|
```
|
|
|
|
---
|
|
|
|
## Verification (after all slices)
|
|
|
|
Run these checks to confirm everything is clean:
|
|
|
|
```bash
|
|
# 1. No tracked build artifacts
|
|
git ls-files build/ | wc -l # expect: 0
|
|
|
|
# 2. No duplicated event keys in logging
|
|
grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__ | wc -l # expect: 0
|
|
|
|
# 3. No _log_ helper functions remain
|
|
grep -rn 'def _log_' app/ --include="*.py" | wc -l # expect: 0
|
|
|
|
# 4. No stale version strings
|
|
grep -rn '0\.0\.1' . --include="*.py" --include="*.json" --include="*.toml" \
|
|
| grep -v node_modules | grep -v .venv | grep -v uv.lock | wc -l # expect: 0
|
|
|
|
# 5. Docs build
|
|
npm --prefix docs/website run build # expect: success, no broken links
|
|
|
|
# 6. All tests pass
|
|
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/
|
|
```
|
|
|
|
---
|
|
|
|
## Slice Summary
|
|
|
|
| # | Slice | Files Changed | Estimated Impact |
|
|
|---|-------|---------------|------------------|
|
|
| ~~1~~ | ~~Repo hygiene~~ | ~~120 files~~ | ~~-18,380 lines~~ **DONE** |
|
|
| 2 | `structured_log()` utility | 2 files | +50 lines |
|
|
| 3 | Migrate ingestion logging | 1 file | -300 lines |
|
|
| 4 | Migrate remaining logging | ~20 files | -180 lines |
|
|
| 5 | Logging noise/readability | ~5 files | -30 lines, better output |
|
|
| 6 | Ruff + mypy in CI | 2 files | +50 lines config |
|
|
| 7 | CodeQL + Dependabot | 2 files | +40 lines config |
|
|
| 8 | Version fix + semantic-release | 3 files | +40 lines config |
|
|
| 9 | Docs rebuild | full replacement | ~20 new files |
|
|
| 10 | Ingestion decomposition | 2 files | net zero (move) |
|