justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

refactor: extract pagination engine and publication upsert from ingestion service

Browse source 
Split app/services/ingestion/application.py (2,955 lines) into three
focused modules:

- page_fetch.py (219 lines): PageFetcher class for single-page fetch,
  parse-or-layout-error handling, and retry with backoff
- pagination.py (486 lines): PaginationEngine class for multi-page
  orchestration, loop state management, and short-circuit detection
- publication_upsert.py (239 lines): module-level functions for
  resolve_publication, upsert_profile_publications, and all
  find/create/update helpers

Also fixes two external import paths in portability/ that incorrectly
sourced normalize_title and build_publication_url from application.py
instead of fingerprints.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 13:47:23 +01:00
parent 4741a2da50
commit 9a7fa5004e
8 changed files with 1169 additions and 885 deletions
Download patch file Download diff file Expand all files Collapse all files

7
tests/integration/test_run_lifecycle_consistency.py
View file

@ -326,7 +326,6 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
db_session.add_all([run, publication])
await db_session.commit()
service = ScholarIngestionService(source=object())
call_count = 0
async def _resolve_publication_stub(*_args: Any, **_kwargs: Any) -> Publication:
@ -336,7 +335,9 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
return publication
raise RuntimeError("mid_page_failure")
monkeypatch.setattr(service, "_resolve_publication", _resolve_publication_stub)
from app.services.ingestion import publication_upsert
monkeypatch.setattr(publication_upsert, "resolve_publication", _resolve_publication_stub)
from app.db.models import ScholarProfile
from app.services.scholar.parser_types import PublicationCandidate
@ -368,7 +369,7 @@ async def test_partial_discovery_exception_keeps_new_pub_count_consistent(
]
with pytest.raises(RuntimeError, match="mid_page_failure"):
await service._upsert_profile_publications(
await publication_upsert.upsert_profile_publications(
db_session,
run=run,
scholar=scholar,