refactor: extract pagination engine and publication upsert from ingestion service
Split app/services/ingestion/application.py (2,955 lines) into three focused modules: - page_fetch.py (219 lines): PageFetcher class for single-page fetch, parse-or-layout-error handling, and retry with backoff - pagination.py (486 lines): PaginationEngine class for multi-page orchestration, loop state management, and short-circuit detection - publication_upsert.py (239 lines): module-level functions for resolve_publication, upsert_profile_publications, and all find/create/update helpers Also fixes two external import paths in portability/ that incorrectly sourced normalize_title and build_publication_url from application.py instead of fingerprints.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
4741a2da50
commit
9a7fa5004e
8 changed files with 1169 additions and 885 deletions
|
|
@ -3,7 +3,7 @@ from __future__ import annotations
|
|||
import hashlib
|
||||
from typing import Any
|
||||
|
||||
from app.services.ingestion.application import normalize_title
|
||||
from app.services.ingestion.fingerprints import normalize_title
|
||||
from app.services.portability.constants import (
|
||||
MAX_IMPORT_PUBLICATIONS,
|
||||
MAX_IMPORT_SCHOLARS,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
|
||||
from app.db.models import Publication, ScholarProfile, ScholarPublication
|
||||
from app.services.doi.normalize import normalize_doi
|
||||
from app.services.ingestion.application import build_publication_url, normalize_title
|
||||
from app.services.ingestion.fingerprints import build_publication_url, normalize_title
|
||||
from app.services.portability.normalize import (
|
||||
_normalize_citation_count,
|
||||
_normalize_optional_text,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue