ci: add ruff linting and mypy type checking
Add ruff and mypy to dev dependencies with configuration in pyproject.toml. Add a lint CI job that runs ruff check, ruff format --check, and mypy. Auto-fix import sorting and formatting across the codebase. Exclude alembic/versions from linting (auto-generated migrations). Ignore B008 (FastAPI Depends pattern) and RUF001 (unicode in user-facing strings). 21 ruff lint errors and 50 mypy errors remain for manual review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
399276ea69
commit
bf04c77aa9
137 changed files with 3066 additions and 1900 deletions
18
.github/workflows/ci.yml
vendored
18
.github/workflows/ci.yml
vendored
|
|
@ -20,10 +20,28 @@ jobs:
|
|||
- name: Enforce env contract parity
|
||||
run: python3 scripts/check_env_contract.py
|
||||
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [repo-hygiene]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: astral-sh/setup-uv@v4
|
||||
- run: uv sync --extra dev
|
||||
- name: Ruff check
|
||||
run: uv run ruff check .
|
||||
- name: Ruff format check
|
||||
run: uv run ruff format --check .
|
||||
- name: Mypy
|
||||
run: uv run mypy app/ --ignore-missing-imports
|
||||
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- repo-hygiene
|
||||
- lint
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
|
|
|
|||
628
MODERNIZATION_PLAN.md
Normal file
628
MODERNIZATION_PLAN.md
Normal file
|
|
@ -0,0 +1,628 @@
|
|||
# Scholarr Modernization Plan — Slices 2-10
|
||||
|
||||
> **Slice 1 is complete** (commit `7736cab`): removed 114 stale `build/` files, fixed `.gitignore`, committed pending deletions, replaced `npm install` with `npm ci` in CI.
|
||||
>
|
||||
> Each slice below is a self-contained prompt designed for an LLM executor. Execute in order — each slice depends on the prior ones being complete.
|
||||
|
||||
---
|
||||
|
||||
## Slice 2: Create `structured_log()` Utility
|
||||
|
||||
**Why:** The codebase has 19 `_log_*` helper functions (422 lines) and 138 instances of duplicated `"event"` keys in logging calls. A single utility eliminates all of it.
|
||||
|
||||
**Files to create:**
|
||||
- `app/logging_utils.py`
|
||||
|
||||
**Files to modify:**
|
||||
- `tests/unit/test_logging.py`
|
||||
|
||||
**Context files (read but don't modify):**
|
||||
- `app/logging_config.py` — line 88 has `"event": getattr(record, "event", record.getMessage())` which means when no `event` key is in `extra`, it falls back to `getMessage()`. This is the mechanism that makes `structured_log()` work without duplicating the event name.
|
||||
- `app/logging_context.py`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. The current logging pattern has a DRY problem:
|
||||
|
||||
1. Every log call duplicates the event name as both the message and in extra["event"]:
|
||||
`logger.info("event.name", extra={"event": "event.name", "key": value})`
|
||||
|
||||
2. There are 19 `_log_*` helper functions (422 lines total) that just wrap logger calls with typed signatures to build extra dicts.
|
||||
|
||||
3. Metrics data (metric_name/metric_value) is mixed into log extra dicts but has no consumer.
|
||||
|
||||
Create `app/logging_utils.py`:
|
||||
|
||||
```python
|
||||
"""Structured logging utility — eliminates boilerplate across domain services."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
|
||||
def structured_log(
|
||||
logger: logging.Logger,
|
||||
level: str,
|
||||
event: str,
|
||||
/,
|
||||
**fields: Any,
|
||||
) -> None:
|
||||
"""Emit a structured log entry.
|
||||
|
||||
The event name is passed as the log message. The JsonLogFormatter in
|
||||
logging_config.py extracts it via record.getMessage() when no explicit
|
||||
'event' key exists in extra — so we do NOT duplicate it.
|
||||
|
||||
Usage:
|
||||
structured_log(logger, "info", "ingestion.run_started", user_id=1, scholar_count=5)
|
||||
"""
|
||||
fields.pop("metric_name", None)
|
||||
fields.pop("metric_value", None)
|
||||
|
||||
log_method = getattr(logger, level.lower())
|
||||
log_method(event, extra=fields)
|
||||
```
|
||||
|
||||
Verify that `app/logging_config.py` line 88 already handles this correctly — when no `event` key is in extra, it falls back to getMessage() which returns the event string passed as the first argument. No changes to logging_config.py should be needed.
|
||||
|
||||
Add tests in `tests/unit/test_logging.py` (append to existing file):
|
||||
- Test that `structured_log()` produces a log record where the JsonLogFormatter outputs the event name correctly
|
||||
- Test that `structured_log()` works with ConsoleLogFormatter
|
||||
- Test that metric_name/metric_value fields are stripped from output
|
||||
- Test that extra fields (user_id, scholar_id, etc.) appear in the formatted output
|
||||
|
||||
Commit message: "refactor: add structured_log utility to eliminate logging boilerplate"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 3: Migrate Ingestion Logging to `structured_log()`
|
||||
|
||||
**Why:** `app/services/domains/ingestion/application.py` is 3,089 lines. 8 `_log_*` helpers consume ~239 lines. 34 inline logger calls duplicate event names and include dead metric fields. This slice removes ~300 lines.
|
||||
|
||||
**Files to modify:**
|
||||
- `app/services/domains/ingestion/application.py`
|
||||
|
||||
**Context files:**
|
||||
- `app/logging_utils.py` (created in Slice 2)
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. The file `app/services/domains/ingestion/application.py` (3,089 lines) has 8 `_log_*` helper functions consuming ~239 lines, plus 34 inline logger calls with duplicated `"event":` keys and mixed `metric_name`/`metric_value` fields.
|
||||
|
||||
Migrate ALL logging in this file to use `structured_log()` from `app.logging_utils`.
|
||||
|
||||
1. Add `from app.logging_utils import structured_log` at the top.
|
||||
|
||||
2. Delete these 8 helper methods entirely:
|
||||
- `_log_request_delay_coercion` (line ~123, 21 lines)
|
||||
- `_log_run_started` (line ~310, 36 lines)
|
||||
- `_log_scholar_parsed` (line ~383, 28 lines)
|
||||
- `_log_alert_thresholds` (line ~1013, 47 lines)
|
||||
- `_log_safety_transition` (line ~1093, 42 lines)
|
||||
- `_log_run_completed` (line ~1178, 28 lines)
|
||||
- `_attempt_log_entry` (line ~1837, 16 lines)
|
||||
- `_page_log_entry` (line ~1999, 21 lines)
|
||||
|
||||
3. Replace every call to these deleted helpers with an inline `structured_log()` call. Example:
|
||||
|
||||
Before:
|
||||
```python
|
||||
self._log_run_started(
|
||||
user_id=user_id,
|
||||
trigger_type=trigger_type,
|
||||
scholar_count=scholar_count,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
After:
|
||||
```python
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_started",
|
||||
user_id=user_id,
|
||||
trigger_type=trigger_type.value,
|
||||
scholar_count=scholar_count,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
4. For ALL remaining inline `logger.info/warning/debug/exception()` calls:
|
||||
- Replace with `structured_log()` where the call uses `extra={"event": ..., ...}` pattern
|
||||
- Do NOT convert `logger.exception()` calls — keep them but remove the duplicated `"event"` key from their extra dict
|
||||
- Remove `metric_name` and `metric_value` from all calls
|
||||
|
||||
5. Do NOT change any business logic. Only logging call sites.
|
||||
|
||||
Expected outcome: ~300 lines removed. All existing tests must pass unchanged.
|
||||
|
||||
Commit message: "refactor: migrate ingestion service logging to structured_log"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 4: Migrate All Remaining Services to `structured_log()`
|
||||
|
||||
**Why:** 11 more `_log_*` helpers remain across 8 files (183 lines total), plus inline logger calls with the same duplication pattern in ~20 files.
|
||||
|
||||
**Files to modify (all have `_log_*` helpers to delete):**
|
||||
- `app/services/domains/ingestion/scheduler.py` — `_log_queue_item_resolved` (line ~518, 21 lines)
|
||||
- `app/services/domains/arxiv/rate_limit.py` — `_log_request_scheduled` (199), `_log_request_completed` (216), `_log_cooldown_activated` (235)
|
||||
- `app/services/domains/arxiv/client.py` — `_log_cache_event` (283), `_log_request_skipped_for_cooldown` (299)
|
||||
- `app/services/domains/publications/pdf_resolution_pipeline.py` — `_log_arxiv_skip` (148)
|
||||
- `app/services/domains/unpaywall/application.py` — `_log_resolution_summary` (194)
|
||||
- `app/api/routers/publications.py` — `_log_retry_pdf_result` (343)
|
||||
- `app/api/routers/settings.py` — `_log_settings_update` (103)
|
||||
- `app/main.py` — `_log_startup_build_marker` (53)
|
||||
|
||||
**Files to modify (inline `extra={"event":...}` pattern only — no helpers to delete):**
|
||||
- `app/http/middleware.py`
|
||||
- `app/db/session.py`
|
||||
- `app/auth/runtime.py`
|
||||
- `app/security/csrf.py`
|
||||
- `app/api/routers/scholars.py`
|
||||
- `app/api/routers/runs.py`
|
||||
- `app/api/routers/admin_dbops.py`
|
||||
- `app/api/routers/admin.py`
|
||||
- `app/api/routers/auth.py`
|
||||
- `app/api/routers/publications.py`
|
||||
- `app/services/domains/scholars/application.py`
|
||||
- `app/services/domains/runs/events.py`
|
||||
- `app/services/domains/openalex/client.py`
|
||||
- `app/services/domains/openalex/matching.py`
|
||||
- `app/services/domains/crossref/application.py`
|
||||
- `app/services/domains/publications/dedup.py`
|
||||
- `app/services/domains/publications/enrichment.py`
|
||||
- `app/services/domains/publications/pdf_queue.py`
|
||||
- `app/services/domains/scholar/source.py`
|
||||
- `app/services/domains/arxiv/gateway.py`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. Slice 3 migrated `ingestion/application.py` to `structured_log()`. Now do the same for ALL remaining files.
|
||||
|
||||
1. Delete these `_log_*` helper functions and replace their call sites with inline `structured_log()`:
|
||||
|
||||
- `app/services/domains/ingestion/scheduler.py:~518` — `_log_queue_item_resolved` (21 lines)
|
||||
- `app/services/domains/arxiv/rate_limit.py:~199` — `_log_request_scheduled` (17 lines)
|
||||
- `app/services/domains/arxiv/rate_limit.py:~216` — `_log_request_completed` (19 lines)
|
||||
- `app/services/domains/arxiv/rate_limit.py:~235` — `_log_cooldown_activated` (13 lines)
|
||||
- `app/services/domains/arxiv/client.py:~283` — `_log_cache_event` (16 lines)
|
||||
- `app/services/domains/arxiv/client.py:~299` — `_log_request_skipped_for_cooldown` (13 lines)
|
||||
- `app/services/domains/publications/pdf_resolution_pipeline.py:~148` — `_log_arxiv_skip` (11 lines)
|
||||
- `app/services/domains/unpaywall/application.py:~194` — `_log_resolution_summary` (21 lines)
|
||||
- `app/api/routers/publications.py:~343` — `_log_retry_pdf_result` (24 lines)
|
||||
- `app/api/routers/settings.py:~103` — `_log_settings_update` (15 lines)
|
||||
- `app/main.py:~53` — `_log_startup_build_marker` (13 lines)
|
||||
|
||||
2. In ALL files under `app/` that have inline `logger.info/warning/debug("event.name", extra={"event": "event.name", ...})` calls:
|
||||
- Replace with `structured_log(logger, "level", "event.name", key=value, ...)`
|
||||
- Add `from app.logging_utils import structured_log` import
|
||||
- Remove `metric_name`/`metric_value` from all calls
|
||||
- Keep `logger.exception()` calls but remove the duplicated `"event"` key from their extra dicts
|
||||
|
||||
3. In `app/services/domains/openalex/client.py`, the lines that log `response.text` (lines ~87, ~137):
|
||||
- Truncate: `response.text[:500]`
|
||||
|
||||
4. Do NOT change any business logic. Only logging call sites.
|
||||
|
||||
Verify: `grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__` should return 0 matches.
|
||||
Verify: `grep -rn 'def _log_' app/ --include="*.py"` should return 0 matches.
|
||||
|
||||
Commit message: "refactor: migrate all remaining logging to structured_log, remove boilerplate helpers"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 5: Logging Relevance Audit — Reduce Noise, Improve Readability
|
||||
|
||||
**Why:** Self-hosted users need clear, actionable logs. Current logging has verbose debug noise (per-HTTP-request, per-publication), overly long event names, and no console display optimization.
|
||||
|
||||
**Files to modify:**
|
||||
- `app/services/domains/scholar/source.py`
|
||||
- `app/services/domains/ingestion/application.py`
|
||||
- `app/logging_config.py`
|
||||
- Any files with event names longer than ~40 chars
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. The logging has been migrated to `structured_log()` (slices 2-4). Now audit log RELEVANCE and READABILITY for self-hosted users running this in Docker.
|
||||
|
||||
1. **Remove redundant debug logs:**
|
||||
- `app/services/domains/scholar/source.py`: Remove the 3 debug-level HTTP fetch events (fetch_started, search_fetch_started, publication_fetch_started). Keep only `fetch_succeeded` at DEBUG. The HTTP middleware already logs request.started/completed.
|
||||
- `app/services/domains/ingestion/application.py`: Remove per-publication `publication.discovered` and `publication.created` DEBUG logs. The run_completed summary already reports totals.
|
||||
|
||||
2. **Shorten overly long event names** (search all structured_log calls in app/):
|
||||
- `ingestion.request_delay_coerced_to_policy_floor` → `ingestion.delay_coerced`
|
||||
- `ingestion.safety_cooldown_cleared` → `ingestion.cooldown_cleared`
|
||||
- Any other names longer than ~40 chars — shorten while preserving meaning
|
||||
|
||||
3. **Improve console readability** in `app/logging_config.py` `ConsoleLogFormatter.format()`:
|
||||
- Add a short-name mapping for common context fields in console output ONLY (JSON keeps full names):
|
||||
```python
|
||||
_CONSOLE_SHORT_KEYS = {
|
||||
"user_id": "user",
|
||||
"scholar_id": "scholar",
|
||||
"crawl_run_id": "run",
|
||||
"run_id": "run",
|
||||
}
|
||||
```
|
||||
- Apply in the `for key in sorted(payload.keys())` loop
|
||||
|
||||
4. Do NOT remove any WARNING or ERROR level logs. Only remove/demote DEBUG/INFO that are redundant.
|
||||
5. Do NOT change business logic.
|
||||
|
||||
Commit message: "refactor: reduce log noise, improve event naming and console readability"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 6: Add Ruff + Mypy to CI
|
||||
|
||||
**Why:** No Python linting or type checking in CI. Code quality regressions go undetected.
|
||||
|
||||
**Files to modify:**
|
||||
- `pyproject.toml`
|
||||
- `.github/workflows/ci.yml`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. Add Python linting and type checking to CI.
|
||||
|
||||
1. In `pyproject.toml`, add ruff configuration:
|
||||
```toml
|
||||
[tool.ruff]
|
||||
target-version = "py312"
|
||||
line-length = 120
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["E", "F", "W", "I", "UP", "B", "SIM", "RUF"]
|
||||
ignore = ["E501"]
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
known-first-party = ["app"]
|
||||
```
|
||||
|
||||
2. In `pyproject.toml`, add mypy configuration:
|
||||
```toml
|
||||
[tool.mypy]
|
||||
python_version = "3.12"
|
||||
ignore_missing_imports = true
|
||||
check_untyped_defs = false
|
||||
warn_unused_ignores = true
|
||||
```
|
||||
|
||||
3. In `.github/workflows/ci.yml`, add a `lint` job after `repo-hygiene` and before `test`:
|
||||
```yaml
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
needs: [repo-hygiene]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- uses: astral-sh/setup-uv@v4
|
||||
- run: uv sync --extra dev
|
||||
- name: Ruff check
|
||||
run: uv run ruff check .
|
||||
- name: Ruff format check
|
||||
run: uv run ruff format --check .
|
||||
- name: Mypy
|
||||
run: uv run mypy app/ --ignore-missing-imports
|
||||
```
|
||||
|
||||
4. Update the `test` job's `needs` to include `lint`.
|
||||
|
||||
5. Add `ruff` and `mypy` to dev dependencies in `pyproject.toml` under `[project.optional-dependencies]`.
|
||||
|
||||
6. Run `ruff check .` locally and fix auto-fixable issues with `ruff check --fix .`. For unfixable issues, add targeted `noqa` comments only if fixing would change behavior.
|
||||
|
||||
Commit message: "ci: add ruff linting and mypy type checking"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 7: Add CodeQL + Dependabot
|
||||
|
||||
**Why:** No security scanning. Missed vulnerabilities, no automated dependency updates.
|
||||
|
||||
**Files to create:**
|
||||
- `.github/workflows/codeql.yml`
|
||||
- `.github/dependabot.yml`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. Add security scanning.
|
||||
|
||||
1. Create `.github/workflows/codeql.yml`:
|
||||
```yaml
|
||||
name: CodeQL
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
schedule:
|
||||
- cron: "30 5 * * 1"
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
security-events: write
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
language: [python, javascript]
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: github/codeql-action/init@v3
|
||||
with:
|
||||
languages: ${{ matrix.language }}
|
||||
- uses: github/codeql-action/autobuild@v3
|
||||
- uses: github/codeql-action/analyze@v3
|
||||
```
|
||||
|
||||
2. Create `.github/dependabot.yml`:
|
||||
```yaml
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: pip
|
||||
directory: /
|
||||
schedule:
|
||||
interval: weekly
|
||||
- package-ecosystem: npm
|
||||
directory: /frontend
|
||||
schedule:
|
||||
interval: weekly
|
||||
- package-ecosystem: github-actions
|
||||
directory: /
|
||||
schedule:
|
||||
interval: weekly
|
||||
```
|
||||
|
||||
Commit message: "ci: add CodeQL security scanning and Dependabot"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 8: Fix Version Remnants + Adopt Semantic Release
|
||||
|
||||
**Why:** Version `0.1.0` is hardcoded in 5 places (including `crossref/application.py:294`). `0.0.1` exists in `docs/website/package.json`. No CHANGELOG, no GitHub Releases.
|
||||
|
||||
**Files to modify:**
|
||||
- `pyproject.toml`
|
||||
- `app/services/domains/crossref/application.py`
|
||||
|
||||
**Files to create:**
|
||||
- `.github/workflows/release.yml`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. Set up automated versioning with python-semantic-release.
|
||||
|
||||
1. Fix hardcoded version in `app/services/domains/crossref/application.py` (line ~294):
|
||||
```python
|
||||
# Before:
|
||||
etiquette = Etiquette(settings.app_name, "0.1.0", "https://scholarr.local", email)
|
||||
|
||||
# After:
|
||||
from importlib.metadata import version as pkg_version
|
||||
_APP_VERSION = pkg_version("scholarr")
|
||||
# then in function:
|
||||
etiquette = Etiquette(settings.app_name, _APP_VERSION, "https://scholarr.local", email)
|
||||
```
|
||||
|
||||
2. In `pyproject.toml`, add semantic-release config:
|
||||
```toml
|
||||
[tool.semantic_release]
|
||||
version_toml = ["pyproject.toml:project.version"]
|
||||
version_variables = ["frontend/package.json:version"]
|
||||
branch = "main"
|
||||
build_command = ""
|
||||
commit_message = "chore(release): v{version}"
|
||||
```
|
||||
|
||||
3. Create `.github/workflows/release.yml`:
|
||||
```yaml
|
||||
name: Release
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository == 'JustinZeus/scholarr'
|
||||
permissions:
|
||||
contents: write
|
||||
id-token: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- run: pip install python-semantic-release
|
||||
- name: Semantic Release
|
||||
id: release
|
||||
run: semantic-release publish
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
4. Add `python-semantic-release` to dev dependencies in `pyproject.toml`.
|
||||
|
||||
Note: `docs/website/package.json` version (`0.0.1`) is addressed in Slice 9 (docs rebuild). `frontend/package-lock.json` and `uv.lock` versions auto-update on next `npm install`/`uv sync`.
|
||||
|
||||
Commit message: "ci: adopt python-semantic-release, fix hardcoded version strings"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 9: Docs — Delete and Rebuild from Scratch
|
||||
|
||||
**Why:** Current docs have split authority (two api-contract.md files), orphaned pages, stale version (`0.0.1`), and missing critical docs (testing, deployment, configuration reference). Clean slate is faster than migration.
|
||||
|
||||
**Files to delete:**
|
||||
- `docs/` (entire directory)
|
||||
|
||||
**Files to create:**
|
||||
- Complete new `docs/` tree
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. Delete the entire `docs/` directory and rebuild from scratch.
|
||||
|
||||
Target IA:
|
||||
|
||||
docs/
|
||||
├── index.md # Landing page: what is scholarr, quick links
|
||||
├── user/
|
||||
│ ├── overview.md # What scholarr does, key concepts
|
||||
│ ├── getting-started.md # Install (docker compose), first run, add first scholar
|
||||
│ └── configuration.md # ALL env vars from .env.example, organized by category
|
||||
├── developer/
|
||||
│ ├── overview.md # Dev quickstart
|
||||
│ ├── architecture.md # FastAPI + SQLAlchemy + Vue 3, domain service boundaries
|
||||
│ ├── local-development.md # docker-compose.dev.yml, hot reload, running tests
|
||||
│ ├── contributing.md # PR process, conventional commits, code standards from agents.md
|
||||
│ ├── ingestion.md # Ingestion pipeline: parsing, rate limiting, safety gates
|
||||
│ ├── frontend-theme-inventory.md # Theme tokens, Tailwind integration
|
||||
│ └── testing.md # Test tiers (unit/integration/smoke), markers, fixtures, how to run
|
||||
├── operations/
|
||||
│ ├── overview.md # Ops quickstart
|
||||
│ ├── deployment.md # Production Docker, scaling, health checks
|
||||
│ ├── database-runbook.md # Backup, restore, integrity checks, migration procedures
|
||||
│ ├── scrape-safety-runbook.md # Rate limiting, cooldowns, CAPTCHA handling
|
||||
│ └── arxiv-runbook.md # ArXiv rate limits, cache, query patterns
|
||||
├── reference/
|
||||
│ ├── overview.md # Reference index
|
||||
│ ├── api.md # CANONICAL: envelope spec + endpoints + DTO contract
|
||||
│ ├── environment.md # Env var quick-reference (links to user/configuration.md)
|
||||
│ └── changelog.md # Placeholder: "auto-generated by semantic-release"
|
||||
└── website/
|
||||
├── docusaurus.config.js
|
||||
├── sidebars.js # ALL pages listed, no orphans
|
||||
├── package.json # version: "0.1.0"
|
||||
└── src/css/custom.css
|
||||
|
||||
Content guidelines:
|
||||
- Self-contained, scannable docs (headers, bullets, tables)
|
||||
- Frontmatter: `title`, `sidebar_position`
|
||||
- `configuration.md`: read `.env.example`, organize every variable with description, type, default, example
|
||||
- `testing.md`: pytest markers from pyproject.toml (`integration`, `db`, `migrations`, `schema`, `smoke`), container-based runner from agents.md, fixture org under tests/fixtures/
|
||||
- `api.md`: merge old `developer/api-contract.md` (DTO structure) and `reference/api-contract.md` (envelope spec). Include exact envelope shapes:
|
||||
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
|
||||
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
|
||||
- Publication semantics (modes, pagination, identifiers)
|
||||
- Scholar portability endpoints
|
||||
- `deployment.md`: practical Docker commands from docker-compose.yml
|
||||
- `database-runbook.md`: reference scripts/db/ (backup_full.sh, restore_dump.sh, check_integrity.py, repair_publication_links.py)
|
||||
|
||||
Docusaurus config:
|
||||
- url: "https://justinzeus.github.io"
|
||||
- baseUrl: "/scholarr/"
|
||||
- organizationName: "JustinZeus"
|
||||
- projectName: "scholarr"
|
||||
- onBrokenLinks: "throw"
|
||||
- docs path: ".." (reads from docs/ parent)
|
||||
- Exclude: "website/**", "README.md"
|
||||
|
||||
Commit message: "docs: rebuild documentation from scratch with clean IA"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice 10: Begin `ingestion/application.py` Decomposition
|
||||
|
||||
**Why:** After logging cleanup (slices 3+5), the file is ~2,700-2,800 lines. Still the largest source file by 3x. This begins decomposition with the safest extract: safety gate logic.
|
||||
|
||||
**Files to modify:**
|
||||
- `app/services/domains/ingestion/application.py`
|
||||
|
||||
**Files to create:**
|
||||
- `app/services/domains/ingestion/safety.py`
|
||||
|
||||
**Prompt:**
|
||||
```
|
||||
You are working on the scholarr repository. `app/services/domains/ingestion/application.py` is ~2,700-2,800 lines (after logging cleanup). It contains the entire ingestion orchestration in a single class.
|
||||
|
||||
Extract the FIRST logical chunk: safety gate logic.
|
||||
|
||||
1. Read `app/services/domains/ingestion/application.py` fully.
|
||||
|
||||
2. Identify safety/cooldown methods:
|
||||
- `_enforce_safety_gate`
|
||||
- `_raise_safety_blocked_start`
|
||||
- Cooldown activation/clearing methods
|
||||
- Alert threshold evaluation methods
|
||||
|
||||
3. Create `app/services/domains/ingestion/safety.py`:
|
||||
- Move identified methods to standalone functions or a small class
|
||||
- Accept explicit parameters (no implicit self.xxx state)
|
||||
- Keep same function signatures where possible
|
||||
|
||||
4. Update `application.py` to import from `safety.py`.
|
||||
|
||||
5. Run full test suite: `docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/`
|
||||
All tests must pass unchanged.
|
||||
|
||||
Constraints from `agents.md`:
|
||||
- Max 50 lines per function
|
||||
- No flat files in `app/services/` root — all within `app/services/domains/ingestion/`
|
||||
- Fail fast, early returns, guard clauses
|
||||
|
||||
Commit message: "refactor: extract ingestion safety gate logic to dedicated module"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification (after all slices)
|
||||
|
||||
Run these checks to confirm everything is clean:
|
||||
|
||||
```bash
|
||||
# 1. No tracked build artifacts
|
||||
git ls-files build/ | wc -l # expect: 0
|
||||
|
||||
# 2. No duplicated event keys in logging
|
||||
grep -rn '"event":' app/ --include="*.py" | grep -v __pycache__ | wc -l # expect: 0
|
||||
|
||||
# 3. No _log_ helper functions remain
|
||||
grep -rn 'def _log_' app/ --include="*.py" | wc -l # expect: 0
|
||||
|
||||
# 4. No stale version strings
|
||||
grep -rn '0\.0\.1' . --include="*.py" --include="*.json" --include="*.toml" \
|
||||
| grep -v node_modules | grep -v .venv | grep -v uv.lock | wc -l # expect: 0
|
||||
|
||||
# 5. Docs build
|
||||
npm --prefix docs/website run build # expect: success, no broken links
|
||||
|
||||
# 6. All tests pass
|
||||
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app pytest tests/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Slice Summary
|
||||
|
||||
| # | Slice | Files Changed | Estimated Impact |
|
||||
|---|-------|---------------|------------------|
|
||||
| ~~1~~ | ~~Repo hygiene~~ | ~~120 files~~ | ~~-18,380 lines~~ **DONE** |
|
||||
| 2 | `structured_log()` utility | 2 files | +50 lines |
|
||||
| 3 | Migrate ingestion logging | 1 file | -300 lines |
|
||||
| 4 | Migrate remaining logging | ~20 files | -180 lines |
|
||||
| 5 | Logging noise/readability | ~5 files | -30 lines, better output |
|
||||
| 6 | Ruff + mypy in CI | 2 files | +50 lines config |
|
||||
| 7 | CodeQL + Dependabot | 2 files | +40 lines config |
|
||||
| 8 | Version fix + semantic-release | 3 files | +40 lines config |
|
||||
| 9 | Docs rebuild | full replacement | ~20 new files |
|
||||
| 10 | Ingestion decomposition | 2 files | net zero (move) |
|
||||
|
|
@ -1,13 +1,13 @@
|
|||
from logging.config import fileConfig
|
||||
import os
|
||||
from logging.config import fileConfig
|
||||
|
||||
from alembic import context
|
||||
from sqlalchemy import pool
|
||||
from sqlalchemy.engine import Connection
|
||||
from sqlalchemy.ext.asyncio import async_engine_from_config
|
||||
|
||||
from app.db.base import metadata
|
||||
from alembic import context
|
||||
from app.db import models # noqa: F401
|
||||
from app.db.base import metadata
|
||||
|
||||
config = context.config
|
||||
|
||||
|
|
|
|||
|
|
@ -8,10 +8,11 @@ Create Date: 2026-02-16 17:10:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260216_0001"
|
||||
down_revision: str | Sequence[str] | None = None
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-16 17:40:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260216_0002"
|
||||
down_revision: str | Sequence[str] | None = "20260216_0001"
|
||||
|
|
@ -31,4 +32,3 @@ def upgrade() -> None:
|
|||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("users", "is_admin")
|
||||
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-16 23:45:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260216_0003"
|
||||
down_revision: str | Sequence[str] | None = "20260216_0002"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-17 10:15:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260217_0004"
|
||||
down_revision: str | Sequence[str] | None = "20260216_0003"
|
||||
|
|
@ -22,9 +23,7 @@ def upgrade() -> None:
|
|||
inspector = sa.inspect(bind)
|
||||
columns = {column["name"] for column in inspector.get_columns("ingestion_queue_items")}
|
||||
indexes = {index["name"] for index in inspector.get_indexes("ingestion_queue_items")}
|
||||
checks = {
|
||||
constraint["name"] for constraint in inspector.get_check_constraints("ingestion_queue_items")
|
||||
}
|
||||
checks = {constraint["name"] for constraint in inspector.get_check_constraints("ingestion_queue_items")}
|
||||
|
||||
if "status" not in columns:
|
||||
op.add_column(
|
||||
|
|
@ -80,9 +79,7 @@ def downgrade() -> None:
|
|||
inspector = sa.inspect(bind)
|
||||
columns = {column["name"] for column in inspector.get_columns("ingestion_queue_items")}
|
||||
indexes = {index["name"] for index in inspector.get_indexes("ingestion_queue_items")}
|
||||
checks = {
|
||||
constraint["name"] for constraint in inspector.get_check_constraints("ingestion_queue_items")
|
||||
}
|
||||
checks = {constraint["name"] for constraint in inspector.get_check_constraints("ingestion_queue_items")}
|
||||
|
||||
if "ix_ingestion_queue_status_next_attempt" in indexes:
|
||||
op.drop_index("ix_ingestion_queue_status_next_attempt", table_name="ingestion_queue_items")
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-17 16:20:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260217_0005"
|
||||
down_revision: str | Sequence[str] | None = "20260217_0004"
|
||||
|
|
@ -70,9 +71,7 @@ def upgrade() -> None:
|
|||
"crawl_runs",
|
||||
["user_id", "idempotency_key"],
|
||||
unique=True,
|
||||
postgresql_where=sa.text(
|
||||
"idempotency_key IS NOT NULL AND trigger_type = 'manual'::run_trigger_type"
|
||||
),
|
||||
postgresql_where=sa.text("idempotency_key IS NOT NULL AND trigger_type = 'manual'::run_trigger_type"),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-17 18:30:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260217_0006"
|
||||
down_revision: str | Sequence[str] | None = "20260217_0005"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ Create Date: 2026-02-17 18:55:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260217_0007"
|
||||
down_revision: str | Sequence[str] | None = "20260217_0006"
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Create Date: 2026-02-19 14:58:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260219_0008"
|
||||
|
|
@ -21,7 +21,7 @@ depends_on: str | Sequence[str] | None = None
|
|||
|
||||
TABLE_NAME = "user_settings"
|
||||
DEFAULT_NAV_VISIBLE_PAGES_SQL = (
|
||||
"'[\"dashboard\",\"scholars\",\"publications\",\"settings\",\"style-guide\",\"runs\",\"users\"]'::jsonb"
|
||||
'\'["dashboard","scholars","publications","settings","style-guide","runs","users"]\'::jsonb'
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Create Date: 2026-02-19 21:20:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260219_0009"
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Create Date: 2026-02-19 22:20:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260219_0010"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-20 13:35:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "20260220_0011"
|
||||
down_revision: str | Sequence[str] | None = "20260219_0010"
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Create Date: 2026-02-20 23:35:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "20260220_0012"
|
||||
down_revision: str | Sequence[str] | None = "20260220_0011"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-21 12:25:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "20260221_0013"
|
||||
down_revision: str | Sequence[str] | None = "20260220_0012"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-21 14:35:00
|
|||
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "20260221_0014"
|
||||
|
|
|
|||
|
|
@ -9,9 +9,9 @@ from __future__ import annotations
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260221_0015"
|
||||
|
|
@ -30,11 +30,7 @@ LEGACY_CHECK_NAME_MIN_2 = "ck_user_settings_request_delay_seconds_min_2"
|
|||
def _check_constraints() -> set[str]:
|
||||
bind = op.get_bind()
|
||||
inspector = sa.inspect(bind)
|
||||
return {
|
||||
str(item.get("name"))
|
||||
for item in inspector.get_check_constraints(TABLE_NAME)
|
||||
if item.get("name")
|
||||
}
|
||||
return {str(item.get("name")) for item in inspector.get_check_constraints(TABLE_NAME) if item.get("name")}
|
||||
|
||||
|
||||
def _drop_check_if_exists(name: str) -> None:
|
||||
|
|
@ -48,12 +44,7 @@ def _drop_check_if_exists(name: str) -> None:
|
|||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.execute(
|
||||
sa.text(
|
||||
"UPDATE user_settings SET request_delay_seconds = 2 "
|
||||
"WHERE request_delay_seconds < 2"
|
||||
)
|
||||
)
|
||||
op.execute(sa.text("UPDATE user_settings SET request_delay_seconds = 2 WHERE request_delay_seconds < 2"))
|
||||
|
||||
_drop_check_if_exists(CHECK_NAME_MIN_1)
|
||||
_drop_check_if_exists(LEGACY_CHECK_NAME_MIN_1)
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-22 12:45:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision: str = "20260222_0016"
|
||||
down_revision: str | Sequence[str] | None = "20260221_0015"
|
||||
|
|
|
|||
|
|
@ -5,26 +5,29 @@ Revises: fa0c5e3262d5
|
|||
Create Date: 2026-02-24 19:10:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '20260224_0018'
|
||||
down_revision: str | Sequence[str] | None = 'fa0c5e3262d5'
|
||||
revision: str = "20260224_0018"
|
||||
down_revision: str | Sequence[str] | None = "fa0c5e3262d5"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.add_column('publications', sa.Column('openalex_enriched', sa.Boolean(), server_default=sa.text('false'), nullable=False))
|
||||
op.add_column(
|
||||
"publications", sa.Column("openalex_enriched", sa.Boolean(), server_default=sa.text("false"), nullable=False)
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_column('publications', 'openalex_enriched')
|
||||
op.drop_column("publications", "openalex_enriched")
|
||||
# ### end Alembic commands ###
|
||||
|
|
|
|||
|
|
@ -5,14 +5,14 @@ Revises: 20260224_0018
|
|||
Create Date: 2026-02-24 22:20:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '20260224_0019'
|
||||
down_revision: str | Sequence[str] | None = '20260224_0018'
|
||||
revision: str = "20260224_0019"
|
||||
down_revision: str | Sequence[str] | None = "20260224_0018"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
|
|
|||
|
|
@ -5,22 +5,23 @@ Revises: 20260224_0019
|
|||
Create Date: 2026-02-24 22:50:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = '20260224_0020'
|
||||
down_revision: str | Sequence[str] | None = '20260224_0019'
|
||||
revision: str = "20260224_0020"
|
||||
down_revision: str | Sequence[str] | None = "20260224_0019"
|
||||
branch_labels: str | Sequence[str] | None = None
|
||||
depends_on: str | Sequence[str] | None = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column('publications', sa.Column('openalex_last_attempt_at', sa.DateTime(timezone=True), nullable=True))
|
||||
op.add_column("publications", sa.Column("openalex_last_attempt_at", sa.DateTime(timezone=True), nullable=True))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column('publications', 'openalex_last_attempt_at')
|
||||
op.drop_column("publications", "openalex_last_attempt_at")
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-25 09:10:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260225_0021"
|
||||
|
|
@ -56,9 +56,7 @@ def upgrade() -> None:
|
|||
"crawl_runs",
|
||||
["user_id"],
|
||||
unique=True,
|
||||
postgresql_where=sa.text(
|
||||
"status IN ('running'::run_status, 'resolving'::run_status)"
|
||||
),
|
||||
postgresql_where=sa.text("status IN ('running'::run_status, 'resolving'::run_status)"),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-25 10:00:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260225_0022"
|
||||
|
|
|
|||
|
|
@ -7,9 +7,9 @@ Create Date: 2026-02-26 12:40:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260226_0023"
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ Create Date: 2026-02-26 14:20:00.000000
|
|||
|
||||
from collections.abc import Sequence
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "20260226_0024"
|
||||
|
|
|
|||
|
|
@ -1,2 +1 @@
|
|||
from __future__ import annotations
|
||||
|
||||
|
|
|
|||
|
|
@ -82,4 +82,3 @@ def register_api_exception_handlers(app: FastAPI) -> None:
|
|||
message="Request validation failed.",
|
||||
details=exc.errors(),
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,2 +1 @@
|
|||
from __future__ import annotations
|
||||
|
||||
|
|
|
|||
|
|
@ -86,7 +86,14 @@ async def create_user(
|
|||
message=str(exc),
|
||||
) from exc
|
||||
|
||||
structured_log(logger, "info", "api.admin.user_created", admin_user_id=int(admin_user.id), target_user_id=int(created_user.id), target_is_admin=bool(created_user.is_admin))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.user_created",
|
||||
admin_user_id=int(admin_user.id),
|
||||
target_user_id=int(created_user.id),
|
||||
target_is_admin=bool(created_user.is_admin),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_serialize_user(created_user),
|
||||
|
|
@ -111,11 +118,7 @@ async def set_user_active(
|
|||
code="user_not_found",
|
||||
message="User not found.",
|
||||
)
|
||||
if (
|
||||
int(target_user.id) == int(admin_user.id)
|
||||
and bool(target_user.is_active)
|
||||
and not bool(payload.is_active)
|
||||
):
|
||||
if int(target_user.id) == int(admin_user.id) and bool(target_user.is_active) and not bool(payload.is_active):
|
||||
raise ApiException(
|
||||
status_code=400,
|
||||
code="cannot_deactivate_self",
|
||||
|
|
@ -126,7 +129,14 @@ async def set_user_active(
|
|||
user=target_user,
|
||||
is_active=bool(payload.is_active),
|
||||
)
|
||||
structured_log(logger, "info", "api.admin.user_active_updated", admin_user_id=int(admin_user.id), target_user_id=int(updated_user.id), is_active=bool(updated_user.is_active))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.user_active_updated",
|
||||
admin_user_id=int(admin_user.id),
|
||||
target_user_id=int(updated_user.id),
|
||||
is_active=bool(updated_user.is_active),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_serialize_user(updated_user),
|
||||
|
|
@ -166,7 +176,13 @@ async def reset_user_password(
|
|||
user=target_user,
|
||||
password_hash=auth_service.hash_password(validated_password),
|
||||
)
|
||||
structured_log(logger, "info", "api.admin.user_password_reset", admin_user_id=int(admin_user.id), target_user_id=int(target_user.id))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.user_password_reset",
|
||||
admin_user_id=int(admin_user.id),
|
||||
target_user_id=int(target_user.id),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={"message": f"Password reset: {target_user.email}"},
|
||||
|
|
|
|||
|
|
@ -10,14 +10,14 @@ from app.api.errors import ApiException
|
|||
from app.api.responses import success_payload
|
||||
from app.api.schemas import (
|
||||
AdminDbIntegrityEnvelope,
|
||||
AdminPdfQueueBulkEnqueueEnvelope,
|
||||
AdminPdfQueueRequeueEnvelope,
|
||||
AdminPdfQueueEnvelope,
|
||||
AdminDbRepairJobsEnvelope,
|
||||
AdminRepairPublicationNearDuplicatesEnvelope,
|
||||
AdminRepairPublicationNearDuplicatesRequest,
|
||||
AdminPdfQueueBulkEnqueueEnvelope,
|
||||
AdminPdfQueueEnvelope,
|
||||
AdminPdfQueueRequeueEnvelope,
|
||||
AdminRepairPublicationLinksEnvelope,
|
||||
AdminRepairPublicationLinksRequest,
|
||||
AdminRepairPublicationNearDuplicatesEnvelope,
|
||||
AdminRepairPublicationNearDuplicatesRequest,
|
||||
)
|
||||
from app.db.models import DataRepairJob, User
|
||||
from app.db.session import get_db_session
|
||||
|
|
@ -25,8 +25,8 @@ from app.logging_utils import structured_log
|
|||
from app.services.domains.dbops import (
|
||||
collect_integrity_report,
|
||||
list_repair_jobs,
|
||||
run_publication_near_duplicate_repair,
|
||||
run_publication_link_repair,
|
||||
run_publication_near_duplicate_repair,
|
||||
)
|
||||
from app.services.domains.publications import application as publication_service
|
||||
|
||||
|
|
@ -136,7 +136,15 @@ async def get_integrity_report(
|
|||
admin_user: User = Depends(get_api_admin_user),
|
||||
):
|
||||
report = await collect_integrity_report(db_session)
|
||||
structured_log(logger, "info", "api.admin.db.integrity_checked", admin_user_id=int(admin_user.id), status=report.get("status"), failure_count=len(report.get("failures", [])), warning_count=len(report.get("warnings", [])))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.integrity_checked",
|
||||
admin_user_id=int(admin_user.id),
|
||||
status=report.get("status"),
|
||||
failure_count=len(report.get("failures", [])),
|
||||
warning_count=len(report.get("warnings", [])),
|
||||
)
|
||||
return success_payload(request, data=report)
|
||||
|
||||
|
||||
|
|
@ -151,7 +159,14 @@ async def get_repair_jobs(
|
|||
admin_user: User = Depends(get_api_admin_user),
|
||||
):
|
||||
jobs = await list_repair_jobs(db_session, limit=limit)
|
||||
structured_log(logger, "info", "api.admin.db.repair_jobs_listed", admin_user_id=int(admin_user.id), limit=int(limit), job_count=len(jobs))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.repair_jobs_listed",
|
||||
admin_user_id=int(admin_user.id),
|
||||
limit=int(limit),
|
||||
job_count=len(jobs),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={"jobs": [_serialize_repair_job(job) for job in jobs]},
|
||||
|
|
@ -186,7 +201,9 @@ async def get_pdf_queue(
|
|||
status=normalized_status,
|
||||
)
|
||||
structured_log(
|
||||
logger, "info", "api.admin.db.pdf_queue_listed",
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.pdf_queue_listed",
|
||||
admin_user_id=int(admin_user.id),
|
||||
page=int(resolved_page),
|
||||
page_size=int(resolved_limit),
|
||||
|
|
@ -232,7 +249,14 @@ async def requeue_pdf_lookup(
|
|||
message="Publication not found.",
|
||||
)
|
||||
status, message = _requeue_response_state(queued=result.queued)
|
||||
structured_log(logger, "info", "api.admin.db.pdf_requeued", admin_user_id=int(admin_user.id), publication_id=int(publication_id), queued=bool(result.queued))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.pdf_requeued",
|
||||
admin_user_id=int(admin_user.id),
|
||||
publication_id=int(publication_id),
|
||||
queued=bool(result.queued),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
@ -260,7 +284,15 @@ async def requeue_all_missing_pdfs(
|
|||
request_email=admin_user.email,
|
||||
limit=limit,
|
||||
)
|
||||
structured_log(logger, "info", "api.admin.db.pdf_queue_requeue_all", admin_user_id=int(admin_user.id), limit=int(limit), requested_count=int(result.requested_count), queued_count=int(result.queued_count))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.pdf_queue_requeue_all",
|
||||
admin_user_id=int(admin_user.id),
|
||||
limit=int(limit),
|
||||
requested_count=int(result.requested_count),
|
||||
queued_count=int(result.queued_count),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
@ -301,7 +333,9 @@ async def trigger_publication_link_repair(
|
|||
message=str(exc),
|
||||
) from exc
|
||||
structured_log(
|
||||
logger, "info", "api.admin.db.publication_link_repair_triggered",
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.publication_link_repair_triggered",
|
||||
admin_user_id=int(admin_user.id),
|
||||
scope_mode=payload.scope_mode,
|
||||
target_user_id=int(payload.user_id) if payload.user_id is not None else None,
|
||||
|
|
@ -340,7 +374,16 @@ async def trigger_publication_near_duplicate_repair(
|
|||
code="invalid_near_duplicate_repair_request",
|
||||
message=str(exc),
|
||||
) from exc
|
||||
structured_log(logger, "info", "api.admin.db.dedup_repair_triggered", admin_user_id=int(admin_user.id), dry_run=bool(payload.dry_run), selected_cluster_count=len(payload.selected_cluster_keys), job_id=int(result["job_id"]), status=result["status"])
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.admin.db.dedup_repair_triggered",
|
||||
admin_user_id=int(admin_user.id),
|
||||
dry_run=bool(payload.dry_run),
|
||||
selected_cluster_count=len(payload.selected_cluster_keys),
|
||||
job_id=int(result["job_id"]),
|
||||
status=result["status"],
|
||||
)
|
||||
return success_payload(request, data=result)
|
||||
|
||||
|
||||
|
|
@ -381,12 +424,17 @@ async def drop_all_publications(
|
|||
await db_session.execute(delete(PublicationPdfJobEvent))
|
||||
await db_session.execute(delete(PublicationPdfJob))
|
||||
await db_session.execute(delete(Publication))
|
||||
await db_session.execute(
|
||||
update(ScholarProfile).values(baseline_completed=False)
|
||||
)
|
||||
await db_session.execute(update(ScholarProfile).values(baseline_completed=False))
|
||||
await db_session.commit()
|
||||
|
||||
structured_log(logger, "warning", "api.admin.db.all_publications_dropped", admin_user_id=int(admin_user.id), admin_email=admin_user.email, deleted_count=int(total_publications))
|
||||
structured_log(
|
||||
logger,
|
||||
"warning",
|
||||
"api.admin.db.all_publications_dropped",
|
||||
admin_user_id=int(admin_user.id),
|
||||
admin_email=admin_user.email,
|
||||
deleted_count=int(total_publications),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@ import logging
|
|||
from fastapi import APIRouter, Depends, Request
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.errors import ApiException
|
||||
from app.api.deps import get_api_current_user
|
||||
from app.api.errors import ApiException
|
||||
from app.api.responses import success_payload
|
||||
from app.api.schemas import (
|
||||
AuthMeEnvelope,
|
||||
|
|
@ -16,14 +16,14 @@ from app.api.schemas import (
|
|||
LoginRequest,
|
||||
MessageEnvelope,
|
||||
)
|
||||
from app.auth.deps import get_auth_service, get_login_rate_limiter
|
||||
from app.logging_utils import structured_log
|
||||
from app.auth.rate_limit import SlidingWindowRateLimiter
|
||||
from app.auth import runtime as auth_runtime
|
||||
from app.auth.deps import get_auth_service, get_login_rate_limiter
|
||||
from app.auth.rate_limit import SlidingWindowRateLimiter
|
||||
from app.auth.service import AuthService
|
||||
from app.auth.session import set_session_user
|
||||
from app.db.models import User
|
||||
from app.db.session import get_db_session
|
||||
from app.logging_utils import structured_log
|
||||
from app.security.csrf import ensure_csrf_token
|
||||
from app.services.domains.users import application as user_service
|
||||
|
||||
|
|
@ -37,7 +37,13 @@ def _login_limiter_key_and_email(request: Request, payload: LoginRequest) -> tup
|
|||
|
||||
|
||||
def _raise_rate_limited(normalized_email: str, retry_after_seconds: int) -> None:
|
||||
structured_log(logger, "warning", "api.auth.login_rate_limited", email=normalized_email, retry_after_seconds=retry_after_seconds)
|
||||
structured_log(
|
||||
logger,
|
||||
"warning",
|
||||
"api.auth.login_rate_limited",
|
||||
email=normalized_email,
|
||||
retry_after_seconds=retry_after_seconds,
|
||||
)
|
||||
raise ApiException(
|
||||
status_code=429,
|
||||
code="rate_limited",
|
||||
|
|
@ -200,7 +206,9 @@ async def logout(
|
|||
):
|
||||
current_user = await auth_runtime.get_authenticated_user(request, db_session)
|
||||
auth_runtime.invalidate_session(request)
|
||||
structured_log(logger, "info", "api.auth.logout", user_id=int(current_user.id) if current_user is not None else None)
|
||||
structured_log(
|
||||
logger, "info", "api.auth.logout", user_id=int(current_user.id) if current_user is not None else None
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from typing import Literal
|
||||
|
||||
from fastapi import APIRouter, Depends, Path, Query, Request
|
||||
|
|
@ -183,7 +183,7 @@ def _resolve_publications_snapshot(
|
|||
snapshot: str | None,
|
||||
) -> tuple[datetime, str]:
|
||||
if snapshot is None:
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
return now_utc, now_utc.isoformat()
|
||||
try:
|
||||
parsed = datetime.fromisoformat(snapshot)
|
||||
|
|
@ -194,8 +194,8 @@ def _resolve_publications_snapshot(
|
|||
message="Invalid publications snapshot cursor.",
|
||||
) from exc
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
normalized = parsed.astimezone(timezone.utc)
|
||||
parsed = parsed.replace(tzinfo=UTC)
|
||||
normalized = parsed.astimezone(UTC)
|
||||
return normalized, normalized.isoformat()
|
||||
|
||||
|
||||
|
|
@ -420,7 +420,9 @@ async def mark_all_publications_read(
|
|||
db_session,
|
||||
user_id=current_user.id,
|
||||
)
|
||||
structured_log(logger, "info", "api.publications.mark_all_read", user_id=current_user.id, updated_count=updated_count)
|
||||
structured_log(
|
||||
logger, "info", "api.publications.mark_all_read", user_id=current_user.id, updated_count=updated_count
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
@ -440,18 +442,20 @@ async def mark_selected_publications_read(
|
|||
db_session: AsyncSession = Depends(get_db_session),
|
||||
current_user: User = Depends(get_api_current_user),
|
||||
):
|
||||
selection_pairs = sorted(
|
||||
{
|
||||
(int(item.scholar_profile_id), int(item.publication_id))
|
||||
for item in payload.selections
|
||||
}
|
||||
)
|
||||
selection_pairs = sorted({(int(item.scholar_profile_id), int(item.publication_id)) for item in payload.selections})
|
||||
updated_count = await publication_service.mark_selected_as_read_for_user(
|
||||
db_session,
|
||||
user_id=current_user.id,
|
||||
selections=selection_pairs,
|
||||
)
|
||||
structured_log(logger, "info", "api.publications.mark_selected_read", user_id=current_user.id, requested_count=len(selection_pairs), updated_count=updated_count)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.publications.mark_selected_read",
|
||||
user_id=current_user.id,
|
||||
requested_count=len(selection_pairs),
|
||||
updated_count=updated_count,
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
@ -486,7 +490,9 @@ async def retry_publication_pdf(
|
|||
pdf_status=current.pdf_status,
|
||||
)
|
||||
structured_log(
|
||||
logger, "info", "api.publications.retry_pdf",
|
||||
logger,
|
||||
"info",
|
||||
"api.publications.retry_pdf",
|
||||
user_id=current_user.id,
|
||||
scholar_profile_id=payload.scholar_profile_id,
|
||||
publication_id=publication_id,
|
||||
|
|
@ -523,7 +529,15 @@ async def toggle_publication_favorite(
|
|||
publication_id=publication_id,
|
||||
is_favorite=payload.is_favorite,
|
||||
)
|
||||
structured_log(logger, "info", "api.publications.favorite", user_id=current_user.id, scholar_profile_id=payload.scholar_profile_id, publication_id=publication_id, is_favorite=bool(payload.is_favorite))
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.publications.favorite",
|
||||
user_id=current_user.id,
|
||||
scholar_profile_id=payload.scholar_profile_id,
|
||||
publication_id=publication_id,
|
||||
is_favorite=bool(payload.is_favorite),
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={
|
||||
|
|
|
|||
|
|
@ -5,13 +5,14 @@ import re
|
|||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.api.deps import get_api_current_user
|
||||
from app.api.errors import ApiException
|
||||
from app.api.responses import success_payload
|
||||
from fastapi.responses import StreamingResponse
|
||||
from app.api.runtime_deps import get_ingestion_service
|
||||
from app.api.schemas import (
|
||||
ManualRunEnvelope,
|
||||
QueueClearEnvelope,
|
||||
|
|
@ -29,7 +30,6 @@ from app.services.domains.runs import application as run_service
|
|||
from app.services.domains.runs.events import event_generator
|
||||
from app.services.domains.settings import application as user_settings_service
|
||||
from app.settings import settings
|
||||
from app.api.runtime_deps import get_ingestion_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -77,10 +77,7 @@ def _normalize_idempotency_key(raw_value: str | None) -> str | None:
|
|||
raise ApiException(
|
||||
status_code=400,
|
||||
code="invalid_idempotency_key",
|
||||
message=(
|
||||
"Invalid Idempotency-Key. Use 8-128 characters from: "
|
||||
"A-Z a-z 0-9 . _ : -"
|
||||
),
|
||||
message=("Invalid Idempotency-Key. Use 8-128 characters from: A-Z a-z 0-9 . _ : -"),
|
||||
)
|
||||
return candidate
|
||||
|
||||
|
|
@ -130,11 +127,7 @@ def _normalize_attempt_log(value: Any) -> list[dict[str, Any]]:
|
|||
"cstart": _int_value(item.get("cstart"), 0),
|
||||
"state": _str_value(item.get("state")),
|
||||
"state_reason": _str_value(item.get("state_reason")),
|
||||
"status_code": (
|
||||
_int_value(item.get("status_code"))
|
||||
if item.get("status_code") is not None
|
||||
else None
|
||||
),
|
||||
"status_code": (_int_value(item.get("status_code")) if item.get("status_code") is not None else None),
|
||||
"fetch_error": _str_value(item.get("fetch_error")),
|
||||
}
|
||||
)
|
||||
|
|
@ -155,19 +148,12 @@ def _normalize_page_logs(value: Any) -> list[dict[str, Any]]:
|
|||
"cstart": _int_value(item.get("cstart"), 0),
|
||||
"state": _str_value(item.get("state")) or "unknown",
|
||||
"state_reason": _str_value(item.get("state_reason")),
|
||||
"status_code": (
|
||||
_int_value(item.get("status_code"))
|
||||
if item.get("status_code") is not None
|
||||
else None
|
||||
),
|
||||
"status_code": (_int_value(item.get("status_code")) if item.get("status_code") is not None else None),
|
||||
"publication_count": _int_value(item.get("publication_count"), 0),
|
||||
"attempt_count": _int_value(item.get("attempt_count"), 0),
|
||||
"has_show_more_button": _bool_value(item.get("has_show_more_button"), False),
|
||||
"articles_range": _str_value(item.get("articles_range")),
|
||||
"warning_codes": [
|
||||
str(code)
|
||||
for code in (warning_codes if isinstance(warning_codes, list) else [])
|
||||
],
|
||||
"warning_codes": [str(code) for code in (warning_codes if isinstance(warning_codes, list) else [])],
|
||||
}
|
||||
)
|
||||
return normalized
|
||||
|
|
@ -179,19 +165,11 @@ def _normalize_debug(value: Any) -> dict[str, Any] | None:
|
|||
marker_counts = value.get("marker_counts_nonzero")
|
||||
warning_codes = value.get("warning_codes")
|
||||
return {
|
||||
"status_code": (
|
||||
_int_value(value.get("status_code"))
|
||||
if value.get("status_code") is not None
|
||||
else None
|
||||
),
|
||||
"status_code": (_int_value(value.get("status_code")) if value.get("status_code") is not None else None),
|
||||
"final_url": _str_value(value.get("final_url")),
|
||||
"fetch_error": _str_value(value.get("fetch_error")),
|
||||
"body_sha256": _str_value(value.get("body_sha256")),
|
||||
"body_length": (
|
||||
_int_value(value.get("body_length"))
|
||||
if value.get("body_length") is not None
|
||||
else None
|
||||
),
|
||||
"body_length": (_int_value(value.get("body_length")) if value.get("body_length") is not None else None),
|
||||
"has_show_more_button": (
|
||||
_bool_value(value.get("has_show_more_button"), False)
|
||||
if value.get("has_show_more_button") is not None
|
||||
|
|
@ -199,10 +177,7 @@ def _normalize_debug(value: Any) -> dict[str, Any] | None:
|
|||
),
|
||||
"articles_range": _str_value(value.get("articles_range")),
|
||||
"state_reason": _str_value(value.get("state_reason")),
|
||||
"warning_codes": [
|
||||
str(code)
|
||||
for code in (warning_codes if isinstance(warning_codes, list) else [])
|
||||
],
|
||||
"warning_codes": [str(code) for code in (warning_codes if isinstance(warning_codes, list) else [])],
|
||||
"marker_counts_nonzero": {
|
||||
str(key): _int_value(count, 0)
|
||||
for key, count in (marker_counts.items() if isinstance(marker_counts, dict) else [])
|
||||
|
|
@ -241,9 +216,7 @@ def _normalize_scholar_result(value: Any) -> dict[str, Any]:
|
|||
"publication_count": _int_value(value.get("publication_count"), 0),
|
||||
"start_cstart": _int_value(value.get("start_cstart"), 0),
|
||||
"continuation_cstart": (
|
||||
_int_value(value.get("continuation_cstart"))
|
||||
if value.get("continuation_cstart") is not None
|
||||
else None
|
||||
_int_value(value.get("continuation_cstart")) if value.get("continuation_cstart") is not None else None
|
||||
),
|
||||
"continuation_enqueued": _bool_value(value.get("continuation_enqueued"), False),
|
||||
"continuation_cleared": _bool_value(value.get("continuation_cleared"), False),
|
||||
|
|
@ -289,7 +262,9 @@ async def _load_safety_state(
|
|||
await db_session.commit()
|
||||
await db_session.refresh(user_settings)
|
||||
structured_log(
|
||||
logger, "info", "api.runs.safety_cooldown_cleared",
|
||||
logger,
|
||||
"info",
|
||||
"api.runs.safety_cooldown_cleared",
|
||||
user_id=user_id,
|
||||
reason=previous_safety_state.get("cooldown_reason"),
|
||||
cooldown_until=previous_safety_state.get("cooldown_until"),
|
||||
|
|
@ -299,7 +274,9 @@ async def _load_safety_state(
|
|||
|
||||
def _raise_manual_runs_disabled(*, user_id: int, safety_state: dict[str, Any]) -> None:
|
||||
structured_log(
|
||||
logger, "warning", "api.runs.manual_blocked_policy",
|
||||
logger,
|
||||
"warning",
|
||||
"api.runs.manual_blocked_policy",
|
||||
user_id=user_id,
|
||||
policy={"manual_run_allowed": False},
|
||||
safety_state=safety_state,
|
||||
|
|
@ -463,7 +440,9 @@ async def _execute_manual_run(
|
|||
|
||||
def _raise_manual_blocked_safety(*, exc, user_id: int) -> None:
|
||||
structured_log(
|
||||
logger, "info", "api.runs.manual_blocked_safety",
|
||||
logger,
|
||||
"info",
|
||||
"api.runs.manual_blocked_safety",
|
||||
user_id=user_id,
|
||||
reason=exc.safety_state.get("cooldown_reason"),
|
||||
cooldown_until=exc.safety_state.get("cooldown_until"),
|
||||
|
|
@ -677,9 +656,10 @@ async def run_manual(
|
|||
await db_session.commit()
|
||||
|
||||
# Kick off background execution
|
||||
from app.db.session import get_session_factory
|
||||
import asyncio
|
||||
|
||||
from app.db.session import get_session_factory
|
||||
|
||||
asyncio.create_task(
|
||||
ingest_service.execute_run(
|
||||
session_factory=get_session_factory(),
|
||||
|
|
@ -714,7 +694,7 @@ async def run_manual(
|
|||
"reused_existing_run": False,
|
||||
"idempotency_key": idempotency_key,
|
||||
"safety_state": await _load_safety_state(db_session, user_id=current_user.id),
|
||||
}
|
||||
},
|
||||
)
|
||||
except ingestion_service.RunBlockedBySafetyPolicyError as exc:
|
||||
await db_session.rollback()
|
||||
|
|
@ -740,7 +720,6 @@ async def run_manual(
|
|||
_raise_manual_failed(exc=exc, user_id=current_user.id)
|
||||
|
||||
|
||||
|
||||
@router.get(
|
||||
"/queue/items",
|
||||
response_model=QueueListEnvelope,
|
||||
|
|
@ -891,7 +870,4 @@ async def stream_run_events(
|
|||
code="run_not_found",
|
||||
message="Run not found.",
|
||||
)
|
||||
return StreamingResponse(
|
||||
event_generator(run_id),
|
||||
media_type="text/event-stream"
|
||||
)
|
||||
return StreamingResponse(event_generator(run_id), media_type="text/event-stream")
|
||||
|
|
|
|||
|
|
@ -27,8 +27,8 @@ from app.logging_utils import structured_log
|
|||
from app.services.domains.ingestion import queue as ingestion_queue_service
|
||||
from app.services.domains.portability import application as import_export_service
|
||||
from app.services.domains.scholar import rate_limit as scholar_rate_limit
|
||||
from app.services.domains.scholars import application as scholar_service
|
||||
from app.services.domains.scholar.source import ScholarSource
|
||||
from app.services.domains.scholars import application as scholar_service
|
||||
from app.settings import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -82,14 +82,18 @@ async def _enqueue_initial_scrape_job_for_scholar(
|
|||
except Exception:
|
||||
await db_session.rollback()
|
||||
structured_log(
|
||||
logger, "warning", "api.scholars.initial_scrape_enqueue_failed",
|
||||
logger,
|
||||
"warning",
|
||||
"api.scholars.initial_scrape_enqueue_failed",
|
||||
user_id=user_id,
|
||||
scholar_profile_id=profile.id,
|
||||
)
|
||||
return False
|
||||
|
||||
structured_log(
|
||||
logger, "info", "api.scholars.initial_scrape_enqueued",
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.initial_scrape_enqueued",
|
||||
user_id=user_id,
|
||||
scholar_profile_id=profile.id,
|
||||
reason=INITIAL_SCHOLAR_SCRAPE_QUEUE_REASON,
|
||||
|
|
@ -120,9 +124,7 @@ def _serialize_scholar(profile) -> dict[str, object]:
|
|||
"is_enabled": bool(profile.is_enabled),
|
||||
"baseline_completed": bool(profile.baseline_completed),
|
||||
"last_run_dt": profile.last_run_dt,
|
||||
"last_run_status": (
|
||||
profile.last_run_status.value if profile.last_run_status is not None else None
|
||||
),
|
||||
"last_run_status": (profile.last_run_status.value if profile.last_run_status is not None else None),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -139,7 +141,9 @@ async def _hydrate_scholar_metadata_if_needed(
|
|||
should_skip, remaining_seconds = _is_create_hydration_rate_limited()
|
||||
if should_skip:
|
||||
structured_log(
|
||||
logger, "info", "api.scholars.create_metadata_hydration_skipped",
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.create_metadata_hydration_skipped",
|
||||
reason="scholar_request_throttle_active",
|
||||
user_id=user_id,
|
||||
scholar_profile_id=profile.id,
|
||||
|
|
@ -158,14 +162,18 @@ async def _hydrate_scholar_metadata_if_needed(
|
|||
)
|
||||
except TimeoutError:
|
||||
structured_log(
|
||||
logger, "info", "api.scholars.create_metadata_hydration_skipped",
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.create_metadata_hydration_skipped",
|
||||
reason="create_timeout",
|
||||
user_id=user_id,
|
||||
scholar_profile_id=profile.id,
|
||||
)
|
||||
except Exception:
|
||||
structured_log(
|
||||
logger, "warning", "api.scholars.create_metadata_hydration_failed",
|
||||
logger,
|
||||
"warning",
|
||||
"api.scholars.create_metadata_hydration_failed",
|
||||
user_id=user_id,
|
||||
scholar_profile_id=profile.id,
|
||||
)
|
||||
|
|
@ -185,9 +193,7 @@ def _search_kwargs() -> dict[str, object]:
|
|||
"cooldown_block_threshold": settings.scholar_name_search_cooldown_block_threshold,
|
||||
"cooldown_seconds": settings.scholar_name_search_cooldown_seconds,
|
||||
"retry_alert_threshold": settings.scholar_name_search_alert_retry_count_threshold,
|
||||
"cooldown_rejection_alert_threshold": (
|
||||
settings.scholar_name_search_alert_cooldown_rejections_threshold
|
||||
),
|
||||
"cooldown_rejection_alert_threshold": (settings.scholar_name_search_alert_cooldown_rejections_threshold),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -299,8 +305,7 @@ async def import_scholars_and_publications(
|
|||
status_code=400,
|
||||
code="invalid_import_schema_version",
|
||||
message=(
|
||||
"Import schema version is not supported. "
|
||||
f"Expected {import_export_service.EXPORT_SCHEMA_VERSION}."
|
||||
f"Import schema version is not supported. Expected {import_export_service.EXPORT_SCHEMA_VERSION}."
|
||||
),
|
||||
)
|
||||
try:
|
||||
|
|
@ -393,7 +398,15 @@ async def search_scholars(
|
|||
message=str(exc),
|
||||
) from exc
|
||||
|
||||
structured_log(logger, "info", "api.scholars.search.completed", user_id=current_user.id, query=query.strip(), candidate_count=len(parsed.candidates), state=parsed.state.value)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.search.completed",
|
||||
user_id=current_user.id,
|
||||
query=query.strip(),
|
||||
candidate_count=len(parsed.candidates),
|
||||
state=parsed.state.value,
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_search_response_data(query, parsed),
|
||||
|
|
@ -422,7 +435,14 @@ async def toggle_scholar(
|
|||
message="Scholar not found.",
|
||||
)
|
||||
updated = await scholar_service.toggle_scholar_enabled(db_session, profile=profile)
|
||||
structured_log(logger, "info", "api.scholars.toggled", user_id=current_user.id, scholar_profile_id=updated.id, is_enabled=updated.is_enabled)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.toggled",
|
||||
user_id=current_user.id,
|
||||
scholar_profile_id=updated.id,
|
||||
is_enabled=updated.is_enabled,
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_serialize_scholar(updated),
|
||||
|
|
@ -455,7 +475,9 @@ async def delete_scholar(
|
|||
profile=profile,
|
||||
upload_dir=settings.scholar_image_upload_dir,
|
||||
)
|
||||
structured_log(logger, "info", "api.scholars.deleted", user_id=current_user.id, scholar_profile_id=scholar_profile_id)
|
||||
structured_log(
|
||||
logger, "info", "api.scholars.deleted", user_id=current_user.id, scholar_profile_id=scholar_profile_id
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data={"message": "Scholar deleted."},
|
||||
|
|
@ -499,7 +521,9 @@ async def update_scholar_image_url(
|
|||
message=str(exc),
|
||||
) from exc
|
||||
|
||||
structured_log(logger, "info", "api.scholars.image_url_updated", user_id=current_user.id, scholar_profile_id=updated.id)
|
||||
structured_log(
|
||||
logger, "info", "api.scholars.image_url_updated", user_id=current_user.id, scholar_profile_id=updated.id
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_serialize_scholar(updated),
|
||||
|
|
@ -541,7 +565,15 @@ async def upload_scholar_image(
|
|||
) from exc
|
||||
|
||||
image_size = len(image_bytes)
|
||||
structured_log(logger, "info", "api.scholars.image_uploaded", user_id=current_user.id, scholar_profile_id=updated.id, content_type=image.content_type, size_bytes=image_size)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"api.scholars.image_uploaded",
|
||||
user_id=current_user.id,
|
||||
scholar_profile_id=updated.id,
|
||||
content_type=image.content_type,
|
||||
size_bytes=image_size,
|
||||
)
|
||||
return success_payload(
|
||||
request,
|
||||
data=_serialize_scholar(updated),
|
||||
|
|
|
|||
|
|
@ -85,7 +85,9 @@ async def _clear_expired_cooldown_with_log(
|
|||
await db_session.commit()
|
||||
await db_session.refresh(user_settings)
|
||||
structured_log(
|
||||
logger, "info", "api.settings.safety_cooldown_cleared",
|
||||
logger,
|
||||
"info",
|
||||
"api.settings.safety_cooldown_cleared",
|
||||
user_id=user_id,
|
||||
reason=previous_safety_state.get("cooldown_reason"),
|
||||
cooldown_until=previous_safety_state.get("cooldown_until"),
|
||||
|
|
@ -164,7 +166,9 @@ async def update_settings(
|
|||
user_settings=updated,
|
||||
)
|
||||
structured_log(
|
||||
logger, "info", "api.settings.updated",
|
||||
logger,
|
||||
"info",
|
||||
"api.settings.updated",
|
||||
user_id=current_user.id,
|
||||
auto_run_enabled=updated.auto_run_enabled,
|
||||
run_interval_minutes=updated.run_interval_minutes,
|
||||
|
|
|
|||
|
|
@ -671,7 +671,7 @@ class AdminRepairPublicationLinksRequest(BaseModel):
|
|||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_scope(self) -> "AdminRepairPublicationLinksRequest":
|
||||
def validate_scope(self) -> AdminRepairPublicationLinksRequest:
|
||||
if self.scope_mode == "single_user" and self.user_id is None:
|
||||
raise ValueError("user_id is required when scope_mode=single_user.")
|
||||
if self.scope_mode == "all_users" and self.user_id is not None:
|
||||
|
|
@ -680,10 +680,7 @@ class AdminRepairPublicationLinksRequest(BaseModel):
|
|||
expected = "REPAIR ALL USERS"
|
||||
provided = (self.confirmation_text or "").strip()
|
||||
if provided != expected:
|
||||
raise ValueError(
|
||||
"confirmation_text must equal 'REPAIR ALL USERS' "
|
||||
"when applying a repair to all users."
|
||||
)
|
||||
raise ValueError("confirmation_text must equal 'REPAIR ALL USERS' when applying a repair to all users.")
|
||||
return self
|
||||
|
||||
|
||||
|
|
@ -735,7 +732,7 @@ class AdminRepairPublicationNearDuplicatesRequest(BaseModel):
|
|||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_apply_mode(self) -> "AdminRepairPublicationNearDuplicatesRequest":
|
||||
def validate_apply_mode(self) -> AdminRepairPublicationNearDuplicatesRequest:
|
||||
if self.dry_run:
|
||||
return self
|
||||
if not self.selected_cluster_keys:
|
||||
|
|
@ -744,8 +741,7 @@ class AdminRepairPublicationNearDuplicatesRequest(BaseModel):
|
|||
provided = (self.confirmation_text or "").strip()
|
||||
if provided != expected:
|
||||
raise ValueError(
|
||||
"confirmation_text must equal 'MERGE SELECTED DUPLICATES' "
|
||||
"when applying near-duplicate merges."
|
||||
"confirmation_text must equal 'MERGE SELECTED DUPLICATES' when applying near-duplicate merges."
|
||||
)
|
||||
return self
|
||||
|
||||
|
|
|
|||
|
|
@ -1,2 +1 @@
|
|||
"""Authentication package for scholarr."""
|
||||
|
||||
|
|
|
|||
|
|
@ -24,4 +24,3 @@ def get_login_rate_limiter() -> SlidingWindowRateLimiter:
|
|||
max_attempts=settings.login_rate_limit_attempts,
|
||||
window_seconds=settings.login_rate_limit_window_seconds,
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -6,8 +6,8 @@ from fastapi import Request
|
|||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.auth.session import clear_session_user, get_session_user, set_session_user
|
||||
from app.logging_utils import structured_log
|
||||
from app.db.models import User
|
||||
from app.logging_utils import structured_log
|
||||
from app.security.csrf import CSRF_SESSION_KEY
|
||||
from app.services.domains.users import application as user_service
|
||||
|
||||
|
|
|
|||
|
|
@ -16,4 +16,3 @@ class PasswordService:
|
|||
return bool(self._hasher.verify(password_hash, password))
|
||||
except (InvalidHashError, VerificationError):
|
||||
return False
|
||||
|
||||
|
|
|
|||
|
|
@ -21,9 +21,7 @@ class AuthService:
|
|||
normalized_email = email.strip().lower()
|
||||
if not normalized_email or not password:
|
||||
return None
|
||||
result = await db_session.execute(
|
||||
select(User).where(User.email == normalized_email)
|
||||
)
|
||||
result = await db_session.execute(select(User).where(User.email == normalized_email))
|
||||
user = result.scalar_one_or_none()
|
||||
if user is None or not user.is_active:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ from dataclasses import dataclass
|
|||
|
||||
from starlette.requests import Request
|
||||
|
||||
|
||||
SESSION_USER_ID_KEY = "auth_user_id"
|
||||
SESSION_USER_EMAIL_KEY = "auth_user_email"
|
||||
SESSION_USER_IS_ADMIN_KEY = "auth_user_is_admin"
|
||||
|
|
|
|||
|
|
@ -1,9 +1,8 @@
|
|||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import MetaData, event
|
||||
from sqlalchemy.orm import DeclarativeBase
|
||||
|
||||
|
||||
NAMING_CONVENTION = {
|
||||
"ix": "ix_%(column_0_label)s",
|
||||
"uq": "uq_%(table_name)s_%(column_0_name)s",
|
||||
|
|
@ -21,7 +20,7 @@ class Base(DeclarativeBase):
|
|||
def _set_updated_at_before_update(_mapper, _connection, target) -> None:
|
||||
# Keep audit timestamps current for ORM-managed updates.
|
||||
if hasattr(target, "updated_at"):
|
||||
target.updated_at = datetime.now(timezone.utc)
|
||||
target.updated_at = datetime.now(UTC)
|
||||
|
||||
|
||||
metadata = Base.metadata
|
||||
|
|
|
|||
212
app/db/models.py
212
app/db/models.py
|
|
@ -62,18 +62,10 @@ class User(Base):
|
|||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
email: Mapped[str] = mapped_column(String(255), nullable=False, unique=True)
|
||||
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
is_active: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("true")
|
||||
)
|
||||
is_admin: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
is_active: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("true"))
|
||||
is_admin: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class UserSetting(Base):
|
||||
|
|
@ -89,18 +81,10 @@ class UserSetting(Base):
|
|||
),
|
||||
)
|
||||
|
||||
user_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("users.id", ondelete="CASCADE"), primary_key=True
|
||||
)
|
||||
auto_run_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
run_interval_minutes: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, server_default=text("1440")
|
||||
)
|
||||
request_delay_seconds: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, server_default=text("10")
|
||||
)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), primary_key=True)
|
||||
auto_run_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
run_interval_minutes: Mapped[int] = mapped_column(Integer, nullable=False, server_default=text("1440"))
|
||||
request_delay_seconds: Mapped[int] = mapped_column(Integer, nullable=False, server_default=text("10"))
|
||||
nav_visible_pages: Mapped[list[str]] = mapped_column(
|
||||
JSONB,
|
||||
nullable=False,
|
||||
|
|
@ -120,12 +104,8 @@ class UserSetting(Base):
|
|||
crossref_api_token: Mapped[str | None] = mapped_column(String(255))
|
||||
crossref_api_mailto: Mapped[str | None] = mapped_column(String(255))
|
||||
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class ScholarProfile(Base):
|
||||
|
|
@ -136,9 +116,7 @@ class ScholarProfile(Base):
|
|||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
scholar_id: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
display_name: Mapped[str | None] = mapped_column(String(255))
|
||||
profile_image_url: Mapped[str | None] = mapped_column(Text)
|
||||
|
|
@ -146,22 +124,14 @@ class ScholarProfile(Base):
|
|||
profile_image_upload_path: Mapped[str | None] = mapped_column(Text)
|
||||
last_initial_page_fingerprint_sha256: Mapped[str | None] = mapped_column(String(64))
|
||||
last_initial_page_checked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
is_enabled: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("true")
|
||||
)
|
||||
baseline_completed: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
is_enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("true"))
|
||||
baseline_completed: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
last_run_dt: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
last_run_status: Mapped[RunStatus | None] = mapped_column(
|
||||
RUN_STATUS_DB_ENUM,
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class CrawlRun(Base):
|
||||
|
|
@ -172,51 +142,29 @@ class CrawlRun(Base):
|
|||
"uq_crawl_runs_user_active",
|
||||
"user_id",
|
||||
unique=True,
|
||||
postgresql_where=text(
|
||||
"status IN ('running'::run_status, 'resolving'::run_status)"
|
||||
),
|
||||
postgresql_where=text("status IN ('running'::run_status, 'resolving'::run_status)"),
|
||||
),
|
||||
Index(
|
||||
"uq_crawl_runs_user_manual_idempotency_key",
|
||||
"user_id",
|
||||
"idempotency_key",
|
||||
unique=True,
|
||||
postgresql_where=text(
|
||||
"idempotency_key IS NOT NULL AND trigger_type = 'manual'::run_trigger_type"
|
||||
),
|
||||
postgresql_where=text("idempotency_key IS NOT NULL AND trigger_type = 'manual'::run_trigger_type"),
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(primary_key=True)
|
||||
user_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("users.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
trigger_type: Mapped[RunTriggerType] = mapped_column(
|
||||
RUN_TRIGGER_TYPE_DB_ENUM, nullable=False
|
||||
)
|
||||
status: Mapped[RunStatus] = mapped_column(
|
||||
RUN_STATUS_DB_ENUM, nullable=False
|
||||
)
|
||||
start_dt: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
user_id: Mapped[int] = mapped_column(ForeignKey("users.id", ondelete="CASCADE"), nullable=False)
|
||||
trigger_type: Mapped[RunTriggerType] = mapped_column(RUN_TRIGGER_TYPE_DB_ENUM, nullable=False)
|
||||
status: Mapped[RunStatus] = mapped_column(RUN_STATUS_DB_ENUM, nullable=False)
|
||||
start_dt: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
end_dt: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
scholar_count: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, server_default=text("0")
|
||||
)
|
||||
new_pub_count: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, server_default=text("0")
|
||||
)
|
||||
scholar_count: Mapped[int] = mapped_column(Integer, nullable=False, server_default=text("0"))
|
||||
new_pub_count: Mapped[int] = mapped_column(Integer, nullable=False, server_default=text("0"))
|
||||
idempotency_key: Mapped[str | None] = mapped_column(String(128))
|
||||
error_log: Mapped[dict] = mapped_column(
|
||||
JSONB, nullable=False, server_default=text("'{}'::jsonb")
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
error_log: Mapped[dict] = mapped_column(JSONB, nullable=False, server_default=text("'{}'::jsonb"))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class Publication(Base):
|
||||
|
|
@ -237,28 +185,16 @@ class Publication(Base):
|
|||
title_raw: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
title_normalized: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
year: Mapped[int | None] = mapped_column(Integer)
|
||||
citation_count: Mapped[int] = mapped_column(
|
||||
Integer, nullable=False, server_default=text("0")
|
||||
)
|
||||
citation_count: Mapped[int] = mapped_column(Integer, nullable=False, server_default=text("0"))
|
||||
author_text: Mapped[str | None] = mapped_column(Text)
|
||||
venue_text: Mapped[str | None] = mapped_column(Text)
|
||||
pub_url: Mapped[str | None] = mapped_column(Text)
|
||||
pdf_url: Mapped[str | None] = mapped_column(Text)
|
||||
canonical_title_hash: Mapped[str | None] = mapped_column(
|
||||
String(64), nullable=True, index=True
|
||||
)
|
||||
openalex_enriched: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
openalex_last_attempt_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
canonical_title_hash: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
|
||||
openalex_enriched: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
openalex_last_attempt_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class PublicationIdentifier(Base):
|
||||
|
|
@ -300,12 +236,8 @@ class PublicationIdentifier(Base):
|
|||
server_default=text("0"),
|
||||
)
|
||||
evidence_url: Mapped[str | None] = mapped_column(Text)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class PublicationPdfJob(Base):
|
||||
|
|
@ -343,12 +275,8 @@ class PublicationPdfJob(Base):
|
|||
last_requested_by_user_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("users.id", ondelete="SET NULL"),
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class PublicationPdfJobEvent(Base):
|
||||
|
|
@ -372,9 +300,7 @@ class PublicationPdfJobEvent(Base):
|
|||
source: Mapped[str | None] = mapped_column(String(32))
|
||||
failure_reason: Mapped[str | None] = mapped_column(String(64))
|
||||
message: Mapped[str | None] = mapped_column(Text)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class ScholarPublication(Base):
|
||||
|
|
@ -392,18 +318,10 @@ class ScholarPublication(Base):
|
|||
ForeignKey("publications.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
is_read: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
is_favorite: Mapped[bool] = mapped_column(
|
||||
Boolean, nullable=False, server_default=text("false")
|
||||
)
|
||||
first_seen_run_id: Mapped[int | None] = mapped_column(
|
||||
ForeignKey("crawl_runs.id", ondelete="SET NULL")
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
is_read: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
is_favorite: Mapped[bool] = mapped_column(Boolean, nullable=False, server_default=text("false"))
|
||||
first_seen_run_id: Mapped[int | None] = mapped_column(ForeignKey("crawl_runs.id", ondelete="SET NULL"))
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class IngestionQueueItem(Base):
|
||||
|
|
@ -458,12 +376,8 @@ class IngestionQueueItem(Base):
|
|||
last_error: Mapped[str | None] = mapped_column(Text)
|
||||
dropped_reason: Mapped[str | None] = mapped_column(String(128))
|
||||
dropped_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class DataRepairJob(Base):
|
||||
|
|
@ -503,12 +417,8 @@ class DataRepairJob(Base):
|
|||
error_text: Mapped[str | None] = mapped_column(Text)
|
||||
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class AuthorSearchRuntimeState(Base):
|
||||
|
|
@ -532,12 +442,8 @@ class AuthorSearchRuntimeState(Base):
|
|||
nullable=False,
|
||||
server_default=text("false"),
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class ArxivRuntimeState(Base):
|
||||
|
|
@ -546,12 +452,8 @@ class ArxivRuntimeState(Base):
|
|||
state_key: Mapped[str] = mapped_column(String(64), primary_key=True)
|
||||
next_allowed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
cooldown_until: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
created_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class ArxivQueryCacheEntry(Base):
|
||||
|
|
@ -570,12 +472,8 @@ class ArxivQueryCacheEntry(Base):
|
|||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
)
|
||||
cached_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
cached_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
||||
|
||||
class AuthorSearchCacheEntry(Base):
|
||||
|
|
@ -594,9 +492,5 @@ class AuthorSearchCacheEntry(Base):
|
|||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
)
|
||||
cached_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, server_default=func.now()
|
||||
)
|
||||
cached_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from collections.abc import AsyncIterator
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
|
|
@ -32,7 +32,9 @@ def _normalized_pool_mode(raw_mode: str) -> str:
|
|||
if mode in {"null", "queue"}:
|
||||
return mode
|
||||
structured_log(
|
||||
logger, "warning", "db.invalid_pool_mode_fallback",
|
||||
logger,
|
||||
"warning",
|
||||
"db.invalid_pool_mode_fallback",
|
||||
database_pool_mode=raw_mode,
|
||||
fallback_mode="queue",
|
||||
)
|
||||
|
|
@ -55,7 +57,9 @@ def get_engine() -> AsyncEngine:
|
|||
|
||||
_engine = create_async_engine(settings.database_url, **engine_kwargs)
|
||||
structured_log(
|
||||
logger, "info", "db.engine_initialized",
|
||||
logger,
|
||||
"info",
|
||||
"db.engine_initialized",
|
||||
pool_mode=pool_mode,
|
||||
)
|
||||
return _engine
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from secrets import token_urlsafe
|
||||
import logging
|
||||
import time
|
||||
from secrets import token_urlsafe
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
|
|
@ -63,7 +63,9 @@ class RequestLoggingMiddleware(BaseHTTPMiddleware):
|
|||
should_log = self._log_requests and not self._is_skipped_path(request.url.path)
|
||||
if should_log:
|
||||
structured_log(
|
||||
logger, "debug", "request.started",
|
||||
logger,
|
||||
"debug",
|
||||
"request.started",
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
)
|
||||
|
|
@ -86,7 +88,9 @@ class RequestLoggingMiddleware(BaseHTTPMiddleware):
|
|||
response.headers[REQUEST_ID_HEADER] = request_id
|
||||
if should_log:
|
||||
structured_log(
|
||||
logger, "debug", "request.completed",
|
||||
logger,
|
||||
"debug",
|
||||
"request.completed",
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
status_code=response.status_code,
|
||||
|
|
@ -164,11 +168,7 @@ class SecurityHeadersMiddleware(BaseHTTPMiddleware):
|
|||
|
||||
csp_policy = self._csp_policy_for_path(request.url.path)
|
||||
if self._csp_enabled and csp_policy:
|
||||
csp_header = (
|
||||
"Content-Security-Policy-Report-Only"
|
||||
if self._csp_report_only
|
||||
else "Content-Security-Policy"
|
||||
)
|
||||
csp_header = "Content-Security-Policy-Report-Only" if self._csp_report_only else "Content-Security-Policy"
|
||||
response.headers.setdefault(csp_header, csp_policy)
|
||||
|
||||
hsts = self._strict_transport_security_value()
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from app.logging_context import get_request_id
|
||||
|
|
@ -102,7 +102,9 @@ class JsonLogFormatter(logging.Formatter):
|
|||
if key.lower() in self._redact_fields:
|
||||
return "[REDACTED]"
|
||||
if isinstance(value, dict):
|
||||
return {nested_key: self._redact_value(nested_key, nested_value) for nested_key, nested_value in value.items()}
|
||||
return {
|
||||
nested_key: self._redact_value(nested_key, nested_value) for nested_key, nested_value in value.items()
|
||||
}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [self._redact_value(key, item) for item in value]
|
||||
return value
|
||||
|
|
@ -177,7 +179,7 @@ def _normalize_level(level: str) -> int:
|
|||
|
||||
|
||||
def _format_timestamp(created_ts: float) -> str:
|
||||
dt = datetime.fromtimestamp(created_ts, tz=timezone.utc)
|
||||
dt = datetime.fromtimestamp(created_ts, tz=UTC)
|
||||
return dt.strftime("%Y-%m-%d %H:%M:%SZ")
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ from __future__ import annotations
|
|||
|
||||
from contextvars import ContextVar
|
||||
|
||||
|
||||
_request_id_ctx: ContextVar[str | None] = ContextVar("request_id", default=None)
|
||||
|
||||
|
||||
|
|
@ -12,4 +11,3 @@ def get_request_id() -> str | None:
|
|||
|
||||
def set_request_id(value: str | None) -> None:
|
||||
_request_id_ctx.set(value)
|
||||
|
||||
|
|
|
|||
22
app/main.py
22
app/main.py
|
|
@ -1,7 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
|
|
@ -12,9 +12,7 @@ from starlette.middleware.sessions import SessionMiddleware
|
|||
from app.api.errors import register_api_exception_handlers
|
||||
from app.api.media import router as media_router
|
||||
from app.api.router import router as api_router
|
||||
from app.api.runtime_deps import get_ingestion_service, get_scholar_source
|
||||
from app.db.session import check_database
|
||||
from app.db.session import close_engine
|
||||
from app.db.session import check_database, close_engine
|
||||
from app.http.middleware import (
|
||||
RequestLoggingMiddleware,
|
||||
SecurityHeadersMiddleware,
|
||||
|
|
@ -54,19 +52,25 @@ scheduler_service = SchedulerService(
|
|||
@asynccontextmanager
|
||||
async def lifespan(_: FastAPI):
|
||||
structured_log(
|
||||
logger, "info", "app.startup_build_marker",
|
||||
logger,
|
||||
"info",
|
||||
"app.startup_build_marker",
|
||||
build_marker=BUILD_MARKER,
|
||||
frontend_enabled=settings.frontend_enabled,
|
||||
scheduler_enabled=settings.scheduler_enabled,
|
||||
log_format=settings.log_format,
|
||||
)
|
||||
|
||||
from app.db.session import get_session_factory
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db.session import get_session_factory
|
||||
|
||||
try:
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as session:
|
||||
await session.execute(text("UPDATE crawl_runs SET status = 'failed' WHERE status::text IN ('running', 'resolving')"))
|
||||
await session.execute(
|
||||
text("UPDATE crawl_runs SET status = 'failed' WHERE status::text IN ('running', 'resolving')")
|
||||
)
|
||||
await session.commit()
|
||||
structured_log(logger, "info", "app.startup_orphaned_runs_cleaned")
|
||||
except Exception as e:
|
||||
|
|
@ -107,9 +111,7 @@ app.add_middleware(
|
|||
content_security_policy_report_only=settings.security_csp_report_only,
|
||||
strict_transport_security_enabled=settings.security_strict_transport_security_enabled,
|
||||
strict_transport_security_max_age=settings.security_strict_transport_security_max_age,
|
||||
strict_transport_security_include_subdomains=(
|
||||
settings.security_strict_transport_security_include_subdomains
|
||||
),
|
||||
strict_transport_security_include_subdomains=(settings.security_strict_transport_security_include_subdomains),
|
||||
strict_transport_security_preload=settings.security_strict_transport_security_preload,
|
||||
)
|
||||
app.include_router(api_router)
|
||||
|
|
|
|||
|
|
@ -1,2 +1 @@
|
|||
"""Security middleware and helpers for scholarr."""
|
||||
|
||||
|
|
|
|||
|
|
@ -38,7 +38,9 @@ class CSRFMiddleware(BaseHTTPMiddleware):
|
|||
session_token = request.session.get(CSRF_SESSION_KEY)
|
||||
if not session_token:
|
||||
structured_log(
|
||||
logger, "warning", "csrf.missing_session_token",
|
||||
logger,
|
||||
"warning",
|
||||
"csrf.missing_session_token",
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
)
|
||||
|
|
@ -56,7 +58,9 @@ class CSRFMiddleware(BaseHTTPMiddleware):
|
|||
|
||||
if not request_token or not compare_digest(str(session_token), str(request_token)):
|
||||
structured_log(
|
||||
logger, "warning", "csrf.invalid_token",
|
||||
logger,
|
||||
"warning",
|
||||
"csrf.invalid_token",
|
||||
method=request.method,
|
||||
path=request.url.path,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,2 +1 @@
|
|||
"""Service layer for scholarr application workflows."""
|
||||
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from app.services.domains.arxiv.gateway import (
|
||||
build_arxiv_query,
|
||||
get_arxiv_gateway,
|
||||
)
|
||||
from app.services.domains.arxiv.errors import ArxivRateLimitError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import asdict
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import json
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import asdict
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, func, select
|
||||
|
|
@ -33,12 +33,9 @@ async def get_cached_feed(
|
|||
) -> ArxivFeed | None:
|
||||
timestamp = _as_utc(now_utc)
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as db_session:
|
||||
async with db_session.begin():
|
||||
async with session_factory() as db_session, db_session.begin():
|
||||
result = await db_session.execute(
|
||||
select(ArxivQueryCacheEntry).where(
|
||||
ArxivQueryCacheEntry.query_fingerprint == query_fingerprint
|
||||
)
|
||||
select(ArxivQueryCacheEntry).where(ArxivQueryCacheEntry.query_fingerprint == query_fingerprint)
|
||||
)
|
||||
entry = result.scalar_one_or_none()
|
||||
return await _validate_cached_entry(db_session, entry=entry, now_utc=timestamp)
|
||||
|
|
@ -54,8 +51,7 @@ async def set_cached_feed(
|
|||
) -> None:
|
||||
timestamp = _as_utc(now_utc)
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as db_session:
|
||||
async with db_session.begin():
|
||||
async with session_factory() as db_session, db_session.begin():
|
||||
await _write_cached_entry(
|
||||
db_session,
|
||||
query_fingerprint=query_fingerprint,
|
||||
|
|
@ -142,9 +138,7 @@ async def _write_cached_entry(
|
|||
) -> None:
|
||||
ttl = max(float(ttl_seconds), 0.0)
|
||||
result = await db_session.execute(
|
||||
select(ArxivQueryCacheEntry).where(
|
||||
ArxivQueryCacheEntry.query_fingerprint == query_fingerprint
|
||||
)
|
||||
select(ArxivQueryCacheEntry).where(ArxivQueryCacheEntry.query_fingerprint == query_fingerprint)
|
||||
)
|
||||
existing = result.scalar_one_or_none()
|
||||
if ttl <= 0.0:
|
||||
|
|
@ -177,30 +171,22 @@ async def _prune_cache_entries(
|
|||
now_utc: datetime,
|
||||
max_entries: int,
|
||||
) -> None:
|
||||
await db_session.execute(
|
||||
delete(ArxivQueryCacheEntry).where(ArxivQueryCacheEntry.expires_at <= now_utc)
|
||||
)
|
||||
await db_session.execute(delete(ArxivQueryCacheEntry).where(ArxivQueryCacheEntry.expires_at <= now_utc))
|
||||
bounded_max_entries = int(max_entries)
|
||||
if bounded_max_entries <= 0:
|
||||
return
|
||||
count_result = await db_session.execute(
|
||||
select(func.count()).select_from(ArxivQueryCacheEntry)
|
||||
)
|
||||
count_result = await db_session.execute(select(func.count()).select_from(ArxivQueryCacheEntry))
|
||||
entry_count = int(count_result.scalar_one() or 0)
|
||||
overflow = max(0, entry_count - bounded_max_entries)
|
||||
if overflow <= 0:
|
||||
return
|
||||
stale_result = await db_session.execute(
|
||||
select(ArxivQueryCacheEntry.query_fingerprint)
|
||||
.order_by(ArxivQueryCacheEntry.cached_at.asc())
|
||||
.limit(overflow)
|
||||
select(ArxivQueryCacheEntry.query_fingerprint).order_by(ArxivQueryCacheEntry.cached_at.asc()).limit(overflow)
|
||||
)
|
||||
stale_keys = [str(row[0]) for row in stale_result.all()]
|
||||
if stale_keys:
|
||||
await db_session.execute(
|
||||
delete(ArxivQueryCacheEntry).where(
|
||||
ArxivQueryCacheEntry.query_fingerprint.in_(stale_keys)
|
||||
)
|
||||
delete(ArxivQueryCacheEntry).where(ArxivQueryCacheEntry.query_fingerprint.in_(stale_keys))
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -275,9 +261,9 @@ def _as_string_list(value: object) -> list[str]:
|
|||
|
||||
def _as_utc(value: datetime | None) -> datetime:
|
||||
if value is None:
|
||||
return datetime.now(timezone.utc)
|
||||
return datetime.now(UTC)
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value.replace(tzinfo=UTC)
|
||||
return value
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Awaitable, Callable
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
import httpx
|
||||
|
||||
|
|
@ -103,9 +103,13 @@ class ArxivClient:
|
|||
if self._cache_enabled:
|
||||
cached = await get_cached_feed(query_fingerprint=query_fingerprint)
|
||||
if cached is not None:
|
||||
structured_log(logger, "info", "arxiv.cache_hit", query_fingerprint=query_fingerprint, source_path=source_path)
|
||||
structured_log(
|
||||
logger, "info", "arxiv.cache_hit", query_fingerprint=query_fingerprint, source_path=source_path
|
||||
)
|
||||
return cached
|
||||
structured_log(logger, "info", "arxiv.cache_miss", query_fingerprint=query_fingerprint, source_path=source_path)
|
||||
structured_log(
|
||||
logger, "info", "arxiv.cache_miss", query_fingerprint=query_fingerprint, source_path=source_path
|
||||
)
|
||||
return await run_with_inflight_dedupe(
|
||||
query_fingerprint=query_fingerprint,
|
||||
fetch_feed=lambda: self._fetch_live_feed(
|
||||
|
|
@ -216,13 +220,13 @@ async def _request_arxiv_feed(
|
|||
cooldown_status = await get_arxiv_cooldown_status()
|
||||
if cooldown_status.is_active:
|
||||
structured_log(
|
||||
logger, "warning", "arxiv.request_skipped_cooldown",
|
||||
logger,
|
||||
"warning",
|
||||
"arxiv.request_skipped_cooldown",
|
||||
source_path=source_path,
|
||||
cooldown_remaining_seconds=float(cooldown_status.remaining_seconds),
|
||||
)
|
||||
raise ArxivRateLimitError(
|
||||
f"arXiv global cooldown active ({cooldown_status.remaining_seconds:.0f}s remaining)"
|
||||
)
|
||||
raise ArxivRateLimitError(f"arXiv global cooldown active ({cooldown_status.remaining_seconds:.0f}s remaining)")
|
||||
|
||||
async def _fetch() -> httpx.Response:
|
||||
timeout_value = _timeout_seconds(timeout_seconds)
|
||||
|
|
@ -272,5 +276,3 @@ def _source_path_from_params(params: dict[str, object]) -> str:
|
|||
if "id_list" in params:
|
||||
return ARXIV_SOURCE_PATH_LOOKUP_IDS
|
||||
return ARXIV_SOURCE_PATH_UNKNOWN
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@ from __future__ import annotations
|
|||
|
||||
import logging
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Protocol
|
||||
import unicodedata
|
||||
from typing import TYPE_CHECKING, Protocol
|
||||
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.arxiv.client import ArxivClient
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import logging
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
import httpx
|
||||
from sqlalchemy import select, text
|
||||
|
|
@ -47,13 +47,11 @@ async def run_with_global_arxiv_limit(
|
|||
|
||||
|
||||
async def get_arxiv_cooldown_status(*, now_utc: datetime | None = None) -> ArxivCooldownStatus:
|
||||
timestamp = _normalize_datetime(now_utc) or datetime.now(timezone.utc)
|
||||
timestamp = _normalize_datetime(now_utc) or datetime.now(UTC)
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as db_session:
|
||||
result = await db_session.execute(
|
||||
select(ArxivRuntimeState.cooldown_until).where(
|
||||
ArxivRuntimeState.state_key == ARXIV_RUNTIME_STATE_KEY
|
||||
)
|
||||
select(ArxivRuntimeState.cooldown_until).where(ArxivRuntimeState.state_key == ARXIV_RUNTIME_STATE_KEY)
|
||||
)
|
||||
cooldown_until = _normalize_datetime(result.scalar_one_or_none())
|
||||
remaining_seconds = _cooldown_remaining_seconds(cooldown_until, now_utc=timestamp)
|
||||
|
|
@ -85,10 +83,14 @@ async def _run_serialized_fetch(
|
|||
source_path=source_path,
|
||||
)
|
||||
structured_log(
|
||||
logger, "info", "arxiv.request_completed",
|
||||
logger,
|
||||
"info",
|
||||
"arxiv.request_completed",
|
||||
status_code=int(response.status_code),
|
||||
wait_seconds=wait_seconds,
|
||||
cooldown_remaining_seconds=_cooldown_remaining_seconds(runtime_state.cooldown_until, now_utc=datetime.now(timezone.utc)),
|
||||
cooldown_remaining_seconds=_cooldown_remaining_seconds(
|
||||
runtime_state.cooldown_until, now_utc=datetime.now(UTC)
|
||||
),
|
||||
source_path=source_path,
|
||||
)
|
||||
return response, hit_rate_limit
|
||||
|
|
@ -106,9 +108,7 @@ async def _acquire_arxiv_lock(db_session: AsyncSession) -> None:
|
|||
|
||||
async def _load_runtime_state_for_update(db_session: AsyncSession) -> ArxivRuntimeState:
|
||||
result = await db_session.execute(
|
||||
select(ArxivRuntimeState)
|
||||
.where(ArxivRuntimeState.state_key == ARXIV_RUNTIME_STATE_KEY)
|
||||
.with_for_update()
|
||||
select(ArxivRuntimeState).where(ArxivRuntimeState.state_key == ARXIV_RUNTIME_STATE_KEY).with_for_update()
|
||||
)
|
||||
state = result.scalar_one_or_none()
|
||||
if state is not None:
|
||||
|
|
@ -124,13 +124,27 @@ async def _wait_for_allowed_slot_or_raise(
|
|||
*,
|
||||
source_path: str,
|
||||
) -> float:
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
cooldown_seconds = _cooldown_remaining_seconds(runtime_state.cooldown_until, now_utc=now_utc)
|
||||
if cooldown_seconds > 0:
|
||||
structured_log(logger, "info", "arxiv.request_scheduled", wait_seconds=0.0, source_path=source_path, cooldown_remaining_seconds=cooldown_seconds)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"arxiv.request_scheduled",
|
||||
wait_seconds=0.0,
|
||||
source_path=source_path,
|
||||
cooldown_remaining_seconds=cooldown_seconds,
|
||||
)
|
||||
raise ArxivRateLimitError(f"arXiv global cooldown active ({cooldown_seconds:.0f}s remaining)")
|
||||
wait_seconds = _next_allowed_wait_seconds(runtime_state.next_allowed_at, now_utc=now_utc)
|
||||
structured_log(logger, "info", "arxiv.request_scheduled", wait_seconds=wait_seconds, source_path=source_path, cooldown_remaining_seconds=0.0)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"arxiv.request_scheduled",
|
||||
wait_seconds=wait_seconds,
|
||||
source_path=source_path,
|
||||
cooldown_remaining_seconds=0.0,
|
||||
)
|
||||
if wait_seconds > 0:
|
||||
await asyncio.sleep(wait_seconds)
|
||||
return wait_seconds
|
||||
|
|
@ -142,13 +156,15 @@ def _record_post_response_state(
|
|||
response_status: int,
|
||||
source_path: str,
|
||||
) -> bool:
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
runtime_state.next_allowed_at = now_utc + timedelta(seconds=_min_interval_seconds())
|
||||
if response_status == 429:
|
||||
cooldown_seconds = _cooldown_seconds()
|
||||
runtime_state.cooldown_until = now_utc + timedelta(seconds=cooldown_seconds)
|
||||
structured_log(
|
||||
logger, "warning", "arxiv.cooldown_activated",
|
||||
logger,
|
||||
"warning",
|
||||
"arxiv.cooldown_activated",
|
||||
cooldown_remaining_seconds=cooldown_seconds,
|
||||
source_path=source_path,
|
||||
)
|
||||
|
|
@ -176,7 +192,7 @@ def _normalize_datetime(value: datetime | None) -> datetime | None:
|
|||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value.replace(tzinfo=UTC)
|
||||
return value
|
||||
|
||||
|
||||
|
|
@ -186,5 +202,3 @@ def _min_interval_seconds() -> float:
|
|||
|
||||
def _cooldown_seconds() -> float:
|
||||
return max(float(settings.arxiv_rate_limit_cooldown_seconds), 0.0)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -9,8 +9,8 @@ from typing import TYPE_CHECKING
|
|||
|
||||
from crossref.restful import Etiquette, Works
|
||||
|
||||
from app.services.domains.doi.normalize import normalize_doi
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.doi.normalize import normalize_doi
|
||||
from app.settings import settings
|
||||
|
||||
if TYPE_CHECKING:
|
||||
|
|
@ -38,7 +38,7 @@ def _rate_limit_wait(min_interval_seconds: float) -> None:
|
|||
|
||||
|
||||
def _normalized_tokens(value: str) -> list[str]:
|
||||
lowered = value.lower().replace("’", "'").replace("“", "\"").replace("”", "\"")
|
||||
lowered = value.lower().replace("’", "'").replace("“", '"').replace("”", '"')
|
||||
lowered = NON_ALNUM_RE.sub(" ", lowered)
|
||||
return [token for token in TOKEN_RE.findall(lowered) if len(token) >= 3]
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
from app.services.domains.dbops.application import run_publication_link_repair
|
||||
from app.services.domains.dbops.integrity import collect_integrity_report
|
||||
from app.services.domains.dbops.near_duplicate_repair import (
|
||||
run_publication_near_duplicate_repair,
|
||||
)
|
||||
from app.services.domains.dbops.integrity import collect_integrity_report
|
||||
from app.services.domains.dbops.query import list_repair_jobs
|
||||
|
||||
__all__ = [
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import delete, exists, func, select, update
|
||||
|
|
@ -8,7 +8,6 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
|
||||
from app.db.models import DataRepairJob, IngestionQueueItem, Publication, ScholarProfile, ScholarPublication
|
||||
|
||||
|
||||
REPAIR_STATUS_PLANNED = "planned"
|
||||
REPAIR_STATUS_RUNNING = "running"
|
||||
REPAIR_STATUS_COMPLETED = "completed"
|
||||
|
|
@ -18,7 +17,7 @@ SCOPE_MODE_ALL_USERS = "all_users"
|
|||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _normalize_scope_mode(scope_mode: str) -> str:
|
||||
|
|
@ -105,11 +104,7 @@ async def _count_orphan_publications(db_session: AsyncSession) -> int:
|
|||
stmt = (
|
||||
select(func.count())
|
||||
.select_from(Publication)
|
||||
.where(
|
||||
~exists(
|
||||
select(1).where(ScholarPublication.publication_id == Publication.id)
|
||||
)
|
||||
)
|
||||
.where(~exists(select(1).where(ScholarPublication.publication_id == Publication.id)))
|
||||
)
|
||||
result = await db_session.execute(stmt)
|
||||
return int(result.scalar_one() or 0)
|
||||
|
|
@ -158,9 +153,7 @@ def _job_summary(
|
|||
|
||||
async def _delete_links_for_targets(db_session: AsyncSession, *, target_scholar_profile_ids: list[int]) -> int:
|
||||
result = await db_session.execute(
|
||||
delete(ScholarPublication).where(
|
||||
ScholarPublication.scholar_profile_id.in_(target_scholar_profile_ids)
|
||||
)
|
||||
delete(ScholarPublication).where(ScholarPublication.scholar_profile_id.in_(target_scholar_profile_ids))
|
||||
)
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
|
|
@ -171,9 +164,7 @@ async def _delete_queue_for_targets(
|
|||
user_id: int | None,
|
||||
target_scholar_profile_ids: list[int],
|
||||
) -> int:
|
||||
stmt = delete(IngestionQueueItem).where(
|
||||
IngestionQueueItem.scholar_profile_id.in_(target_scholar_profile_ids)
|
||||
)
|
||||
stmt = delete(IngestionQueueItem).where(IngestionQueueItem.scholar_profile_id.in_(target_scholar_profile_ids))
|
||||
if user_id is not None:
|
||||
stmt = stmt.where(IngestionQueueItem.user_id == user_id)
|
||||
result = await db_session.execute(stmt)
|
||||
|
|
@ -203,9 +194,7 @@ async def _reset_scholar_tracking_state(
|
|||
|
||||
async def _delete_orphan_publications(db_session: AsyncSession) -> int:
|
||||
result = await db_session.execute(
|
||||
delete(Publication).where(
|
||||
~exists(select(1).where(ScholarPublication.publication_id == Publication.id))
|
||||
)
|
||||
delete(Publication).where(~exists(select(1).where(ScholarPublication.publication_id == Publication.id)))
|
||||
)
|
||||
return int(result.rowcount or 0)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import func, select, text
|
||||
|
|
@ -168,7 +168,7 @@ async def collect_integrity_report(db_session: AsyncSession) -> dict[str, Any]:
|
|||
|
||||
return {
|
||||
"status": _status_from_issues(failures=failures, warnings=warnings),
|
||||
"checked_at": datetime.now(timezone.utc).isoformat(),
|
||||
"checked_at": datetime.now(UTC).isoformat(),
|
||||
"failures": failures,
|
||||
"warnings": warnings,
|
||||
"checks": checks,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -17,7 +17,7 @@ NEAR_DUP_DEFAULT_MAX_CLUSTERS = 25
|
|||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _normalized_cluster_keys(values: list[str] | None) -> list[str]:
|
||||
|
|
|
|||
|
|
@ -16,7 +16,5 @@ async def list_repair_jobs(
|
|||
limit: int = 50,
|
||||
) -> list[DataRepairJob]:
|
||||
bounded = _bounded_limit(limit)
|
||||
result = await db_session.execute(
|
||||
select(DataRepairJob).order_by(DataRepairJob.created_at.desc()).limit(bounded)
|
||||
)
|
||||
result = await db_session.execute(select(DataRepairJob).order_by(DataRepairJob.created_at.desc()).limit(bounded))
|
||||
return list(result.scalars())
|
||||
|
|
|
|||
|
|
@ -1,10 +1,10 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import or_, select, text
|
||||
|
|
@ -19,6 +19,11 @@ from app.db.models import (
|
|||
ScholarProfile,
|
||||
ScholarPublication,
|
||||
)
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.arxiv.errors import ArxivRateLimitError
|
||||
from app.services.domains.doi.normalize import first_doi_from_texts
|
||||
from app.services.domains.ingestion import queue as queue_service
|
||||
from app.services.domains.ingestion import safety as run_safety_service
|
||||
from app.services.domains.ingestion.constants import (
|
||||
FAILED_STATES,
|
||||
FAILURE_BUCKET_BLOCKED,
|
||||
|
|
@ -30,9 +35,6 @@ from app.services.domains.ingestion.constants import (
|
|||
RESUMABLE_PARTIAL_REASONS,
|
||||
RUN_LOCK_NAMESPACE,
|
||||
)
|
||||
from app.services.domains.arxiv.errors import ArxivRateLimitError
|
||||
from app.services.domains.doi.normalize import first_doi_from_texts
|
||||
from app.services.domains.publication_identifiers import application as identifier_service
|
||||
from app.services.domains.ingestion.fingerprints import (
|
||||
_build_body_excerpt,
|
||||
_dedupe_publication_candidates,
|
||||
|
|
@ -43,8 +45,6 @@ from app.services.domains.ingestion.fingerprints import (
|
|||
canonical_title_for_dedup,
|
||||
normalize_title,
|
||||
)
|
||||
from app.services.domains.ingestion import queue as queue_service
|
||||
from app.services.domains.ingestion import safety as run_safety_service
|
||||
from app.services.domains.ingestion.types import (
|
||||
PagedLoopState,
|
||||
PagedParseResult,
|
||||
|
|
@ -56,17 +56,17 @@ from app.services.domains.ingestion.types import (
|
|||
RunProgress,
|
||||
ScholarProcessingOutcome,
|
||||
)
|
||||
from app.services.domains.settings import application as user_settings_service
|
||||
from app.services.domains.publication_identifiers import application as identifier_service
|
||||
from app.services.domains.runs.events import run_events
|
||||
from app.services.domains.scholar.parser import (
|
||||
ParseState,
|
||||
ParsedProfilePage,
|
||||
ParseState,
|
||||
PublicationCandidate,
|
||||
ScholarParserError,
|
||||
parse_profile_page,
|
||||
)
|
||||
from app.services.domains.scholar.source import FetchResult, ScholarSource
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.settings import application as user_settings_service
|
||||
from app.settings import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -147,18 +147,20 @@ class ScholarIngestionService:
|
|||
user_id: int,
|
||||
trigger_type: RunTriggerType,
|
||||
) -> None:
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
previous = run_safety_service.get_safety_event_context(user_settings, now_utc=now_utc)
|
||||
if run_safety_service.clear_expired_cooldown(user_settings, now_utc=now_utc):
|
||||
await db_session.commit()
|
||||
await db_session.refresh(user_settings)
|
||||
structured_log(
|
||||
logger, "info", "ingestion.cooldown_cleared",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.cooldown_cleared",
|
||||
user_id=user_id,
|
||||
reason=previous.get("cooldown_reason"),
|
||||
cooldown_until=previous.get("cooldown_until"),
|
||||
)
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
if run_safety_service.is_cooldown_active(user_settings, now_utc=now_utc):
|
||||
await self._raise_safety_blocked_start(
|
||||
db_session,
|
||||
|
|
@ -183,7 +185,9 @@ class ScholarIngestionService:
|
|||
)
|
||||
await db_session.commit()
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.safety_policy_blocked_run_start",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.safety_policy_blocked_run_start",
|
||||
user_id=user_id,
|
||||
trigger_type=trigger_type.value,
|
||||
reason=safety_state.get("cooldown_reason"),
|
||||
|
|
@ -204,14 +208,9 @@ class ScholarIngestionService:
|
|||
start_cstart_by_scholar_id: dict[int, int] | None,
|
||||
) -> tuple[set[int] | None, dict[int, int]]:
|
||||
filtered_scholar_ids = (
|
||||
{int(value) for value in scholar_profile_ids}
|
||||
if scholar_profile_ids is not None
|
||||
else None
|
||||
{int(value) for value in scholar_profile_ids} if scholar_profile_ids is not None else None
|
||||
)
|
||||
start_cstart_map = {
|
||||
int(key): max(0, int(value))
|
||||
for key, value in (start_cstart_by_scholar_id or {}).items()
|
||||
}
|
||||
start_cstart_map = {int(key): max(0, int(value)) for key, value in (start_cstart_by_scholar_id or {}).items()}
|
||||
return filtered_scholar_ids, start_cstart_map
|
||||
|
||||
async def _load_target_scholars(
|
||||
|
|
@ -521,7 +520,9 @@ class ScholarIngestionService:
|
|||
page_logs=paged_parse_result.page_logs,
|
||||
)
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.scholar_parse_failed",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.scholar_parse_failed",
|
||||
scholar_profile_id=scholar.id,
|
||||
scholar_id=scholar.scholar_id,
|
||||
state=paged_parse_result.parsed_page.state.value,
|
||||
|
|
@ -679,7 +680,7 @@ class ScholarIngestionService:
|
|||
max_pages_per_scholar: int,
|
||||
page_size: int,
|
||||
) -> tuple[datetime, PagedParseResult, dict[str, Any]]:
|
||||
run_dt = datetime.now(timezone.utc)
|
||||
run_dt = datetime.now(UTC)
|
||||
paged_parse_result = await self._fetch_and_parse_all_pages_with_retry(
|
||||
scholar=scholar,
|
||||
run=run,
|
||||
|
|
@ -698,7 +699,9 @@ class ScholarIngestionService:
|
|||
self._apply_first_page_profile_metadata(scholar=scholar, paged_parse_result=paged_parse_result, run_dt=run_dt)
|
||||
parsed_page = paged_parse_result.parsed_page
|
||||
structured_log(
|
||||
logger, "info", "ingestion.scholar_parsed",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.scholar_parsed",
|
||||
user_id=user_id,
|
||||
crawl_run_id=run.id,
|
||||
scholar_profile_id=scholar.id,
|
||||
|
|
@ -796,9 +799,7 @@ class ScholarIngestionService:
|
|||
scholar_results=progress.scholar_results,
|
||||
scholar_profile_id=scholar_profile_id,
|
||||
)
|
||||
next_succeeded, next_failed, next_partial = ScholarIngestionService._result_counters(
|
||||
outcome.result_entry
|
||||
)
|
||||
next_succeeded, next_failed, next_partial = ScholarIngestionService._result_counters(outcome.result_entry)
|
||||
if prior_index is None:
|
||||
progress.scholar_results.append(outcome.result_entry)
|
||||
ScholarIngestionService._adjust_progress_counts(
|
||||
|
|
@ -809,9 +810,7 @@ class ScholarIngestionService:
|
|||
)
|
||||
return
|
||||
previous_entry = progress.scholar_results[prior_index]
|
||||
prev_succeeded, prev_failed, prev_partial = ScholarIngestionService._result_counters(
|
||||
previous_entry
|
||||
)
|
||||
prev_succeeded, prev_failed, prev_partial = ScholarIngestionService._result_counters(previous_entry)
|
||||
progress.scholar_results[prior_index] = outcome.result_entry
|
||||
ScholarIngestionService._adjust_progress_counts(
|
||||
progress=progress,
|
||||
|
|
@ -829,7 +828,7 @@ class ScholarIngestionService:
|
|||
exc: Exception,
|
||||
) -> ScholarProcessingOutcome:
|
||||
scholar.last_run_status = RunStatus.FAILED
|
||||
scholar.last_run_dt = datetime.now(timezone.utc)
|
||||
scholar.last_run_dt = datetime.now(UTC)
|
||||
logger.exception(
|
||||
"ingestion.scholar_unexpected_failure",
|
||||
extra={
|
||||
|
|
@ -933,7 +932,7 @@ class ScholarIngestionService:
|
|||
) -> None:
|
||||
pre_apply_state = run_safety_service.get_safety_event_context(
|
||||
user_settings,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
)
|
||||
safety_state, cooldown_trigger_reason = run_safety_service.apply_run_safety_outcome(
|
||||
user_settings,
|
||||
|
|
@ -944,11 +943,13 @@ class ScholarIngestionService:
|
|||
network_failure_threshold=alert_summary.network_failure_threshold,
|
||||
blocked_cooldown_seconds=settings.ingestion_safety_cooldown_blocked_seconds,
|
||||
network_cooldown_seconds=settings.ingestion_safety_cooldown_network_seconds,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
)
|
||||
if cooldown_trigger_reason is not None:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.safety_cooldown_entered",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.safety_cooldown_entered",
|
||||
user_id=user_id,
|
||||
crawl_run_id=int(run.id),
|
||||
reason=cooldown_trigger_reason,
|
||||
|
|
@ -962,7 +963,9 @@ class ScholarIngestionService:
|
|||
)
|
||||
elif pre_apply_state.get("cooldown_active") and not safety_state.get("cooldown_active"):
|
||||
structured_log(
|
||||
logger, "info", "ingestion.cooldown_cleared",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.cooldown_cleared",
|
||||
user_id=user_id,
|
||||
crawl_run_id=int(run.id),
|
||||
reason=pre_apply_state.get("cooldown_reason"),
|
||||
|
|
@ -979,7 +982,7 @@ class ScholarIngestionService:
|
|||
alert_summary: RunAlertSummary,
|
||||
idempotency_key: str | None,
|
||||
) -> None:
|
||||
run.end_dt = datetime.now(timezone.utc)
|
||||
run.end_dt = datetime.now(UTC)
|
||||
if run.status != RunStatus.CANCELED:
|
||||
run.status = self._resolve_run_status(
|
||||
scholar_count=len(scholars),
|
||||
|
|
@ -1062,7 +1065,26 @@ class ScholarIngestionService:
|
|||
new_publication_count=run.new_pub_count,
|
||||
)
|
||||
|
||||
async def _initialize_run_for_user(self, db_session: AsyncSession, *, user_id: int, trigger_type: RunTriggerType, scholar_profile_ids: set[int] | None, start_cstart_by_scholar_id: dict[int, int] | None, request_delay_seconds: int, network_error_retries: int, retry_backoff_seconds: float, rate_limit_retries: int, rate_limit_backoff_seconds: float, max_pages_per_scholar: int, page_size: int, idempotency_key: str | None, alert_blocked_failure_threshold: int, alert_network_failure_threshold: int, alert_retry_scheduled_threshold: int) -> tuple[Any, CrawlRun, list[ScholarProfile], dict[int, int]]:
|
||||
async def _initialize_run_for_user(
|
||||
self,
|
||||
db_session: AsyncSession,
|
||||
*,
|
||||
user_id: int,
|
||||
trigger_type: RunTriggerType,
|
||||
scholar_profile_ids: set[int] | None,
|
||||
start_cstart_by_scholar_id: dict[int, int] | None,
|
||||
request_delay_seconds: int,
|
||||
network_error_retries: int,
|
||||
retry_backoff_seconds: float,
|
||||
rate_limit_retries: int,
|
||||
rate_limit_backoff_seconds: float,
|
||||
max_pages_per_scholar: int,
|
||||
page_size: int,
|
||||
idempotency_key: str | None,
|
||||
alert_blocked_failure_threshold: int,
|
||||
alert_network_failure_threshold: int,
|
||||
alert_retry_scheduled_threshold: int,
|
||||
) -> tuple[Any, CrawlRun, list[ScholarProfile], dict[int, int]]:
|
||||
user_settings = await self._load_user_settings_for_run(
|
||||
db_session,
|
||||
user_id=user_id,
|
||||
|
|
@ -1116,7 +1138,9 @@ class ScholarIngestionService:
|
|||
alert_retry_scheduled_threshold: int,
|
||||
) -> CrawlRun:
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_started",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.run_started",
|
||||
user_id=user_id,
|
||||
trigger_type=trigger_type.value,
|
||||
scholar_count=len(scholars),
|
||||
|
|
@ -1143,9 +1167,7 @@ class ScholarIngestionService:
|
|||
except IntegrityError as exc:
|
||||
if _is_active_run_integrity_error(exc):
|
||||
await db_session.rollback()
|
||||
raise RunAlreadyInProgressError(
|
||||
f"Run already in progress for user_id={user_id}."
|
||||
) from exc
|
||||
raise RunAlreadyInProgressError(f"Run already in progress for user_id={user_id}.") from exc
|
||||
raise
|
||||
return run
|
||||
|
||||
|
|
@ -1177,8 +1199,11 @@ class ScholarIngestionService:
|
|||
await db_session.refresh(run)
|
||||
if run.status == RunStatus.CANCELED:
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_canceled",
|
||||
run_id=run.id, user_id=user_id,
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.run_canceled",
|
||||
run_id=run.id,
|
||||
user_id=user_id,
|
||||
)
|
||||
return progress
|
||||
await self._wait_between_scholars(index=index, request_delay_seconds=request_delay_seconds)
|
||||
|
|
@ -1217,8 +1242,11 @@ class ScholarIngestionService:
|
|||
await db_session.refresh(run)
|
||||
if run.status == RunStatus.CANCELED:
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_canceled",
|
||||
run_id=run.id, user_id=user_id,
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.run_canceled",
|
||||
run_id=run.id,
|
||||
user_id=user_id,
|
||||
)
|
||||
break
|
||||
await self._wait_between_scholars(index=index, request_delay_seconds=request_delay_seconds)
|
||||
|
|
@ -1263,7 +1291,9 @@ class ScholarIngestionService:
|
|||
)
|
||||
if alert_summary.alert_flags["blocked_failure_threshold_exceeded"]:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.alert_blocked_failure_threshold_exceeded",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.alert_blocked_failure_threshold_exceeded",
|
||||
user_id=user_id,
|
||||
crawl_run_id=int(run.id),
|
||||
blocked_failure_count=alert_summary.blocked_failure_count,
|
||||
|
|
@ -1271,7 +1301,9 @@ class ScholarIngestionService:
|
|||
)
|
||||
if alert_summary.alert_flags["network_failure_threshold_exceeded"]:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.alert_network_failure_threshold_exceeded",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.alert_network_failure_threshold_exceeded",
|
||||
user_id=user_id,
|
||||
crawl_run_id=int(run.id),
|
||||
network_failure_count=alert_summary.network_failure_count,
|
||||
|
|
@ -1279,7 +1311,9 @@ class ScholarIngestionService:
|
|||
)
|
||||
if alert_summary.alert_flags["retry_scheduled_threshold_exceeded"]:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.alert_retry_scheduled_threshold_exceeded",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.alert_retry_scheduled_threshold_exceeded",
|
||||
user_id=user_id,
|
||||
crawl_run_id=int(run.id),
|
||||
retries_scheduled_count=failure_summary.retries_scheduled_count,
|
||||
|
|
@ -1319,7 +1353,9 @@ class ScholarIngestionService:
|
|||
effective_delay = self._effective_request_delay_seconds(request_delay_seconds)
|
||||
if effective_delay != _int_or_default(request_delay_seconds, effective_delay):
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.delay_coerced",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.delay_coerced",
|
||||
user_id=user_id,
|
||||
requested_request_delay_seconds=_int_or_default(request_delay_seconds, 0),
|
||||
effective_request_delay_seconds=effective_delay,
|
||||
|
|
@ -1332,8 +1368,12 @@ class ScholarIngestionService:
|
|||
request_delay_seconds=effective_delay,
|
||||
network_error_retries=network_error_retries,
|
||||
retry_backoff_seconds=retry_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries if rate_limit_retries is not None else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds if rate_limit_backoff_seconds is not None else settings.ingestion_rate_limit_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries
|
||||
if rate_limit_retries is not None
|
||||
else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds
|
||||
if rate_limit_backoff_seconds is not None
|
||||
else settings.ingestion_rate_limit_backoff_seconds,
|
||||
max_pages_per_scholar=max_pages_per_scholar,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
|
@ -1388,8 +1428,12 @@ class ScholarIngestionService:
|
|||
request_delay_seconds=request_delay_seconds,
|
||||
network_error_retries=network_error_retries,
|
||||
retry_backoff_seconds=retry_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries if rate_limit_retries is not None else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds if rate_limit_backoff_seconds is not None else settings.ingestion_rate_limit_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries
|
||||
if rate_limit_retries is not None
|
||||
else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds
|
||||
if rate_limit_backoff_seconds is not None
|
||||
else settings.ingestion_rate_limit_backoff_seconds,
|
||||
max_pages_per_scholar=max_pages_per_scholar,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
|
@ -1427,7 +1471,9 @@ class ScholarIngestionService:
|
|||
run.status = RunStatus.RESOLVING
|
||||
await db_session.commit()
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_completed",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.run_completed",
|
||||
user_id=user_id,
|
||||
crawl_run_id=run.id,
|
||||
status=run.status.value,
|
||||
|
|
@ -1471,7 +1517,8 @@ class ScholarIngestionService:
|
|||
|
||||
scholar_ids = [s.id for s in scholars]
|
||||
scholars_result = await db_session.execute(
|
||||
select(ScholarProfile).where(ScholarProfile.id.in_(scholar_ids))
|
||||
select(ScholarProfile)
|
||||
.where(ScholarProfile.id.in_(scholar_ids))
|
||||
.order_by(ScholarProfile.created_at.asc(), ScholarProfile.id.asc())
|
||||
)
|
||||
return run, user_settings, list(scholars_result.scalars().all())
|
||||
|
|
@ -1481,7 +1528,7 @@ class ScholarIngestionService:
|
|||
run_to_fail = await cleanup_session.get(CrawlRun, run_id)
|
||||
if run_to_fail:
|
||||
run_to_fail.status = RunStatus.FAILED
|
||||
run_to_fail.end_dt = datetime.now(timezone.utc)
|
||||
run_to_fail.end_dt = datetime.now(UTC)
|
||||
run_to_fail.error_log["terminal_exception"] = str(exc)
|
||||
await cleanup_session.commit()
|
||||
|
||||
|
|
@ -1573,8 +1620,12 @@ class ScholarIngestionService:
|
|||
request_delay_seconds=self._effective_request_delay_seconds(request_delay_seconds),
|
||||
network_error_retries=network_error_retries,
|
||||
retry_backoff_seconds=retry_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries if rate_limit_retries is not None else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds if rate_limit_backoff_seconds is not None else settings.ingestion_rate_limit_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries
|
||||
if rate_limit_retries is not None
|
||||
else settings.ingestion_rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds
|
||||
if rate_limit_backoff_seconds is not None
|
||||
else settings.ingestion_rate_limit_backoff_seconds,
|
||||
max_pages_per_scholar=max_pages_per_scholar,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
|
@ -1628,7 +1679,9 @@ class ScholarIngestionService:
|
|||
await db_session.commit()
|
||||
|
||||
structured_log(
|
||||
logger, "info", "ingestion.run_completed",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.run_completed",
|
||||
user_id=user_id,
|
||||
crawl_run_id=run.id,
|
||||
status=run.status.value,
|
||||
|
|
@ -1663,8 +1716,7 @@ class ScholarIngestionService:
|
|||
return await self._source.fetch_profile_html(scholar_id)
|
||||
return FetchResult(
|
||||
requested_url=(
|
||||
"https://scholar.google.com/citations"
|
||||
f"?hl=en&user={scholar_id}&cstart={cstart}&pagesize={page_size}"
|
||||
f"https://scholar.google.com/citations?hl=en&user={scholar_id}&cstart={cstart}&pagesize={page_size}"
|
||||
),
|
||||
status_code=None,
|
||||
final_url=None,
|
||||
|
|
@ -1682,8 +1734,7 @@ class ScholarIngestionService:
|
|||
)
|
||||
return FetchResult(
|
||||
requested_url=(
|
||||
"https://scholar.google.com/citations"
|
||||
f"?hl=en&user={scholar_id}&cstart={cstart}&pagesize={page_size}"
|
||||
f"https://scholar.google.com/citations?hl=en&user={scholar_id}&cstart={cstart}&pagesize={page_size}"
|
||||
),
|
||||
status_code=None,
|
||||
final_url=None,
|
||||
|
|
@ -1702,7 +1753,10 @@ class ScholarIngestionService:
|
|||
) -> bool:
|
||||
if parsed_page.state == ParseState.NETWORK_ERROR:
|
||||
return network_attempt_count <= network_error_retries
|
||||
if parsed_page.state == ParseState.BLOCKED_OR_CAPTCHA and parsed_page.state_reason == "blocked_http_429_rate_limited":
|
||||
if (
|
||||
parsed_page.state == ParseState.BLOCKED_OR_CAPTCHA
|
||||
and parsed_page.state_reason == "blocked_http_429_rate_limited"
|
||||
):
|
||||
return rate_limit_attempt_count <= rate_limit_retries
|
||||
return False
|
||||
|
||||
|
|
@ -1725,7 +1779,9 @@ class ScholarIngestionService:
|
|||
attempt_label = network_attempt_count
|
||||
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.scholar_retry_scheduled",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.scholar_retry_scheduled",
|
||||
scholar_id=scholar_id,
|
||||
cstart=cstart,
|
||||
attempt_count=attempt_label,
|
||||
|
|
@ -1763,20 +1819,25 @@ class ScholarIngestionService:
|
|||
if parsed_page.state == ParseState.NETWORK_ERROR:
|
||||
network_attempts += 1
|
||||
total_attempts = network_attempts
|
||||
elif parsed_page.state == ParseState.BLOCKED_OR_CAPTCHA and parsed_page.state_reason == "blocked_http_429_rate_limited":
|
||||
elif (
|
||||
parsed_page.state == ParseState.BLOCKED_OR_CAPTCHA
|
||||
and parsed_page.state_reason == "blocked_http_429_rate_limited"
|
||||
):
|
||||
rate_limit_attempts += 1
|
||||
total_attempts = rate_limit_attempts
|
||||
else:
|
||||
total_attempts = network_attempts + rate_limit_attempts + 1
|
||||
|
||||
attempt_log.append({
|
||||
attempt_log.append(
|
||||
{
|
||||
"attempt": total_attempts,
|
||||
"cstart": cstart,
|
||||
"state": parsed_page.state.value,
|
||||
"state_reason": parsed_page.state_reason,
|
||||
"status_code": fetch_result.status_code,
|
||||
"fetch_error": fetch_result.error,
|
||||
})
|
||||
}
|
||||
)
|
||||
|
||||
if not self._should_retry_page_fetch(
|
||||
parsed_page=parsed_page,
|
||||
|
|
@ -1995,7 +2056,8 @@ class ScholarIngestionService:
|
|||
) -> None:
|
||||
state.pages_attempted += 1
|
||||
state.attempt_log.extend(page_attempt_log)
|
||||
state.page_logs.append({
|
||||
state.page_logs.append(
|
||||
{
|
||||
"page": state.pages_attempted,
|
||||
"cstart": state.current_cstart,
|
||||
"state": parsed_page.state.value,
|
||||
|
|
@ -2006,7 +2068,8 @@ class ScholarIngestionService:
|
|||
"has_show_more_button": parsed_page.has_show_more_button,
|
||||
"warning_codes": parsed_page.warnings,
|
||||
"attempt_count": len(page_attempt_log),
|
||||
})
|
||||
}
|
||||
)
|
||||
state.fetch_result = fetch_result
|
||||
state.parsed_page = parsed_page
|
||||
|
||||
|
|
@ -2052,7 +2115,8 @@ class ScholarIngestionService:
|
|||
)
|
||||
first_page_fingerprint_sha256 = build_initial_page_fingerprint(parsed_page)
|
||||
attempt_log = list(first_attempt_log)
|
||||
page_logs = [{
|
||||
page_logs = [
|
||||
{
|
||||
"page": 1,
|
||||
"cstart": start_cstart,
|
||||
"state": parsed_page.state.value,
|
||||
|
|
@ -2063,7 +2127,8 @@ class ScholarIngestionService:
|
|||
"has_show_more_button": parsed_page.has_show_more_button,
|
||||
"warning_codes": parsed_page.warnings,
|
||||
"attempt_count": len(first_attempt_log),
|
||||
}]
|
||||
}
|
||||
]
|
||||
return fetch_result, parsed_page, first_page_fingerprint_sha256, attempt_log, page_logs
|
||||
|
||||
async def _paginate_loop(
|
||||
|
|
@ -2098,7 +2163,9 @@ class ScholarIngestionService:
|
|||
await db_session.refresh(run)
|
||||
if run.status == RunStatus.CANCELED:
|
||||
structured_log(
|
||||
logger, "info", "ingestion.pagination_canceled",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.pagination_canceled",
|
||||
run_id=run.id,
|
||||
)
|
||||
self._set_truncated_state(
|
||||
|
|
@ -2217,12 +2284,17 @@ class ScholarIngestionService:
|
|||
rate_limit_backoff_seconds: float,
|
||||
max_pages: int,
|
||||
page_size: int,
|
||||
previous_initial_page_fingerprint_sha256: str | None = None
|
||||
previous_initial_page_fingerprint_sha256: str | None = None,
|
||||
) -> PagedParseResult:
|
||||
bounded_max_pages = max(1, int(max_pages))
|
||||
bounded_page_size = max(1, int(page_size))
|
||||
fetch_result, parsed_page, first_page_fingerprint_sha256, attempt_log, page_logs = (
|
||||
await self._fetch_initial_page_context(
|
||||
(
|
||||
fetch_result,
|
||||
parsed_page,
|
||||
first_page_fingerprint_sha256,
|
||||
attempt_log,
|
||||
page_logs,
|
||||
) = await self._fetch_initial_page_context(
|
||||
scholar_id=scholar.scholar_id,
|
||||
start_cstart=start_cstart,
|
||||
bounded_page_size=bounded_page_size,
|
||||
|
|
@ -2230,7 +2302,7 @@ class ScholarIngestionService:
|
|||
retry_backoff_seconds=retry_backoff_seconds,
|
||||
rate_limit_retries=rate_limit_retries,
|
||||
rate_limit_backoff_seconds=rate_limit_backoff_seconds,
|
||||
))
|
||||
)
|
||||
shortcut_result = self._short_circuit_initial_page(
|
||||
start_cstart=start_cstart,
|
||||
previous_initial_page_fingerprint_sha256=previous_initial_page_fingerprint_sha256,
|
||||
|
|
@ -2275,9 +2347,7 @@ class ScholarIngestionService:
|
|||
*,
|
||||
run_id: int,
|
||||
) -> bool:
|
||||
check_result = await db_session.execute(
|
||||
select(CrawlRun.status).where(CrawlRun.id == run_id)
|
||||
)
|
||||
check_result = await db_session.execute(select(CrawlRun.status).where(CrawlRun.id == run_id))
|
||||
status = check_result.scalar_one_or_none()
|
||||
if status is None:
|
||||
raise RuntimeError(f"Missing crawl_run for run_id={run_id}.")
|
||||
|
|
@ -2302,12 +2372,10 @@ class ScholarIngestionService:
|
|||
)
|
||||
from app.services.domains.openalex.matching import find_best_match
|
||||
|
||||
run_result = await db_session.execute(
|
||||
select(CrawlRun.user_id).where(CrawlRun.id == run_id)
|
||||
)
|
||||
run_result = await db_session.execute(select(CrawlRun.user_id).where(CrawlRun.id == run_id))
|
||||
user_id = run_result.scalar_one()
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
now = datetime.now(UTC)
|
||||
cooldown_threshold = now - timedelta(days=7)
|
||||
|
||||
stmt = (
|
||||
|
|
@ -2355,21 +2423,28 @@ class ScholarIngestionService:
|
|||
)
|
||||
except OpenAlexBudgetExhaustedError:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.openalex_budget_exhausted",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.openalex_budget_exhausted",
|
||||
run_id=run_id,
|
||||
)
|
||||
break # Stop all enrichment — budget won't reset until midnight UTC
|
||||
except OpenAlexRateLimitError:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.openalex_rate_limited",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.openalex_rate_limited",
|
||||
run_id=run_id,
|
||||
)
|
||||
await asyncio.sleep(60)
|
||||
continue
|
||||
except Exception as e:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.openalex_enrichment_failed",
|
||||
error=str(e), run_id=run_id,
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.openalex_enrichment_failed",
|
||||
error=str(e),
|
||||
run_id=run_id,
|
||||
)
|
||||
continue
|
||||
|
||||
|
|
@ -2406,7 +2481,9 @@ class ScholarIngestionService:
|
|||
merge_count = await sweep_identifier_duplicates(db_session)
|
||||
if merge_count:
|
||||
structured_log(
|
||||
logger, "info", "ingestion.identifier_dedup_sweep",
|
||||
logger,
|
||||
"info",
|
||||
"ingestion.identifier_dedup_sweep",
|
||||
merged_count=merge_count,
|
||||
run_id=run_id,
|
||||
)
|
||||
|
|
@ -2444,7 +2521,9 @@ class ScholarIngestionService:
|
|||
return True
|
||||
except ArxivRateLimitError:
|
||||
structured_log(
|
||||
logger, "warning", "ingestion.arxiv_rate_limited",
|
||||
logger,
|
||||
"warning",
|
||||
"ingestion.arxiv_rate_limited",
|
||||
run_id=run_id,
|
||||
publication_id=int(publication.id),
|
||||
detail="arXiv temporarily disabled for remaining enrichment pass",
|
||||
|
|
@ -2506,6 +2585,7 @@ class ScholarIngestionService:
|
|||
enriched: list[PublicationCandidate] = []
|
||||
|
||||
import re
|
||||
|
||||
for i in range(0, len(publications), batch_size):
|
||||
batch = publications[i : i + batch_size]
|
||||
|
||||
|
|
@ -2526,13 +2606,11 @@ class ScholarIngestionService:
|
|||
|
||||
query = "|".join(t for t in titles)
|
||||
try:
|
||||
openalex_works = await client.get_works_by_filter(
|
||||
{"title.search": query}, limit=batch_size * 3
|
||||
)
|
||||
openalex_works = await client.get_works_by_filter({"title.search": query}, limit=batch_size * 3)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"ingestion.openalex_enrichment_failed",
|
||||
extra={"error": str(e), "batch_size": len(batch), "scholar_id": scholar.id}
|
||||
extra={"error": str(e), "batch_size": len(batch), "scholar_id": scholar.id},
|
||||
)
|
||||
openalex_works = []
|
||||
|
||||
|
|
@ -2541,7 +2619,7 @@ class ScholarIngestionService:
|
|||
target_title=p.title,
|
||||
target_year=p.year,
|
||||
target_authors=p.authors_text or (scholar.display_name or scholar.scholar_id),
|
||||
candidates=openalex_works
|
||||
candidates=openalex_works,
|
||||
)
|
||||
if match:
|
||||
new_p = PublicationCandidate(
|
||||
|
|
@ -2627,7 +2705,7 @@ class ScholarIngestionService:
|
|||
"pub_url": publication.pub_url,
|
||||
"scholar_profile_id": scholar.id,
|
||||
"scholar_label": scholar.display_name or scholar.scholar_id,
|
||||
"first_seen_at": datetime.now(timezone.utc).isoformat(),
|
||||
"first_seen_at": datetime.now(UTC).isoformat(),
|
||||
"new_publication_count": int(run.new_pub_count or 0),
|
||||
},
|
||||
)
|
||||
|
|
@ -2647,9 +2725,7 @@ class ScholarIngestionService:
|
|||
) -> Publication | None:
|
||||
if not cluster_id:
|
||||
return None
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(Publication.cluster_id == cluster_id)
|
||||
)
|
||||
result = await db_session.execute(select(Publication).where(Publication.cluster_id == cluster_id))
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def _find_publication_by_fingerprint(
|
||||
|
|
@ -2658,9 +2734,7 @@ class ScholarIngestionService:
|
|||
*,
|
||||
fingerprint: str,
|
||||
) -> Publication | None:
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(Publication.fingerprint_sha256 == fingerprint)
|
||||
)
|
||||
result = await db_session.execute(select(Publication).where(Publication.fingerprint_sha256 == fingerprint))
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -2685,9 +2759,7 @@ class ScholarIngestionService:
|
|||
canonical_title_hash: str,
|
||||
) -> Publication | None:
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(
|
||||
Publication.canonical_title_hash == canonical_title_hash
|
||||
)
|
||||
select(Publication).where(Publication.canonical_title_hash == canonical_title_hash)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
|
@ -2816,9 +2888,7 @@ class ScholarIngestionService:
|
|||
) -> tuple[str | None, int]:
|
||||
if outcome == "partial":
|
||||
reason = (pagination_truncated_reason or "").strip()
|
||||
if reason in RESUMABLE_PARTIAL_REASONS or reason.startswith(
|
||||
RESUMABLE_PARTIAL_REASON_PREFIXES
|
||||
):
|
||||
if reason in RESUMABLE_PARTIAL_REASONS or reason.startswith(RESUMABLE_PARTIAL_REASON_PREFIXES):
|
||||
return reason, queue_service.normalize_cstart(
|
||||
continuation_cstart if continuation_cstart is not None else fallback_cstart
|
||||
)
|
||||
|
|
@ -2852,13 +2922,9 @@ class ScholarIngestionService:
|
|||
"has_show_more_button": parsed_page.has_show_more_button,
|
||||
"has_operation_error_banner": parsed_page.has_operation_error_banner,
|
||||
"warning_codes": parsed_page.warnings,
|
||||
"marker_counts_nonzero": {
|
||||
key: value for key, value in parsed_page.marker_counts.items() if value > 0
|
||||
},
|
||||
"marker_counts_nonzero": {key: value for key, value in parsed_page.marker_counts.items() if value > 0},
|
||||
"body_length": len(fetch_result.body),
|
||||
"body_sha256": hashlib.sha256(fetch_result.body.encode("utf-8")).hexdigest()
|
||||
if fetch_result.body
|
||||
else None,
|
||||
"body_sha256": hashlib.sha256(fetch_result.body.encode("utf-8")).hexdigest() if fetch_result.body else None,
|
||||
"body_excerpt": _build_body_excerpt(fetch_result.body),
|
||||
"attempt_log": attempt_log,
|
||||
}
|
||||
|
|
@ -2876,9 +2942,7 @@ class ScholarIngestionService:
|
|||
user_id: int,
|
||||
) -> bool:
|
||||
result = await db_session.execute(
|
||||
text(
|
||||
"SELECT pg_try_advisory_xact_lock(:namespace, :user_key)"
|
||||
),
|
||||
text("SELECT pg_try_advisory_xact_lock(:namespace, :user_key)"),
|
||||
{
|
||||
"namespace": RUN_LOCK_NAMESPACE,
|
||||
"user_key": int(user_id),
|
||||
|
|
|
|||
|
|
@ -3,8 +3,8 @@ from __future__ import annotations
|
|||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
import unicodedata
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from app.services.domains.ingestion.constants import (
|
||||
|
|
@ -14,7 +14,7 @@ from app.services.domains.ingestion.constants import (
|
|||
TITLE_ALNUM_RE,
|
||||
WORD_RE,
|
||||
)
|
||||
from app.services.domains.scholar.parser import ParseState, ParsedProfilePage, PublicationCandidate
|
||||
from app.services.domains.scholar.parser import ParsedProfilePage, ParseState, PublicationCandidate
|
||||
|
||||
# Scholar-specific noise patterns stripped before canonical comparison.
|
||||
# Applied in order; each targets a different Scholar metadata injection style.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -115,7 +115,7 @@ async def upsert_job(
|
|||
run_id: int | None,
|
||||
delay_seconds: int,
|
||||
) -> IngestionQueueItem:
|
||||
now = datetime.now(timezone.utc)
|
||||
now = datetime.now(UTC)
|
||||
next_attempt_dt = now + timedelta(seconds=max(0, int(delay_seconds)))
|
||||
item = await _get_item_for_user_scholar(
|
||||
db_session,
|
||||
|
|
@ -171,9 +171,7 @@ async def delete_job_by_id(
|
|||
*,
|
||||
job_id: int,
|
||||
) -> bool:
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return False
|
||||
|
|
@ -222,10 +220,8 @@ async def increment_attempt_count(
|
|||
*,
|
||||
job_id: int,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
@ -239,10 +235,8 @@ async def reset_attempt_count(
|
|||
*,
|
||||
job_id: int,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
@ -259,10 +253,8 @@ async def reschedule_job(
|
|||
reason: str,
|
||||
error: str | None = None,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
@ -282,10 +274,8 @@ async def mark_retrying(
|
|||
job_id: int,
|
||||
reason: str | None = None,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
@ -305,10 +295,8 @@ async def mark_dropped(
|
|||
reason: str,
|
||||
error: str | None = None,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
@ -329,10 +317,8 @@ async def mark_queued_now(
|
|||
reason: str,
|
||||
reset_attempt_count: bool = False,
|
||||
) -> IngestionQueueItem | None:
|
||||
now = datetime.now(timezone.utc)
|
||||
result = await db_session.execute(
|
||||
select(IngestionQueueItem).where(IngestionQueueItem.id == job_id)
|
||||
)
|
||||
now = datetime.now(UTC)
|
||||
result = await db_session.execute(select(IngestionQueueItem).where(IngestionQueueItem.id == job_id))
|
||||
item = result.scalar_one_or_none()
|
||||
if item is None:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
from app.db.models import UserSetting
|
||||
|
|
@ -18,7 +18,7 @@ _COUNTER_LAST_EVALUATED_RUN_ID = "last_evaluated_run_id"
|
|||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _safe_int(value: Any, default: int = 0) -> int:
|
||||
|
|
@ -41,8 +41,8 @@ def _normalize_datetime(value: datetime | None) -> datetime | None:
|
|||
if value is None:
|
||||
return None
|
||||
if value.tzinfo is None:
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
return value.astimezone(timezone.utc)
|
||||
return value.replace(tzinfo=UTC)
|
||||
return value.astimezone(UTC)
|
||||
|
||||
|
||||
def _state_dict(settings: UserSetting) -> dict[str, Any]:
|
||||
|
|
@ -100,9 +100,7 @@ def _recommended_action(reason: str | None) -> str | None:
|
|||
"increase request delay, and avoid repeated manual retries."
|
||||
)
|
||||
if reason == COOLDOWN_REASON_NETWORK_FAILURE_THRESHOLD:
|
||||
return (
|
||||
"Network failures crossed the threshold. Verify connectivity and retry after cooldown."
|
||||
)
|
||||
return "Network failures crossed the threshold. Verify connectivity and retry after cooldown."
|
||||
return None
|
||||
|
||||
|
||||
|
|
@ -159,14 +157,10 @@ def _update_run_counters(
|
|||
counters[_COUNTER_LAST_NETWORK_FAILURE_COUNT] = bounded_network_failures
|
||||
counters[_COUNTER_LAST_EVALUATED_RUN_ID] = int(run_id)
|
||||
counters[_COUNTER_CONSECUTIVE_BLOCKED_RUNS] = (
|
||||
int(counters[_COUNTER_CONSECUTIVE_BLOCKED_RUNS]) + 1
|
||||
if bounded_blocked_failures > 0
|
||||
else 0
|
||||
int(counters[_COUNTER_CONSECUTIVE_BLOCKED_RUNS]) + 1 if bounded_blocked_failures > 0 else 0
|
||||
)
|
||||
counters[_COUNTER_CONSECUTIVE_NETWORK_RUNS] = (
|
||||
int(counters[_COUNTER_CONSECUTIVE_NETWORK_RUNS]) + 1
|
||||
if bounded_network_failures > 0
|
||||
else 0
|
||||
int(counters[_COUNTER_CONSECUTIVE_NETWORK_RUNS]) + 1 if bounded_network_failures > 0 else 0
|
||||
)
|
||||
return bounded_blocked_failures, bounded_network_failures
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -17,24 +17,22 @@ from app.db.models import (
|
|||
UserSetting,
|
||||
)
|
||||
from app.db.session import get_session_factory
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.ingestion import queue as queue_service
|
||||
from app.services.domains.ingestion.application import (
|
||||
RunAlreadyInProgressError,
|
||||
RunBlockedBySafetyPolicyError,
|
||||
ScholarIngestionService,
|
||||
)
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.settings import application as user_settings_service
|
||||
from app.services.domains.scholar.source import LiveScholarSource
|
||||
from app.services.domains.settings import application as user_settings_service
|
||||
from app.settings import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _request_delay_floor_seconds() -> int:
|
||||
return user_settings_service.resolve_request_delay_minimum(
|
||||
settings.ingestion_min_request_delay_seconds
|
||||
)
|
||||
return user_settings_service.resolve_request_delay_minimum(settings.ingestion_min_request_delay_seconds)
|
||||
|
||||
|
||||
def _effective_request_delay_seconds(value: int | None) -> int:
|
||||
|
|
@ -96,7 +94,9 @@ class SchedulerService:
|
|||
return
|
||||
self._task = asyncio.create_task(self._run_loop(), name="scholarr-scheduler")
|
||||
structured_log(
|
||||
logger, "info", "scheduler.started",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.started",
|
||||
tick_seconds=self._tick_seconds,
|
||||
network_error_retries=self._network_error_retries,
|
||||
retry_backoff_seconds=self._retry_backoff_seconds,
|
||||
|
|
@ -143,7 +143,7 @@ class SchedulerService:
|
|||
candidates = await self._load_candidates()
|
||||
if not candidates:
|
||||
return
|
||||
now = datetime.now(timezone.utc)
|
||||
now = datetime.now(UTC)
|
||||
for candidate in candidates:
|
||||
if not await self._is_due(candidate, now=now):
|
||||
continue
|
||||
|
|
@ -170,10 +170,12 @@ class SchedulerService:
|
|||
def _candidate_from_row(row: tuple, *, now_utc: datetime) -> _AutoRunCandidate | None:
|
||||
user_id, run_interval_minutes, request_delay_seconds, cooldown_until, cooldown_reason = row
|
||||
if cooldown_until is not None and cooldown_until.tzinfo is None:
|
||||
cooldown_until = cooldown_until.replace(tzinfo=timezone.utc)
|
||||
cooldown_until = cooldown_until.replace(tzinfo=UTC)
|
||||
if cooldown_until is not None and cooldown_until > now_utc:
|
||||
structured_log(
|
||||
logger, "info", "scheduler.run_skipped_safety_cooldown_precheck",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.run_skipped_safety_cooldown_precheck",
|
||||
user_id=int(user_id),
|
||||
reason=cooldown_reason,
|
||||
cooldown_until=cooldown_until,
|
||||
|
|
@ -192,7 +194,7 @@ class SchedulerService:
|
|||
if not settings.ingestion_automation_allowed:
|
||||
return []
|
||||
rows = await self._load_candidate_rows()
|
||||
now_utc = datetime.now(timezone.utc)
|
||||
now_utc = datetime.now(UTC)
|
||||
candidates: list[_AutoRunCandidate] = []
|
||||
for row in rows:
|
||||
candidate = self._candidate_from_row(row, now_utc=now_utc)
|
||||
|
|
@ -216,9 +218,7 @@ class SchedulerService:
|
|||
if last_run is None:
|
||||
return True
|
||||
|
||||
next_due_dt = last_run + timedelta(
|
||||
minutes=candidate.run_interval_minutes
|
||||
)
|
||||
next_due_dt = last_run + timedelta(minutes=candidate.run_interval_minutes)
|
||||
return now >= next_due_dt
|
||||
|
||||
async def _run_candidate_ingestion(
|
||||
|
|
@ -252,7 +252,9 @@ class SchedulerService:
|
|||
except RunBlockedBySafetyPolicyError as exc:
|
||||
await session.rollback()
|
||||
structured_log(
|
||||
logger, "info", "scheduler.run_skipped_safety_cooldown",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.run_skipped_safety_cooldown",
|
||||
user_id=candidate.user_id,
|
||||
reason=exc.safety_state.get("cooldown_reason"),
|
||||
cooldown_until=exc.safety_state.get("cooldown_until"),
|
||||
|
|
@ -269,7 +271,9 @@ class SchedulerService:
|
|||
if run_summary is None:
|
||||
return
|
||||
structured_log(
|
||||
logger, "info", "scheduler.run_completed",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.run_completed",
|
||||
user_id=candidate.user_id,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
|
|
@ -278,7 +282,7 @@ class SchedulerService:
|
|||
)
|
||||
|
||||
async def _drain_continuation_queue(self) -> None:
|
||||
now = datetime.now(timezone.utc)
|
||||
now = datetime.now(UTC)
|
||||
session_factory = get_session_factory()
|
||||
async with session_factory() as session:
|
||||
jobs = await queue_service.list_due_jobs(
|
||||
|
|
@ -305,7 +309,9 @@ class SchedulerService:
|
|||
await session.commit()
|
||||
if dropped is not None:
|
||||
structured_log(
|
||||
logger, "warning", "scheduler.queue_item_dropped_max_attempts",
|
||||
logger,
|
||||
"warning",
|
||||
"scheduler.queue_item_dropped_max_attempts",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
scholar_profile_id=job.scholar_profile_id,
|
||||
|
|
@ -352,7 +358,9 @@ class SchedulerService:
|
|||
await session.commit()
|
||||
if dropped is not None:
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_dropped_scholar_unavailable",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_dropped_scholar_unavailable",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
scholar_profile_id=job.scholar_profile_id,
|
||||
|
|
@ -371,8 +379,11 @@ class SchedulerService:
|
|||
)
|
||||
await recovery_session.commit()
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_deferred_lock",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_deferred_lock",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
)
|
||||
|
||||
async def _reschedule_queue_job_safety_cooldown(
|
||||
|
|
@ -395,7 +406,9 @@ class SchedulerService:
|
|||
)
|
||||
await recovery_session.commit()
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_deferred_safety_cooldown",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_deferred_safety_cooldown",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
reason=exc.safety_state.get("cooldown_reason"),
|
||||
|
|
@ -423,8 +436,12 @@ class SchedulerService:
|
|||
)
|
||||
await recovery_session.commit()
|
||||
structured_log(
|
||||
logger, "warning", "scheduler.queue_item_dropped_after_exception",
|
||||
queue_item_id=job.id, user_id=job.user_id, attempt_count=queue_item.attempt_count,
|
||||
logger,
|
||||
"warning",
|
||||
"scheduler.queue_item_dropped_after_exception",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
attempt_count=queue_item.attempt_count,
|
||||
)
|
||||
return
|
||||
delay_seconds = queue_service.compute_backoff_seconds(
|
||||
|
|
@ -489,15 +506,23 @@ class SchedulerService:
|
|||
await session.commit()
|
||||
if queue_item is None:
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_resolved",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id, status=run_summary.status.value,
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_resolved",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
)
|
||||
return
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_progressed",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id, status=run_summary.status.value,
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_progressed",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
attempt_count=int(queue_item.attempt_count),
|
||||
)
|
||||
return
|
||||
|
|
@ -505,28 +530,50 @@ class SchedulerService:
|
|||
if queue_item is None:
|
||||
await session.commit()
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_resolved",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id, status=run_summary.status.value,
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_resolved",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
)
|
||||
return
|
||||
if int(queue_item.attempt_count) >= self._continuation_max_attempts:
|
||||
await queue_service.mark_dropped(session, job_id=job.id, reason="max_attempts_after_run")
|
||||
await session.commit()
|
||||
structured_log(
|
||||
logger, "warning", "scheduler.queue_item_dropped_max_attempts_after_run",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
logger,
|
||||
"warning",
|
||||
"scheduler.queue_item_dropped_max_attempts_after_run",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
attempt_count=queue_item.attempt_count,
|
||||
run_id=run_summary.crawl_run_id, status=run_summary.status.value,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
)
|
||||
return
|
||||
delay_seconds = queue_service.compute_backoff_seconds(base_seconds=self._continuation_base_delay_seconds, attempt_count=int(queue_item.attempt_count), max_seconds=self._continuation_max_delay_seconds)
|
||||
await queue_service.reschedule_job(session, job_id=job.id, delay_seconds=delay_seconds, reason=queue_item.reason, error=queue_item.last_error)
|
||||
delay_seconds = queue_service.compute_backoff_seconds(
|
||||
base_seconds=self._continuation_base_delay_seconds,
|
||||
attempt_count=int(queue_item.attempt_count),
|
||||
max_seconds=self._continuation_max_delay_seconds,
|
||||
)
|
||||
await queue_service.reschedule_job(
|
||||
session,
|
||||
job_id=job.id,
|
||||
delay_seconds=delay_seconds,
|
||||
reason=queue_item.reason,
|
||||
error=queue_item.last_error,
|
||||
)
|
||||
await session.commit()
|
||||
structured_log(
|
||||
logger, "info", "scheduler.queue_item_rescheduled_failed",
|
||||
queue_item_id=job.id, user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id, status=run_summary.status.value,
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.queue_item_rescheduled_failed",
|
||||
queue_item_id=job.id,
|
||||
user_id=job.user_id,
|
||||
run_id=run_summary.crawl_run_id,
|
||||
status=run_summary.status.value,
|
||||
attempt_count=int(queue_item.attempt_count),
|
||||
delay_seconds=delay_seconds,
|
||||
)
|
||||
|
|
@ -556,7 +603,9 @@ class SchedulerService:
|
|||
)
|
||||
if processed > 0:
|
||||
structured_log(
|
||||
logger, "info", "scheduler.pdf_queue_drain_completed",
|
||||
logger,
|
||||
"info",
|
||||
"scheduler.pdf_queue_drain_completed",
|
||||
processed_count=processed,
|
||||
)
|
||||
except Exception:
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Mapping
|
||||
|
||||
import httpx
|
||||
from tenacity import (
|
||||
|
|
@ -23,11 +21,13 @@ class OpenAlexClientError(Exception):
|
|||
|
||||
class OpenAlexRateLimitError(OpenAlexClientError):
|
||||
"""Transient rate limit (too many requests per second)."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class OpenAlexBudgetExhaustedError(OpenAlexClientError):
|
||||
"""Daily API budget exhausted — retrying is futile until midnight UTC."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
|
|
@ -79,9 +79,7 @@ class OpenAlexClient:
|
|||
if response.status_code == 429:
|
||||
remaining = response.headers.get("X-RateLimit-Remaining-USD", "")
|
||||
if remaining == "0" or remaining.startswith("-"):
|
||||
raise OpenAlexBudgetExhaustedError(
|
||||
"Daily API budget exhausted; retrying won't help until midnight UTC"
|
||||
)
|
||||
raise OpenAlexBudgetExhaustedError("Daily API budget exhausted; retrying won't help until midnight UTC")
|
||||
raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex work by DOI")
|
||||
if response.status_code >= 400:
|
||||
logger.warning("OpenAlex API error: %s %s", response.status_code, response.text[:500])
|
||||
|
|
@ -129,9 +127,7 @@ class OpenAlexClient:
|
|||
if response.status_code == 429:
|
||||
remaining = response.headers.get("X-RateLimit-Remaining-USD", "")
|
||||
if remaining == "0" or remaining.startswith("-"):
|
||||
raise OpenAlexBudgetExhaustedError(
|
||||
"Daily API budget exhausted; retrying won't help until midnight UTC"
|
||||
)
|
||||
raise OpenAlexBudgetExhaustedError("Daily API budget exhausted; retrying won't help until midnight UTC")
|
||||
raise OpenAlexRateLimitError("Rate limit exceeded fetching OpenAlex works list")
|
||||
if response.status_code >= 400:
|
||||
logger.warning("OpenAlex API error (filters=%s): %s %s", filters, response.status_code, response.text[:500])
|
||||
|
|
|
|||
|
|
@ -81,8 +81,7 @@ def find_best_match(
|
|||
|
||||
# Extract all candidates within the tiebreaker margin
|
||||
top_scored_candidates = [
|
||||
(score, cand) for score, cand in scored_candidates
|
||||
if best_score - score <= TIEBREAKER_MARGIN
|
||||
(score, cand) for score, cand in scored_candidates if best_score - score <= TIEBREAKER_MARGIN
|
||||
]
|
||||
|
||||
if len(top_scored_candidates) == 1:
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Any, Mapping
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ from app.services.domains.portability.publication_import import (
|
|||
_upsert_imported_publication,
|
||||
)
|
||||
from app.services.domains.portability.scholar_import import _upsert_imported_scholars
|
||||
from app.services.domains.portability.types import ImportExportError, ImportedPublicationInput
|
||||
from app.services.domains.portability.types import ImportedPublicationInput, ImportExportError
|
||||
|
||||
|
||||
async def import_user_data(
|
||||
|
|
@ -58,8 +58,8 @@ async def import_user_data(
|
|||
|
||||
__all__ = [
|
||||
"EXPORT_SCHEMA_VERSION",
|
||||
"MAX_IMPORT_SCHOLARS",
|
||||
"MAX_IMPORT_PUBLICATIONS",
|
||||
"MAX_IMPORT_SCHOLARS",
|
||||
"ImportExportError",
|
||||
"ImportedPublicationInput",
|
||||
"export_user_data",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import select
|
||||
|
|
@ -11,7 +11,7 @@ from app.services.domains.portability.constants import EXPORT_SCHEMA_VERSION
|
|||
|
||||
|
||||
def _exported_at_iso() -> str:
|
||||
return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
|
||||
return datetime.now(UTC).replace(microsecond=0).isoformat()
|
||||
|
||||
|
||||
def _serialize_export_scholar(profile: ScholarProfile) -> dict[str, Any]:
|
||||
|
|
@ -58,9 +58,7 @@ async def export_user_data(
|
|||
user_id: int,
|
||||
) -> dict[str, Any]:
|
||||
scholars_result = await db_session.execute(
|
||||
select(ScholarProfile)
|
||||
.where(ScholarProfile.user_id == user_id)
|
||||
.order_by(ScholarProfile.id.asc())
|
||||
select(ScholarProfile).where(ScholarProfile.user_id == user_id).order_by(ScholarProfile.id.asc())
|
||||
)
|
||||
publication_result = await db_session.execute(
|
||||
select(
|
||||
|
|
|
|||
|
|
@ -104,6 +104,4 @@ def _validate_import_sizes(
|
|||
if len(scholars) > MAX_IMPORT_SCHOLARS:
|
||||
raise ImportExportError(f"Import exceeds max scholars ({MAX_IMPORT_SCHOLARS}).")
|
||||
if len(publications) > MAX_IMPORT_PUBLICATIONS:
|
||||
raise ImportExportError(
|
||||
f"Import exceeds max publications ({MAX_IMPORT_PUBLICATIONS})."
|
||||
)
|
||||
raise ImportExportError(f"Import exceeds max publications ({MAX_IMPORT_PUBLICATIONS}).")
|
||||
|
|
|
|||
|
|
@ -6,8 +6,8 @@ from sqlalchemy import select
|
|||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models import Publication, ScholarProfile, ScholarPublication
|
||||
from app.services.domains.ingestion.application import build_publication_url, normalize_title
|
||||
from app.services.domains.doi.normalize import normalize_doi
|
||||
from app.services.domains.ingestion.application import build_publication_url, normalize_title
|
||||
from app.services.domains.portability.normalize import (
|
||||
_normalize_citation_count,
|
||||
_normalize_optional_text,
|
||||
|
|
@ -22,9 +22,7 @@ async def _find_publication_by_cluster(
|
|||
*,
|
||||
cluster_id: str,
|
||||
) -> Publication | None:
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(Publication.cluster_id == cluster_id)
|
||||
)
|
||||
result = await db_session.execute(select(Publication).where(Publication.cluster_id == cluster_id))
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
|
|
@ -33,9 +31,7 @@ async def _find_publication_by_fingerprint(
|
|||
*,
|
||||
fingerprint_sha256: str,
|
||||
) -> Publication | None:
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(Publication.fingerprint_sha256 == fingerprint_sha256)
|
||||
)
|
||||
result = await db_session.execute(select(Publication).where(Publication.fingerprint_sha256 == fingerprint_sha256))
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -15,9 +15,7 @@ async def _load_user_scholar_map(
|
|||
*,
|
||||
user_id: int,
|
||||
) -> dict[str, ScholarProfile]:
|
||||
result = await db_session.execute(
|
||||
select(ScholarProfile).where(ScholarProfile.user_id == user_id)
|
||||
)
|
||||
result = await db_session.execute(select(ScholarProfile).where(ScholarProfile.user_id == user_id))
|
||||
profiles = list(result.scalars().all())
|
||||
return {profile.scholar_id: profile for profile in profiles}
|
||||
|
||||
|
|
@ -70,9 +68,7 @@ async def _upsert_imported_scholars(
|
|||
for item in scholars:
|
||||
try:
|
||||
scholar_id = scholar_service.validate_scholar_id(str(item["scholar_id"]))
|
||||
display_name = scholar_service.normalize_display_name(
|
||||
str(item.get("display_name") or "")
|
||||
)
|
||||
display_name = scholar_service.normalize_display_name(str(item.get("display_name") or ""))
|
||||
override_url = scholar_service.normalize_profile_image_url(
|
||||
_normalize_optional_text(item.get("profile_image_override_url"))
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
from app.services.domains.publication_identifiers.application import (
|
||||
DisplayIdentifier,
|
||||
display_identifier_for_publication_id,
|
||||
derive_display_identifier_from_values,
|
||||
display_identifier_for_publication_id,
|
||||
overlay_pdf_queue_items_with_display_identifiers,
|
||||
overlay_publication_items_with_display_identifiers,
|
||||
sync_identifiers_for_publication_fields,
|
||||
|
|
@ -10,8 +10,8 @@ from app.services.domains.publication_identifiers.application import (
|
|||
|
||||
__all__ = [
|
||||
"DisplayIdentifier",
|
||||
"display_identifier_for_publication_id",
|
||||
"derive_display_identifier_from_values",
|
||||
"display_identifier_for_publication_id",
|
||||
"overlay_pdf_queue_items_with_display_identifiers",
|
||||
"overlay_publication_items_with_display_identifiers",
|
||||
"sync_identifiers_for_publication_fields",
|
||||
|
|
|
|||
|
|
@ -55,7 +55,9 @@ def _fallback_candidates_from_values(
|
|||
if doi:
|
||||
normalized_doi = normalize_doi(doi)
|
||||
if normalized_doi:
|
||||
candidates.append(_candidate(IdentifierKind.DOI, doi, normalized_doi, "legacy_doi", CONFIDENCE_HIGH, pub_url))
|
||||
candidates.append(
|
||||
_candidate(IdentifierKind.DOI, doi, normalized_doi, "legacy_doi", CONFIDENCE_HIGH, pub_url)
|
||||
)
|
||||
candidates.extend(_url_identifier_candidates(values=values, source="legacy_urls"))
|
||||
return _dedup_candidates(candidates)
|
||||
|
||||
|
|
@ -332,10 +334,13 @@ async def _existing_identifier_by_kind(
|
|||
kind: str,
|
||||
) -> PublicationIdentifier | None:
|
||||
result = await db_session.execute(
|
||||
select(PublicationIdentifier).where(
|
||||
select(PublicationIdentifier)
|
||||
.where(
|
||||
PublicationIdentifier.publication_id == publication_id,
|
||||
PublicationIdentifier.kind == kind,
|
||||
).order_by(PublicationIdentifier.confidence_score.desc()).limit(1)
|
||||
)
|
||||
.order_by(PublicationIdentifier.confidence_score.desc())
|
||||
.limit(1)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
|
@ -380,7 +385,9 @@ def _overlay_publication_item(
|
|||
item: PublicationListItem,
|
||||
display_identifier: DisplayIdentifier | None,
|
||||
) -> PublicationListItem:
|
||||
fallback = display_identifier or derive_display_identifier_from_values(doi=None, pub_url=item.pub_url, pdf_url=item.pdf_url)
|
||||
fallback = display_identifier or derive_display_identifier_from_values(
|
||||
doi=None, pub_url=item.pub_url, pdf_url=item.pdf_url
|
||||
)
|
||||
return replace(item, display_identifier=fallback)
|
||||
|
||||
|
||||
|
|
@ -447,8 +454,7 @@ def _best_display_identifier_map(rows: list[PublicationIdentifier]) -> dict[int,
|
|||
return {
|
||||
publication_id: display
|
||||
for publication_id, display in (
|
||||
(publication_id, _best_display_identifier(candidates))
|
||||
for publication_id, candidates in grouped.items()
|
||||
(publication_id, _best_display_identifier(candidates)) for publication_id, candidates in grouped.items()
|
||||
)
|
||||
if display is not None
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from app.services.domains.publications.counts import (
|
||||
count_for_user,
|
||||
count_favorite_for_user,
|
||||
count_for_user,
|
||||
count_latest_for_user,
|
||||
count_unread_for_user,
|
||||
)
|
||||
from app.services.domains.publications.listing import (
|
||||
list_for_user,
|
||||
retry_pdf_for_user,
|
||||
list_unread_for_user,
|
||||
)
|
||||
from app.services.domains.publications.enrichment import (
|
||||
hydrate_pdf_enrichment_state,
|
||||
schedule_missing_pdf_enrichment_for_user,
|
||||
schedule_retry_pdf_enrichment_for_row,
|
||||
)
|
||||
from app.services.domains.publications.listing import (
|
||||
list_for_user,
|
||||
list_unread_for_user,
|
||||
retry_pdf_for_user,
|
||||
)
|
||||
from app.services.domains.publications.modes import (
|
||||
MODE_ALL,
|
||||
MODE_LATEST,
|
||||
|
|
@ -23,6 +23,13 @@ from app.services.domains.publications.modes import (
|
|||
MODE_UNREAD,
|
||||
resolve_publication_view_mode,
|
||||
)
|
||||
from app.services.domains.publications.pdf_queue import (
|
||||
count_pdf_queue_items,
|
||||
enqueue_all_missing_pdf_jobs,
|
||||
enqueue_retry_pdf_job_for_publication_id,
|
||||
list_pdf_queue_items,
|
||||
list_pdf_queue_page,
|
||||
)
|
||||
from app.services.domains.publications.queries import (
|
||||
get_latest_run_id_for_user,
|
||||
get_publication_item_for_user,
|
||||
|
|
@ -33,42 +40,35 @@ from app.services.domains.publications.read_state import (
|
|||
mark_selected_as_read_for_user,
|
||||
set_publication_favorite_for_user,
|
||||
)
|
||||
from app.services.domains.publications.pdf_queue import (
|
||||
count_pdf_queue_items,
|
||||
enqueue_all_missing_pdf_jobs,
|
||||
enqueue_retry_pdf_job_for_publication_id,
|
||||
list_pdf_queue_page,
|
||||
list_pdf_queue_items,
|
||||
)
|
||||
from app.services.domains.publications.types import PublicationListItem, UnreadPublicationItem
|
||||
|
||||
__all__ = [
|
||||
"MODE_ALL",
|
||||
"MODE_UNREAD",
|
||||
"MODE_LATEST",
|
||||
"MODE_NEW",
|
||||
"MODE_UNREAD",
|
||||
"PublicationListItem",
|
||||
"UnreadPublicationItem",
|
||||
"resolve_publication_view_mode",
|
||||
"get_latest_run_id_for_user",
|
||||
"publications_query",
|
||||
"get_publication_item_for_user",
|
||||
"list_for_user",
|
||||
"list_unread_for_user",
|
||||
"retry_pdf_for_user",
|
||||
"hydrate_pdf_enrichment_state",
|
||||
"schedule_retry_pdf_enrichment_for_row",
|
||||
"list_pdf_queue_items",
|
||||
"list_pdf_queue_page",
|
||||
"count_favorite_for_user",
|
||||
"count_for_user",
|
||||
"count_latest_for_user",
|
||||
"count_pdf_queue_items",
|
||||
"count_unread_for_user",
|
||||
"enqueue_all_missing_pdf_jobs",
|
||||
"enqueue_retry_pdf_job_for_publication_id",
|
||||
"schedule_missing_pdf_enrichment_for_user",
|
||||
"count_for_user",
|
||||
"count_favorite_for_user",
|
||||
"count_unread_for_user",
|
||||
"count_latest_for_user",
|
||||
"get_latest_run_id_for_user",
|
||||
"get_publication_item_for_user",
|
||||
"hydrate_pdf_enrichment_state",
|
||||
"list_for_user",
|
||||
"list_pdf_queue_items",
|
||||
"list_pdf_queue_page",
|
||||
"list_unread_for_user",
|
||||
"mark_all_unread_as_read_for_user",
|
||||
"mark_selected_as_read_for_user",
|
||||
"publications_query",
|
||||
"resolve_publication_view_mode",
|
||||
"retry_pdf_for_user",
|
||||
"schedule_missing_pdf_enrichment_for_user",
|
||||
"schedule_retry_pdf_enrichment_for_row",
|
||||
"set_publication_favorite_for_user",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import hashlib
|
||||
import logging
|
||||
from typing import Iterable
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import delete, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -113,9 +113,7 @@ async def _load_publication(
|
|||
*,
|
||||
publication_id: int,
|
||||
) -> Publication | None:
|
||||
result = await db_session.execute(
|
||||
select(Publication).where(Publication.id == publication_id)
|
||||
)
|
||||
result = await db_session.execute(select(Publication).where(Publication.id == publication_id))
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
|
||||
|
|
@ -160,9 +158,7 @@ async def _migrate_scholar_links(
|
|||
dup_links = dup_links_result.scalars().all()
|
||||
|
||||
winner_profiles_result = await db_session.execute(
|
||||
select(ScholarPublication.scholar_profile_id).where(
|
||||
ScholarPublication.publication_id == winner_id
|
||||
)
|
||||
select(ScholarPublication.scholar_profile_id).where(ScholarPublication.publication_id == winner_id)
|
||||
)
|
||||
winner_profiles: set[int] = {row for (row,) in winner_profiles_result}
|
||||
|
||||
|
|
@ -346,11 +342,7 @@ def _candidate_from_row(
|
|||
|
||||
|
||||
def _normalized_tokens(tokens: Iterable[str]) -> set[str]:
|
||||
return {
|
||||
token
|
||||
for token in tokens
|
||||
if len(token) >= NEAR_DUP_MIN_TOKEN_LENGTH and token not in NEAR_DUP_STOPWORDS
|
||||
}
|
||||
return {token for token in tokens if len(token) >= NEAR_DUP_MIN_TOKEN_LENGTH and token not in NEAR_DUP_STOPWORDS}
|
||||
|
||||
|
||||
def _cluster_candidate_groups(
|
||||
|
|
|
|||
|
|
@ -4,12 +4,12 @@ import logging
|
|||
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.publications.pdf_queue import (
|
||||
enqueue_missing_pdf_jobs,
|
||||
enqueue_retry_pdf_job,
|
||||
overlay_pdf_job_state,
|
||||
)
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.publications.types import PublicationListItem
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -30,7 +30,9 @@ async def schedule_missing_pdf_enrichment_for_user(
|
|||
rows=items,
|
||||
max_items=max_items,
|
||||
)
|
||||
structured_log(logger, "info", "publications.enrichment.scheduled", user_id=user_id, publication_count=len(queued_ids))
|
||||
structured_log(
|
||||
logger, "info", "publications.enrichment.scheduled", user_id=user_id, publication_count=len(queued_ids)
|
||||
)
|
||||
return len(queued_ids)
|
||||
|
||||
|
||||
|
|
@ -47,7 +49,14 @@ async def schedule_retry_pdf_enrichment_for_row(
|
|||
request_email=request_email,
|
||||
row=item,
|
||||
)
|
||||
structured_log(logger, "info", "publications.enrichment.retry_scheduled", user_id=user_id, publication_id=item.publication_id, queued=queued)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"publications.enrichment.retry_scheduled",
|
||||
user_id=user_id,
|
||||
publication_id=item.publication_id,
|
||||
queued=queued,
|
||||
)
|
||||
return queued
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -51,10 +51,7 @@ async def list_for_user(
|
|||
snapshot_before=snapshot_before,
|
||||
)
|
||||
)
|
||||
rows = [
|
||||
publication_list_item_from_row(row, latest_run_id=latest_run_id)
|
||||
for row in result.all()
|
||||
]
|
||||
rows = [publication_list_item_from_row(row, latest_run_id=latest_run_id) for row in result.all()]
|
||||
return await identifier_service.overlay_publication_items_with_display_identifiers(
|
||||
db_session,
|
||||
items=rows,
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime, timedelta
|
||||
|
||||
from sqlalchemy import Select, and_, func, literal, or_, select, union_all
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
|
@ -17,6 +17,7 @@ from app.db.models import (
|
|||
User,
|
||||
)
|
||||
from app.db.session import get_session_factory
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.publication_identifiers import application as identifier_service
|
||||
from app.services.domains.publication_identifiers.types import DisplayIdentifier
|
||||
from app.services.domains.publications.pdf_resolution_pipeline import (
|
||||
|
|
@ -27,7 +28,6 @@ from app.services.domains.unpaywall.application import (
|
|||
FAILURE_RESOLUTION_EXCEPTION,
|
||||
OaResolutionOutcome,
|
||||
)
|
||||
from app.logging_utils import structured_log
|
||||
from app.settings import settings
|
||||
|
||||
PDF_STATUS_UNTRACKED = "untracked"
|
||||
|
|
@ -85,7 +85,7 @@ class PdfQueuePage:
|
|||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
return datetime.now(timezone.utc)
|
||||
return datetime.now(UTC)
|
||||
|
||||
|
||||
def _publication_ids(rows: list[PublicationListItem]) -> list[int]:
|
||||
|
|
@ -472,7 +472,13 @@ async def _resolve_publication_row(
|
|||
# Propagate upward so the batch loop can stop immediately.
|
||||
raise
|
||||
except Exception as exc: # pragma: no cover - defensive network boundary
|
||||
structured_log(logger, "warning", "publications.pdf_queue.resolve_failed", publication_id=row.publication_id, error=str(exc))
|
||||
structured_log(
|
||||
logger,
|
||||
"warning",
|
||||
"publications.pdf_queue.resolve_failed",
|
||||
publication_id=row.publication_id,
|
||||
error=str(exc),
|
||||
)
|
||||
outcome = _failed_outcome(row=row)
|
||||
arxiv_rate_limited = False
|
||||
await _persist_outcome(
|
||||
|
|
@ -514,9 +520,19 @@ async def _run_resolution_task(
|
|||
)
|
||||
if arxiv_rate_limited and arxiv_lookup_allowed:
|
||||
arxiv_lookup_allowed = False
|
||||
structured_log(logger, "warning", "pdf_queue.arxiv_batch_disabled", detail="arXiv temporarily disabled for remaining batch after rate limit")
|
||||
structured_log(
|
||||
logger,
|
||||
"warning",
|
||||
"pdf_queue.arxiv_batch_disabled",
|
||||
detail="arXiv temporarily disabled for remaining batch after rate limit",
|
||||
)
|
||||
except OpenAlexBudgetExhaustedError:
|
||||
structured_log(logger, "warning", "pdf_queue.budget_exhausted", detail="Stopping PDF resolution batch — OpenAlex daily budget exhausted")
|
||||
structured_log(
|
||||
logger,
|
||||
"warning",
|
||||
"pdf_queue.budget_exhausted",
|
||||
detail="Stopping PDF resolution batch — OpenAlex daily budget exhausted",
|
||||
)
|
||||
break
|
||||
|
||||
|
||||
|
|
@ -679,7 +695,7 @@ async def _missing_pdf_candidates(
|
|||
limit: int,
|
||||
) -> list[PublicationListItem]:
|
||||
bounded_limit = max(1, min(int(limit), 5000))
|
||||
now = datetime.now(timezone.utc)
|
||||
now = datetime.now(UTC)
|
||||
cooldown_threshold = now - timedelta(days=7)
|
||||
|
||||
result = await db_session.execute(
|
||||
|
|
@ -701,10 +717,7 @@ async def _missing_pdf_candidates(
|
|||
.order_by(Publication.updated_at.desc(), Publication.id.desc())
|
||||
.limit(bounded_limit)
|
||||
)
|
||||
return [
|
||||
_queue_candidate_from_publication(publication)
|
||||
for publication in result.scalars()
|
||||
]
|
||||
return [_queue_candidate_from_publication(publication) for publication in result.scalars()]
|
||||
|
||||
|
||||
async def enqueue_all_missing_pdf_jobs(
|
||||
|
|
@ -902,9 +915,7 @@ async def count_pdf_queue_items(
|
|||
if normalized_status == PDF_STATUS_UNTRACKED:
|
||||
result = await db_session.execute(_untracked_queue_count_select())
|
||||
return int(result.scalar_one() or 0)
|
||||
tracked_result = await db_session.execute(
|
||||
_tracked_queue_count_select(status=normalized_status)
|
||||
)
|
||||
tracked_result = await db_session.execute(_tracked_queue_count_select(status=normalized_status))
|
||||
tracked_count = int(tracked_result.scalar_one() or 0)
|
||||
if normalized_status is not None:
|
||||
return tracked_count
|
||||
|
|
@ -946,9 +957,7 @@ async def drain_ready_jobs(
|
|||
limit: int,
|
||||
max_attempts: int,
|
||||
) -> int:
|
||||
result = await db_session.execute(
|
||||
select(User.id).where(User.is_active.is_(True)).order_by(User.id.asc()).limit(1)
|
||||
)
|
||||
result = await db_session.execute(select(User.id).where(User.is_active.is_(True)).order_by(User.id.asc()).limit(1))
|
||||
system_user_id = result.scalar_one_or_none()
|
||||
if system_user_id is None:
|
||||
return 0
|
||||
|
|
|
|||
|
|
@ -1,15 +1,15 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.arxiv.application import ArxivRateLimitError
|
||||
from app.services.domains.arxiv.guards import arxiv_skip_reason_for_item
|
||||
from app.services.domains.openalex.client import OpenAlexBudgetExhaustedError
|
||||
from app.services.domains.publications.types import PublicationListItem
|
||||
from app.services.domains.unpaywall.application import OaResolutionOutcome, resolve_publication_oa_outcomes
|
||||
from app.logging_utils import structured_log
|
||||
from app.settings import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -66,6 +66,7 @@ async def _openalex_outcome(
|
|||
return None
|
||||
|
||||
import re
|
||||
|
||||
safe_title = re.sub(r"[^\w\s]", " ", row.title)
|
||||
safe_title = " ".join(safe_title.split())
|
||||
if not safe_title:
|
||||
|
|
@ -107,12 +108,24 @@ async def _arxiv_outcome(
|
|||
from app.services.domains.arxiv.application import discover_arxiv_id_for_publication
|
||||
|
||||
if not allow_lookup:
|
||||
structured_log(logger, "info", "pdf_resolution.arxiv_skipped", publication_id=int(row.publication_id), skip_reason="batch_arxiv_cooldown_active")
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"pdf_resolution.arxiv_skipped",
|
||||
publication_id=int(row.publication_id),
|
||||
skip_reason="batch_arxiv_cooldown_active",
|
||||
)
|
||||
return None
|
||||
|
||||
skip_reason = arxiv_skip_reason_for_item(item=row)
|
||||
if skip_reason is not None:
|
||||
structured_log(logger, "info", "pdf_resolution.arxiv_skipped", publication_id=int(row.publication_id), skip_reason=skip_reason)
|
||||
structured_log(
|
||||
logger,
|
||||
"info",
|
||||
"pdf_resolution.arxiv_skipped",
|
||||
publication_id=int(row.publication_id),
|
||||
skip_reason=skip_reason,
|
||||
)
|
||||
return None
|
||||
|
||||
try:
|
||||
|
|
@ -134,7 +147,6 @@ async def _arxiv_outcome(
|
|||
return None
|
||||
|
||||
|
||||
|
||||
async def _oa_outcome(
|
||||
*,
|
||||
row: PublicationListItem,
|
||||
|
|
|
|||
|
|
@ -228,9 +228,7 @@ def publication_list_item_from_row(
|
|||
is_read=bool(is_read),
|
||||
is_favorite=bool(is_favorite),
|
||||
first_seen_at=created_at,
|
||||
is_new_in_latest_run=(
|
||||
latest_run_id is not None and int(first_seen_run_id or 0) == latest_run_id
|
||||
),
|
||||
is_new_in_latest_run=(latest_run_id is not None and int(first_seen_run_id or 0) == latest_run_id),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -17,11 +17,7 @@ def _normalized_selection_pairs(selections: list[tuple[int, int]]) -> set[tuple[
|
|||
|
||||
|
||||
def _scoped_scholar_ids_query(*, user_id: int):
|
||||
return (
|
||||
select(ScholarProfile.id)
|
||||
.where(ScholarProfile.user_id == user_id)
|
||||
.scalar_subquery()
|
||||
)
|
||||
return select(ScholarProfile.id).where(ScholarProfile.user_id == user_id).scalar_subquery()
|
||||
|
||||
|
||||
async def mark_all_unread_as_read_for_user(
|
||||
|
|
|
|||
|
|
@ -25,21 +25,21 @@ from app.services.domains.runs.types import (
|
|||
)
|
||||
|
||||
__all__ = [
|
||||
"QUEUE_STATUS_DROPPED",
|
||||
"QUEUE_STATUS_QUEUED",
|
||||
"QUEUE_STATUS_RETRYING",
|
||||
"QUEUE_STATUS_DROPPED",
|
||||
"QueueListItem",
|
||||
"QueueClearResult",
|
||||
"QueueListItem",
|
||||
"QueueTransitionError",
|
||||
"clear_queue_item_for_user",
|
||||
"drop_queue_item_for_user",
|
||||
"extract_run_summary",
|
||||
"get_manual_run_by_idempotency_key",
|
||||
"get_queue_item_for_user",
|
||||
"get_run_for_user",
|
||||
"list_queue_items_for_user",
|
||||
"list_recent_runs_for_user",
|
||||
"list_runs_for_user",
|
||||
"get_run_for_user",
|
||||
"get_manual_run_by_idempotency_key",
|
||||
"list_queue_items_for_user",
|
||||
"get_queue_item_for_user",
|
||||
"retry_queue_item_for_user",
|
||||
"drop_queue_item_for_user",
|
||||
"clear_queue_item_for_user",
|
||||
"queue_status_counts_for_user",
|
||||
"retry_queue_item_for_user",
|
||||
]
|
||||
|
|
|
|||
|
|
@ -1,14 +1,16 @@
|
|||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, AsyncGenerator, Dict, Set
|
||||
from collections.abc import AsyncGenerator
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RunEventPublisher:
|
||||
def __init__(self) -> None:
|
||||
# Maps run_id to a set of subscriber queues
|
||||
self._subscribers: Dict[int, Set[asyncio.Queue]] = {}
|
||||
self._subscribers: dict[int, set[asyncio.Queue]] = {}
|
||||
|
||||
def subscribe(self, run_id: int) -> asyncio.Queue:
|
||||
if run_id not in self._subscribers:
|
||||
|
|
@ -28,10 +30,7 @@ class RunEventPublisher:
|
|||
if run_id not in self._subscribers:
|
||||
return
|
||||
|
||||
message = {
|
||||
"type": event_type,
|
||||
"data": data
|
||||
}
|
||||
message = {"type": event_type, "data": data}
|
||||
|
||||
# Fan-out to all active subscribers for this run
|
||||
for queue in list(self._subscribers[run_id]):
|
||||
|
|
@ -40,8 +39,10 @@ class RunEventPublisher:
|
|||
except asyncio.QueueFull:
|
||||
logger.warning(f"Subscriber queue full for run {run_id}, dropping message")
|
||||
|
||||
|
||||
run_events = RunEventPublisher()
|
||||
|
||||
|
||||
async def event_generator(run_id: int) -> AsyncGenerator[str, None]:
|
||||
queue = run_events.subscribe(run_id)
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -41,9 +41,7 @@ async def get_queue_item_for_user(
|
|||
queue_item_id: int,
|
||||
) -> QueueListItem | None:
|
||||
result = await db_session.execute(
|
||||
queue_item_select(user_id=user_id)
|
||||
.where(IngestionQueueItem.id == queue_item_id)
|
||||
.limit(1)
|
||||
queue_item_select(user_id=user_id).where(IngestionQueueItem.id == queue_item_id).limit(1)
|
||||
)
|
||||
row = result.one_or_none()
|
||||
if row is None:
|
||||
|
|
|
|||
|
|
@ -35,9 +35,7 @@ async def list_runs_for_user(
|
|||
.limit(limit)
|
||||
)
|
||||
if failed_only:
|
||||
stmt = stmt.where(
|
||||
CrawlRun.status.in_([RunStatus.FAILED, RunStatus.PARTIAL_FAILURE])
|
||||
)
|
||||
stmt = stmt.where(CrawlRun.status.in_([RunStatus.FAILED, RunStatus.PARTIAL_FAILURE]))
|
||||
result = await db_session.execute(stmt)
|
||||
return list(result.scalars().all())
|
||||
|
||||
|
|
|
|||
|
|
@ -24,9 +24,7 @@ def _summary_int_dict(summary: dict[str, Any], key: str) -> dict[str, int]:
|
|||
if not isinstance(value, dict):
|
||||
return {}
|
||||
return {
|
||||
str(item_key): _safe_int(item_value, 0)
|
||||
for item_key, item_value in value.items()
|
||||
if isinstance(item_key, str)
|
||||
str(item_key): _safe_int(item_value, 0) for item_key, item_value in value.items() if isinstance(item_key, str)
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -34,11 +32,7 @@ def _summary_bool_dict(summary: dict[str, Any], key: str) -> dict[str, bool]:
|
|||
value = summary.get(key)
|
||||
if not isinstance(value, dict):
|
||||
return {}
|
||||
return {
|
||||
str(item_key): bool(item_value)
|
||||
for item_key, item_value in value.items()
|
||||
if isinstance(item_key, str)
|
||||
}
|
||||
return {str(item_key): bool(item_value) for item_key, item_value in value.items() if isinstance(item_key, str)}
|
||||
|
||||
|
||||
def _retry_counts(summary: dict[str, Any]) -> dict[str, int]:
|
||||
|
|
|
|||
|
|
@ -79,9 +79,7 @@ class ScholarAuthorSearchParser(HTMLParser):
|
|||
return
|
||||
|
||||
affiliation = normalize_space("".join(self._candidate["aff_parts"])) or None
|
||||
email_domain = _extract_verified_email_domain(
|
||||
normalize_space("".join(self._candidate["eml_parts"])) or None
|
||||
)
|
||||
email_domain = _extract_verified_email_domain(normalize_space("".join(self._candidate["eml_parts"])) or None)
|
||||
cited_by_text = normalize_space("".join(self._candidate["cby_parts"]))
|
||||
cited_by_match = re.search(r"\d+", cited_by_text)
|
||||
cited_by_count = int(cited_by_match.group(0)) if cited_by_match else None
|
||||
|
|
@ -97,10 +95,7 @@ class ScholarAuthorSearchParser(HTMLParser):
|
|||
|
||||
profile_url = build_absolute_scholar_url(self._candidate["name_href"])
|
||||
if not profile_url:
|
||||
profile_url = (
|
||||
"https://scholar.google.com/citations"
|
||||
f"?hl=en&user={scholar_id}"
|
||||
)
|
||||
profile_url = f"https://scholar.google.com/citations?hl=en&user={scholar_id}"
|
||||
|
||||
self.candidates.append(
|
||||
ScholarSearchCandidate(
|
||||
|
|
|
|||
|
|
@ -3,40 +3,29 @@ from __future__ import annotations
|
|||
from app.services.domains.scholar.author_rows import (
|
||||
ScholarAuthorSearchParser,
|
||||
count_author_search_markers,
|
||||
parse_scholar_id_from_href,
|
||||
)
|
||||
from app.services.domains.scholar.parser_constants import SCRIPT_STYLE_RE
|
||||
from app.services.domains.scholar.parser_types import (
|
||||
ParseState,
|
||||
ParsedAuthorSearchPage,
|
||||
ParsedProfilePage,
|
||||
PublicationCandidate,
|
||||
ScholarDomInvariantError,
|
||||
ScholarMalformedDataError,
|
||||
ScholarParserError,
|
||||
ScholarSearchCandidate,
|
||||
)
|
||||
from app.services.domains.scholar.parser_utils import (
|
||||
strip_tags,
|
||||
)
|
||||
from app.services.domains.scholar.profile_rows import (
|
||||
ScholarRowParser,
|
||||
count_markers,
|
||||
extract_articles_range,
|
||||
extract_profile_image_url,
|
||||
extract_profile_name,
|
||||
extract_rows,
|
||||
has_operation_error_banner,
|
||||
has_show_more_button,
|
||||
parse_citation_count,
|
||||
parse_cluster_id_from_href,
|
||||
parse_publications,
|
||||
parse_year,
|
||||
)
|
||||
from app.services.domains.scholar.source import FetchResult
|
||||
from app.services.domains.scholar.state_detection import (
|
||||
classify_block_or_captcha_reason,
|
||||
classify_network_error_reason,
|
||||
detect_author_search_state,
|
||||
detect_state,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -12,7 +12,6 @@ from app.services.domains.scholar.parser_types import PublicationCandidate
|
|||
from app.services.domains.scholar.parser_utils import (
|
||||
attr_class,
|
||||
attr_href,
|
||||
attr_src,
|
||||
build_absolute_scholar_url,
|
||||
normalize_space,
|
||||
strip_tags,
|
||||
|
|
@ -260,7 +259,7 @@ def has_show_more_button(html: str) -> bool:
|
|||
|
||||
def has_operation_error_banner(html: str) -> bool:
|
||||
lowered = html.lower()
|
||||
if "id=\"gsc_a_err\"" not in lowered and "id='gsc_a_err'" not in lowered:
|
||||
if 'id="gsc_a_err"' not in lowered and "id='gsc_a_err'" not in lowered:
|
||||
return False
|
||||
return "can't perform the operation now" in lowered or "cannot perform the operation now" in lowered
|
||||
|
||||
|
|
|
|||
|
|
@ -17,18 +17,12 @@ SCHOLAR_PROFILE_URL = "https://scholar.google.com/citations"
|
|||
DEFAULT_PAGE_SIZE = 100
|
||||
|
||||
DEFAULT_USER_AGENTS = [
|
||||
(
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
|
||||
),
|
||||
("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"),
|
||||
(
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7) AppleWebKit/605.1.15 "
|
||||
"(KHTML, like Gecko) Version/18.1 Safari/605.1.15"
|
||||
),
|
||||
(
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) "
|
||||
"Gecko/20100101 Firefox/131.0"
|
||||
),
|
||||
("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:131.0) Gecko/20100101 Firefox/131.0"),
|
||||
]
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
|
@ -44,8 +38,7 @@ class FetchResult:
|
|||
|
||||
|
||||
class ScholarSource(Protocol):
|
||||
async def fetch_profile_html(self, scholar_id: str) -> FetchResult:
|
||||
...
|
||||
async def fetch_profile_html(self, scholar_id: str) -> FetchResult: ...
|
||||
|
||||
async def fetch_profile_page_html(
|
||||
self,
|
||||
|
|
@ -53,16 +46,14 @@ class ScholarSource(Protocol):
|
|||
*,
|
||||
cstart: int,
|
||||
pagesize: int,
|
||||
) -> FetchResult:
|
||||
...
|
||||
) -> FetchResult: ...
|
||||
|
||||
async def fetch_author_search_html(
|
||||
self,
|
||||
query: str,
|
||||
*,
|
||||
start: int,
|
||||
) -> FetchResult:
|
||||
...
|
||||
) -> FetchResult: ...
|
||||
|
||||
|
||||
class LiveScholarSource:
|
||||
|
|
@ -145,7 +136,9 @@ class LiveScholarSource:
|
|||
@staticmethod
|
||||
def _network_error_result(requested_url: str, exc: URLError) -> FetchResult:
|
||||
structured_log(
|
||||
logger, "warning", "scholar_source.fetch_network_error",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_source.fetch_network_error",
|
||||
requested_url=requested_url,
|
||||
)
|
||||
return FetchResult(
|
||||
|
|
@ -159,7 +152,9 @@ class LiveScholarSource:
|
|||
@staticmethod
|
||||
def _http_error_result(requested_url: str, exc: HTTPError) -> FetchResult:
|
||||
structured_log(
|
||||
logger, "warning", "scholar_source.fetch_http_error",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_source.fetch_http_error",
|
||||
requested_url=requested_url,
|
||||
status_code=exc.code,
|
||||
)
|
||||
|
|
@ -176,7 +171,9 @@ class LiveScholarSource:
|
|||
body = response.read().decode("utf-8", errors="replace")
|
||||
status_code = getattr(response, "status", 200)
|
||||
structured_log(
|
||||
logger, "debug", "scholar_source.fetch_succeeded",
|
||||
logger,
|
||||
"debug",
|
||||
"scholar_source.fetch_succeeded",
|
||||
requested_url=requested_url,
|
||||
status_code=status_code,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,13 +1,11 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import replace
|
||||
from datetime import datetime
|
||||
from datetime import timedelta
|
||||
from datetime import timezone
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
from dataclasses import replace
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import delete, func, select, text
|
||||
|
|
@ -15,9 +13,10 @@ from sqlalchemy.exc import IntegrityError
|
|||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from app.db.models import AuthorSearchCacheEntry, AuthorSearchRuntimeState, ScholarProfile
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.scholar.parser import (
|
||||
ParseState,
|
||||
ParsedAuthorSearchPage,
|
||||
ParseState,
|
||||
ScholarParserError,
|
||||
parse_author_search_page,
|
||||
parse_profile_page,
|
||||
|
|
@ -34,33 +33,30 @@ from app.services.domains.scholars.constants import (
|
|||
DEFAULT_AUTHOR_SEARCH_COOLDOWN_REJECTION_ALERT_THRESHOLD,
|
||||
DEFAULT_AUTHOR_SEARCH_COOLDOWN_SECONDS,
|
||||
DEFAULT_AUTHOR_SEARCH_INTERVAL_JITTER_SECONDS,
|
||||
MAX_AUTHOR_SEARCH_LIMIT,
|
||||
DEFAULT_AUTHOR_SEARCH_MIN_INTERVAL_SECONDS,
|
||||
DEFAULT_AUTHOR_SEARCH_RETRY_ALERT_THRESHOLD,
|
||||
MAX_AUTHOR_SEARCH_LIMIT,
|
||||
SEARCH_CACHED_BLOCK_REASON,
|
||||
SEARCH_COOLDOWN_REASON,
|
||||
SEARCH_DISABLED_REASON,
|
||||
)
|
||||
from app.logging_utils import structured_log
|
||||
from app.services.domains.scholars.exceptions import ScholarServiceError
|
||||
from app.services.domains.scholars.search_hints import (
|
||||
_merge_warnings,
|
||||
_policy_blocked_author_search_result,
|
||||
_trim_author_search_result,
|
||||
resolve_profile_image,
|
||||
scrape_state_hint,
|
||||
)
|
||||
from app.services.domains.scholars.uploads import (
|
||||
_ensure_upload_root,
|
||||
_resolve_upload_path,
|
||||
_safe_remove_upload,
|
||||
resolve_upload_file_path,
|
||||
)
|
||||
from app.services.domains.scholars.validators import (
|
||||
normalize_display_name,
|
||||
normalize_profile_image_url,
|
||||
validate_scholar_id,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
@ -247,7 +243,7 @@ async def _cache_get_author_search_result(
|
|||
return None
|
||||
expires_at = entry.expires_at
|
||||
if expires_at.tzinfo is None:
|
||||
expires_at = expires_at.replace(tzinfo=timezone.utc)
|
||||
expires_at = expires_at.replace(tzinfo=UTC)
|
||||
if expires_at <= now_utc:
|
||||
await db_session.delete(entry)
|
||||
return None
|
||||
|
|
@ -305,27 +301,19 @@ async def _prune_author_search_cache(
|
|||
now_utc: datetime,
|
||||
max_entries: int,
|
||||
) -> None:
|
||||
await db_session.execute(
|
||||
delete(AuthorSearchCacheEntry).where(AuthorSearchCacheEntry.expires_at <= now_utc)
|
||||
)
|
||||
await db_session.execute(delete(AuthorSearchCacheEntry).where(AuthorSearchCacheEntry.expires_at <= now_utc))
|
||||
bounded_max_entries = max(1, int(max_entries))
|
||||
count_result = await db_session.execute(
|
||||
select(func.count()).select_from(AuthorSearchCacheEntry)
|
||||
)
|
||||
count_result = await db_session.execute(select(func.count()).select_from(AuthorSearchCacheEntry))
|
||||
entry_count = int(count_result.scalar_one() or 0)
|
||||
overflow = max(0, entry_count - bounded_max_entries)
|
||||
if overflow <= 0:
|
||||
return
|
||||
stale_keys_result = await db_session.execute(
|
||||
select(AuthorSearchCacheEntry.query_key)
|
||||
.order_by(AuthorSearchCacheEntry.cached_at.asc())
|
||||
.limit(overflow)
|
||||
select(AuthorSearchCacheEntry.query_key).order_by(AuthorSearchCacheEntry.cached_at.asc()).limit(overflow)
|
||||
)
|
||||
stale_keys = [str(row[0]) for row in stale_keys_result.all()]
|
||||
if stale_keys:
|
||||
await db_session.execute(
|
||||
delete(AuthorSearchCacheEntry).where(AuthorSearchCacheEntry.query_key.in_(stale_keys))
|
||||
)
|
||||
await db_session.execute(delete(AuthorSearchCacheEntry).where(AuthorSearchCacheEntry.query_key.in_(stale_keys)))
|
||||
|
||||
|
||||
def _is_author_search_block_state(parsed: ParsedAuthorSearchPage) -> bool:
|
||||
|
|
@ -340,7 +328,7 @@ def _author_search_cooldown_remaining_seconds(
|
|||
if cooldown_until is None:
|
||||
return 0
|
||||
if cooldown_until.tzinfo is None:
|
||||
cooldown_until = cooldown_until.replace(tzinfo=timezone.utc)
|
||||
cooldown_until = cooldown_until.replace(tzinfo=UTC)
|
||||
remaining_seconds = int((cooldown_until - now_utc).total_seconds())
|
||||
return max(0, remaining_seconds)
|
||||
|
||||
|
|
@ -432,7 +420,9 @@ def _normalize_author_search_inputs(query: str, limit: int) -> tuple[str, int, s
|
|||
|
||||
def _disabled_search_result(*, normalized_query: str, bounded_limit: int) -> ParsedAuthorSearchPage:
|
||||
structured_log(
|
||||
logger, "warning", "scholar_search.disabled_by_configuration",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_search.disabled_by_configuration",
|
||||
query=normalized_query,
|
||||
)
|
||||
return _policy_blocked_author_search_result(
|
||||
|
|
@ -452,13 +442,15 @@ def _normalize_runtime_cooldown_state(
|
|||
cooldown_until = runtime_state.cooldown_until
|
||||
updated = False
|
||||
if cooldown_until.tzinfo is None:
|
||||
cooldown_until = cooldown_until.replace(tzinfo=timezone.utc)
|
||||
cooldown_until = cooldown_until.replace(tzinfo=UTC)
|
||||
runtime_state.cooldown_until = cooldown_until
|
||||
updated = True
|
||||
if now_utc < cooldown_until:
|
||||
return updated
|
||||
structured_log(
|
||||
logger, "info", "scholar_search.cooldown_expired",
|
||||
logger,
|
||||
"info",
|
||||
"scholar_search.cooldown_expired",
|
||||
cooldown_until_utc=cooldown_until.isoformat(),
|
||||
)
|
||||
runtime_state.cooldown_until = None
|
||||
|
|
@ -494,7 +486,9 @@ def _emit_cooldown_threshold_alert(
|
|||
if bool(runtime_state.cooldown_alert_emitted):
|
||||
return True
|
||||
structured_log(
|
||||
logger, "error", "scholar_search.cooldown_rejection_threshold_exceeded",
|
||||
logger,
|
||||
"error",
|
||||
"scholar_search.cooldown_rejection_threshold_exceeded",
|
||||
query=normalized_query,
|
||||
cooldown_rejection_count=int(runtime_state.cooldown_rejection_count),
|
||||
threshold=threshold,
|
||||
|
|
@ -518,7 +512,9 @@ def _cooldown_block_result(
|
|||
cooldown_rejection_alert_threshold=cooldown_rejection_alert_threshold,
|
||||
)
|
||||
structured_log(
|
||||
logger, "warning", "scholar_search.cooldown_active",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_search.cooldown_active",
|
||||
query=normalized_query,
|
||||
cooldown_remaining_seconds=cooldown_remaining_seconds,
|
||||
cooldown_until_utc=runtime_state.cooldown_until.isoformat() if runtime_state.cooldown_until else None,
|
||||
|
|
@ -549,7 +545,9 @@ async def _cache_hit_result(
|
|||
if cached is None:
|
||||
return None
|
||||
structured_log(
|
||||
logger, "info", "scholar_search.cache_hit",
|
||||
logger,
|
||||
"info",
|
||||
"scholar_search.cache_hit",
|
||||
query=normalized_query,
|
||||
state=cached.state.value,
|
||||
state_reason=cached.state_reason,
|
||||
|
|
@ -576,7 +574,7 @@ def _throttle_sleep_seconds(
|
|||
else:
|
||||
last_live_request_at = runtime_state.last_live_request_at
|
||||
if last_live_request_at.tzinfo is None:
|
||||
last_live_request_at = last_live_request_at.replace(tzinfo=timezone.utc)
|
||||
last_live_request_at = last_live_request_at.replace(tzinfo=UTC)
|
||||
runtime_state.last_live_request_at = last_live_request_at
|
||||
updated = True
|
||||
enforced_wait_seconds = (
|
||||
|
|
@ -603,7 +601,9 @@ async def _wait_for_author_search_throttle(
|
|||
if sleep_seconds <= 0.0:
|
||||
return updated
|
||||
structured_log(
|
||||
logger, "info", "scholar_search.throttle_wait",
|
||||
logger,
|
||||
"info",
|
||||
"scholar_search.throttle_wait",
|
||||
query=normalized_query,
|
||||
sleep_seconds=round(sleep_seconds, 3),
|
||||
)
|
||||
|
|
@ -659,7 +659,9 @@ def _with_retry_warnings(
|
|||
if retry_scheduled_count < threshold:
|
||||
return merged
|
||||
structured_log(
|
||||
logger, "warning", "scholar_search.retry_threshold_exceeded",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_search.retry_threshold_exceeded",
|
||||
query=normalized_query,
|
||||
retry_scheduled_count=retry_scheduled_count,
|
||||
threshold=threshold,
|
||||
|
|
@ -688,19 +690,23 @@ def _apply_block_circuit_breaker(
|
|||
return merged_parsed
|
||||
runtime_state.consecutive_blocked_count = int(runtime_state.consecutive_blocked_count) + 1
|
||||
structured_log(
|
||||
logger, "warning", "scholar_search.block_detected",
|
||||
logger,
|
||||
"warning",
|
||||
"scholar_search.block_detected",
|
||||
query=normalized_query,
|
||||
state_reason=merged_parsed.state_reason,
|
||||
consecutive_blocked_count=int(runtime_state.consecutive_blocked_count),
|
||||
)
|
||||
if int(runtime_state.consecutive_blocked_count) < max(1, int(cooldown_block_threshold)):
|
||||
return merged_parsed
|
||||
runtime_state.cooldown_until = datetime.now(timezone.utc) + timedelta(seconds=max(60, int(cooldown_seconds)))
|
||||
runtime_state.cooldown_until = datetime.now(UTC) + timedelta(seconds=max(60, int(cooldown_seconds)))
|
||||
runtime_state.consecutive_blocked_count = 0
|
||||
runtime_state.cooldown_rejection_count = 0
|
||||
runtime_state.cooldown_alert_emitted = False
|
||||
structured_log(
|
||||
logger, "error", "scholar_search.cooldown_activated",
|
||||
logger,
|
||||
"error",
|
||||
"scholar_search.cooldown_activated",
|
||||
query=normalized_query,
|
||||
cooldown_until_utc=runtime_state.cooldown_until.isoformat() if runtime_state.cooldown_until else None,
|
||||
)
|
||||
|
|
@ -739,11 +745,11 @@ async def _cooldown_or_cache_result(
|
|||
) -> tuple[ParsedAuthorSearchPage | None, bool]:
|
||||
runtime_state_updated = _normalize_runtime_cooldown_state(
|
||||
runtime_state,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
)
|
||||
cooldown_remaining_seconds = _author_search_cooldown_remaining_seconds(
|
||||
runtime_state,
|
||||
datetime.now(timezone.utc),
|
||||
datetime.now(UTC),
|
||||
)
|
||||
if cooldown_remaining_seconds > 0:
|
||||
return (
|
||||
|
|
@ -759,18 +765,35 @@ async def _cooldown_or_cache_result(
|
|||
cached_result = await _cache_hit_result(
|
||||
db_session,
|
||||
query_key=query_key,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
normalized_query=normalized_query,
|
||||
bounded_limit=bounded_limit,
|
||||
)
|
||||
return cached_result, runtime_state_updated
|
||||
|
||||
|
||||
async def _perform_live_author_search(db_session: AsyncSession, *, source: ScholarSource, runtime_state: AuthorSearchRuntimeState, normalized_query: str, query_key: str, network_error_retries: int, retry_backoff_seconds: float, min_interval_seconds: float, interval_jitter_seconds: float, retry_alert_threshold: int, cooldown_block_threshold: int, cooldown_seconds: int, blocked_cache_ttl_seconds: int, cache_ttl_seconds: int, cache_max_entries: int) -> tuple[ParsedAuthorSearchPage, bool]:
|
||||
async def _perform_live_author_search(
|
||||
db_session: AsyncSession,
|
||||
*,
|
||||
source: ScholarSource,
|
||||
runtime_state: AuthorSearchRuntimeState,
|
||||
normalized_query: str,
|
||||
query_key: str,
|
||||
network_error_retries: int,
|
||||
retry_backoff_seconds: float,
|
||||
min_interval_seconds: float,
|
||||
interval_jitter_seconds: float,
|
||||
retry_alert_threshold: int,
|
||||
cooldown_block_threshold: int,
|
||||
cooldown_seconds: int,
|
||||
blocked_cache_ttl_seconds: int,
|
||||
cache_ttl_seconds: int,
|
||||
cache_max_entries: int,
|
||||
) -> tuple[ParsedAuthorSearchPage, bool]:
|
||||
runtime_state_updated = await _wait_for_author_search_throttle(
|
||||
runtime_state=runtime_state,
|
||||
normalized_query=normalized_query,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
min_interval_seconds=min_interval_seconds,
|
||||
interval_jitter_seconds=interval_jitter_seconds,
|
||||
)
|
||||
|
|
@ -780,7 +803,7 @@ async def _perform_live_author_search(db_session: AsyncSession, *, source: Schol
|
|||
network_error_retries=network_error_retries,
|
||||
retry_backoff_seconds=retry_backoff_seconds,
|
||||
)
|
||||
runtime_state.last_live_request_at = datetime.now(timezone.utc)
|
||||
runtime_state.last_live_request_at = datetime.now(UTC)
|
||||
merged = _with_retry_warnings(
|
||||
parsed,
|
||||
retry_warnings=retry_warnings,
|
||||
|
|
@ -806,12 +829,30 @@ async def _perform_live_author_search(db_session: AsyncSession, *, source: Schol
|
|||
parsed=merged,
|
||||
ttl_seconds=float(ttl_seconds),
|
||||
max_entries=cache_max_entries,
|
||||
now_utc=datetime.now(timezone.utc),
|
||||
now_utc=datetime.now(UTC),
|
||||
)
|
||||
return merged, True
|
||||
|
||||
|
||||
async def search_author_candidates(*, source: ScholarSource, db_session: AsyncSession, query: str, limit: int, network_error_retries: int = 1, retry_backoff_seconds: float = 1.0, search_enabled: bool = True, cache_ttl_seconds: int = 21_600, blocked_cache_ttl_seconds: int = DEFAULT_AUTHOR_SEARCH_BLOCKED_CACHE_TTL_SECONDS, cache_max_entries: int = DEFAULT_AUTHOR_SEARCH_CACHE_MAX_ENTRIES, min_interval_seconds: float = DEFAULT_AUTHOR_SEARCH_MIN_INTERVAL_SECONDS, interval_jitter_seconds: float = DEFAULT_AUTHOR_SEARCH_INTERVAL_JITTER_SECONDS, cooldown_block_threshold: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_BLOCK_THRESHOLD, cooldown_seconds: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_SECONDS, retry_alert_threshold: int = DEFAULT_AUTHOR_SEARCH_RETRY_ALERT_THRESHOLD, cooldown_rejection_alert_threshold: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_REJECTION_ALERT_THRESHOLD) -> ParsedAuthorSearchPage:
|
||||
async def search_author_candidates(
|
||||
*,
|
||||
source: ScholarSource,
|
||||
db_session: AsyncSession,
|
||||
query: str,
|
||||
limit: int,
|
||||
network_error_retries: int = 1,
|
||||
retry_backoff_seconds: float = 1.0,
|
||||
search_enabled: bool = True,
|
||||
cache_ttl_seconds: int = 21_600,
|
||||
blocked_cache_ttl_seconds: int = DEFAULT_AUTHOR_SEARCH_BLOCKED_CACHE_TTL_SECONDS,
|
||||
cache_max_entries: int = DEFAULT_AUTHOR_SEARCH_CACHE_MAX_ENTRIES,
|
||||
min_interval_seconds: float = DEFAULT_AUTHOR_SEARCH_MIN_INTERVAL_SECONDS,
|
||||
interval_jitter_seconds: float = DEFAULT_AUTHOR_SEARCH_INTERVAL_JITTER_SECONDS,
|
||||
cooldown_block_threshold: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_BLOCK_THRESHOLD,
|
||||
cooldown_seconds: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_SECONDS,
|
||||
retry_alert_threshold: int = DEFAULT_AUTHOR_SEARCH_RETRY_ALERT_THRESHOLD,
|
||||
cooldown_rejection_alert_threshold: int = DEFAULT_AUTHOR_SEARCH_COOLDOWN_REJECTION_ALERT_THRESHOLD,
|
||||
) -> ParsedAuthorSearchPage:
|
||||
normalized_query, bounded_limit, query_key = _normalize_author_search_inputs(query, limit)
|
||||
if not search_enabled:
|
||||
return _disabled_search_result(
|
||||
|
|
@ -924,17 +965,13 @@ async def set_profile_image_upload(
|
|||
normalized_content_type = (content_type or "").strip().lower()
|
||||
extension = ALLOWED_IMAGE_UPLOAD_CONTENT_TYPES.get(normalized_content_type)
|
||||
if extension is None:
|
||||
raise ScholarServiceError(
|
||||
"Unsupported image type. Use JPEG, PNG, WEBP, or GIF."
|
||||
)
|
||||
raise ScholarServiceError("Unsupported image type. Use JPEG, PNG, WEBP, or GIF.")
|
||||
|
||||
if not image_bytes:
|
||||
raise ScholarServiceError("Uploaded image file is empty.")
|
||||
|
||||
if len(image_bytes) > max_upload_bytes:
|
||||
raise ScholarServiceError(
|
||||
f"Uploaded image exceeds {max_upload_bytes} bytes."
|
||||
)
|
||||
raise ScholarServiceError(f"Uploaded image exceeds {max_upload_bytes} bytes.")
|
||||
|
||||
upload_root = _ensure_upload_root(upload_dir, create=True)
|
||||
user_dir = upload_root / str(profile.user_id)
|
||||
|
|
|
|||
|
|
@ -32,39 +32,25 @@ SEARCH_CACHED_BLOCK_REASON = "search_temporarily_disabled_from_cached_blocked_re
|
|||
|
||||
STATE_REASON_HINTS: dict[str, str] = {
|
||||
SEARCH_DISABLED_REASON: (
|
||||
"Scholar name search is currently disabled by service policy. "
|
||||
"Add scholars by profile URL or Scholar ID."
|
||||
"Scholar name search is currently disabled by service policy. Add scholars by profile URL or Scholar ID."
|
||||
),
|
||||
SEARCH_COOLDOWN_REASON: (
|
||||
"Scholar name search is temporarily paused after repeated block responses. "
|
||||
"Use Scholar URL/ID adds until cooldown expires."
|
||||
),
|
||||
SEARCH_CACHED_BLOCK_REASON: (
|
||||
"A recent blocked response was cached to reduce traffic. "
|
||||
"Retry later or add by Scholar URL/ID."
|
||||
"A recent blocked response was cached to reduce traffic. Retry later or add by Scholar URL/ID."
|
||||
),
|
||||
"network_dns_resolution_failed": (
|
||||
"DNS resolution failed while reaching scholar.google.com. "
|
||||
"Verify container DNS/network and retry."
|
||||
),
|
||||
"network_timeout": (
|
||||
"Request timed out before Google Scholar responded. "
|
||||
"Increase delay/backoff and retry."
|
||||
),
|
||||
"network_tls_error": (
|
||||
"TLS handshake/certificate validation failed. "
|
||||
"Verify outbound TLS/network configuration."
|
||||
),
|
||||
"blocked_http_429_rate_limited": (
|
||||
"Google Scholar rate-limited the request. "
|
||||
"Slow request cadence and retry later."
|
||||
"DNS resolution failed while reaching scholar.google.com. Verify container DNS/network and retry."
|
||||
),
|
||||
"network_timeout": ("Request timed out before Google Scholar responded. Increase delay/backoff and retry."),
|
||||
"network_tls_error": ("TLS handshake/certificate validation failed. Verify outbound TLS/network configuration."),
|
||||
"blocked_http_429_rate_limited": ("Google Scholar rate-limited the request. Slow request cadence and retry later."),
|
||||
"blocked_unusual_traffic_detected": (
|
||||
"Google Scholar flagged traffic as unusual. "
|
||||
"Increase delay/jitter and reduce concurrent scraping."
|
||||
"Google Scholar flagged traffic as unusual. Increase delay/jitter and reduce concurrent scraping."
|
||||
),
|
||||
"blocked_accounts_redirect": (
|
||||
"Request was redirected to Google Account sign-in. "
|
||||
"Treat as access block and retry later."
|
||||
"Request was redirected to Google Account sign-in. Treat as access block and retry later."
|
||||
),
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from app.db.models import ScholarProfile
|
||||
from app.services.domains.scholar.parser import ParseState, ParsedAuthorSearchPage
|
||||
from app.services.domains.scholar.parser import ParsedAuthorSearchPage, ParseState
|
||||
from app.services.domains.scholars.constants import (
|
||||
MAX_AUTHOR_SEARCH_LIMIT,
|
||||
STATE_REASON_HINTS,
|
||||
|
|
|
|||
|
|
@ -27,9 +27,7 @@ def normalize_profile_image_url(value: str | None) -> str | None:
|
|||
return None
|
||||
|
||||
if len(candidate) > MAX_IMAGE_URL_LENGTH:
|
||||
raise ScholarServiceError(
|
||||
f"Image URL must be {MAX_IMAGE_URL_LENGTH} characters or fewer."
|
||||
)
|
||||
raise ScholarServiceError(f"Image URL must be {MAX_IMAGE_URL_LENGTH} characters or fewer.")
|
||||
|
||||
parsed = urlparse(candidate)
|
||||
if parsed.scheme.lower() not in {"http", "https"} or not parsed.netloc:
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue