justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

removed old UI

Browse source
This commit is contained in:
Justin Visser 2026-02-17 15:21:08 +01:00
parent 4433d7d2c4
commit f71841e922
62 changed files with 411 additions and 5815 deletions
Download patch file Download diff file Expand all files Collapse all files

223
README.md
View file

@ -1,41 +1,80 @@
# scholarr
Container-first, self-hosted scholar tracking in the spirit of the `*arr` ecosystem, with strict multi-user isolation and an intentionally small operational footprint.
API-first, self-hosted scholar tracking backend in the spirit of the `*arr` ecosystem.
The legacy server-rendered UI has been removed. This repository now focuses on core ingestion, scheduling, and multi-user API functionality.
## Current Scope
- Admin-managed users only (no public signup)
- Session auth with CSRF protection and login rate limiting
- Per-user scholar tracking list (add, enable/disable, delete)
- Per-user automation settings
- Manual per-user ingestion runs from dashboard
- Per-user run history and "new since last run" publication stream
- Publications library view with `new` and `all` modes plus scholar filtering
- Dedicated run diagnostics pages (`/runs`, `/runs/{id}`) with failed-only filtering
- Continuation queue visibility + controls on runs page (`retry`, `drop`, `clear`)
- Baseline-aware discovery tracking (`new` = discovered in latest completed run)
- Read state is user-controlled (`Mark All Read`), including baseline items
- Multi-page profile ingestion (`cstart`) to capture full publication lists
- Dashboard rendered via a presentation-layer view model and section template registry
- Structured application logging with request IDs and field redaction
- Ingestion overlap guard via Postgres advisory locks
- Network-error retry support for ingestion attempts
- Automatic continuation queue for resumable/limit-hit scholar runs
- Background scheduler for per-user automatic runs
- User self-service password changes
- Admin user management (create users, activate/deactivate, reset passwords)
- PostgreSQL + Alembic migrations + containerized test workflow
- Versioned backend API (`/api/v1`) for frontend decoupling (same-origin cookie session)
- Multi-user accounts with admin-managed user lifecycle
- Same-origin cookie sessions with CSRF enforcement
- API auth flows (login, logout, me, password change)
- Admin user management API
- Scholar CRUD per user
- Per-user ingestion settings
- Manual runs with idempotency support
- Run history, run detail diagnostics, and continuation queue actions
- Publication listing (`new` / `all`) and mark-all-read
- Ingestion scheduler + continuation queue retries
- Structured logging with request IDs + redaction
- PostgreSQL + Alembic migrations
- Container-first development workflow with `uv`
## Tech Stack
## Functionality Tracking
- Python 3.12+
- FastAPI
- PostgreSQL 15+
- SQLAlchemy 2 + Alembic
- Jinja2 templates + modular theme tokens
- `uv` for dependency and runtime workflow
- Docker / Docker Compose for development and CI parity
Planned and supported backend functionality is tracked in:
- `AGENTS.MD`
## API Base
- Base path: `/api/v1`
- Success envelope:
- `{"data": ..., "meta": {"request_id": "..."}}`
- Error envelope:
- `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
## Auth & Session Model
- Session transport: same-origin cookie session (`HttpOnly`, `SameSite=Lax`)
- CSRF:
- Required for unsafe methods (`POST`, `PUT`, `PATCH`, `DELETE`) via `X-CSRF-Token`
- Bootstrap token via `GET /api/v1/auth/csrf`
## API Surface
- Auth:
- `GET /api/v1/auth/csrf`
- `POST /api/v1/auth/login`
- `GET /api/v1/auth/me`
- `POST /api/v1/auth/change-password`
- `POST /api/v1/auth/logout`
- Admin users:
- `GET /api/v1/admin/users`
- `POST /api/v1/admin/users`
- `PATCH /api/v1/admin/users/{id}/active`
- `POST /api/v1/admin/users/{id}/reset-password`
- Scholars:
- `GET /api/v1/scholars`
- `POST /api/v1/scholars`
- `PATCH /api/v1/scholars/{id}/toggle`
- `DELETE /api/v1/scholars/{id}`
- Settings:
- `GET /api/v1/settings`
- `PUT /api/v1/settings`
- Runs:
- `GET /api/v1/runs`
- `GET /api/v1/runs/{id}`
- `POST /api/v1/runs/manual` (`Idempotency-Key` supported)
- `GET /api/v1/runs/queue/items`
- `POST /api/v1/runs/queue/{id}/retry`
- `POST /api/v1/runs/queue/{id}/drop`
- `DELETE /api/v1/runs/queue/{id}`
- Publications:
- `GET /api/v1/publications`
- `POST /api/v1/publications/mark-all-read`
- Ops:
- `GET /healthz`
## Quick Start
@ -45,7 +84,7 @@ Container-first, self-hosted scholar tracking in the spirit of the `*arr` ecosys
cp .env.example .env
```
2. Bootstrap the first admin on startup (recommended for first run):
2. Optional bootstrap admin on startup:
```bash
export BOOTSTRAP_ADMIN_ON_START=1
@ -53,28 +92,20 @@ export BOOTSTRAP_ADMIN_EMAIL=admin@example.com
export BOOTSTRAP_ADMIN_PASSWORD=change-me-now
```
3. Start the stack:
3. Start stack:
```bash
docker compose up --build
```
4. Open:
4. Health check:
- Login: `http://localhost:8000/login`
- Dashboard: `http://localhost:8000/`
- Healthcheck: `http://localhost:8000/healthz`
5. After first successful admin login, disable auto-bootstrap:
```bash
export BOOTSTRAP_ADMIN_ON_START=0
```text
http://localhost:8000/healthz
```
## Admin Bootstrap (Manual)
You can create/update an admin account without restarting the app:
```bash
docker compose run --rm app uv run python scripts/bootstrap_admin.py \
--email admin@example.com \
@ -82,18 +113,8 @@ docker compose run --rm app uv run python scripts/bootstrap_admin.py \
--force-password
```
Notes:
- If the user does not exist, this creates an active admin account.
- If the user exists, it ensures admin + active flags.
- `--force-password` resets the password for existing users.
## Test Workflow
- Integration/smoke tests run against `TEST_DATABASE_URL`.
- If `TEST_DATABASE_URL` is empty, test runs derive an isolated database from `DATABASE_URL` by appending `_test`.
- This keeps app data in `DATABASE_URL` untouched during test runs.
- Parser regression tests are pinned to captured real-world fixture pages in `tests/fixtures/scholar/regression`.
- Unit tests:
```bash
@ -106,101 +127,29 @@ docker compose run --rm app uv run pytest tests/unit
docker compose run --rm app uv run pytest -m integration
```
- Smoke checks (container + DB + migration sanity):
- Smoke checks:
```bash
./scripts/smoke_compose.sh
```
CI runs migration, unit, and integration checks on push/PR via `.github/workflows/ci.yml`.
## Scholar Ingestion Notes
- Current ingestion targets public profile pages: `/citations?user=<id>`.
- Runs are intentionally conservative and prioritize reliability over aggressive scraping.
- Ingestion follows pagination (`cstart`) to collect all reachable profile pages.
- If pagination cannot be completed (max pages hit, cursor stall, or page-state failure), the scholar result is marked partial with explicit debug context.
- Resumable partial/failure states are automatically queued and retried by the scheduler with bounded backoff.
- Queue items keep explicit status (`queued`, `retrying`, `dropped`) and last-error context for UI diagnostics.
- Failed scholar attempts persist structured debug context in `crawl_runs.error_log` (state reason, fetch metadata, marker evidence, and compact response excerpt).
- Only resumable states are retried automatically (`network_error` and continuation-eligible pagination truncations); blocked/layout states are recorded immediately as failures.
## API v1 (Frontend Prep)
- Base path: `/api/v1`
- Auth model: same-origin cookie session (no token auth added)
- CSRF model: required for unsafe methods (`POST`, `PUT`, `PATCH`, `DELETE`) via `X-CSRF-Token` header.
- Bootstrap CSRF via `GET /api/v1/auth/csrf` (works for anonymous and authenticated sessions).
- You can also fetch `csrf_token` from `GET /api/v1/auth/me` after login.
- Manual run idempotency is supported via `Idempotency-Key` header on `POST /api/v1/runs/manual`.
- Response envelopes:
- Success: `{"data": ..., "meta": {"request_id": "..."}}`
- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
- Initial API domains exposed:
- `auth`: `/auth/csrf`, `/auth/login`, `/auth/me`, `/auth/change-password`, `/auth/logout`
- `admin users`: `/admin/users` (list/create), `/admin/users/{id}/active`, `/admin/users/{id}/reset-password`
- `scholars`: list/create/toggle/delete
- `settings`: get/update
- `runs`: list/detail/manual trigger + queue list/retry/drop/clear
- `publications`: list + mark-all-read
- Queue action semantics:
- `retry` is valid only for dropped items; retrying/queued states return `409`.
- `drop` returns the updated queue item (status `dropped`); dropping an already dropped item returns `409`.
- `clear` only works for dropped items and returns `{queue_item_id, previous_status, status="cleared"}`.
## Logging
- Default output is concise console logs to stdout.
- Timestamp format is compact UTC: `YYYY-MM-DD HH:MM:SSZ`.
- Every request gets an `X-Request-ID` (propagated if caller provides one).
- Sensitive fields are redacted before log emission (`password`, `csrf_token`, `cookie`, etc.).
- Configure via env:
- `LOG_LEVEL` (default `INFO`)
- `LOG_FORMAT` (`console` or `json`, default `console`)
- `LOG_REQUESTS` (`1`/`0`, default `1`)
- `LOG_UVICORN_ACCESS` (`1`/`0`, default `0`)
- `LOG_REQUEST_SKIP_PATHS` (comma-separated prefixes, default `/healthz,/static/`)
- `LOG_REDACT_FIELDS` (comma-separated additional keys)
- `SCHEDULER_ENABLED` (`1`/`0`, default `1`)
- `SCHEDULER_TICK_SECONDS` (default `60`)
- `INGESTION_NETWORK_ERROR_RETRIES` (default `1`)
- `INGESTION_RETRY_BACKOFF_SECONDS` (default `1.0`)
- `INGESTION_MAX_PAGES_PER_SCHOLAR` (default `30`)
- `INGESTION_PAGE_SIZE` (default `100`)
- `INGESTION_CONTINUATION_QUEUE_ENABLED` (`1`/`0`, default `1`)
- `INGESTION_CONTINUATION_BASE_DELAY_SECONDS` (default `120`)
- `INGESTION_CONTINUATION_MAX_DELAY_SECONDS` (default `3600`)
- `INGESTION_CONTINUATION_MAX_ATTEMPTS` (default `6`)
- `SCHEDULER_QUEUE_BATCH_SIZE` (default `10`)
Configurable env vars:
## Optional Local `uv` Workflow
```bash
uv sync --extra dev
uv run pytest
uv run uvicorn app.main:app --reload
```
Update the lockfile after dependency changes:
```bash
uv lock
```
## Theming
- Theme registry: `app/theme.py`
- Semantic tokens: `app/static/theme.css`
- App styling: `app/static/app.css`
- Theme persistence: `app/static/theme.js`
- Dashboard presentation contract: `app/presentation/dashboard.py`
- `LOG_LEVEL` (default `INFO`)
- `LOG_FORMAT` (`console` or `json`, default `console`)
- `LOG_REQUESTS` (`1`/`0`, default `1`)
- `LOG_UVICORN_ACCESS` (`1`/`0`, default `0`)
- `LOG_REQUEST_SKIP_PATHS` (default `/healthz`)
- `LOG_REDACT_FIELDS` (additional redact keys)
## Project Layout
```text
app/ FastAPI app, auth/security modules, templates, services, DB wiring, presentation view-models
app/web/ Router modules, shared web helpers, and request middleware
app/ FastAPI app (API routers, auth, services, db, middleware)
alembic/ Migration environment and versions
scripts/ Entrypoint, DB wait, bootstrap, smoke automation
scripts/ Entrypoint, db wait/bootstrap, smoke automation
tests/ Unit, integration, and smoke suites
planning/ Scope and implementation planning notes
```