scholarr/AGENTS.MD

# scholarr Agent Handbook

This file is the consolidated source of truth for project scope, constraints, scrape contract, and UI implementation flow.

Raw probe artifacts in `planning/scholar_probe_tmp/notes/*.json` and `planning/scholar_probe_tmp/notes/*.md` remain historical evidence and fixture references.

## 1. Product Objective

- Build a self-hosted scholar tracking system ("scholarr") with reliable, low-cost scraping and clear, actionable UI.
- Keep the MVP scope intentionally small.
- Prioritize reliability and ease of use over advanced/fancy processing.

## 2. Locked Constraints and Preferences

- Multi-user system with strict tenant isolation.
- Users are admin-created only.
- Users can change their own password.
- Admin features are shown only to admin users.
- Same-origin cookie session model with CSRF protection.
- Container-first development workflow (`docker compose`, `uv`).
- Test while developing; avoid shipping untested flows.
- Keep backend modular and DRY.
- UI is being rebuilt from scratch against API contracts.

## 3. Current Backend Architecture

- Runtime: Python + FastAPI.
- DB: PostgreSQL + SQLAlchemy + Alembic migrations.
- Scheduler: background scheduler + continuation queue processing.
- Auth:
  - Argon2 password hashing.
  - Session cookie (`HttpOnly`, `SameSite=Lax`, secure flag configurable).
  - CSRF required for unsafe methods via `X-CSRF-Token`.
- Logging:
  - structured request logging with request IDs.
  - redaction controls for sensitive fields.

## 4. Scraping Contract (Probe-Based)

Status: sufficient for MVP implementation with graceful degradation.

Target endpoint:
- `https://scholar.google.com/citations?hl=en&user=<scholar_id>`

Primary selectors:
- Row: `tr.gsc_a_tr`
- Title/link: `a.gsc_a_at`
- Citation count: `a.gsc_a_ac`
- Year: `span.gsc_a_h` (fallback regex on year cell text)
- Metadata lines: first/second `div.gs_gray` -> authors/venue
- Cluster ID: from `citation_for_view=<user>:<cluster_id>` in title URL

Required parser states:
- `ok`
- `no_results`
- `blocked_or_captcha`
- `layout_changed`
- `network_error`

Required page flags:
- `has_show_more_button`
- `articles_range`

Quality assumptions from probe:
- title/cluster_id/citation/authors coverage observed near 100%
- year and venue may be missing and must remain nullable

Guardrails:
- Never crash the full run when one scholar fails.
- Persist structured failure/debug information for diagnosis.
- Handle inaccessible/redirected IDs as states, not exceptions.

## 5. Current API Scope

Base path: `/api/v1`

Envelope model:
- success: `{"data": ..., "meta": {"request_id": "..."}}`
- error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`

Auth/session:
- `GET /api/v1/auth/csrf`
- `POST /api/v1/auth/login`
- `GET /api/v1/auth/me`
- `POST /api/v1/auth/change-password`
- `POST /api/v1/auth/logout`

Admin users:
- `GET /api/v1/admin/users`
- `POST /api/v1/admin/users`
- `PATCH /api/v1/admin/users/{id}/active`
- `POST /api/v1/admin/users/{id}/reset-password`

Scholars:
- `GET /api/v1/scholars`
- `POST /api/v1/scholars`
- `PATCH /api/v1/scholars/{id}/toggle`
- `DELETE /api/v1/scholars/{id}`

Settings:
- `GET /api/v1/settings`
- `PUT /api/v1/settings`

Runs/diagnostics/queue:
- `GET /api/v1/runs`
- `GET /api/v1/runs/{id}`
- `POST /api/v1/runs/manual` (`Idempotency-Key` supported)
- `GET /api/v1/runs/queue/items`
- `POST /api/v1/runs/queue/{id}/retry`
- `POST /api/v1/runs/queue/{id}/drop`
- `DELETE /api/v1/runs/queue/{id}`

Publications:
- `GET /api/v1/publications`
- `POST /api/v1/publications/mark-all-read`

Ops:
- `GET /healthz`

## 6. Required UI Behavior (MVP)

Core UX entities:
- scholars
- publications (`all` and `new`)
- runs + run diagnostics
- queue visibility/actions
- settings/account
- admin users (role-gated)

Critical semantics:
- Keep `is_new_in_latest_run` separate from `is_read`.
- First-time ingestion should not imply user read-state.
- Surface partial runs and warnings clearly.
- Show loading, empty, and error states explicitly.

## 7. UI Information Architecture

Public:
- Login

Authenticated:
- Dashboard
- Scholars
- Publications
- Settings
- Admin Users (admin only)

## 8. End-to-End UI Flow Plan

## 8.1 Session bootstrap

1. On load call `GET /api/v1/auth/csrf`.
2. Call `GET /api/v1/auth/me`.
3. Route to login or dashboard based on auth state.

## 8.2 Login/logout

1. Login form submits `POST /api/v1/auth/login` with `X-CSRF-Token`.
2. Store CSRF token from response for future unsafe requests.
3. Logout calls `POST /api/v1/auth/logout`.

## 8.3 Dashboard



## 8.4 Scholars

1. List via `GET /api/v1/scholars`.
2. Add/toggle/delete actions mapped to scholar endpoints.
3. Per-row quick links:
   - all publications filtered by scholar
   - new publications filtered by scholar

## 8.5 Publications

## 8.6 Runs and diagnostics (admin)

1. List runs from `GET /api/v1/runs`.
2. Run details from `GET /api/v1/runs/{id}`:
   - scholar result state/reason
   - warnings
   - page logs + attempt logs
   - debug metadata
3. Queue operations from queue endpoints (`retry`, `drop`, `clear`).

## 8.7 Settings + account

1. User ingest settings from `/api/v1/settings`.
2. Password change via `/api/v1/auth/change-password`.

## 8.8 Admin user management

1. Show section only when `current_user.is_admin=true`.
2. Use admin user endpoints for create/activate/deactivate/reset password.

## 9. Development Method

- API-contract-first frontend.
- Vertical slices (complete flow before moving on).
- Shared API client module:
  - always send credentials
  - inject CSRF header for unsafe methods
  - normalize envelope parsing and error handling
- Keep frontend modules separate (`auth`, `scholars`, `publications`, `runs`, `settings`, `admin`).
- Write tests while implementing each slice.

## 10. Definition of Done (UI Readiness)

- All main flows above implemented and mapped to existing API endpoints.
- Role-gated admin experience enforced in UI.
- Distinct new-vs-read publication semantics visible.
- Run/queue diagnostics visible and actionable.
- Reliable loading/empty/error states with request-id surfaced for support.
- No dependence on removed server-rendered UI layer.

## 11. Open Items (Known Future Work)

- Scholar discovery by real name (search Google Scholar candidates, select one, then add scholar ID).
- API contract freeze/changelog discipline for external UI clients.
- Additional API-first smoke coverage for end-to-end UI critical flows.