m

AGENTS.MD

@@ -1,314 +1,28 @@
-# scholarr Agent Handbook
-
-This file is the consolidated source of truth for project scope, constraints, scrape contract, and UI implementation flow.
-
-## 1. Product Objective
-
-- Build a self-hosted scholar tracking system ("scholarr") with reliable, low-cost scraping and clear, actionable UI.
-- Keep the MVP scope intentionally small.
-- Prioritize reliability and ease of use over advanced/fancy processing.
-
-## 2. Locked Constraints and Preferences
-
-- Multi-user system with strict tenant isolation.
-- Users are admin-created only.
-- Users can change their own password.
-- Admin features are shown only to admin users.
-- Same-origin cookie session model with CSRF protection.
-- Container-first development workflow (`docker compose`, `uv`).
-- Test while developing; avoid shipping untested flows.
-- Keep backend modular and DRY.
-- UI is being rebuilt from scratch against API contracts.
-
-## 3. Current Backend Architecture
-
-- Runtime: Python + FastAPI.
-- DB: PostgreSQL + SQLAlchemy + Alembic migrations.
-- Scheduler: background scheduler + continuation queue processing.
-- Auth:
-  - Argon2 password hashing.
-  - Session cookie (`HttpOnly`, `SameSite=Lax`, secure flag configurable).
-  - CSRF required for unsafe methods via `X-CSRF-Token`.
-- Logging:
-  - structured request logging with request IDs.
-  - redaction controls for sensitive fields. 
-
-## 4. Scraping Contract (Probe-Based)
-
-Status: sufficient for MVP implementation with graceful degradation.
-
-Target endpoint:
-- `https://scholar.google.com/citations?hl=en&user=<scholar_id>`
-
-Primary selectors:
-- Row: `tr.gsc_a_tr`
-- Title/link: `a.gsc_a_at`
-- Citation count: `a.gsc_a_ac`
-- Year: `span.gsc_a_h` (fallback regex on year cell text)
-- Metadata lines: first/second `div.gs_gray` -> authors/venue
-- Cluster ID: from `citation_for_view=<user>:<cluster_id>` in title URL
-
-Required parser states:
-- `ok`
-- `no_results`
-- `blocked_or_captcha`
-- `layout_changed`
-- `network_error`
-
-Required page flags:
-- `has_show_more_button`
-- `articles_range`
-
-Quality assumptions from probe:
-- title/cluster_id/citation/authors coverage observed near 100%
-- year and venue may be missing and must remain nullable
-
-Guardrails:
-- Never crash the full run when one scholar fails.
-- Persist structured failure/debug information for diagnosis.
-- Handle inaccessible/redirected IDs as states, not exceptions.
-
-## 5. Current API Scope
-
-Base path: `/api/v1`
-
-Envelope model:
-- success: `{"data": ..., "meta": {"request_id": "..."}}`
-- error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
-
-Auth/session:
-- `GET /api/v1/auth/csrf`
-- `POST /api/v1/auth/login`
-- `GET /api/v1/auth/me`
-- `POST /api/v1/auth/change-password`
-- `POST /api/v1/auth/logout`
-
-Admin users:
-- `GET /api/v1/admin/users`
-- `POST /api/v1/admin/users`
-- `PATCH /api/v1/admin/users/{id}/active`
-- `POST /api/v1/admin/users/{id}/reset-password`
-
-Scholars:
-- `GET /api/v1/scholars`
-- `POST /api/v1/scholars`
-- `GET /api/v1/scholars/search`
-- `PATCH /api/v1/scholars/{id}/toggle`
-- `PUT /api/v1/scholars/{id}/image/url`
-- `POST /api/v1/scholars/{id}/image/upload`
-- `GET /api/v1/scholars/{id}/image/upload`
-- `DELETE /api/v1/scholars/{id}/image`
-- `DELETE /api/v1/scholars/{id}`
-
-Settings:
-- `GET /api/v1/settings`
-- `PUT /api/v1/settings`
-
-Runs/diagnostics/queue:
-- `GET /api/v1/runs`
-- `GET /api/v1/runs/{id}`
-- `POST /api/v1/runs/manual` (`Idempotency-Key` supported)
-- `GET /api/v1/runs/queue/items`
-- `POST /api/v1/runs/queue/{id}/retry`
-- `POST /api/v1/runs/queue/{id}/drop`
-- `DELETE /api/v1/runs/queue/{id}`
-
-Publications:
-- `GET /api/v1/publications`
-- `POST /api/v1/publications/mark-read`
-- `POST /api/v1/publications/mark-all-read`
-
-Ops:
-- `GET /healthz`
-
-## 5.1 Runtime Semantics (Locked)
-
-Manual run idempotency:
-- `Idempotency-Key` is optional on `POST /api/v1/runs/manual`.
-- Idempotency scope is `(user_id, idempotency_key)`.
-- Replaying the same key for the same user returns the same run payload after completion.
-- If the keyed run is still in progress, respond with `409 run_in_progress` and include `run_id`.
-- Enforcement is database-backed, not best-effort in-memory logic.
-- Frontend triggers generate keys internally; key input is not user-facing.
-
-Continuation queue attempt policy:
-- `attempt_count` is a failed-attempt budget, not a total execution counter.
-- Increment attempts only when a continuation execution fails to make progress.
-- Reset attempts to `0` after successful progress (success or resumable partial progress).
-- Drop queue items only after failed-attempt threshold is reached.
-
-Scholar no-change skip policy:
-- Persist first-page fingerprint snapshots per scholar.
-- If the first-page fingerprint is unchanged for a normal run (`cstart=0`), skip deep pagination/upsert work.
-- Record run scholar result `state_reason` as `no_change_initial_page_signature`.
-
-Scholar create naming policy:
-- Do not accept custom display names during scholar create.
-- Hydrate names from scraped profile metadata.
-
-Scholar name-search safety policy:
-- Treat name search as best-effort only.
-- Apply single-flight execution, pacing/jitter, cache, and cooldown circuit breaker after repeated blocked responses.
-- Prefer Scholar URL/ID adds for reliable ingestion onboarding.
-
-## 6. Required UI Behavior (MVP)
-
-Core UX entities:
-- scholars
-- publications (`all` and `new`)
-- runs + run diagnostics
-- queue visibility/actions
-- settings/account
-- admin users (role-gated)
-
-Critical semantics:
-- Keep `is_new_in_latest_run` separate from `is_read`.
-- First-time ingestion should not imply user read-state.
-- Surface partial runs and warnings clearly.
-- Show loading, empty, and error states explicitly.
-- Support both light and dark modes with consistent status semantics.
-
-## 7. UI Information Architecture
-
-Public:
-- Login
-
-Authenticated:
-- Dashboard
-- Scholars
-- Publications
-- Settings
-- Admin Users (admin only)
-
-## 8. End-to-End UI Flow Plan
-
-## 8.1 Session bootstrap
-
-1. On load call `GET /api/v1/auth/csrf`.
-2. Call `GET /api/v1/auth/me`.
-3. Route to login or dashboard based on auth state.
-
-## 8.2 Login/logout
-
-1. Login form submits `POST /api/v1/auth/login` with `X-CSRF-Token`.
-2. Store CSRF token from response for future unsafe requests.
-3. Logout calls `POST /api/v1/auth/logout`.
-
-## 8.3 Dashboard
-1. Fetch latest publications summary from `GET /api/v1/publications?mode=new&limit=20`.
-2. Fetch recent runs from `GET /api/v1/runs?limit=5`.
-3. Fetch queue status from `GET /api/v1/runs/queue/items?limit=200` (aggregate in UI).
-4. Show:
-   - new publications count (`new_count`)
-   - total tracked publications (`total_count`)
-   - latest run status/started time
-   - queue badges (`queued`, `retrying`, `dropped`)
-5. Admin users additionally see a diagnostics shortcut block to full runs/queue page.
-6. Every dashboard error panel must display request-id for support.
-
-
-## 8.4 Scholars
-
-1. List via `GET /api/v1/scholars`.
-2. Add/toggle/delete actions mapped to scholar endpoints.
-3. Search by name via `GET /api/v1/scholars/search` with candidate add actions.
-4. Profile image controls:
-   - scraped image fallback from Scholar metadata
-   - custom URL override
-   - image upload + reset to scraped/default
-5. Per-row quick links:
-   - all publications filtered by scholar
-   - new publications filtered by scholar
-
-## 8.5 Publications
-
-1. Default view mode is `new` (`GET /api/v1/publications?mode=new`).
-2. Support mode toggle:
-   - `new`: scoped to latest completed run
-   - `all`: full history
-3. Support scholar filter via `scholar_profile_id` query param.
-4. Render both badges independently:
-   - `is_new_in_latest_run`
-   - `is_read`
-5. "Mark all read" action calls `POST /api/v1/publications/mark-all-read` with CSRF.
-6. Show explicit empty states:
-   - no publications tracked
-   - no new publications in latest run
-   - filtered scholar has no items
-7. Surface run recency context when mode is `new` (latest completed run semantics).
-
-## 8.6 Runs and diagnostics (admin)
-
-1. List runs from `GET /api/v1/runs`.
-2. Run details from `GET /api/v1/runs/{id}`:
-   - scholar result state/reason
-   - warnings
-   - page logs + attempt logs
-   - debug metadata
-3. Queue operations from queue endpoints (`retry`, `drop`, `clear`).
-
-## 8.7 Settings + account
-
-1. User ingest settings from `/api/v1/settings`.
-2. Password change via `/api/v1/auth/change-password`.
-
-## 8.8 Admin user management
-
-1. Show section only when `current_user.is_admin=true`.
-2. Use admin user endpoints for create/activate/deactivate/reset password.
-
-## 9. Development Method
-
-- API-contract-first frontend.
-- Vertical slices (complete flow before moving on).
-- Shared API client module:
-  - always send credentials
-  - inject CSRF header for unsafe methods
-  - normalize envelope parsing and error handling
-- Keep frontend modules separate (`auth`, `scholars`, `publications`, `runs`, `settings`, `admin`).
-- Write tests while implementing each slice.
-
-## 10. Definition of Done (UI Readiness)
-
-- All main flows above implemented and mapped to existing API endpoints.
-- Role-gated admin experience enforced in UI.
-- Distinct new-vs-read publication semantics visible.
-- Run/queue diagnostics visible and actionable.
-- Reliable loading/empty/error states with request-id surfaced for support.
-- No dependence on removed server-rendered UI layer.
-
-## 11. Open Items (Known Future Work)
-
-- API contract freeze/changelog discipline for external UI clients.
-- Additional API-first smoke coverage for end-to-end UI critical flows.
-- Optional background image enrichment/refresh jobs for already-tracked scholars.
-
-## 12. UI Rebuild Program Artifacts
-
-- Phase 0 through Phase 3 decisions are consolidated into this handbook and `README.md`.
-- Local scratch notes, probes, and temporary planning artifacts are intentionally not tracked in git.
-
-## 13. Frontend Scaffold Status (Implemented)
-
-- Frontend scaffold path: `frontend/`
-- Implemented core:
-  - App shell + routing + auth/role route guards
-  - Shared API client with envelope parsing, credentials, and CSRF header injection
-  - Pinia stores for auth, theme, and global UI errors
-  - Tailwind-first design foundation with light/dark themes and pre-paint theme initialization
-  - Page shells for login, dashboard, scholars, publications, runs, run detail, settings, and admin users
-  - Initial feature modules under `frontend/src/features/*` mapped to current API endpoints
-- Deployment boundary:
-  - No in-repo reverse-proxy implementation
-  - Frontend edge/proxy routing is managed externally (for example, Caddy)
-
-## 14. Frontend Quality Gates (Phase 3 Implemented)
-
-- CI now includes frontend quality gates in `.github/workflows/ci.yml`:
-  - API contract drift check (`scripts/check_frontend_api_contract.py`)
-  - frontend typecheck (`npm run typecheck`)
-  - frontend unit tests (`npm run test:run`)
-  - frontend build (`npm run build`)
-- Frontend test harness:
-  - `vitest` configured in `frontend/vitest.config.ts`
-  - baseline tests in `frontend/src/lib/api/envelope.test.ts` and `frontend/src/lib/api/errors.test.ts`
+# AI Agent Instructions: Scholarr
+
+Adhere strictly to these constraints.
+
+## 1. Coding Standards (Strict Enforcement)
+* **Function Length:** Maximum 50 lines of code per function. Break down complex logic into small, testable, single-responsibility functions.
+* **DRY (Don't Repeat Yourself):** Abstract repetitive logic immediately. No duplicate boilerplate for database queries, API responses, or error handling.
+* **Negative Space Programming:** Utilize explicit assertions and constraints to define invalid states. Fail fast and early. Do not allow silent failures or cascading malformed data, especially in DOM parsing.
+* **Cyclomatic Complexity:** Flatten logic. Use early returns and guard clauses instead of deep nesting.
+
+## 2. Domain Architecture & Data Model
+* **Data Isolation:** Scholar tracking is **user-scoped**. Validate mapping/join tables; never assume global links between users and Scholar IDs.
+* **Data Deduplication:** Publications are **global records**. Deduplicate via Scholar cluster ID and normalized fingerprinting prior to database insertion.
+* **State Management:** Visibility and "read/unread" states exist exclusively on the scholar-publication link table, not the global publication table.
+* **API Contract:** Exact envelope format required:
+    * Success: `{"data": ..., "meta": {"request_id": "..."}}`
+    * Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
+
+## 3. Scrape Safety & Rate Limiting (Immutable)
+These limits prevent IP bans and are not to be optimized away.
+* **Minimum Delay:** Enforce `INGESTION_MIN_REQUEST_DELAY_SECONDS` (default 2s) between all external requests.
+* **Anti-Detection:** Default to direct ID or profile URL ingestion. Name searches trigger CAPTCHAs.
+* **Cooldowns:** Respect `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS` (1800s) and `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS` (900s) upon threshold breaches.
+
+## 4. Current Environment & Stack
+* **Backend:** Python 3.12+, FastAPI, SQLAlchemy (Async/asyncpg), Alembic.
+* **Frontend:** TypeScript, React, Vite.
+* **Infrastructure:** Multi-stage Docker.