Big changes

docs/reference/api_contract.md

@@ -0,0 +1,29 @@
+# API Contract
+
+## Envelope Invariant
+
+All API responses under `/api/v1` use one of these envelopes:
+
+- Success: `{"data": ..., "meta": {"request_id": "..."}}`
+- Error: `{"error": {"code": "...", "message": "...", "details": ...}, "meta": {"request_id": "..."}}`
+
+`meta.request_id` must be present for success and error responses.
+
+## Publications Semantics
+
+- `GET /api/v1/publications` supports `mode=all|unread|latest`.
+- `mode=new` is currently accepted as a compatibility alias for `latest`.
+- Pagination controls: `page`, `page_size` (with backward-compatible `limit`/`offset` support).
+- Pagination fields in response: `page`, `page_size`, `has_prev`, `has_next`, `total_count`.
+- `unread` represents read-state (`is_read=false`).
+- `latest` represents discovery-state (`first seen in the latest completed run`).
+
+Publication payloads expose:
+- `pub_url` (canonical scholar detail URL)
+- `doi` (normalized DOI)
+- `pdf_url` (resolved OA PDF when available)
+
+## Scholar Portability
+
+- `GET /api/v1/scholars/export` exports tracked scholars and scholar-publication link state.
+- `POST /api/v1/scholars/import` imports that payload while preserving global publication deduplication.

docs/reference/environment.md

@@ -0,0 +1,68 @@
+# Environment Reference and Audit
+
+This document is the source-of-truth audit for what remains configurable versus internal defaults.
+
+## Decision Summary
+
+- Keep in `.env.example`: all runtime and compose controls currently exposed by `app/settings.py` plus compose/bootstrap extras.
+- Move to internal defaults only: none at this time.
+- Internal fallback behavior still exists in code to keep local/test startup resilient when values are omitted.
+
+## Configurable Variables (By Section)
+
+### Compose + Database
+
+`POSTGRES_DB`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `DATABASE_URL`, `TEST_DATABASE_URL`, `SCHOLARR_IMAGE`
+
+### App Runtime + Networking
+
+`APP_NAME`, `APP_HOST`, `APP_PORT`, `APP_HOST_PORT`, `APP_RELOAD`, `MIGRATE_ON_START`, `FRONTEND_ENABLED`, `FRONTEND_DIST_DIR`
+
+### Database Pool
+
+`DATABASE_POOL_MODE`, `DATABASE_POOL_SIZE`, `DATABASE_POOL_MAX_OVERFLOW`, `DATABASE_POOL_TIMEOUT_SECONDS`
+
+### Frontend Dev Overrides
+
+`FRONTEND_HOST_PORT`, `CHOKIDAR_USEPOLLING`, `VITE_DEV_API_PROXY_TARGET`
+
+### Auth + Session
+
+`SESSION_SECRET_KEY`, `SESSION_COOKIE_SECURE`, `LOGIN_RATE_LIMIT_ATTEMPTS`, `LOGIN_RATE_LIMIT_WINDOW_SECONDS`
+
+### HTTP Security Headers + CSP
+
+`SECURITY_HEADERS_ENABLED`, `SECURITY_X_CONTENT_TYPE_OPTIONS`, `SECURITY_X_FRAME_OPTIONS`, `SECURITY_REFERRER_POLICY`, `SECURITY_PERMISSIONS_POLICY`, `SECURITY_CROSS_ORIGIN_OPENER_POLICY`, `SECURITY_CROSS_ORIGIN_RESOURCE_POLICY`, `SECURITY_CSP_ENABLED`, `SECURITY_CSP_POLICY`, `SECURITY_CSP_DOCS_POLICY`, `SECURITY_CSP_REPORT_ONLY`, `SECURITY_STRICT_TRANSPORT_SECURITY_ENABLED`, `SECURITY_STRICT_TRANSPORT_SECURITY_MAX_AGE`, `SECURITY_STRICT_TRANSPORT_SECURITY_INCLUDE_SUBDOMAINS`, `SECURITY_STRICT_TRANSPORT_SECURITY_PRELOAD`
+
+### Logging
+
+`LOG_LEVEL`, `LOG_FORMAT`, `LOG_REQUESTS`, `LOG_UVICORN_ACCESS`, `LOG_REQUEST_SKIP_PATHS`, `LOG_REDACT_FIELDS`
+
+### Scheduler + Ingestion Safety
+
+`SCHEDULER_ENABLED`, `SCHEDULER_TICK_SECONDS`, `SCHEDULER_QUEUE_BATCH_SIZE`, `INGESTION_AUTOMATION_ALLOWED`, `INGESTION_MANUAL_RUN_ALLOWED`, `INGESTION_MIN_RUN_INTERVAL_MINUTES`, `INGESTION_MIN_REQUEST_DELAY_SECONDS`, `INGESTION_NETWORK_ERROR_RETRIES`, `INGESTION_RETRY_BACKOFF_SECONDS`, `INGESTION_MAX_PAGES_PER_SCHOLAR`, `INGESTION_PAGE_SIZE`, `INGESTION_ALERT_BLOCKED_FAILURE_THRESHOLD`, `INGESTION_ALERT_NETWORK_FAILURE_THRESHOLD`, `INGESTION_ALERT_RETRY_SCHEDULED_THRESHOLD`, `INGESTION_SAFETY_COOLDOWN_BLOCKED_SECONDS`, `INGESTION_SAFETY_COOLDOWN_NETWORK_SECONDS`, `INGESTION_CONTINUATION_QUEUE_ENABLED`, `INGESTION_CONTINUATION_BASE_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_DELAY_SECONDS`, `INGESTION_CONTINUATION_MAX_ATTEMPTS`
+
+### Scholar Images + Name Search Safety
+
+`SCHOLAR_IMAGE_UPLOAD_DIR`, `SCHOLAR_IMAGE_UPLOAD_MAX_BYTES`, `SCHOLAR_NAME_SEARCH_ENABLED`, `SCHOLAR_NAME_SEARCH_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_BLOCKED_CACHE_TTL_SECONDS`, `SCHOLAR_NAME_SEARCH_CACHE_MAX_ENTRIES`, `SCHOLAR_NAME_SEARCH_MIN_INTERVAL_SECONDS`, `SCHOLAR_NAME_SEARCH_INTERVAL_JITTER_SECONDS`, `SCHOLAR_NAME_SEARCH_COOLDOWN_BLOCK_THRESHOLD`, `SCHOLAR_NAME_SEARCH_COOLDOWN_SECONDS`, `SCHOLAR_NAME_SEARCH_ALERT_RETRY_COUNT_THRESHOLD`, `SCHOLAR_NAME_SEARCH_ALERT_COOLDOWN_REJECTIONS_THRESHOLD`
+
+### OA Enrichment + PDF Resolution
+
+`UNPAYWALL_ENABLED`, `UNPAYWALL_EMAIL`, `UNPAYWALL_TIMEOUT_SECONDS`, `UNPAYWALL_MIN_INTERVAL_SECONDS`, `UNPAYWALL_MAX_ITEMS_PER_REQUEST`, `UNPAYWALL_RETRY_COOLDOWN_SECONDS`, `UNPAYWALL_PDF_DISCOVERY_ENABLED`, `UNPAYWALL_PDF_DISCOVERY_MAX_CANDIDATES`, `UNPAYWALL_PDF_DISCOVERY_MAX_HTML_BYTES`, `PDF_AUTO_RETRY_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_FIRST_INTERVAL_SECONDS`, `PDF_AUTO_RETRY_MAX_ATTEMPTS`, `CROSSREF_ENABLED`, `CROSSREF_MAX_ROWS`, `CROSSREF_TIMEOUT_SECONDS`, `CROSSREF_MIN_INTERVAL_SECONDS`, `CROSSREF_MAX_LOOKUPS_PER_REQUEST`
+
+### Startup Bootstrap + DB Wait
+
+`BOOTSTRAP_ADMIN_ON_START`, `BOOTSTRAP_ADMIN_EMAIL`, `BOOTSTRAP_ADMIN_PASSWORD`, `BOOTSTRAP_ADMIN_FORCE_PASSWORD`, `DB_WAIT_TIMEOUT_SECONDS`, `DB_WAIT_INTERVAL_SECONDS`
+
+## Drift Guard
+
+CI enforces env parity with:
+
+```bash
+python3 scripts/check_env_contract.py
+```
+
+The check fails when:
+- `app/settings.py` references a variable missing from `.env.example`.
+- `.env.example` contains an unknown key (outside the approved compose/bootstrap extras).
+- `.env.example` contains duplicate keys.