diff --git a/README.md b/README.md index c05beaa..f0f3fac 100644 --- a/README.md +++ b/README.md @@ -1,94 +1,139 @@ -# scholarr -
-Self-hosted scholar tracking with a single app image (API + frontend). It automates publication discovery, parses Google Scholar, gathers external metadata via Crossref & arXiv APIs, and autonomously downloads Open Access PDFs. +Scholarr + +# Scholarr + +**Self-hosted academic publication tracker.** + +Track Google Scholar profiles, discover new papers automatically, +resolve open-access PDFs, and stay on top of the literature you care about. [![CI](https://img.shields.io/github/actions/workflow/status/justinzeus/scholarr/ci.yml?style=for-the-badge)](https://github.com/JustinZeus/scholarr/actions/workflows/ci.yml) [![Docker Pulls](https://img.shields.io/docker/pulls/justinzeus/scholarr?style=for-the-badge&logo=docker)](https://hub.docker.com/r/justinzeus/scholarr) -[![Docker Image](https://img.shields.io/badge/docker-justinzeus%2Fscholarr-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://hub.docker.com/r/justinzeus/scholarr) +[![Docs](https://img.shields.io/badge/docs-scholarr-2e8555?style=for-the-badge)](https://justinzeus.github.io/scholarr/)
-## Key Features -- **Automated Ingestion**: Background tasks regularly scrape author profiles to pull the latest publications. -- **Resilient Parsing**: Custom DOM parsing logic falls back heavily across changes, backed by a dynamic `429 Too Many Requests` backoff cycle. -- **Identifier Subsystem**: Isolates DOIs, PMIDs, and arXiv IDs independently of PDFs mapping fallback external APIs sequentially. -- **Unpaywall PDF Search**: Leverages a fully automated resolution queue to grab public PDFs via the Unpaywall API on identified entries. -- **Beautiful UI**: Polished Vue 3 frontend that ships natively inside the same Docker container as the FastAPI backend. +--- -## Architecture Data Flow -```mermaid -graph LR - HTML(Google Scholar) --> |Scraper| Ingestion - Ingestion --> |Extract DOI| DB[(Database)] - Ingestion -.-> |Fallback API Search| Crossref/arXiv - Crossref/arXiv --> DB - - DB --> |Identified DOIs| PDFQueue(Unpaywall/PDF Workers) - PDFQueue --> |Download & Store File| Storage -``` + + +

+ Publications dashboard +

+ + + +## Why Scholarr? + +Most researchers track new papers by manually checking Google Scholar, setting up email alerts, or juggling RSS feeds. Scholarr replaces all of that with a single self-hosted service: + +- **Add scholars once** -- by profile URL, Scholar ID, or name search +- **Publications appear automatically** -- a background scheduler scrapes profiles on a configurable interval +- **Open-access PDFs are resolved for you** -- Unpaywall and arXiv are queried automatically when a DOI is found +- **Everything is deduplicated** -- publications are global records; no duplicates across scholars +- **Your data stays yours** -- fully self-hosted, export/import your entire library at any time + +## Features + +| | | +|---|---| +| **Automated Ingestion** | Background scheduler with configurable intervals, continuation queue, and multi-page pagination | +| **Identifier Resolution** | Cross-references arXiv, Crossref, and OpenAlex to gather DOIs, arXiv IDs, PMIDs | +| **PDF Discovery** | Resolves open-access PDFs via Unpaywall API and arXiv, with automatic retry queue | +| **Scrape Safety** | Rate limiting, cooldowns, and backoff strategies that prevent IP bans -- these are safety floors, not optional | +| **Multi-User** | Session-based auth, admin user management, user-scoped scholar tracking | +| **Theming** | 7 color presets with light/dark mode, tokenized component system | +| **Import / Export** | Portable scholar data with full publication and read-state preservation | +| **Single Container** | FastAPI backend + Vue 3 frontend ship as one Docker image | ## Quick Start -1. Copy env template: - ```bash +# 1. Clone and configure +git clone https://github.com/JustinZeus/scholarr.git +cd scholarr cp .env.example .env + +# 2. Set required secrets in .env +# POSTGRES_PASSWORD= +# SESSION_SECRET_KEY= + +# 3. Start +docker compose up -d + +# 4. Open http://localhost:8000 ``` -2. Set required values in `.env`: -- `POSTGRES_PASSWORD` -- `SESSION_SECRET_KEY` - -3. Start stack: +To bootstrap an admin account on first run, add to `.env`: ```bash -docker compose pull -docker compose up -d +BOOTSTRAP_ADMIN_ON_START=1 +BOOTSTRAP_ADMIN_EMAIL=admin@example.com +BOOTSTRAP_ADMIN_PASSWORD= ``` -Open: -- App/API: `http://localhost:8000` -- Health: `http://localhost:8000/healthz` +## How It Works + +```mermaid +graph LR + Scholar[Google Scholar] -->|Scrape HTML| Parser[Parser & Fingerprinting] + Parser -->|Deduplicate| DB[(PostgreSQL)] + Parser -.->|Identify| APIs[arXiv / Crossref / OpenAlex] + APIs --> DB + DB -->|DOIs| PDF[PDF Resolution] + PDF -->|Unpaywall / arXiv| DB + DB --> UI[Vue 3 Dashboard] +``` + +## Tech Stack + +| Layer | Technology | +|-------|------------| +| Backend | Python 3.12, FastAPI, SQLAlchemy 2.0 (async), Alembic | +| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS | +| Database | PostgreSQL 15 | +| Infrastructure | Multi-stage Docker, Docker Compose | ## Documentation -Complete documentation is published at: -- https://justinzeus.github.io/scholarr/ +Full documentation: **[justinzeus.github.io/scholarr](https://justinzeus.github.io/scholarr/)** -You can also read the developer/architecture breakdowns directly within the repository under: -- `docs/developer/architecture.md` -- `docs/developer/ingestion.md` -- `docs/developer/api-contract.md` +| Section | Covers | +|---------|--------| +| [User Guide](docs/user/overview.md) | Installation, configuration, all environment variables | +| [Developer Guide](docs/developer/overview.md) | Architecture, local dev, contributing, testing | +| [Operations](docs/operations/overview.md) | Deployment, database runbook, scrape safety | +| [API Reference](docs/reference/api.md) | Envelope spec, all endpoints, DTO contracts | -## Quality Gates +## Contributing -Backend: +Scholarr uses [conventional commits](https://www.conventionalcommits.org/) and [semantic versioning](https://semver.org/). See the [contributing guide](docs/developer/contributing.md) for PR process and code standards. ```bash -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration +# Dev environment +docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build + +# Run tests (always in containers) +docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \ + python -m pytest ``` -Frontend: +## License -```bash -# ALL tests/builds MUST be run within the docker container. Do not run raw npm commands on the host. -# For automated agents checking outputs, redirect stdout/stderr to files (e.g. > /tmp/npm_typecheck.log 2>&1) -# The temp files /tmp/npm_test.log, /tmp/npm_lint.log, /tmp/npm_typecheck.log, and /tmp/pytest_eval.log are pre-authorized for overwrite. - -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm install -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run typecheck -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run test:run -docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run build -``` - -Contract and env checks: - -```bash -python3 scripts/check_frontend_api_contract.py -python3 scripts/check_env_contract.py -./scripts/check_no_generated_artifacts.sh -``` -thanks arxiv \ No newline at end of file +See [LICENSE](LICENSE) for details. diff --git a/docs/assets/.gitkeep b/docs/assets/.gitkeep new file mode 100644 index 0000000..e69de29