justinzeus/scholarr
1
0
Fork 0
Code Issues 1 Pull requests 18 Projects Releases Packages Wiki Activity Actions

docs: rewrite README with project hook, feature table, and screenshot placeholders

Browse source 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Justin Visser 2026-02-27 10:45:19 +01:00
parent 0c5b199b76
commit c9b9d892f4
2 changed files with 108 additions and 63 deletions
Download patch file Download diff file Expand all files Collapse all files

169
README.md
View file

@ -1,94 +1,139 @@
# scholarr
<div align="center">
Self-hosted scholar tracking with a single app image (API + frontend). It automates publication discovery, parses Google Scholar, gathers external metadata via Crossref & arXiv APIs, and autonomously downloads Open Access PDFs.
<img src="frontend/public/scholar_logo.png" alt="Scholarr" width="120" />
# Scholarr
**Self-hosted academic publication tracker.**
Track Google Scholar profiles, discover new papers automatically,
resolve open-access PDFs, and stay on top of the literature you care about.
[![CI](https://img.shields.io/github/actions/workflow/status/justinzeus/scholarr/ci.yml?style=for-the-badge)](https://github.com/JustinZeus/scholarr/actions/workflows/ci.yml)
[![Docker Pulls](https://img.shields.io/docker/pulls/justinzeus/scholarr?style=for-the-badge&logo=docker)](https://hub.docker.com/r/justinzeus/scholarr)
[![Docker Image](https://img.shields.io/badge/docker-justinzeus%2Fscholarr-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://hub.docker.com/r/justinzeus/scholarr)
[![Docs](https://img.shields.io/badge/docs-scholarr-2e8555?style=for-the-badge)](https://justinzeus.github.io/scholarr/)
</div>
## Key Features
- **Automated Ingestion**: Background tasks regularly scrape author profiles to pull the latest publications.
- **Resilient Parsing**: Custom DOM parsing logic falls back heavily across changes, backed by a dynamic `429 Too Many Requests` backoff cycle.
- **Identifier Subsystem**: Isolates DOIs, PMIDs, and arXiv IDs independently of PDFs mapping fallback external APIs sequentially.
- **Unpaywall PDF Search**: Leverages a fully automated resolution queue to grab public PDFs via the Unpaywall API on identified entries.
- **Beautiful UI**: Polished Vue 3 frontend that ships natively inside the same Docker container as the FastAPI backend.
---
## Architecture Data Flow
```mermaid
graph LR
HTML(Google Scholar) --> |Scraper| Ingestion
Ingestion --> |Extract DOI| DB[(Database)]
Ingestion -.-> |Fallback API Search| Crossref/arXiv
Crossref/arXiv --> DB
<!-- TODO: Replace these placeholders with actual screenshots.
Drop PNG/JPEG files into docs/assets/ and update the paths below.
DB --> |Identified DOIs| PDFQueue(Unpaywall/PDF Workers)
PDFQueue --> |Download & Store File| Storage
```
Recommended screenshots:
1. Dashboard / publications list (light or dark mode)
2. Scholar profile view
3. Run detail with progress
Ideal dimensions: 1200-1400px wide, PNG or WebP.
-->
<p align="center">
<img src="docs/assets/screenshot-dashboard.png" alt="Publications dashboard" width="720" />
</p>
<!--
<p align="center">
<img src="docs/assets/screenshot-scholars.png" alt="Scholar profiles" width="360" />
<img src="docs/assets/screenshot-run.png" alt="Ingestion run" width="360" />
</p>
-->
## Why Scholarr?
Most researchers track new papers by manually checking Google Scholar, setting up email alerts, or juggling RSS feeds. Scholarr replaces all of that with a single self-hosted service:
- **Add scholars once** -- by profile URL, Scholar ID, or name search
- **Publications appear automatically** -- a background scheduler scrapes profiles on a configurable interval
- **Open-access PDFs are resolved for you** -- Unpaywall and arXiv are queried automatically when a DOI is found
- **Everything is deduplicated** -- publications are global records; no duplicates across scholars
- **Your data stays yours** -- fully self-hosted, export/import your entire library at any time
## Features
| | |
|---|---|
| **Automated Ingestion** | Background scheduler with configurable intervals, continuation queue, and multi-page pagination |
| **Identifier Resolution** | Cross-references arXiv, Crossref, and OpenAlex to gather DOIs, arXiv IDs, PMIDs |
| **PDF Discovery** | Resolves open-access PDFs via Unpaywall API and arXiv, with automatic retry queue |
| **Scrape Safety** | Rate limiting, cooldowns, and backoff strategies that prevent IP bans -- these are safety floors, not optional |
| **Multi-User** | Session-based auth, admin user management, user-scoped scholar tracking |
| **Theming** | 7 color presets with light/dark mode, tokenized component system |
| **Import / Export** | Portable scholar data with full publication and read-state preservation |
| **Single Container** | FastAPI backend + Vue 3 frontend ship as one Docker image |
## Quick Start
1. Copy env template:
```bash
# 1. Clone and configure
git clone https://github.com/JustinZeus/scholarr.git
cd scholarr
cp .env.example .env
# 2. Set required secrets in .env
# POSTGRES_PASSWORD=<secure-password>
# SESSION_SECRET_KEY=<random-32-char-string>
# 3. Start
docker compose up -d
# 4. Open http://localhost:8000
```
2. Set required values in `.env`:
- `POSTGRES_PASSWORD`
- `SESSION_SECRET_KEY`
3. Start stack:
To bootstrap an admin account on first run, add to `.env`:
```bash
docker compose pull
docker compose up -d
BOOTSTRAP_ADMIN_ON_START=1
BOOTSTRAP_ADMIN_EMAIL=admin@example.com
BOOTSTRAP_ADMIN_PASSWORD=<secure-password>
```
Open:
- App/API: `http://localhost:8000`
- Health: `http://localhost:8000/healthz`
## How It Works
```mermaid
graph LR
Scholar[Google Scholar] -->|Scrape HTML| Parser[Parser & Fingerprinting]
Parser -->|Deduplicate| DB[(PostgreSQL)]
Parser -.->|Identify| APIs[arXiv / Crossref / OpenAlex]
APIs --> DB
DB -->|DOIs| PDF[PDF Resolution]
PDF -->|Unpaywall / arXiv| DB
DB --> UI[Vue 3 Dashboard]
```
## Tech Stack
| Layer | Technology |
|-------|------------|
| Backend | Python 3.12, FastAPI, SQLAlchemy 2.0 (async), Alembic |
| Frontend | TypeScript, Vue 3, Vite, Tailwind CSS |
| Database | PostgreSQL 15 |
| Infrastructure | Multi-stage Docker, Docker Compose |
## Documentation
Complete documentation is published at:
- https://justinzeus.github.io/scholarr/
Full documentation: **[justinzeus.github.io/scholarr](https://justinzeus.github.io/scholarr/)**
You can also read the developer/architecture breakdowns directly within the repository under:
- `docs/developer/architecture.md`
- `docs/developer/ingestion.md`
- `docs/developer/api-contract.md`
| Section | Covers |
|---------|--------|
| [User Guide](docs/user/overview.md) | Installation, configuration, all environment variables |
| [Developer Guide](docs/developer/overview.md) | Architecture, local dev, contributing, testing |
| [Operations](docs/operations/overview.md) | Deployment, database runbook, scrape safety |
| [API Reference](docs/reference/api.md) | Envelope spec, all endpoints, DTO contracts |
## Quality Gates
## Contributing
Backend:
Scholarr uses [conventional commits](https://www.conventionalcommits.org/) and [semantic versioning](https://semver.org/). See the [contributing guide](docs/developer/contributing.md) for PR process and code standards.
```bash
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest tests/unit
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app uv run pytest -m integration
# Dev environment
docker compose -f docker-compose.yml -f docker-compose.dev.yml up --build
# Run tests (always in containers)
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm app \
python -m pytest
```
Frontend:
## License
```bash
# ALL tests/builds MUST be run within the docker container. Do not run raw npm commands on the host.
# For automated agents checking outputs, redirect stdout/stderr to files (e.g. > /tmp/npm_typecheck.log 2>&1)
# The temp files /tmp/npm_test.log, /tmp/npm_lint.log, /tmp/npm_typecheck.log, and /tmp/pytest_eval.log are pre-authorized for overwrite.
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm install
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run typecheck
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run test:run
docker compose -f docker-compose.yml -f docker-compose.dev.yml run --rm frontend npm run build
```
Contract and env checks:
```bash
python3 scripts/check_frontend_api_contract.py
python3 scripts/check_env_contract.py
./scripts/check_no_generated_artifacts.sh
```
thanks arxiv
See [LICENSE](LICENSE) for details.

0
docs/assets/.gitkeep Normal file
View file