From 41e8714ca3fb7721d8cff6caaaa8740e6170e6b9 Mon Sep 17 00:00:00 2001 From: justinzeus Date: Thu, 20 Aug 2026 11:34:31 +0200 Subject: [PATCH] add the rewrite charter and its decision records --- runbooks/rewrite/charter.md | 162 ++++++++++++++++++++ runbooks/rewrite/identity-calibration.md | 187 +++++++++++++++++++++++ runbooks/rewrite/identity-model.md | 83 ++++++++++ runbooks/rewrite/testing-strategy.md | 87 +++++++++++ 4 files changed, 519 insertions(+) create mode 100644 runbooks/rewrite/charter.md create mode 100644 runbooks/rewrite/identity-calibration.md create mode 100644 runbooks/rewrite/identity-model.md create mode 100644 runbooks/rewrite/testing-strategy.md diff --git a/runbooks/rewrite/charter.md b/runbooks/rewrite/charter.md new file mode 100644 index 0000000..0ba29f5 --- /dev/null +++ b/runbooks/rewrite/charter.md @@ -0,0 +1,162 @@ +# Scholarr rewrite charter **[live 2026-08-20]** + +Tracker: https://git.justintime.lol/justinzeus/scholarr/issues + +Durable doctrine for the Scholarr rewrite: what was decided and why, and the +constraints every piece of work inherits. It holds no status, no next action and +no progress notes; those live on the issues. Migrated from the retired +`server-docs/runbooks/scholarr-rewrite/` on 2026-08-20, which is now git history. + +## Goal + +A self-hosted author-watchlist and publication tracker. A user follows a set of +authors; the service tracks what those authors publish and surfaces +legally-available open-access PDFs. + +**Dual purpose, and both halves are first-class:** + +- **Daily driver for Justin's dad** (he already follows hundreds of authors; the + v1 product was built for exactly this use). +- **The one repo Justin wants to be genuinely useful to the general public**, a + real public FOSS project, not a personal-tool-that-happens-to-be-on-GitHub. + Design and polish are held to that bar. + +## IMMUTABLE DOCTRINE: no Google Scholar network contact, ever + +This is the entire reason the rewrite exists. It is not a preference or a +default; it is a hard invariant of the new system. + +- **Zero Google Scholar network contact from the service, ever.** No scraping, no + automated fetches from `scholar.google.com`, no headless-browser calls, no + CAPTCHA-solving, no anti-bot or evasion machinery of any kind. There is no code + path in the rewrite that reaches a Google endpoint. +- **Scholar IDs and Scholar URLs live on only as inert imported metadata and + human-clickable links.** A stored Scholar profile URL is a link a *person* can + click in a browser; the service itself never dereferences it. +- **Official APIs only.** Acquisition and enrichment go through documented, + sanctioned APIs: + - **OpenAlex**, primary works + authors source. + - **Crossref**, **arXiv**, **Unpaywall**, metadata, preprints, legal + open-access PDF resolution. + - **ORCID**, identity anchor for disambiguating authors. + - **Semantic Scholar**, resolved **not** a v1 requirement (2026-07-22 + calibration verdict). Possible later addition only. +- **Every provider is a good citizen.** Per-provider **persisted** rate limits, + cooldowns, and **user-visible source health** are part of the design from the + first spec, not bolted on later. Etiquette (contact `mailto`, modest request + rates, backoff) is mandatory. + +**Rationale (2026-07-21):** the v1 product was Scholar-first. Google login-gated +and CAPTCHA-challenged the scrapes, the box got blocked and IP-banned, name +search had to be **permanently disabled** in the UI, and development died +**2026-03-07** in the middle of Scholar-breakage firefighting. Justin's stated +requirement for the rewrite is: **"no risk of Google getting mad at my box, or +the service just stopping suddenly."** The doctrine above is the direct, +non-negotiable answer to that requirement. + +## Locked decisions *(2026-07-21, decided by Justin)* + +| Area | Decision | +|---|---| +| **Language / stack** | Backend **Go**, frontend **Vue**. | +| **Database** | **SQLite only** (no Postgres). The concurrency design must fit SQLite from the start, it is not a placeholder to be swapped later. | +| **Root entity** | **`FollowedAuthor`** (source-agnostic), with **`AuthorSourceIdentity`** rows attached (OpenAlex / ORCID / Semantic Scholar / Scholar-imported). Scholar is **demoted** to an import reference plus a manual fallback identity, never the root. | +| **Auth (v1)** | Three modes, all in v1: (1) **internal username/password**, always available, the zero-dependency default; (2) **generic OIDC**, where Authentik is *one* provider among many and the integration is never Authentik-specific; (3) **trusted-header / forward-auth** mode. | +| **Multi-user** | **Confirmed requirement.** A **global, deduplicated publication store** shared across users, with **per-user follow links and per-user read state**. | +| **Hosting** | **GitHub is primary** and the public face. Personal identity only: **`justinzeus` / `justinfvisser@gmail.com`**, never the maxdoro work identity. **Forgejo keeps a copy** at `https://git.justintime.lol`. | +| **Repos** | The old Forgejo repo `justinzeus/scholarr` was renamed **`scholarr-legacy`** (kept as reference: specs, regression fixtures, git history). The rewrite is a **fresh `scholarr` repo with clean history**. | +| **Design values** | **Easy onboarding**: first run must be trivial, import Scholar URLs or search for authors, confirm, done. **Config-driven design throughout**: a conductarr-style declarative config, a single config file driving behavior. | + +## Still open, deliberately + +Recorded as open, not resolved here. Each is an owner decision, not an +implementation detail. + +- **Release and distribution mechanics.** `ghcr` versus GitHub Releases; a single + self-contained binary with an embedded frontend versus container-first. +- **License choice for a public FOSS product.** The legacy repo ships **MIT** + (`Copyright (c) 2026 JustinZeus`). Whether the rewrite keeps MIT or picks + another OSI license is a conscious decision, not a carry-over by inertia. +- **Semantic Scholar** stays a possible later addition. Resolved as out of scope + for v1, not resolved as never. + +## Working conventions + +Binding on any agent editing this repo. + +- **Commit messages: plain imperative, no trailers of any kind.** No + `Co-Authored-By`, no `Generated-with`, no sign-off, no attribution lines. +- **Stage only your own files by name.** Never `git add -A` or `git add .`. +- **No em dashes in any written output.** Owner rule. Use commas, parentheses, or + a spaced hyphen. Applies to docs, commits and code comments. +- **Immutable doctrine applies to every artifact.** No code path, spec, test or + tool may reach a Google endpoint. +- **UI QA is manual, done by Justin. No browser automation.** Do not add or run + Playwright, Selenium or headless-browser UI tests. Smoke via build, curl, unit + and integration tests only; UI review is Justin's, against the frozen design + contract. +- **Secrets never in git or logs.** API keys (for example `OPENALEX_API_KEY`), the + safety dump, and any real email address stay out of the repo, out of logs, and + out of these docs. Reference secrets by variable name and escrow location, + never by value. + +## Assets + +Everything is on the tank server unless stated. None of the originals are +modified. + +- **Legacy checkout:** `/opt/stacks/scholarr`, a clone of Forgejo + `justinzeus/scholarr-legacy` at commit `f501ea4`, the last commit before + development stopped (Python FastAPI + Vue, ~120 commits 2026-02-16 to + 2026-03-07). Its `.backup/`, `.experiment/` and `design/` directories are + locally git-excluded via `.git/info/exclude`. +- **Rewrite checkout:** `/opt/stacks/scholarr-rewrite`. Primary + `https://github.com/JustinZeus/scholarr` (`origin`), Forgejo copy + `https://git.justintime.lol/justinzeus/scholarr` (`forgejo`). Bootstrap + completion commit `ba7592e`. +- **Safety dump (the seed dataset):** + `/opt/stacks/scholarr/.backup/scholarr-pgdump-20260721.dump`, ~6.3 MB + `pg_dump` custom format, verified restorable. **Contains real email + addresses.** Locally git-excluded and must never be committed or published + anywhere. +- **Frozen UI spec (FROZEN 2026-07-22):** `/opt/stacks/scholarr-rewrite/design/`, + holding `DESIGN.md` (the contract), `tokens.css`, `component-reference.html`, + `assets/` and `reference/`. `reference/support.js` is proof-only and never + ships. Review records stay in the git-excluded legacy `design/` directory. +- **Calibration workspace:** `/opt/stacks/scholarr/.experiment/`, git-excluded, + cache-first and resumable. + +**The legacy dataset is an identity seed and a partial floor, not ground truth.** +492 followed-author rows collapsing to 253 distinct people, 25,698 publications, +~47k author-to-publication links, last successful crawl 2026-03-12. It was +harvested by a scraper that was already being blocked, enrichment was half done, +and **97 profiles never completed a baseline crawl**. Calibrate against it; do +not treat it as correct or complete. + +## The spec freeze gate + +The data-model spec at `docs/specs/data-model.md` is **reviewed and revised but +not frozen**. Its Card C.1 independent architecture review (2026-07-23, four +adversarial dimensions over the `e36e486` draft) returned **NO-FREEZE with seven +blockers**, all spec-text gaps rather than design flaws, and the revision +applying every finding is committed at `a8c5939`. + +Freezing means the owner has answered **D1 through D7** (D2 split into D2a-D2d), +approved any resulting edits, and marked the spec frozen. **No implementation and +no Card C.2 work begins before that freeze.** That rule is not enforced by +convention any more: the implementation issue carries real Forgejo dependencies +on all seven decision issues, so Forgejo itself refuses to close it while any +decision is open. + +The **UI spec is already frozen** and needs no freeze card. + +## Related + +- [identity-calibration.md](identity-calibration.md), the calibration method and + its completed results. +- [identity-model.md](identity-model.md), the identity and duplicate-prevention + decision record. Its deterministic merge-target rule governs every author + merge, so the survivor is not assumed to be the already-followed record. +- [testing-strategy.md](testing-strategy.md), the testing strategy and release + gates decision record. +- Work tracking doctrine: `server-docs/runbooks/tracking/charter.md`. diff --git a/runbooks/rewrite/identity-calibration.md b/runbooks/rewrite/identity-calibration.md new file mode 100644 index 0000000..0cad07e --- /dev/null +++ b/runbooks/rewrite/identity-calibration.md @@ -0,0 +1,187 @@ +# Scholarr rewrite - identity calibration **[live 2026-07-22]** + +Part of the [scholarr-rewrite](README.md) runbook. Read the doctrine there first. + +## Purpose + +**This is calibration, not a gate.** An earlier framing treated a coverage experiment as the +decision point for whether to go API-first at all. That question is settled: **API-first is +doctrine** (no Google Scholar network contact, ever - see the runbook README). This document +does not re-litigate it. + +What calibration *does* measure: + +- **Identity-mapping quality** - how reliably a Scholar-imported author (name, plus whatever + identifiers the old dataset holds) maps to the correct **OpenAlex** author, so the + `FollowedAuthor` → `AuthorSourceIdentity` mapping in the rewrite is grounded in real numbers + rather than optimism. +- **Honest coverage expectations** - per-author publication coverage relative to the old + dataset, so onboarding and the "did we find everything?" UX can set expectations truthfully + instead of implying completeness the APIs cannot deliver. + +The output feeds two things: the [Semantic Scholar open question](README.md#open-questions) +(include it only if a real, material gap shows here), and the honesty of the onboarding / +source-health UX in the spec set. + +## Constraints + +- **No Google contact, ever.** Calibration reads only from the recovered old DB (for the known + authors and their known publication titles) and from **official APIs**. Nothing in this + method touches `scholar.google.com`. +- **Polite OpenAlex etiquette.** Use the `mailto` contact parameter on every request and keep a + modest request rate with backoff. Calibration is a good citizen just like the service will be. + +## Method + +Input: the **492 authors** from the safety dump +(`/opt/stacks/scholarr/.backup/scholarr-pgdump-20260721.dump`; see the runbook's Assets +section for provenance and the real-email handling rule). For each author: + +1. **Candidate search.** Query **OpenAlex `/authors`** by the author's name to get a candidate + list. +2. **Works-overlap voting.** Pull each candidate's works and compare against the **author's + known publication titles from the old DB**. The candidate whose works overlap the known + titles most strongly wins the vote. Overlap is title-based (identifier-based where the old DB + happens to have an identifier), which is exactly the disambiguation signal a same-name + collision needs. +3. **Classify the mapping** into one of: + - **auto-match** - a single clear winner, high overlap; safe to map without human review. + - **needs-review** - ambiguous (multiple plausible candidates, or a weak winner); surface to + a human. + - **unmatched** - no acceptable candidate found. +4. **Measure per-author title coverage** - of the author's known titles, what fraction appear + in the matched OpenAlex author's works. This is the coverage-expectation number. + +**Freshness note (record, do not resolve):** sources differ in lag. **OpenAlex** typically lags +**days to weeks** behind publication; **arXiv** is **near-instant** for preprints. The sync +engine and the "recently published" UX must account for this per-source, and the coverage +numbers here should be read with the OpenAlex lag in mind (a very recent title missing from +OpenAlex is a freshness artifact, not a coverage failure). + +**Caveat carried from the dataset facts:** the old titles are themselves a partial floor (the +scraper was already being blocked; 97 profiles never completed a baseline). "Coverage vs the old +DB" therefore measures agreement with an incomplete reference, not agreement with ground truth. +Report it as such. + +## Results + +**Calibration COMPLETE 2026-07-22.** Full run over all 492 dump rows finished +(completion fraction 1.0, 1287 OpenAlex requests). The 2026-07-21 attempt stalled at 75/492 +under an OpenAlex IP-level rate block; the completion run used the recovered polite-pool API +key (separate rate-limit pool) and saw **zero HTTP 429** at a <= 2 requests/second pace. Raw +artifacts live under `/opt/stacks/scholarr/.experiment/` (git-excluded): `summary.json`, +`authors_matched.csv`, `authors_results.json`, and the experiment `RESULTS.md`. + +### Headline (492 followed-author rows) + +97 rows are **no-data** shells (no display_name and 0 publications in the dump, so nothing to +name-search). The other **395 matchable** rows are the denominator for the match rates. + +| Class | Count | % of matchable | % of all 492 | +|---|---|---|---| +| auto-match | 352 | 89.1% | 71.5% | +| needs-review | 20 | 5.1% | 4.1% | +| unmatched | 23 | 5.8% | 4.7% | +| no-data shell | 97 | n/a | 19.7% | + +The 89.1% auto-match rate held: the 75-author partial run showed 87.1%, and the full run came +in slightly higher at 89.1%, so the early number was representative, not lucky. + +### Per user (percentages of that user's matchable rows) + +| user | total | matchable | auto | needs-review | unmatched | no-data | +|---|---|---|---|---|---|---| +| Justin (1) | 242 | 240 | 215 (89.6%) | 12 (5.0%) | 13 (5.4%) | 2 | +| dad (2) | 250 | 155 | 137 (88.4%) | 8 (5.2%) | 10 (6.5%) | 95 | + +Auto-match quality is essentially identical for both users; the only large gap is no-data +shells, which are almost entirely dad's un-crawled follows (95 of the 97). + +### Over the 253 distinct authors (after cross-user dedup) + +The 492 rows collapse to **253 distinct authors** (see dedup finding below). Counting each +person once, taking the better-resolved copy when the two users disagree: + +| Class | Count | % of distinct-matchable (246) | % of 253 | +|---|---|---|---| +| auto-match | 222 | 90.2% | 87.7% | +| needs-review | 11 | 4.5% | 4.3% | +| unmatched | 13 | 5.3% | 5.1% | +| no-data shell | 7 | n/a | 2.8% | + +Deduplicating raises the effective auto-match rate to **90.2%** and, more importantly, cuts +the no-data shells from 97 rows to just **7 true shells** (people un-crawled for *every* +following user). The other 90 no-data rows are the same person that another user already has +fully resolved. + +### Title coverage of auto-matches + +Fraction of ALL an author's dump titles found in the matched OpenAlex author's works (up to +600 works fetched per candidate). + +- Overall title coverage: min 0.398, **Q1 0.730, median 0.800, Q3 0.872**, max 1.0, mean 0.789. +- Recent (year >= 2024) coverage, over the 345 auto-matches that have 2024+ dump pubs: + **Q1 0.769, median 0.882, Q3 1.0**, mean 0.856. + +Recent coverage sits above overall coverage, which is the expected shape: the older long tail +of a dump has more title-formatting drift and more pre-OpenAlex-era gaps than recent work. +Read all coverage against the two standing caveats: the OpenAlex publication lag (days to +weeks) and the fact that the dump titles are an incomplete floor, not ground truth. + +### Failure-mode patterns + +- **No-data shells (97 rows / 7 distinct):** the single largest bucket, and almost entirely a + dedup artifact. See the dedup finding. +- **Common-name collisions:** 20 needs-review-or-unmatched rows returned >= 8 candidates + (crowded name space). These are the genuine hard cases: e.g. Bin Wang, Liang Meng, Ying + Yang, ZHANG Kai (10 candidates, no clear works-overlap winner), and near-miss reviews like + Sander van der Linden and Stephan Lewandowsky (top ~0.33 with a close runner-up). Short, + common, or heavily-shared names are where works-overlap voting earns its place. +- **Non-latin / alt-script names:** only 3 rows carry CJK characters in the display_name; + 2 of those (Haotian Zhang 张昊天) did not auto-match, while Xiaomin Sun (孙晓敏) did. + Volume is tiny, but the parenthesized-alt-script pattern is a real edge case for name + cleaning. +- **Sparse / titled-name authors:** a handful of true 0-candidate misses are sparse profiles + or names carrying academic titles the cleaner does not fully strip (e.g. "Dr. phil. Marius + Jais" leaves a "phil." fragment; EJ Horberg and Joseph Yap Haw are genuinely thin). These + are correct "unmatched" outcomes, not throttle artifacts: the 4 zero-candidate cases flagged + as throttle-corrupted in the partial run were re-searched cleanly on the keyed pool, and one + of them (Marina Milyavskaya, 107 dump pubs) recovered to a confident auto-match at 0.867. +- **Prolific authors (>= 200 dump pubs) with < 60% coverage:** 5. This is the OpenAlex + 600-works fetch cap and title-formatting drift, not a wrong match (scores are still high). +- **Review queue is small and mostly easy:** 20 needs-review total, 17 with a top score + >= 0.3 (likely-correct, quick human confirm). + +### Cross-user dedup finding + +The 492 profile rows cover only **253 distinct scholar_ids**, and **239** of those are +followed by BOTH Justin and dad. Concretely: 82 authors are auto-matched for one user but a +bare no-data shell for the other, so a shared resolved identity would immediately cure 90 of +the 97 no-data shells. This is direct evidence for the [identity model](identity-model.md): +key `FollowedAuthor` on a resolved identifier and share one canonical resolution across users, +with per-user follow rows on top. It roughly halves the true match-plus-review workload. + +### Provider-etiquette lessons (feed the provider-contracts spec) + +The run doubled as a live test of OpenAlex etiquette, and every finding argues for +**per-provider persisted rate clocks as a core feature, not a nicety**: + +- **A ~2 requests/second sustained ceiling is the safe operating point.** The 2026-07-21 + bursts that ran hotter triggered the block; the 2026-07-22 completion run held <= 2 req/s + with exponential backoff and saw zero 429s across 1287 requests. +- **Keyed and anonymous pools are separate.** The recovered polite-pool API key sailed + through while anonymous requests from the same IP were still blocked. The rewrite should + treat "has a provider key" as its own rate-limit lane and persist the key per provider. +- **IP-level blocks outlast the daily reset.** The 2026-07-21 block from bursty requests + survived the daily quota reset and kept returning 429 to anonymous probes into the next day. + A rate limiter that only reasons about a rolling daily quota is not enough; the clock and any + cooldown must be **persisted across process restarts** so a restart cannot re-burst straight + into a fresh block. + +### Semantic Scholar call + +**No material gap that justifies adding Semantic Scholar for coverage.** OpenAlex alone +auto-matches ~90% of distinct authors with a median title coverage of 0.80, and the residual +misses are common-name disambiguation and genuinely sparse profiles that a second aggregator +would not obviously fix. Semantic Scholar stays a possible later addition, not a v1 +requirement. See the [open question](README.md#open-questions). diff --git a/runbooks/rewrite/identity-model.md b/runbooks/rewrite/identity-model.md new file mode 100644 index 0000000..52d25f1 --- /dev/null +++ b/runbooks/rewrite/identity-model.md @@ -0,0 +1,83 @@ +# Scholarr rewrite - identity model **[live 2026-07-23]** + +This is an agreed direction from the 2026-07-21 discussion with Justin. It is now formalized in the +review-ready, unfrozen draft at `/opt/stacks/scholarr-rewrite/docs/specs/data-model.md`. Part of the +[scholarr-rewrite](README.md) runbook; read the doctrine there first. + +This document records how the rewrite decides *who an author is* and how it keeps the same person +from turning into two records. It is a decision record, not the final spec: it feeds the identity / +identifier-first-dedup spec when that spec is frozen. + +## Root principle + +Identity is anchored to **source identifiers**; names are **labels only**. Nothing is ever keyed +on a name. Two records are the same author because they share a resolved identifier, never because +they share a string. + +## Same name, different people + +Supported **by construction**: distinct external IDs are distinct `FollowedAuthor` records, so two +genuinely different people who happen to share a name simply resolve to two records and never +collide. + +UX around this: + +- **Disambiguation evidence at add time.** When a user is picking a candidate, show the deciding + evidence side by side: affiliation, fields, active years, sample works, and the identifiers + themselves. The user confirms a specific person, not a name. +- **One soft, non-blocking notice** when confirming a candidate whose name matches an + already-followed author with a **different** ID: "same person or different?" It does not block + the confirm; it just makes the collision visible. +- **The answer persists as a "confirmed different" tombstone**, so that exact pair is never + re-flagged again. + +## Duplicate prevention: three mechanisms for three causes + +Duplicates have three distinct causes, and each gets its own mechanism. + +1. **Exact re-follows** (the same identifier followed again). Killed by a **uniqueness constraint on + `(source, external_id)`** plus the shared-identity model: there is one canonical author record + and per-user follow rows, so a second user following the same person **attaches to the existing + resolution** rather than creating a new author. + +2. **The same human reached via different identifiers.** `AuthorSourceIdentity` is a **list on one + author**: a Scholar-imported shell, an OpenAlex ID, and an ORCID can all attach to a single + `FollowedAuthor` record. When a shell is later identified as an already-followed ID, the + resolution becomes a **MERGE**, **never a new + author**. The survivor is not assumed to be the already-followed record: the spec's + deterministic target rule governs every author merge (resolved beats shell, more strong + identities, older `created_at`, lowest `public_id`), regardless of which record was the review + subject. Merges are **audited and undoable**, because a wrong merge (collapsing two real people + into one) is the worst mistake the system can make. + +3. **OpenAlex split profiles** (one real person holding more than one OpenAlex ID). One author may + hold multiple OpenAlex IDs via an **"also this profile"** action. The **works-overlap scoring** + used for identity calibration doubles as the duplicate detector here: it raises a **"possible + duplicate" review card** that resolves to either **merge** or **keep-separate** (keep-separate + writes the **confirmed-different** tombstone described above). + +## Review-queue hygiene + +The review queue is keyed by the **author entity**, never by the raising event. This keeps the +queue honest instead of flooding it. + +- A shell that fails to resolve across ten syncs is **one card**, not ten. +- **Pairwise duplicate cards spawn once per pair**, unless genuinely new evidence appears (for + example, a newly shared identifier changes the picture). +- **Four card types:** + - **identify shell** - a Scholar-imported shell that has no resolved ID yet. + - **confirm ambiguous match** - the soft "same person or different?" from add time. + - **possible duplicate** - the works-overlap detector flagged two records as maybe the same + person. + - **not-this-person fallout** - cleanup when a prior resolution turns out to be wrong. + +## Honest limit + +Shells with **no name and no works** cannot be proven distinct. They stay **human-decided**, and +the UI **says so plainly** rather than guessing. The system does not pretend to resolve identity it +has no evidence for. + +## Related + +- The works-overlap scoring reused here as a duplicate detector: [identity-calibration.md](identity-calibration.md). +- Root entity and `AuthorSourceIdentity` decision: [README.md](README.md) (Locked decisions). diff --git a/runbooks/rewrite/testing-strategy.md b/runbooks/rewrite/testing-strategy.md new file mode 100644 index 0000000..4938524 --- /dev/null +++ b/runbooks/rewrite/testing-strategy.md @@ -0,0 +1,87 @@ +# Scholarr rewrite - testing strategy **[live 2026-07-21]** + +This is an agreed direction from the 2026-07-21 discussion with Justin, to be formalized during +spec freeze. Part of the [scholarr-rewrite](README.md) runbook; read the doctrine there first. + +This document records how the rewrite proves itself correct. It is a decision record, not the final +spec: it feeds the spec set and defines the release gates below. + +## Core principle + +**Deterministic tests against recorded reality.** The service's hard logic (matching, dedup, +scheduling, backoff) must be testable with **zero network**. Nothing that matters to correctness +depends on a live provider being reachable at test time. + +## Golden corpus + +Two recorded artifacts become **permanent fixtures in the repo** (anonymized where needed): + +- The **recovered v1 database**: 25,698 real publications, 492 profiles with real edge cases. +- The **calibration run's cached OpenAlex responses**. + +These are the ground for deterministic replay: the matcher and dedup logic are exercised against +real data and real provider payloads without touching the network. + +## The five pillars + +1. **Pure logic** (normalization, fingerprinting, identifier parsing, confidence scoring): + - table tests; + - **property-based tests** for the invariants that must always hold: normalization is + idempotent, dedup is order-independent, merge is commutative; + - **Go native fuzzing** on every parser that touches external text or paste input. + +2. **Provider clients:** + - **contract tests** replaying recorded HTTP fixtures via `httptest`, including the ugly cases: + `429` with and without `Retry-After`, truncated JSON, empty pages; + - the **rate limiter and cooldown state machine** tested with a **fake clock**; + - the **2026-07-21 OpenAlex IP block becomes a regression case** so the system's handling of an + IP-level block is proven, not hoped. + +3. **Sync engine** - executable invariants: + - one active run per user; + - idempotent re-runs; + - additive-only baseline import; + - crash-mid-run leaves resumable state; + - **race detector in CI**, plus a **concurrent-write hammer test**, because single-writer + discipline is the main risk of the SQLite-only design. + +4. **Migrations:** + - every migration tested **up and down** against a **real-database copy**; + - plus a **seeded prior-version DB in CI**, so upgrades are proven **pre-release**. + +5. **End-to-end smoke:** + - the binary boots in CI with a **seeded SQLite** and a **stub provider**; + - a **scripted curl** run walks the full API flow: onboard, import, sync, review-resolve, read, + export. + - **No browser automation.** UI QA is manual by Justin against the frozen design contract + (`/opt/stacks/scholarr-rewrite/design/DESIGN.md`). See the working-conventions block + in [README.md](README.md#conventions-for-any-agent-working-this-project). + +## Dry runs are a product feature + +Not test-only scaffolding, actual product behavior that also serves as the QA instrument: + +- **bulk import previews** before creating anything; +- **`--dry-run`** on sync and on destructive admin actions. + +## Acceptance + +**Shadow-run** with the real ~253-author list for **2 to 3 weeks**, alongside the dad's existing +JS-script habit, and **diff what each caught**. This is the **release flip criterion** and follows +the conductarr-soak pattern. + +## Release gates, in order + +1. unit + property + fuzz green; +2. race detector green; +3. migration up/down green; +4. e2e smoke green; +5. **golden-corpus ratchet** - matcher changes may not regress the recorded auto-match baseline; +6. tagged release with a **tested backup/restore path documented**. + +## Related + +- The recovered v1 DB and calibration cache that become fixtures: [README.md](README.md) (Assets), + [identity-calibration.md](identity-calibration.md). +- Manual-QA rule (no browser automation): a working convention of this project, stated in + [README.md](README.md#conventions-for-any-agent-working-this-project).