justinzeus/scholarr
1
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

add the rewrite charter and its decision records

Browse source
This commit is contained in:
justinzeus 2026-08-20 11:34:31 +02:00
parent a8c5939804
commit 41e8714ca3
4 changed files with 519 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

162
runbooks/rewrite/charter.md Normal file
View file

@ -0,0 +1,162 @@
# Scholarr rewrite charter **[live 2026-08-20]**
Tracker: https://git.justintime.lol/justinzeus/scholarr/issues
Durable doctrine for the Scholarr rewrite: what was decided and why, and the
constraints every piece of work inherits. It holds no status, no next action and
no progress notes; those live on the issues. Migrated from the retired
`server-docs/runbooks/scholarr-rewrite/` on 2026-08-20, which is now git history.
## Goal
A self-hosted author-watchlist and publication tracker. A user follows a set of
authors; the service tracks what those authors publish and surfaces
legally-available open-access PDFs.
**Dual purpose, and both halves are first-class:**
- **Daily driver for Justin's dad** (he already follows hundreds of authors; the
v1 product was built for exactly this use).
- **The one repo Justin wants to be genuinely useful to the general public**, a
real public FOSS project, not a personal-tool-that-happens-to-be-on-GitHub.
Design and polish are held to that bar.
## IMMUTABLE DOCTRINE: no Google Scholar network contact, ever
This is the entire reason the rewrite exists. It is not a preference or a
default; it is a hard invariant of the new system.
- **Zero Google Scholar network contact from the service, ever.** No scraping, no
automated fetches from `scholar.google.com`, no headless-browser calls, no
CAPTCHA-solving, no anti-bot or evasion machinery of any kind. There is no code
path in the rewrite that reaches a Google endpoint.
- **Scholar IDs and Scholar URLs live on only as inert imported metadata and
human-clickable links.** A stored Scholar profile URL is a link a *person* can
click in a browser; the service itself never dereferences it.
- **Official APIs only.** Acquisition and enrichment go through documented,
sanctioned APIs:
- **OpenAlex**, primary works + authors source.
- **Crossref**, **arXiv**, **Unpaywall**, metadata, preprints, legal
open-access PDF resolution.
- **ORCID**, identity anchor for disambiguating authors.
- **Semantic Scholar**, resolved **not** a v1 requirement (2026-07-22
calibration verdict). Possible later addition only.
- **Every provider is a good citizen.** Per-provider **persisted** rate limits,
cooldowns, and **user-visible source health** are part of the design from the
first spec, not bolted on later. Etiquette (contact `mailto`, modest request
rates, backoff) is mandatory.
**Rationale (2026-07-21):** the v1 product was Scholar-first. Google login-gated
and CAPTCHA-challenged the scrapes, the box got blocked and IP-banned, name
search had to be **permanently disabled** in the UI, and development died
**2026-03-07** in the middle of Scholar-breakage firefighting. Justin's stated
requirement for the rewrite is: **"no risk of Google getting mad at my box, or
the service just stopping suddenly."** The doctrine above is the direct,
non-negotiable answer to that requirement.
## Locked decisions *(2026-07-21, decided by Justin)*
| Area | Decision |
|---|---|
| **Language / stack** | Backend **Go**, frontend **Vue**. |
| **Database** | **SQLite only** (no Postgres). The concurrency design must fit SQLite from the start, it is not a placeholder to be swapped later. |
| **Root entity** | **`FollowedAuthor`** (source-agnostic), with **`AuthorSourceIdentity`** rows attached (OpenAlex / ORCID / Semantic Scholar / Scholar-imported). Scholar is **demoted** to an import reference plus a manual fallback identity, never the root. |
| **Auth (v1)** | Three modes, all in v1: (1) **internal username/password**, always available, the zero-dependency default; (2) **generic OIDC**, where Authentik is *one* provider among many and the integration is never Authentik-specific; (3) **trusted-header / forward-auth** mode. |
| **Multi-user** | **Confirmed requirement.** A **global, deduplicated publication store** shared across users, with **per-user follow links and per-user read state**. |
| **Hosting** | **GitHub is primary** and the public face. Personal identity only: **`justinzeus` / `justinfvisser@gmail.com`**, never the maxdoro work identity. **Forgejo keeps a copy** at `https://git.justintime.lol`. |
| **Repos** | The old Forgejo repo `justinzeus/scholarr` was renamed **`scholarr-legacy`** (kept as reference: specs, regression fixtures, git history). The rewrite is a **fresh `scholarr` repo with clean history**. |
| **Design values** | **Easy onboarding**: first run must be trivial, import Scholar URLs or search for authors, confirm, done. **Config-driven design throughout**: a conductarr-style declarative config, a single config file driving behavior. |
## Still open, deliberately
Recorded as open, not resolved here. Each is an owner decision, not an
implementation detail.
- **Release and distribution mechanics.** `ghcr` versus GitHub Releases; a single
self-contained binary with an embedded frontend versus container-first.
- **License choice for a public FOSS product.** The legacy repo ships **MIT**
(`Copyright (c) 2026 JustinZeus`). Whether the rewrite keeps MIT or picks
another OSI license is a conscious decision, not a carry-over by inertia.
- **Semantic Scholar** stays a possible later addition. Resolved as out of scope
for v1, not resolved as never.
## Working conventions
Binding on any agent editing this repo.
- **Commit messages: plain imperative, no trailers of any kind.** No
`Co-Authored-By`, no `Generated-with`, no sign-off, no attribution lines.
- **Stage only your own files by name.** Never `git add -A` or `git add .`.
- **No em dashes in any written output.** Owner rule. Use commas, parentheses, or
a spaced hyphen. Applies to docs, commits and code comments.
- **Immutable doctrine applies to every artifact.** No code path, spec, test or
tool may reach a Google endpoint.
- **UI QA is manual, done by Justin. No browser automation.** Do not add or run
Playwright, Selenium or headless-browser UI tests. Smoke via build, curl, unit
and integration tests only; UI review is Justin's, against the frozen design
contract.
- **Secrets never in git or logs.** API keys (for example `OPENALEX_API_KEY`), the
safety dump, and any real email address stay out of the repo, out of logs, and
out of these docs. Reference secrets by variable name and escrow location,
never by value.
## Assets
Everything is on the tank server unless stated. None of the originals are
modified.
- **Legacy checkout:** `/opt/stacks/scholarr`, a clone of Forgejo
`justinzeus/scholarr-legacy` at commit `f501ea4`, the last commit before
development stopped (Python FastAPI + Vue, ~120 commits 2026-02-16 to
2026-03-07). Its `.backup/`, `.experiment/` and `design/` directories are
locally git-excluded via `.git/info/exclude`.
- **Rewrite checkout:** `/opt/stacks/scholarr-rewrite`. Primary
`https://github.com/JustinZeus/scholarr` (`origin`), Forgejo copy
`https://git.justintime.lol/justinzeus/scholarr` (`forgejo`). Bootstrap
completion commit `ba7592e`.
- **Safety dump (the seed dataset):**
`/opt/stacks/scholarr/.backup/scholarr-pgdump-20260721.dump`, ~6.3 MB
`pg_dump` custom format, verified restorable. **Contains real email
addresses.** Locally git-excluded and must never be committed or published
anywhere.
- **Frozen UI spec (FROZEN 2026-07-22):** `/opt/stacks/scholarr-rewrite/design/`,
holding `DESIGN.md` (the contract), `tokens.css`, `component-reference.html`,
`assets/` and `reference/`. `reference/support.js` is proof-only and never
ships. Review records stay in the git-excluded legacy `design/` directory.
- **Calibration workspace:** `/opt/stacks/scholarr/.experiment/`, git-excluded,
cache-first and resumable.
**The legacy dataset is an identity seed and a partial floor, not ground truth.**
492 followed-author rows collapsing to 253 distinct people, 25,698 publications,
~47k author-to-publication links, last successful crawl 2026-03-12. It was
harvested by a scraper that was already being blocked, enrichment was half done,
and **97 profiles never completed a baseline crawl**. Calibrate against it; do
not treat it as correct or complete.
## The spec freeze gate
The data-model spec at `docs/specs/data-model.md` is **reviewed and revised but
not frozen**. Its Card C.1 independent architecture review (2026-07-23, four
adversarial dimensions over the `e36e486` draft) returned **NO-FREEZE with seven
blockers**, all spec-text gaps rather than design flaws, and the revision
applying every finding is committed at `a8c5939`.
Freezing means the owner has answered **D1 through D7** (D2 split into D2a-D2d),
approved any resulting edits, and marked the spec frozen. **No implementation and
no Card C.2 work begins before that freeze.** That rule is not enforced by
convention any more: the implementation issue carries real Forgejo dependencies
on all seven decision issues, so Forgejo itself refuses to close it while any
decision is open.
The **UI spec is already frozen** and needs no freeze card.
## Related
- [identity-calibration.md](identity-calibration.md), the calibration method and
its completed results.
- [identity-model.md](identity-model.md), the identity and duplicate-prevention
decision record. Its deterministic merge-target rule governs every author
merge, so the survivor is not assumed to be the already-followed record.
- [testing-strategy.md](testing-strategy.md), the testing strategy and release
gates decision record.
- Work tracking doctrine: `server-docs/runbooks/tracking/charter.md`.

187
runbooks/rewrite/identity-calibration.md Normal file
View file

@ -0,0 +1,187 @@
# Scholarr rewrite - identity calibration **[live 2026-07-22]**
Part of the [scholarr-rewrite](README.md) runbook. Read the doctrine there first.
## Purpose
**This is calibration, not a gate.** An earlier framing treated a coverage experiment as the
decision point for whether to go API-first at all. That question is settled: **API-first is
doctrine** (no Google Scholar network contact, ever - see the runbook README). This document
does not re-litigate it.
What calibration *does* measure:
- **Identity-mapping quality** - how reliably a Scholar-imported author (name, plus whatever
identifiers the old dataset holds) maps to the correct **OpenAlex** author, so the
`FollowedAuthor` → `AuthorSourceIdentity` mapping in the rewrite is grounded in real numbers
rather than optimism.
- **Honest coverage expectations** - per-author publication coverage relative to the old
dataset, so onboarding and the "did we find everything?" UX can set expectations truthfully
instead of implying completeness the APIs cannot deliver.
The output feeds two things: the [Semantic Scholar open question](README.md#open-questions)
(include it only if a real, material gap shows here), and the honesty of the onboarding /
source-health UX in the spec set.
## Constraints
- **No Google contact, ever.** Calibration reads only from the recovered old DB (for the known
authors and their known publication titles) and from **official APIs**. Nothing in this
method touches `scholar.google.com`.
- **Polite OpenAlex etiquette.** Use the `mailto` contact parameter on every request and keep a
modest request rate with backoff. Calibration is a good citizen just like the service will be.
## Method
Input: the **492 authors** from the safety dump
(`/opt/stacks/scholarr/.backup/scholarr-pgdump-20260721.dump`; see the runbook's Assets
section for provenance and the real-email handling rule). For each author:
1. **Candidate search.** Query **OpenAlex `/authors`** by the author's name to get a candidate
list.
2. **Works-overlap voting.** Pull each candidate's works and compare against the **author's
known publication titles from the old DB**. The candidate whose works overlap the known
titles most strongly wins the vote. Overlap is title-based (identifier-based where the old DB
happens to have an identifier), which is exactly the disambiguation signal a same-name
collision needs.
3. **Classify the mapping** into one of:
- **auto-match** - a single clear winner, high overlap; safe to map without human review.
- **needs-review** - ambiguous (multiple plausible candidates, or a weak winner); surface to
a human.
- **unmatched** - no acceptable candidate found.
4. **Measure per-author title coverage** - of the author's known titles, what fraction appear
in the matched OpenAlex author's works. This is the coverage-expectation number.
**Freshness note (record, do not resolve):** sources differ in lag. **OpenAlex** typically lags
**days to weeks** behind publication; **arXiv** is **near-instant** for preprints. The sync
engine and the "recently published" UX must account for this per-source, and the coverage
numbers here should be read with the OpenAlex lag in mind (a very recent title missing from
OpenAlex is a freshness artifact, not a coverage failure).
**Caveat carried from the dataset facts:** the old titles are themselves a partial floor (the
scraper was already being blocked; 97 profiles never completed a baseline). "Coverage vs the old
DB" therefore measures agreement with an incomplete reference, not agreement with ground truth.
Report it as such.
## Results
**Calibration COMPLETE 2026-07-22.** Full run over all 492 dump rows finished
(completion fraction 1.0, 1287 OpenAlex requests). The 2026-07-21 attempt stalled at 75/492
under an OpenAlex IP-level rate block; the completion run used the recovered polite-pool API
key (separate rate-limit pool) and saw **zero HTTP 429** at a <= 2 requests/second pace. Raw
artifacts live under `/opt/stacks/scholarr/.experiment/` (git-excluded): `summary.json`,
`authors_matched.csv`, `authors_results.json`, and the experiment `RESULTS.md`.
### Headline (492 followed-author rows)
97 rows are **no-data** shells (no display_name and 0 publications in the dump, so nothing to
name-search). The other **395 matchable** rows are the denominator for the match rates.
| Class | Count | % of matchable | % of all 492 |
|---|---|---|---|
| auto-match | 352 | 89.1% | 71.5% |
| needs-review | 20 | 5.1% | 4.1% |
| unmatched | 23 | 5.8% | 4.7% |
| no-data shell | 97 | n/a | 19.7% |
The 89.1% auto-match rate held: the 75-author partial run showed 87.1%, and the full run came
in slightly higher at 89.1%, so the early number was representative, not lucky.
### Per user (percentages of that user's matchable rows)
| user | total | matchable | auto | needs-review | unmatched | no-data |
|---|---|---|---|---|---|---|
| Justin (1) | 242 | 240 | 215 (89.6%) | 12 (5.0%) | 13 (5.4%) | 2 |
| dad (2) | 250 | 155 | 137 (88.4%) | 8 (5.2%) | 10 (6.5%) | 95 |
Auto-match quality is essentially identical for both users; the only large gap is no-data
shells, which are almost entirely dad's un-crawled follows (95 of the 97).
### Over the 253 distinct authors (after cross-user dedup)
The 492 rows collapse to **253 distinct authors** (see dedup finding below). Counting each
person once, taking the better-resolved copy when the two users disagree:
| Class | Count | % of distinct-matchable (246) | % of 253 |
|---|---|---|---|
| auto-match | 222 | 90.2% | 87.7% |
| needs-review | 11 | 4.5% | 4.3% |
| unmatched | 13 | 5.3% | 5.1% |
| no-data shell | 7 | n/a | 2.8% |
Deduplicating raises the effective auto-match rate to **90.2%** and, more importantly, cuts
the no-data shells from 97 rows to just **7 true shells** (people un-crawled for *every*
following user). The other 90 no-data rows are the same person that another user already has
fully resolved.
### Title coverage of auto-matches
Fraction of ALL an author's dump titles found in the matched OpenAlex author's works (up to
600 works fetched per candidate).
- Overall title coverage: min 0.398, **Q1 0.730, median 0.800, Q3 0.872**, max 1.0, mean 0.789.
- Recent (year >= 2024) coverage, over the 345 auto-matches that have 2024+ dump pubs:
**Q1 0.769, median 0.882, Q3 1.0**, mean 0.856.
Recent coverage sits above overall coverage, which is the expected shape: the older long tail
of a dump has more title-formatting drift and more pre-OpenAlex-era gaps than recent work.
Read all coverage against the two standing caveats: the OpenAlex publication lag (days to
weeks) and the fact that the dump titles are an incomplete floor, not ground truth.
### Failure-mode patterns
- **No-data shells (97 rows / 7 distinct):** the single largest bucket, and almost entirely a
dedup artifact. See the dedup finding.
- **Common-name collisions:** 20 needs-review-or-unmatched rows returned >= 8 candidates
(crowded name space). These are the genuine hard cases: e.g. Bin Wang, Liang Meng, Ying
Yang, ZHANG Kai (10 candidates, no clear works-overlap winner), and near-miss reviews like
Sander van der Linden and Stephan Lewandowsky (top ~0.33 with a close runner-up). Short,
common, or heavily-shared names are where works-overlap voting earns its place.
- **Non-latin / alt-script names:** only 3 rows carry CJK characters in the display_name;
2 of those (Haotian Zhang 张昊天) did not auto-match, while Xiaomin Sun (孙晓敏) did.
Volume is tiny, but the parenthesized-alt-script pattern is a real edge case for name
cleaning.
- **Sparse / titled-name authors:** a handful of true 0-candidate misses are sparse profiles
or names carrying academic titles the cleaner does not fully strip (e.g. "Dr. phil. Marius
Jais" leaves a "phil." fragment; EJ Horberg and Joseph Yap Haw are genuinely thin). These
are correct "unmatched" outcomes, not throttle artifacts: the 4 zero-candidate cases flagged
as throttle-corrupted in the partial run were re-searched cleanly on the keyed pool, and one
of them (Marina Milyavskaya, 107 dump pubs) recovered to a confident auto-match at 0.867.
- **Prolific authors (>= 200 dump pubs) with < 60% coverage:** 5. This is the OpenAlex
600-works fetch cap and title-formatting drift, not a wrong match (scores are still high).
- **Review queue is small and mostly easy:** 20 needs-review total, 17 with a top score
>= 0.3 (likely-correct, quick human confirm).
### Cross-user dedup finding
The 492 profile rows cover only **253 distinct scholar_ids**, and **239** of those are
followed by BOTH Justin and dad. Concretely: 82 authors are auto-matched for one user but a
bare no-data shell for the other, so a shared resolved identity would immediately cure 90 of
the 97 no-data shells. This is direct evidence for the [identity model](identity-model.md):
key `FollowedAuthor` on a resolved identifier and share one canonical resolution across users,
with per-user follow rows on top. It roughly halves the true match-plus-review workload.
### Provider-etiquette lessons (feed the provider-contracts spec)
The run doubled as a live test of OpenAlex etiquette, and every finding argues for
**per-provider persisted rate clocks as a core feature, not a nicety**:
- **A ~2 requests/second sustained ceiling is the safe operating point.** The 2026-07-21
bursts that ran hotter triggered the block; the 2026-07-22 completion run held <= 2 req/s
with exponential backoff and saw zero 429s across 1287 requests.
- **Keyed and anonymous pools are separate.** The recovered polite-pool API key sailed
through while anonymous requests from the same IP were still blocked. The rewrite should
treat "has a provider key" as its own rate-limit lane and persist the key per provider.
- **IP-level blocks outlast the daily reset.** The 2026-07-21 block from bursty requests
survived the daily quota reset and kept returning 429 to anonymous probes into the next day.
A rate limiter that only reasons about a rolling daily quota is not enough; the clock and any
cooldown must be **persisted across process restarts** so a restart cannot re-burst straight
into a fresh block.
### Semantic Scholar call
**No material gap that justifies adding Semantic Scholar for coverage.** OpenAlex alone
auto-matches ~90% of distinct authors with a median title coverage of 0.80, and the residual
misses are common-name disambiguation and genuinely sparse profiles that a second aggregator
would not obviously fix. Semantic Scholar stays a possible later addition, not a v1
requirement. See the [open question](README.md#open-questions).

83
runbooks/rewrite/identity-model.md Normal file
View file

@ -0,0 +1,83 @@
# Scholarr rewrite - identity model **[live 2026-07-23]**
This is an agreed direction from the 2026-07-21 discussion with Justin. It is now formalized in the
review-ready, unfrozen draft at `/opt/stacks/scholarr-rewrite/docs/specs/data-model.md`. Part of the
[scholarr-rewrite](README.md) runbook; read the doctrine there first.
This document records how the rewrite decides *who an author is* and how it keeps the same person
from turning into two records. It is a decision record, not the final spec: it feeds the identity /
identifier-first-dedup spec when that spec is frozen.
## Root principle
Identity is anchored to **source identifiers**; names are **labels only**. Nothing is ever keyed
on a name. Two records are the same author because they share a resolved identifier, never because
they share a string.
## Same name, different people
Supported **by construction**: distinct external IDs are distinct `FollowedAuthor` records, so two
genuinely different people who happen to share a name simply resolve to two records and never
collide.
UX around this:
- **Disambiguation evidence at add time.** When a user is picking a candidate, show the deciding
evidence side by side: affiliation, fields, active years, sample works, and the identifiers
themselves. The user confirms a specific person, not a name.
- **One soft, non-blocking notice** when confirming a candidate whose name matches an
already-followed author with a **different** ID: "same person or different?" It does not block
the confirm; it just makes the collision visible.
- **The answer persists as a "confirmed different" tombstone**, so that exact pair is never
re-flagged again.
## Duplicate prevention: three mechanisms for three causes
Duplicates have three distinct causes, and each gets its own mechanism.
1. **Exact re-follows** (the same identifier followed again). Killed by a **uniqueness constraint on
`(source, external_id)`** plus the shared-identity model: there is one canonical author record
and per-user follow rows, so a second user following the same person **attaches to the existing
resolution** rather than creating a new author.
2. **The same human reached via different identifiers.** `AuthorSourceIdentity` is a **list on one
author**: a Scholar-imported shell, an OpenAlex ID, and an ORCID can all attach to a single
`FollowedAuthor` record. When a shell is later identified as an already-followed ID, the
resolution becomes a **MERGE**, **never a new
author**. The survivor is not assumed to be the already-followed record: the spec's
deterministic target rule governs every author merge (resolved beats shell, more strong
identities, older `created_at`, lowest `public_id`), regardless of which record was the review
subject. Merges are **audited and undoable**, because a wrong merge (collapsing two real people
into one) is the worst mistake the system can make.
3. **OpenAlex split profiles** (one real person holding more than one OpenAlex ID). One author may
hold multiple OpenAlex IDs via an **"also this profile"** action. The **works-overlap scoring**
used for identity calibration doubles as the duplicate detector here: it raises a **"possible
duplicate" review card** that resolves to either **merge** or **keep-separate** (keep-separate
writes the **confirmed-different** tombstone described above).
## Review-queue hygiene
The review queue is keyed by the **author entity**, never by the raising event. This keeps the
queue honest instead of flooding it.
- A shell that fails to resolve across ten syncs is **one card**, not ten.
- **Pairwise duplicate cards spawn once per pair**, unless genuinely new evidence appears (for
example, a newly shared identifier changes the picture).
- **Four card types:**
- **identify shell** - a Scholar-imported shell that has no resolved ID yet.
- **confirm ambiguous match** - the soft "same person or different?" from add time.
- **possible duplicate** - the works-overlap detector flagged two records as maybe the same
person.
- **not-this-person fallout** - cleanup when a prior resolution turns out to be wrong.
## Honest limit
Shells with **no name and no works** cannot be proven distinct. They stay **human-decided**, and
the UI **says so plainly** rather than guessing. The system does not pretend to resolve identity it
has no evidence for.
## Related
- The works-overlap scoring reused here as a duplicate detector: [identity-calibration.md](identity-calibration.md).
- Root entity and `AuthorSourceIdentity` decision: [README.md](README.md) (Locked decisions).

87
runbooks/rewrite/testing-strategy.md Normal file
View file

@ -0,0 +1,87 @@
# Scholarr rewrite - testing strategy **[live 2026-07-21]**
This is an agreed direction from the 2026-07-21 discussion with Justin, to be formalized during
spec freeze. Part of the [scholarr-rewrite](README.md) runbook; read the doctrine there first.
This document records how the rewrite proves itself correct. It is a decision record, not the final
spec: it feeds the spec set and defines the release gates below.
## Core principle
**Deterministic tests against recorded reality.** The service's hard logic (matching, dedup,
scheduling, backoff) must be testable with **zero network**. Nothing that matters to correctness
depends on a live provider being reachable at test time.
## Golden corpus
Two recorded artifacts become **permanent fixtures in the repo** (anonymized where needed):
- The **recovered v1 database**: 25,698 real publications, 492 profiles with real edge cases.
- The **calibration run's cached OpenAlex responses**.
These are the ground for deterministic replay: the matcher and dedup logic are exercised against
real data and real provider payloads without touching the network.
## The five pillars
1. **Pure logic** (normalization, fingerprinting, identifier parsing, confidence scoring):
- table tests;
- **property-based tests** for the invariants that must always hold: normalization is
idempotent, dedup is order-independent, merge is commutative;
- **Go native fuzzing** on every parser that touches external text or paste input.
2. **Provider clients:**
- **contract tests** replaying recorded HTTP fixtures via `httptest`, including the ugly cases:
`429` with and without `Retry-After`, truncated JSON, empty pages;
- the **rate limiter and cooldown state machine** tested with a **fake clock**;
- the **2026-07-21 OpenAlex IP block becomes a regression case** so the system's handling of an
IP-level block is proven, not hoped.
3. **Sync engine** - executable invariants:
- one active run per user;
- idempotent re-runs;
- additive-only baseline import;
- crash-mid-run leaves resumable state;
- **race detector in CI**, plus a **concurrent-write hammer test**, because single-writer
discipline is the main risk of the SQLite-only design.
4. **Migrations:**
- every migration tested **up and down** against a **real-database copy**;
- plus a **seeded prior-version DB in CI**, so upgrades are proven **pre-release**.
5. **End-to-end smoke:**
- the binary boots in CI with a **seeded SQLite** and a **stub provider**;
- a **scripted curl** run walks the full API flow: onboard, import, sync, review-resolve, read,
export.
- **No browser automation.** UI QA is manual by Justin against the frozen design contract
(`/opt/stacks/scholarr-rewrite/design/DESIGN.md`). See the working-conventions block
in [README.md](README.md#conventions-for-any-agent-working-this-project).
## Dry runs are a product feature
Not test-only scaffolding, actual product behavior that also serves as the QA instrument:
- **bulk import previews** before creating anything;
- **`--dry-run`** on sync and on destructive admin actions.
## Acceptance
**Shadow-run** with the real ~253-author list for **2 to 3 weeks**, alongside the dad's existing
JS-script habit, and **diff what each caught**. This is the **release flip criterion** and follows
the conductarr-soak pattern.
## Release gates, in order
1. unit + property + fuzz green;
2. race detector green;
3. migration up/down green;
4. e2e smoke green;
5. **golden-corpus ratchet** - matcher changes may not regress the recorded auto-match baseline;
6. tagged release with a **tested backup/restore path documented**.
## Related
- The recovered v1 DB and calibration cache that become fixtures: [README.md](README.md) (Assets),
[identity-calibration.md](identity-calibration.md).
- Manual-QA rule (no browser automation): a working convention of this project, stated in
[README.md](README.md#conventions-for-any-agent-working-this-project).