docs: write the README and map the build workflow

README.md

@@ -1,35 +1,168 @@
 # discovery-by-llm
 
-This is a demo that serves as a proof of concept for LLM utilization for music discovery, specifically using the Spotify API. The form-factor is an LLM-chat like experience, with direct Spotify integration.
+Music discovery in a chat form: describe the moment, get verified Spotify
+tracks with a reason per track, save the result as a playlist.
 
-> Work in progress. This file is filled in during the build.
+Spotify removed its recommendation and audio-intelligence endpoints for new
-> Chronological build log (Dutch): [docs/logboek.md](docs/logboek.md).
+apps, so in this PoC the LLM fills that role. It interprets the request and
+proposes candidates; Spotify verifies them and supplies the taste profile,
+album art and playback. The eval suite compares the output against bare
+Spotify search on the same questions.
 
-## Demo
+> Chronological build log, including what was dropped and why (Dutch):
+> [docs/logboek.md](docs/logboek.md).
 
-*(to follow: hosted instance + local `docker compose up --build`, with and
+## Running it
-without API keys)*
+
+**Hosted instance** (credentials in the submission mail): a live deployment
+bound to my own Spotify account, with real personalisation and playlist
+writes. Spotify dev mode caps an app at 5 allowlisted users, so a reviewer
+cannot log in with their own account; I can allowlist a reviewer account on
+request.
+
+**Without any keys:**
+
+```
+docker compose up --build
+```
+
+Demo mode replays recorded API cassettes through the same pipeline and
+streaming path as live mode. The suggestion chips map to the recorded
+scenarios; any other input replays the nearest scenario, with a banner
+naming it. Playlist writes are simulated and labeled.
+
+**Live with your own keys:** copy `.env.example` to `.env`, set
+`APP_MODE=live`, a Spotify client id (redirect URI
+`http://127.0.0.1:8888/callback`) and an Anthropic API key, then
+`docker compose up --build` and log in via Spotify.
 
 ## How it works
 
-*(to follow: pipeline diagram and module map)*
+```
+query ──► call 1: intent + candidates ──► grounding ──► call 2: rerank ──► cards
+            (LLM, structured output)      (Spotify        (LLM, streamed
+                    ▲                       search          per track)
+            taste profile                   fan-out)            │
+            (cached, /me/top + saved)          │           playlist write
+                                          verified pool
+```
+
+- **Call 1** interprets the request (mood, activity, era, language,
+  familiarity) and proposes 35 candidate tracks, informed by a compressed
+  taste profile fetched once per session.
+- **Grounding** resolves every candidate against Spotify search in a
+  bounded concurrent fan-out with early stop, a request deadline and a
+  name-to-id cache. Unverified candidates are dropped. Misses and
+  mismatches are logged as separate rates.
+- **Call 2** reranks the verified pool and streams one justification per
+  track as NDJSON events; cards render as they arrive.
+- Only track ids from the verified pool can reach the user. In live
+  testing at familiarity `new`, the model fabricated 12 of 32 candidates;
+  the matcher dropped all 12 and the user still received 15 verified
+  tracks.
+- An honest limit on discovery depth: "new to you" means "not in your top
+  or saved tracks", and the intent prompt favors well-known tracks from
+  adjacent scenes because those verify reliably (chasing obscure work is
+  where the model starts inventing titles). A listener deep in those
+  scenes will recognize a fair share of the results. Going deeper needs a
+  repair loop and a better novelty signal; see the last section.
+- Refinement turns rerank the existing pool; turn 2 makes zero Spotify
+  calls.
 
 ## Choices
 
-*(to follow)*
+- **Two LLM calls around a deterministic grounding step**, no agent loop:
+  predictable latency and every stage testable on its own.
+- **Hand-rolled Spotify client** (~200 lines over httpx). The endpoint
+  surface is small and the retry, rate-limit and caching behavior is where
+  the engineering lives. Retry policy is per endpoint: bounded Retry-After
+  honor on reads, one token refresh, and no retries on playlist writes
+  (side effects would be ambiguous).
+- **Ports and adapters.** Demo mode is a second implementation of the same
+  ports. The domain model never sees Spotify JSON.
+- **NDJSON over a streamed POST.** The request has a body and EventSource
+  cannot POST. Typed events (metadata / track / warning / error / done)
+  with a strict client-side phase machine.
+- **Degradation is explicit.** Below the grounding floor the app returns
+  what resolved plus a warning; quota exhaustion (QUOTA_EXCEEDED) is
+  recognized and never retried; live mode never falls back to fixtures.
+- **Hosted seed session.** The public instance installs a session from an
+  escrowed refresh token at startup, so every visitor shares my account,
+  including playlist writes. HTTP basic auth at the edge is what makes
+  that acceptable. Playlists carry a fixed name prefix for bulk cleanup.
+- **No embeddings.** Search is the only entry point into the catalog; an
+  index does not pay for itself inside a PoC.
 
-## Performance and optimisations
+## Performance
 
-*(to follow)*
+Time to first card is 20 to 30 seconds, measured through the UI and in the
+eval suite; the spread is LLM latency. Streaming keeps the wait honest:
+metadata arrives early, cards render one by one, progress states name what
+is happening.
+
+Under the hood:
+
+- Search `limit` is capped at 10 since Feb 2026, so resolving 35 candidates
+  requires a fan-out: bounded concurrency, early stop once the rerank pool
+  is full, request deadline, partial results on quota exhaustion.
+- Name-to-id resolution cache and a session-scoped taste-profile cache.
+- Anthropic prompt caching between calls.
+- Per-request counters (Spotify calls, cache hits, LLM tokens) in
+  structured logs, surfaced in the dev panel.
+
+Bare search matches words; it has no notion of mood, era, energy or who is
+asking. The pipeline answers from the listener's actual scenes. Across all
+8 eval scenarios the pipeline delivered 11 to 15 verified tracks from 8 to
+20 distinct artists; on the discovery scenario, all 15 were outside the
+listener's top and saved tracks on Spotify. A sample (full table via
+`eval/run_eval.py --baseline`; the eval checks are plain code assertions,
+no LLM judging):
+
+| "Energy for the gym" | Pipeline                                 | Bare search                                                    |
+| -------------------- | ---------------------------------------- | -------------------------------------------------------------- |
+| 1                    | Baddadan by Chase & Status, Bou, Flowdan | Redbone (with GloRilla) by Lil Baby                            |
+| 2                    | Tough Talk by Chase & Status, Kwengface  | Rock That Body by Black Eyed Peas                              |
+| 3                    | Solar System by Sub Focus                | Instigator by Inpatient, Ren, Chris Webby                      |
+| 8                    | DJ Turn It Up by Dimension               | Gym Power Beat by Ultimate Fitness Playlist Power Workout Trax |
 
 ## Where to look
 
-*(to follow)*
+| Criterion                  | Where                                                                                                  |
+| -------------------------- | ------------------------------------------------------------------------------------------------------ |
+| Code quality and structure | `backend/app/` (domain / ports / adapters / pipeline / api), `AGENTS.md`                               |
+| AI application             | `backend/app/prompts.py`, `backend/app/adapters/anthropic/llm.py`, `backend/app/pipeline/grounding.py` |
+| Spotify integration        | `backend/app/adapters/spotify/` (PKCE, retry policy, mapping)                                          |
+| Performance                | `backend/app/pipeline/grounding.py` (fan-out, cache, early stop), `backend/app/observability/`         |
+| Streaming contract         | `backend/app/api/recommendations.py`, `frontend/src/lib/recommendationStream.ts`                       |
+| Evaluation                 | `eval/` (runner, scenarios, baseline arm, recorded fixtures)                                           |
+| Pragmatic cuts over time   | `docs/logboek.md`                                                                                      |
 
 ## Method
 
-*(to follow)*
+About one hour of preparation (API research, a design spike) and about
+eight hours of hands-on build time across three days. The build was
+agent-assisted: coding agents implemented scoped tracks from briefs I
+wrote, while I owned the architecture, the API contracts, every code
+review, every merge and all live verification against the real APIs. Work
+landed in reviewed batches, which is why commits arrive in bursts. The
+logboek documents the process as it happened; the setup behind it (server,
+Forgejo/GitHub mirror, deployment, how the agents are used) is mapped in
+[docs/workflow.md](docs/workflow.md).
 
-## What I cut / what I would do next
+## What I cut, and what would come next
 
-*(to follow)*
+- A faster model on the intent call, which would cut most of the time to
+  first card; the grounding stage already measures the trade-off to watch
+  (fabrication rate), so the experiment is cheap to run safely.
+- Deeper discovery. A repair loop that re-prompts the model for candidates
+  that failed verification would let the prompt chase less obvious work
+  without losing the pool, and a novelty signal beyond exact-id exclusion
+  (artist-level familiarity, recently played) would push results past
+  "famous in an adjacent scene".
+- Refinement turns that re-ground; today turn 2 reranks the existing pool.
+- Chat history persistence. Conversations are client-side only and the
+  server keeps no conversation state; sessions and caches are in-process,
+  Redis is the named production step.
+- Playback queue control, an i18n framework (copy is already centralized),
+  per-stage byte snapshots in the eval, and a semantic index over grounded
+  candidates.

docs/logboek.md

@@ -2,7 +2,9 @@
 
 Bijgehouden tijdens de bouw. Per stap: wat ik deed, waarom, wat ik heb laten
 vallen.
+
 ## Dag 1 - korte sessie in de avond
+
 ### Opzet
 
 Wat ik deed:
@@ -35,6 +37,7 @@ Wat ik heb laten vallen of uitgesteld:
 - Geen apart beslisdocument. De motivering staat in de README en hier.
 
 ## Dag 2
+
 ### Spotify-koppeling
 
 Wat ik deed:
@@ -227,7 +230,7 @@ Waarom:
   main landt.
 - Styling is scoped CSS per component bovenop globale design tokens
   (custom properties). Scoped omdat machine-geconverteerde componenten
-  om style leakage te voorkomen, en  omdat het bouwpakket al gewone CSS 
+  om style leakage te voorkomen, en omdat het bouwpakket al gewone CSS
   per component leverde: een utility
   framework had een rewrite plus een extra dependency gekost. De
   tokens houden het thema op 1 plek, dus het bekende scoped-CSS risico
@@ -262,7 +265,7 @@ Wat ik deed:
   gefragmenteerde chunks, error-terminaliteit, de request-caps en
   abort-races vastlegt; draait mee in CI.
 - Alle user-facing tekst uit de componenten geextraheerd naar 1 getypeerde
-  messages-module (i18n-klaar zonder er nu een framework voor mee te nemen), 
+  messages-module (i18n-klaar zonder er nu een framework voor mee te nemen),
   alle magic numbers vervangen door
   benoemde constants met per waarde een reden (de request-caps verwijzen
   expliciet naar de backend schema-bounds), en dichtgeschreven TypeScript
@@ -275,6 +278,7 @@ Waarom:
   contractkritische randen vastzet, voordat het richting main gaat.
 
 ## Dag 3 - avond
+
 ### Live eval run
 
 Wat ik deed:
@@ -397,6 +401,8 @@ Wat ik heb laten vallen of uitgesteld:
 - Byte-snapshots per pipeline-stap; de checks hierboven en de cassettes
   zijn nu het bewijs.
 
+## Dag 3 - afronding
+
 ### Demo mode
 
 Wat ik deed:
@@ -438,3 +444,31 @@ Waarom:
 
 - Minder scopes vragen dan je gebruikt is netter richting de reviewer en
   richting Spotify.
+
+### Documentatie
+
+Wat ik deed:
+
+- De README geschreven: wat het is, drie manieren om het te draaien, hoe
+  de pipeline werkt, de keuzes, eerlijke performance-cijfers (eerste kaart
+  20 tot 30 s, gemeten via de UI en de eval), een stukje van de
+  vergelijking met kale search, en een "where to look" tabel per
+  beoordelingscriterium.
+- docs/workflow.md toegevoegd: hoe ik werk. De server, Forgejo met een
+  GitHub-mirror, Komodo en Caddy voor de gehoste instantie, en hoe ik
+  agents inzet: implementatie parallel en goedkoop, ontwerp, review en
+  verificatie serieel en bij mij.
+- De gehoste preview bouwt nu bij elke merge opnieuw vanaf main.
+
+Waarom:
+
+- De reviewer leest de README het eerst; die moet kort zijn en naar de
+  rest wijzen in plaats van alles zelf te vertellen. En de werkwijze
+  uitleggen is eerlijker dan hem laten raden waarom commits in bursts
+  binnenkomen.
+
+Wat ik heb laten vallen of uitgesteld:
+
+- Een sneller model op de intent call (zou de wachttijd flink verlagen;
+  de fabrication rate is de afweging om dan te meten) staat als vervolg
+  in de README, niet gebouwd.

docs/workflow.md

@@ -0,0 +1,62 @@
+# How I work
+
+A short map of the setup behind this repo, because the process says as much
+as the code.
+
+## The environment
+
+Everything was built over SSH from a laptop (I was house-sitting for most
+of it) against my home server, a Debian box that runs my self-hosted
+infrastructure. The server carries the whole toolchain: uv and node for
+fast local feedback, Docker for the deliverable, and the deployment stack
+below. No cloud dev environment involved.
+
+## Source control and CI
+
+- Origin is my self-hosted Forgejo instance, with a push mirror to GitHub;
+  the repo you are reading is the mirror and shows the same history and
+  the same green CI.
+- CI runs on every push: ruff, mypy and pytest for the backend; typecheck,
+  lint, build and vitest for the frontend. It has been green since the
+  first commit.
+
+## Deployment
+
+- The hosted instance is a single container on the same server, built from
+  this repo's Dockerfile and managed with Komodo (a self-hosted container
+  control plane). It is rebuilt from main as the build advances.
+- Routing and TLS come from caddy-docker-proxy: the container carries its
+  route as labels, Caddy picks them up. The streaming endpoint needed one
+  deliberate setting there (no response buffering), verified through the
+  public URL.
+- HTTP basic auth sits at the edge because the instance runs against my
+  personal Spotify account.
+
+## AI-assisted building
+
+I work with coding agents, and this project was built that way end to end:
+
+- One orchestrating session (Claude Code harness) holds the plan and the
+  state. It briefs implementation agents (codex CLI, Opus 5 subagents) that
+  build scoped tracks in their own git worktrees, in parallel.
+- Contracts come first: the API schemas and the event protocol were frozen
+  before frontend and backend lanes ran in parallel against them.
+- Parallel lanes buy time where it matters. While I was working on the
+  backend, a separate lane generated a ready-to-implement UI (design
+  tokens, component structure, copy) against the frozen event protocol, so
+  frontend implementation started from a settled design instead of a blank
+  page.
+- Everything that lands is reviewed by me, commit by commit, in my editor
+  before it goes in. Voice-carrying text (this file, the README, the
+  logboek) I write or rewrite myself.
+- Verification against the real APIs is deliberate and budgeted: the
+  Spotify development quota is a daily account-level budget, so live test
+  runs are planned, sequential and measured rather than sprayed.
+- Review also runs as a tool: adversarial review passes over the diffs
+  produced findings that became fix batches; the useful findings and the
+  rejected ones are both traceable in the logboek.
+
+The result is that implementation is cheap and parallel, while design,
+review and verification stay serial and human. The logboek
+([logboek.md](logboek.md)) records that rhythm as it happened, including
+what was dropped under time pressure.