diff --git a/README.md b/README.md index 37706ce..9eb0c91 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,168 @@ # discovery-by-llm -This is a demo that serves as a proof of concept for LLM utilization for music discovery, specifically using the Spotify API. The form-factor is an LLM-chat like experience, with direct Spotify integration. +Music discovery in a chat form: describe the moment, get verified Spotify +tracks with a reason per track, save the result as a playlist. -> Work in progress. This file is filled in during the build. -> Chronological build log (Dutch): [docs/logboek.md](docs/logboek.md). +Spotify removed its recommendation and audio-intelligence endpoints for new +apps, so in this PoC the LLM fills that role. It interprets the request and +proposes candidates; Spotify verifies them and supplies the taste profile, +album art and playback. The eval suite compares the output against bare +Spotify search on the same questions. -## Demo +> Chronological build log, including what was dropped and why (Dutch): +> [docs/logboek.md](docs/logboek.md). -*(to follow: hosted instance + local `docker compose up --build`, with and -without API keys)* +## Running it + +**Hosted instance** (credentials in the submission mail): a live deployment +bound to my own Spotify account, with real personalisation and playlist +writes. Spotify dev mode caps an app at 5 allowlisted users, so a reviewer +cannot log in with their own account; I can allowlist a reviewer account on +request. + +**Without any keys:** + +``` +docker compose up --build +``` + +Demo mode replays recorded API cassettes through the same pipeline and +streaming path as live mode. The suggestion chips map to the recorded +scenarios; any other input replays the nearest scenario, with a banner +naming it. Playlist writes are simulated and labeled. + +**Live with your own keys:** copy `.env.example` to `.env`, set +`APP_MODE=live`, a Spotify client id (redirect URI +`http://127.0.0.1:8888/callback`) and an Anthropic API key, then +`docker compose up --build` and log in via Spotify. ## How it works -*(to follow: pipeline diagram and module map)* +``` +query ──► call 1: intent + candidates ──► grounding ──► call 2: rerank ──► cards + (LLM, structured output) (Spotify (LLM, streamed + ▲ search per track) + taste profile fan-out) │ + (cached, /me/top + saved) │ playlist write + verified pool +``` + +- **Call 1** interprets the request (mood, activity, era, language, + familiarity) and proposes 35 candidate tracks, informed by a compressed + taste profile fetched once per session. +- **Grounding** resolves every candidate against Spotify search in a + bounded concurrent fan-out with early stop, a request deadline and a + name-to-id cache. Unverified candidates are dropped. Misses and + mismatches are logged as separate rates. +- **Call 2** reranks the verified pool and streams one justification per + track as NDJSON events; cards render as they arrive. +- Only track ids from the verified pool can reach the user. In live + testing at familiarity `new`, the model fabricated 12 of 32 candidates; + the matcher dropped all 12 and the user still received 15 verified + tracks. +- An honest limit on discovery depth: "new to you" means "not in your top + or saved tracks", and the intent prompt favors well-known tracks from + adjacent scenes because those verify reliably (chasing obscure work is + where the model starts inventing titles). A listener deep in those + scenes will recognize a fair share of the results. Going deeper needs a + repair loop and a better novelty signal; see the last section. +- Refinement turns rerank the existing pool; turn 2 makes zero Spotify + calls. ## Choices -*(to follow)* +- **Two LLM calls around a deterministic grounding step**, no agent loop: + predictable latency and every stage testable on its own. +- **Hand-rolled Spotify client** (~200 lines over httpx). The endpoint + surface is small and the retry, rate-limit and caching behavior is where + the engineering lives. Retry policy is per endpoint: bounded Retry-After + honor on reads, one token refresh, and no retries on playlist writes + (side effects would be ambiguous). +- **Ports and adapters.** Demo mode is a second implementation of the same + ports. The domain model never sees Spotify JSON. +- **NDJSON over a streamed POST.** The request has a body and EventSource + cannot POST. Typed events (metadata / track / warning / error / done) + with a strict client-side phase machine. +- **Degradation is explicit.** Below the grounding floor the app returns + what resolved plus a warning; quota exhaustion (QUOTA_EXCEEDED) is + recognized and never retried; live mode never falls back to fixtures. +- **Hosted seed session.** The public instance installs a session from an + escrowed refresh token at startup, so every visitor shares my account, + including playlist writes. HTTP basic auth at the edge is what makes + that acceptable. Playlists carry a fixed name prefix for bulk cleanup. +- **No embeddings.** Search is the only entry point into the catalog; an + index does not pay for itself inside a PoC. -## Performance and optimisations +## Performance -*(to follow)* +Time to first card is 20 to 30 seconds, measured through the UI and in the +eval suite; the spread is LLM latency. Streaming keeps the wait honest: +metadata arrives early, cards render one by one, progress states name what +is happening. + +Under the hood: + +- Search `limit` is capped at 10 since Feb 2026, so resolving 35 candidates + requires a fan-out: bounded concurrency, early stop once the rerank pool + is full, request deadline, partial results on quota exhaustion. +- Name-to-id resolution cache and a session-scoped taste-profile cache. +- Anthropic prompt caching between calls. +- Per-request counters (Spotify calls, cache hits, LLM tokens) in + structured logs, surfaced in the dev panel. + +Bare search matches words; it has no notion of mood, era, energy or who is +asking. The pipeline answers from the listener's actual scenes. Across all +8 eval scenarios the pipeline delivered 11 to 15 verified tracks from 8 to +20 distinct artists; on the discovery scenario, all 15 were outside the +listener's top and saved tracks on Spotify. A sample (full table via +`eval/run_eval.py --baseline`; the eval checks are plain code assertions, +no LLM judging): + +| "Energy for the gym" | Pipeline | Bare search | +| -------------------- | ---------------------------------------- | -------------------------------------------------------------- | +| 1 | Baddadan by Chase & Status, Bou, Flowdan | Redbone (with GloRilla) by Lil Baby | +| 2 | Tough Talk by Chase & Status, Kwengface | Rock That Body by Black Eyed Peas | +| 3 | Solar System by Sub Focus | Instigator by Inpatient, Ren, Chris Webby | +| 8 | DJ Turn It Up by Dimension | Gym Power Beat by Ultimate Fitness Playlist Power Workout Trax | ## Where to look -*(to follow)* +| Criterion | Where | +| -------------------------- | ------------------------------------------------------------------------------------------------------ | +| Code quality and structure | `backend/app/` (domain / ports / adapters / pipeline / api), `AGENTS.md` | +| AI application | `backend/app/prompts.py`, `backend/app/adapters/anthropic/llm.py`, `backend/app/pipeline/grounding.py` | +| Spotify integration | `backend/app/adapters/spotify/` (PKCE, retry policy, mapping) | +| Performance | `backend/app/pipeline/grounding.py` (fan-out, cache, early stop), `backend/app/observability/` | +| Streaming contract | `backend/app/api/recommendations.py`, `frontend/src/lib/recommendationStream.ts` | +| Evaluation | `eval/` (runner, scenarios, baseline arm, recorded fixtures) | +| Pragmatic cuts over time | `docs/logboek.md` | ## Method -*(to follow)* +About one hour of preparation (API research, a design spike) and about +eight hours of hands-on build time across three days. The build was +agent-assisted: coding agents implemented scoped tracks from briefs I +wrote, while I owned the architecture, the API contracts, every code +review, every merge and all live verification against the real APIs. Work +landed in reviewed batches, which is why commits arrive in bursts. The +logboek documents the process as it happened; the setup behind it (server, +Forgejo/GitHub mirror, deployment, how the agents are used) is mapped in +[docs/workflow.md](docs/workflow.md). -## What I cut / what I would do next +## What I cut, and what would come next -*(to follow)* +- A faster model on the intent call, which would cut most of the time to + first card; the grounding stage already measures the trade-off to watch + (fabrication rate), so the experiment is cheap to run safely. +- Deeper discovery. A repair loop that re-prompts the model for candidates + that failed verification would let the prompt chase less obvious work + without losing the pool, and a novelty signal beyond exact-id exclusion + (artist-level familiarity, recently played) would push results past + "famous in an adjacent scene". +- Refinement turns that re-ground; today turn 2 reranks the existing pool. +- Chat history persistence. Conversations are client-side only and the + server keeps no conversation state; sessions and caches are in-process, + Redis is the named production step. +- Playback queue control, an i18n framework (copy is already centralized), + per-stage byte snapshots in the eval, and a semantic index over grounded + candidates. diff --git a/docs/logboek.md b/docs/logboek.md index 1ee58cb..851e9b1 100644 --- a/docs/logboek.md +++ b/docs/logboek.md @@ -2,7 +2,9 @@ Bijgehouden tijdens de bouw. Per stap: wat ik deed, waarom, wat ik heb laten vallen. + ## Dag 1 - korte sessie in de avond + ### Opzet Wat ik deed: @@ -35,6 +37,7 @@ Wat ik heb laten vallen of uitgesteld: - Geen apart beslisdocument. De motivering staat in de README en hier. ## Dag 2 + ### Spotify-koppeling Wat ik deed: @@ -227,7 +230,7 @@ Waarom: main landt. - Styling is scoped CSS per component bovenop globale design tokens (custom properties). Scoped omdat machine-geconverteerde componenten - om style leakage te voorkomen, en omdat het bouwpakket al gewone CSS + om style leakage te voorkomen, en omdat het bouwpakket al gewone CSS per component leverde: een utility framework had een rewrite plus een extra dependency gekost. De tokens houden het thema op 1 plek, dus het bekende scoped-CSS risico @@ -262,7 +265,7 @@ Wat ik deed: gefragmenteerde chunks, error-terminaliteit, de request-caps en abort-races vastlegt; draait mee in CI. - Alle user-facing tekst uit de componenten geextraheerd naar 1 getypeerde - messages-module (i18n-klaar zonder er nu een framework voor mee te nemen), + messages-module (i18n-klaar zonder er nu een framework voor mee te nemen), alle magic numbers vervangen door benoemde constants met per waarde een reden (de request-caps verwijzen expliciet naar de backend schema-bounds), en dichtgeschreven TypeScript @@ -275,6 +278,7 @@ Waarom: contractkritische randen vastzet, voordat het richting main gaat. ## Dag 3 - avond + ### Live eval run Wat ik deed: @@ -397,6 +401,8 @@ Wat ik heb laten vallen of uitgesteld: - Byte-snapshots per pipeline-stap; de checks hierboven en de cassettes zijn nu het bewijs. +## Dag 3 - afronding + ### Demo mode Wat ik deed: @@ -438,3 +444,31 @@ Waarom: - Minder scopes vragen dan je gebruikt is netter richting de reviewer en richting Spotify. + +### Documentatie + +Wat ik deed: + +- De README geschreven: wat het is, drie manieren om het te draaien, hoe + de pipeline werkt, de keuzes, eerlijke performance-cijfers (eerste kaart + 20 tot 30 s, gemeten via de UI en de eval), een stukje van de + vergelijking met kale search, en een "where to look" tabel per + beoordelingscriterium. +- docs/workflow.md toegevoegd: hoe ik werk. De server, Forgejo met een + GitHub-mirror, Komodo en Caddy voor de gehoste instantie, en hoe ik + agents inzet: implementatie parallel en goedkoop, ontwerp, review en + verificatie serieel en bij mij. +- De gehoste preview bouwt nu bij elke merge opnieuw vanaf main. + +Waarom: + +- De reviewer leest de README het eerst; die moet kort zijn en naar de + rest wijzen in plaats van alles zelf te vertellen. En de werkwijze + uitleggen is eerlijker dan hem laten raden waarom commits in bursts + binnenkomen. + +Wat ik heb laten vallen of uitgesteld: + +- Een sneller model op de intent call (zou de wachttijd flink verlagen; + de fabrication rate is de afweging om dan te meten) staat als vervolg + in de README, niet gebouwd. diff --git a/docs/workflow.md b/docs/workflow.md new file mode 100644 index 0000000..2f5d25b --- /dev/null +++ b/docs/workflow.md @@ -0,0 +1,62 @@ +# How I work + +A short map of the setup behind this repo, because the process says as much +as the code. + +## The environment + +Everything was built over SSH from a laptop (I was house-sitting for most +of it) against my home server, a Debian box that runs my self-hosted +infrastructure. The server carries the whole toolchain: uv and node for +fast local feedback, Docker for the deliverable, and the deployment stack +below. No cloud dev environment involved. + +## Source control and CI + +- Origin is my self-hosted Forgejo instance, with a push mirror to GitHub; + the repo you are reading is the mirror and shows the same history and + the same green CI. +- CI runs on every push: ruff, mypy and pytest for the backend; typecheck, + lint, build and vitest for the frontend. It has been green since the + first commit. + +## Deployment + +- The hosted instance is a single container on the same server, built from + this repo's Dockerfile and managed with Komodo (a self-hosted container + control plane). It is rebuilt from main as the build advances. +- Routing and TLS come from caddy-docker-proxy: the container carries its + route as labels, Caddy picks them up. The streaming endpoint needed one + deliberate setting there (no response buffering), verified through the + public URL. +- HTTP basic auth sits at the edge because the instance runs against my + personal Spotify account. + +## AI-assisted building + +I work with coding agents, and this project was built that way end to end: + +- One orchestrating session (Claude Code harness) holds the plan and the + state. It briefs implementation agents (codex CLI, Opus 5 subagents) that + build scoped tracks in their own git worktrees, in parallel. +- Contracts come first: the API schemas and the event protocol were frozen + before frontend and backend lanes ran in parallel against them. +- Parallel lanes buy time where it matters. While I was working on the + backend, a separate lane generated a ready-to-implement UI (design + tokens, component structure, copy) against the frozen event protocol, so + frontend implementation started from a settled design instead of a blank + page. +- Everything that lands is reviewed by me, commit by commit, in my editor + before it goes in. Voice-carrying text (this file, the README, the + logboek) I write or rewrite myself. +- Verification against the real APIs is deliberate and budgeted: the + Spotify development quota is a daily account-level budget, so live test + runs are planned, sequential and measured rather than sprayed. +- Review also runs as a tool: adversarial review passes over the diffs + produced findings that became fix batches; the useful findings and the + rejected ones are both traceable in the logboek. + +The result is that implementation is cheap and parallel, while design, +review and verification stay serial and human. The logboek +([logboek.md](logboek.md)) records that rhythm as it happened, including +what was dropped under time pressure.