removed old UI

This commit is contained in:
Justin Visser 2026-02-17 15:21:08 +01:00
parent 4433d7d2c4
commit f71841e922
62 changed files with 411 additions and 5815 deletions

View file

@ -1,13 +0,0 @@
# Scholar Scrape Probe (Temporary)
Purpose: short-lived workspace for information gathering and planning.
Rules:
- No production code changes here.
- Use this for HTML samples, parsing experiments, and extraction notes.
- Delete this directory after we lock the scrape contract and parser plan.
Structure:
- `fixtures/` saved HTML samples for parser validation
- `notes/` extraction decisions, edge cases, and risk log
- `scripts/` one-off probe scripts used only during planning

View file

@ -1,71 +0,0 @@
# Probe Findings -> Phase 1 Input
Generated from live probe run: `run_20260216T182334Z`.
## What is stable enough to build on
- Public profile endpoint works for anonymous access:
- `GET /citations?hl=en&user=<scholar_id>`
- Core row structure is present and parseable:
- row: `tr.gsc_a_tr`
- title + detail URL: `a.gsc_a_at`
- citation count: `a.gsc_a_ac`
- year: `span.gsc_a_h` (fallback: `td.gsc_a_y` text regex)
- metadata lines: first/second `div.gs_gray` => authors/venue
- `cluster_id` can be extracted reliably from `citation_for_view=<user>:<cluster_id>` in title URLs.
## What can break / where to degrade gracefully
- Invalid or inaccessible profile IDs may redirect to Google sign-in page.
- Treat as `blocked_or_captcha` / `inaccessible` state, not parser crash.
- `Show more` is present for tested profiles.
- We currently parse first page only.
- Because robots currently disallows `citations?*cstart=`, deep pagination should not be assumed.
- Some rows are missing year or venue text.
- Keep nullable fields and avoid failing dedupe for these gaps.
## Observed quality from probe
- Parsed publication rows: 57 across 4 accessible profiles.
- Field coverage:
- title: 100%
- cluster_id: 100%
- citation_count: 100%
- authors_text: 100%
- year: 98.2%
- venue_text: 94.7%
## Recommended parser contract for implementation
- Input -> `PublicationCandidate`
- `title` (required)
- `cluster_id` (required when present in URL; expected high coverage)
- `year` (nullable)
- `citation_count` (nullable/int default fallback 0)
- `authors_text` (nullable)
- `venue_text` (nullable)
- `title_url` (nullable)
- Page-level parse status enum:
- `ok`, `no_results`, `blocked_or_captcha`, `layout_changed`, `network_error`
- Page-level flags:
- `has_show_more_button`
- `articles_range` string (e.g. `Articles 120`)
## Immediate test plan unlocked by this probe
- Add fixture-driven unit tests using captured HTML from `fixtures/run_20260216T182334Z`.
- Add assertions for:
- row count > 0 on known-good fixtures
- profile name extraction
- cluster_id extraction from title URLs
- nullable handling for year/venue
- blocked/inaccessible page classification
- show-more partial warning classification
## Phase 1 implementation guardrails
- Keep requests low-rate with jitter and explicit timeout.
- Persist parser status per scholar run.
- If `has_show_more_button=True`, mark run as partial and show this in UI.
- Never fail entire run because one scholar page is blocked or malformed.