removed old UI
This commit is contained in:
parent
4433d7d2c4
commit
f71841e922
62 changed files with 411 additions and 5815 deletions
|
|
@ -1,92 +0,0 @@
|
|||
# MVP Implementation Reminders (From Scholar Probe)
|
||||
|
||||
Last validated probe run: `planning/scholar_probe_tmp/notes/probe_report_run_20260216T182334Z.md`.
|
||||
|
||||
## Decision: Are we ready?
|
||||
|
||||
Yes, for the scoped MVP.
|
||||
|
||||
- We have stable selectors and field coverage for first-page profile parsing.
|
||||
- We have explicit failure modes for blocked/inaccessible pages.
|
||||
- We have enough fixtures to start parser + ingestion tests immediately.
|
||||
|
||||
Not guaranteed (and intentionally out of MVP):
|
||||
|
||||
- Full historical completeness from profile pagination.
|
||||
- Robots currently disallows `citations?*cstart=`; treat deep pagination as out of scope unless policy changes.
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
- Keep app container-first (`docker compose`) and test while developing.
|
||||
- Keep feature scope small; prefer deterministic behavior over clever scraping.
|
||||
- Never crash a whole run because one scholar page fails.
|
||||
- Preserve strict tenant isolation for all run/output/read-state data.
|
||||
|
||||
## Scraping Contract
|
||||
|
||||
Target URL:
|
||||
|
||||
- `https://scholar.google.com/citations?hl=en&user=<scholar_id>`
|
||||
|
||||
Primary parse markers:
|
||||
|
||||
- Row: `tr.gsc_a_tr`
|
||||
- Title/link: `a.gsc_a_at`
|
||||
- Citation count: `a.gsc_a_ac`
|
||||
- Year: `span.gsc_a_h` (fallback year regex)
|
||||
- Metadata lines: first/second `div.gs_gray` => authors/venue
|
||||
- Cluster id from `citation_for_view=<user>:<cluster_id>`
|
||||
|
||||
## Required Parse States
|
||||
|
||||
Use and persist one of:
|
||||
|
||||
- `ok`
|
||||
- `no_results`
|
||||
- `blocked_or_captcha`
|
||||
- `layout_changed`
|
||||
- `network_error`
|
||||
|
||||
Plus these page-level flags:
|
||||
|
||||
- `has_show_more_button`
|
||||
- `articles_range`
|
||||
|
||||
## Data Quality Expectations
|
||||
|
||||
Observed from probe sample:
|
||||
|
||||
- `title`: 100%
|
||||
- `cluster_id`: 100%
|
||||
- `citation_count`: 100%
|
||||
- `authors_text`: 100%
|
||||
- `year`: 98.2%
|
||||
- `venue_text`: 94.7%
|
||||
|
||||
Implications:
|
||||
|
||||
- `year` and `venue_text` must remain nullable.
|
||||
- Dedupe must not depend solely on `year`/`venue_text` presence.
|
||||
|
||||
## Run Semantics
|
||||
|
||||
- First successful run per scholar = baseline.
|
||||
- If `has_show_more_button=true`, label result as partial in run status/UI.
|
||||
- Invalid/inaccessible scholar IDs are expected and should become structured run errors, not exceptions.
|
||||
|
||||
## Testing Priorities (Do Before Feature Expansion)
|
||||
|
||||
- Fixture parser unit tests using probe HTML snapshots.
|
||||
- Parse-state classification tests (ok/blocked/layout/no_results).
|
||||
- Dedupe integration tests (`cluster_id` first, fingerprint fallback).
|
||||
- Tenant isolation tests for run records and read-state.
|
||||
- Smoke test for manual run path through Docker.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
Pause implementation and re-probe if:
|
||||
|
||||
- Selector markers drop out unexpectedly.
|
||||
- Login/redirect pages become frequent for valid IDs.
|
||||
- Robots policy for profile endpoints changes.
|
||||
|
||||
|
|
@ -1,13 +0,0 @@
|
|||
# Scholar Scrape Probe (Temporary)
|
||||
|
||||
Purpose: short-lived workspace for information gathering and planning.
|
||||
|
||||
Rules:
|
||||
- No production code changes here.
|
||||
- Use this for HTML samples, parsing experiments, and extraction notes.
|
||||
- Delete this directory after we lock the scrape contract and parser plan.
|
||||
|
||||
Structure:
|
||||
- `fixtures/` saved HTML samples for parser validation
|
||||
- `notes/` extraction decisions, edge cases, and risk log
|
||||
- `scripts/` one-off probe scripts used only during planning
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
# Probe Findings -> Phase 1 Input
|
||||
|
||||
Generated from live probe run: `run_20260216T182334Z`.
|
||||
|
||||
## What is stable enough to build on
|
||||
|
||||
- Public profile endpoint works for anonymous access:
|
||||
- `GET /citations?hl=en&user=<scholar_id>`
|
||||
- Core row structure is present and parseable:
|
||||
- row: `tr.gsc_a_tr`
|
||||
- title + detail URL: `a.gsc_a_at`
|
||||
- citation count: `a.gsc_a_ac`
|
||||
- year: `span.gsc_a_h` (fallback: `td.gsc_a_y` text regex)
|
||||
- metadata lines: first/second `div.gs_gray` => authors/venue
|
||||
- `cluster_id` can be extracted reliably from `citation_for_view=<user>:<cluster_id>` in title URLs.
|
||||
|
||||
## What can break / where to degrade gracefully
|
||||
|
||||
- Invalid or inaccessible profile IDs may redirect to Google sign-in page.
|
||||
- Treat as `blocked_or_captcha` / `inaccessible` state, not parser crash.
|
||||
- `Show more` is present for tested profiles.
|
||||
- We currently parse first page only.
|
||||
- Because robots currently disallows `citations?*cstart=`, deep pagination should not be assumed.
|
||||
- Some rows are missing year or venue text.
|
||||
- Keep nullable fields and avoid failing dedupe for these gaps.
|
||||
|
||||
## Observed quality from probe
|
||||
|
||||
- Parsed publication rows: 57 across 4 accessible profiles.
|
||||
- Field coverage:
|
||||
- title: 100%
|
||||
- cluster_id: 100%
|
||||
- citation_count: 100%
|
||||
- authors_text: 100%
|
||||
- year: 98.2%
|
||||
- venue_text: 94.7%
|
||||
|
||||
## Recommended parser contract for implementation
|
||||
|
||||
- Input -> `PublicationCandidate`
|
||||
- `title` (required)
|
||||
- `cluster_id` (required when present in URL; expected high coverage)
|
||||
- `year` (nullable)
|
||||
- `citation_count` (nullable/int default fallback 0)
|
||||
- `authors_text` (nullable)
|
||||
- `venue_text` (nullable)
|
||||
- `title_url` (nullable)
|
||||
- Page-level parse status enum:
|
||||
- `ok`, `no_results`, `blocked_or_captcha`, `layout_changed`, `network_error`
|
||||
- Page-level flags:
|
||||
- `has_show_more_button`
|
||||
- `articles_range` string (e.g. `Articles 1–20`)
|
||||
|
||||
## Immediate test plan unlocked by this probe
|
||||
|
||||
- Add fixture-driven unit tests using captured HTML from `fixtures/run_20260216T182334Z`.
|
||||
- Add assertions for:
|
||||
- row count > 0 on known-good fixtures
|
||||
- profile name extraction
|
||||
- cluster_id extraction from title URLs
|
||||
- nullable handling for year/venue
|
||||
- blocked/inaccessible page classification
|
||||
- show-more partial warning classification
|
||||
|
||||
## Phase 1 implementation guardrails
|
||||
|
||||
- Keep requests low-rate with jitter and explicit timeout.
|
||||
- Persist parser status per scholar run.
|
||||
- If `has_show_more_button=True`, mark run as partial and show this in UI.
|
||||
- Never fail entire run because one scholar page is blocked or malformed.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue