How the new scraping system works
Each step says how it works. Anything not decided yet sits in orange, next to the step it belongs to.
The whole flow
Website → find the best source → build a scraper → read dates, prices and labels → check the scraper → keep it live or park it → hand over to TypeScript → match place, organiser and duplicates → save → review list.
Every night: run every live scraper, check it, save what passed, rebuild what broke.
Where Jev is used. Jev is a fast AI that answers yes/no and pick-one questions with a probability. It answers: is this feed item a real event? · is this item an event? · are these two the same event? · free? family? members only?
Every Jev answer is stored with its probabilities, so a threshold can change without asking again.
When Jev is unsure, a Claude agent is asked as one more signal — never the only gate.
Finding sources
dog.py sniffs a site for data machines can read, so we copy less off web pages.
- Looks for RSS / Atom feeds, iCal calendars, Google Calendar embeds, WordPress event APIs and JSON-LD on event pages.
- Keeps a source only if it has at least one future event: STRUCTURED, QUIET, BLOCKED or HTML-ONLY.
- Test 1 — same events? Jev asks per feed item: “is this one of the page's events?”
- Test 2 — same count? Far fewer than the page means a partial feed: rejected.
- Test 3 — same values? A sample is compared with the page. Hour-off times, wrong dates or stale feeds: rejected.
- Test 4 — real fields? It needs a start date and time, not just “published on”.
- The best source that passes all four wins. None pass → copy from the web page.
- GATE every sniff NOT BUILTA source is used only if it passes all 4 tests; the answers are logged.
- Decided: all ~586 sites are rebuilt before the site goes back on. Only the order they're built in is still open.
Of 20 sites picked at random, 1 blocked us (5%); the true share could be up to a quarter.
A source is one of the four things we store: the website we scrape. One source can list many organisers, like a council “what's on”.
Open
4 open Which sites count as sources, and who adds and retires them.
Do we label each source by kind?
- own site · ticket site · listings site · council page · social
Matching treats a listings site differently. The O2 Academy scraper reads a LiveNation feed for five Oxford venues — a listings site inside a “venue”. A first split of today's scrapers: 238 venue sites, 11 listings, 18 unsure (old data).
Do we scrape ticket sites (Glee, WeGotTickets) as second sources?
- Yes
- No — venue and organiser sites only
Without them “a ticket site wins on price” never fires: The Bullingdon's page gives no prices at all.
The keep rule needs one future event. What about a blocked site, or a page that gives only a rule?
- Keep it, marked
- Drop it until a dated event shows
ox.ac.uk returns 403; The Gardeners Arms says “every Sunday at 8.30” and lists no dates.
Who adds a site — sets up its place and organiser before its scraper runs — and who retires a dead one?
No step does this yet, and the fleet keeps changing after the first build.
How it runs
Can a setup step give each source its kind, fixed place, fixed organiser and town without a person — and how often do items' real place or organiser differ from the fixed one?
Method
- 21 sources (5 → 8 → 13 → 21), incl. O2/LiveNation, Bullingdon, Town Hall, SOFO, Henley Literary Festival, ANHSO, OAC, Didcot First, St Andrew's, Flair Dance.
- Kind: poc-b's location rule + a Jev pick-one; a person checks the 21.
- Fixed place by a plain Google Places lookup (not P17's matcher); count sources that resolve without a person.
- From P04/bulk2 items, count where the page states another place or organiser ('Location:', 'Presented by').
- Domain sharing: code over every targets/*.yaml, grouped by registrable domain — no sampling.
Measure and decision rule
Share set up with no person; click-weighted share of items whose true place ≠ fixed place; share with several organisers; number of shared domains. Decision: items off the fixed place >2% click-weighted → the shape carries each event's own location (Q 4.64 option 1); the same test for a per-item organiser; store domain+path if several domains host more than one organiser.
Stop rule
Stop at 13 if the setup step needs a person for more than half.
How it runs
How much do second sources — ticket sites (Glee, WeGotTickets, Skiddle, TicketSource, Eventbrite) and feed-only APIs (LiveNation) — add that venue pages lack, can feed-only items be verified against venue pages, and when sources disagree, which is right?
Method
- Step 0: robots.txt and terms of service per platform; drop any platform that forbids scraping.
- For the top-20 venues' events plus 21 gig/comedy events, follow the ticket links on the venue page and compare price, start, title and line-up.
- LiveNation: page through the API until 'total' is reached; match items to the O2 Academy, New Theatre, Old Fire Station and Bullingdon pages (all pages) by date + title.
- A person looks at mismatches only and marks which source is right from the page evidence; where the evidence conflicts, it is 'unresolved', never a win.
- Count events where the venue has no price and the second source does.
Measure and decision rule
Fill-rate gain per field; disagreement rate; wins per source and the unresolved share; share of feed items matched to a venue page. Decision: ticket sites fill price for >20% of top-20 events → scrape the permitted ones; the per-field trust order comes from resolved wins only; a feed with ≥95% matched items that adds events → the 'verify via venue pages' option for CAN'T VERIFY feeds, otherwise scrape the pages.
Stop rule
One pass.
How it runs
Do the four source tests accept good feeds and reject partial, hour-off, stale, wrong-field and non-event feeds across the census fleet?
Method
- Choose STRUCTURED sites from the dog-census ledger — its truth is AI-adjudicated and dog was tuned on it, so it only picks sites, never grades.
- Split sites dev/held-out before anything runs.
- Good-feed truth by code: match feed items to the page by exact title + date; the person checks only the mismatches.
- Plant by code per feed copy: truncate to 20%, shift times +1 h, move dates back a year (stale), replace startDate with a 'published' date, add non-event items (news posts, closures).
- Run the four tests; log each Jev Test-1 answer.
Measure and decision rule
False accepts of planted-bad feeds (must be 0, with an upper bound); false rejects of good feeds; Test-1 accuracy per item. Decision: false rejects >10% → loosen Test 2/3 tolerances on the dev sites only, then re-score on held-out.
Stop rule
Don't grow past 8 sites while any planted-bad feed is accepted.
Later — 2 small things
- Which of the 578 old scrapers move first — suggested: feed sites, then the most-clicked (
scripts/venue-clicks.py), then the rest. - Run every old scraper once for a first health map — they run unchecked today.
Building a scraper
A scraper only copies text off the page, word for word. It never reads dates, times or prices.
- One shared reader turns the copied text into dates, times and prices — the same way for every site.
- No regex, no chopping text, no word lists, no date maths. A site's quirks live in that site's file only.
- A Claude agent writes the copy-only scraper. Max 3 tries.
- A lint checks the code for banned tricks. Fail → back to the agent.
- Run it. The raw copied text goes to the reader, then to the checker.
- Tested on 9 sites, built in 1–3 tries. On pages never seen, 6 of 19 had their date only in unmarked text, which copying misses.
- GATE every build NOT BUILTLint 0 — the scraper only copies text.
- GATE every PR that touches the lint NOT BUILTUnit tests: every banned trick is caught; a known-clean scraper passes. Test version catches 80 of 80 planted, 51 of 52 unseen; not yet passing.
Do now
- Put these rules, the lint, and the scraper contract + template into the repo.
- Try 5 more sites — the messiest ones — with no human help.
Open
2 open
Some of the messiest sites can't pass as copy-only scrapers. What happens to them?
- Hand-written scrapers, checked like any other
- Dropped
The site stays dark until every scraper passes, so one site that can never pass keeps it dark.
Who may allow an exception to these rules?
- An agent — the rules rot
- A person — a person is back in the loop
How it runs
Can a Claude agent, with no human help and in at most 3 tries, build a lint-clean copy-only scraper for the messiest sites — and what does each build cost?
Method
- Fix the site list first (a list, not a POC) — 13 sites in clusters so later POCs have inputs: an overlap cluster (Bullingdon + Glee + LiveNation/O2 + WeGotTickets), a second one (OAC + CyclAbility), a listings site that names many places (a council what's-on or Daily Info), plus messy singles — dates in images/PDFs, pagination, Spektrix/Wix widgets, rule-only (Gardeners Arms), WooCommerce (SOFO), festival detail pages (Henley Literary Festival), a feed (an iCal source). Every site here goes to P37's dev half.
- Grow 5 → 8 → 13. The build agent gets only the contract, the template and the lint.
- First-5 kill check (early, not waiting for P37): lint 0 + P02 phase A's checker passes + a person checks 5 events per site against the page (~0.5 h).
- From 8 on, success also means matching the person's truth on 5 sampled events per site (the person checks them; P37 dev labels are reused where they exist).
- Log tries used, lint failures, drift into reading dates, exceptions asked for, tokens/$ and minutes. Mark every failed site 'park-likely' with its class (image/PDF, widget, blocked, rule-only …).
- For each failed class, one hand-written attempt (an agent with the copy-only rule lifted, or a person), checked by P02 and a 5-event person check, logging its build and check cost — the evidence for Q 4.3.
- If ≥80% succeed at 13, extend to 21–34 sites with the cheapest settings to feed P22 and P30.
Measure and decision rule
Share meeting the success definition; tries distribution; park-likely rate by site class; hand-written cost and pass/fail per failed class; $ and minutes per build (a ceiling for the fleet). Decision: <50% at 5 → stop and rethink before the ~15 dependent POCs. ≥80% of 13 → proceed to the fleet build order and the 21–34 extension. 50–80% → fix the contract/template for the failing class and re-run on fresh sites. The park-likely rate and hand-written cost per class go to the founder (founder_calls 1).
Stop rule
Stop at any level where under 50% succeed. After a prompt change, only fresh sites count.
How it runs
Does the lint catch banned tricks it was not written against, without flagging clean scrapers?
Method
- Mine real banned snippets (strptime, regex date parsing, word lists, all-day defaults, ISO literals like '2099-12-31T12:00:00', old imports) from the 578 old scrapers by AST.
- Inject them by code into clean scrapers with renamed variables and indirect data flow; whoever writes the injector doesn't see the lint source.
- Clean corpus: the 2 copytest scrapers now, P04's clean builds when they exist.
- Arms: today's lint; semgrep taint mode or a libcst pass.
- Score per injected violation (per line), not per file.
Measure and decision rule
Per-violation catch rate per trick family (ISO-literal fake dates named separately); false positives on clean scrapers. Decision: any family under 90% caught → add the dataflow arm to the lint rather than more names; any false positive on clean code → fix before it gates builds.
Stop rule
One pass over all mutation kinds; no rule edits while scoring.
Later — 2 small things
- Following page 2 / next month / hidden widget data · iCal, JSON-LD and APIs untested.
- Lint gaps: it checks variable names, not where values go; misses word lists, all-day defaults, old imports, fake dates; must lint the reader too; the Duckling imports need allowing; Tap Social uses a word list.
Dates and times
One reader, plain rules, for every site.
- Dates from feeds (ISO format) → Python's own exact date reader.
- Dates in sentences → Duckling, a maintained library that reads dates, times and prices (UK mode, plain Python).
- Exactly one possible answer, or blank with the reason. Blank beats wrong.
- Nothing invented: no made-up year, time, midnight or all-day. No time on the page → no time. “Midnight” written on the page is a real time.
- Rebuilt as plain rules; several start times → blank. On the sites it was fixed on: 1,454 of 1,544 values right, none false.
- The reader no longer asks Jev; an unsure reading is left blank.
- Unseen sites: the test stopped 6 times by site 12 on values judged wrong (one later ruled blank); each fix was a general rule.
- GATE every PR that touches the reader NOT BUILTThe reader passes a test set built from real pages — 0 wrong; blanks allowed. It includes doors-only times, feed midnights, rules with no dates and listings with no year. Plus the made-up hard strings.
- CHECK every PR NOT BUILTUnit tests for London time, 24-hour feed times, year rollover and date ranges.
- Decided: an event needs a start time to be shown. No start time → it stays off the site (a blank start costs the event, like a blank date). Exhibitions and other runs count their stated opening hours as their time.
Open
6 open How to read times that aren't a plain start — no time, doors, midnights, am/pm, rules without dates, no year, past events.
A feed says 00:00–23:59, or 23:59, when it means “no time”. Who spots it?
- The reader treats those as no time
- Trust the feed
Oxford City AC's feed puts a relay championship at midnight; LiveNation puts a Glee show at 23:59. Both would become real start times.
A page only says “doors 7pm” — what is the start?
- Doors is the start
- No start
The Bullingdon lists only doors; Glee says doors 6:45, last entry 7:15; Oxford Castle's header says “Open today 10am–5:30pm” on every page.
A time with no am or pm — a feed's “7:30”, a pub's “8.30 sharp” — morning or evening?
The test key reads 07:30; it may mean 19:30.
A page gives only a rule — “Every Sunday night at 8.30” — and no dates. What does the reader hand over?
- The rule as written; dates are made from it later
- Blank — the event is left off
The Gardeners Arms quiz and Oxford Castle's ghost tour (the venue with 29% of clicks) are rules only. Blank means they never appear.
A listing gives no year (“Open until 1 Nov”, “Sat 3 Oct”). Is the weekday enough to pick it?
- Yes, when only one year fits
- No — blank, or follow the link for the year
Otherwise the Ashmolean's free Gallery 8 show is rejected; today it only breaks at new year.
Past events are copied too (Tap Social: 837, mostly past). What happens to one we have never seen?
- Turned away at the hand-over, logged as “past”
- Saved, then marked “passed” by the clock like any other
The test must use the end date: an exhibition that opened in March is still on.
How it runs
How often do feeds write 'no time' as 00:00, 23:59 or 00:00–23:59:59, and can one generic rule turn those into 'no time' without erasing real midnight events?
Method
- Scan every STRUCTURED source in the census (JSON-LD, iCal, WordPress events API, LiveNation API); count items starting at 00:00 or 23:59, ending 23:59:59, or at 22:59Z, and feed prices of 0 / 0.0.
- Code compares each such feed time with the event's own page text: the page shows a different time → placeholder; the page shows no time → no-time; the page shows midnight → real. A person looks only at ambiguous pages.
- Real midnight negatives come from real late-night sources in the feeds (club nights, New Year's Eve, midnight mass in church feeds); no synthetic negatives.
- Candidate rules: r1 00:00 start + 23:59 end on the same day → no time; r2 a 23:59 start → no time; r3 per source, 00:00 → no time only if the source never gives other times; iCal VALUE=DATE via the icalendar library. For price 0: blank unless the page says free.
- Pages showing more than one time (a sidebar or header time could be picked up) are counted separately as ambiguous and go to a person.
Measure and decision rule
Placeholder prevalence per pattern; each rule's false 'no time' on real midnights and its misses; feed-zero prevalence and how often the page shows a real price. Decision: a rule with 0 false blanks on the real negatives and ≥95% placeholder catch goes into the reader; a feed zero that is ever wrong is read as blank unless the page says free.
Stop rule
One census pass; the negatives are fixed before any rule is tried.
How it runs
How often do real pages give only a doors, last-entry or opening-hours time, an am/pm-less time, or a date with no year — and what does each option in those questions produce on them?
Method
- Doors: from bulk2 pages and JSON-LD doorTime vs startDate at gig and comedy venues (O2, Bullingdon, Jericho Tavern, Wheatsheaf, Glee, Castle), count doors-only events; where both are given, measure the gap.
- am/pm: code removes the am/pm from times whose am/pm is known (P37 truth + feeds); run Jev at ≥0.8 vs blank.
- No year: within a ~20-month window a weekday + day-month is unique by calendar maths, so don't re-test it. Instead count by code (click-weighted) the dates whose stated weekday contradicts the day-month in every candidate year (venue typos), and yearless dates with no weekday ('Open until 1 Nov').
Measure and decision rule
Prevalence of doors-only (click-weighted) and the share where doors-as-start would be ≥30 min off; am/pm wrong count at ≥0.8 (must be 0); weekday/date conflict rate; yearless-without-weekday rate. Decision: am/pm resolved only if 0 wrong; a weekday that conflicts means blank, never a guess; yearless-without-weekday items follow the link for the year, or blank if the rate is tiny; doors-only prevalence at the top 20 decides whether 'Doors 7pm' must be displayed.
Stop rule
One pass over bulk2 plus the P08 sites.
How it runs
Can the reader turn page words into the right shape (list / rule / run / hours) with the right rule or range and no invented dates?
Method
- Take 13 series-heavy sources (from P10 and the walk-through, walk-through pages in dev).
- A person writes, for each item, the shape and the rule or range exactly as the page states it (an RRULE or start–end), not a list of dates.
- The reader emits shape + rule. Expand both the person's rule and the reader's rule with python-dateutil rrule / icalendar (battle-tested, independent of the system under test) and compare the dates.
- Invented dates are checked by code: every emitted date must follow from a date, range or rule stated in the page text.
- Opening hours: count exhibitions whose own page gives hours vs only a Visit page.
Measure and decision rule
Shape accuracy; rule/range match; invented dates (must be 0); exhibitions with hours on their own page. Decision: 0 invented dates and correct shapes on held-out → the reader emits rules; otherwise rule items stay blank and the founder decides the rule-only Q with its click cost (Castle 29%).
Stop rule
5 → 8 → 13 sources; stop at 8 if invented dates remain.
How it runs
Are Jev's probabilities calibrated enough that one threshold per question is safe, and how much do answers change between repeated asks and across model versions?
Method
- Freeze ~300 questions with person truth from P37 (dates, prices, labels) now; add P21's duplicate questions when they exist.
- Calibration: reliability curves (accuracy by probability band) per question type; choose each threshold on dev, report on held-out.
- Ask each question 3 times now, logging the versioned model id.
- Cross-version arm, opportunistic: ts_models.md lists only jev-1.13.0 today, so the primary gate is the weekly frozen-set re-ask against person truth; run a cross-version comparison only when a second version exists.
- Count argmax flips, flips across the threshold, and right→wrong flips.
Measure and decision rule
Calibration error per question type and the chosen threshold; same-model flip rate; cross-version flip rate; right→wrong flips. Decision: any question type whose accepted answers are wrong above ~1% at 0.8 gets its own higher threshold; a 'material flip' (crosses the threshold and changes the accepted answer) with the same model means the gate needs a tolerance; any right→wrong flip across versions blocks the version (pin the old one).
Stop rule
3 repeats; stop if the same-model flip rate is 0 after 2.
How it runs
Can one set of real pages, labelled once and split by page into dev, a working held-out slice, a sealed reserve and a final lockbox, serve as the known-good truth for every POC that needs a person's reading?
Method
- Unit: a page. Per page, code picks ≤3 events at random; the person labels those events' fields and every date/price candidate inside those events' blocks. Count pages and sites separately: 70 pages from ~45 sites (one listing page, plus a detail page where the site has them).
- Fix P04's site list first (a list, not a POC). Every site P04 iterates prompts on goes in dev, as do all walk-through (E1–E13) pages.
- Split by page before any labelling: dev 25 · working held-out 12 (for POC decisions) · sealed reserve 8 (opened only to re-score a fix on fresh pages, e.g. P08) · final lockbox 25 (opened once, only for the numbers the founder signs off — P01's relight-facing number among them).
- Save a snapshot of each page with its hash, so every POC reads the same bytes.
- Held-out pages (45): the person labels, in one tool, per sampled event: date, start, end, time kind (start / doors / last entry / hours), place as written, organiser as written, every price amount with its role, free, members-only, min age, status (on / cancelled / postponed), public event yes/no, title as shown AND clean title (decorations removed, each decoration recorded in its own field), the event's own link, and for rules/ranges the rule or range as written. Code tags the candidates; the person marks each as this event's or not, and adds any candidate the tagger missed — that count is itself a reader-recall number.
- Dev 25: two independent extractors — a Claude quote-first extractor and a code extractor (JSON-LD/microdata where present, else a second model family) — with the person adjudicating only disagreements. Dev truth is for tuning only. Note: because one extractor is the P01 checker, P01's prompt tuning on dev drifts toward its own readings; acceptable for dev, never for a score.
- Anchor: on 10 working-slice pages, also run the two extractors and count 'both agreed but the person says wrong'; publish that rate with the set.
- Freeze with a version id; a relabel makes a new version, never an edit.
Measure and decision rule
Labelling minutes per page; the both-agreed-but-wrong rate; candidates the tagger missed per page; held-out positives per field and label (under 20 → the consuming POC tops up with code plants). Decision: if two-extractor agreement is wrong on >2% of anchored fields, dev truth for those fields also goes to the person. Reachable bounds are published with the set: 0 misses on the 25-page lockbox gives a cluster-aware 95% upper bound of ~12%; a ≤5% bound needs ~60 lockbox pages (≈ +9 person-hours) — the founder can buy that (see founder_calls).
Stop rule
One labelling pass. The reserve and lockbox are opened only as stated; once opened, a slice is spent.
How it runs
Is the Duckling route (pyduckling-native-phihos, a single-maintainer fork with native wheels) safe to build the reader on, or should a fallback be chosen before P08?
Method
- Check the fork's release history, maintainer count and open issues; install it in a clean Linux py3.12 container the way CI would.
- Check downloads, dependents and the licence (Duckling is BSD; check what the fork's wheels bundle, e.g. the GHC runtime).
- Check the fallbacks: the official Duckling HTTP server in Docker, and dateparser alone.
- Run all three on the existing 72-case set and the 62 hard strings (dev material, direction only), and time them.
Measure and decision rule
Installs on CI yes/no; last release date; agreement between routes on the 72 + 62 strings; ms per string. Decision: installs and was released within ~12 months → keep, pinned; otherwise switch to the Docker server pinned to the same Duckling version (answers diffed on the 134 strings, not assumed equal) or dateparser if it matches on ≥95% of those strings. The choice is made before P08 starts.
Stop rule
Time-box 1–2 hours.
Later — 3 small things
- End times: Jev 9 / 10; a “last time” rule 10 / 10 but wrong on 12 of 20 events with no end.
- Make the test set a gate on every reader change (not enforced yet). Jev model changes no longer affect the reader.
- TypeSafe can change rate limits without notice — no plan if Jev gets slow, dear, or goes away.
Prices
Plain rules read the ticket price from the stated amounts; Jev no longer decides.
- Several amounts → rules drop fees, deposits, sponsorship and prizes; which one is the price stays open (below).
- “Free” = £0 · “From £12.50” = the lowest price only.
- GATE every PR that touches prices NOT BUILTThe price test set from real pages — 0 wrong; “free”, “from”, booking-fee, concession, member-free and gift-voucher cases included.
Open
Several real prices — which is the price?
- The cheapest anyone pays, booking fee in
- The standard adult ticket, booking fee in
- The standard adult ticket, fee left out
A £19.50 Glee show would read “From £12.50” (the concession). WeGotTickets adds a £1.20 fee. Members-free entry, Gift Aid, a book's RRP and a membership fee must never count.
How it runs
Superseded: prices are now read by rules, without Jev. On real pages with planted traps, does 'Jev per amount + the free rule' give zero wrong prices, and what does each 'which price' option show?
Method
- Truth: P37's per-amount role labels (ticket / concession / fee / member / voucher / RRP / membership / donation) and free, on held-out pages; the jev-price-poc v2 fixtures are dev only.
- Plant traps by code (their v2 list), inside the event's own block too.
- Compute each 'which price' option's displayed price twice: from the person's labels and from Jev's labels.
Measure and decision rule
Wrong prices accepted at the chosen confidence (must be 0, with the 95% upper bound); false 'free' (must be 0); coverage and the share sent to Claude as secondary numbers; per option, events whose display is >20% off the standard adult ticket. Decision: Jev (or the cascade) is kept only at 0 accepted-wrong; everything below the threshold is blank; coverage below ~70% is reported to the founder as a blank-price cost, not fixed by loosening. The options table goes to the founder.
Stop rule
8 → 13 → 21 pages; stop if a false 'free' persists at 13.
Later — 1 small thing
- One price (“Tickets £12”) — is the top price blank, or also £12?
Labels and categories
Worked out in Python, before the hand-over.
- Labels: Jev asks per new event — public or members only? · free? · family-friendly?
- Code turns the answers into event fields. The site never reads Jev directly.
- GATE before any label goes on the site NOT BUILTEach label passes a test set checked by a person, never AI-written answers.
- GATE whenever Jev's model changes NOT BUILTRe-ask a saved set of Jev questions; any answer that flips blocks the new model until checked.
Open
4 open Whether an item is a public event at all, who spots cancellations, and how categories work.
Who decides “free” — the price rules, or Jev's “free?” question?
- The price rules
- Jev's “free?” question
Both answer it today, so they could disagree.
Is this a public event at all? Nothing asks that of web-page items.
- Jev asks per new item
- Leave it to the review list
Didcot First's board meeting and the Ashmolean's gallery-closure notice would go live as events.
Who reads “cancelled” and “postponed”?
- A Jev question per event: on, cancelled or postponed?
- Nobody — the words are only kept as copied
Oxford Town Hall's cancelled organ recital stayed live on the old site. SOFO's page says “POSTPONED” in the title and “cancelled” in the text.
Categories were left open on purpose. What reads them, and how many?
- Jev
- A small tuned model
Category links only keep working if the list stays, or each old one redirects. Lessons from last time: one main category · “don't know” allowed · ~8–10 categories, not 21 · test answers checked by a person · look at what changes over every event before trusting a score.
How it runs
Can Jev answer members-only, free, family, minimum age, 'is this a public event at all?' and 'on / cancelled / postponed?' with zero wrong at the chosen threshold on a person-checked set?
Method
- Truth: P37 labels; the walk-through items (Didcot board meeting, Ashmolean closure, Town Hall cancellation, SOFO postponement, ANHSO Zoom) are dev only.
- Top up held-out positives by code to ≥20 per label: inject 'Members only', 'Age 18+', 'Board meeting — agenda', 'Gallery closed', '- CANCELLED' / 'POSTPONED:' into real items; move a real cancellation into the body only.
- Ask each Jev question; pick the threshold on dev; score held-out.
Measure and decision rule
Precision and recall per label, and wrong answers accepted at the threshold (with the 95% upper bound), reported separately for natural and planted positives. Decision: a label with 0 accepted-wrong and usable recall on natural (not planted) held-out positives can go on the site; others stay off. If the cancelled question catches every plant with 0 false alarms, the 'Jev per event' option is adopted and the hand-over carries the status text.
Stop rule
Stop a label as soon as it has a wrong answer at ≥0.8 on held-out; report and redesign that question, don't re-tune on held-out.
How it runs
With ~8–10 categories and 'don't know' allowed, does Jev pick-one match the parked catv2 model on a person-checked set — and which old category links carry traffic?
Method
- Fix the category list first from P28's traffic pull (keep the slugs that earned clicks) plus the plan's lessons (one main category, 'don't know', ~8–10).
- If every new category is a union of old categories, map catv2's output many-to-one and keep it as an arm; otherwise drop the catv2 arm and measure Jev alone.
- Draw ≥200 events across the fleet from bulk2; a person labels them against the fixed list.
- Paired McNemar test if both arms run.
Measure and decision rule
Accuracy per arm (paired test when both run). Decision: Jev, unless catv2 (unions only) is significantly better (p < 0.05) by ≥5 points; categories with meaningful old clicks keep their slug or redirect. Runs after relight unless the founder wants category pages at relight.
Stop rule
One pass on the held-out set.
Later — 2 small things
- An earlier model sits on a parked branch (
catv2/parked-do-not-merge) — real gain ~+5 points, tested only on AI-written answers · 409 old chats about it not searched. - Age limits — 18+, 14+, (5+) in a title — and sources disagree: The Bullingdon says 18+, Glee 14+, for the same show.
Checking a scraper
validate-scraper-output.py works on any scraper: it reads only the scraper's output and the real page. Agents never grade themselves — only scripts write verdicts.
snapshot.pyopens the page in a hidden browser and saves a small copy.verbatim_check.py— is every copied text really on the page? Exact match.segment.pycuts the page into one block per event.check-missed.py— does the page show a time or price we left blank?count-site-events.py— did we drop events?verdict.py— OK · FAILING · BROKEN · QUIET (really nothing on) · PARTIAL (fewer than the site) · CAN'T VERIFY (on no page a person sees) · CAN'T CHECK (our side failed — never blamed on the scraper).- Zero wrong to pass.
- GATE every PR that touches the checker NOT BUILTCode plants errors in saved pages — a date +1 day, a sidebar date, a dropped event; all must be caught. Test built; checker fails it.
- GATE every PR that touches a scraper NOT BUILTThe scraper runs on its saved pages offline and must match its expected output.
Checking each read value appears in the copied text blocked 11 wrong values, no right ones; it can't tell a start from another listed date.
Do now
- Run the word-for-word check on Henley and Tap Social.
- An empty page is QUIET, not BROKEN. CAN'T VERIFY is not wrong.
- Follow every page when counting — O2 has 6 pages; counting page 1 alone is the old bug.
Open
A feed with no page a person sees is CAN'T VERIFY for ever, and only OK saves. Does it ever save?
- Yes, if its items match the venue pages that exist
- No — scrape the web pages instead
LiveNation's feed behind the O2 Academy scraper. It also returns 20 of 105 events, and there is no page to count against.
How it runs
Can a generic step tell a value that belongs to this event from one elsewhere on the page (sidebar date, related product's £0.00, gift voucher, a past exhibition's range, a stale banner)?
Method
- Arms and pass marks written before any run: (a) segment-block containment; (b) Jev per value, 'does this value belong to <title>?', with the block text; (c) (a) + (b); baseline: the first value on the page.
- Use the gold set (P37) candidate labels: every date/price candidate on each page is already marked as belonging to the event or not. Scoring uses held-out pages only (the split is by page).
- Plant by code both outside the event's block (a copied sidebar) and inside it (a related product or voucher line inserted in the same card, as SOFO and Castle really do).
Measure and decision rule
Wrong attributions accepted at ≥0.8, with the 95% upper bound computed per candidate (assumes candidates are independent; the per-page bound, which is wider, is reported alongside); blank rate. Decision: an arm whose upper bound is ≤2% with ≤20% blank goes into both the reader and the checker, and R 5.12 becomes a measured risk. Otherwise a person-check sample stays mandatory for prices and dates on sidebar-heavy templates.
Stop rule
5 → 8 → 13 pages; stop if every arm shows a wrong accept at 8.
How it runs
Does validate-scraper-output catch code-planted errors on messy saved pages, and how often does it fail a correct scraper through Jev noise alone?
Method
- Test the planned split checker (snapshot / verbatim / segment / check-missed / count / verdict) as wired from today's validate-scraper-output.py; name the commit under test so results carry to the rewrite.
- Phase A (now): controls are Henley and Tap Social. Code plants one lie per copy: date +1, hour +1, sidebar date substituted, one event dropped, a fake event added, page 2 not followed (O2's 6 pages), a time blanked where the page shows one, an empty page (must be QUIET), 0 items from a non-empty page (must be BROKEN), a fetch timeout / Jev 5xx / snapshot failure (must be CAN'T CHECK, never blamed on the scraper), and a correct rule block that expands to many dates (must not be 'dropped' or 'missed').
- Measure the per-answer Jev error rate on the controls, then set N by simulating the binomial over nights and fleet size, not by counting reruns.
- Phase B: once P04 builds exist, add 3 of them as extra controls and re-run the same plants.
Measure and decision rule
Catch rate per plant kind (100% for every non-sidebar kind); the verdict for the empty / zero-items / our-side-failed / rule-block plants; per-answer Jev error rate (from Henley and Tap Social, the tuned sites, so likely optimistic) and the simulated false-FAILING rate for each N. Decision: any non-sidebar plant missed → the checker can't grade P04 or gate go-live until fixed. N = the smallest value whose simulated false fail is under 1 in 100 site-nights at fleet size.
Stop rule
Phase A is one pass; fix and re-run only failures. Phase B waits for P04.
How it runs
Does segment.py cut listing and detail pages into one block per event, measured against independent block truth rather than listing pages alone?
Method
- Split templates dev/held-out; the existing 48 listing samples segment.py was tuned on are dev only.
- Use STRUCTURED sites whose JSON-LD lists each event as independent block truth: every JSON-LD event should map to exactly one block. A person checks only the disagreements.
- Add 5 → 8 → 13 messy HTML-only pages (detail pages, calendars, tabs) where a person marks blocks.
Measure and decision rule
Share of held-out events in exactly one block; merged and split blocks; JSON-LD-vs-display count mismatches as their own class (truth noise, not segment errors). Decision: ≥98% on held-out templates → trusted; otherwise the count and missed checks must not use blocks on that template class.
Stop rule
Don't grow past 8 while any template class is under 90%.
Later — 3 small things
- Fail rule: N problems over 2 nights, so one noisy Jev answer never fails a scraper · N not set · simulated only: works at ~50+ events a night with Jev + Claude, Jev alone needs ~100+, small venues unsolved · starts as a to-do list, blocks only once its own error rate is measured.
- Split the checker out of today's 1,099-line file · “you may also like” blocks leak in, month-name lists, a date format it misreads · price-missed question untested · some dates sit in hidden page attributes.
- Titles, descriptions and images have no reader or check.
Keeping a scraper
A scraper that passes goes live; one that keeps failing is parked.
- Clean = lint 0 · every copied text on the page · nothing missed · nothing dropped.
- Pass → live.
- 3 fails → parked.
- Decided: new data reaches the live site only through the new system — a new, empty database with one writer. The site stays dark until every scraper is rebuilt and the quality check passes. Old scrapers do not keep feeding today's system meanwhile, so today's system (NG) and its registry retire.
scraper_status: one row per scraper — a site can have two, like the Ashmolean's exhibitions and events — with source, verdict, nights in a row, last built, last passed.- GATE every build NOT BUILTGo live only if lint 0, every copied text is on the page, nothing missed, nothing dropped.
Do now
- Move the matching scripts and checked test pairs out of /tmp into the repo, next to the scraper code.
Open
2 open Where the code lives, and what happens to parked sites.
Where exactly does that code live?
packages/scraper/match/
Never under plans/.
Parked sites pile up with no owner. What happens to them?
- A weekly list for a person
- Drop sites nobody visits
How it runs
Of the ~586 target sites today, how many are live with future events, quiet, blocked, dead or moved — and how many old scrapers still work?
Method
- From bulk2 (1,497 pages fetched 2026-09-27) + urls.txt: HTTP status, redirects to another domain, parked domains, empty pages; join dog-census verdicts.
- Break and dormant rate per scraper from packages/scraper/data/state.json git history (207 commits, 139 days), direction only.
- Run old scrapers with --no-ingest into scratch output on 5 → 8 → … → 89 sites, after confirming no other path in the run writes to a database.
- Keep rule: apply 'at least one future event' to BLOCKED and rule-only sites and record what each Q option does to them.
- Weight everything by clicks.
Measure and decision rule
Share per status, raw and click-weighted; blocked share among the top 20; break and dormant rate per night. Decision: sets the build order and the number of builds; if blocked or rule-only sites are common in the top 20, the keep rule needs 'keep, marked'.
Stop rule
Stop the old-scraper sample at 89.
Later — 1 small thing
- Save a test page + expected output per scraper, so changes can be tested offline.
Python → TypeScript → Drizzle
Python works things out. TypeScript decides and saves.
- The shape of an event is written once in TypeScript (Zod). The Python copy is generated from it. CI fails if they differ.
- Small and flat: no IDs, slugs or status — those belong to TypeScript.
- Python writes one file per run, one event per line: copied fields, read values, labels.
- A place guess is only a guess (“probably venue 42, 93% sure”). Python may read the database to suggest, never to decide.
- TypeScript checks every event against the shape. A bad one is rejected with the reason.
- Drizzle creates the tables, in a new empty database.
- GATE every PR NOT BUILTThe Python copy of the shape matches the Zod shape — CI fails if not.
- GATE every run NOT BUILTEvery item is checked against the shape; a bad one goes to rejected_items with the reason.
Tested on 6 sites, all passed: the copied shape gains a subtitle and a list of every other labelled fact — doors, age, ticket link.
Facts inside one sentence — a room inside the venue, “postponed” inside the title — are copied whole; the reader separates them.
Do now
- Write the shape, generate the Python copy, create the empty tables.
Open
3 open What the hand-over carries, what makes an item the same tomorrow, and whether Python ever writes.
The shape isn't written yet. What must it carry? The tables already have a column for each of these:
- a place for every event — its own location, or one fixed venue the scraper declares
- venue and organiser as separate fields
- the source's own event id, and a real link per event (Tap Social's is “javascript:;”)
- “cancelled” / “postponed” text
- a rule or range as written (“Every Sunday at 8.30”, “8 Oct – 11 Apr”), and support acts
- the fields the reader couldn't read, and why
- which run each item came from
A fixed venue is wrong when the event is elsewhere: SOFO's concert is at a church 700 m away; Henley's talks move between stages.
A page gives no event id and no link of its own. What makes an item the same item tomorrow?
- The scraper builds a key from date + title
- The link when it's real, else date + title
The Bullingdon, ANHSO and The Gardeners Arms have neither. A title key breaks when “- CANCELLED” or “(18+)” is added; a link key breaks when the link holds the date (SOFO).
Does Python ever write to the database — its log tables (builds, runs, Jev answers, evidence), or events?
- Never — it writes files; TypeScript loads everything through Drizzle
- Only its own log tables, in their own schema; TypeScript writes events
- Log tables stay as files next to each scraper's output
~63 Python files run raw SQL today, and the drift check only guards Drizzle's tables.
How it runs
Does one Zod shape, generated into Python, carry every fact real items from messy sites need without loss — and does CI catch drift between the two copies?
Method
- Draft the Zod shape (the repo pins zod ^3.24, so zod-to-json-schema is the right generator; Zod 4's native toJSONSchema if the repo upgrades) → datamodel-code-generator for pydantic.
- Seed the missing-field list from walk-through holes H1, H4, H8, H10, H21, H23, H28.
- Push the E1–E13 items through now; add P04 items when they exist.
- Plant drift (rename a field in Zod only) → CI must fail. Plant bad items → each must be rejected with its reason.
- Past items: count past-by-start vs past-by-end in Tap Social + Henley + an exhibition source; apply both Q 4.52 options.
Measure and decision rule
Every seeded hole has a field or an explicit 'not carried' decision; drift caught (100%); bad items rejected with a reason (100%); running exhibitions dropped per Q 4.52 option. Decision: the Q 4.64 field list is what the holes and items needed; the Q 4.52 option that drops 0 running exhibitions wins.
Stop rule
Stop when two consecutive new sites add no new missing field.
How it runs
Which identity key (source id, link, date+title, or link-else-date+title) keeps the same event the same across nights and edits, without ever joining two different events?
Method
- Planted truth (the scored part): code edits real items — add '- CANCELLED', '(18+)', 'SOLD OUT', 'POSTPONED:'; move 18:30 → 17:45; change the date inside a URL (SOFO); add a 1pm/7pm twin. Splits and collisions are known by construction.
- New-scraper arm: run P04's scrapers on two same-day fetches of the same pages plus the planted edits; test each key on what new scrapers emit.
- Real history, direction only: ng.wedge_raw_items (read-only) and packages/scraper/output/runs (299 runs; output/ is gitignored, so there is no git history of items) — report key-change counts without claiming which are true splits.
- Count real date/time moves in output/runs.
Measure and decision rule
Split and collision rate per key on planted and new-scraper data (collisions must be 0); key-change counts on real history as direction. Decision: the key with 0 collisions and the lowest split rate answers the identity Q.
Stop rule
One pass per key over all data; no key tuning on the planted set.
How it runs
Do the planned tables, written as Drizzle migrations, apply to an empty database and back, and do their constraints reject planted bad rows while accepting real awkward ones?
Method
- Write the schema in Drizzle, including the audit schema (check drizzle-kit multi-schema support and whether db-drift.yml would cover it).
- Apply to PGlite for speed; migrate up, down, up.
- Planted bad rows must fail: end < start, price_max < price_min, a duplicate untimed date, a bad status, a physical place with no address or parent.
- Real awkward rows from E1–E13 must pass: a room with a parent, an online event with a town, a 186-day exhibition, two scrapers on one site.
- Role test on Docker Postgres: only the writer role can write events; a planted write from another role is refused.
Measure and decision rule
All planted rows rejected, all real rows accepted, up/down clean, role test passes. Decision: where one real row needed two overlapping tables, merge them; if the audit schema escapes the drift check, keep logs as files or bring them under Drizzle (answers Q 6.13).
Stop rule
One pass; fix and re-run failures.
Later — 2 small things
- A raw inbox table next to the file? · stamp each file with the shape version · how far ahead the page shows tonight · event images · how AI labels are stored.
- Build order: places → organisers → matching + saving → series → safety limits + review list → point the scrapers at it.
Database schema
New tables in the same Supabase project — users, newsletter sign-ups and logins live there and are kept. Drizzle creates every table. Nothing is ever hard-deleted. Every table has created_at; tables that change also have updated_at.
- Rules the database enforces, not just the code: every status and kind is a fixed list (CHECK) · end ≥ start on dates · price_max ≥ price_min · a physical place needs a street or postcode, or a parent place (a room) · one row per (event, date, time), with a blank time counted once (
NULLS NOT DISTINCT— Postgres otherwise treats blanks as all different). - Indexes: fuzzy search on
events.title_normandplace_names.name_norm(pg_trgm) ·event_dates(date)for day, week and tonight pages ·events(town_id, status)·event_sources(source_id, source_event_id)· every foreign key. - GATE every PR NOT BUILTMigrations run on an empty database, and back.
- GATE every PR The schema files match the live database (the existing drift check).
Tested on made-up cases and 4 real festivals: 175 event shapes alone, and 10,287 pairs among 144 of them, fit after small changes; no new tables.
Do now
- Write these as Drizzle migrations, empty, alongside the hand-over shape.
Open
2 open Old AI scores, and how to store sold out and save-the-date.
Sold out and “save the date” — new status values (“full”, “provisional”), or a separate flag and a label?
- New status values
- A sold-out flag on events and dates; “save the date” as a label
A date can be both sold out and cancelled, which one status can't hold.
Today's events and organisations carry AI scores (discoverability, uniqueness, cultural value, accessibility, capacity). Keep them?
- Drop them
- Carry the idea over as new columns
Only the admin panel reads them today. Dropping them needs no new column; keeping the idea does.
| Table | Status | Key columns |
|---|---|---|
places | new | id · slug (unique) · name · kind (building / online / tbc) · parent_place_id → places (rooms) · street · street_2 · city · county · postcode · country_code · lat · lng · town_id → towns (required) · google_place_id (a clue, not unique) · venue_type · description · image_url · website · opening_hours · status (unverified / active / closed) · created_at · updated_at · deleted_at PROPOSED town may be empty for online or TBC places. |
place_names | new | place_id · name_norm · source_id? · unique (name_norm, source_id) — every name a place has been called |
place_lookups | new | query_text (unique) · Google result (jsonb) · place_id? · looked_up_at — so Google is asked once per name |
organisers | new | id · slug (unique) · name · kind · description · website · contact_email · telephone · image_url · town_id → towns · status (unverified / active / closed) · created_at · updated_at · deleted_at |
organiser_ids | new | organiser_id · type (charity / company / website) · value · unique (type, value) |
place_organisers | new | place_id · organiser_id · role? · key (place_id, organiser_id, role) |
sources | new | id · scraper_id (unique) · url · event_page_url (where the scraper looks) · domain · kind? · fixed_place_id → places · fixed_organiser_id → organisers · town_id → towns (where events with no venue show) · created_at · updated_at |
events | new | id (the permanent ID used in links) · title · title_norm · description · category (empty = don't know) · categories (a second one, only if very sure) · is_free · is_family · members_only · min_age (as stated: 18+, 14+) · place_id → places (empty = no venue) · town_id → towns (required — an event with no venue still shows on its town's page) · organiser_id → organisers · status (scheduled / cancelled / postponed / passed / removed) · parent_event_id (the festival or programme it sits in) · merged_into_id → events · price_min / max_pence · price_display · image_url · ticket_url · link_url (the best page to send people to) · created_at · updated_at · deleted_at PROPOSED labels in the page's words, seen on real festivals · made-up cases: a maximum age, a livestream's link to its in-person show. |
event_dates | new | id · event_id · date (London) · start_time (empty = no time, never a fake all-day) · time_kind (start / doors / last entry / opening hours) · end_date · end_time · subtitle (this date's line-up or performer) · ticket_url · price_min / max_pence · year_stated (did the page give the year?) · previous_start (to spot a moved date) · status (one date cancelled) · stale flag · first_seen_night · last_seen_night · unique (event_id, date, start_time) PROPOSED a date's own festival link and place, or several places; festival and town lists count them · other times, like doors · labels like “captioned”. |
series | new | event_id (key — a repeating event is an event with one series row) · shape (list / rule / run / hours) · rule (RRULE) · page_words (“every Wednesday”, kept as written) · rule_from (stated on the page / worked out) · starts_on · ends_on · daily_open · daily_close · seen_until (how far ahead the page showed) · stored_until (how far ahead we store dates) · created_at · updated_at PROPOSED listed every date? (if not, “unknown”, not “no”) · calendars the rule skips, like term time · runs with no start date. |
series_exceptions | new | event_id · date · kind (cancelled / moved / added) · detail · unique (event_id, date) |
event_sources | new | event_id · source_id · event_date_id? → event_dates (when the source lists one date) · source_event_id · source_url · first_run_id · last_run_id · last_seen_night · said (jsonb: what this source last said) · blanks (jsonb: fields it couldn't read, and why) · unique (source_id, source_event_id) |
event_changes | new, add-only | id · event_id · event_date_id? · field · old · new · source_id · run_id · actor (scraper / admin / merge) · reason · at |
field_locks | new | event_id · field · value · locked_by → users · reason · locked_at · expires_at? · key (event_id, field) |
url_redirects | new | old_path (key) · target_kind (event / place / gone) · target_id — merged events, moved dates, old links |
match_decisions | new, add-only | id · kind (place / organiser / duplicate) · left_ref · right_ref · decision · method (rule / score / jev / person) · score · run_id · decided_by — the remembered answers |
review_items | new | id · kind (duplicate / place / organiser / safety_stop) · refs (jsonb) · status · match_decision_id · decided_by · decided_at |
run_stats | new | run_id · source_id · items_in · new · changed · gone · rejected · stopped · stop_reason |
audit.builds | new | id · scraper_id · attempt · model · lint_ok · lint_failures (jsonb) · code_sha · at |
audit.runs | new | id · scraper_id · build_id · started · finished · verdict (OK … CAN'T CHECK) · failing_fields (jsonb) · items · jsonl_sha · shape_version |
audit.jev_answers | new | id · run_id · question_key · question_version · subject_hash · answer · probabilities (jsonb) · model · decided_by |
audit.evidence | new | id · run_id · url · snapshot_path · sha · http_status · fetched_at · keep_until |
audit.scraper_status | new | scraper_id (key) · source_id · source_kind (STRUCTURED / QUIET / BLOCKED / HTML-ONLY) · verdict · nights_in_row · fails · state (live / parked) · last_built · last_passed |
rejected_items | new, add-only | id · run_id · source_id · item (jsonb, as handed over) · reason (bad shape / no date …) · at — where “rejected, with the reason” goes |
alerts | new | id · kind (safety stop / many sites broke / Jev down / CAN'T CHECK streak) · source_id? · detail (jsonb) · raised_at · cleared_at · cleared_by |
fleet_snapshots | new, add-only | night · live · failing · broken · quiet · parked · cant_check · events_saved · events_changed — the nightly summary, one row a night |
quality_runs | new, add-only | id · kind (relight / weekly) · sample_size · wrong · blank · guessed · passed · run_at — one row per quality check |
quality_items | new, add-only | id · quality_run_id → quality_runs · event_id → events · field · ours · page_value · page_quote (word for word) · verdict (right / wrong / blank / guessed) · checked_by (agent / person) · planted (a known error?) |
towns | keep | id · name · slug (unique) · county · lat · lng · parent_town_id → towns · timezone · intro_copy · city_id → cities · created_at PROPOSED an “Oxfordshire” row, for festivals across the county. |
cities | keep | fixed reference data |
users, newsletter_subscribers, contact_messages | keep, with their rows | people's data, not scraped data — starting again must not wipe these, or logins |
submissions | change | venue_id → place_id · event_id → the new events |
events, venues, addresses, organisations (today's) | retire | kept read-only as an archive, then removed · their scores (score_discoverability, score_uniqueness, score_cultural_value, score_accessibility, score_capacity_signal), confidence and canonical_hash have no new column |
ng.* (NG's tables) | retire | their ideas live on in event_dates, series and series_exceptions; no rows carried over |
Later — 2 small things
- Open questions from the steps land on these tables: several showings and moved dates → event_dates · organiser roles and a main organiser → place_organisers / events · website as domain or domain + path → organiser_ids · what counts as one place → places.parent_place_id · festivals → events.parent_event_id · categories → events.category · undo a merge → events.merged_into_id · lock expiry → field_locks · how far ahead to store → series.stored_until · a raw inbox per run → an optional audit.run_items.
- Some tables overlap: place_names vs aliases, match_decisions vs review_items, Jev answers logged by both matching and labels.
Places
A place is a building: our own ID, with the Google place ID kept as a clue, never the key. Rooms sit inside a building.
- “Online” and “TBC” are states of their own — never looked up, never a pin in the town centre.
- No location on the event → a venue's own site uses its venue; a listings site gets “no venue”, honestly.
- Seen it before? Same scraper, last run → update it.
- A new place name → ask Google Places once and remember the answer. Buildings only, never organiser names.
- Why this matters: 592 of 602 old “venues” are really organiser names; 7 organisers sit on one dot in central Oxford.
- Every place match — merged, sent to review, or new — is logged with its score and the rules it passed.
- GATE every PR that touches place matching NOT BUILTChecked place pairs + a held-back set never tuned on: 0 wrong merges. Planted look-alikes (two St Mary's) must stay apart.
Tested: Jev answers once per pair of place names — same building, room inside, or different — remembered. Duplicates found: 3 → 27 of 40.
Jev once called O2 Academy2, a room, the same building as O2 Academy. Fix tested: any doubt in a group blocks merging.
Open
3 open How two names become one place, what counts as one place, and when to clean up the old venues.
How do two names become one place?
- Score every clue — name, postcode, distance, Google ID, organiser — with a hard rule: same postcode or within 300 m. Tested on 23 checked pairs: 5 of 11 same places merged, 0 of 12 different ones.
- Exact: our saved names → Google's ID → still unclear, a person checks
Decides what matching code we write.
What counts as one place?
“Main Hall” at the Town Hall, the Bullingdon's “front bar”, the Ashmolean's “Gallery 8”, a stage in a marquee at Phyllis Court, units in one complex, a park vs its building.
Clean up the old venues first, or after?
- First
- After
Matching against today's bad venues copies their mistakes.
How it runs
Which place-matching method, and do the organiser rules, give 0 wrong merges on held-back pairs from new output plus listings sources, including planted look-alikes and shared-domain traps?
Method
- Place strings from 3–5 listings sources (a council what's-on, Daily Info, LiveNation's five venues), P04's listings site, and bulk2 text — not only venue-own sites, which name themselves.
- One person session labels ~100 place pairs and ~60 organiser pairs; ~30 place and ~20 organiser pairs held back.
- Plant: two St Mary's, St Mary Magdalene Woodstock vs Oxford, Glee Club Oxford vs The Bullingdon, O2 Academy vs Academy2; colleges on ox.ac.uk, two Greene King pubs, the promoter Divine Schism.
- Places — arm A: score every clue with the hard rule (same postcode or within 300 m), Splink. Arm B: exact saved names → Google ID → person. Compare Google Places with postcodes.io/Nominatim.
- Organisers: run the plan's rules in code; check Charity Commission API key access first.
- Re-look-up 50 old google_place_ids for churn (old rows, direction only).
Measure and decision rule
Wrong merges for places and for organisers (each must be 0, with the upper bound); misses; review items; Google $; ID churn (direction). Decision: both place arms will likely show 0 wrong on ~30 pairs, so the real decider is fewer review items at 0 wrong; organiser rules ship as the gate only at 0 wrong, and a rule with a wrong merge is dropped, not patched per site.
Stop rule
Stop any arm at its first wrong merge on held-back pairs.
How it runs
Can a generic step classify a place string as building, room in a building, online or TBC, and do the schema's address rules hold for real rooms?
Method
- Now: collect every sub-space and non-place string across 30 sources (from P10 and bulk2); a person labels building / room-of-X / online / TBC; Jev pick-one vs the Google Places result type.
- Now: for items marked TBC or place-unknown, count how often the detail page does name a place (9.3b).
- After P26: insert the results into the local schema; count CHECK violations and events left with no town.
Measure and decision rule
Classification errors that would cause a wrong merge or a wrong pin (target 0); TBC items that actually had a place; CHECK violations; events with no town. Decision: adopt 'a place = its own entrance/address; rooms are children' if ≥95% of labels fit; if TBC items often have a place on the detail page, the scraper must follow the detail page before 'TBC' is allowed; add sources.town_id if any real online/TBC event has no town.
Stop rule
One pass over the 30 sources.
Later — 1 small thing
- One “Online” place, or one per organiser? · score cut-offs, set once it runs on clean scraper output · free map alternatives · re-test Google itself · old lookup code to reuse · no fix yet for the 3 missed places.
Organisers
An organiser is who runs the event — its own thing, linked many-to-many with places.
- A pub running its own quiz = one place + one organiser, linked — not one merged row.
- Its IDs live in their own table: charity number, company number, website.
- A charity or company number matches on its own. A website alone never does.
- Website + exact name → same organiser. An event on the organiser's own site → the site's owner.
- Listings pages never guess · “presented by arrangement with …” is ignored · shared platforms (Greene King) are on a denylist.
- A partial match → review list. The answer is remembered.
- GATE every PR that touches organiser matching NOT BUILTChecked organiser pairs, including shared-domain traps: 0 wrong merges.
Tested on made-up data: a name printed as the venue at two postcodes is treated as the organiser, placed only by the event's own address.
Open
4 open Who counts as the organiser, what we store for them, and whether they get their own matching step.
Does each place–organiser link have a role?
- organiser · co-organiser · host · sponsor
- No role
The page names someone other than the site's owner. Who is the organiser?
- The name on the page; one main organiser
- The site's owner, unless a person changes it
- Several, each with a role
The Bullingdon hosts Glee's comedy and promoters' gigs (“Presented by: Divine Schism”); SOFO runs a concert for a church. One fixed organiser per source gets these wrong.
What do we store for a website?
- The domain
- Domain + path (a college vs its quad)
Is organiser matching its own step?
- Its own step
- One clue inside place and duplicate matching
Later — 1 small thing
- An organiser-kind field · look up charity numbers now or later · when “site owner” becomes sure · who keeps the denylist, and which pub-chain domains go on it.
Duplicates
A duplicate becomes a second source on the same event — never deleted. Missing one is safe; a wrong merge makes an event vanish.
- Tidy: trim titles, drop “SOLD OUT”, London time, clean links. “Part 1” kept as its own field.
- Candidates: same London day. Never two items from one scraper in the same run.
- Exact rules: same link + start + title → same. Same place + start + title → same.
- Scoring (Splink) on title, time, venue, distance, organiser, link. Titles must be similar; starts within 30 min.
- Jev: “same, different, or same show at a different time?”
- Not sure → saved as new, shown on the site, flagged.
- Test: 94 website-checked pairs; 40 duplicates on 4 sites.
- GATE every PR that touches duplicate matching NOT BUILTThe 94 checked pairs + ~30 held back + planted traps: 0 wrong merges. Test passes; catches 0 of 8 unseen real duplicates; not automatic yet.
Do now
- Build it: tidy → seen it → place → duplicate → Jev → save.
- 0 wrong merges on those 94 pairs at every change — plus ~30 new pairs kept aside, never tuned on.
Open
3 open When two events merge on their own, when a person decides, and whether a livestream is the same event.
Can a high score merge events from two different websites by itself?
- Yes, automatically
- A person confirms, until that pair of sites has earned trust
Jev is unsure whether two events are the same. Does a person decide?
- Yes — it waits on the review list for a person
- No — keep them apart; nobody works the list
The aim is as little human work as possible; a person could keep up (~5 a day). Keeping apart costs a duplicate on the site.
An in-person talk and its livestream at the same time — one event or two?
- One event, two ways to attend
- Two events, one of them Online
Henley's Sebastian Faulks talk has a livestream twin. Jev's three answers have no “same show, different format”, so it may merge them.
How it runs
On fresh pairs never tuned on — cross-source overlaps plus planted traps — does tidy → rules → Splink → Jev keep 0 wrong merges, and how many real duplicates slip through?
Method
- Build ≥40 held-out pairs from P04's overlap clusters (Bullingdon / Glee / LiveNation / WeGotTickets, OAC / CyclAbility, Henley in-person / livestream); a person labels them from the pages; never used for tuning.
- Walk-through traps (Part 1 / Part 2, 1pm / 7pm, '- CANCELLED', livestream twin, fake 23:59) are planted by code as a regression set, reported separately, not as held-out.
- Add a 4th Jev option, 'same show, different format'; compare with the 3-option question.
- Shuffle the arrival order 20 times; outputs must be identical.
- Band sample: 50 future-dated pairs from the auto-merge band (pages still live), person-checked.
Measure and decision rule
Wrong merges (must be 0, with the upper bound), recall on real duplicates, output differences across shuffles (must be 0), and second-agent vs person agreement on the held-out pairs (for the weekly check). Decision on Q 7.76: three coarse classes — venue-own × ticket site, venue-own × listings, venue-own × venue-own. In each, score-band merges go to the review list until the class has 0 wrong in ≥150 confirmed pairs (upper bound ~2%); then it may auto-merge. The ongoing person minutes per week are estimated from P22's volume and shown to the founder. Exact-key merges (shared booking id or ticket URL) don't need the bound.
Stop rule
Stop at the first wrong merge on the new held-out pairs; report, don't re-tune on them.
How it runs
With the chosen matching, how many review items does a night produce on future-dated new output, and how many duplicates stay on the site if nobody works the list?
Method
- Run the chosen matchers (P17/P21) over the old fleet's --no-ingest output from P32's runs (future dates only), plus P04's 21–34 new builds if the extension ran.
- Run twice, the second time with remembered answers, for the repeat share.
- Count review items per kind and visible duplicates under 'keep apart'; fit the count against the number of overlapping source pairs, so it can be projected to 586.
Measure and decision rule
Items per night by kind, repeat share, duplicates shown if unattended, projected count at 586. Decision: ≤~5 a day projected → a person works the list; >20 a day → tighten the matchers or accept keep-apart with the duplicate count shown to the founder.
Stop rule
One run plus one replay.
Later — 1 small thing
- “Part 1” ≠ “Part 2” not built · how many confirms earn trust · the auto-merge band over live data was never sampled (763 event, 224 venue pairs) · extra clues only if a clean re-test misses: rare words, booking-link id, embeddings to find (never to decide).
Series
Four shapes: a list of dates · a rule (“every Tuesday”) · a run of shows · opening hours.
- A series is stored with its real dates as rows, plus an exceptions table. Opening hours stay a range.
- No invented dates: only what the page says, or what its stated rule clearly gives.
- Each night, the page's dates are compared with what's stored — only as far ahead as the page showed.
- A rule change affects only the future. One-off vs series is worked out fresh each night.
- GATE every PR that touches series NOT BUILTSaved pages from several nights in a row: dates that roll forward stay one series; a moved or cancelled date is caught.
Tested: a rule becomes dates only if it clearly repeats; 406 of 487 rule-like texts were left blank, never guessed.
Open
5 open What makes one event with many dates, how moves and rules work, and how exhibitions and festivals fit.
What makes one event with several dates, rather than several events?
- Same title at the same place — each date may carry its own line-up, link and price
- Anything that differs per date (line-up, booking link) makes its own event
One show at 1pm and 7pm; Glee's 49 Saturdays at The Bullingdon, each with its own acts and booking id; the Town Hall's monthly organ recital with a new organist. Until then the reader leaves several dates blank.
A date or time moves. What happens?
- A new event — cheap; 0 moves seen in 47 nights
- One update to a permanent date ID
The 47 nights covered only 6 scrapers, so read that as a direction, not a number. Wild Goose moved Macbeth from 18:30 to 17:45.
Who turns the page's words into a repeat rule, and picks list / rule / run?
- TypeScript, from the scraped shape
- The Python reader, before the hand-over
A weekly quiz must not merge into one row. Oxford Castle's ghost tour says both “1 Jan – 31 Dec 2026” and “Every Friday from 18:30” — a year-long run is the wrong shape.
Whose opening hours does an exhibition show — its own page's, or the building's?
- The place's hours, unless the page gives its own
- Only hours on the event's page
The Ashmolean's exhibition pages give no hours; they sit on its Visit page, which no scraper reads, and change by day — a series holds one open and close time.
How are festivals and umbrella programmes modelled, and how do they show?
- A festival page with its events under it — the layers fit 4 real festivals, even Open Doors four levels deep
- Plus a festival link on each date: Ghost Fest claims only two Fridays of a weekly ghost tour
No scraped item is the festival itself, so something must make that parent row. The Ashmolean's talks and tours belong to an exhibition; Oxford Castle's Ghost Fest is an umbrella.
How it runs
On real multi-date pages, how many events, cards and duplicates does each 'one event with several dates' option produce — as an evidence table for the founder, not a winner?
Method
- Apply both options to Glee Saturdays, the Town Hall organ series, a Playhouse double showing, Ashmolean Aphrodite, and CyclAbility.
- Count events, event pages, cards per day page, per-date fields that would be lost, and what a moved or cancelled date does under each.
Measure and decision rule
The table itself. No decision rule: it goes to the founder.
Stop rule
Five cases.
How it runs
Across every saved target page, how common is each hard shape that the open questions hinge on, weighted by clicks?
Method
- Take a stratified sample of ~300 sites (source kind × click band); fetch 1–2 detail pages per site alongside the bulk2 listing page, since doors, prices, members-only, 'presented by' and rooms live on detail pages.
- Ask one Jev yes/no question per shape per page (Jev has no pick-many): rule-only, multi-showing, run/exhibition, festival/umbrella, room named, event away from the site's venue, online, TBC, members-only, cancelled/postponed, doors-only, no-year dates, fee/concession prices, 'presented by' / several organisers, no per-event link.
- A person checks 40 pages (drawn from P37's labelled pages where possible) plus, for each rare shape, pages the tagger flagged, so every shape has positives to measure recall on.
- Weight by clicks (scripts/venue-clicks.py).
Measure and decision rule
Count and click-weighted share per shape, with an interval; per-shape recall on positives (not raw agreement, which rare shapes hide). Decision: a shape under ~1% click-weighted moves its Qs to LATER only if the pipeline's default output for that shape is blank, never a guess; a shape over 10% moves its Qs to the front and into P29's site list.
Stop rule
Stop if tagger agreement on a shape is under 80% — report that shape as unmeasured, not as a count.
How it runs
Replaying several nights of the same pages, does the nightly compare keep rolling dates as one series, catch moved or cancelled dates, and notice a rule-only event that stops?
Method
- Night sequences: real multi-night data (ng.wedge_raw_items via the read-only DSN, output/runs) plus synthetic nights made by code from today's pages: roll a date forward, drop one, move 18:30 → 17:45, cancel one date, delete a rule page.
- Run the TypeScript compare into P26's local database.
- From the real history, measure how often an old event changes, to check the 1/30 rotation's catch delay.
Measure and decision rule
Planted changes detected correctly (target 100%) and false changes (0) on the synthetic nights, which are the gate. The real history only estimates how often old events change, for the 1/30 rotation's catch delay. Decision: if a stopped rule goes unnoticed, add a 'page still states the rule' check; if old events change often, raise the rotation.
Stop rule
One replay per scenario; no calendar waiting.
Later — 1 small thing
- How far ahead to store (8 weeks to 12 months) · “term-time only” · window sizes and caps.
Saving events
Only events from scrapers whose verdict is OK are saved.
- Only changed fields are written. A bad run can be undone.
- Which source wins: the venue's own site — except a ticket site wins on price and sold-out.
- Admin edits are locked. A scraper never overwrites them.
- “Cancelled” → still shown, marked. “Sold out” is not a status.
- Gone from the page is never a delete. Future + gone → review. Past + gone N nights → soft-delete, only if the scraper is OK.
- Passed → a nightly job marks it by the clock.
- Event links use our permanent ID; the title part is decoration. A merged event's old link redirects.
- Safety stop: a run that would change far more than normal for a site stops — nothing saved, alert. What counts as “far more” is open below.
- New sites get a tight cap · an alarm when many sites break at once · override: one run, logged, operator only.
- GATE every run NOT BUILTSafety stops: a run that changes far more than normal for a site → nothing saved, alert.
- GATE every PR NOT BUILTUnit tests: which source wins each field, locks never overwritten, gone ≠ deleted.
Every merge and removal must reach anything else that keeps a list of live events.
Open
8 open Undoing mistakes, unclear places, cancellations, safety stops, and which source wins.
How do we undo a wrong merge?
- Re-run from the kept source rows
- Log an “unmerge”
An event with an unclear place?
- Publish it as “TBC”
- Hold it back
A series date that looks wrong?
- Hide it
- Show a warning
- Keep it off “what's on Saturday”
Do admin locks expire?
Keep every value every source gave, winners and losers?
- Yes — each source's last word, in event_sources.said
- Only the winning value
Undoing a wrong merge by re-running from kept source rows needs every value kept.
What trips the safety stop?
- Any time change, and any drop to zero
- Many time changes at once; a drop to zero only while stored dates are still ahead
With “any”, Wild Goose moving Macbeth 18:30 → 17:45 saves nothing for that site that night and the move never shows; Henley Literary Festival trips it every year when it ends.
“Cancelled” can sit on the event, on one date, or as a series exception — and the nightly “passed” flip overwrites it. One rule?
- Cancelled lives on the date; “passed” is a separate field, so both show
- One status per event; “passed” wins
Oxford Town Hall's cancelled organ recital would later read as a normal past concert, and SOFO's postponed concert as one that happened.
Where does the organiser's own site rank, and which source wins the title and time?
The Bullingdon says doors 7:00; Glee — organiser and ticket site — says 6:45. WeGotTickets' title names an act the venue page leaves out.
How it runs
On a throwaway local Postgres, do saving only changed fields, locks, 'gone is never a delete', merge→unmerge, cancelled vs passed, and undoing a bad run behave as the plan says?
Method
- Prototype the TypeScript save module against P26's schema in a throwaway local Postgres (PGlite or Docker) — never dev or prod Supabase.
- Scripted scenarios: two-source merge then unmerge (both Q options); locked title vs a scraper change; a cancelled date passes the 'passed' job; a bad run (all prices ×100) undone via event_changes; a launch rollback (3 bad nights, then restore to night-0); url_redirects after an unmerge.
- Run every undo scenario twice: with 8.6 = keep every source's last value (event_sources.said) and 8.6 = keep only the winning value. Record what each option can't restore, and whether the trust order (P36) can be re-evaluated later.
Measure and decision rule
Pass/fail per scenario; minutes to undo 1 run and 3 nights. Decision: the undo option that restores links, locks and redirects exactly wins; the status model that shows both 'cancelled' and 'passed' wins; the rollback time answers the switch Q. Q 8.6: what each option loses (merges that can't be undone, trust-order changes that can't be replayed) and its storage size — the evidence 8.6 needs.
Stop rule
All scenarios once; fix and re-run only the failures.
How it runs
Which safety-stop rule stops real disasters (a site dropping to zero, all times shifted) without stopping legitimate change (a moved show, a festival ending, a series rolling)?
Method
- Real break rate: packages/scraper/data/state.json git history (207 commits over 139 days; per-scraper consecutive failures and zero runs), direction only.
- Replay the available multi-night data (ng.* read-only, output/runs).
- Plant disasters by code: a login wall (0 events), a timezone bug moving every time +1 h, a scraper picking sidebar dates, half the fleet broken on one night, and a stalled loop (no fleet_snapshots row) that must raise an alert.
- Plant legitimate changes: festival end, one moved show, a weekly roll.
- Run both Q options and per-site thresholds; count trips.
- Arm 'far more than normal for a site' (the plan's current step): per-site normal from that site's own history; test k × normal thresholds.
- Cold-start arm: an empty database and new scrapers mean no site has a 'normal' at relight. Replay a new site's first nights under the tight cap (8.28) and measure after how many nights its 'normal' is stable enough to use.
- Flag for the plan: the Q's two options ('any time change…' / 'many time changes…') no longer match the step's 'far more than normal' wording.
Measure and decision rule
Disasters stopped (100%), legitimate changes stopped (target 0), stall alert fires, per arm (the Q's two options and 'far more than normal'); nights until a new site's normal is usable; the many-sites alarm threshold from the state.json break rate. Decision: the arm that meets both marks answers the Q; the cold-start result sets how long a new site stays under the tight cap.
Stop rule
One replay per rule.
Later — 1 small thing
- Jev changing its answer · misses before delete · the many-sites alarm threshold (15–20% is a guess) · do safety stops go on the review list · do locked fields count as changes · a daily cap on new cancellations.
Review list
One list: possible duplicates · unclear places · unclear organisers · safety stops.
- Repeat questions are answered from remembered answers; only new or tricky items reach the list.
- A possible duplicate still shows on the site while it waits. Every decision is logged and can be undone.
- One item at a time, options side by side, one tap — reuses the admin list page.
- Expected size.
- CHECK weekly NOT BUILTA sample of the AI's own decisions is checked by a person or a second agent; a drop in agreement raises an alert.
Open
Later — 1 small thing
- Count new vs repeat questions per night once it runs.
Frontend (the website)
The site we have is reused. Its queries move onto Drizzle.
- Either way nothing is copied into a second store, so nothing can drift out of step.
- What it reads today: day / week / tonight / town pages → events, venues, addresses, towns · event page → also organisations (must exist, or the page 404s) · category pages → one of 16 fixed categories · venue pages and map → venues, addresses, organisations (a venue's town comes from its organiser) · search → events joined to venues (fuzzy title match) · sitemap and newsletter → events joined to venues.
- What it writes: the admin Remove button · public submissions (stored, never turned into events) · newsletter sign-ups · accounts · contact messages.
- Links today: events
/events/{title}-{date}(the slug is the only key) · venues/venues/{slug}· towns/{town}· categories/{town}/events/{category}· no organiser pages. - New event links:
/events/{id}-{title}. The ID finds the event; an out-of-date title part redirects. - New things the pages must show: a cancelled or postponed event stays, with a CANCELLED or POSTPONED badge · a date with no time never shows 00:00 · “Online”, “Venue TBC” and “No venue” as real states, not a pin in the town centre · “Free”, “From £12.50”, family-friendly and members-only as labels and filters.
- Admin pages: the review list screen — one item, options side by side, one tap · Remove and edits become locks, shown as locked · a page per scraper showing its verdict, last run and why it failed.
- GATE every PR, before deploy Playwright tests on every public page.
- GATE every PR NOT BUILTEnd to end: saved pages → the whole pipeline → an empty database → pages shown in a browser, with the expected events on them.
Two ways to move the pages: rewrite each query onto the new tables, or first add views named like today's tables (events, venues, addresses, organisations) that show the new data in the old shape — about 45 of the 49 queries would keep working unchanged.
Do now
- Fix the queries that break: search, sitemap and newsletter drop events with no venue · event pages 404 with no organiser · cancelled events are hidden (show them, marked) · the Remove button must lock the event, not edit it · a venue's town comes from the place.
Open
4 open Old links, how repeating events and exhibitions show, organiser pages, and members-only events.
Load the list of old links once, so they redirect?
- Yes — past events get the “gone” page, clear matches get a redirect
- No — every old link breaks
Almost every old event link and most old venue links break otherwise, including the ones Google sends people to (592 of 602 old “venues” are really organiser names, so only name matches can redirect). Town links keep working. It's only the addresses, not the old data.
In day and week lists, does a repeating event get a card on each of its dates? And a long exhibition?
- A card on each date; a long run once, as “on now”
- A card on each date and each open day
An event has one page. Cards per day once filled the old site with one exhibition day after day; showing it once hides it from later days (the Ashmolean's Aphrodite runs 8 Oct – 11 Apr).
Do organisers get their own pages now that they're a real thing?
- Yes — an organiser page lists everything they run
- Not yet
Members-only events — show them with a badge, or leave them off public lists?
- Show, with a badge
- Leave them off
ANHSO's Zoom talks are for members only. Doors-only times and “open until 1 Nov” runs have no display rule yet either.
How it runs
How much search and landing traffic do old event, venue, category and per-date links carry, and how many old venue slugs can be name-matched to new places?
Method
- GSC (docs/gsc.md) clicks and impressions by URL pattern for 2026-05-18 → 2026-08-17 — the site has served a 503 since 2026-08-18, so a window ending today would mostly show the dark period. PostHog landing pages for the same window, filtered on $host (docs/posthog.md).
- Note as a time-sensitive fact, not a proof unit: the value of redirects decays while Google keeps seeing 503s.
- After P17: match old venue slugs to new places by code (slug-token overlap + the same Google place); a person checks only the unsure ones.
- Click share of per-date and per-day exhibition cards vs event pages; read memory growth-and-business-verdicts before judging organiser pages.
Measure and decision rule
Click share by pattern; correct name-match rate. Decision: if old links carry more than ~5% of clicks, or GSC shows indexed event URLs still getting clicks, load the redirects; organiser pages only if venue pages earned traffic.
Stop rule
One pull.
How it runs
Do saved pages from a handful of messy sites flow through scraper → reader → checker → hand-over → matching → save → empty database → pages, and show the expected events in a browser?
Method
- 5 then 8 sites: Gardeners Arms, Bullingdon, Town Hall, Ashmolean, OAC, then SOFO, Henley Literary Festival, ANHSO.
- An agent drafts the expected site output; a person corrects and signs it off against the pages (~1–1.5 h). The walk-through trace is not the answer key.
- Extend P40's harness: run everything offline from saved pages into a fresh local DB and the local web app.
- Playwright asserts the cards: cancelled badge, no 00:00, TBC/online states, Free label, exhibition shown once, no duplicate.
Measure and decision rule
Expected events present and correct per page; failures by stage. Decision: every failing stage gets a factory issue; relight readiness can't be claimed until 8 of 8 pass.
Stop rule
5 sites first; don't add the next 3 while any of the 5 fails.
How it runs
Does a thin end-to-end path — two sites, a stub matcher — expose join bugs (dropped no-venue events, 404s with no organiser, hidden cancellations) before the big parts are built?
Method
- Gardeners Arms (rule-only, no id) and Oxford Town Hall (cancelled, room, detail pages) from saved pages.
- It builds its own two copy-only scrapers (no P04 dependency) and uses today's interpret.py (before P38/P08 settle the reader) → P15 shape → a stub save (exact-key upsert, no matching) into P26's local schema → the local web app.
- Playwright checks the two sites' cards against expected output that a person has signed off.
Measure and decision rule
Stage-by-stage pass/fail; the list of join failures. Decision: every join failure becomes a fix before P25/P27 are built.
Stop rule
Two sites only.
Later — 1 small thing
- People's event submissions are stored but nothing ever turns one into an event ·
museums-oxford.tshard-codes old venue links · organiser pages don't exist yet · festivals have no page of their own · category links break unless today's 16 stay or each old one redirects.
Site queries on Drizzle
Every query the site runs moves onto Drizzle, so its types come from the schema — not from a label we promise is true.
- Today: 48 queries in 12 files are hand-written SQL with a type label (
sql<TonightEvent[]>). Nothing checks the label against the database, so a wrong column shows blanks and nothing fails. - After: queries use Drizzle's query builder on the new schema. Rename or drop a column and
pnpm typecheckfails before anything ships. - The same schema file feeds the hand-over shape, the tables and the site — one source of truth from scraper to page.
- Most queries live in
packages/web/src/lib/queries/(events, venues, search, towns, metrics); the rest are the sitemap, newsletter, admin pages and one description API. - Things raw SQL does that need care in Drizzle: fuzzy search (the pg_trgm
%match) · London-time date maths · the “tonight / this weekend” windows. These use small typedsqlpieces inside the builder, with the result type still coming from the schema. - GATE every PR Typecheck — the types come from the schema.
- GATE every PR that touches a query NOT BUILTOld and new queries return the same rows on the same data.
Do now
- List every query and the page that uses it; write each one's Drizzle version against the new schema, with a test comparing old and new results on the same data.
Open
2 open Rewrite everything first or step by step, and whether admin moves too.
Rewrite every query before the site goes back on, or add the old-shape views first and move page by page?
- Rewrite everything first — no views
- Views first, then page by page
Views are a second thing to keep in step; rewriting first is more work before the site comes back.
Does the admin area move too, or only the public pages?
How it runs
Can the site's queries run unchanged on old-shape views over the new schema, and how does that compare with a Drizzle rewrite on results and speed?
Method
- Inventory every sql<…> query (62 hits under packages/web/src) with its page, including admin.
- Generate new-schema rows from P04 output, scaled to the old site's volume (~35k events, volume target only).
- For day / week / tonight / search: write the old-shape view and a rewritten Drizzle query; check they return the same rows under a written old→new column mapping; EXPLAIN ANALYZE both.
Measure and decision rule
p95 latency of the view vs the Drizzle query on the big four; rows equal under the mapping; the query count by page. Decision: the drizzle section already prefers the typed rewrite, so this only decides whether views are fast enough as a temporary bridge (within 1.5× of the rewrite) — the inventory sizes the rewrite work for the founder.
Stop rule
The four biggest queries only.
Later — 1 small thing
- A lint rule that bans new raw
sql<…>queries in the web app, so it can't drift back.
The nightly loop
- Run every live scraper and read the text.
- Check new and changed events, plus a rotating 1/30th of old ones.
- Decide: OK / QUIET → keep · FAILING 2 nights or BROKEN → rebuild.
- Save the events of scrapers that passed.
- Rebuild: the queue is a database query, not a file. Max 10 a night.
- Summary: live · failing · broken · quiet · parked · can't check.
- Every week: check Jev against screenshots · re-sniff parked sites for a new feed.
- CHECK nightly NOT BUILTThe fleet summary is compared with last night; a jump in failing or broken scrapers raises an alert.
- CHECK weekly NOT BUILTJev is checked against screenshots.
Open
2 open Where it runs, and who owns it when it stalls.
Where does it run?
Nightly GitHub jobs are paused, the Jev key must live there, and politeness limits aren't planned.
Who owns the loop when it stalls?
It has stopped before and looked finished.
How it runs
What do a nightly run of the whole fleet and a first build of every scraper cost in time, money and rate limits, and where can the nightly run live?
Method
- Fetch + snapshot time per site from P32's old-scraper runs (up to 89 sites, incl. detail pages).
- Reader + checker + Jev time and $ per item from P04's 13 builds.
- Combine per item to 586 sites, with an interval.
- Ask TypeSafe for rate limits and quotas in writing; no load test against their API.
- robots.txt census over urls.txt, and the fetch count per night including detail pages; propose a per-host rate.
- If a GitHub Actions timing test is wanted, a separate workflow_dispatch that writes nothing (the nightly crons stay disabled).
- Build cost: P04's $/build (a ceiling) × 586, plus rebuild churn from the state.json break rate (P24/P32) against the 10-a-night cap.
Measure and decision rule
Hours and $ per night, $ for the first build, rebuild queue vs cap — each as an interval; robots disallows among targets; fetches per night. Decision: under ~3 h on GitHub Actions with headroom → GitHub Actions, else a box. The cost interval goes to the founder.
Stop rule
One pass.
Later — 2 small things
- CAN'T CHECK → keep yesterday's, retry, alert if it lasts.
- Audit tables, all add-only: builds · runs · jev_answers · evidence · scraper_status · match_decisions · event_changes · run_stats · review_items — some overlap (venue aliases vs venue names, match decisions vs review decisions, Jev answers logged twice), and how long evidence is kept is open.
The quality bar
How we know the site is good enough to turn back on.
- An event is wrong if any of 8 things is wrong: really on (not past, not a hidden cancellation) · date · start time · place · price · title · link · shown once. Blank is never wrong; a guess always is.
- Code checks every event for known bad shapes — end before start, past shown as upcoming, the same event twice, junk titles, one exhibition as a card per day. Target: 0.
- A sample of 400 is checked against the venues' real pages: 200 from the 20 most-clicked venues (70% of clicks, Oxford Castle alone 29%), 200 random with at most 2 per venue — one bad scraper breaks all its events at once. The halves are scored separately.
- At the top 20 venues, also count missed events — ones on their page we don't have — so the system can't look good by listing less.
- A Claude agent writes down each event's facts from the page before it sees ours, quoting the page word for word. Jev can't grade here — it's part of what's being tested.
- Keeping the checker honest: a person checks 60 real events by hand once; code (never an agent) plants errors in copies — day +1, hour +1, swapped place, wrong price — and the checker must catch 95% before a run counts. How much a person looks at is set once the proof-of-concept runs show how often the checker is wrong.
- If a page changed since we scraped it, compare with the page as it is now — that's what a visitor sees.
- Also counted, but not making an event wrong: organiser, end time, labels, description, image — and blanks, so “blank beats wrong” can't quietly empty the site.
- Goal: as few wrong as possible. Every code check at 0 and no guessed values are hard rules. Each run reports how many of the 400 are wrong, and what that proves.
- After launch: 100 events a week (50 by clicks, 50 random) plus the code checks. A venue with 2 wrong in one run is taken off. 11+ wrong in 100, or any code check above 0 two runs running → the site goes dark again.
- GATE before relight, then weekly NOT BUILTThe 400-event sample against real pages: code checks at 0, no guesses, wrong count and what it proves reported. Test built; missed its bar.
Tested: the sample grader caught 91 of 96 planted mistakes, just under 95%. After fixes, no right event marked wrong — on tuning data only.
Do now
- Design proof-of-concept runs of the checker first. They measure how often it is wrong, which sets how much human checking is needed.
Open
2 open How blanks and repeating events are scored.
Are blank start times always fine?
- Yes
- Fewer than 1 in 5 at the top 20 venues
How is a repeating event or a long run scored — once, or per date?
- Once: the rule or range must match the page
- Each shown date is its own check
The Gardeners Arms page says only “every Sunday at 8.30”; the sample picks events, but the site shows dates.
How it runs
Superseded: the rebuilt reader uses no Jev. On a test set built from real messy pages, not made-up strings, does the reader (exact ISO + the date library + Jev picking the start at 0.8) give zero wrong dates and times, and how many blanks?
Method
- Library route fixed by P38 before anything runs.
- Truth: P37's date, start, end and time-kind labels. Dev pages (walk-through included) for fixes; held-out pages for the reported number.
- Code plants on held-out copies: sidebar times, header opening hours, doors times, swapped am/pm.
- Arms: reader alone; reader + Claude on unsure cases.
- Plot accuracy by Jev probability band for the start-time question (the full calibration is P31); pick the threshold on dev.
- After any generic fix, only fresh held-out pages count (P37's sealed reserve).
- Report the blank-start rate at the top-20 venues separately.
Measure and decision rule
Wrong count on held-out with its 95% upper bound (working slice 12 pages; reserve 8, a few hundred date/time fields); blank rate overall and at the top-20 venues; share sent to Claude; accuracy by probability band. Decision: 0 wrong → this set becomes the reader gate's fixture. Any wrong → a generic fix, then re-score on fresh pages only. A top-20 blank-start rate over 20% goes to the founder with the numbers. Thresholds chosen here are provisional until P31's calibration.
Stop rule
Don't grow past the working slice while any item is wrong; a fix is re-scored only on P37's sealed reserve.
How it runs
How often does the relight checker (a Claude agent that writes the page's facts before it sees ours) pass a wrong event or fail a right one — especially when the wrong value was lifted from elsewhere on the same page?
Method
- Subjects and truth come from the gold set (P37): its person-read held-out pages are the only scored pages, and its dev pages (all walk-through pages included) are the only place prompts are tuned. Base records are built from the person's values, so every plant has a known answer. Where old prod rows are used as extra unplanted subjects, their 'natural error' agreement is reported as direction only — old errors don't predict new-system errors.
- Code plants errors from a fixed seed and list written before the run, in two families, reported separately. (1) Same-page substitutions: code picks a date, time, price or place string that exists elsewhere on the same snapshot (sidebar date, related product's £0.00, voucher price, header opening hours, a past exhibition's range, 'doors' for start) and swaps it into our record. (2) Random lies: day +1, hour +1, swapped place, wrong price, past shown as upcoming, a blank replaced by a guess. Keep unplanted copies as controls.
- Changed-page arm: code plants an edit into a copy of 10 page snapshots (date moved, 'CANCELLED' added) and the checker compares with the edited page, as the plan's 'compare with the page as it is now' rule requires. No calendar-time re-snapshot is needed.
- Missed-event count: on 8 top-20 venue listing pages, the checker extracts the full event list independently; a person counts the events on those 8 pages once; code deletes events from our list as plants.
- 'Shown once' is not judged per page; it moves to P35's code check.
- Run the checker twice to measure run-to-run disagreement.
- Report blanks per field (start, place, price), so 'blank beats wrong' can't quietly empty the site.
Measure and decision rule
For each plant family and field: catches and the cluster-aware 95% upper bound on the miss rate (working slice during development; the lockbox, opened once, for the number the founder sees); false 'wrong' on clean controls; missed-event count error vs the person; run-to-run disagreement; blank rate per field. Bounds are stated up front: 0 misses on 25 lockbox pages bounds the miss rate at ~12%, so ≤5% can't be shown without ~60 lockbox pages. Decision bands (evidence for the founder's human-checking call, not the call itself): upper bound ≤12% on both families → a person adjudicates only the checker's 'wrong' calls plus a fixed audit sample; ≤25% → a person also re-checks 'right' calls in the family that misses, at the top-20 half; >25% on same-page substitutions → the agent checker can't gate relight for that field until P03's attribution step is in it. If the founder wants the ≤5% band, the lockbox grows to ~60 pages first.
Stop rule
Develop on the working slice (grow 4 → 8 → 12 pages); stop and report if same-page substitution misses exceed 30% there. Open the lockbox once. Never re-tune after opening it.
How it runs
Do the pass marks (12 wrong in 400 before relight; 11+ in 100 weekly) still prove what they claim when errors cluster by venue and field, and when the checker itself misses some errors?
Method
- Sweep, today: per-venue error distributions (from none to one bad venue carrying all errors), top-20 click shares, checker miss rates 0–15%, the plan's 200 + 200 sampling design and alternatives (per-venue caps, larger samples).
- Compute the real confidence behind '≥95% right' and the false 'go dark' rate of the weekly rule for each setting.
- Re-run once when P01 gives a measured miss rate.
Measure and decision rule
Actual confidence of each rule across the sweep; the designs that stay ≥95%/95% across it. Decision: pick the sample design that is robust across the sweep before P01 fixes the relight sample; if none is, cap events per venue in the top-20 half or raise the sample.
Stop rule
One simulation run.
How it runs
How often are titles decorated or junk and links fake or not per-event, and do a generic title tidy and a link-validity rule fix them without harming good ones?
Method
- On P04 output and bulk2 items, count by code: links that don't resolve, are 'javascript:'/'#', are identical across items, or equal the listing URL; titles with status/age/price decorations or dates embedded.
- Plant by code: swap in the listing URL, a javascript: link, the same link on every item; add decorations to clean titles.
- Test a generic link rule (resolves, differs per event, not the listing URL — else blank and link_url falls back to the listing page, marked as such) and a generic title tidy whose status words come from P13's status question, not a word list.
- Score the tidy against P37's clean-title label (and the planted decorations), the link rule against P37's link label. Link resolution needs live HTTP; results from saved pages may be stale and are marked so.
Measure and decision rule
Junk rate per field, click-weighted; planted catch rate (target 100%); clean titles the tidy changes wrongly and good links rejected (must be 0). Decision: a rule with 0 harm goes into the checker and the hand-over; otherwise titles are kept as written and decorations become separate fields.
Stop rule
One pass per rule on dev, one on held-out.
Later — 2 small things
- How much human checking, and who does it — decided after the proof-of-concept runs. A second AI alone is ruled out: nothing would check the checker.
- Cost per full run: agents ~$20–60 and 1–2 hours with 5 at once · a person 1–2 hours · weekly: ~$10 and 30 minutes · the hand-checked set: ~3 hours once. Pass-mark maths: to be 95% sure of at least 95% right, the most wrong allowed is 4 in 200, 8 in 300, 12 in 400, 20 in 600.
Switching over
Start again from an empty database. Old data gives ideas only, never numbers.
- Nothing is carried over from the old database.
- The site stays dark until the new system is ready — every scraper rebuilt and the quality check passed. Today's system is switched off, never run alongside, so only one thing ever writes live events.
Open
3 open Nightly bad-data checks, the total cost, and how to go back after launch.
The known-bad-data checks — exhibitions re-listed daily, repeats with no end, end before start — are they a gate on every nightly save?
- Yes, every night
- Only in the weekly quality run
What does the whole thing cost — a first build of every scraper by a Claude agent, and a month of nightly runs?
Estimated: first build ~$200–300 if every site is as easy as the 9 tried, more likely $300–800; nightly runs ~$5–60 a month.
If the new system goes wrong after the site is back on, how do we go back?
Safety stops cover one bad night, not a bad launch.
How it runs
Do the quality code checks run at 0 false alarms on correct output and catch planted bad shapes, so they can gate every nightly save?
Method
- Write the checks as SQL over P26's local DB: end before start, past as upcoming, the same event shown twice ('shown once', moved here from P01), junk titles, an exhibition as a card per day.
- Run them on P04's person-checked correct output.
- Plant bad shapes by code.
- Every alarm on rows the person hasn't checked goes to a person (or a planted-truth check) before it is counted as a false alarm.
Measure and decision rule
False alarms (must be 0 to gate nightly); catches (100%). Decision: 0 false alarms on 13 sites → gate every night; otherwise weekly only.
Stop rule
One pass.
Later — 2 small things
- Sites open, move and die after the first build; the fleet is never finished.
- Quick measurements: how often sites really break · false alarms a night · how common “every Wednesday” is.
Tests and gates
Every test in one list, collected from the steps above. GATE blocks — the change doesn't ship, or the run doesn't save. CHECK warns. Every gate needs a known-good set, errors planted by code, and data it was never tuned on. No agent grades its own work.
| Step | Kind | When | What it proves | Built |
|---|---|---|---|---|
| 02 Finding sources | GATE | every sniff | A source is used only if it passes all 4 tests; the answers are logged. | not built |
| 03 Building a scraper | GATE | every build | Lint 0 — the scraper only copies text. | not built |
| 03 Building a scraper | GATE | every PR that touches the lint | Unit tests: every banned trick is caught; a known-clean scraper passes. Test version catches 80 of 80 planted, 51 of 52 unseen; not yet passing. | not built |
| 04 Dates and times | GATE | every PR that touches the reader | The reader passes a test set built from real pages — 0 wrong; blanks allowed. It includes doors-only times, feed midnights, rules with no dates and listings with no year. Plus the made-up hard strings. | not built |
| 04 Dates and times | CHECK | every PR | Unit tests for London time, 24-hour feed times, year rollover and date ranges. | not built |
| 05 Prices | GATE | every PR that touches prices | The price test set from real pages — 0 wrong; “free”, “from”, booking-fee, concession, member-free and gift-voucher cases included. | not built |
| 06 Labels and categories | GATE | before any label goes on the site | Each label passes a test set checked by a person, never AI-written answers. | not built |
| 06 Labels and categories | GATE | whenever Jev's model changes | Re-ask a saved set of Jev questions; any answer that flips blocks the new model until checked. | not built |
| 07 Checking a scraper | GATE | every PR that touches the checker | Code plants errors in saved pages — a date +1 day, a sidebar date, a dropped event; all must be caught. Test built; checker fails it. | not built |
| 07 Checking a scraper | GATE | every PR that touches a scraper | The scraper runs on its saved pages offline and must match its expected output. | not built |
| 08 Keeping a scraper | GATE | every build | Go live only if lint 0, every copied text is on the page, nothing missed, nothing dropped. | not built |
| 09 Python → TypeScript → Drizzle | GATE | every PR | The Python copy of the shape matches the Zod shape — CI fails if not. | not built |
| 09 Python → TypeScript → Drizzle | GATE | every run | Every item is checked against the shape; a bad one goes to rejected_items with the reason. | not built |
| 10 Database schema | GATE | every PR | Migrations run on an empty database, and back. | not built |
| 10 Database schema | GATE | every PR | The schema files match the live database (the existing drift check). | built |
| 11 Places | GATE | every PR that touches place matching | Checked place pairs + a held-back set never tuned on: 0 wrong merges. Planted look-alikes (two St Mary's) must stay apart. | not built |
| 12 Organisers | GATE | every PR that touches organiser matching | Checked organiser pairs, including shared-domain traps: 0 wrong merges. | not built |
| 13 Duplicates | GATE | every PR that touches duplicate matching | The 94 checked pairs + ~30 held back + planted traps: 0 wrong merges. Test passes; catches 0 of 8 unseen real duplicates; not automatic yet. | not built |
| 14 Series | GATE | every PR that touches series | Saved pages from several nights in a row: dates that roll forward stay one series; a moved or cancelled date is caught. | not built |
| 15 Saving events | GATE | every run | Safety stops: a run that changes far more than normal for a site → nothing saved, alert. | not built |
| 15 Saving events | GATE | every PR | Unit tests: which source wins each field, locks never overwritten, gone ≠ deleted. | not built |
| 16 Review list | CHECK | weekly | A sample of the AI's own decisions is checked by a person or a second agent; a drop in agreement raises an alert. | not built |
| 17 Frontend (the website) | GATE | every PR, before deploy | Playwright tests on every public page. | built |
| 17 Frontend (the website) | GATE | every PR | End to end: saved pages → the whole pipeline → an empty database → pages shown in a browser, with the expected events on them. | not built |
| 18 Site queries on Drizzle | GATE | every PR | Typecheck — the types come from the schema. | built |
| 18 Site queries on Drizzle | GATE | every PR that touches a query | Old and new queries return the same rows on the same data. | not built |
| 19 The nightly loop | CHECK | nightly | The fleet summary is compared with last night; a jump in failing or broken scrapers raises an alert. | not built |
| 19 The nightly loop | CHECK | weekly | Jev is checked against screenshots. | not built |
| 20 The quality bar | GATE | before relight, then weekly | The 400-event sample against real pages: code checks at 0, no guesses, wrong count and what it proves reported. Test built; missed its bar. | not built |
POC runs
40 small experiments, each settling or testing items above, in run order. They need about 29.25 hours of a person's time in all. Batch 0 runs first; at most five agents run at once. Each one has a known-good set, errors planted by code, and data it was never tuned on. No agent grades its own work.
Your calls these feed
- What happens to sites that can't pass as copy-only scrapers, and does a parked site block relight? — evidence from P04, P10, P32, P30
- Hand-written scrapers for the classes that fail, checked like any other (cost per class from P04)
- Drop sites that can't pass
- How a parked site counts: blocks relight / counts as rebuilt with a recorded reason / counts only while parked sites stay under an agreed click share
- Who labels the POC gold set (P37), needed first — about 12–13 person-hours. — evidence from P37
- Arjun
- A paid checker
- Split between them
- How much human checking the relight and weekly quality runs need. — evidence from P01, P34, P37
- Which price is 'the price' (cheapest, standard adult, fee in or out). — evidence from P12, P36
- One event with several dates vs several events; cards per date vs once for long runs. — evidence from P42, P28
- Blank rate the site can live with (start time, price, place). — evidence from P08, P01, P12
- No cap: blank always beats wrong
- Cap blank start times at 1 in 5 at the top-20 venues (cost: the fixes or hand checks P08 says that needs)
- Cap per field (start, price, place) at the rates P08/P12/P01 measure plus a margin
| Batch | POC | What it settles | Value | Effort | Person-hours | Depends on |
|---|---|---|---|---|---|---|
| 0 | P38 Date library dependency spike (Dates and times) | Dates in sentences → Duckling, a maintained… | ●●●●○ | S | 0 | — |
| 0 | P34 Pass-mark maths under clustering (The quality bar) | A sample of 400 is checked against the venues'… · Goal: as few wrong as possible. Every code… +2 more | ●●●○○ | S | 0 | — |
| 0 | P02 Scraper checker planted-error test (Checking a scraper) | Counting assumes one dated item per block. A… | ●●●●● | M | 0 | — (partly P04, P41) |
| 0 | P04 Copy-only builds on messy sites (Building a scraper) | Only 9 sites tried. In the long tail, dates sit… · Who may allow an exception to these rules? +2 more | ●●●●● | L | 2 | P02 (partly P37) |
| 0 | P15 Hand-over shape round trip (Python → TypeScript → Drizzle) | The shape isn't written yet. What must it… · Past events are copied too (Tap Social: 837… | ●●●●○ | M | 0 | — |
| 1 | P37 Shared person-labelled gold page set (Dates and times) | The reader passes a test set built from real… · The price test set from real pages — 0 wrong… +2 more | ●●●●● | M | 12.75 | — (partly P04) |
| 1 | P41 Segmenting pages into event blocks (Checking a scraper) | segment.py cuts the page into one block per… | ●●●○○ | S | 0.5 | — |
| 1 | P05 Lint catches unseen tricks (Building a scraper) | A lint checks the code for banned tricks. Fail… · Lint 0 — the scraper only copies text. +3 more | ●●●○○ | S–M | 0 | — (partly P04) |
| 1 | P28 Old links and page traffic (Frontend (the website)) | Load the list of old links once, so they… · In day and week lists, does a repeating event… +2 more | ●●●○○ | S | 0.25 | — (partly P17) |
| 1 | P07 Feed fake-midnight times (Dates and times) | A feed says 00:00–23:59, or 23:59, when it… · Copies can stay apart and show twice: a feed's… +1 more | ●●●○○ | S | 0.25 | — |
| 1 | P32 Fleet liveness map (Keeping a scraper) | Parked sites pile up with no owner. What… · The keep rule needs one future event. What… | ●●●○○ | S | 0 | — |
| 1 | P24 Safety stop replay (Saving events) | What trips the safety stop? · Safety-stop numbers come from 6 sites only… +1 more | ●●●●○ | M | 0 | — |
| 2 | P26 Schema migrations and constraints (Python → TypeScript → Drizzle) | Does Python ever write to the database — its… · Five things write live events today. Leaving a… | ●●●●○ | M | 0 | P15 |
| 2 | P40 Walking skeleton on two sites (Frontend (the website)) | End to end: saved pages → the whole pipeline →… · Fix the queries that break: search, sitemap and… +1 more | ●●●●○ | M | 0.25 | P15, P26 |
| 2 | P01 Quality checker honesty test (The quality bar) | How is a repeating event or a long run scored… | ●●●●● | M | 0.75 | P37 |
| 2 | P03 This event's value, not the sidebar's (Checking a scraper) | The check proves the text is on the page, not… · A zero is not “free”: LiveNation's feed gives a… | ●●●●○ | M | 0 | P37, P41 |
| 2 | P08 Real-page date test set (The quality bar) | The latest reader has read only 2 new sites… · Are blank start times always fine? | ●●●●● | L | 0 | P37, P38, P03 |
| 2 | P12 Prices on real pages (Prices) | Without Jev, fewer prices are filled: on two… · Several real prices — which is the price? | ●●●●○ | M | 0 | P37, P38, P03 |
| 2 | P31 Jev calibration, drift and noise (Dates and times) | Re-ask a saved set of Jev questions; any answer… · Make the test set a gate on every reader change… +4 more | ●●●●○ | S–M | 0 | P37 (partly P21) |
| 2 | P10 Page-shape census across fleet (Series) | How are festivals and umbrella programmes… · Members-only events — show them with a badge… | ●●●●○ | M | 1 | — |
| 3 | P09 Doors, am/pm and missing years (Dates and times) | A page only says “doors 7pm” — what is the… · A time with no am or pm — a feed's “7:30”, a… +1 more | ●●●●○ | M | 0 | P08, P37 |
| 3 | P11 Rules and runs from page words (Dates and times) | A page gives only a rule — “Every Sunday night… · Who turns the page's words into a repeat rule… +1 more | ●●●●● | M | 0.75 | P08, P10 |
| 3 | P42 One event or many — evidence (Series) | What makes one event with several dates, rather… · In day and week lists, does a repeating event… | ●●●○○ | S | 0 | P11 |
| 3 | P13 Labels, public-event and cancelled questions (Labels and categories) | No label has been tested yet. · Is this a public event at all? Nothing asks… +1 more | ●●●●○ | M | 0.25 | P37 |
| 3 | P39 Title and link reader check (The quality bar) | An event is wrong if any of 8 things is wrong… · Titles, descriptions and images have no reader… +1 more | ●●●●○ | S | 0 | P04, P37, P13 |
| 3 | P16 Same item across nights (Python → TypeScript → Drizzle) | A page gives no event id and no link of its… · A date or time moves. What happens? | ●●●●○ | M | 0 | P04 |
| 3 | P06 Source sniff four tests (Finding sources) | Test 1 — same events? Jev asks per feed item… · Test 2 — same count? Far fewer than the page… +5 more | ●●●●○ | M | 0.5 | — |
| 3 | P19 Source setup dry run (Finding sources) | Do we label each source by kind? · Who adds a site — sets up its place and… +2 more | ●●●●○ | M | 1 | P10 |
| 3 | P17 Place and organiser matching bake-off (Places) | How do two names become one place? · Google IDs change, and some point at the wrong… +3 more | ●●●●○ | L | 2 | P04, P19 |
| 3 | P18 Rooms, online and TBC places (Places) | What counts as one place? · An event with an unclear place? | ●●●○○ | S | 0.5 | P10 (partly P26) |
| 3 | P21 Duplicate matching on held-out pairs (Duplicates) | The rules were tuned on the same 94 pairs… · The order items arrive in decides the winner… +2 more | ●●●●● | L | 2 | P04, P16 |
| 4 | P23 Series nightly replay (Series) | A page with only a rule shows no dates ahead… | ●●●●○ | M | 0 | P11, P16, P26 |
| 4 | P25 Save, undo and cancel rules (Saving events) | How do we undo a wrong merge? · “Cancelled” can sit on the event, on one date… +2 more | ●●●●○ | M | 0 | P26 |
| 4 | P27 Old-shape views vs Drizzle rewrite (Site queries on Drizzle) | Rewrite every query before the site goes back… · The event queries are the biggest and most… +2 more | ●●●●○ | S–M | 0 | P26, P04 |
| 4 | P22 Review list size on future dates (Duplicates) | That size was measured on past rows; future… · If nobody works the list, every unsure pair… +1 more | ●●●○○ | M | 0 | P21, P17, P32 |
| 4 | P30 Fleet run time and cost (The nightly loop) | At ~586 sites: rate limits, and a Jev backlog… · Where does it run? +1 more | ●●●●○ | M | 0 | P04, P02, P32, P24 |
| 4 | P35 Bad-shape code checks (Switching over) | The known-bad-data checks — exhibitions… | ●●○○○ | S | 0.25 | P26, P04 |
| 4 | P36 Second sources vs venue pages (Finding sources) | Do we scrape ticket sites (Glee, WeGotTickets)… · Where does the organiser's own site rank, and… +2 more | ●●●○○ | M | 1.5 | P10 |
| 5 | P29 End-to-end on saved pages (Frontend (the website)) | End to end: saved pages → the whole pipeline →… · The scraper runs on its saved pages offline and… +4 more | ●●●●● | L | 1.25 | P40, P04, P08, P11, P15, P21, P25, P26, P27, P39 |
| 5 | P14 Categories bake-off (Labels and categories) | Categories were left open on purpose. What… | ●●○○○ | M | 1.5 | P28 |