Findings
One line per experiment, from FINDINGS.md.
| experiment | what we tested | result | what it means | evidence |
|---|---|---|---|---|
| F1-finding | Is "we find 1 in 10 events because of splitting" true? Tested dates vs links, a splitter, and the whole finder on top-20 and 20 fresh sites | The 1-in-10 does not hold (T5 missed 224/266 rows but 171 were 2 sites). 287 sites show upcoming dates; on 62 the crawler found 0 event links. Splitter v1 43/55 found, 6 fake; v2 (code cuts, Jev reads) 52/55, 6 fake. Whole finder on top 20: 62% (236/382); after 3 fixes 84% (tuned on same sites). On 20 fresh sites: 57% old finder, 61% fixed | Splitting is a real gap on about 60-90 sites, not the main leak. Hand fixes per site gain 22 points in-sample and 4 out of sample | artifacts/sniff-threads/F1-finding/RESULT.md |
| R1-robots | Does robots.py match a blind key on real robots.txt files, and do the fetchers use it? | Parser 230/230 allow/deny and 33/33 crawl-delay on 33 files. Status fail: fetchers do not check browser sub-requests, redirects, or share spacing. 578 files fetched: 187 use wildcards, 107 have Crawl-delay | The parser is right; the fetch paths around it are not safe yet | artifacts/sniff-threads/R1-robots/RESULT.md |
| T1-top20 | Do we find at least 90% of events on the top-20 venues' listing pages? | FAIL: 388 of 515 = 75%. O2 Academy 20/105, Literary Festival 0/30, Town Hall 16/28, Curzon blocked. The other 14 venues about 100% | Fetch needs API paging to the end, event-link vs tag-link, and stop-on-"no more". Estimated 92% if fixed | artifacts/sniff-threads/T1-top20/RESULT.md |
| T1b-fetchloop | Fetch code on 6 held-back sites, up to 3 rounds | Fail: r1 72/107 = 67.3%, r2 84/107 = 78.5% (bar 90%). Planted bugs had no effect, so not a proper run | Weak on cards with no own page, week/month steppers, and picking the real title | artifacts/sniff-threads/T1b-fetchloop/RESULT.md |
| T2-calendars | Do calendar-embed sites come through the dog? | FAIL on the line: 287 of 386 items = 74% (bar 80%). Per site 100/100/100/94/37%. Jev private-vs-public on 90 titles: 10 of 11 private caught | One fixable dog gap: it skipped some calendars in an embed | artifacts/sniff-threads/T2-calendars/RESULT.md |
| T2b-calloop | Same, held-back first, with a booking filter | Real events through 76/78 = 97.4%; round 2 78/78. Private dropped 0/0 (no test site had real bookings) | Finding passes once every calendar id is read; booking filter still untested | artifacts/sniff-threads/T2b-calloop/RESULT.md |
| T3-splitter | Saved per-site patterns applied by plain code on 5 list-only sites | Pass: 5/5 sites gave the exact block count on both fetches; 0 blocks lost a date | A saved pattern splits these pages, but patterns were hand-written and pages only minutes apart | artifacts/sniff-threads/T3-splitter/RESULT.md |
| T3b-flowtest | Does the split-role flow (scorer, plant, blind critic) work? | Pass (flow): planted Swan pattern showed as 2 vs 3 and the critic flagged it. Wallingford 3/3, Sewell 1 vs 19 (definition gap) | The roles catch things; the scorer matched by title only and never compared dates | artifacts/sniff-threads/T3b-flowtest/RESULT.md |
| T5-whole | Whole chain start to end on 20 random held-back sites | Partial. Shown 43 events; wrong 25 of 39 (64%; 16/39 if "none" counted as blank). Missed 238 of 266 key events (89%), 109 of 118 distinct titles. Jev about $0.23. Politeness breached on Schwarzman (217 pages) | Finding is the weak part, not reading. Show rule needs "can anyone go?". Stdlib robots parser ignores wildcards | artifacts/sniff-threads/T5-whole/RESULT.md |
| T6-onerequest | Ask all ~150 questions in one request vs ~46 | Pass: title 46/47 both, start 38/39 both, date 37/49 both; full agrees with core on 2893/2933 answers. Cost about $0.05 vs $0.024 per page | One big request loses no quality, costs about 2x | artifacts/sniff-threads/T6-onerequest/RESULT.md |
| T7-reach | Which of 586 sites are unreachable, and why? | Partial: 9.0% unreachable (blocked 25, broken 15, robots 7, booking box 3, login 2). Spot check 20/20 agreed, but a live check of 12 unreachable calls found 4 reachable | Reach is not the design risk; about 5-7% truly unreachable | artifacts/sniff-threads/T7-reach/RESULT.md |
| W1-options | Is the right date/time among the value finder's options? | Pass (narrow): dates 44/46 = 95.7%, times 39/39 = 100% on pages that give one. Mean 5.5 date and 3.2 time options per page | Ranges covered; wrong extra options (e.g. "0.33g" as 00:33) still need a check | artifacts/sniff-threads/W1-options/RESULT.md |
| W2-trim | Does trimming long pages halve Jev's unsure rate? | Fail: unsure 38 to 34 (need 19). Fix round: 18/68 fields unsure untrimmed vs 21/68 trimmed; sure-right 0.72 to 0.66 | Half the "long event pages" are not single events; length is not the cause | artifacts/sniff-threads/W2-trim/RESULT.md |
| W3-pagetype | Does "What is this page?" match a blind key? | Partial pass: 23/25 = 92% in round 2; not_events 5/5 | Works as one question; key was an AI, not a person | artifacts/sniff-threads/W3-pagetype/RESULT.md |
| W4-status | Zero false "cancelled/postponed" on 30 pages plus 15 trap pages | Fail: round 1 one false change, round 2 two; caught 5/7 then 6/7 | A "same event?" check still confuses sibling or past sessions | artifacts/sniff-threads/W4-status/RESULT.md |
| W5-next | Does clicking "next" find more than following links? | Fail on the bar: more on 7/10 sites (bar 8/10); 0 runaways. Found 168/376 (45%) vs 33/376 (9%) first view | Clicking is essential and stops safely, but load-more buttons and steppers fall short | artifacts/sniff-threads/W5-next/RESULT.md |
| W6-current | Staying-current rule drops removed events without wrong drops | Pass: 0 real wrong drops, 21/21 must-drops, 119/119 site-nights | Rule works; a real >30% removal makes every night "bad" forever | artifacts/sniff-threads/W6-current/RESULT.md |
| W7-ids | Same event keeps the same ID across old and new snapshots | Partial: 72/72 pairs kept, 0 wrong merges; but 68/72 share the exact link and 0 reschedules were in the sample | Link-based IDs work; no-link and reschedule paths barely tested | artifacts/sniff-threads/W7-ids/RESULT.md |
| W8-places | Place/organiser name matcher | Fail: 0 real wrong merges, true pairs merged 48/74 = 64.9% (bar 80%). 132 Jev calls | Safe but misses variants; normalise case and match any group member | artifacts/sniff-threads/W8-places/RESULT.md |
| W9-dupes | Cross-site duplicate matching | Pass: 0 real wrong merges, 13/16 = 81% true pairs caught; 23 near-misses kept apart | Design holds on real pairs; start-time tension remains | artifacts/sniff-threads/W9-dupes/RESULT.md |
| W10-repeats | Regex parsing of repeating-event wording | Fail: 18/26 exact (69%); 6 wrong-date examples + 1 false expand | Have Jev fill a structured rule; code expands | artifacts/sniff-threads/W10-repeats/RESULT.md |
| W11-fingerprints | Page fingerprint to skip unchanged pages | Fail: false "changed" 9/56 = 16% (bar 5%); 2 real main-content changes hidden (price, "SOLD OUT") | Fingerprint extracted facts, not trimmed text | artifacts/sniff-threads/W11-fingerprints/RESULT.md |
| W12-checks | Every plan check fires on its plant and stays silent on clean | Partial: 14/15 planted fired (miss = the plant), clean silent 15/15 | Test was easier than real; blocking checks fail open when a field is missing | artifacts/sniff-threads/W12-checks/RESULT.md |
| W13-qualitybar | Checker flags planted errors in event records | Pass (narrow): round 2 21/22 = 95.5% caught, 0/22 clean flagged | Reachable but fragile (one more miss = 90.9%); fail-open on missing facts | artifacts/sniff-threads/W13-qualitybar/RESULT.md |
| W14-realcode | Ported real code matches the tested script | Pass: 957/970 answers = 98.7%; critic found requests byte-identical on 19/20 pages | The event sniff exists as real code; pin the Jev version | artifacts/sniff-threads/W14-realcode/RESULT.md |
| W15-descriptions | Get the full description, first paragraph to last | Fail: mean recall 0.949 (met), 16/20 pages at 0.9 (bar 17), mean extra 0.166 (bar 0.10) | First/last-paragraph method fixes lost openings; extra text is the cost | artifacts/sniff-threads/W15-descriptions/RESULT.md |
| W16-series | Series-name question | Fail: 6/17 = 35% series pages right; 0 invented names on 17 no-series pages | Safe but misses most; low priority | artifacts/sniff-threads/W16-series/RESULT.md |
| W17-sitesniff | Improved website sniff on 40 held-back sites | Fail on "where events live" 77.5% (bar 90%); has_upcoming 87.5%, next_page 92.5%, booking_box 90% | New questions fix most false "no events"; "where" needs a fix round | artifacts/sniff-threads/W17-sitesniff/RESULT.md |
| C2-chain | Chain test | No RESULT.md; folder holds worker/chain.py only | Nothing to report | artifacts/sniff-threads/C2-chain/ |
| QUESTIONS | Which existing questions are below 95% right or tied to no field | 19 dev pages, 146 questions: venue 5/13 (38%), series 1/4, doors 6/10, oxford 12/18. GSC pull covers 2026-05-20 to 2026-08-17 | Triage list from a small sample, not a verdict | artifacts/sniff-threads/QUESTIONS/existing_check.md, gsc.md |
| DOG2 | A rebuilt dog that also reads events (calendars, feeds, page data) vs the old dog and August census | Retired 2026-09-29. On the dog's job it agreed with the census on about 77% of sites (partial full run: agree 304 / disagree 91) vs the old dog's 97%. Event-page finder found 31/31 and 12/12 in two blind tests. 47 tests | Reading events is the nightly scraper's job. Keep the old dog made polite; keep DOG2 readers for later | artifacts/sniff-threads/DOG2/archive-dog2/README.md, artifacts/sniff-threads/DOG2/STATE.md |
| TEMP_poc-s1, TEMP_poc-s2 | Copy-only scrapers built by an agent (6 sites) | s2: 4 PASS, sofo failed only on a kill-list error; negative controls 5/5. s1 has a kill line only, no result section | Copy-only scrapers can be built within 1-2 tries | artifacts/plan37/TEMP_poc-s2/KILL.md, TEMP_poc-s1/KILL.md |
| TEMP_poc-s3 | Do the scrapers survive unseen pages? | 5 sites robust; Ashmolean brittle (1 of 5 detail pages returned a title as a date) | Detail-page rules overfit hand-picked pages | artifacts/plan37/TEMP_poc-s3/KILL.md |
| TEMP_poc-s4 | Stricter build, sealed unseen pages | FAIL: Ashmolean 2/3 sealed pages flagged wrong-field ("PAST FREE DISPLAY", "Gallery 8" as date) | Copy-only alone still copies wrong text into date fields | artifacts/plan37/TEMP_poc-s4/KILL.md |
| TEMP_poc-s5 | Candidate collector for dates | FAIL: true date in a candidate on 13/19 pages (oumnh 0/4, ashmolean 3/5); 0 main-field traps | Collector misses dates on some sites | artifacts/plan37/TEMP_poc-s5/KILL.md |
| TEMP_poc-c1 | Contract v2 carrying all facts copy-only | PASS: 6 sites, within 2 tries each; 7/7 negative controls (castle 5/5 events, bullingdon 229/229) | Contract v2 fits the gaps found in s2 | artifacts/plan37/TEMP_poc-c1/KILL.md |
| TEMP_fastfail-castle | Copy-only scraper for one ghost-tour page in 3 tries | Kill line only (no result section) | See poc-c1 for the castle result | artifacts/plan37/TEMP_fastfail-castle/KILL.md |
| TEMP_poc-r1 | Why the reader leaves blanks (4,700 blocks, 870 sites) | All blanks FAIL on price (78.4%, top 6 causes); counting only blanks where a value is stated: date 91.8%, time 90.9%, price 83.9% PASS | Biggest causes: time ranges, year-less dates, date ranges | artifacts/plan37/TEMP_poc-r1/RESULTS.md |
| TEMP_poc-r2 | Reader v2 with nine fixes | PASS: 13-event wrong 2 to 0; date blanks 2125 to 1787; K1 hold-out (189 events) date right 3 to 20 | Fixes work; F4 only safe with F8 | artifacts/plan37/TEMP_poc-r2/KILL.md |
| TEMP_poc-r3 | Rules against placeholder times (00:00, 23:59) | FAIL as written: 33 times changed; clause 3 wiped one real time (Urzila Carlson 20:00). Clauses 1, 2, 4 PASS | Empty-field clause is too greedy | artifacts/plan37/TEMP_poc-r3/KILL.md |
| TEMP_poc-r6 | LLM reader with grounding checks vs rules reader | KILL: held-out (220 events) 9 wrong vs 0 for rules v2; 13-event 1 vs 0 | Rules reader with Jev picks beats LLM-with-checks | artifacts/plan37/TEMP_poc-r6/KILL.md |
| TEMP_poc-r7 | Add A and B to v2 | FAIL as pre-registered: v2+A blanks 309 to 291 (5.8%); v2+B had 1 wrong, a verified key error. Corrected: v2+A+B 225 blanks | Blank cut real; key errors happen | artifacts/plan37/TEMP_poc-r7/KILL.md |
| TEMP_poc-r8 | v2+A+B on a fresh key (204 events, 26 sites) | PASS: 0 reader errors, blanks 389 to 203 (-48%) | Holds on fresh sites | artifacts/plan37/TEMP_poc-r8/KILL.md |
| TEMP_reader-v6, v7, v8 | Later reader versions on all sets | v6 PASS 0 wrong on every set; v7 PASS (K1 blanks 217 to 213); v8.1 fills s03c 1584 to 1078 blanks, 0 wrong | Reader kept at 0 wrong while blanks fall | artifacts/plan37/TEMP_reader-v6/KILL.md, TEMP_reader-v7/KILL.md, TEMP_reader-v8/KILL.md |
| TEMP_fastfail-reader | Reader on 13 hard cases with Duckling and Jev | PASS on the line: 2 wrong (limit 2), both placeholder times (23:59, 00:00) | Placeholders are the reader's main trap | artifacts/plan37/TEMP_fastfail-reader/KILL.md |
| TEMP_poc-k1, TEMP_poc-k2 | Answer keys from structured data (K1) and two-reader agreement (K2) | K1: 35 of 337 events with a real structured time (10%) conflict with their own page text. K2: on 21 plain-HTML sites 76 blocks, date agreed 26 (34%), time 13 (17%), price 6 (8%) | Structured data is a key with a 10% error; two readers agreeing cannot catch a bad block | artifacts/plan37/TEMP_poc-k1/README.md, TEMP_poc-k2/README.md |
| TEMP_poc-jev1 | Jev choosing between date candidates | KILLED: 0 wrong, but only 1 real gain lost-then-found (5 post-hoc) vs bar of 10; 62 Jev calls | Ceiling is the span cutter and year rule, not Jev | artifacts/plan37/TEMP_poc-jev1/PROGRESS.md |
| TEMP_poc-e1, e2, e3 | Whole line end to end on 3 sets | All KILL: E1 342 events 0 wrong scored but Aphrodite 19:00 (price read as clock); E2 3 wrong on fresh set (01:00, 2027-01-01); E3 5 wrong dates on the sealed set (last date of a 3-date list) | Each run finds a new wrong shape; fixes were made after each | artifacts/plan37/TEMP_poc-e1/REPORT.md, TEMP_poc-e2/REPORT.md, TEMP_poc-e3/REPORT.md |
| TEMP_poc-m, m5, m6, m7, m8, m9 | Matching the same event across sources, building on each other | m: held-out 288 items, baseline recall 4/34. m5 KILLED (O2 Academy and Academy2 fused). m6 PASS, fresh recall 13/16, but too easy. m7 PASS, 0 wrong on 454 items. m8 and m9 KILLED on new worlds (e.g. Emo Night in two rooms; m9 base 20 wrong pairs) | No matcher config was safe on every world; Jev merges wrongly across rooms and early/late shows | artifacts/plan37/TEMP_poc-m/KILL.md (and m5-m9) |
| TEMP_poc-m10 | Collision rate on live prod data | ng.listing_showtimes 2 rows; events.address_id 2,298 of 32,434; scraper fleet has almost no redundant coverage | Data too thin to measure collision rate directly | artifacts/plan37/TEMP_poc-m10/KILL.md |
| TEMP_fastfail-match | Matchers on 293 Bullingdon/Glee/LiveNation items | FAIL (strict): Jev matcher 0 wrong, 19/40 duplicates; trained Splink 3 wrong, 1/40; candidate rule 38/40 | A trained model merged different rooms; Jev matcher safe but low recall | artifacts/plan37/TEMP_fastfail-match/KILL.md |
| TEMP_fastfail-ids | Do a site's own IDs give a free answer key (>=200 cases, >=90% stable)? | Kill line only in the folder; scan scripts included | Not stated in the file | artifacts/plan37/TEMP_fastfail-ids/KILL.md |
| TEMP_poc-ser1 | Expand weekly rules into dates | KILL on Bullingdon: 8-week expansion gave 2 of 16 dates on nights that do not happen; 2 and 4 week windows 0 wrong | A rule does not know which weeks are skipped | artifacts/plan37/TEMP_poc-ser1/RESULT.txt |
| TEMP_poc-cost1 | Cost estimate from what POCs logged | Scraper build about $205-$265 once for 586 sites (110k-135k tokens/site); nightly fetch about $0 API | Estimate, not a measurement | artifacts/plan37/TEMP_poc-cost1/KILL.md |
| TEMP_skeleton | Whole line on 13 sites | 1,018 rows scraped, 45 shown; reach 44/63 (70%) of events with a start time, 45/85 (53%) of all. Stop rule hit: "Comedy Gala" shown 17:00 (office hours) vs page 19:30 | Line runs end to end but reach is short and one false value ships | artifacts/plan37/TEMP_skeleton/KILL.md |
| TEMP_schema-stress | 78 made-up pages from 13 ingredients (2 rounds) against the data model | Kill line trips on scenario 7 (one event at two venues); round 2: 78/78 round-1 pairs fit; new kind: a page with doors and start time | Fits after about 16 small changes, no new tables. Self-graded | artifacts/plan37/TEMP_schema-stress/TEMP_report.md |
| TEMP_umbrella | Festivals on real sites | Kill line trips on umbrella 4 of 5 (Ghost Fest): 2 new columns needed, limit 1 | Festival including some dates of a recurring event needs one more link | artifacts/plan37/TEMP_umbrella/TEMP_report.md |
| TEMP_gate-lint | Lint that catches scrapers that interpret text | Catches 80/80 planted tricks, 51/52 real bad scrapers | False-positive rate on clean scrapers unmeasured (none held out) | artifacts/plan37/TEMP_gate-lint/TEMP_report.md |
| TEMP_gate-checker | Checker catches planted errors in scraper output | Final sealed exam 8 unseen sites: about 90% of planted errors caught, correct output wrongly failed on 7 of 8 sites. Catches 9/9 real bad scrapers (201/210 wrong events). Loop: about 2,760 Jev calls; answer keys wrong 4 times | Not fit to block; ok as an alarm | artifacts/plan37/TEMP_gate-checker/TEMP_LEDGER.md, TEMP_LOOP-REPORT.md |
| TEMP_gate-duplicates | Matching never wrongly merges | 0 wrong merges on all sets with Jev, CANCELLED traps 20/20; finds 0/8 real held-out duplicates | Safe, low recall | artifacts/plan37/TEMP_gate-duplicates/TEMP_report.md |
| TEMP_duckling-lab | Duckling parser for dates, times, money | 64/64 real prices found (median 2 amounts per page). Fails on en-dash ranges, "2-hour", "8.30 sharp", "£19.00" read as 19:00 | Good at finding values; Jev picks which is the price | artifacts/plan37/TEMP_duckling-lab/TEMP_REPORT.md |
| TEMP_rehearsal | 40 frozen sites and a grader | 1 of 20 random sites blocked (5%); grader catches 91/96 planted mistakes, 0 right events marked wrong (tuned data only) | Practice set ready; grader tuned on its own data | artifacts/plan37/TEMP_rehearsal/ |
| TEMP_exam | Fair exam: 20 unseen sites through the whole line (batches) | Verdict NOT READY: 5 wrong shown values (all prices); 0 wrong dates/times; about 45 lost events. Scorer caught 46/56 planted mistakes (82%), errors all lenient | Fails reach and the 0-wrong line | artifacts/plan37/TEMP_exam/GOAL.md |
| TEMP_tune, TEMP_tune2 | Tune reader v8.7 and exam on sealed random sites | v8.7 fills 41 to 52 of 67 (on tuned sets). Random4 batch 1: 0 wrong, reach 3/15 (20%). Across 3 unseen exams: 54%, 56%, about 20-33% | Hand-tuned rules do not generalise; 0 false dates/times throughout | artifacts/plan37/TEMP_tune/GOAL.md, artifacts/plan37/TEMP_tune2/ |
| TEMP_fresh20 | Reader v8.4 on 20 fresh sites | 1,544 values re-judged: 1,454 right, 90 not right (none a false published value), 64 cannot-tell | Zero false published values held through s13-s20 stop rule | artifacts/plan37/TEMP_fresh20/KILL.md |
| TEMP_timetable | Repeating events: how common, how to store | About 3 in 10 sites state a repeat; rule expansion 2/27 wrong at 4 weeks, 11/66 at 12; series.py 3/40 | Rules cannot make firm dates | artifacts/plan37/TEMP_timetable/TEMP_STATE.md |
| TEMP_open-jev | Free local model instead of Jev | Plumb alone: 41/153 items before cap; 0 sure-and-different from Jev; runtime degrades to about 14 s/item | Stay on Jev; Plumb only a fallback | artifacts/plan37/TEMP_open-jev/README.md |
| TEMP_page-update | Writer/critic loop (3 rounds) that updated the plan page | Critic 6: lint 0 fails; 0 MISSING; word cap 0 over; 2 CONTRADICTS | Page checked against sources | artifacts/plan37/TEMP_page-update/critic-6.md |
| Jev answer banks (dev / held) | The question set asked of saved event pages | bank_dev 290 sites, 284 events; bank_held 276 sites, 301 events; bank_out 566 sites, 586 events; 17 sites had errors | The banks behind the tune / held split | artifacts/plan38/TEMP_evidence/bank_dev.jsonl, bank_held.jsonl, bank_out.jsonl |
| check_tidy / check_messy | Blind check of Jev answers on 10 tidy and 10 messy pages | Tidy: 396 RIGHT, 26 UNCLEAR, 38 WRONG. Messy: 410 RIGHT, 23 UNCLEAR, 30 WRONG | Messy pages did not score worse than tidy pages | artifacts/plan38/TEMP_evidence/check_tidy/, check_messy/ |
| Site sniff, dedup, pagination | Site sniff over 227 sites; pairwise dedup; next-click paging | sitesniff_out 227 sites; sniff_out 152 rows; misses.json 7 entries | Working files for the site sniff, dedup and paging tests | artifacts/plan38/TEMP_evidence/sitesniff_out.jsonl, dedup/, pagination/ |
| candidate-audit-1 | Is the right date / start / price among the options the candidate step offers Jev (first 24)? Key = the JSON-LD Event each saved event page carries about itself; 4483 saved event pages scanned, 1218 keyed (86 sites); no Jev, no network | Right value offered: date 96.3% of 1205 pages, start 93.6% of 1038, price_low 98.4% of 308, price_high 97.4% of 77. Misses by pattern: recurring rule only 29 pages / 3 sites, bare "10:00" with no am/pm 46 / 7, wrong year 7 / 6, shortened range 6 / 3, written time not parsed 19 / 3, "7–9:30pm" 1. Left out as suspect keys: 72 start times exactly 1 hour off (12 sites label UK time as UTC), 54 + 32 prices never written on the page. Fake date options (lone weekday or bare time) on 991 of 1218 pages, 1.51 per page | On pages from sites that publish structured data, the candidate step usually offers the right value; its main fixable gaps are bare 24h clocks and recurring rules, and it adds distractors to most pages. Keyed pages are tidy-site pages, so this overstates accuracy on HTML-only sites (where the 110-event review found the date problems) | scraper/audit/candidates.py, data/audits/candidates.json (uv run python -m audit) |
| pick-audit-1 | Follow each value through offered -> Jev's pick (sure = conf >= 0.8) -> card, using Jev answers already saved (run 2026-09-28-original + evidence bank, jev-latest, today = 28 Sep 2026) on the 148 pages (80 sites) that carry their own JSON-LD; suspect keys left out as in candidate-audit-1 | Date (143 pages): offered 95.8%, sure+right 74.1%, sure+wrong 4.9% (6 of 7 wrong when the right date was offered: picking the next date among several, year, range); card right 82 / wrong 2. Start (111): offered 92.8%, sure+right 73.9%, sure+wrong 6.3% (mostly doors vs start, or keys still 1 hour off); card right 67 / wrong 4. Right value offered but Jev unsure: 31 dates, 21 starts (shown blank). Prices (32/31 pages): offered 100%, "sure+wrong" 19% but nearly all are the key including booking fees (key £18.70, Jev £17) | On tidy pages a wrong date or time is rare and mostly a picking problem (several dates, doors vs start), not a candidate problem; the bigger loss is right answers left blank as unsure (~20%), a threshold/calibration question (P31). The JSON-LD price key can't judge prices: it includes fees the question excludes | scraper/audit/picks.py, data/audits/picks.json (uv run python -m audit.picks) |
| venue-audit-1 | Is the event's place (the JSON-LD location name) among the venue options Jev is offered (shorts, first 25)? Same 1218 keyed pages as candidate-audit-1; 774 have a usable place name | Offered cleanly on 38.2% of 774 pages (per-site 45.7%) vs 93-98% for date / start / price. Misses: place written as a named element the option regex skips (- link "X":, - heading "X" [level=5], - row "X", - img "X") 240 pages / 20 sites; only in a heading 150 / 14; past option 25 35; place + address over 60 characters 31; glued to a date or time 14; inside a title or sentence 7; 83 keys not on the page. Applied in scraper/reader/page.py short_texts(): read named elements too and order options from the main heading outwards (as the reader already does for dates) -> 66.0% clean (date / start / price unchanged); most of the rest sit at options 25-50 and are the site's own name (header or footer) | Venue is the weakest candidate by far, and one general cause (page_bits only reads role: text lines) explains most of it; the fix touches only the venue question (the only user of shorts) | scraper/audit/candidates.py (uv run python -m audit), data/audits/candidates.json |
| venue-rerun-1 | Does the venue option fix (short_texts) change what Jev answers? Jev jev-1.13.0, all 110 app events (incl. held, viewing only), same model both arms: run 2026-10-02-baseline (old page_bits) vs 2026-10-02-venue-fix | "Venue not found" 34 -> 32 of 110; review-flagged venue-missing events with a venue 21 -> 28 of 49. Same model: 9 gained (7 real venues, 2 with label or sentence words: "Location The Hub", "Step into the shadows of Oxford Castle Prison…"), 7 lost (4 real: AirHop, Cowley Workers Club, Dorchester Abbey, Oxford - Phoenix; 3 junk or out of area). Cost $0.04 per run | Better options lift the venue only a little on these pages: Jev's confidence drops when more options compete (AirHop 0.95 -> none), and options carry label / sentence words. Next general steps: clean option text (label words, sentences), and test a larger venue max | data/runs/2026-10-02-baseline, data/runs/2026-10-02-venue-fix |
| date-fix-1 | General date-candidate fixes in scraper/reader/values.py: spell out shortened ranges ("20 to 25 October", "Thu 16 - Sun 19 Sept") and day lists ("1st & 15th October") before Duckling; whole-day exclusive ends; a match becomes a date only when its words say a day + month (or d/m, today, tonight, tomorrow); a bare "19:30" kept as a time | Candidate audit (1218 keyed pages): start offered 93.6% -> 98.1%; date 96.3% -> 96.5% (6 shortened ranges + 1 year fixed; 6 dates lost were lucky guesses from a lone weekday on "every Thursday" pages or a calendar grid); fake date options 1.51 per page on 991 pages -> 0.09 on 75. Jev rerun, same model (2026-10-02-venue-fix -> 2026-10-02-date-fix, 110 events): 14 dates changed; Blenheim and Swalcliffe now start on the first day, Old Fire Station recovered (20 Oct), invented dates gone on 2 events, 3 past events now dated (hidden as past); 1 multi-date page (Hooky Players 20 Nov) went blank (confirm said no). "No date found" 24 -> 21 vs the old-code run | Date candidates are now clean; what remains is Jev confidence (21 of 110 events have no sure date with jev-1.13.0) and recurring / multi-date events, which need the event-shape decision | data/runs/2026-10-02-date-fix, scraper/audit/candidates.py |
| date-code-picks-1 | Code, not Jev, makes the date comparison. Date question reworded literal ("Which of these is a date on which {E} takes place?") plus one yes/no per candidate date (dates, each: date); reader/ask.py choose_dates takes Jev's pick + every sure yes and keeps the next on or after today, else the latest; every date of the event saved as belongs. Tried first: yes/no per date alone (run 2026-10-02-date-each) | Yes/no alone: sure dates 89 -> 74 of 110 (Jev hesitates on single-date yes/no: Double Red Duke's correct date got yes 0.31; a more literal wording was worse). Combined (run 2026-10-02-date-code-picks): sure dates 89 -> 101 of 110, starts 80 -> 83, 27 events with several dates found; 16 dates changed (pop-up date replaced by the real 4 Nov, Hooky 19 Nov, daily tours dated today, past events dated and hidden as past); Browse lists 80 vs 72. Weekly courses with only start-end written show the last session (Phase 5) | Jev is strong at picking from a list, weak at one-date yes/no; using both and letting code compare dates gives more sure dates and every date of multi-date events | questions/pick.txt, scraper/reader/ask.py, data/runs/2026-10-02-date-code-picks |
| venue-clean-1 | Clean every venue option before Jev sees it (scraper/reader/page.py clean_venue): icon glyphs, a leading label ("Location", "Venue :"), and a leading / trailing run of date-time words, cut only when the run holds a weekday, month or clock time | Candidate audit: venue offered cleanly 66.0% -> 69.6% of 774 pages; glued to a date/time 14 -> 6 pages. Jev rerun (2026-10-02-date-code-picks -> 2026-10-02-venue-clean, same model): "Venue not found" 32 -> 30 of 110; venues with date/time/label words 7 -> 5; 10 venues changed, e.g. "9 October @ 7:00 pm - 10:30 pm Littlebury Hotel, Bicester" -> "Littlebury Hotel, Bicester", "Location The Hub" -> "The Hub", AirHop now has its address; sure dates 101 -> 100 | Option text cleaning carries straight through to the card. Left: venues inside a sentence ("New Years Eve concert – 6pm at Dorchester Abbey"), and venues past option 25 (often the site's own name) | data/runs/2026-10-02-venue-clean |
| venue-phrases-1 | A venue option that is a phrase naming a place after "at" / "@" / "in" is offered as just that place (scraper/reader/page.py place_in_phrase): capitalised words, two of them or a place word (Hall, Abbey, Theatre…), stopping at the first lowercase word; a leading "the" allowed | Candidate audit: venue offered cleanly 69.6% -> 71.2% (tune 71.1%, held 71.2%). Jev rerun (2026-10-02-venue-clean -> 2026-10-02-venue-phrases2): "Venue not found" 30 -> 25; sentence venues 2 -> 0 ("…6pm at Dorchester Abbey" -> "Dorchester Abbey", "…in the historic Holywell Music Room" -> "Holywell Music Room", "…at the Jude the Obscure in Oxford" -> "Jude the Obscure"); Fairytale Farm x2, The Barn, Dorchester Abbey newly found; dates unchanged | Examples behind these rules came partly from held-site pages (the first 20 UI events are held); tune and held moved alike, so no sign of over-fitting, but the next rule should be built from tune pages only | data/runs/2026-10-02-venue-phrases2 |
| on-sale-1 | New literal question on_sale in questions/fixed.txt ("Can people book a place or buy a ticket for {E} right now?": open / not yet / closed / sold out / waiting list / no booking / not stated); the card button (app/lib/listing.ts cta) follows it when sure, else the older booking answers; "Book tickets" only with a price, else "Book a place". Built on tune pages only (5), checked on all 110 | Run 2026-10-02-on-sale vs 2026-10-02-venue-phrases2: 11 buttons changed; "Tickets on sale soon" on 6 events whose pages say sales open later (Hooky x2, Queen's College "Booking will open in Michaelmas Term", Blenheim "box office opens January 2027"…); a waitlist-only page now "Join the waiting list" (was "Sold out"). Review-flagged booking buttons still wrong: 2 of 16 (a groups-only page Jev is unsure about; a held page with a "Registrations Closed" banner Jev read as open, 0.94) | One literal question fixes most "Book tickets" errors; a closed-registration banner is still missed | questions/fixed.txt, data/runs/2026-10-02-on-sale |
| tags-1 | Card tags came from event_type_detail / music_genre in questions/full.txt, which the reader no longer asks: runs overlaid on the original answers kept those stale tags, and their options had no comedy / opera / race (comedy night -> "Gig", HMS Pinafore -> "Play", a half marathon -> "Match"). Fix: format and genre questions in questions/fixed.txt with full option lists; tagsFor in app/lib/listing.ts shows readable words and skips a format that repeats the category; a run's answers replace the original ones instead of overlaying them; no "unknown" category label. Built on 7 tune pages | Run 2026-10-02-tags: comedy nights lose "Gig"; HMS Pinafore "Opera"; Oxford Half Marathon "Race"; Blenheim horse trials loses "Match"; a Beatles tribute "Concert · Tribute act" (was "rock"); music cards "Concert · Folk / Choral / Classical / World music" | Tags now come only from questions the reader asks today, in words people read | questions/fixed.txt, data/runs/2026-10-02-tags |
| desc-1 | Description options (scraper/reader/page.py para_texts, only the desc question uses them): prose blocks of 8+ words from paragraph, text, strong, main and long heading lines; a paragraph split into child lines (bold, links) joined back; long paragraphs kept to 600 characters; short headings, site boilerplate (cookies, privacy, newsletter…) and mostly-date lines left out; ordered from the main heading outwards. Before: only - paragraph: … lines of 80-600 characters, in page order | Candidate audit (JSON-LD description as key, 313 checkable pages): right description offered 56.3% -> 95.8%. Jev rerun (2026-10-02-tags -> 2026-10-02-desc-final): sure descriptions 61 -> 81 of 110; pages with no description options 14 -> 0; 29 gained, 9 lost (all now unsure, 0.23-0.79: more real paragraphs compete); 1 junk shown, no titles. Two intermediate runs found and fixed: titles / cookie text offered, long paragraphs dropped | Descriptions were missing mostly because the option builder only read one kind of line; the remaining loss is Jev choosing between several good paragraphs | data/runs/2026-10-02-desc-final |
| doors-1 | Show the doors time on the card (app/lib/listing.ts doorsText): only when Jev is sure the page gives a separate doors time and sure of the time, and it falls 1-180 minutes before the start. Display only, no rerun | On run 2026-10-02-desc-final: doors shown on 7 events, all 7 matching their page ("Doors: 7pm | Support act: 7.15pm | Main act: 8pm", "doors open 7:30pm", "Doors at 6:30pm, show 7:30pm"…); 0 listed pages that write "doors" with a time and show none |
| structured-1 | Phase 2 offline: use the page's own JSON-LD Event. Checked first where Jev and the JSON-LD disagree on the 110 app events (39 carry JSON-LD): dates 5 of 38, starts 6 of 29. Then scraper/reader/structured.py fill_gaps: a JSON-LD value fills a field only when Jev left it blank or unsure AND the same value is among the page's own options; wired into reader/event_sniff.py and rerun/core.py | Disagreements: Jev right on most (JSON-LD start an hour off x4 — UK time labelled UTC; a series' next date instead of this page's 28 Nov; an end time "17:00 - 21:00" given as the start), JSON-LD better on 1 (a per-performance page), 3 debatable — so JSON-LD never overrules Jev. Run 2026-10-02-structured (Jev cached, $0): 7 fields filled (Bridge nightclub price £5; Unicorn lowest price £16.50; Hadsham Farm description; 2 values Jev had just under 0.8 confirmed); 1 questionable (a £10 that may be a deposit) | Offline, page JSON-LD adds little and is not reliable enough to lead; the bigger structured sources (calendar files, event APIs) are live-only and need the polite live fetch decision | scraper/reader/structured.py, data/runs/2026-10-02-structured |
| checks-1 | Phase 6 sanity checks before any card is drawn (app/lib/checks.ts, applied in lib/browse.ts for Browse and the event page; the event page lists what was left out): a date more than 2 years ahead, a 00:00 / 23:59 start, "free" with a price, a price over £1,000, venue or description that is site boilerplate, and a title that is a section name ("Events", "What's on", "News") or a site notice ("Privacy Notice", "Website Cookies") -> event not listed | 110 app events (newest run): 1 event trips a check (a note: Jev says the title is only the organiser's name). 586 evidence-bank events (older questions): 11 trip — 4 section-name titles and 2 cookie / privacy pages now left out, 5 notes; no date, time or price check fired | The checks are a safety net: the candidate and question fixes already stop impossible dates, times and prices reaching the card on these pages | — |
| review-2 | Independent re-review of all 110 events by 6 Sonnet agents (same method as the first review: card vs saved screenshot, headless), on the newest run 2026-10-02-structured; left-out decisions judged too | Events showing a serious error 39 -> 7 (35% -> 6%); serious issues 47 -> 8; all issues 288 -> 104; fully clean events 4 -> 24; 27 left out correctly, 4 wrongly. Remaining by cause: multi-day / recurring shown as one date 24 events, "Venue not found" 18 (often the site's own venue), price details 17 (mostly minor: tiers, VAT, child price as lowest), descriptions 14 (minor). Serious left: 2 wrong prices from structured-1 (a £10 deposit, a "2 for £5" drinks offer), weekly courses shown at a past or last session (2), an empty "No activities found" page listed, a tagline venue, a cinema run left out, one wrong start time | The fixes hold up independently; what remains is mostly the one-date event shape (Phase 5) | scratchpad/event-review-v2/ (session), this table's earlier rows |
| review-fixes-1 | Fixes for review-2's serious issues, all general: JSON-LD never fills prices (reader/structured.py); a weekly course or run of days with a past first date and a future last date gets its next session from code (reader/ask.py choose_dates, when Jev is sure of repeats); "of" joins the venue phrase rule only inside a sentence and before a place word (reader/page.py); a page whose description says nothing is scheduled is left out (app/lib/checks.ts) | Run 2026-10-03-review-fixes ($0.001, Jev cached): the deposit and drinks-offer prices gone; both weekly courses (Sunningwell, Ayres House) now Wed 30 Sep, the real next session (was left out / last session); "Oxford Castle Prison" instead of the tagline; the empty listing page left out. Venue audit 71.2% -> 71.1% (one page; real "University of …" names kept); dates unchanged. 79 listed, 31 left out. Still open: the cinema run (Jev unsure of any one showing) and multi-date display — Phase 5 | Serious issues left from review-2: 2 of 8 (#64 cinema run, #78 a multi-day exhibition's time) | data/runs/2026-10-03-review-fixes |
| venue-site-1 | "Venue not found" on 18 events in review-2, mostly where the venue is the website itself (The Kenton, Story Museum, Corn Exchange, Modern Art Oxford, Jericho Tavern). Keyed pages: the venue is the site's own name on 365 of 774, and 109 of the 224 misses. Fixes, all general: (1) the site's name from data/sites.json is one extra venue option, always inside the 25 (reader/page.py with_site_name; Jev still decides); (2) the card shows venue + address when Jev is sure of both and the address adds to the name (app/lib/listing.ts venueText); (3) the borderline venue second look in questions/confirm.txt (venue2) had never run — confirm() only knew dates and times and looked for a field named "venue2"; a confirm block now names its field (for: venue) and text picks are checked as written | Candidate audit: venue offered cleanly 71.1% -> 82.6%. Listed events with "Venue not found": 14 -> 8 (site option) -> 7 (second look); review-2's 18 missing venues: 12 now found. Venue + address: "Oxford Town Hall, St. Aldate's, Oxford, OX1 1BX"; a church that meets at a school shows both. The second look stays cautious: it confirms "The Story Museum" (yes 0.97) but not "Oxford Botanic Garden & Arboretum" for events at Harcourt Arboretum, nor "Bodleian Libraries" for a course at the Weston Library. Dates 102 -> 101 | Left (7): a sub-venue of the site (Harcourt Arboretum, Weston Library), a meeting point, a venue after "to" ("returning to Oxford Town Hall") | data/runs/2026-10-03-site-venue, data/runs/2026-10-03-venue-confirm |
| heldout-1 | The first measurement on pages no fix was built on: 200 saved event pages drawn at random (seed 20261004) from 108 held-out sites, at most 2 per site, none of the 110 app events (rerun --from-crawl --split held, $0.085); preview cards built with the app's own code; 5 Sonnet agents judged each against the saved page text (no screenshots for these pages). Held pages used for scoring only | Only 123 of 200 saved "event pages" are single real events (the rest: listings, news, shop, info, error pages — almost all correctly left out). 78 listed: 26 clean (33%); serious issue on 15 = 19% (95% range 12-29%), of which 10 are a missing title (Jev unsure of the title, card has none) — excluding those, 6% (3-14%): 3 wrong dates (a sibling session's date, a run's end date), 2 non-events listed, 1 price ("Free" that needs a day ticket). Listed but should not be: 4 (2-12%). Real events wrongly left out: 6 of 123 (2-10%): "other events" sidebars read as a list (2), Henley-on-Thames read as not near Oxford (2 across batches), recurring with no single date. All issues: venue missing 17, multi-date collapsed 17, title 14, time 9, price 7 | Field accuracy holds on unseen pages except titles; the next general gaps are titles (unsure), "Oxfordshire" read as "near Oxford city", sidebars making a page look like a list, and the one-date shape | data/runs/2026-10-04-heldout-sample, scratchpad/heldout-review/ (session) |
| title-confirm-1 | Title second look (title2, for: title in questions/confirm.txt): a borderline title pick gets "Is "{label}" the name of {E}?"; a sure yes promotes it. Built on a fresh random dev sample of 100 tune pages (55 sites, seed 20261005, runs 2026-10-04-tune100-before / -title), then measured once on the held-out sample (scoring only) | Tune dev: sure titles 70 -> 73 of 100; 17 second looks — real names confirmed ("Soundscaping the Wonder of Birds"), site headings rejected ("Contact Us", "Privacy Policy", "Our Story", "Room detail"). Held-out (2026-10-04-heldout-title, $0.003): sure titles 143 -> 152 of 200, all 9 new titles real event names; of heldout-1's 10 missing-title cards, 3 now titled (Harwell film club, Harwell history group, an Ashmolean exhibition). Left: names in a series label ("Event Series: Cake Cafe"), a seminar and a wellbeing page where Jev stays unsure | Serious issues on held-out listed cards: 15 -> 12 of 78 (~15%; ~6% excluding a missing title) | data/runs/2026-10-04-heldout-title |
| area-kind-title-1 | Three general fixes from heldout-1, built on tune pages only (dev sample 2026-10-05-tune100-v2 + the 90 tune app events), measured once on the held-out sample (2026-10-05-heldout-v2, scoring only): (1) Oxfordshire, not "near Oxford": oxford question reworded to the county with its towns, and the venue / address postcode decides in code when Jev is sure of one (OX*, RG9 Henley, SN7 Faringdon in; RG8, SN6, GL7/54/56, HP14/17/18, CV36 border -> Jev; any other postcode out; app/lib/listing.ts oxfordshireByPostcode); (2) kind question tells Jev to ignore side panels ("other events", "you may also like"); (3) title options add a name given after a label line ("Event Series:" then - link "Cake Cafe") and strip "Event:" / "Series:" prefixes (reader/page.py title_options; pick.txt title from: titles) | Tune: sure titles on app tune events 81 -> 89 ("Event: Angeline Morrison…" -> "Angeline Morrison + Goblin Band"); 8-9 more sidebar pages read as one event. Held-out vs the reviewers' verdicts: real events wrongly left out 6 -> 1 (Henley Literary Festival, Tadmarton Heath, two wine tastings with sidebars, a library Lego club); missing titles 10 -> 5; wrongly listed 4 -> 3 (a members' meeting now out); 1 new wrong listing (a "what's on today" page); real single events listed 76 -> 80 of 123 | Recovers real Oxfordshire events without letting non-events through, bar one day-listing page | data/runs/2026-10-05-heldout-v2 |
| rule-dates-1 | A page that gives only a rule ("Every Fri & Sat during Aug–Oct, 2–3pm") had no date options, so no date. scraper/reader/values.py: rule_dates turns "every <weekdays> [during <months>]" into the next two matching dates on or after today (a month window with no year = the coming one); _share_ampm makes "2–3pm" / "10 – 11am" / "11–1pm" give both times; the chosen date keeps the rule's words (inferred_from) and the card says "check with the organiser" | Ashmolean tour: no date -> Fri 2 Oct, 2pm (was the issue the user raised); Oxford Castle "Every Friday, from 18:30" -> Fri 2 Oct (was a wrong 1 Jan); a weekly choir -> Wed 30 Sep. Audit: date offered 96.5% -> 97.1%, start 98.1% -> 98.2%. Sure dates: tune dev 66 -> 67, app events 100 -> 101, starts 83 -> 85 | — | data/runs/2026-10-05-rule-dates |
| several-dates-1 | The "Events with several dates" proposal, implemented in the snapshot: new date_shape question (once / separate performances / daily run / repeating); reader/ask.py treats a sure daily run like a run of days (today when it is on) and keeps which words each date came from (belongs_words); the card (app/lib/listing.ts runText, alsoOn) shows a daily run as "Sat 24 Oct – Sun 1 Nov · daily 10am" or "Until …", separate dates as "Also on Fri 20, Sat 21 Nov" (first 4 + "and N more"), a written range as "Runs until Sun 25 Oct", a weekly course as "Weekly until Wed 25 Nov"; rule-derived dates keep the "check with the organiser" note. Two over-eager "until" cases found and fixed (a time range "13:30-14:30" and a rule's month window "Aug–Oct" are not date ranges) | App events (2026-10-05-several-dates2): 18 listed events now show their other dates or run (Hooky Players also on 20 and 21 Nov; a cinema's 5 showings; a 6-week course; Fairytale Farm "Sat 24 Oct – Sun 1 Nov · daily 10am"; Blenheim horse trials as a run). Held-out (2026-10-05-heldout-v3, scoring only): 14 of the 17 cards the reviewers flagged as collapsing several dates now show them. Left: an exhibition and a gallery run shown as one date, a theatre run as "also on" its end date | The biggest remaining gap from review-2 is now mostly closed in the snapshot; the live site already has the three shapes to receive it | data/runs/2026-10-05-several-dates2, data/runs/2026-10-05-heldout-v3 |
| card-facts-1 | More of Jev's answers on the card, each only when sure: age ("Ages" line, from a new age pick over age phrases the page gives — reader/page.py age_texts), "Concessions available", "Also online" / "Online only", "Outdoors" and "For children" tags; the "Parking" tag removed (its question only asks whether parking is mentioned; flagged wrong twice in review-2). Checked by an independent agent against the page text on 63 listed events (app + held-out) | 73 facts: 62 correct (85%). Online 3/3, for children 3/3, outdoors 13/14, concessions 24/28 (a member price from another film in a sidebar x2, "under 2s free" x2), age 19/25 (a stray "D" from "aged", "not suitable for under 3s" shown as "Under 3s" x2, a supervision rule, a price tier). Fixed, all general: the age line shows only "N+", "N–M years", "Adults only" ("not suitable for under N" -> "N+"; bare under/over phrases not shown; phrases next to £ or "free" not offered); the concessions question now asks about this event's own tickets and excludes free entry for babies | Show a new fact only after it has been measured; age needed two rounds | data/runs/2026-10-05-card-facts3, scratchpad/facts_review.jsonl (session) |
| question-prune-1 | Pruned the question set from 59 to 42 (fixed 43 -> 29, pick 16 -> 13): dropped 17 that nothing used or that newer questions replaced (other_events, more_link, has_support, series, multi_day, all_day, fee_given, time_given, sessions, rule_only, access_perf, parking, sold_out and waitlist (both covered by on_sale), support, image, ticket_link). Put six unused answers to work: cancelled status now hides (was a label only; dry run listed "…- CANCELLED"), pay-what-you-can and a sure booking fee on the price, "(doors)"/"(last entry)" when that is the page's only time, a "we assumed the year" note when the page gives none, and page_old / contradicts as "check before listing" flags. Reworded public (No only when the page says so) and status (look for cancelled/postponed words). Also: a sure "repeats on a pattern" leaning weekly, first and last dates on the same weekday, now counts as weekly (two weekly courses that began in September were hidden as past when the "how often" answer sat at 0.78) | Before -> after, same pages: tune 40 -> 39 listed, held-out 86 -> 85, dry-run sample 99 -> 100, app 80 -> 80; the rest of the cards barely moved (a few field changes per 100 pages, mostly borderline answers moving either side of 0.8). The one change per set is a borderline page (kind 0.48–0.51, or list vs one). | Fewer questions, same quality; the cancelled, weekly-course and price fixes address dry-run findings | data/runs/2026-10-05-prune2-{tune100,heldout,dry150,app} vs -tune100-facts, -heldout-facts2, -dryrun-4073, -card-facts3 |
| venue-missing-1 | The dry run's biggest gap: "Venue not found" on 25 of 100 random listed cards. Two general causes, both fixed in reader/page.py / reader/ask.py: (1) the place was never offered — it sat inside a longer line ("Venue: Olivier Hall, …", "31 Jan | Sheldonian Theatre, Broad Street; Barbican Concert Hall", "This talk is in the Headley Lecture Theatre and online") or past the 25 options on a long page behind cookie-banner text. New venue_phrases offers labelled places, "a | b" parts with a place word, and places after at/in in sentences, first; banner/button text and bare place words ("Museum", "Theatre") are no longer offered; "At the X" -> "X", "Centre's Jerusalem Room" -> "Jerusalem Room"; more place words (arboretum, quad, lecture theatre…). (2) Jev picked the right place but neither the pick nor the second look was sure (often a venue's own site, where the page never names the place): when both lean yes (>= 0.6) the venue counts as sure (venue only) | Venue filled, same pages: tune 68 -> 76, held-out 115 -> 138, dry-run sample 105 -> 125, app 86 -> 93. Of the 25 dry-run misses, 16 now filled (4 pages give no venue at all). Where the page has its own JSON-LD place, agreement unchanged (all "disagreements" are the same place written differently). Independent agent check of all 89 new/changed venues: 79 correct (89%), 5 right but messy, 4 wrong, 1 unclear; fixed since: bare "Museum"/"Theatre" (2 wrong), two places joined, "Centre's" |
| dates-collapsed-1 | The dry run's second gap: several dates or times collapsed (31 date/time issues on 22 of 100 random cards). General fixes: (1) a range whose year is written once ("Tue 6 Oct - Sat 10 Oct 2026") gives the first date that year too (values.py _RANGE_YEAR; it read 6 Oct 2020 from the weekday); (2) one end of a written range brings the other in (ask.py choose_dates, same source words), so "28 Nov 2026 to 3 Jan 2027" no longer shows its last day as the date, and a range reads "Runs until"; (3) a new per-time yes/no (pick.txt starts) plus a reworded sessions question: a one-day event Jev is sure repeats that day shows "11am & 2pm" (no end time); a programme of activities does not; one clock read two ways (4am/4pm) is one time; one sure time fills an unsure start only on a one-day event whose pick agrees | Blind A/B check by an independent agent of all 32 date lines that changed across tune, held-out, dry-run sample and app (560 pages), unblinded after: new better 19, old better 8, same 5 (after fixing the 2 cinema cases it found: one sure time was another showing's). Now right: Pegasus x2, Dancin', Corn Exchange, Oxford Phil "11am & 2pm" etc.; New Theatre runs; Abbey retreat Wed 11 – Fri 13; IF Oxford 10am–6pm. Still wrong in new: 6 cards where Jev's other answers moved because the question set changed (Tadmarton county, a past/other-branch event, an end time, a rule-only group left out) — answer drift, not this code | Adding a question moves Jev's other answers a little; measured here as about 1 card in 100 worse vs about 3 in 100 better | data/runs/2026-10-05-dates3-*, scratchpad/when_review.jsonl (session) |
| titles-1 | Dry-run "(no title)" / wrong title (6 of 100 cards, 8 cases). General fixes in reader/page.py title_options: a heading ending in ":" joined with the next ("All Souls Seminars…: Davitt Moroney…"); the page's own og:title / <title> (minus the site name) offered, for pages with no heading for the event; a fixture line ("Under 18s vs. Wallingford") offered; cookie, landmark ("Breadcrumb", "Footer") and button headings left out; a heading with a date on the end also offered without it. App displayTitle cuts a trailing date ("…GALLERY TOUR 1 OCT 2026") since the card shows the date on its own line | One-page tests: 4 of the 5 retested now right (Holywell, Blavatnik, Yogalina, Ashmolean); the fixture page still unsure (title is a "Match Overview:" line). Measured at scale in dryrun-2 | data/runs/2026-10-05-titles-one | |
| dryrun-2 | Readiness check: all 4,070 dry-run pages re-read with every fix so far (prune, venue, dates, titles; $1.42), then a FRESH random 100 listed + 50 left-out cards (none looked at before) reviewed by 5 independent agents against the page text, same rubric as the first dry run | Serious errors on listed cards 16/100 -> 16/100 (13 excluding a booking-fee bug from question-prune-1, since fixed: a ticket price picked again as the fee, "£10 + £10 booking fee"). Clean cards 29 -> 31; serious-or-missing 49 -> 50; real events wrongly left out 7/50 -> 3/50. Field fill rose (venue 81% -> 88%, title 94% -> 95%) but the serious-error rate on a fresh sample did not fall. Remaining serious causes: wrong date on multi-date/series pages (8), listed when out of area / online / not an event (4), wrongly left out (3), price (3 = the fee bug) | Fixing patterns found in one sample does not carry to the next: the long tail moves. Not ready by the < 5% bar | data/runs/2026-10-05-dryrun2, scratchpad/dry2_e*_{items,issues}.jsonl (session) |
| stop-guessing-1 | Founder rule: an event is listed only with a date AND a start time. Plus "stop guessing" (app only, no re-run): lib/browse.ts writtenDate lists only a date the page writes (a computed weekly next session or "every Friday" rule date becomes the first written date to come, else none; a daily run on today keeps today, its range being written); lib/listing.ts hides "No start time found", "Online only", and needs positive evidence of Oxfordshire (inOxfordshire: in-county postcode, an Oxfordshire place in venue/address, Jev >= 0.9, else the site's home area from its other events — oxonSites: >= 2 in-county events, 3:1 over out-of-county); "Weekly until" dropped, "Runs until" only for a written range, else "More dates on the venue's page" | Re-scored on the dryrun-2 fresh sample already marked by reviewers: serious errors on listed cards 13/100 -> 6/82 (7%); hidden: 7 serious-error cards, 9 with minor issues only, 2 clean (an online livestream, a ride with start point TBC). All 4,070 pages: listed 1,890 -> 1,487 (no start time 502, area not evidenced 385, online 90). The 6 left are Jev picking the wrong date on multi-date pages, at confidence 0.89–1.0 — no threshold separates them from right ones | Halves serious errors at the cost of ~20% fewer listings; still above the 5% bar; the rest needs a different date question, not a threshold | scratchpad/rescore.json (session) |
| date-pairs-1 | The general date fix: Jev confirms each written date-and-time pair for the event instead of picking one date and one time separately. values.py pair_candidates pairs every date (today or later) with each time written in the same text, 2 before or 6 after (time matched as written, incl. "7.30 p.m."; "Sat 6 & Sun 7 Feb" now reads both days); pick.txt when asks yes/no per pair; ask.py choose_occurrences: the sure pairs are the event's occurrences (the next is the card's date and start, the rest listed with their own times; others Jev leans yes to, >= 0.5, listed once one is sure). Stop-guessing rules with it: a date and time Jev firmly rejects as a pair is not shown; when Jev leans yes to some pairs and the separate picks match none, neither is shown; the end of a written range that began before today is not shown as the next date (also in app writtenDate). App: occurrences, "11am & 2pm" from same-day pairs, "Also on Sun 7 Feb, 3pm" | Blind A/B by an independent agent on all 27 date lines that changed across the 300 reviewed pages, unblinded after: new better 15, old better 11, same 1; wrong 11 -> 6. The 6: 4 real events now NOT LISTED where the old card showed a wrong date or time (Kendrew Barn, Blavatnik, Dancin', TORCH: wrong shown -> not shown), 1 not listed that was nearly right (Yarnton, Jev rejected 9:45 against "9:30 arrival"), 1 course showing its last session (Sunningwell; fixed since). Old-only wrongs fixed: Old Fire Station Sat 7:30/Sun 3pm, North Wall 27 Nov – 3 Jan, Ayres House 28 Oct, mosque sessions at other venues dropped, two cinema times now shown | Wrong dates shown fall sharply; the cost is a few real events not listed when the page's date or time is ambiguous — the founder's rule. Next: measure on a fresh sample | data/runs/2026-10-05-pairs2-300, scratchpad/pairs_review.jsonl (session) |
| answer-key-1 | Built the locked answer key (data-quality plan step 2): 150 never-used pages (75 tune sites, 75 held, 2 max per site, seed 20261007) labelled by a sonnet pass per ~/oxfordcompass-keys/card-key-1/GUIDE.md, key fields re-labelled independently by haiku, the 42 disagreements settled by sonnet against the page (41 sided with the first pass). Locked outside the repo ($KEYS_DIR/card-key-1/key.json, read-only, sha256 in LOCKED). New scorer app/scripts/score-cards.ts builds cards with the app's own rules and prints counts + failing ids (never key values) | Baseline (run 2026-10-07-key-baseline, current code): listed 56, serious errors 9 (16%) — tune 2/23 (9%), held 7/33 (21%); coverage 48 of 56 real events (86%). Dates and times: 0 wrong on listed real events. The errors are almost all listing decisions: 8 listed that should not be (3 members/invite-only, 3 with only a doors or no start time, 2 list pages), 1 venue with date text glued on; 8 real events left out (mostly "no date/start time found") | The answer key says the date work landed: the remaining problem is deciding WHAT to list (private, doors-only, list pages), not reading dates. The haiku second pass was careless (most of its disagreements were its own errors); a sonnet pair is worth the cost for future keys | ~/oxfordcompass-keys/card-key-1/, data/runs/2026-10-07-key-baseline |
| listing-decisions-1 | Fixed listing decisions, tuned on the answer key's tune half only: a sure doors-only time is not a start time ("Only a doors time, no start time"); "not open to the public" hides from 0.5 (was 0.9; the question only says No when the page says members/invited only). Tried and reverted (no gain on the key): "already happened" needing 0.9 (it relisted a self-contradicting page with a wrong date), and a direct "does the page say when it starts?" question (held 6 -> 5 but tune 1 -> 2 from answer drift; overall unchanged) | Scored on card-key-1: tune 2/23 (9%) -> 1/21 (5%); held 7/33 (21%) -> 6/32 (19%); all 9/56 (16%) -> 7/53 (13%); coverage 48 -> 47 of 56. Held left: 3 listed with no real start time, 1 list page, 1 not public, 1 messy venue | The tune half reached the bar but the held half barely moved: the tune half had few of the held half's error types. At 26-30 real events per half, one card is 3-4 points, so small rule changes can't be told apart from noise. Next: per-site checks (step 3) and a bigger key | data/runs/2026-10-07-key-baseline, -key-start (reverted) |
| site-health-1 | Per-site check (plan step 3), code only, no agents: scraper/audit/sites.py --run <id> judges each site across all its pages — agreement of Jev's sure date / start / venue with the site's own JSON-LD (a 1h JSON-LD slip counts as agreement), and repeated values (the same date+time on most pages; the same title on most pages when it is the site's name or the dates repeat too). Writes data/audits/site-health.json; app/lib/site-health.ts leaves out events from "needs_work" sites in Browse, Check, event pages and the scorer (--site-health) | On the 4,070-page dry run: 8 of 274 sites need work (148 pages), e.g. a museum whose every page read the same sidebar date and title, a centre where dates agreed with its own data on 2 of 14 pages. One false flag seen (a venue named two ways). Answer key: held 6/32 -> 5/31 serious, tune unchanged 1/21, no real events lost; all 7/53 -> 6/52 (11.5%) | The site check catches whole sites that read wrongly, cheaply, but only 5 key pages sit on flagged sites; the remaining errors are page-level listing judgements (no real start time, list pages, private). Signals need sites with JSON-LD or repeats to judge at all | data/audits/site-health.json |
| listing-decisions-2 | Three listing fixes. (1) Start time only from a confirmed date-and-time pair: tried strict (hide when pairs were offered and none confirmed) and soft (also the picked time is in no offered pair) on the key's tune half with existing answers — strict listed 13 of 26 real events (was 20), soft 16 of 26, neither removed an error; removed. (2) what gains "form" (a booking/enquiry/hire form), hidden as not an event. (3) pair_candidates: a date with no time nearby is paired with every time on the page when the page has 4 or fewer (tables, a time far from its date) | Re-run after the credit top-up (2026-10-07-key-listing2, $0.06): the fixes work mechanically — pages offered date-time pairs 82 -> 97, confirmed pairs 40 -> 43, one booking form recognised (conf 1.0) — but the key's scores are identical: tune 1/21, held 5/31, all 6/52 (11.5%), coverage 47/56. The new pairs landed on events already listed, and the form page was already left out | Kept (no drift, no loss, more dates confirmed). The key's remaining errors are not these kinds: on 150 pages the next gains need a bigger key to even see | data/runs/2026-10-07-key-listing2 |
| key-300 + missed-dates-1 | Answer key grown to 300 pages (card-key-2, provisional: 150 new pages labelled in one careful pass from trimmed page text, ~0.65M agent tokens; 30-page spot-check stopped to save tokens). Baseline (2026-10-08-key300): listed 95, serious 11 (11.6%), coverage 86/105, 0 wrong dates/times. Then the missed dates and times, causes found on the tune half only: right values picked just under 0.8 with the second look leaning yes. Fix: the "pick and second look both >= 0.6 count as sure" rule, until now venue-only, now applies to every field with a second look (date, start, title) | Tune: listed real 37 -> 39 of 51, errors 4 -> 4. Held (untouched): 49 -> 50 of 54, errors 7 -> 7. All: coverage 86 -> 89 of 105, serious 11.6% -> 11.2% | General, not a patch: the held half improved too with no new errors. Remaining misses are pages that contradict themselves (left out on purpose) and plain Jev misreads | data/runs/2026-10-08-key300-lean |
| wrong-listings-1 | Wrong listings, causes found on the key's tune half only: 2 of 4 were pages Jev called a list at 0.98 that stayed listed because "several dates" was also sure (an exception meant for one film's showings). Fix: kind=list at >= 0.95 hides even with sure multi_dates. The other 2 were one-offs (a course with no dated session, a venue spelled another way) and were not patched | Tune: serious 4/42 -> 2/40, coverage 39/51 unchanged. Held: 7/56 -> 7/56, coverage 50/54 unchanged. All: 11.2% -> 9.4% | No harm on either half, but the held half had nothing for it to act on, so not yet shown to be general. Held's remaining wrong listings are other kinds (no real date or start time, private) | data/runs/2026-10-08-key300-lean |
| wrong-listings-2 | Re-split the 300-page key by site (seed 20261009) so both halves hold every error type (held half no longer fully unseen; original split kept as split_orig). On the new tune half: positive evidence of ONE event (an answered but unsure kind is not listed), "probably members only" (audience leans members >= 0.5 and public not sure), date/time text cut off the end of a venue. Scorer fix: venues match when their names (first comma part) match ("Summertown Arts, Oxford" vs "Summertown Arts, 294A Banbury Road") — a measuring bug, it changes no card | New tune: serious 6/44 -> 1/40 (part of that is the scorer fix), coverage 40 -> 39 of 48. New held: 3/52 -> 3/52, coverage 49/57 unchanged. All 300: 9.4% -> 4.3% (4 of 92), coverage 88 of 105, 0 wrong dates/times | Under the 5% bar on the key for the first time, but the held half did not move again, and it was partly seen before the re-split: confirm on fresh pages before calling it ready | data/runs/2026-10-08-key300-lean |
| fresh-proof-1 | Spot-check of the 300-page key: a second labeller agreed on listable 29/30 and on every date-time of the 11 events both listed (the one difference: an event booked through library membership) -> card-key-2 locked. Final proof: 100 fresh pages from 100 different sites, never looked at (card-key-3, one careful labelling pass from trimmed text), scored with every fix so far | Fresh: 35 real events; listed 27, scorer says 4 serious (14.8%) — but 2 are scorer strictness on venues ("Main Hall" for West Oxford Community Centre = room without building; "Oxford Town Hall, Museum of Oxford" is right, the museum is in the Town Hall), so 2 real (7.4%): a dance event with no location, a village-hall event with no start time. 0 wrong dates/times. Coverage only 25 of 35 (71%): left out by caution (members-only guesses x2, area not evidenced x2, two good events on "needs work" sites, an unsure page kind, library membership, no date/time found x2) | On truly fresh pages: about 7% wrong (2 of 27; with 27 cards the range is roughly 1–24%) and we miss ~3 in 10 real events. The caution rules now cost more coverage than they save errors on fresh sites | data/runs/2026-10-08-fresh100, ~/oxfordcompass-keys/card-key-3/ |
| loosen-caution-1 | Tried loosening each caution rule that cost real events on the fresh 100, scored on the 300-page key and the fresh 100 together (rules only, no re-run): "probably members only" at 0.8 (was 0.5); "one event" at 0.6 (was 0.8); Jev "in Oxfordshire" yes at 0.8 (was 0.9); site check without the venue signal; no site check at all | Members 0.8: +2 real, +2 wrong (fresh 4 -> 6 wrong). One-event 0.6: +1 real, +1 wrong. Oxfordshire 0.8: no change. Site check without venue: fresh +1 real, key-300 +2 wrong. No site check: key-300 +2 real / +2 wrong, fresh +2 real / 0 wrong. None kept; thresholds now named constants (MEMBERS_AT, ONE_AT, OXON_YES_AT) | The caution rules are paying their way: each loosening lets in about one wrong card per real event it recovers. The misses are the price of the error rate, not slack — recovering them needs better answers (reading), not lower bars | — |
| strip-chrome-1 | Showed Jev only the event's part of the page: reader/page.py strip_chrome drops the page's own landmarks (menus, header, sidebars, pop-ups, footer; footer postcode lines kept as address evidence). 82-85% of pages mark menus/header/footer. Re-ran all 400 key pages twice ($0.25) | Full strip: key-300 serious 4/92 -> 7/85, real listed 88 -> 81; fresh 4/27 -> 5/25, 25 -> 22. Mild strip (menus, pop-ups, footer only): key-300 8/90, 84 real; fresh 5/27, 24 real. Lost cards were mostly "no date / start time found": many sites put the date box or showtimes in a header or sidebar | Worse in every form, so switched off (STRIP_CHROME=0; code kept). Page landmarks don't reliably separate the event from the rest | data/runs/2026-10-08-key300-strip*, -fresh100-strip* |
| sure-none-1 | Looking at the wrong times on the key: Jev already receives the page as its outline with nesting kept (not flat text), and on the doors-only page it answered correctly — start "none" 0.88, time_kind doors 1.0, every date-time pair and start time "no". Our own structured.fill_gaps then filled the start from the page's JSON-LD (the doors time) because it counted a sure "none" as open. Fix: a sure "none" is not open; no JSON-LD start when Jev is sure the time is doors/opening hours/last entry or rejected every pair | key-300: serious 4/92 -> 2/90, real listed 88 unchanged. Fresh 100: unchanged (4/27; its wrong times come from elsewhere). Re-run was cached ($0) | Some "Jev errors" are our code overruling Jev; check the pipeline's own steps before blaming the reader. Structure is already in Jev's input | data/runs/2026-10-08-key300-fill |
| override-audit-1 | Audit of every step where our code changes or adds to Jev's answers: each switched off in turn (AUDIT_OFF=fill,lean,siblings,sessions,onetime,conflict,pairs; cached re-runs, $0) and scored on key-300 + fresh 100 | Today: 2 wrong / 88 real; 4 / 25. Gap-filler off: 1 / 86; 3 / 24 — but its "errors" are a cinema showtime the labeller may have missed and a correct venue the scorer is strict on, while it adds 3 real events: keep. Lean promotion off: -3 real: keep. Sessions, one-time fill: -1 real each when off: keep. Contradiction blanking: no measured effect, kept as a safety net. Range siblings off: +1 real on fresh, no error change: neutral. Pairs off: 3 / 88; 3 / 23: mixed, keep | After the sure-none fix no override step is a net source of wrong cards. The remaining wrong cards are Jev confirming a wrong pair (Holton 12:00), a course listed as one date, and debatable labels | data/runs/2026-10-08-audit-* |
| year-and-rules-1 | Missing year, measured before changing anything: a date with no year already gets the coming year (Duckling), and Jev is about as sure on it (91% of 1,126 dryrun-2 cards) as when the year is written (93% of 2,210), so "assume this year" was already in place. Changes: the year note is a bold amber box ("Year not stated on the page — we've assumed 2026. Check with the organiser before you go."); lib/checks.ts 1b drops a date written without a year that lands more than a year ahead (the year came from the weekday, i.e. last year's page: "Saturday 4th October" -> 2031); lib/listing.ts splits "No date found" into no date / only a repeating rule / dates on the page, none confirmed / its date failed a check; reader/values.py rule_dates reads a year and "end/mid/late" in the month window ("Every Fri & Sat during Oct 2026 – end Apr 2027" gave October only, then Oct 2027) | All 4,880 cards (final-app, final-review, dryrun-2) before vs after: 0 change listed/unlisted or their date; only reason wording (final-review: 51 -> "none confirmed", 25 -> "only a repeating rule"). All 291 rule sentences in data/crawl: 2 change, both fixes ("every Wednesday during September 2026" no longer adds Sep 2027). Held back on final-review: 437/700 (no date 177, already happened 130, not confirmed Oxfordshire 116, not sure one event 82, no start time 66) | The rule-reader change takes effect on the next Jev run. Rule-only events stay hidden (stop-guessing-1) until Arjun decides whether to list them with the warning | scratchpad (session) |
| oxford-area-1 | "Not confirmed to be in Oxfordshire" holds back 116 of 700 final-review cards (16 for that reason alone). Looser rules measured on key-300 + fresh-100 (in_oxfordshire labels; tune vs held) and every card they flip on final-review and dryrun-2. Address/venue at >= 0.5: +9 in-county on the keys, 0/26 out-of-county, but on the runs it let through a page about an Edinburgh event via the site's footer address, and the site-home fallback cascaded (a dance company's London and Milton Keynes events passed). Venue only, >= 0.5, never overriding Jev's sure "no" and never counted towards a site's home: +9 in-county on the keys, 0/26 out; 28 flips on the runs, all genuinely in Oxfordshire | Correct but worth nothing today: 0 cards newly listed on any run — every card it fixes is also held for another reason (no date, members only, a list, not an event). The 16 area-only cards are pages with no venue at all (a city farm, a cycling club) or venues that don't name a town ("Pegasus Theatre", "Mill Theatre"), which need a venue gazetteer, not a looser threshold | Not shipped (reverted); the rule is kept in the session scratchpad if fresh data changes the picture. Cheap test first, as with the time-outlier check | scratchpad (session) |
| dog-only-1 | Is the dog needed? Per site (707, 1 Oct corpus): did the scraper save any event pages (crawl manifests), and does the dog's note list a source with upcoming events (future > 0)? | 381 sites: the scraper found event pages (108 of them also have dog data, so the dog is a cross-check there). 31 sites: no event pages, the dog is the only way in — 6 calendar files (adderbury-village-events, littlemore-village-hall, oxford-poetry-library, …) and 25 feeds / page data (o2-academy-oxford, st-aldates-church, oxford-clayhouse, outdoor-cinema-oxford, rugby and cricket fixtures, …). 11: an embedded calendar neither reads. 263: nothing upcoming either way. 21 not scraped | Dropping the dog loses at most 31 sites (8% of the 412 with events), incl. O2 Academy. The dog's "upcoming" counts aren't checked, so 31 is an upper bound | Keep it narrowly as the source for those sites, or replace it with a reader for just them; for the other 381 Jev reads the pages |
| new121-1 | Arjun's 121 new sites (1 Oct corpus; 59 with saved event pages, 538 pages) re-run with today's questions: rerun --sites-file … --run-id 2026-10-15-new121, $0.19, 0 errors. Listing rules applied, compared with Arjun's 2026-10-01-new120 (older questions, no date-time pair questions) | Listed 200/538 (was 184). "No date found" 208 -> 83: the new questions find the dates, and most of those are past (already happened 81 -> 161). 28 of the sites have a listed card. Spot check, 15 random listed cards vs their saved pages: title, date and start time right on all 15 (venue blank on 2) | The rules hold up on sites never tuned on. No answer keys for these sites, so 15/15 is a small sample, not a measured error rate | data/runs/2026-10-15-new121 |
| site-questions-2 | Website questions for every site (Arjun, 15 Oct call: the website screen "should be a lot better"): only 93 of the app's 293 sites had website answers and 20 had a screenshot. scraper/site_sniff.py asks the 44 questions in questions/site.json (exported from app/lib/site-questions.ts) about each site's saved events page; scraper/site_shots.py renders each saved events page offline (network blocked, small built-in stylesheet); the site page gains a "This website at a glance" summary (sure answers stated, unsure greyed with %, flags for a person) above the raw answers | 672 of 707 sites answered for $0.16 (35 have no saved page); all 293 app sites now have answers; 670 of 707 have a screenshot (649 rendered offline + 20 live), 208 MB in data/. Offline renders lack the site's own styling and pictures | Website-level answers now cover the fleet, so questions like "is this a once-a-year event?" can be added and read per site. Real-looking screenshots of homepage/about pages need a live fetch (contact address + go-ahead) | data/app/site-answers.json, data/app/site-shots.json |
| brute-force-1 | Arjun (14 Oct): "add as many questions as you think, whittle them down later". Event questions 50 -> 87: the 16 pruned in question-prune-1 reinstated (ticket_link, image, support, sold_out, waitlist, series, multi_day, all_day, access_perf, parking, …) plus 21 new (annual, season, booking_deadline, places_limited, pay_on_door, free_for_kids, duration_given, performers_named, organiser_named, bring, step_free, accessible_toilet, relaxed, dogs, food_drink, language, weather, getting_there, refund, photo_id, newsletter), appended to the end of the files. Website questions 44 -> 57 (once_a_year, event_months, last_updated, regular_programme, many_organisers, venue_pages, hire, newsletter, social_only, contact, accessibility_info, tickets_on_site, reviews) | Arjun's 121 new sites (538 pages), same pages as 2026-10-15-new121: listed 200 -> 199 (2 off, 1 on, 0 listed dates/times changed) — answer noise from a longer request, ~0.5%; one real loss (international-piano-series/event_10 lost its 19:30 start). Cost per page +22% (9.3k -> 11.3k tokens), $0.25 for 538 pages. Websites (672): once a year 28, one short period 22, regular programme 176, tickets on their own site 6 | Brute force costs little and barely moves listings. Nothing on the card uses the new answers yet; trim later by measuring which ones are ever sure and useful | data/runs/2026-10-16-bf-new121, data/app/site-answers.json |
| REFETCH-2026-10-10 | Can the saved sites be fetched again and compared, page by page and event by event, with what we hold? | 733 sites looked at, 705 answered. 12,353 event pages saved (cap 60 per site, listing links to 400, ticket-seller pages followed under their own robots.txt), 12,297 read into events, 2,662 public (5 Oct export: 7,885 events, 2,100 public). Events in both: 6,193, of which 1,610 differ in some field. Noise floors measured: page text fetched twice the same night differs by under 3% of lines (59 pairs, max 2.5%); on pages with IDENTICAL text 17.7% of events still differ in some field (place 7.3%, start 5.2%, public 2.7%, title 2.1%, price 0.9%, sold out 0.2%). Of 299 still-to-come events no longer on their site's list, 269 pages were still up and 12 gone. 88 sites print 3+ upcoming dates on the listing page and have no event links. The date parser lost dates for whole 40-page chunks under machine load (57% of the newer-question answers on the first pass, 0 after three retries) | A re-fetch works and resumes, but "off the list" is not "cancelled", and field changes below the identical-text floor are the reader, not the venue: only sold out, price and last date are clean change signals. Listing-only sites remain unread by any step | data/corpus-2026-10-10/ (compare.json, sites_table.csv, empty_sites.json), data/export-2026-10-10/compare_events.json, scripts/corpus_*.py, dist/corpus-2026-10-10.zip |
| question-tree-1 | Arjun, 16 Oct: 40 -> 400 questions, hierarchical. Question tree (questions/tree_build.py): 71 event facts from 468 small yes/no questions (Jev Nouls), combined in code (scraper/tree_sniff.py) as the Jev docs advise; tested on 100 events from the 10 Oct re-fetch (one per site), then compared with the flat 88-question set on the same 100 pages | First pass: 28% of facts "unsure", mostly broad gates ("does the page mention places for the event?" is always yes) with every specific sign a sure no; one bad question ("last year or next year") made 2027 events "annual". Fixed: narrower gates, the gate wins, every yes-sign a sure no -> "not stated", family / drop-in / beginners rewritten: unsure 28% -> 7%. Lost "yes" answers were mostly wrong before (a lunchtime recital "for beginners"). Tree vs flat on the same 100 events: 114 sure facts per event vs 49.5; where both are sure they agree on ~450 of 453 comparisons (free 56/56, refund 95/95, booking 63/63, category 72/83); tree $0.046 vs flat $0.055 per 100 pages (flat needs a second follow-up request). Tree's "not stated" vs flat's "no" for "not mentioned" accounts for most one-sided answers (parking 93). Glass-fusing workshop: tree says pay-what-you-decide (page: suggested £7.50), the 10 Oct export said free | The tree gives about 2.3x the facts for less money and agrees with the flat set where both answer. It gives yes/no facts only: the flat set's picks (the actual date, time, venue text, price amounts, ticket link, image) are still needed, so the tree adds to the flat set, it doesn't replace it | data/tree-runs/tree-100events-v2, data/runs/2026-10-16-flat-100 |
| question-tree-2 | Question tree scored against the answer keys for the first time (card-key-2 + card-key-3, 400 pages, scraper/tree_score.py; tuned on tune, scored on held). Then three facts improved on tune: public (gate widened to "or invites people to book", 3 more yes-signs), age given (all ages / must come with an adult), in Oxfordshire (Oxfordshire towns and villages, colleges/museums) | Right when decided, held (tune): in Oxfordshire 100% (100%), free 100% (100%), still to come 100% (98%), start time given 98% (96%), more than one date 94% (90%), real event 96% (91%), kind of page 93% (98%). After the fix: public decided 27% -> 89% at 91% right (tune 25% -> 80%, 94%); age given 80% -> 92% right; in Oxfordshire decided 77% -> 86% at 100%. Scoring trap: the keys leave occurrences empty for past and other unlisted pages, so start time is only scored where the key records it (was showing a false 68%) | The tree is accurate where it commits; the work is now making it commit more often. 400 key pages cost $0.19 per run | data/tree-runs/tree-keys-v2, tree-keys-v3 |
| question-tree-3 | "Real event" and "kind of page" left about a quarter of key pages undecided. Every undecided "real event" page was a non-event whose gate (does it describe something to attend?) was a sure no, so the answer was "not stated"; for this fact that means "no" (new per-fact gate_no), plus 3 more no-signs. "Shop" now excludes event tickets; "past archive" means a list of past events | Held (tune): real event decided 76% -> 93% (76% -> 93%), right 98% (98%); kind of page decided 73% -> 86% (72% -> 84%), right 95% (99%). Other facts within a page or two | Decision rates now 79-100% on held pages, right 91-100% | data/tree-runs/tree-keys-v4 |
| question-tree-big-1 | First big run of the question tree (478 questions, 71 facts): all 2,662 public upcoming events (start >= 10 Oct) in Arjun's 10 Oct export, from 260 sites, with today = 10 Oct | $1.25, 245 s, 0 errors. Facts: 22% yes, 8% no, 62% not stated, 8% unsure. Real event 2,556, still to come 1,969, start time 2,352, in Oxfordshire 2,350, public 2,085; booking needed 1,556, ticket link 1,097, free 315, pay what you can 64, sold out 69; family 428, kids 109, adults only 200; wheelchair 82, hearing 108, relaxed 28; food 312, bar 216, parking 133; Christmas 204, Halloween 50, half-term 87; top categories community 596, music 594, talk 344, family activity 320. Flags against Arjun's "public" list: 4 not a real event, 9 already happened, 14 outside Oxfordshire | Facts for every public event for about $1.25 a run. answers.json is 109 MB (every small answer); facts.jsonl (facts only, with title and link) is the file to share | data/tree-runs/tree-public-2662/{answers.json, facts.jsonl} (not in git) |
| rescue-1 | Why only ~2,186 of 12,297 events (10 Oct export) are publishable, and two fixes for "no sure date" / "no sure title". 585 events are held for only those reasons. Date: scraper/date_check.py asks 6 small yes/no questions about the one picked date (3 yes-signs, 3 no-signs: another event's date, a publish/booking date, a past edition), only where the picked date is upcoming and 0.2-0.8 sure. Title: scraper/title_fallback.py takes the page's level-1 heading, else og:title / <title>, never the organisation's name, generic headings or calendar headings | Key pages first (only 8 fit): 2 confirmed, both right; all 5 wrong dates stayed unsure. 10 Oct: 218 checked, $0.02 -> 53 confirmed (spot-checked: dates written as ranges, e.g. "27th - 29th August 2027"), 78 rejected, 87 unsure. Title fallback: 110 "won back", but 45 are Big Feastival artist pages (one festival, not separate events), 12 Henley Choral Society with junk titles ("422440 … Poster Spring 2020"); realistic good titles ~30-40. Net: ~80-90 more publishable, ~60 still upcoming on 19 Oct (+3-4%) | Dates and titles are not the main reason the count is low. Coverage is: 268 of 733 sites gave no event pages (listing-only and calendar-only sites), and the 40/60-events-per-site cap. Title fallback is not wired into listing until festival-lineup and junk titles are handled | session scratchpad |
| meta-question-1 | Arjun's meta question (19 Oct): scraper/meta_sniff.py, a second Jev pass with the page + the tree's 71 answers. It asks "is this answer wrong?" for each decided fact, "now is it true?" for each unsure fact, and 15 yes/no gap questions ("does the page give <detail> the answers don't record?"). Jev can't write free text, so "what did we miss?" has to be yes/no | 10 tune-split key pages from tree-keys-v4 (7 with a tree mistake), $0.0013 total (~3,000 tokens and ~46 questions per page). 8 flags, all on facts the keys don't score; checked by hand, 7-8 of 8 are real tree errors: "Hearing loops? No" read as hearing support (x2), a footer "Accessibility Statement" link read as access info (x4), a running time read as an end time, a timed workshop read as all day. 0 of 48 right key answers wrongly flagged. Caught 0 of the 3 key-scored mistakes (several dates x2, a start time). Settled 1 unsure fact, correctly. Gap questions: only "other" fired (4 pages), so the specific gaps add little | Good as a cheap error-finder for the tree's own questions: it shows which tree questions to fix (hearing, access_info). Not a fix for core-data mistakes. ~$1.60 for all 12,297 events | data/tree-runs/meta-10 |
| question-tree-4 | Fixed the two tree questions the meta question caught (meta-question-1). hearing: yes-signs now need the support to EXIST ("there IS a hearing loop") plus no-signs for "Hearing loop: No" / no BSL or captions. access_info: gate and yes-signs limited to the main part of the page, with no-signs for "only a menu/footer link" and "only the website's own accessibility statement"; "access contact" now means a contact for access needs, not any contact details (that question said yes to every page with a phone number); two plain yes-signs added ("says whether it is wheelchair accessible / step-free, either way", "tells people with access needs what to do") | 173 key pages (every page where the old tree answered either fact, plus 20 others), v4 -> v8, $0.08 a run (4 runs, ~$0.27). access_info "yes": 142 -> 22. By hand, ~17 of the 22 are real (was ~17 of 142, 12% -> ~78%), and 16 of 17 pages whose main text has access details still say yes. Still wrong: menu links "Visit & Access", a screen-reader widget. hearing "yes": 14 -> 1, and the 13 dropped were all wrong ("Hearing loops? No" x2, "Subtitled" on a different film in the other-films list x3+, opera surtitles). Other facts moved on 390 of 11,937 answers between runs: Jev's run-to-run noise, not this change | Next tree run should use the new questions (tree_events.json now 484 questions) | data/tree-runs/tree-access-v5..v8 |
| question-tree-5 | Fixed has_end and has_doors, the next facts the meta question flagged (meta-100: 7 and 3 flags). has_end counted a duration as an end time (a yes-sign said "finish time OR a duration"), and also picked up venue opening hours and video lengths. Now: a clock end time or a start-end range, with no-signs for "only the venue's opening hours", "only a running time" and "only a video length". has_doors: now only a stated doors time, with a no-sign for "only a start-end range". Two signs I tried were dropped because Jev said yes to them when the page had nothing of the kind ("is the only 'until' a closing date?", "a separate arrival or entry time?") | All 400 key pages, v4 -> v10, $0.20 (plus $0.20 for the v9 try). has_doors "yes": 62 -> 18, still 12 of the 12 pages with a written doors time. has_end "yes": 227 -> 193; "yes" with no time range on the page 60 -> 33; time ranges answered yes 167 -> 160 (some of those ranges are opening hours) | Tree is now 488 questions. Signs phrased "is the ONLY X…" are risky: Jev can say yes when there is no X at all | data/tree-runs/tree-keys-v9, -v10 |
| no-cap-1 | Removed the per-site event limits (decided on the 19 Oct call: "there should be no cap on events per site"). fresh.py (made the 10 Oct download): 60 pages/site, 1,200 total and the 30-min site budget are gone unless asked for; listing next/load-more guards raised to 200/100 (runaway stops for endless calendars, not caps). crawler.py: 40 events/site, 3 next pages, 15-min budget gone. In place of a cap, sitemap links are filtered by sitemap_keep(): past year in the link, not updated for a year, or on a shared ticket site (DesignMyNight, ATG) a link not naming this venue | 10 Oct discovery data: the 60-page cap cut only 14 listing links (one site) but 160,000 sitemap links; 129,142 not updated for a year, 8,645 a past year, 17,914 other venues on shared ticket sites (DesignMyNight lists Aberdeen, Manchester...). Kept: 8,670 listing + 8,971 sitemap links (~17,600 pages vs 13,518 fetched). Biggest gains: oxford-life-drawing 1,098, pump-house-project 848, tcg-coffee 696, merton-college 480, ultimate-picture-palace 355. Tests by a separate agent: tests/test_no_cap.py, 139 pass with test_fresh/test_crawler | Takes effect on the next live download only (live fetching still off; needs a real FRESH_CONTACT). Known edge: an own-site listing under a /venues/<slug>/ path is treated as a shared host | data/corpus-2026-10-10/batch-*/discovery.json |
| storage-1 | Where to store downloads, and do we need the screenshots? (19 Oct call, task 8) | 10 Oct download = 2.85 GB: screenshots 2.03 GB (71%, 12,425 JPEGs, avg 163 KB, 1366 x ~2,800 px), page HTML 0.70 GB (24%), page text 0.11 GB (4%, what Jev reads). Screenshots are used only by the review app and the oracle OCR check; the reader and the question tree read the text, and structured data comes from the HTML. Re-encoding 150 sample screenshots: WebP 800 px wide q60 = 43 KB (27% of now) and still readable; same size as WebP = 65%; JPEG 800 wide = 49%. 1 Oct -> 10 Oct, of pages fetched both times 3,978 were the same or nearly (same 2,725 + minor 1,253) and 2,150 changed, so most pages repeat between downloads. A no-cap download (~20,000 pages) would be ~4.5 GB as now, ~2 GB with WebP shots | Keep the text and HTML of every page version (cheap, and needed to re-ask new questions). Keep a screenshot only when the page's text changed, as 800 px WebP: it is the evidence of what the page said (sold out, cancelled), the input for OCR on poster-only events, and what the review app shows. Store unchanged pages as a pointer to the earlier copy. Store downloads in object storage (Cloudflare R2 or Backblaze B2), not git or the Pi; GitHub releases cap a file at 2 GB. data/corpus-2026-10-10 (2.9 GB) duplicates data/crawl-2026-10-10 and ~/Downloads/corpus-2026-10-10.zip | data/crawl-2026-10-10, data/corpus-2026-10-10/compare.json |
| storage-2 | Smaller screenshots and none for unchanged pages, in the download (fresh.py). Screenshots: taken as JPEG bytes, scaled to 800 px wide, saved as WebP q60 (falls back to the old JPEG if Pillow is missing). Unchanged pages: the new text is compared with the previous snapshot's copy (--prev, default data/crawl); under 3% of lines differing (the same-night noise floor from REFETCH-2026-10-10) means no new screenshot and entry "same_as" the copy that has one (chains collapse to one step). Text and HTML are still saved for every page. corpus_merge keeps the shot's extension | A real 105 KB screenshot -> 23 KB; 150-sample average 27% of the old size. On real data, the 10 Oct pages also in the 1 Oct download: 1,818 unchanged (no screenshot), 2,022 changed, 6,076 new. Tests by a separate agent: tests/test_fresh_shots.py (24) + updated test_fresh.py; 231 pass | Takes effect on the next live download. Write-up for Arjun: Download Storage artifact | data/crawl-2026-10-10 |
| meta-question-2 | The meta question again, on the same 100 tune-split key pages, now with the fixed tree (tree-keys-v10) | $0.016. Flags 44 -> 10. The fixed questions no longer get flagged (access info 24 -> 0, end time 7 -> 0, doors 3 -> 0). The 10 left are spread out, at most 2 per fact (has_date 2, bar 2, then 1 each for relaxed, is_real_event, concessions, age_given, all_day, adults_only) | No weak question stands out on this sample; the next one would need a bigger or different sample | data/tree-runs/meta-100-v10 |
| one-page-1 | One-page listing sites: split the listing page into one block per event, so each block can be read like a small event page. scraper/listing_split.py on the page's accessibility tree: drop banner/navigation/footer; the smallest repeated container (list item, article, row) holding a date is a block; a block with 3+ dates is cut again at lines opening with a date or day name, taking a short title line above with it; a flat page is cut at headings and date lines. No Jev, no site-specific code | The 88 sites Arjun found (3+ upcoming dates on the listing, no event links; data/corpus-2026-10-10/empty_sites.json) print 643 upcoming dates (median 5 per site). Split: 672 blocks; 71 sites give 2+ blocks, 10 give 1, 7 give none; 11 blocks are over 1,200 chars (still merged). Clean on pages with repeated containers (one block per competition, comedy night, talk, with date, time and place). Dates are not events: one orchestra page lists ~20 members' rehearsal dates and one public concert. Splitter rules came from 2 tune sites (a flat orchestra page, a museum page with all dates in one container); the output of 3 held sites was displayed while checking but nothing was changed for them. The 88 split 32 tune / 29 held / 26 unassigned | Next: read each block with the existing reader (title, date, time, public?) on tune sites only, then score on held sites. Tests for listing_split.py still to be written by a separate agent | /private scratch blocks88.json (rebuild with the command in the file's docstring) |
| one-page-2 | Read each one-page-listing block with the existing reader (scripts/read_chunk.py, today = 10 Oct). Blocks are written as small pages by listing_split.py --pages-dir; the block's title line (first short line with no date, time or button words, else a dated line with the date taken out) is offered as its heading, and the reader decides. Fixed on the way: read_chunk.py built fewer candidates than the current reader needs and failed every page with KeyError('titles'), which would also have broken the next bulk read | Tune (28 sites, 168 blocks, $0.09 over 2 passes): 72 look listable (event, sure upcoming date, not closed, sure title) from 19 sites. Held (27 sites, 243 blocks, $0.08, never looked at while building): 79 look listable from 19 sites. Hand check of 20 held listable: date right 20/20; title right 11, right but with the venue address stuck on 3 (title and place on one line on the page), wrong 6 (4 from one site's "5 / 20: Monday 9 November Open" table, "Noon" for an auction, "Zoom in" an image button). So ~150 more upcoming events from the 55 tune+held sites; the 26 unassigned sites not read yet | Dates are reliable; title choice is the weak part. Title fixes must come from tune sites (the held failures above are not to be tuned on). Tests for listing_split.py and the read_chunk.py fix still to be written by a separate agent | data/blocks-2026-10-10/, data/blocks-2026-10-10.{tune,held}.json; read output in the session scratchpad |
| one-page-3 | Better event names for one-page listing blocks, tuned on tune sites only. title_line now skips label-like lines (ending ':', in brackets, lower-case start, emails, long sentences, generic words like "Matches" / "Forthcoming films", the organisation's own name from sites.json) and, for a line holding both name and date ("Free Beer! Open Tap Tasting Evening – Wednesday 14th October"), keeps the parts that are not date, day, time or price. Also: date-then-title pages keep each title with its own date (bug found by the tester) | Tune: listable 72 -> 79. Held (scored, not tuned): 79 -> 83 listable. Same 20 held events by hand: dates still 20/20; "Noon" (an auction) now has no title and drops out; one site's class table still gives "Open" (x4) and an image button still gives "Zoom in"; others unchanged (14 of 19 usable). Unassigned sites read for the first time: 42 listable from 14 sites. All three splits: ~204 upcoming events from 50 sites that nothing read before, $0.28 in all | Names improved on the sites tuned on, barely on held: the remaining held errors are layouts the tune sites don't have. Dates are the reliable part. Tests: tests/test_listing_split.py, 34, by a separate agent | data/blocks-2026-10-10*, read output in the session scratchpad |