Models · Our evals
The Speedrunning AI Evalboard
Real work tasks, run on camera, every finding checked by hand. Five suites are defined; the boards fill as Prem records the runs. We never publish a score without its date, recording and write-up.
AP invoice extraction
0 runs · Full write-up
Extract 10 dummy supplier invoices (PDF and image, mixed layouts, two in INR with GST) into a sheet: supplier, GSTIN/VAT ID, invoice number, date, due date, currency, line items, subtotal, tax, total.
| # | MODEL + EFFORT | SCORE | VALID / PARTIAL / WRONG | RECALL | TIME (S) | COST | OUTPUT TOKENS | EVIDENCE |
|---|---|---|---|---|---|---|---|---|
| No runs yet: first runs land with Prem’s test videos. | ||||||||
Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.
SCORING RUBRIC
- Each of 10 fields per invoice scored valid (exact), partial (right value, wrong format) or wrong/missing
- Score = valid fields ÷ all fields × 100
- Recall = fields extracted ÷ fields present
- Totals must reconcile with line items; a mismatch counts as wrong
CHECKED BY HAND
- Every extracted field is compared with the source invoice by a person
- GST/VAT numbers and totals are re-added by hand
- Hallucinated fields (values not on the invoice) are counted as wrong and noted
Landing page build
0 runs · Full write-up
From a one-page brief for a fictional Indian D2C brand, build a responsive landing page (hero, three benefits, pricing in INR, FAQ, signup form) as a single deployable folder.
| # | MODEL + EFFORT | SCORE | VALID / PARTIAL / WRONG | RECALL | TIME (S) | COST | OUTPUT TOKENS | EVIDENCE |
|---|---|---|---|---|---|---|---|---|
| No runs yet: first runs land with Prem’s test videos. | ||||||||
Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.
SCORING RUBRIC
- 20-item checklist: every section present, works at 390px and 1440px, no horizontal scroll, form validates, accessible labels, Lighthouse accessibility ≥ 90
- Score = checklist items passed ÷ 20 × 100
- Time = wall-clock until the first deployable build
CHECKED BY HAND
- Opened in a real browser at both widths by a person
- Copy is checked against the brief for invented claims
- Keyboard-only pass through the form
n8n / automation workflow
0 runs · Full write-up
From a written spec (new form lead → enrich → score → route to a sheet and a Slack-style webhook, with error handling), produce a working n8n workflow export.
| # | MODEL + EFFORT | SCORE | VALID / PARTIAL / WRONG | RECALL | TIME (S) | COST | OUTPUT TOKENS | EVIDENCE |
|---|---|---|---|---|---|---|---|---|
| No runs yet: first runs land with Prem’s test videos. | ||||||||
Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.
SCORING RUBRIC
- 12-item checklist: imports cleanly, every node configured, runs end to end on 5 test payloads, errors routed, no hard-coded secrets
- Score = checklist items passed ÷ 12 × 100
CHECKED BY HAND
- Workflow imported and run on the 5 payloads by a person
- Outputs compared with expected rows
- Credentials and secrets inspected
Research brief with sources
0 runs · Full write-up
Write a 600-word brief on a fixed current AI question with at least 8 cited sources.
| # | MODEL + EFFORT | SCORE | VALID / PARTIAL / WRONG | RECALL | TIME (S) | COST | OUTPUT TOKENS | EVIDENCE |
|---|---|---|---|---|---|---|---|---|
| No runs yet: first runs land with Prem’s test videos. | ||||||||
Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.
SCORING RUBRIC
- Citation accuracy: each cited claim checked against its source — valid (supported), partial (source related but claim overstated), wrong (unsupported or source missing)
- Score = valid citations ÷ all citations × 100
- Recall = key facts covered from a fixed list of 10
CHECKED BY HAND
- Every citation opened and read by a person
- Dead links and fabricated sources counted as wrong
- Dates checked for staleness
Bug fix in a small repo
0 runs · Full write-up
Fix a seeded bug in a small open-source-style Python/TypeScript repo given a failing issue report; the hidden test suite must pass without breaking existing tests.
| # | MODEL + EFFORT | SCORE | VALID / PARTIAL / WRONG | RECALL | TIME (S) | COST | OUTPUT TOKENS | EVIDENCE |
|---|---|---|---|---|---|---|---|---|
| No runs yet: first runs land with Prem’s test videos. | ||||||||
Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.
SCORING RUBRIC
- Score = 100 if all hidden tests pass, 50 if the target test passes but another breaks, 0 otherwise
- Valid / partial / wrong across 5 seeded bugs
- Cost and output tokens recorded from the provider's usage report
CHECKED BY HAND
- Diff read by a person for test-gaming (edited tests, skipped asserts)
- Hidden tests run locally
- Unrelated changes noted
How a run gets on the board
1 · Same task, same inputs
Each suite fixes the inputs (dummy invoices, a brief, a spec, a repo) so models face identical work.
2 · Recorded
Runs are screen-recorded. The recording link sits on the row.
3 · Checked by hand
Every field, claim or test result is checked by a person against the rubric. Valid, partial and wrong are counted separately.
4 · Dated and sourced
Each row carries the run date, model, effort level, time, cost and output tokens, plus a write-up. Builder Pro members get early access to new runs.