Models · Our evals

The Speedrunning AI Evalboard

Real work tasks, run on camera, every finding checked by hand. Five suites are defined; the boards fill as Prem records the runs. We never publish a score without its date, recording and write-up.

AP invoice extraction

0 runs · Full write-up

Extract 10 dummy supplier invoices (PDF and image, mixed layouts, two in INR with GST) into a sheet: supplier, GSTIN/VAT ID, invoice number, date, due date, currency, line items, subtotal, tax, total.

#MODEL + EFFORTSCOREVALID / PARTIAL / WRONGRECALLTIME (S)COSTOUTPUT TOKENSEVIDENCE
No runs yet: first runs land with Prem’s test videos.

Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.

SCORING RUBRIC

  • Each of 10 fields per invoice scored valid (exact), partial (right value, wrong format) or wrong/missing
  • Score = valid fields ÷ all fields × 100
  • Recall = fields extracted ÷ fields present
  • Totals must reconcile with line items; a mismatch counts as wrong

CHECKED BY HAND

  • Every extracted field is compared with the source invoice by a person
  • GST/VAT numbers and totals are re-added by hand
  • Hallucinated fields (values not on the invoice) are counted as wrong and noted

Landing page build

0 runs · Full write-up

From a one-page brief for a fictional Indian D2C brand, build a responsive landing page (hero, three benefits, pricing in INR, FAQ, signup form) as a single deployable folder.

#MODEL + EFFORTSCOREVALID / PARTIAL / WRONGRECALLTIME (S)COSTOUTPUT TOKENSEVIDENCE
No runs yet: first runs land with Prem’s test videos.

Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.

SCORING RUBRIC

  • 20-item checklist: every section present, works at 390px and 1440px, no horizontal scroll, form validates, accessible labels, Lighthouse accessibility ≥ 90
  • Score = checklist items passed ÷ 20 × 100
  • Time = wall-clock until the first deployable build

CHECKED BY HAND

  • Opened in a real browser at both widths by a person
  • Copy is checked against the brief for invented claims
  • Keyboard-only pass through the form

n8n / automation workflow

0 runs · Full write-up

From a written spec (new form lead → enrich → score → route to a sheet and a Slack-style webhook, with error handling), produce a working n8n workflow export.

#MODEL + EFFORTSCOREVALID / PARTIAL / WRONGRECALLTIME (S)COSTOUTPUT TOKENSEVIDENCE
No runs yet: first runs land with Prem’s test videos.

Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.

SCORING RUBRIC

  • 12-item checklist: imports cleanly, every node configured, runs end to end on 5 test payloads, errors routed, no hard-coded secrets
  • Score = checklist items passed ÷ 12 × 100

CHECKED BY HAND

  • Workflow imported and run on the 5 payloads by a person
  • Outputs compared with expected rows
  • Credentials and secrets inspected

Research brief with sources

0 runs · Full write-up

Write a 600-word brief on a fixed current AI question with at least 8 cited sources.

#MODEL + EFFORTSCOREVALID / PARTIAL / WRONGRECALLTIME (S)COSTOUTPUT TOKENSEVIDENCE
No runs yet: first runs land with Prem’s test videos.

Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.

SCORING RUBRIC

  • Citation accuracy: each cited claim checked against its source — valid (supported), partial (source related but claim overstated), wrong (unsupported or source missing)
  • Score = valid citations ÷ all citations × 100
  • Recall = key facts covered from a fixed list of 10

CHECKED BY HAND

  • Every citation opened and read by a person
  • Dead links and fabricated sources counted as wrong
  • Dates checked for staleness

Bug fix in a small repo

0 runs · Full write-up

Fix a seeded bug in a small open-source-style Python/TypeScript repo given a failing issue report; the hidden test suite must pass without breaking existing tests.

#MODEL + EFFORTSCOREVALID / PARTIAL / WRONGRECALLTIME (S)COSTOUTPUT TOKENSEVIDENCE
No runs yet: first runs land with Prem’s test videos.

Source: Speedrunning AI runs recorded by Prem and checked by hand; each row links its recording and write-up.

SCORING RUBRIC

  • Score = 100 if all hidden tests pass, 50 if the target test passes but another breaks, 0 otherwise
  • Valid / partial / wrong across 5 seeded bugs
  • Cost and output tokens recorded from the provider's usage report

CHECKED BY HAND

  • Diff read by a person for test-gaming (edited tests, skipped asserts)
  • Hidden tests run locally
  • Unrelated changes noted

How a run gets on the board

1 · Same task, same inputs

Each suite fixes the inputs (dummy invoices, a brief, a spec, a repo) so models face identical work.

2 · Recorded

Runs are screen-recorded. The recording link sits on the row.

3 · Checked by hand

Every field, claim or test result is checked by a person against the rubric. Valid, partial and wrong are counted separately.

4 · Dated and sourced

Each row carries the run date, model, effort level, time, cost and output tokens, plus a write-up. Builder Pro members get early access to new runs.