RESOURCE / REPRODUCIBILITY

A benchmark template developers can actually repeat.

A useful AI coding benchmark is one another developer can repeat. This template captures the repository state, task, model, cost assumptions, and review criteria before results are known.

Download and use the worksheet

Download the Markdown template, copy it for each tool and task, and publish the filled version with the starting commit and test commands. Keep failed runs and human interventions in the record.

Choose a public repository and immutable starting commit.
Write a task with observable acceptance criteria and a time limit.
Record tool version, model, settings, plan, and pricing source date.
Run the same task from the same clean state.
Publish test output, review notes, total spend assumptions, and limitations.

What to report

Separate mechanical measures, such as tests passed or elapsed time, from subjective code-review judgments. If the tool exposes only a subscription allowance rather than exact per-task cost, say so. Do not convert unlike usage units into a universal score.

Repository + commit: ____
Task and acceptance tests: ____
Tool / version / model: ____
Prompt and settings: ____
Run log and test results: ____
Human corrections: ____
Price source and date: ____
Limitations: ____

Related research

Use the cost comparison to locate current plan sources and the calculator to frame subscription cost per session.