A coding agent benchmark should help a developer choose a tool for real work. A single impressive task or a speed claim without its setup rarely does that. A useful comparison gives each CLI the same repository, task, starting commit, model conditions, and acceptance checks, then reports how much human review was needed to keep the result.
Decide what “better” means before the run
A tool can finish quickly and still produce a patch that needs substantial repair. Another can use more model calls but leave a smaller, safer change. Choose the outcome that matters for your team: accepted tasks, correct tests, time to a reviewed result, usage cost, or a balance of those measures.
Do not combine different questions into one unexplained score. If you want to compare model quality, keep the CLI and prompt conditions as close as possible. If you want to compare complete products, say that each product includes its own model access, routing, tools, and plan. Those are different experiments.
| Task type | Example acceptance check |
|---|---|
| Bug fix | A regression test fails before the change and passes after it |
| Test writing | New cases cover expected, edge, and failure behavior |
| Contained feature | The requested behavior works without unrelated changes |
| Repository task | The agent finds the owning package and runs its documented checks |
Keep the starting conditions fair
Use the same clean commit, task description, acceptance criteria, and access to project instructions. Record the CLI version, model identifier, provider, account path, region, plan, and settings. If a tool requires a different setup, document it rather than pretending the products ran under identical conditions.
Separate cold cache runs from warm cache runs. Keep the same task and wait interval when testing cache behavior. Do not let one product reuse a context that another product had to build from scratch. If a model changes during the test, mark the run and repeat it when a comparable version is available.
Track outcomes, cost, and review effort separately
For every run, record whether the acceptance test passed, what files changed, the number of retries, the time to a reviewable diff, and the time a person spent correcting it. For token billed products, record input, cached input, cache writes, and output separately when the provider exposes them. For a subscription allowance, record the actual usage meter instead of inventing a token conversion.
Read each diff before scoring it. A benchmark that counts generated code as success without checking behavior rewards output volume rather than useful work. Note false starts, unrun checks, unrelated changes, and any result a developer rejected.
| Metric | Record |
|---|---|
| Task result | Acceptance checks passed and changes accepted |
| Time | Time to a reviewable patch and time spent on human corrections |
| Usage | Displayed credits or provider token fields, with their unit |
| Caching | Cold or warm state, cache reads, cache writes, and denominator |
| Reliability | Retries, failed checks, and results that needed repair |
Report enough detail for someone else to repeat it
Publish the date, repository commit, task prompts, product and version, model, plan or API path, settings, commands, and scoring rules. Include the number of attempts and the full range of outcomes. If you publish a percentage such as a cache hit rate or a usage advantage, define its numerator and denominator and show the source records used to calculate it.
Avoid turning a benchmark from one task into a universal ranking. Results can change with the language, repository size, model release, cache state, and test suite. State what you tested, what you did not test, and where the result should not be generalized.
- Choose a small set of representative tasks and acceptance checks.
- Run each product from the same clean repository state.
- Record model, plan, usage, cache state, time, and corrections.
- Repeat runs and publish the method with the results.
Use a template instead of a leaderboard guess
LucentraCode has a developer benchmark template for recording the repository, prompt, model, tool, tests, cost source, and limitations. The cost calculator can help frame subscription spend from your own assumptions. Include LucentraCode and any alternatives you use under the same task rules, then let the resulting evidence support the conclusion.