A coding agent that succeeds once may still be difficult to rely on. It might take different actions on a repeated task, skip a check, make unrelated edits, or need several corrections before the code is useful. Measure reliability across a small set of repeated repository tasks rather than relying on one impressive result.
Define what a reliable result means
For one task, reliability may mean that the same acceptance test passes and the patch stays within the requested scope. For another, it may mean that the agent asks for a missing requirement instead of inventing one. Decide what matters before you compare products.
Track failed commands, missing tests, retries, unrelated changes, and human corrections. Keep the agent’s final explanation separate from the evidence. A confident summary is not a substitute for a passing check or a reviewed diff.
- Did the requested behavior work?
- Did the agent change only files needed for the task?
- Did it report failed or skipped checks accurately?
- Could a developer understand and maintain the patch?
Repeat tasks under controlled conditions
Use the same starting commit, prompt, model, repository instructions, and acceptance criteria for each attempt. Separate cold and warm cache runs, and note any product or model version change. Repeat enough times to see whether a result is typical or an exception.
Do not treat different tools as identical when their models, account access, or permission defaults differ. If the question is which complete product works best for a team, compare the product workflow. If the question is model quality, hold the CLI and request path as constant as you can.
Use reliability results to improve the workflow
If failures repeat, find the cause before changing models. Missing instructions, broad tasks, unavailable dependencies, unclear permissions, and weak acceptance tests can all create unreliable outcomes. Improve one part of the setup and repeat the evaluation so you know whether it helped.
Report the task set, sample size, tool versions, run conditions, and failure categories. A useful result tells a developer what to try and where the evidence stops. It does not turn a limited internal test into a universal ranking.
- Choose representative tasks and define success before each run.
- Record successes, failures, retries, and review corrections.
- Group failures by cause rather than blaming the model alone.
- Publish the setup and limits so the comparison can be repeated.