# Reproducible AI coding benchmark

Copy this file for each task and tool. Fill in every field before publishing results. Do not invent missing measurements.

## Task definition

- Task ID and date:
- Public repository URL and license:
- Starting commit SHA:
- Task prompt (exact text):
- Acceptance criteria:
- Test commands and expected outcomes:
- Time or attempt limit:

## Environment

- Operating system and version:
- Language runtime and dependency lockfile:
- Tool, version, and interface (CLI/editor/cloud):
- Model identifier and selection mode:
- Relevant settings, permissions, and network access:
- Account plan and region:

## Pricing assumptions

- Official pricing source URL:
- Source verification date:
- Base subscription and billing period:
- Included allowance and limit notes:
- Extra usage charges, if any:
- Whether exact per-task cost is available:

## Execution record

| Attempt | Start/end time | Prompt or intervention | Outcome | Usage/cost evidence |
| --- | --- | --- | --- | --- |
| 1 | | | | |

- Raw run log URL or archived file:
- Resulting commit or patch URL:
- Test output URL or archived file:
- Human edits after the tool completed:

## Review

- Acceptance tests passed:
- New regression tests:
- Code review findings:
- Reverts or retries:
- Task completion judgment and reviewer:

## Limitations

- Any unavailable data:
- Sources of variance, including model changes or rate limits:
- Why results may not generalize:
- Date of final review: