# openspec/changes/wf-evaluate-an-agent/proposal.md

## Why

Agents are compared on demos and benchmarks, not on how they behave in your systems. You buy the agent that behaved well in your code, not the one with the best demo.

## What Changes

- Add the `evaluate-an-agent` workflow to the PromptReports workflow library, run by the workflow runner as an agent acting for a person.
- PromptCompute: Register each candidate and run it in listen mode (today: Built).
- RepoOps: Attribute incidents to each agent's sessions (today: Built).
- PromptCompute: Compare records per action (today: Planned).
- Gate `grant.create` through PromptCompute approvals; each gated step runs only after `verifyResume` passes, with `final_args`.
- Write a signed record at the end of each run and count one workflow request.

## Impact

- Products: PromptCompute, RepoOps.
- Outside the family: GitHub.
- State: Planned. Being built now. The records exist; the comparison view in the agent registry is still to build, and it will show no figures until runs exist.
- Plan row: LDG-2279.

# openspec/changes/wf-evaluate-an-agent/tasks.md

## 1. Connect and authorize

- [ ] 1.1 Confirm the org has a connection for each of: PromptCompute, RepoOps. Stop and tell the user which is missing.
- [ ] 1.2 Get a PromptReports ID agent token for each product the run calls (RFC 8693 exchange). Where a product does not accept ID tokens yet, use its connect key.
- [ ] 1.3 Register policy for `grant.create` with `require_approval` as the default.

## 2. Build the steps

- [ ] 2.1 PromptCompute: Register each candidate and run it in listen mode.
- [ ] 2.2 RepoOps: Attribute incidents to each agent's sessions.
- [ ] 2.3 PromptCompute: Compare records per action. Blocked: this part is Planned; until it ships, record the step's evidence as `missing` and continue.

## 3. Verify and record

- [ ] 3.1 Run a dry run with the user's inputs: Agents = Two or more registered agents; Trial length = 14 days.
- [ ] 3.2 Run it once for real; approve each gate as the user.
- [ ] 3.3 Measure the outcomes: Incidents per 100 actions, per agent; Share of actions a person had to approve; Cost per merged change.
- [ ] 3.4 Confirm the run left a signed record with a task id and one workflow request counted.

# openspec/changes/wf-evaluate-an-agent/specs/workflow-evaluate-an-agent/spec.md

## ADDED Requirements

### Requirement: Comparisons use the same actions

The comparison SHALL count only action types that every candidate attempted.

#### Scenario: Comparisons use the same actions

- **WHEN** one agent never attempted fix.apply
- **THEN** fix.apply is left out of the comparison and the view says why

### Requirement: Evidence stays honest

Each step SHALL report what it saw as measured, inferred or missing, and SHALL never report missing as measured.

#### Scenario: A product part is not built

- **WHEN** a step depends on a part that is Planned or Proposed
- **THEN** the step records `missing`, names the part, and the approval card shows the gap

### Requirement: One run is one workflow request

The runner SHALL count one workflow request when the first step starts, and SHALL NOT count retries, resumes or dry runs.

#### Scenario: A gated step resumes after two days

- **WHEN** a person approves a step and the run resumes
- **THEN** the org's usage does not change