PPromptReports
Sign in
Use cases/Evaluate an agent before you buy it/OpenSpec plan

Plan: Evaluate an agent before you buy it

Plannedopenspec/changes/wf-evaluate-an-agentLDG-2279

Written for an AI agent to fetch and follow: it says what to connect, which steps to build with which product, what a person must approve, and how to check the result. Steps that depend on a part that is not built say so, and tell the agent to record the gap instead of guessing.

Raw URL: https://promptreports.ai/use-cases/evaluate-an-agent/openspec.md

URL pattern: /use-cases/<id>/openspec.md, all plans at /use-cases/index.json. Checkboxes on this page are for reading only.

openspec/changes/wf-evaluate-an-agent/proposal.md

Why

Agents are compared on demos and benchmarks, not on how they behave in your systems. You buy the agent that behaved well in your code, not the one with the best demo.

What Changes

  • Add the evaluate-an-agent workflow to the PromptReports workflow library, run by the workflow runner as an agent acting for a person.
  • PromptCompute: Register each candidate and run it in listen mode (today: Built).
  • RepoOps: Attribute incidents to each agent's sessions (today: Built).
  • PromptCompute: Compare records per action (today: Planned).
  • Gate grant.create through PromptCompute approvals; each gated step runs only after verifyResume passes, with final_args.
  • Write a signed record at the end of each run and count one workflow request.

Impact

  • Products: PromptCompute, RepoOps.
  • Outside the family: GitHub.
  • State: Planned. Being built now. The records exist; the comparison view in the agent registry is still to build, and it will show no figures until runs exist.
  • Plan row: LDG-2279.
openspec/changes/wf-evaluate-an-agent/tasks.md

1. Connect and authorize

  • 1.1 Confirm the org has a connection for each of: PromptCompute, RepoOps. Stop and tell the user which is missing.
  • 1.2 Get a PromptReports ID agent token for each product the run calls (RFC 8693 exchange). Where a product does not accept ID tokens yet, use its connect key.
  • 1.3 Register policy for grant.create with require_approval as the default.

2. Build the steps

  • 2.1 PromptCompute: Register each candidate and run it in listen mode.
  • 2.2 RepoOps: Attribute incidents to each agent's sessions.
  • 2.3 PromptCompute: Compare records per action. Blocked: this part is Planned; until it ships, record the step's evidence as missing and continue.

3. Verify and record

  • 3.1 Run a dry run with the user's inputs: Agents = Two or more registered agents; Trial length = 14 days.
  • 3.2 Run it once for real; approve each gate as the user.
  • 3.3 Measure the outcomes: Incidents per 100 actions, per agent; Share of actions a person had to approve; Cost per merged change.
  • 3.4 Confirm the run left a signed record with a task id and one workflow request counted.
openspec/changes/wf-evaluate-an-agent/specs/workflow-evaluate-an-agent/spec.md

ADDED Requirements

Requirement: Comparisons use the same actions

The comparison SHALL count only action types that every candidate attempted.

Scenario: Comparisons use the same actions

  • WHEN one agent never attempted fix.apply
  • THEN fix.apply is left out of the comparison and the view says why

Requirement: Evidence stays honest

Each step SHALL report what it saw as measured, inferred or missing, and SHALL never report missing as measured.

Scenario: A product part is not built

  • WHEN a step depends on a part that is Planned or Proposed
  • THEN the step records missing, names the part, and the approval card shows the gap

Requirement: One run is one workflow request

The runner SHALL count one workflow request when the first step starts, and SHALL NOT count retries, resumes or dry runs.

Scenario: A gated step resumes after two days

  • WHEN a person approves a step and the run resumes
  • THEN the org's usage does not change