Use cases/Evaluate an agent before you buy it

Evaluate an agent before you buy it

PlannedLDG-2279PromptComputeRepoOps

Being built now. The records exist; the comparison view in the agent registry is still to build, and it will show no figures until runs exist.

Use case

Evaluate an agent before you buy it: a workflow across PromptCompute, RepoOps that starts from a person starts a trial.

Who would use it

Builders choosing between agents or agent vendors.

What value it provides

Run two agents side by side in listen mode and compare their records per action: incidents, approvals needed and cost per outcome.

Why it matters to the user

You buy the agent that behaved well in your code, not the one with the best demo.

The problem

Agents are compared on demos and benchmarks, not on how they behave in your systems.

Expected outcomes

Incidents per 100 actions, per agent
Share of actions a person had to approve
Cost per merged change

Each run records its outcome as met, not met or not measured.

How it runs

1. Register each candidate and run it in listen modePromptComputeBuilt

2. Attribute incidents to each agent's sessionsRepoOpsBuilt

3. Compare records per actionPromptComputePlanned

Products and connections

Product	Role in this workflow	Docs
PromptCompute	Register each candidate and run it in listen mode; Compare records per action	Reference · PromptCompute docs
RepoOps	Attribute incidents to each agent's sessions	Reference · RepoOps docs

Outside the family: GitHub.

What a person approves

Action	Default risk
`grant.create`	high

Action keys are from the PromptCompute catalog.

How to access it from an agent

An agent will run this workflow through the PromptReports MCP server or API, with an ID agent token. The runner will then call each product for it. Proposed

// MCP: https://mcp.promptreports.ai/mcp (Proposed)
start_run({
  workflow: "evaluate-an-agent",
  inputs: { "agents": "Two or more registered agents", "trial_length": "14 days" },
  dry_run: false
})
// returns { run_id, status: "running", approvals_url }
get_run({ run_id })   // poll, or subscribe to run events

curl -X POST https://api.promptreports.ai/v1/runs \
  -H "Authorization: Bearer $PROMPTREPORTS_AGENT_TOKEN" \
  -d '{"workflow":"evaluate-an-agent"}'

Or call the products yourself, step by step:

Product	Endpoint	Tools
PromptCompute	`https://mcp.promptcompute.ai/mcp`	`request_approval` `get_approval` `cancel_approval` `answer_approver`
RepoOps	`https://mcp.repoops.ai/mcp`	`search_brain` `get_pre_edit_context` `propose_brain_entry` `get_incident_attribution`

Set it up

Sign in at app.promptreports.ai with PromptReports ID, and pick your org.
Connect PromptCompute, RepoOps on the Connections page. Link GitHub through the product that uses it.
Register the agent that will run it on the Agents page, and give it a version manifest.
Set the approval policy for grant.create. Start with Require approval; critical actions are never auto.
Fill in the inputs (Agents, Trial length) and choose the trigger: a person starts a trial.
Start a dry run. It will count no workflow request, and it shows which steps would record missing evidence.
Start the run. Approve each gate in the approvals inbox, then read the signed record when it ends.