Evaluate an agent before you buy it: compare agents by what they did in your code

How the agent evaluation workflow will compare agents on their record per action, why it will show an empty state until the runner has recorded runs for each agent, and what is being built now.

Evaluate an agent before you buy it is a workflow for builders choosing between coding or sales agents, or between the vendors who sell them. It will compare agents on their record in your systems: incidents caused per action, how often a person had to approve, and cost per outcome. PromptCompute will hold the comparison in its agent registry, and RepoOps will supply the incidents.

We are building it now and will switch it on as soon as it works. Until the workflow runner has recorded runs for each agent, the comparison will show an empty state that says so. This guide explains why, and what the view will hold once there is data.

The problem

Agents are compared on demos and benchmarks. A demo shows the agent on a task the vendor picked. A benchmark shows it on someone else’s code. Neither tells you how the agent behaves in your repos, with your tests, your review rules and your production traffic.

The behavior that matters shows up later. Which agent’s changes caused incidents, and which one needed a person to step in on every other action? You find out after you have committed to a vendor, and the evidence is scattered across the tools the agent touched.

How the workflow solves it

The workflow will run the candidates side by side, in your systems, under the same rules. Each candidate will be registered in PromptCompute with its own identity and version, and each will run in listen mode. RepoOps attributes each incident to the agent session that caused it. The comparison view will then line the agents up per action type, from their own records.

The rules will be the same for every candidate. The comparison will count only the action types every candidate attempted. If one agent never tried fix.apply, that action will be left out, and the view will say why.

Why it starts empty

A comparison needs records, and the records come from runs. The workflow runner, which does not exist yet, is what will give each agent’s actions one run id across products and record how each run turned out. Until the runner has recorded runs for an agent, there is nothing honest to compare.

So the view will open with an empty state that says it is waiting for recorded runs. It will not show a score built from a guess or from a vendor’s own numbers. That is the same rule the rest of the family follows: a fact that is missing reads missing, never zero and never a pass.

Step by step

When the runner ships, a trial will take these steps. Present tense means the part works in its product today; “will” means it is still to build.

A trial will start when a person starts it and names the candidates.

1. PromptCompute registers each candidate and runs it in listen mode. Each gets an agt_ id, an Ed25519 key and an agv_ version pinned to its manifest.
2. RepoOps attributes incidents to each agent’s sessions. It links a commit to its session through a Session-Id trailer when one exists, or through time and file overlap, and marks the link strong or weak. When two sessions overlap, it does not pick one.
3. PromptCompute will compare the records per action in a comparison view in the agent registry, using only the action types every candidate attempted.

Granting a candidate more access during the trial is an approval like any other, under grant.create.

What the workflow will report

Per agent, the comparison will report:

• Incidents per 100 actions.
• The share of actions a person had to approve.
• The cost per merged change.

Every figure will come from an agent’s recorded runs. An agent with none will show the empty state, not a number.

Why it matters to a builder

You will pick the agent that behaved well in your code, on evidence you collected yourself. The records will also keep their value after the purchase, because RepoOps will count incidents per agent version. When the vendor ships a new version, you will see whether it does better or worse than the one before.

It will change the conversation with a vendor as well. Instead of debating a benchmark, you will be able to show them their agent’s incidents per 100 actions next to a competitor’s, from the same repos over the same period.

How to set it up

GitHub is the only outside connection.

When the runner and the signed-in app ship, setup will be:

1. Sign in with PromptReports ID and pick your org.
2. Connect PromptCompute and RepoOps, and link GitHub through RepoOps.
3. Register each candidate agent with a version manifest.
4. Set the approval policy for grant.create, so no candidate gains access without a person.
5. Choose the trial length and start the trial.
6. Read the comparison as runs build up. It will stay empty for an agent until that agent has recorded runs.

An agent will start a trial through the PromptReports MCP server or API, with the proposed start_run tool.

The workflow is on every plan. Plans are per user per month: $25 for 100 workflow requests, $50 for 500, and Enterprise above 500. There is no free tier. Each candidate agent’s run in a trial will count as one workflow request, so a trial of three agents uses three. The allowance is pooled across your organization.

What is built today and what is planned

The workflows page lists this workflow as planned. We decided on 2026-11-10 to build it and switch it on now, with the empty state described above until the runner has recorded runs per agent.

In the products today:

• PromptCompute registers agents and versions and runs listen mode per environment.
• RepoOps attributes incidents to agent sessions.

Being built:

• A comparison view in the PromptCompute agent registry.
• Incidents and lessons per agent and per version in RepoOps.

Planned:

• The workflow runner. No runner exists today. The plan is a durable runner service in the PromptCompute stack, with one run id across products, approvals as steps, and each run metered as one workflow request. Until it records runs, the comparison has nothing to show.

Read more

• Every workflow and its state
• Pricing
• Should the agent that writes the code also review it?
• PromptCompute and RepoOps