Use cases/Switch models with proof

Switch models with proof

ProposedNo plan row yetPromptComputeRepoOpsContextFileAI Source of Truth

Not in the plan. Agent versions and shadow tests exist in PromptCompute.

Use case

Switch models with proof: a workflow across PromptCompute, RepoOps, ContextFile, AI Source of Truth that starts from a new agent version is registered.

Who would use it

Teams moving an agent to a new model or prompt.

What value it provides

The new agent version runs in a shadow test first and is compared with the old one on approvals, incidents and outcomes. A person approves the switch with the comparison attached.

Why it matters to the user

Model releases come every few weeks. This makes each switch a decision with evidence instead of a gamble.

The problem

A new model changes behavior in ways the tests miss, and the first sign is a production incident.

Expected outcomes

Incidents in the 14 days after a switch
Switches made with a comparison attached: target 100%

Each run records its outcome as met, not met or not measured.

How it runs

1. Register the new version from its manifestPromptComputeBuilt

2. Run it in a Test first shadow testPromptComputeBuilt

3. Compare incidents per versionRepoOpsBuilt

4. Pin the context version both readContextFileBuilt

5. A person approves the switchPromptComputeBuilt

6. Record the comparisonAI Source of TruthProposed

Products and connections

Product	Role in this workflow	Docs
PromptCompute	Register the new version from its manifest; Run it in a Test first shadow test; A person approves the switch	Reference · PromptCompute docs
RepoOps	Compare incidents per version	Reference · RepoOps docs
ContextFile	Pin the context version both read	Reference · ContextFile docs
AI Source of Truth	Record the comparison	Reference · AI Source of Truth docs

Outside the family: Your model provider.

What a person approves

Action	Default risk
`engine.switch`	high

Action keys are from the PromptCompute catalog.

How to access it from an agent

An agent will run this workflow through the PromptReports MCP server or API, with an ID agent token. The runner will then call each product for it. Proposed

// MCP: https://mcp.promptreports.ai/mcp (Proposed)
start_run({
  workflow: "switch-models-with-proof",
  inputs: { "agent": "billing-fixer", "new_manifest": "model id, prompt hash" },
  dry_run: false
})
// returns { run_id, status: "running", approvals_url }
get_run({ run_id })   // poll, or subscribe to run events

curl -X POST https://api.promptreports.ai/v1/runs \
  -H "Authorization: Bearer $PROMPTREPORTS_AGENT_TOKEN" \
  -d '{"workflow":"switch-models-with-proof"}'

Or call the products yourself, step by step:

Product	Endpoint	Tools
PromptCompute	`https://mcp.promptcompute.ai/mcp`	`request_approval` `get_approval` `cancel_approval` `answer_approver`
RepoOps	`https://mcp.repoops.ai/mcp`	`search_brain` `get_pre_edit_context` `propose_brain_entry` `get_incident_attribution`
ContextFile	Built	to confirm
AI Source of Truth	`https://aisourceoftruth.com/v1/family/facts`	to confirm

Set it up

Sign in at app.promptreports.ai with PromptReports ID, and pick your org.
Connect PromptCompute, RepoOps, ContextFile, AI Source of Truth on the Connections page. Link Your model provider through the product that uses it.
Register the agent that will run it on the Agents page, and give it a version manifest.
Set the approval policy for engine.switch. Start with Require approval; critical actions are never auto.
Fill in the inputs (Agent, New manifest) and choose the trigger: a new agent version is registered.
Start a dry run. It will count no workflow request, and it shows which steps would record missing evidence.
Start the run. Approve each gate in the approvals inbox, then read the signed record when it ends.