PPromptReports
Sign in
Use cases/Graded research/OpenSpec plan

Plan: Graded research

Partly builtopenspec/changes/wf-graded-researchLDG-2277

Written for an AI agent to fetch and follow: it says what to connect, which steps to build with which product, what a person must approve, and how to check the result. Steps that depend on a part that is not built say so, and tell the agent to record the gap instead of guessing.

Raw URL: https://promptreports.ai/use-cases/graded-research/openspec.md

URL pattern: /use-cases/<id>/openspec.md, all plans at /use-cases/index.json. Checkboxes on this page are for reading only.

openspec/changes/wf-graded-research/proposal.md

Why

Research calls are rarely graded once the event resolves, so a reader cannot tell a good source from a loud one. You can judge a research source by its record instead of its confidence.

What Changes

  • Add the graded-research workflow to the PromptReports workflow library, run by the workflow runner as an agent acting for a person.
  • 15 Minute Markets: Record each call with its time and odds (today: Planned).
  • StockGaines: Record each forecast (today: Planned).
  • AI Source of Truth: Grade calls when they resolve (today: Planned).
  • AI Source of Truth: Show the scorecard (today: Built).
  • PromptCompute: Gate any trading action for a person (today: Built).
  • Gate trade.order.place, stockgaines.order.place through PromptCompute approvals; each gated step runs only after verifyResume passes, with final_args.
  • Write a signed record at the end of each run and count one workflow request.

Impact

  • Products: 15 Minute Markets, StockGaines, AI Source of Truth, PromptCompute.
  • Outside the family: None beyond the family.
  • State: Partly built. The AI Source of Truth scorecard is live. Three parts to go.
  • Plan row: LDG-2277.
openspec/changes/wf-graded-research/tasks.md

1. Connect and authorize

  • 1.1 Confirm the org has a connection for each of: 15 Minute Markets, StockGaines, AI Source of Truth, PromptCompute. Stop and tell the user which is missing.
  • 1.2 Get a PromptReports ID agent token for each product the run calls (RFC 8693 exchange). Where a product does not accept ID tokens yet, use its connect key.
  • 1.3 Register policy for trade.order.place, stockgaines.order.place with require_approval as the default.

2. Build the steps

  • 2.1 15 Minute Markets: Record each call with its time and odds. Blocked: this part is Planned; until it ships, record the step's evidence as missing and continue.
  • 2.2 StockGaines: Record each forecast. Blocked: this part is Planned; until it ships, record the step's evidence as missing and continue.
  • 2.3 AI Source of Truth: Grade calls when they resolve. Blocked: this part is Planned; until it ships, record the step's evidence as missing and continue.
  • 2.4 AI Source of Truth: Show the scorecard.
  • 2.5 PromptCompute: Gate any trading action for a person.

3. Verify and record

  • 3.1 Run a dry run with the user's inputs: Source = 15 Minute Markets or StockGaines; Markets = Your watch list.
  • 3.2 Run it once for real; approve each gate as the user.
  • 3.3 Measure the outcomes: Calls recorded before resolution: target 100%; Calibration on the scorecard; Trading actions without approval: 0.
  • 3.4 Confirm the run left a signed record with a task id and one workflow request counted.
openspec/changes/wf-graded-research/specs/workflow-graded-research/spec.md

ADDED Requirements

Requirement: Calls are recorded first

A call SHALL be recorded with a timestamp before the event it predicts resolves.

Scenario: Calls are recorded first

  • WHEN a call is filed after resolution
  • THEN it is refused and never counts on the scorecard

Requirement: Evidence stays honest

Each step SHALL report what it saw as measured, inferred or missing, and SHALL never report missing as measured.

Scenario: A product part is not built

  • WHEN a step depends on a part that is Planned or Proposed
  • THEN the step records missing, names the part, and the approval card shows the gap

Requirement: One run is one workflow request

The runner SHALL count one workflow request when the first step starts, and SHALL NOT count retries, resumes or dry runs.

Scenario: A gated step resumes after two days

  • WHEN a person approves a step and the run resumes
  • THEN the org's usage does not change