HTTP API
11 endpoints, all under /api/*. The full OpenAPI 3 schema is at
/api/openapi.json — pipe into openapi-typescript or similar
to generate typed clients.
Read-only
Section titled “Read-only”| Endpoint | Description |
|---|---|
GET /api/health | Liveness + version. Always cheap. |
GET /api/scenarios | Bundled scenarios (skips non-YAML markdown). |
GET /api/scenarios/{name} | Full scenario definition — arms, rubric, policies, task. 404 on unknown. |
GET /api/doctor | Env-only preflight checks. Add ?scenario=<name> for scenario-scoped checks (fixture, asd probe, budget sanity). |
GET /api/sessions | List persisted sessions, newest first. Empty when the server wasn’t started with --state-dir. |
GET /api/sessions/{id} | Full SessionOutcome — arm summaries, decisions, verdict. |
| Endpoint | Description |
|---|---|
POST /api/runs | Blocking A/B + judge. Returns the full SessionOutcome when the judge completes. Body: {scenario, model_override?, judge_model?}. |
POST /api/runs/async | Same, but starts a background task and returns {session_id} immediately. Subscribe to the SSE stream for progress. |
GET /api/runs/{id}/stream | Server-sent events for an in-flight run. Event kinds: decision, arm_finished, verdict_ready, session_failed. 404 if the id has no live sender (run already finished — read /api/sessions/{id} for the completed outcome). |
Example SSE frame:
data: {"event":"decision","session_id":"session-asd-vs-baseline-...","arm_id":"assisted","decision":{"seq":3,"kind":"model_turn","summary":"...","tokens_in":718,"tokens_out":14,"latency_ms":1399}}
data: {"event":"arm_finished","session_id":"...","arm_id":"assisted"}
data: {"event":"verdict_ready","session_id":"...","verdict":{...}}Verdicts
Section titled “Verdicts”| Endpoint | Description |
|---|---|
POST /api/verdicts | Rejudge two arms without re-running the agents. Body: {arm_a: {session_id, arm_id}, arm_b: {session_id, arm_id}, judge_model?}. Same-session (rejudge with new model) and cross-session (compare arms from separate runs of the same scenario) both supported. Cross-scenario refused up front. |
Schema
Section titled “Schema”| Endpoint | Description |
|---|---|
GET /api/openapi.json | OpenAPI 3.1 document. All types (Scenario, Verdict, SessionOutcome, ReferenceFixScore, etc) as #/components/schemas. Regression-tested to always list every route. |
Server sends Access-Control-Allow-Origin: *. Fine for local dev
and same-origin hosting. Add a reverse proxy that scopes CORS if
you’re exposing this on a shared network.
None in v1. See P3 packaging notes for the intended deploy shape
(reverse proxy, TLS termination, optional basic-auth in front of
/api/runs* if the server is exposed beyond localhost).