Crucible over MCP

Crucible over MCP

crucible mcp is a stdio Model Context Protocol server that lets any MCP client (Claude Code, Cursor, Codex, custom SDK) drive Crucible: list scenarios, run A/B sessions, query verdicts, rejudge with a stronger model.

Install

cargo install --path crates/crucible-cli

(or brew install crucible once P3 packaging lands.)

Register with a client

Templates live in packaging/mcp/. Pick the one that matches your client, fix the absolute paths, and drop it in the client’s config.

• Claude Code / Claude Desktop → packaging/mcp/claude-code.json
• Cursor → packaging/mcp/cursor.json
• Codex CLI → packaging/mcp/codex.json

The server takes two flags:

• --scenarios-dir <path> (default scenarios): where scenario *.md files live.
• --state-dir <path> (optional): where sessions.db + summaries land. Required for crucible_run_session, crucible_list_sessions, crucible_get_session, and crucible_rejudge. The read-only tools (list_scenarios, get_scenario, doctor) work without it.

Live tools call OpenAI, so OPENAI_API_KEY must be reachable in the shell that spawns the server.

Tools

Tool	Description
crucible_list_scenarios	Returns {scenarios: [{name, description, arm_ids, reference_fix}, ...]}.
crucible_get_scenario (name)	Full scenario definition.
crucible_doctor (scenario?)	Preflight checks. Env-only when scenario is omitted.
crucible_run_session (scenario, model_override?, judge_model?)	Runs the A/B end-to-end (agent loop → tests → diff → judge). Blocks until the verdict returns. Returns a SessionOutcome. Persists to state_dir.
crucible_list_sessions	Trimmed list of persisted sessions, newest first.
crucible_get_session (session_id)	Full SessionOutcome with all arm decisions and the diff.
crucible_rejudge (arm_a, arm_b, judge_model?)	Judges two arms (same or different sessions of the same scenario) with a new model. Reuses recorded decisions / tests / diffs — no fresh agent runs. Does not persist.

Sample agent prompts

“Run asd-vs-baseline with both arms on openai/gpt-4o-mini and judge with openai/gpt-4o. Tell me who won and why.”

“List the persisted sessions. For the newest one, compare the reference-fix scores of the two arms.”

“Rejudge session session-asd-vs-baseline-... with openai/gpt-4o. Report how the winner and per-dimension scores differ from the stored verdict.”

Downstream use case

If ASD’s agent has Crucible registered as an MCP tool, ASD can run its own regression tests before shipping a new iteration — Crucible becomes ASD’s CI, model-independent and reproducible. Same argument for CTXone or any other agent product that wants to gate releases on behavior.