Skip to content
Crucible

Quickstart

Get from git clone to a real verdict in five minutes.

1. Install

Section titled “1. Install”
Terminal window
curl -sSfL https://raw.githubusercontent.com/AgentStateLabs/AgentStateCrucible/main/install.sh | bash

Set CRUCIBLE_GITLAB_TOKEN if the project is private. Other install paths — Docker, source, homebrew — are on the Install page.

2. Set your provider key

Section titled “2. Set your provider key”
Terminal window
export OPENAI_API_KEY=sk-...

Crucible calls OpenAI’s chat API for both agent turns and the judge. Anthropic works too — set ANTHROPIC_API_KEY and use --model anthropic/claude-sonnet-4-6.

3. Run the bundled A/B

Section titled “3. Run the bundled A/B”
Terminal window
crucible run scenarios/asd-vs-baseline.md \
    --model openai/gpt-4o-mini \
    --state-dir ~/.crucible-state

That kicks a full A/B session:

  • Copies the fixture (a small Python package with a seeded bug) into a per-arm workdir.
  • Runs an agent loop for each arm against the OpenAI API — one with ASD/CTX tools available, one baseline with just filesystem+bash.
  • Executes python3 -m pytest -q on each resulting workdir.
  • Captures the diff and scores it against the reference fix.
  • Invokes the judge to compare the two arms across the scenario’s rubric (correctness, minimality, reasoning quality, efficiency).
  • Prints a JSON verdict and persists everything to ~/.crucible-state/sessions/.

Typical wall clock: 30–90s.

4. Look at the verdict

Section titled “4. Look at the verdict”

The stdout JSON has the shape:

{
  "scenario": "asd-vs-baseline",
  "arms": [
    { "arm_id": "assisted", "test_passed": true, "visible_tokens": 27346,
      "reference_fix_score": { "value": 0.76, "structural": 1.0, "textual": 0.4 } },
    { "arm_id": "baseline", "test_passed": true, "visible_tokens": 8639,
      "reference_fix_score": { "value": 0.77, "structural": 1.0, "textual": 0.41 } }
  ],
  "verdict": {
    "winner": "b",
    "dimensions": [
      { "name": "correctness", "score_a": 8.0, "score_b": 8.0, ... },
      { "name": "efficiency",  "score_a": 6.0, "score_b": 8.0, ... },
      ...
    ]
  }
}

5. Serve the UI

Section titled “5. Serve the UI”
Terminal window
crucible serve --state-dir ~/.crucible-state --web-dir /path/to/web/build
# → http://127.0.0.1:8080

The same session shows up at /sessions. Click into it for the full decision timeline, per-arm diff, and the ability to rejudge with a different model.

6. Register with an MCP client

Section titled “6. Register with an MCP client”

Point Claude Code (or Cursor / Codex) at crucible mcp and Crucible becomes a tool your agent can invoke — run a scenario, get a verdict, gate a release. See MCP tools.