CRUCIBLE // agent testing on ASG primitives

Your agents behave differently every run. Crucible measures how.

Crucible is a self-hosted regression harness for coding agents — sealed A/B sessions, replayable decisions, and judge-scored verdicts. Reproducible enough to gate a release on.

Under the hood it's the AgentStateGraph in a purpose-built shape: scenarios as fixtures, arms as agent configurations, and verdicts written back as commits on the session's main branch. One binary. MCP-native. Zero telemetry.

$ curl -sSfL https://raw.githubusercontent.com/AgentStateLabs/AgentStateCrucible/main/install.sh | bash

Early access: the repository is not public yet, so this installer will not resolve. Once it is: prebuilt binaries for macOS (Apple Silicon + Intel) and Linux (musl x86_64 + aarch64), or cargo install and docker compose.

5-minute quickstart →

4 bundled scenarios 7 MCP tools SSE live-run stream Reference-fix scorer

Bundled scenarios

Primitives

Surfaces

MCP tools

ARCHITECTURE //

One binary. Three surfaces.

Crucible isn't a library you import — it's a self-hosted harness your tools point at. Same core, three shapes: a CLI for humans and CI, an HTTP + SSE API paired with the Lens UI, and an MCP server for agents that want to test themselves.

cli/

crucible

the command line

One binary you can drop into CI: list scenarios, kick a run, tail the SSE stream, and re-judge without re-running. Everything the server does, you can do from a shell.

server/

crucible-server + Lens

HTTP API + live web UI

A local daemon that serves the REST API, streams live decisions over SSE, and pairs with crucible-lens — a SvelteKit UI for browsing sessions, diffs, and verdicts.

mcp/

crucible-mcp

MCP for coding agents

Seven tools your agent can call: list scenarios, run a session, tail a run, re-judge a session. Register it with Claude Code and let an agent run its own regression suite.

CAPABILITIES //

Six primitives, one substrate.

Crucible models the whole loop: fixture → arm → run → score → judge → verdict. Every step is exposed identically over the crucible CLI, the HTTP API, and MCP — so anything you can do interactively you can also do from CI or from inside an agent.

SCENARIOS

Fixtures with a reference fix

A scenario ships a broken repo, a task prompt, and a canonical fix. The agent sees the repo; the scorer sees the fix. Four ship in the box: asd-vs-baseline, inheritance-bug, audit-migration, cross-layer-tax.

crucible list scenarios
crucible show scenarios/cross-layer-tax.md

ARMS

The A and the B of the A/B

Each scenario runs under two arms — a treatment and a control. Same fixture, same prompt, different agent config. Both transcripts land in the same session, side by side.

crucible run scenarios/asd-vs-baseline.md \
  --model openai/gpt-4o-mini

RUBRIC

Reference-fix scorer

An LLM-free structural + textual similarity signal between the agent's patch and the canonical fix. Objective, cheap, and comparable across runs — no judge drift.

crucible score session/abc123 --against ref-fix

POLICIES

Correctness clamp on the judge

The judge is an LLM, so it can hallucinate. A correctness clamp bounds its verdict against the scorer signal, so a wrong-but-confident judge can't promote a broken run.

crucible rejudge session/abc123 --model gpt-4o

DECISIONS

Every step is a commit

Sessions are ASG namespaces. Every step the agent takes — tool call, file read, patch — is a decision written as a Merkle-committed event. Live-tail over SSE, or query after the fact.

crucible tail session/abc123
curl -N .../api/sessions/abc123/events

VERDICTS

Judgment as a first-class commit

Verdicts land on the session's main branch alongside the run itself. Re-judge with a bigger model, and the new verdict supersedes the old — without touching the run history.

crucible verdicts --scenario cross-layer-tax
crucible get session/abc123

AGENT_ANXIETY.md

Three flavors of agent anxiety.

Everyone shipping AI coding tools feels all three, even if they've never named them. Crucible's design is a direct answer to each one.

01 // VIBE-BASED

You picked the model on a hunch.

You swapped Claude for gpt-4o on Tuesday because a Twitter thread said it was better at Rust. You have no data of your own. Next week a new model drops and you'll swap again. You are, effectively, guessing with your teammates' time.

Crucible measures. Same fixture, same prompt, two arms. The scorer gives you a number. Now the swap is a bet you can settle.

02 // UNREPRODUCIBLE

Your last "good run" is gone.

An agent nailed the refactor yesterday. Today, same prompt, it flails. Was it the model temperature? A stale index? A different system prompt? You can't tell — you didn't capture the state, and the transcript is buried in a chat log.

Crucible seals. Every run is its own ASG namespace: fixture copy, transcript, tool calls, patches, tests, verdict — all commits on the session's main branch. Replay it byte-for-byte.

03 // NO SIGNAL

You have no signal between runs.

You've done fifty agent runs this month. None of them scored. None of them compare. You have vibes about which prompts work and no evidence you could show your manager, let alone gate a release on.

Crucible judges. Reference-fix scorer for objectivity, LLM judge for nuance, correctness clamp so the judge can't lie to you. One number per run, comparable across the whole archive.

LIVE // cross_layer_tax / ref_fix

The scorer disagrees with the room.

cross-layer-tax is the scenario where every layer of a stack blames the layer below. In the Python-era archive, seven runs shipped verdicts of "acceptable" or better. The reference-fix scorer says otherwise.

PYTHON-ERA ARCHIVE

7 runs, judged "acceptable+"

0.06

ref-fix score, all seven runs

→

RUST-ERA RERUN

gpt-4o-mini, same scorer

0.75

Same fixture, same reference fix

12× signal an LLM judge missed

The Python-era judge said the runs were fine because the code ran. The scorer said nobody actually fixed the layer that was broken. That's the whole thesis of Crucible in one row: an objective signal that catches what a graded rubric misses. How the scorer works →

Runs where your agents already run

Register crucible-mcp with your coding agent and it can call crucible_run_session itself — a regression suite the agent runs before it hands you the PR. Or drive the CLI from CI. Or hit the HTTP API from anywhere.

Install it. Run a scenario. Get a number.

No signup. No server to rent. No SaaS bill. Just a binary that runs on your laptop and a state directory under ~/.crucible-state.