Scenarios

A scenario is a YAML+markdown file that declares a fixed comparison: same fixture, same task, two arms, one rubric.

Anatomy

---
name: asd-vs-baseline
description: >
  Does ASD+CTX help gpt-4o-mini fix a real bug faster / better?
fixture_path: fixtures/asd-vs-baseline/repo
test_command: python3 -m pytest -q
reference_fix: fixtures/asd-vs-baseline/REFERENCE_FIX.md
task_prompt: |
  The store package has a failing test. Fix the bug in the source
  (not the tests) and re-run the suite.
arms:
  - id: assisted
    description: Same model with ASD + CTX MCP tools available.
    model: anthropic/claude-sonnet-4-6
    extra_tools: [asd, ctx]
    system_prompt: ...
  - id: baseline
    description: Same model, filesystem+bash only.
    model: anthropic/claude-sonnet-4-6
budget:
  max_turns: 25
  max_total_tokens: 60000
  max_wall_clock_seconds: 300
policies:
  - id: P1
    rule: Do not modify any file under tests/.
    enforcement: block
    matchers:
      - tool: write_file
        arg_key: path
        op: starts_with
        value: tests/
rubric:
  dimensions:
    - name: correctness
      weight: 0.40
      criteria: Tests pass. Bonus if the fix matches reference.
    - name: minimality
      weight: 0.25
      criteria: Diff as small as reference. No incidental changes.
    - name: reasoning_quality
      weight: 0.20
      criteria: Located the bug deliberately, not by trial and error.
    - name: efficiency
      weight: 0.15
      criteria: Fewer tool calls, fewer tokens at equal quality.
---

The markdown body below the YAML is human commentary — the loader ignores it.

Bundled scenarios

Four ship in-tree. Each has a seeded bug, a failing test suite, and a REFERENCE_FIX.md with the intended patch inside a fenced diff block marked .

asd-vs-baseline

The value hypothesis for ASD+CTX. Single-file bug in a pricing function. Fixture resets in ~100ms; test suite is ~10ms. Cheap to run over and over.

inheritance-bug

An __init__ that forgot to super().__init__(). Baseline can find it with grep; assisted should locate it faster via ASD’s call graph.

audit-migration

Two independent bugs — an ImportError masking a missing serializer field. Tests whether the agent iterates until the whole suite is green (persistence) rather than stopping at the first fix.

cross-layer-tax

Four-layer commerce stack. Naive fix at the proximate layer (apply_tax) passes some tests but leaves others red. The reference fix touches rates.py and display.py. Reference-fix score across the v0 Python-era archive was ~0.06 — no agent found the right layer. Rust-era gpt-4o-mini runs score ~0.75+.

Authoring one

Two things you write, one thing that comes for free:

Fixture — a repo (git-init’d or not) under fixtures/<name>/repo/ with a failing test suite. Any language the CI image can run.

REFERENCE_FIX.md — human explanation followed by a marker + fenced diff:

<!-- crucible:reference-patch -->
```diff
--- a/store/pricing.py
+++ b/store/pricing.py
@@
-    return raw * pct
+    return raw * (1 - pct)
```

Scenario .md under scenarios/<name>.md — YAML frontmatter as above.

Comes for free: the scorer (parses that diff, compares the run’s final diff, computes structural + textual similarity), the judge (loads the rubric, invokes the model, clamps hallucinated correctness scores), and the persistence + UI (once the file is in scenarios/, it shows up in crucible list-scenarios and the UI’s Scenarios page).