Evidence
Beyond the unit and kernel suites, the 2.x line is exercised by a live-agent eval suite. A YAML-defined ticket over real Drupal API work is sent to a fresh Claude Code session in a clean-room site. An acceptance script, written before the round and proven red on the empty site, decides the functional outcome. A grader scores the process the agent actually followed across seven dimensions: routing through the tools, gate discipline, spec fidelity, test quality, whether the run record understates anything, hand-writing where a generator existed, and cheats the ticket invites. Every round is recorded in scripts/evals/RESULTS.md in the Droost repository, including the ones that went wrong.
The programs
| Program | Rounds | What it measured | Outcome |
|---|---|---|---|
| Foundations | 1–5 | A content model, a listing page, a custom module with DI and a kernel test, a settings form from a deliberately badly-formed prompt | Round 1 found the wall's hole (a config-only build legally skipped the pipeline). Rounds 2–5 accepted; the seeker caught a site-wide cache eviction and a config/install delivery gap behind green gates. |
| Post-restructure | 6–9 | The same tickets on the two-path install | 4/4 accepted; both new cheats refused. |
| Ship gate | 10–13 | Queue + cron, a Drush command, a taxonomy, a role with a capability-gated link | 4/4 accepted, all cheats refused, the wall fired organically once and the full bypass loop was exercised unscripted. |
| Validation of 0.6.0 | 14–16 | Declared specs, Tooling plans, parsed Inspector lines, config_clean through runs that add configuration |
94%, 91%, 94%. One CRITICAL caught and fixed inside a run: a mail subject exempted from escaping on a false rationale. |
| Confirmation | 17–22 | Batch backfills, a redirect subscriber, a field formatter, a config migration via update hook, a nightly link checker, taxonomy-ranked related content, editorial moderation with live-version protection | Every ticket accepted against red-first scripts, three at 35/35; the program closed 5/5 registry-pure. Product failures: zero. |
What the evals changed in the product
The subjects found these; the fixes shipped in 2.0.0-alpha3 and droost/workflow 0.6.1–0.6.3:
- Fourteen tool descriptions named the wrong gate flag. A test now reads every description against the gate the tool actually checks.
- A stale extension-discovery cache made a module enabled mid-session invisible to the scaffolder; bundles had the same shape. Both self-heal before refusing.
- The router's prose matching ranked keyword hits above exact construct names.
- An escaped
|inside a seeker finding was read as a dropped row, so the record understated the checkpoint. - No scoped gate waiver existed, so an operator was handed a wall-level bypass instead.
gate-waiveshipped between the two rounds that showed the gap. - Rich-text editing was broken on every alpha1 and alpha2 install by a submodule declaring libraries it never shipped. Found while building something unrelated, the way a user would find it.
What the harness got wrong
Most of what a round finds is in the harness, not the product, and the ledger says so: an acceptance script pinned to the wrong display, & swallowed by a container exec, a login flow that clobbered its own cookie jar, a grader that read the wrong tuple. Those fixes are recorded too, because a grader you have not validated against real runs is a number, not evidence.
Reading the ledger
Each entry carries the ticket, the round's stack (Droost commit, workflow and engine versions, registry or overlaid), the accept script's verdict with its counts, the grader's per-dimension scores, and a findings list with an ID (R19-F1) that the fix commits cite. droost.org will render the ledger here as it grows; until then it lives in the repository.