12. Evaluation

CMD-9 is done when the scenario suite passes against ud mcp, run by Claude Code with Figma’s skills and with our skills. The suite also gates the 0.7 milestone (“Design a settings screen yields a lint-clean screen using only system components”).

12.1 Scenarios#

Scenarios live in test/agent-scenarios/<id>/ with a scenario.yaml, a fixture design (and library), and optional reference images:

id: S1-login-screen
prompt: "Design a login screen for the mobile app on the Onboarding page."
fixture: fixtures/app.design
libraries: [fixtures/system.design]
skills: [figma, ud]          # run once per skill set
models:                      # one full, dated model name per tier, pinned; aliases (opus, sonnet) are not allowed here
  design: <full model name of the Design tier default at the time of pinning>
  assistant: <full model name of the Assistant tier default at the time of pinning>
runs: 10                     # N per cell
passAt: 8                    # cell passes at ≥ passAt / runs, with zero tool-schema errors in every run
allowedTools: "mcp__ui-designer__*,Read,Grep,Glob"   # passed to claude --allowedTools; ud mcp --allow derived
allow: "read,use_figma,ud_apply,ud_lint,ud_render,ud_help,selection.set"
seed: 1337                   # UD_ID_SEED for deterministic ids
checks:
  - tree: 'frame[name~="(?i)login"]'           # exactly one new top-level frame
    count: 1
  - lint: { node: '$frame', errors: 0 }
  - instancesOnlyFrom: [system.design]          # no locally drawn buttons or fields
  - noRawValues: [fills, itemSpacing, padding]  # bound to variables
  - textStyles: all                             # every text node uses a text style
  - render: { node: '$frame', notBlank: true, noClippedText: true }
  - tools: { forbidden: [unsupported], schemaErrors: 0, maxCalls: 40 }

models names the exact model per tier because the pass rate is a property of a model, not of an alias; the file is updated deliberately when a tier’s default (§7) moves, and the report shows the old and new rate side by side for one cycle.

Every render check has numeric thresholds in the scenario file (notBlank is “≥ 5 % non-background pixels”, noClippedText is “no text node whose glyph bounds exceed its box by > 1 px”, similarity as in S6); a check without a number is a bug in the scenario.

12.2 How it runs#

• Runner. npm run test:agents [-- --scenario S1 --skills ud --tier assistant --runs 3] copies the fixture into a temporary workspace, installs the skill set (ud skill --install, or Figma’s skills from the pinned copy), writes .mcp.json with ud mcp, and runs claude --print --output-format stream-json with the scenario prompt and the scenario’s allowedTools (and the derived ud mcp --allow list), headless (no editor). Then it runs the checks with ud against the result and stores the transcript, the final file and renders. --runs and --tier default to the scenario file; the command-line values are for local iteration and are marked “not a gate” in the report.
• Pinned skills. Figma’s skills live in test/fixtures/skills/figma/ with VERSION recording the upstream commit or release and the date they were copied; our skills come from the bundled copy of the build under test. A pull request that bumps the pinned Figma skills runs S1–S4 and S9 with the figma set and reports the rate change.
• Deterministic ids. The runner sets UD_ID_SEED=<scenario seed> so ud generates node, page and proposal ids from a seeded sequence instead of random bytes. Two runs that make the same calls produce the same file, which makes transcripts replayable (below), diffs between runs readable, and checks able to name ids. UD_ID_SEED is honoured only when set explicitly; it is never set in production.
• Matrix. Each scenario runs for each skill set it lists and for the Design and Assistant tier models named in its file, N = 10 runs per cell. The vscode.lm backend is a separate, non-blocking column.
• Pass rule. Models are not deterministic, so a cell is judged on its rate: a cell passes when at least 8 of its 10 runs pass every check and no run has a tool-schema error (a call rejected with usage, or a parameter outside the published schema; one such error fails the cell, whatever the rate, because it means the tool interface or the skill is wrong, not the model). Design tier cells gate: CMD-9 passes when S1–S4 pass for both skill sets on the Design tier; the 0.7 milestone also needs S5 on the Design tier. Assistant tier cells are advisory: their rates are reported and tracked, and a fall of more than two runs against the last release’s report opens an issue but blocks nothing.
• Panel pipeline test. S10 runs on every pull request with no model and no key: ud.ai.claude.command points at test/agent-scenarios/mock-claude/, which replays a recorded stream-json transcript (one of the stored passing S1 runs, ids deterministic through UD_ID_SEED) at a configurable pace and honours SIGINT. It exercises §6.3–§6.6 end to end — argument construction, including --allowedTools, the settings override and --allow; stream parsing; the proposal growing on the bridge; the change list with its explicit groups; Accept as one undo step; Cancel and failure rollback; the Restricted Mode refusal — in the extension test host. Recorded transcripts are refreshed when the stream-json format or the design loop changes.
• When. Nightly and before every release, not on each pull request (cost and time). The report is a table per scenario × skill set × model with pass rate, tool calls, schema errors, duration, tokens and cost, and links to transcripts. The gate for a release is the last nightly report of the release commit.
• Key and budget. Nightly and release runs use a SoftForge-held project key stored in CI secrets (ANTHROPIC_API_KEY scoped to a ui-designer-eval project), never a personal login, with a monthly budget set by the owner in the provider console and mirrored in test/agent-scenarios/budget.json (cost per nightly matrix is in the report, so the budget can be checked against it). When the month’s budget is reached the nightly run stops with a visible “budget” status instead of failing silently. Contributors run the suite locally with their own key; no key is needed for S10 or the replay tests.
• Replay tests on every pull request. Recorded tool-call sequences from passing runs are replayed against ud mcp without a model (npm run test:agents:replay). They catch regressions in tools, schemas and the Plugin API that would break agents, deterministically and in seconds. They complement the schema snapshot test of CMD-8.
• Skill changes. A pull request that changes a skill runs the scenarios that use that skill (manual trigger, label agents) with the full N = 10 on the Design tier. The merge is blocked when a Design tier cell that passed in the last nightly report falls below 8/10, or when any run has a tool-schema error; Assistant tier changes are reported in the pull request and do not block.