Testing¶

make lint && make test && make test-race && make build && make test-cli

Everything below that talks to an agent uses craze-fake-agent, so no Cursor credentials are needed.

Go tests¶

make test

Or without make:

go test -timeout 5m -v ./...

Frame goldens live in internal/tui/testdata/ and render through the same TUI model as a live session. CI also runs make test-race: -race over every package with real concurrency (the list is the test-race target in the Makefile).

Docs drift tests¶

Four tests fail when the reference pages lose track of the code, so a new command, flag, key or method cannot ship undocumented. They read the docs from the module root, the way internal/engine/gate_table_published_test.go reads protocol.md:

A failure names the missing command, flag, key or method and the section it looked in. Hidden commands and flags are exempt.

Sub-agent scripts¶

The grok-subagent* fake scripts replay the wire shapes captured from a live grok run — snake_case _x.ai/session_notification with attempt_id, update._meta["x.ai/tool"], and child session ids on the same stdio:

Queue scripts¶

The 008 scripts are the long turns the message queue needs: a turn has to outlive several frames before anything can be typed into it.

Sanitized excerpts of those live captures are committed under internal/acp/testdata/grok-queue/ (fallback.jsonl is the whole interject-fallback turn, interject.jsonl a merged one, queue-changed.jsonl the queue broadcast) and are replayed through a real client.

Cursor's task / task-late / tasks scripts cover the receipt-only path: the session synthesizes the same sub-agent lifecycle from the cursor/task receipt, and the TUI view shows what the receipt carried.

Sanitized excerpts of the live captures are committed under internal/acp/testdata/grok-subagent/ (subagent.jsonl, two.jsonl, cancel-late.jsonl for a cancel while the child still runs, cancel-early.jsonl for a cancel after the child completed) and are what the ACP parser tests and a replay through the client run against, so the repo does not depend on the capture directory.

Plugin scripts¶

The fake's commands script advertises a full catalog from session/new (the same 24-entry catalog the slash-menu goldens use), so a plugin row resolves on turn one; nocommands never sends available_commands_update at all, so a session against it stays in the provisional window — every plugin row qualified — for its whole life, which is the one state a real Cursor session only holds for a few seconds and no golden could otherwise capture. callorder is echo with a receipt: a second text block naming every session/prompt and session/cancel the fake has read so far, in arrival order, which is how internal/agent/expand_test.go's catalog-wait tests prove Cancel aborts the wait rather than racing the next queued prompt.

Fixture plugins live in two places: tests/cli/fixtures/probe-plugin/ (one command, probe-echo, that spends its arguments, and one skill, probe-skill, that does not) is what the Python suite and the manual JSON checks drive; internal/tui/testdata/plugins/ carries the same probe plugin for the Go goldens plus alpha and beta, which each ship a rescue command so a name collision — and the qualified spelling it forces — has a fixture of its own.

Every test that starts a session or runs a binary isolates HOME: craze walks the plugin caches under HOME at session start, so a test that didn't would see whatever a developer happens to have installed. In Go that's t.Setenv("HOME", t.TempDir()) in the test's own setup (internal/agent/session_test.go, internal/cli/json_test.go, internal/cli/frame_test.go, internal/cli/tui_test.go, …); in the Python suite it's the autouse isolate_run_env fixture in tests/cli/conftest.py.

HOME alone does not isolate craze's own files: a developer's exported CRAZE_HOME would still decide where the config file and the session index live. A Go test that reads or writes either also sets CRAZE_HOME — to a temp directory, or to "" so the craze directory follows the isolated HOME — and the Python fixture sets it to <tmp>/craze-home.

Neither isolates the native provider's keys. craze ships a model catalog, so every native model table carries the catalog's providers and the environment variables they take a key from (FIREWORKS_API_KEY, META_API_KEY, …): a key exported in your shell would fund a test's session, change its model list and be redacted in its output. The suites unset every one of those variables before any test runs — the TestMain of internal/agent, internal/cli and internal/tui (the last before it captures pristineEnv) through modeltable.CatalogEnvNames(), and isolate_run_env by reading internal/harness/modeltable/catalog.toml itself — so a catalog change needs no edit to either. A test that wants one of them sets it itself. Tests that write their own model files keep exactly their own table: modeltable.Save writes catalog = false for a Table whose NoCatalog is set, raw fixtures (tests/cli/sse_fixture.py's write_native_config, internal/harness's) write the key themselves, and modeltable.LoadWith(dir, nil) loads with no catalog at all.

To check the isolation holds, run the suites with every catalog variable exported to a dummy value:

env ANTHROPIC_API_KEY=dummy-key-0006 FIREWORKS_API_KEY=dummy-key-0001 \
    META_API_KEY=dummy-key-0002 OPENROUTER_API_KEY=dummy-key-0003 \
    ZHIPU_API_KEY=dummy-key-0004 ZAI_API_KEY=dummy-key-0005 make test test-cli

No test reaches Anthropic's API. A load reads the Anthropic key only to bind the account's model list to it (anthropic-models.json), through a seam the modeltable tests replace, and the list's fetcher (internal/anthropicmodels) fetches through a variable its tests replace. Code that does build the real client — the driver, the fetcher's default — takes Anthropic's origin from CRAZE_TEST_ANTHROPIC_API when it is set: a loopback http URL (http://127.0.0.1:<port>) to a test's fake server, and nothing else — any other value is an error, never ignored, so a test that misnames its fake fails instead of sending a key to Anthropic. The TestMain of internal/anthropicmodels, internal/agent, internal/cli and internal/tui points it at a loopback port nothing serves — a native session fetches the account's list as it opens whenever a key funds Anthropic, and craze auth login anthropic fetches it too — and so does tests/cli's conftest.py, for every case, replacing any value exported in your shell (a live check's logging proxy, say). A test that wants Anthropic starts a fake and points the variable at it: t.Setenv in Go, the child's environment in tests/cli, whose anthropic_fixture.py serves the model list's pages and scripted Messages streams and records every request (test_anthropic.py).

To drive a plugin command through the JSON interface:

CRAZE_FAKE_SCRIPT=commands ./bin/craze prompt --json \
  --agent-bin ./bin/craze-fake-agent \
  --plugin-dir tests/cli/fixtures/probe-plugin \
  "/probe-echo banana"

and through craze frame:

./bin/craze frame --cols 100 --rows 30 \
  --agent-bin ./bin/craze-fake-agent --fake-script commands \
  --plugin-dir tests/cli/fixtures/probe-plugin \
  --keys "/probe-echo banana<enter><wait:text:PROBE-COMMAND-EXPANDED>"

Both isolate HOME themselves in the test suites above; run by hand like this, they read whatever plugins your own HOME happens to have cached in addition to the fixture.

craze frame¶

craze frame is a hidden command that runs the real TUI model with no terminal attached, feeds it a key script, and prints the final frame. It is what the goldens render against.

./bin/craze frame --cols 100 --rows 30 \
  --agent-bin ./bin/craze-fake-agent --fake-script todos \
  --keys "go<enter><wait:text:TASKS>"

There is no --workspace: craze frame runs in the current directory. It never reads ~/.craze/config.toml, so a saved theme cannot reach a golden.

--seed-session rows are written inside the isolated HOME, and --continue / --resume resolve against that isolated index, so a replay or picker golden can never see — or touch — a developer's real ~/.craze/sessions.jsonl, or the one an exported CRAZE_HOME points at.

craze frame cannot drive the real cursor-agent. It points HOME at an empty directory and unsets CRAZE_HOME for the duration of the run — that is what keeps a developer's config and skills out of a golden — and cursor-agent loses its credentials along with it. Use ./bin/craze-fake-agent. To drive a live agent in a terminal, use the tmux smoke or just run craze.

In a key script, literal text is typed rune by rune and anything in angle brackets is a token:

<enter> <esc> <tab> <backspace> <delete> <space> <up> <down> <left> <right>
<pgup> <pgdn> <shift-tab> <alt-enter> <ctrl-a>..<ctrl-z> <lt>
<wheel-up> <wheel-down> <click:X,Y> <resize:COLS,ROWS> <sleep:250ms>
<press:X,Y> <motion:X,Y> <release:X,Y> <drag:X1,Y1,X2,Y2> <dblclick:X,Y>
<hover:X,Y>
<paste:one\ntwo>
<wait:idle> <wait:working> <wait:card> <wait:copied> <wait:text:foo> <wait:gone:foo>

<motion:X,Y> stamps the left button, so it models a drag. <hover:X,Y> is motion with nothing held, which is one of the two things a queued message's [send now] [edit] [cancel] strip appears on — the other is the row the keyboard has selected.

--freeze stops the clock and the spinner cycle for the whole run. A frame of a turn in progress is otherwise a race against wall time: the elapsed counter and the spinner glyph both move on their own, so the same script captures a different frame on a slower build. Every golden of a running turn uses it.

CRAZE_FAKE_STEP sets how long each of the fake's long-turn tool steps takes. It is a comma-separated list, one entry per turn with the last repeating: 2s,1ms is a first turn slow enough to type into and later ones that finish at once, which is what a frame of a drained queue needs.

Python CLI suite¶

make test-cli runs the Python suite in tests/cli under uv:

• test_prompt.py — the headless prompt --json path
• test_frame.py — craze frame through the binary, for the same scripts the Go goldens cover
• test_tui.py — a real PTY
• test_auth.py — craze auth login|logout|list against a temp CRAZE_HOME: the messages, the refusals that save nothing, and canaries proving a stored key reaches the provider's wire and nothing else (the no-echo prompt is the Go pty test's, internal/cli/auth_test.go)
• test_sessions.py — the session list in real PTYs sharing one HOME, each in a directory of its own: groups, regrouping, ctrl+x, opening a session in place, quitting from the list, an unreachable host, resuming a saved session. Its Screen plays craze's output, one whole frame at a time, into an 80×24 grid, so a case reads the screen as it stands rather than the bytes that went past

make test-cli

CRAZE_FAKE_LINGER=1 keeps the fake agent alive after its stdin closes, until a signal ends it or 30 seconds pass. Without it the fake exits the moment craze does, however craze exits — killed outright included — so a check that the agent is gone could not tell an agent craze shut down from one it orphaned. The SIGHUP cases in test_host_status.py set it.

CRAZE_FAKE_STUBBORN=1 is that for 90 seconds, and the fake also ignores SIGTERM, SIGINT, SIGHUP and SIGPIPE, so only SIGKILL of its process group ends it sooner: an agent that outlives its host. The spawner's test that kills a detached host's recorded agent groups sets it.

CRAZE_FAKE_STDERR=<line> makes every fake-agent script except hang and hang-ack write that line to stderr once at startup and once per session/prompt. hang and hang-ack stay silent, the same house rule that keeps their behaviour otherwise unchanged. It is the canary the test_tui.py cases for issue #23 use to prove the agent's own stderr is gated on the run having failed.

The exit-two-lines script is the fake's start failure: it reads craze's initialize, leaves it unanswered, writes Error: KEYCHAIN LOCKED and Run unlock and retry. to stderr and exits 1 — an agent that dies at its start the way cursor does on a locked keychain. It reads the request first so the call always fails with the exit's status, never with the broken pipe a write to an agent already gone would get. The start-failure tests (internal/cli/starterr_test.go) use it, with CRAZE_TEST_GUI_SESSION=0 or 1 forcing the macOS login session craze reads (rundir.GUISession) — honoured only in a test binary's own processes — so the login-session hint is tested on every OS. Those that pin the agent's exact words set CRAZE_TEST_STDERR_TAIL_WAIT=10s (acp.StderrWaitEnv, honoured the same way), which stretches craze's two half-second waits for the copy of the agent's stderr — the reaper's before it closes the pipe, and the failed start's — so a starved copy cannot cut the lines. A wait ends as soon as the copy does.

CRAZE_FAKE_SESSION_ID=<id> makes session/new answer that session id instead of fake-session-1. The session index keys a row by provider and that id, so two sessions of one test HOME under the default would be one row; test_sessions.py names each after its workspace.

Verification scripts¶

scripts/verify/ holds the reviewed helpers a plan runs on a committed sha, exported to a short scratch path, never on the working tree. Each prints its usage with --help, ends with a <NAME>_EXIT= line, and writes its output to ~/.cache/craze-verify/<label>/.

A failure is diagnosed, not re-run. A live check against the real cursor-agent, grok, gx or native in a dedicated tmux server, on Linux or macOS, follows the craze-live-smoke skill (.claude/skills/craze-live-smoke/).

tmux smoke¶

tests/cli/tmux_smoke.py drives the real binary in a real terminal at 100x30 and 80x24, one run per case: every fake script, plus cases for the model dialog, the sub-agent view (both providers), a real mouse drag, the queue band (queue) and a grok interjection (grok-interject). It is opt-in and never runs in CI — pytest only collects it when it is named explicitly, and it skips unless tmux is on PATH and CRAZE_TMUX is set.

cd tests/cli && CRAZE_TMUX=1 uv run pytest -v tmux_smoke.py
cd tests/cli && CRAZE_TMUX=1 uv run python tmux_smoke.py --cases echo,todos

Screen captures land in ./smoke-captures/ (git-ignored), overridden with --out or CRAZE_SMOKE_OUT.

The suite is isolated from your machine: its own tmux socket (-L), its own configuration (-f /dev/null, so no ~/.tmux.conf hook can reach the panes), its own HOME, and stub wl-copy/xclip/xsel on the pane's PATH so a copy never touches your real clipboard.

Evaluation harness (eval/)¶

eval/ is a separate evaluation harness for the native provider: sandboxed runs of native, gx, opencode and codex against the same tasks and models, scored objectively and judged blind by codex exec. It is not part of CI or the per-commit gate, and it spends real API money — do not run it as part of an ordinary change.

Only the recording proxy inside eval/ ever holds a real provider key; the sandboxed harnesses it drives see a dummy key and a proxy URL. Nothing under ~/.craze, ~/.grok, ~/.config/opencode, ~/.local/share/opencode or ~/.codex is ever written by a run, only read.

Changes to native's system prompt, tool descriptions or request parameters are measured against this harness before they ship (see discovery/native-harness/08-decisions.md, D-69).

See eval/README.md for the commands (crazeeval validate, run, judge-batch, report, …) and the full shape of a run.

The procedure for running a full evaluation campaign — baseline, lever loop, final, summarize — is eval/RUNBOOK.md. A campaign's rollup (a human README.md plus the compact summary.json that crazeeval summarize makes) is committed under eval/results/<campaign>/; plan 029's are in eval/results/2026-09-plan029/. The full archive (crazeeval archive output — every run and verdict) lives in the private charliek/craze-evals repository instead, and raw runs stay local, never uploaded anywhere.