Test automation¶

How the north star gets verified: tests drive the same workspace operation set users and roostctl drive, and assert on its events and state. No test-only backdoors that can drift from what ships.

Audience: Claude (primary) + the maintainer. Targets: dev Macs, the Pop!_OS (COSMIC/Wayland) box, and CI (Linux + macOS runners).

The three layers¶

tools/README.md is the source of truth for this map; the summary:

Reach for the highest layer that can answer the question. Layer 1 is where most new coverage lands: it needs no compositor, no macOS Accessibility (TCC) grant, and no Wayland pointer mapping, which is exactly what makes "Mac E2E in CI" tractable at all. cargo test / swift test remain the fast first line below all three; tools/perf/ measures cost on a sibling axis rather than correctness.

Principles¶

• Robustness lives in the driver + app affordances, not the test language. Flake resistance comes from waiting on conditions, reading content as text, reproducible rendering, and driving via IPC instead of OS input — none of which depend on what the cases are written in.
• Drive through the control protocol. The IPC socket is the seam. Every UI action reachable only by keyboard or mouse gets an op (palette.*, tab.copy/tab.paste, screenshot) so a test can trigger it without synthetic input.
• No test ever calls sleep. Wait on a condition: roostctl wait (--state, --text, --gone) or the harness's wait_* helpers. wait is poll-backed today even though an in-process iced UI socket now serves events.subscribe (the Mac's still answers not-implemented) — the interface is what tests depend on, so swapping in wire events later changes no test.
• Content as text, pixels only when targeted. Assert with tab.dump, not OCR or whole-window image diffs. Layer-2 checks are targeted color probes ("this cell is amber #f0a040"), never golden full-frame comparisons.
• Isolation by construction. Each test creates its own project and cascade-closes it, and a harness-launched UI runs against a throwaway ROOST_STATE_DIR — a run never touches the developer's real saved tabs.

Test-mode IPC ops¶

ROOST_TEST_MODE=1 at UI launch (CI sets it; make *-ci sets it) unlocks ops that would otherwise be a footgun. Both gated ops answer not-enabled when the flag is absent — a deterministic error rather than silent acceptance. The flag is read once at boot, so a tester cannot toggle the gate mid-session.

Together the first two cover byte-level OSC and reply wiring end to end without needing a real shell — the pattern walk is in tools/roosttest/README.md. tab.dump_resolved is ungated for the same reason the gated pair is vision-compliant: it is a richer read of an existing surface, not a new one.

A sibling gate, ROOST_TEST_PANIC, forces the crash-report + abort path in the Rust UI end to end (=1 panics on the main thread at startup, =thread from a named background thread). It fires right after the panic hook is installed and before the single-instance lock, so it never touches a running instance.

Harness flags¶

Full operational detail lives in tools/roosttest/README.md.

The skip policy (the trustworthiness rule)¶

A skip must mean only "this environment genuinely cannot exercise this" — never "the setup didn't work" or "we didn't turn the mode on." Three helpers in tools/roosttest/util.py enforce it:

• precondition(ok, reason) — a setup precondition (seed config present, OSC 7 cwd tracking working) is a hard failure in fresh mode, where the harness guarantees the environment; a graceful skip otherwise, since an ad-hoc dev UI may genuinely lack the capability.
• skip_on_ci(reason, alt_coverage=…) — for the rare test that cannot run remotely. It must cite where the regression class is otherwise covered.
• cwd_reaches(...) — the shared, scaled cwd poll, replacing per-file copies that ignored ROOST_TEST_TIMEOUT_SCALE.

Every run prints a SKIPS: N summary listing each skipped test and its reason (conftest.py::pytest_terminal_summary), so a run that quietly skipped half the suite can never read as "all green" — the failure mode that motivated the rule.

Make targets¶

make e2e dispatches rather than running the whole tools/roosttest directory: that directory contains modules that deliberately end the UI they drive, which under the iced default would strand every module the session-scoped fixture runs afterward. e2e-iced runs a curated list instead. The enumerated-list trap: a new roosttest module needs adding to ICED_E2E_TESTS in the Makefile and to the corresponding lists in ci.yml, or it never runs on the iced target.

The Mac real-input harness¶

The functional suite's app.key_event and app.context_menu_* ops start after winit and AppKit have had their say, so a bug between the OS and Roost is invisible to them. The Mac real-input harness (tools/input/mac/) closes that gap with real CGEvents and the Accessibility API, driving a Roost-Iced the test launches itself. It covers Option as Meta by side, Shift+Enter, the native right-click popup, full screen, Secure Keyboard Entry, selection auto-scroll, the remembered window frame and SGR mouse clicks, plus two controls on the helper's own foreign key-window guard, against a small fixture app built per run (tools/input/mac/fixtures/key_panel.swift).

• Two modes. On a dev Mac the harness runs in runner mode: its freely rebuilt helper (roost-input-mac) runs as a child of Roost Test Runner.app, a tiny app that is built once, granted Accessibility, Input Monitoring and Screen Recording once, and never rebuilt (a rebuild changes the ad-hoc signature and voids the grants). On GitHub's hosted macOS runner it runs in direct mode (ROOST_REAL_INPUT_MODE=direct): the job's own processes already hold those grants, so there is no grant step. CI runs the whole suite as a blocking step of the macOS/wgpu iced-build-e2e cell (promoted after three consecutive qualifying attempts, plan 074).
• Running it. make e2e-iced-real-input-mac builds the helper and the bundle and runs tools/roosttest/test_real_input_mac.py with ROOST_REQUIRE_REAL_INPUT=1, which turns every "cannot run here" skip (a missing grant, a locked console) into a failure, per the skip policy below.
• Its own UI. The module is marked owns_ui: the shared launcher stands down and each test launches the branch's mac/build/Roost-Iced.app itself, under ROOST_BUNDLE_PROFILE=linux, so it lives in the Roost-linux namespace, apart from an installed Roost-Iced. Because that namespace is one fixed location, a run holds /tmp/roost-real-input.lock and refuses to start while the namespace's socket answers. Each launch gets its own state directory, ROOST_TEST_MODE=1 and a config that keeps local tabs in-process (no roost-session starts) and turns agent hooks off.
• Safety rules. The launched app is found, focused and quit by pid only, after its environment proves it is this launch's; never by bundle id or process name, since the branch build shares the installed app's bundle id. Selection assertions read the private selection pasteboard, never the general one. Every event is preceded by its own target check, and every press is journaled so a failed or interrupted run releases exactly what it held and nothing the person at the desk is holding.
• Other apps may stay open. A key is refused only when another app claims the keyboard with a window in front of Roost's, the way a non-activating panel (Spotlight, a launcher, a password manager) does. A Zed, a second Roost-Iced or any other app that claims the keyboard from a window behind Roost's no longer blocks the keys (#604).
• The screen-lock caveat. Real events need an unlocked console; a locked screen is a skip, or a failure under ROOST_REQUIRE_REAL_INPUT=1. The harness Mac must stay logged in and awake.

The one-time setup, the helper's commands and its accepted limits live in tools/input/mac/test-runner/README.md.

CI lanes¶

The single required check is ci-success, which derives its membership from its own needs: list — ten jobs: the changes path-filter job plus the nine required ones below. (The table's last row, e2e-iced-wayland-drag, is a non-blocking signal lane and is deliberately not one of the ten.) changes and ci-success always run; every other job is conditionally gated on changes' outputs, so a PR pays only for what it touches.

The six host-session client lanes inside iced-build-e2e. They need a UI and a roost-session daemon, so they are in neither ICED_E2E_TESTS nor SESSION_E2E_TESTS and run as their own steps — e2e-host-client, e2e-host-ssh, e2e-host-missing-daemon, e2e-host-local-spawn (added by plan 044 for #397), e2e-host-localhost (added by plan 058 for #460: the literal localhost target, its ↻ Restart verb, and the restart behind it) and e2e-host-bootstrap, each with an X11 and a Wayland twin (in the X11 and Wayland cells respectively). Within a cell they run serialized, never beside one another: they share the client's incarnation counter and the default session socket, so two at once bind probes to each other's terminals and fail looking exactly like a product bug. The Makefile targets and the CI steps both carry that warning.

Two notes on the soft lane: it pins ROOST_REQUIRE_REAL_INPUT=1 so a missing cage, /dev/uinput, or binary fails rather than silently passing as a skip, and it tees diagnostics to an uploaded artifact — continue-on-error hides a failure's conclusion, so the log body has to be the honest record. Promotion to required is tracked separately.

Both required E2E lanes run --roost-fresh with a throwaway ROOST_STATE_DIR and upload diagnostics on failure (the mac lane additionally emits JUnit XML for GitHub's test annotations). Neither reruns failed tests: a genuine intermittent bug must not be masked. The macOS lane additionally scales its timeouts (ROOST_TEST_TIMEOUT_SCALE) and clears any stale socket/lock before launching, since a crashed instance's held single-instance flock is the one cascade mode that wedges the next run.

Proving a flake fix. A red run on a test the commit did not touch is a defect to remove, not a rerun button. A race is fixed in the test by removing the race, never by widening a budget — a budget moves only for a proven runner-load case, and only through ROOST_TEST_TIMEOUT_SCALE, never a bigger bare constant. One green run is not evidence about a race, so the proof is a loop over the built binary — before the fix, to show it reproduces; after, to show it does not — on the OS it was seen on:

BIN=$(cargo test -p roost-engine --features server-vt --test tab_task_test --no-run 2>&1 | sed -n 's/^ *Executable .* (\(.*\))$/\1/p')
for i in $(seq 100); do perl -e 'alarm 120; exec @ARGV' "$BIN" >"/tmp/run-$i.log" 2>&1 || echo "FAIL $i exit=$?"; done

alarm is what makes a hang a counted failure — it reports as exit=142 (macOS has no GNU timeout, and a pending alarm survives exec); any other non-zero exit is the test itself. Taking the path from cargo's own Executable line rather than ls -t matters: a stale hash from another feature set — tab_task_test built without server-vt is an empty binary — would run zero tests and pass. Check that grep 'test result:' /tmp/run-1.log shows a non-zero count, and put the before/after tallies in the PR. #410 is the worked example: a_query_flood… went 8/100 → 0/100 on Linux, and the loop is also what surfaced #409.

Adding coverage¶

• Behavior or content — a Layer 1 case in tools/roosttest/. Use the fixture that yields a clean workspace; assert with tab.dump / tab.list. Remember the enumerated-list trap above.
• Rendering — a Layer 2 check via tools/screenshot/ and pngtool.py. tab.dump is text-only, so anything about color or layout needs pixels.
• Real key/pointer/clipboard behavior — Layer 3. Linux: tools/input/linux/ (uinput/xdotool). Mac: tools/input/mac/ (real CGEvents + Accessibility through the never-rebuilt runner app; see the Mac real-input harness).

Agent verification¶

An agent should be able to verify a change on the machine it has, without a human present. Three skills under .claude/skills/ carry the recipes: mac-test (a local Mac, or a remote Mac over ssh, including the real-input harness), linux-test (Linux from a Mac, in a shed VM) and popos-test (a native Linux dev box). On a Mac, make mac-real-input-check answers "can this Mac take real input right now?" (grants, lock, Secure Input, input layout) before a real-input run. Skills use placeholders ($MAC_HOST, $MAC_REPO, TEST_SHA, <shed>) rather than machine names.

The guiding rule: an agent should be able to go from "I changed X" to "here is the exact command that proves X still works" without guessing.