Calipers

← The sequence

FAIL / WARN / NOTE, a facts bag, and one verdict

09of 09after 05 · 07 · 08640 wordsqa/gate.mjs

The failure it prevents

Five checks with five output formats and five ideas of what "failed" means is five things to remember, and the one you forget is the one that breaks. Meanwhile the audit is a red light: it says pass or fail, so nobody can answer "did last month's refactor make layout shift better or worse" without going back and re-running an old commit. And the check that is disabled because of a known backlog has been disabled for a year, because nothing anywhere records what it is waiting for.

The prompt

Unify this project's design checks behind one severity model, one facts bag and one runner. Assume the individual checks already exist.

**Part 1 — three severities, written down where the checks live.**

- **FAIL** — a contract rule is broken, or the codebase has diverged in a way that costs real work to unwind. Breaks the run.
- **WARN** — a real divergence that might be a deliberate call. Printed; breaks the run only under `--strict`.
- **NOTE** — inventory. What the code does today, so the *next* run's diff shows what moved. Never breaks anything.

Build a small `Report` class with `fail/warn/note(check, where, note, detail?)` and `fact(key, value)`. Findings group by check id when printed. `where` is a location a person can open — `file.tsx:42`, or `route @ 390px`. `detail` is an array of lines printed indented under the finding: the culprit list, the three worst shifts, the stray text blocks. The constructor takes the audit's **intent** in one sentence, printed under the title, so somebody staring at a red run knows what the audit is defending before they start arguing with it.

**Part 2 — the facts bag turns the audit into a time series.**

Alongside findings, every check records its measurements: `route.cls`, `route.headings` as a level string, `route.textEdges` as `[{left, count}]`, viewport widths, element counts. Serialize them under `--json`. This is the difference between a red light and an instrument. Commit `qa/baseline.json` and diff it between runs: "CLS went 0 → 0.04 on three routes" is a conversation; "the audit is red" is not. Add a `--json` diff note in the README showing the two-command version (`node qa/gate.mjs --json > new.json && diff baseline.json new.json`).

Exit code from `print()`: 1 if any FAIL, or if `--strict` and any WARN. Every check script takes the same four flags — `--json`, `--strict`, `--quiet` (hide the NOTE inventory), and the audit's own scope flag.

**Part 3 — one runner, `qa/gate.mjs`.**

Declare the checks as data: `{ name, cmd: [bin, args], blocking }`. Spawn each as a child process, capture stdout+stderr and elapsed ms, and print one table:

```
  pass  typecheck                 4.1s
  warn  lint                      6.8s
        non-blocking — waiting for the pre-existing no-unused-vars backlog
  pass  source audit              0.9s
  pass  probe self-test           9.8s
  FAIL  rendered audit          214.3s
```

Then print the **full output of blocking failures only** — the summary tells you which check, the detail tells you what actually broke, and nobody has to re-run anything to find out.

**The rule that keeps the backlog honest: every non-blocking check must carry a `waitingFor` string, printed under it on every run.** A check that is `blocking: false` with no `waitingFor` is a config error and the gate must refuse to run — otherwise "temporarily non-blocking" becomes permanent and invisible. `waitingFor` turns a disabled rule into a visible IOU, and the list should only ever get shorter.

**Two entry points, and one specific thing that must appear in both.** A fast gate (source audit + typecheck + lint + the probe self-test — seconds) and a rendered gate (fast, plus the browser pass — minutes). **The probe self-test belongs in the fast set**, always, even though it launches a browser: a probe that has stopped matching reports a clean page, so skipping it is skipping the reason to believe every other green result. If it makes the fast gate too slow, cut the fixture count, never the suite.

Wire both into `package.json` using whichever package manager the lockfile indicates, and into CI: the fast one on every push, the rendered one on pull requests and on the default branch.

**Hand back:** the report module, the runner, both entry points, a full run of each with real timings, and the first `baseline.json`. If any check ships non-blocking, list it with its `waitingFor` and tell me what would have to be true to make it blocking.

What it produces

qa/report.mjs (FAIL/WARN/NOTE + facts bag + one exit-code rule + --json/--strict/--quiet), qa/gate.mjs (declarative check list, per-check timing table, full output for blocking failures only, mandatory waitingFor on non-blocking checks), two package scripts, CI wiring, and a committed baseline.json.

How you prove it works

Four probes of the runner itself. (1) Add a check with blocking: false and no waitingFor — the gate must refuse to run rather than print a friendly warning. (2) Add a check that exits 1 and is blocking — the gate must exit 1 and print that check's full output, not just its name. (3) Add one that exits 1 and is non-blocking — the gate must exit 0 and print its waitingFor line. (4) Run --json before and after a real change (widen a container by 20px), diff the two facts bags, and confirm the changed number appears in the diff — if it does not, the checks are recording verdicts and not measurements.