Calipers

← The sequence

Boot it yourself and read it like a reader

02of 09starts here614 wordsqa/harness.mjs

The failure it prevents

The audit only ran when someone remembered to start the dev server first, so it stopped running. When it did run, it scrolled straight to the bottom — every IntersectionObserver-gated component was still unmounted when the measurement happened, so the audit measured an empty page and called it clean. And one route that 404'd took the other nine routes' findings with it, which reads as "the audit is broken" rather than "this route is broken".

The prompt

Build `qa/harness.mjs`: the part of a rendered design audit that boots the app, opens each route, reads it the way a person does, and isolates failures. It contains no checks — it hands measured data to checkers. Use `puppeteer-core` (not `puppeteer`; do not download a second Chrome).

**Find Chrome, and fail loudly if it is missing.** Try `process.env.CHROME_PATH` first, then a platform candidate list (`/Applications/Google Chrome.app/Contents/MacOS/Google Chrome`, `/Applications/Chromium.app/Contents/MacOS/Chromium`, `/usr/bin/google-chrome`, `/usr/bin/chromium`, and the Windows Program Files paths). If none exists, **throw with the full list of paths tried and the CHROME_PATH hint**. Never `process.exit(0)` on a missing browser — a check that silently skips itself in CI is worse than no check, because the green tick is a lie.

**Boot the app on a port nobody is using.** Get one with `net.createServer().listen(0)`, read `address().port`, close it, then spawn the dev server on that port. Detect the package manager from the lockfile (`pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `bun.lockb` → bun, else npm) and read the dev command out of `package.json` scripts — do not hardcode either. Capture stdout and stderr into a string; stream it to stderr only under `--verbose`.

**Wait for it properly.** Poll `GET base` every 400ms with `AbortSignal.timeout(4000)` per attempt against a 90s deadline. Return on the first `res.ok`.

**Reuse a server that is already running.** Many dev servers refuse to start a second instance and print the URL of the existing one. If the boot times out, search the captured log for that message, extract the `http://localhost:NNNN` (strip ANSI escapes), verify it responds within 10s, and measure that one instead — recording `reused: true` and returning a **no-op `stop()`**, because it was not yours to stop. This single behaviour is the difference between an audit people run and an audit that demands they close their editor first. Also support `--base <url>` to skip spawning entirely and measure a preview deployment.

**Read each route the way a reader does.** Not `scrollTo(bottom)`. Measure `document.documentElement.scrollHeight`, divide the scrollable distance into 14 steps (minimum 200px each), and at each step `window.scrollTo({top, behavior:'instant'})` then wait 320ms. Components gated on entering the viewport need to actually enter it, mount, and start. **Time is an explicit input to this audit** — put the step count and dwell in named constants with a comment saying what they are for, so the next person tunes them instead of deleting them.

**Wait for real content, generously.** `waitForSelector` on the app's main content root (`main`, `article`, `[data-testid=…]` — read the app and pick one) with a **120 second** timeout, and write the reason in a comment: a first-request dev compile reported as a failed audit is a false alarm, and nobody trusts a flaky gate twice. Then a short settle wait (~1200ms) before measuring.

**Isolate failures per route.** Wrap each route's measurement in try/catch; on error, record a FAIL for that route naming the error message and continue to the next. One broken route must never suppress nine routes' findings.

**Flags:** `--route <path>` (measure one), `--base <url>`, `--json`, `--strict`, `--quiet`, `--verbose`. Always close the browser and stop the server in a `finally`.

**Route discovery:** enumerate routes from the framework's own file conventions (app/ or pages/ directory walk, a routes manifest, or the sitemap) — and if the project keeps a list of published pages somewhere, read that list rather than copying the slugs into the harness. Print the routes it found before measuring, so a route that silently stopped being audited is visible.

**Hand back:** the file, the printed route list, and one full run against this project showing per-route timings. State the real wall-clock time — if the first run takes four minutes because of a cold compile, say four minutes.

What it produces

qa/harness.mjs — Chrome discovery with a loud failure, free-port dev-server boot with reuse-and-don't-kill, ready-polling, stepped read-through, per-route failure isolation, route discovery, and the standard flag set.

How you prove it works

Three deliberate breakages. (1) Add a route whose component throws; run the harness — that route reports a failure and every other route still reports its findings. (2) Run CHROME_PATH=/nope node qa/harness.mjs — it must exit non-zero with the list of paths tried, never exit 0. (3) Start the project's dev server yourself in another terminal, then run the harness: the output must say it reused your server, and your server must still be running when the harness exits.