Testing — tcm-ts

Testing

A test is a definition named test…, or any definition under a tests namespace. It passes if it returns (or resolves) without throwing. tcm test [namespace] runs them.

// tcm: namespace lib
import { expect, vi } from "vitest";

export const testSortIsStable = () => {
  expect(sortWith(byKey, [/* … */])).toEqual([/* … */]);
};

export const testRetriesOnce = async () => {
  const fetch = vi.fn<() => Promise<number>>().mockRejectedValueOnce(new Error("down")).mockResolvedValue(1);
  expect(await withRetry(fetch)).toBe(1);
  expect(fetch).toHaveBeenCalledTimes(2);
};
  • Vitest, with nothing to set up. Each test definition is one Vitest test. expect, vi (mocks, spies, fake timers) and the rest come from vitest, which ships with tcm. Failures read like Vitest's, diffs included. Tests that use node:assert, or just throw, work too.
  • Where they run. tcm test runs them, and the suites. The watcher, tcm check and VS Code run a scratch file's tests inline.
  • What ran is named. tcm test prints a line per test, (cached) or how long it took. The MCP server's tcm_test lists them by name under their shared namespace: the ones that ran, plus the cached ones when you named tests or something failed (otherwise it counts them).
  • Tests don't see the session. A test or watch expression doesn't see the session's branch (TCM_BRANCH, sessions) or TCM_BRIEF. Code that opens a codebase of its own won't look for the session's branch there, and what the code prints is what it prints anywhere.

Caching

  • Results are cached by hash, and by the Node and Vitest versions. A test's hash covers everything it calls, so only tests whose code or dependencies changed run again.
  • Platform too, when it matters. A test whose code reaches files, the environment (process.env, node:os) or other processes is also cached per platform (darwin-arm64).
  • Clock and network can go stale. Tests that read them can keep an outdated result. --fresh runs everything.
  • Only passes are reused. A failure may come from the environment rather than the code, so a failed test runs again next time, and passes once the cause is fixed, without --fresh. One that fails again with nothing it runs changed says it failed last time too (often a fix that wasn't saved).
  • Watch expressions are cached the same way (values, not errors). One whose code reaches outside the codebase (Date.now(), Math.random(), a file) is evaluated every time (scratch files).
  • Results travel with the code. tcm push sends the branch's test results the remote doesn't have. tcm pull fetches the ones run on your Node version (and platform, where it matters), so a fresh clone doesn't run them again (sync). A codebase that hasn't run tests takes the remote's Vitest version with them. Pulled results are trusted as the remote's code is.

Suites: testing a command as a program

A package's command (its bin) can be tested by running it. Its suites are test definitions in namespaces the package's manifest lists under suites (command → namespaces, relative to the package):

// tcm: namespace greeter
export const manifest = { name: "greeter", bin: { greet: "main" }, suites: { greet: ["e2e"] } };
// tcm: namespace greeter.e2e
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";

export const testGreets = () => {
  const r = spawnSync(process.execPath, [process.env.GREET_BIN!, "tcm"], { encoding: "utf8" });
  assert.equal(r.stdout, "hello, tcm\n");
};
  • tcm test runs suites after the test definitions. Each test is a program of its own (passing if it returns), not run with Vitest. It runs against the codebase's own build of the command, found in <COMMAND>_BIN (GREET_BIN here, MY_TOOL_BIN for my-tool).
  • Side by side. Suites run as many at once as the machine's budget allows, each reported as it finishes.
  • A test still running after ten minutes fails as timed out. The ten minutes start once it has a slot.
--definitions
only the test definitions, not the suites
--suites
only the suites
--bin greet=<path>
run the suites against another build of the command (comma-separated cmd=<path>), such as the package tcm pack makes, or an older build (below)
--fresh
run everything again, cached or not

tcm run <test> runs one suite test as a plain program, once, uncached. It finds the command as suites do, unless the variable is already set (GREET_BIN=<path> tcm run <test>), which it keeps and says so.

How suite results are cached

A suite test's result is cached by the code it actually ran, not by its hash:

  • What ran is measured. V8 coverage of every process the test starts says which definitions executed. The result holds until the test changes or one of those definitions does. A definition it didn't run can change without running it again.
  • Classes count by method. A change to a method the test didn't call keeps the result. The rest of the class (fields, constructor) counts as one.
  • Constants count when code that ran refers to them, if loading them does nothing but make their value (a literal, or a computation that reaches nothing outside the codebase, as for pure constants). Any other value computed when a module loads counts as run.
  • Any build of the same code shares the result. A result from the codebase's own build holds for a packed build with that code. A build without it (an older one) runs the test again.
  • A tcm the test started must exit. If the test starts tcm (a server it stops at the end), the result waits up to 15 seconds for it to exit. One that doesn't leaves the result uncached.
  • Saves during a run don't confuse it. A result is kept for the code as it was when the run started.
  • The last eight versions are kept per suite, so code put back as it was (after another branch's tests ran) is cached again.
  • Only passes are kept, as for test definitions.

One budget for the machine

A suite starts several processes (tcm, the TypeScript checker, Vite, a browser). So every tcm on the machine takes its heavy work from one shared budget of slots: however many runs there are (agents' sessions, merges, verifies), no more of that work runs at once than there are slots.

What takes a slot:

  • a suite test, while it runs;
  • a run of test definitions: one per Vitest project (Node's, the browser's), with that many workers;
  • an app build (tcm build, and the builds tcm start and a deploy make);
  • each of a sync's checker processes while it checks (sync).

Work done inside work that has a slot (a tcm test a suite runs on a codebase of its own, a build inside a build) uses that slot, so nothing waits behind itself.

  • How many: TCM_SLOTS (TCM_SUITE_SLOTS still works). By default, one per performance core plus one per three efficiency cores (every core, on a chip with one kind), at most one per 2 GB of memory, at least 2. A Mac with 4 performance cores, 6 efficiency cores and 32 GB gets 6. A run given a smaller budget (TCM_SLOTS=2 tcm test) uses the first that many slots, shared with everyone else.

  • First come, first served. A run needing several slots gets them together, so two runs never each hold some.

  • Waits are said briefly, on stderr, naming who holds the slots by branch and kind of work:

    waiting for a slot: 6 of 6 in use (3 by session/2026-11-02-b933ee, 2 by session/2026-11-02-4ad254's test definitions, 1 by main's verify), 3 ahead in line (TCM_SLOTS sets how many)

    It repeats only when that changes much (at most once a minute) or after five minutes (still waiting for a slot after 6m20s: …), and says when it gets one (got a slot after 6m30s). The summary gives the total: suites: 233 passed, 0 failed (19 cached), 6m12s waiting for slots.

  • Slots are files in tcm-suite-slots-<uid> in the temporary directory (or TCM_SLOTS_DIR): slot-<i> holds the pid using it, line-… is a place in line. A slot whose process is gone is free: there's nothing to clean up.

Showing it fails before the fix

A fix comes with a test, and the test should fail without the fix.

  • A test definition: run it before you save the fix.
  • A suite test: run it against a build of the command from before the fix, with --bin (tcm_test's bin for an agent, {"greet": "<path>"}). A build without the fix's code runs the test again, whatever passed on the codebase's own build.
tcm test greeter.e2e --bin greet=<the build without the fix>   # fails
tcm test greeter.e2e                                          # the codebase's own build, with the fix: passes

Tests in a project you import

Tests can live beside the code in an imported project too. Write them as exports named test… in a test/ (or tests/) directory, and tcm import --tests brings them in (importing). Helpers there are plain definitions. A *.test.ts file next to them (skipped by import) can register the same exports with another runner.

// test/math.ts
import assert from "node:assert/strict";
import { double } from "../src/math.ts";

export const testDouble = () => assert.equal(double(21), 42);

Where a test runs

A test runs in Node, unless it needs a browser: it, or something it uses or renders, touches the DOM or is a component that keeps state or handles events. Those run in Vitest's browser mode, in the Chrome you have installed (nothing is downloaded).

import { expect } from "vitest";
import { fireEvent, render, screen } from "@testing-library/react";

export const testCounter = () => {
  render(<Counter />);                             // Counter keeps state: this test runs in a browser
  fireEvent.click(screen.getByRole("button"));
  expect(screen.getByRole("button").textContent).toBe("clicked 1");
};
  • Not a browser test: code that only writes a function out as source text (${init} into a generated file), or names a DOM type only in types ((el: Element) => …, x as HTMLElement). Neither runs in the browser (placement).
  • A browser test that also needs Node (node:fs) fails saying what put it in the browser: "The test runs in a browser, since it uses lib.chart, which uses IntersectionObserver". tcm view says the same of the code.
  • Tests go through Vite, as apps do. In a browser test, server functions are reached by hash and answered by the test's server. Testing libraries like @testing-library/react are ordinary imports.

Testing app code in one process

In a Node test, everything runs locally:

  • Server functions are plain function calls. withRequest(init, fn) from the server library runs code as if answering a request (method, URL, headers, cookies, IP), so login flows test with no server.
  • Tables live in memory, one store per test run, shared by the run's tests: give each test its own rows.
  • Jobs: runJobs() runs the queued jobs that are due, inline. runJobs({ all: true }) runs all of them.

The test framework's version

Vitest ships with tcm, as does its browser side (Playwright, driving your installed Chrome), at the versions tcm was tested with. Your tests and the runner use the same Vitest. The lock records its version (tcm packages).

tcm upgrade vitest@<range> picks another version for the codebase, installed in its cache like any package. Tests written against a range tcm's version isn't in get that version the same way. Results are cached by Vitest's version, so a new one runs the tests again.

Built from c482a14e