Developing tcm-ts — tcm-ts

Developing tcm-ts

tcm is developed in tcm. Its code, docs and the repository's own files are all in its codebase, tcmts; the git repository only says how to get it. To propose a change from outside, see contributing.

Working on tcm in tcm

pnpm tcm <command> runs tcm on tcm's codebase, kept in codebase/ (not in git). You can run it from anywhere: paths you give it are relative to where you are.

pnpm tcm patch tcmts.core.semver.compatRange --old "…" --new "…"   # a small change, checked and saved
pnpm tcm edit tcmts.core.semver.compatRange                         # a bigger one, into codebase/scratch.ts
pnpm tcm update                                                     # saved; its dependents re-hashed
pnpm tcm mv tcmts.core.semver.meetsComparator tcmts.core.semver.matchesComparator
pnpm test                                                           # the tests the change reaches

The first run gets the codebase, with its history and releases, from tcm's server. The server is public, so this needs no token. If there's no known-good tcm here yet, it also installs the tcm the server runs. If TCM_TOKEN is set, origin gets that token.

Agents work the same way. The repository's .mcp.json runs tcm mcp --session, so an MCP client opened in the repository gets tcm's tools. Each agent works in a session, a branch of its own, landed on main with tcm_merge when you say so.

The repository's files (README, AGENTS.md, CONTRIBUTING.md, package.json, scripts/codebase.sh, …) are files of the codebase too. Change them in the codebase, then run pnpm repository to write them out, and commit them.

The known-good tcm

pnpm tcm, and so the MCP server, doesn't run the code you're changing. It runs the build the last passing pnpm verify on main tested, in dist/known-good, so a change that breaks tcm can't break the tcm making it. A running tcm mcp picks up a new one by itself (agents).

To run the code being changed:

pnpm test
the codebase's test definitions, and its end-to-end tests against the codebase's own tcm
TCM_LIVE=1 pnpm tcm <command>
the codebase's own tcm as it is now, on main (or on the branch TCM_BRANCH names)
TCM_LIVE=1 pnpm tcm -C /tmp/repro <command>
the same, on another codebase (one made for a repro); -C comes first

TCM_BRANCH=<branch> pnpm tcm … works on a session's branch. Verify reads main unless told otherwise, so you can keep working on a branch while it runs.

Changing the hash format or the store's layout (IR_VERSION in tcmts.core.hash, tcmts.store.SCHEMA): old and new tcm can't open each other's codebases. So:

  1. Bump IR_VERSION (or change SCHEMA).
  2. Add the step from the old format (tcmts.core.migrate.IR_STEPS, or tcmts.store.LAYOUT_STEPS for the layout).
  3. Re-pin the golden hashes (hash format).
  4. Add a codebase in the old format as a fixture (like tcmts.fixtures.codebase-v1), made with the known-good tcm before the bump.

When pnpm verify passes on main, the new tcm migrates tcm's codebase (tcm migrate, keeping the old database beside it) and only then becomes known-good; if migrating fails, neither changes. Until then, TCM_LIVE=1 can't open tcm's codebase: try the new tcm on a test codebase.

Older builds

pnpm verify keeps the builds it replaces, so you can show a bug was already in an older tcm, or that a new test fails without the fix (testing). The one the known-good replaced is dist/known-good-previous. Older ones go to dist/builds/<id>, named by their history entry; the newest four are kept.

pnpm tcm builds
each build: name, history entry, when verified, what changed, its tcm.js
pnpm tcm --build <name> <command>
a command run by that build: known-good, previous, an id (or prefix) from tcm builds, or a tcm.js path. -C <dir> after it runs on another codebase
pnpm tcm test <e2e tests> --bin tcm=<name>
the end-to-end tests against that build
pnpm tcm builds make <entry>
a build of tcm at any history entry (a number back, or a hash), in about fifteen seconds

A build from before a hash format change can't open a codebase in the new format, tcm's own included. Run it on a codebase it makes: --build <it> -C /tmp/repro init.

The codebase

tcmts.core      extraction, canonicalization (the IR), hashing, runtime schemas from types, compiler options
tcmts.store     the SQLite codebase store
tcmts.loader    rendering stored definitions to .tcm/cache modules, running them, the #codebase shim
tcmts.cli       the tcm command, placement inference, sync, merge, web (Vite plugin, dev/build/start), the
                #tcm/host runtime, the wire codec; tests in tcmts.cli.test (end-to-end: tcmts.cli.test.e2e)
tcmts.share     tcm serve's web UI (tcmts.share.App), shipped in the package as a codebase of its own
tcmts.studio    tcm studio's web IDE
tcmts.registry  the registry `tcm registry` runs (tcmts.registry.App), shipped as a codebase of its own
tcmts.atproto   the AT Protocol client tcm and its registry use (identity, loopback OAuth, signed records)
tcmts.react     the tcm library `react` (`tcm lib add tcm react`): routes, links, data, Pages, Stylesheet
tcmts.records   the tcm library `records`: tcm's AT Protocol records, their lexicons, keys and readers
tcmts.server    the tcm library `server`: tables, transactions, jobs, sessions, cookies, the request
tcmts.vscode    the VS Code extension (a thin client for `tcm lsp`), its own package (tcm-ts-vscode)
tcmts.examples  the example projects (polls, notes, userinput, reminders, geometry, effect), kept as written
tcmts.docs      user documentation, one page per feature area
tcmts.site      the docs site: its pages, how they render, and its app (tcmts.site.Docs)
tcmts.design    design notes
tcmts.dev       verifying, building the docs site, deploying the server and the registry, comparing compilers
tcmts.fixtures  what tests take as files: the examples, a docs page, the editor plugin, old-format codebases
tcmts.manifest  the tcm-ts package's manifest
tcmts.deploy    tcm's server and registry, as deployed to Railway

tcm at <file>:<line> names the definition at a line of a stack trace; tcm find and tcm view do the rest.

Rules that keep changes small and safe:

  • Code Codebase calls takes CodebaseCore, an interface the class has, not the class (tcmts.cli.codebase.Codebase). Naming the class in their types makes them all one cycle, re-hashed together. Add what you need to CodebaseCore.
  • Read a definition's IR through tcmts.core.ir (defParts, declarationOf, initializerOf, childOf), never by position. The IR is ["Def", tag, ...], then a lineage salt (["Salt", path]) only if it has one, then its JSX runtime (["Jsx", runtime]) only if it has JSX, then its syntax. So ir[3] is the declaration in some definitions and not others.
  • The server runtime stays self-contained. tcmts.cli.server_api.createServerHost (with createTracer) is written to .tcm/cache/tcm-host.js, so it uses only what comes in through deps. It knows an HttpError by its mark (Symbol.for("tcm.httpError")), so any copy's counts. HOST_PROTOCOLS are the runtime versions this tcm serves (host in a manifest). Code importing the old #tcm/server module is refused with TCM_SERVER_GONE.
  • Roles (tcmts.cli.roles.ROLES: stylesheet, table, every) are found by hash, so a library's table or schedule is known wherever it's used from.
  • TypeScript is TypeScript 7, through tcmts.core.ts. Checker answers cross a process boundary: ask for symbols in batches (prefetched in tcmts.core.analyze).

A command

Each command is a few definitions, so a change to one reaches only the suites that ran it. For tcm find:

tcmts.cli.commands.routeFind
its arguments and options, as a @bomb.sh/router route tree. No descriptions
tcmts.cli.commands.helpFind
its description and its parameters' (commonHelp has shared ones), read by tcm help, --help, usage errors and shell completion
tcmts.cli.run.runFind
its handler, given a Context (its arguments, the codebase opened)

tcmts.cli.main.dispatch does what every command needs (-C, --build, the session, --branch, opening the codebase) and calls the handler. routes(), helps() and handlers() list them by name; they're functions, not constant tables, so a suite doesn't count every command. A new command needs all three (handlers()'s type and testHelpCoversEveryCommand catch a missing one) and a place in SECTIONS, the order tcm help lists them in. Terminal prompts use @clack/prompts (tcmts.cli.ui).

Tests

tcm's tests are test definitions in its own codebase.

  • Tests that call code directly go in a test namespace beside the code. They're cached by hash: a change re-runs only the tests that reach it.
  • End-to-end tests, which run tcm as a program, go in tcmts.cli.test.e2e (helpers in tcmts.cli.test.e2e.harness). They're the tcm-ts package's suites, each run as its own program and cached by the code it ran, so keep each independent. Files they need (an example project, a docs page) come from tcmts.fixtures.
pnpm test
tcm test: the tests a change reaches, the rest cached
pnpm verify
everything a change needs (below)
pnpm examples:check [name…] [--tests]
the examples' files typechecked in seconds (--tests: their tests too). By default the apps and static sites; geometry and Effect only when named. Check Effect after a change to importing or hashing (its first check takes minutes)
pnpm tcm test tcmts.cli.test.e2e.templates
tcm init's templates (tcmts.cli.templates): each made, its tests run, and built, in about 20 seconds
pnpm docs:dev
the docs site as a tcm app (tcm dev tcmts.site.Docs), reloading when a page is updated
pnpm docs:site
the static docs site, with its code browser and examples, into dist/site. Fails on a broken link or anchor, or a page no link reaches
pnpm compare-compilers
checks every definition hashes the same under tsrs as under TypeScript 7, golden tests included
pnpm tcm run tcmts.dev.forgeTimings <codebase dir>
how long the web UI's pages take on a codebase (a cp -c copy of tcm's, say)
pnpm tcm run tcmts.dev.atprotoTrial [tcm.js] [dir]
before a release: tcm's AT Protocol flows (announcing, libraries, attestations, the registry, publishing lexicons) against Bluesky's dev-env, offline. Installs dev-env in dir (default: a temp directory); logs in dir/*.log

The examples must keep working: the end-to-end tests (tcmts.cli.test.e2e.examples) build each from scratch, all but Effect, which only pnpm examples:check effect checks. Tests that render in a browser need Chrome (they're skipped without it).

Verifying a change

Run pnpm verify when a change is done. On a session's branch, run TCM_BRANCH=<branch> pnpm verify: it checks everything there but doesn't make a known-good tcm, which is only ever main's.

It runs four stages side by side, printing each result as it finishes, each test failure as it happens, and progress every 30 seconds:

  1. codebase: nothing needs fixing (tcm todo), every definition re-hashes to itself (tcm verify), and tcm's libraries hold everything they use;
  2. tests: the test definitions (tcm test tcmts --definitions);
  3. docs: the docs site builds, every link and anchor resolving;
  4. package: the tcm-ts package packed from the codebase installs, works far from the repository, and passes the end-to-end tests run against it (the tcm that becomes known-good).

What passed is kept, so each stage redoes only what a change reaches: nothing changed finishes at once, a page or a command takes seconds, and a change every end-to-end test runs (the store, the Codebase class) about two and a half minutes. pnpm verify --fresh checks everything again.

When everything passes on main, the package becomes the known-good tcm.

Side by side: what's in dist/

Each branch's verify builds in a place of its own, so worktrees sharing dist/ verify different branches at once. A second verify of the same branch waits for the first. Making a tcm the known-good one is main's alone, one at a time. Tests share the machine's suite budget.

known-good/
the tcm pnpm tcm and the MCP server run: main's last verified (VERIFIED, verified.json), with its tarball
known-good-previous/, builds/<id>/
the builds it replaced (older builds)
candidates/<branch>/
a verify's own package/, install/ (the tcm it tests) and verify.lock. Removed once it passes; a failed one stays a day, so you can rerun its install/…/tcm.js
verify.lock
held by main's verify while it makes its tcm the known-good one
examples/
the examples' codebases and downloads for the docs site, kept a day after last use
package/, site/
pnpm build's package and pnpm docs:site's site

Packaging

tcm-ts, the npm package, is a package of tcm's codebase, with its manifest in tcmts.manifest (the tcm and tcm-ts bins run tcmts.cli.main.runCli; tcm-ts/core exports the analyzer). pnpm build packs it into dist/package, dependencies pinned at the codebase's locked versions. At a release it's that release; otherwise it's 0.0.0.

Releasing is tcm release:

pnpm tcm release check tcm-ts      # the next version, from the semver checks on tcm-ts's API
pnpm tcm release tcm-ts --draft    # codebase/release-tcm-ts.md: write a summary above the marker
pnpm tcm release tcm-ts            # recorded in the codebase (and pushed with it)
pnpm tcm changelog tcm-ts --out CHANGELOG.md
pnpm build && (cd dist/package && npm publish)

Publishing is done by hand.

tcm's server

tcm's codebase is served by tcm serve at https://tcm.codes (also https://tcm-server-production.up.railway.app): the tcm-server service of the tcm-ts Railway project. It's the codebase's only copy off your machine; pnpm tcm push keeps it current.

/
the home page, and /tcm-ts.tgz, the tcm it runs (which a new checkout installs)
/docs/
the docs site; its code browser (/docs/code/, with a forge per example at /docs/code/<name>/); the examples' downloads and running apps (/docs/examples/<name>/)
/forge/
the codebase's web UI
/registry/
tcm's registry (below)
POSTs to /<route>
the codebase, for tcm pull, push and contribution

Old root addresses redirect (301) to /docs (--moved /docs). A proxied app's path mustn't be forge, registry, docs/code/<name> or docs/examples/<name>: those are matched first.

  • What's deployed is tcmts.deploy.server (a Dockerfile installing the known-good tcm, railway.json, the home page, start.sh). start.sh makes the codebase on the volume (/data) the first time, runs tcm migrate --if-needed, builds the examples in the background, runs the example apps under /docs/examples/<name> (notes and polls reset every six hours, TCM_EXAMPLES_RESET), and serves everything with --public --no-run --insecure (Railway terminates TLS). It needs TCM_TOKEN, the maintainer's token.
  • pnpm server:deploy builds the docs site for /docs and uploads it with the server's files and the known-good tcm's package, from a checkout linked to the project (railway link). The docs served are as of the last deploy.
  • Tokens: anyone can pull. The maintainer's token is the service's TCM_TOKEN; contributors get their own from pnpm tcm user add <handle> (contributors), which push only their own branches. An existing codebase/ connects with pnpm tcm remote add origin https://tcm.codes --token <token>.

tcm's registry

tcm's registry, at https://tcm.codes/registry, is the one tcm lib search asks by default. It's its own Railway service, tcm-hub (tcmts.deploy.registry), so deploying the server doesn't restart it. The server proxies /registry to it over Railway's private network (TCM_REGISTRY_PROXY, default http://tcm-hub.railway.internal:8080; empty for none).

  • start.sh runs tcm registry --host :: --port $PORT --base /registry --dir /data/hub, and refuses to start with TCM_ATPROTO_LOCAL set. TCM_REGISTRY_BASE must match the path the server proxies; TCM_REGISTRY_DIR moves its directory.
  • pnpm registry:deploy uploads it with the known-good tcm's package. A newer tcm keeps its tables, jobs, Jetstream position and mirrored codebase.

Setting up tcm.codes

Done once, by hand:

  1. An empty tcm-hub service in the tcm-ts project, no public domain, a volume at /data, PORT=8080, and JETSTREAM_API_KEY if there is one. Never TCM_ATPROTO_LOCAL.
  2. env -u TCM_BRANCH pnpm registry:deploy, then env -u TCM_BRANCH pnpm server:deploy.
  3. The domain, in the dashboard (railway domain answers "Unauthorized"): on tcm-server, Custom Domain tcm.codes on port 8080. At Marque, where the DNS is, add a CNAME named @ to Railway's target and the _railway-verify TXT record Railway shows (required, as Marque flattens the CNAME). Leave the _atproto and _lexicon records alone.
  4. Check /, /docs/, /forge/ and /registry/ load, railway logs --service tcm-hub shows job ingest done every minute, and tcm lib search <query> ends from https://tcm.codes/registry. Then pnpm repository, and commit.

Publishing tcm's lexicons

For maintainers. The Lexicon schemas of tcm's AT Protocol records (codes.tcm.release, .review, .advisory, .yank, .star, .codebase; see releases, libraries, listed codebases) are in tcmts.records.lexicons. Each is published as a com.atproto.lexicon.schema record in the repository of the DID _lexicon.tcm.codes names (Lexicon spec).

pnpm tcm run tcmts.dev.publishLexicons -- --dry-run      # what it would write, signing in to nothing
pnpm tcm run tcmts.dev.publishLexicons [-- --as tcm.codes]

It signs in only as that DID, asking only for repo:com.atproto.lexicon.schema, and refuses as anyone else. It validates each schema, lists each as new, changed or unchanged, and writes only new and changed ones. After changing a lexicon, the tests in tcmts.dev.lexicons.test check tcm's records against it; publish once the change is on main.

Built from c482a14e