Traces: what the app did, by definition — tcm-ts

Traces: what the app did, by definition

tcm start and tcm dev trace the app with nothing to set up: no SDK, no instrumentation, no exporter. A trace runs from a click in the browser to the SQL statement it ran. tcm traces shows them.

$ tcm traces
what the server spent its time on in the last 24h, by definition as named now (client: a call as the page waited for it)
                  kind  calls  errors    p50    p95    max  total
board.post      client    412       0   14ms   38ms  0.2s    6.8s
board.post        call    412       0  3.1ms  9.4ms   41ms   1.6s
page              page    120       0   12ms   30ms   88ms   1.5s
board.Feed      render    120       0  6.2ms   19ms   52ms  840ms
posts INSERT       sql    412       0 0.31ms  1.2ms  6.0ms  190ms
board.welcome      job    412       3  1.0ms  2.2ms  8.1ms  520ms

worse than the version before
  board.list call  p95 9.4ms → 41ms (4.4× slower)
    since "Board: newest posts first" #k4njkepv, 2026-11-04 10:02:11
    changed: board.sortPosts

A slower version names the change that did it. A version is worse than the one before when its p95 is 1.5× and 2ms more, or 5 points more of its calls fail (two at least), with five calls each. tcm names the definitions whose own code changed (board.sortPosts, not board.list) and the history entry that made the version.

tcm traces <name>
one definition, version by version (below)
--errors
errors by the definition that threw them (below)
--trace <id>
one trace as a tree (an id prefix is enough), with its statements and the jobs it enqueued
--since 30m
how far back (default 24h)
-n, --limit
how many definitions the summary lists (20)

What's recorded

call
server function call from the browser, a stream until it ends: hash, status, error (500s only)
render
server component rendered: hash, an error a <Suspense> caught
page
page rendered on the server: path; its server components inside
http
HTTP route request: hash, method, path, status
job
job run, retries included, its own trace: hash, attempt, link to the call that enqueued it
transaction
transaction: its time, waiting included
sql
statement on the app's tables and sessions: table and verb (posts INSERT), statement, rows
client
browser call to a server function or component: hash, the definition that made it, path, status
view
page load in the browser: path, time to first byte and to DOM ready
navigate
navigation without a load (the react library's, history.pushState)
fn
codebase function call on the server, with tcm dev --trace-calls (below)
error
uncaught error in the page: message, stack, where thrown

An error also names the definition that threw it (tcm.thrown_in). Each span records the build that answered. A request with a W3C traceparent joins the caller's trace, and the answer's traceresponse header names its span.

Across the browser

The page's span is the parent of everything the page does, and the server's span for each call sits inside the browser's:

$ tcm traces --trace 3f2a9c1e
trace 3f2a9c1e5b7d40a1c9e2f6b8d3a7c514  2026-11-05 10:12:01  build #mqyuftuf
    +12ms   0.42ms  page  /
  +0.00ms     88ms    view  /
   +1.20s     14ms      client board.post  from board.Composer, /
   +1.21s    3.1ms        call board.post
   +1.21s   0.31ms          sql posts INSERT  1 row(s)
   +4.02s   0.00ms      navigate  /archive
   +4.05s     21ms        client board.archive  from board.Archive, /archive
   +4.06s    9.8ms          call board.archive
  • The calling definition (from board.Composer) comes from the code, so it follows renames.
  • The browser's clock is corrected to the server's.
  • A page open long continues in a new, linked trace after half an hour or a thousand spans.

Every call: tcm dev --trace-calls

Traces each call to the codebase's functions on the server, so a trace is the whole tree of what ran:

$ tcm traces --trace eeeeeeee
  +0.00ms    4.1ms  call shop.list
  +0.10ms   0.20ms    sql items SELECT  1 row(s)
  +0.40ms    2.2ms    fn shop.names
  +0.52ms   0.31ms      sql items SELECT  2 row(s)
   +2.7ms   0.30ms    fn shop.order
   +2.8ms   0.02ms      fn shop.label
   +2.9ms   0.01ms      fn shop.label
  • Code is unchanged: functions keep their names, properties and source.
  • Only calls inside a span count, up to two thousand a trace.
  • Not traced: calls within one definition's cycle, and tcm dev's first-paint rendering.
  • For development only: calls get slower and traces big.

Where code is read

tcm view (and tcm_view) shows how each definition ran in the last day:

$ tcm view board.post
// board.post #x3hq2mwa : (text: string) => Promise<string>
// runs on the server: it uses lib.server.table
// traced: 412 call(s) in the last day, p50 3.1ms, p95 9.4ms; pages waited p95 38ms for it

An agent's tcm_traces is tcm traces (agents). The studio shows each definition's runs by version, and traces as waterfalls.

Errors, by what threw them

$ tcm traces --errors
errors in the last 24h, by the definition that threw them, as named now

board.parse  Error: no post 17
  41 time(s), 2026-11-04 10:02:11 – 11:40:52, the newest in trace 1c0a77e3
  began with #x3hq2mwa: "Board: posts by slug" #nvz45hx6, 2026-11-04 10:01:40
  through board.post (call), board.Feed (render)

board.Composer in the browser  TypeError: can't read 'title' of undefined
  3 time(s), 2026-11-04 11:02:40 – 11:31:09, the newest in trace 9be41f20
  began with #c4hok2jp: "Board: drafts" #k4njkepv, 2026-11-04 11:00:02
  • Grouped by definition (all versions) and message, ignoring what varies: no post 17 and no post 18 are one.
  • Counted once a trace; through lists the calls and renders it passed.
  • Each names the version it began with and the history entry that made it.
  • Browser errors are included. Production builds keep source maps (.tcm/apps/maps/, never served) to name the definition; chunks built without one are grouped by message only.

By name now, by version

Spans record hashes, so traces show today's names. tcm traces <name> puts its versions side by side, newest first, each with the history entry that made it, then what changed between each pair. Callers from the browser are listed with how long the page waited.

$ tcm traces board.list
board.list in the last 24h
version          calls  errors    p50    p95   max  seen                             since
x3hq2mwa (now)     212       0  4.0ms  7.7ms  21ms  2026-11-04 10:02:11 – 11:40:52  "Board: cache the feed" #nvz45hx6, …
cfi6w4xc          1840       2   38ms   91ms  0.4s  2026-11-03 09:12:40 – 10:01:58  "Board: posts by tag" #c4hok2jp, …

what changed, version by version
  x3hq2mwa ← cfi6w4xc  board.feedCache, board.list changed  p95 91ms → 7.7ms (11.8× faster)

slowest
     0.4s  2026-11-03 17:20:04  trace 9be41f20  cfi6w4xc
     ...

errors, newest first
  2026-11-03 16:02:44  trace 1c0a77e3  Error: database is locked  cfi6w4xc

Sending them elsewhere

Set OpenTelemetry's variables and tcm also sends spans over OTLP (JSON over HTTP), to Honeycomb, Grafana, Jaeger, a Collector or anything that takes OTLP:

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io \
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<key> \
tcm start app.main
OTEL_EXPORTER_OTLP_ENDPOINT
/v1/traces is appended
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
used as is
OTEL_EXPORTER_OTLP_HEADERS (or _TRACES_HEADERS)
key=value,key=value
OTEL_SERVICE_NAME
default: the app's name; service.version is the build
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
unsupported: tcm warns and keeps traces local

Browser spans go as a separate resource (tcm.side: browser). Span names are the names at the time, with tcm.hash. If sending fails, tcm says so once; traces are still kept locally.

Traces from production

Set TCM_TRACES_TOKEN where tcm start runs:

TCM_TRACES_TOKEN=$(openssl rand -hex 24) tcm start app.main

GET /__tcm/traces.sqlite with Authorization: Bearer <token> returns a copy of the traces database (gzipped if asked). No token set: 404; wrong token: 401. The studio reads production's traces this way.

Where they're kept

.tcm/data/traces.sqlite: a week, at most half a million spans. OTEL_SDK_DISABLED=true turns tracing off. tcm test records nothing.

Not yet

  • Spans are per definition, not per call (except with --trace-calls). In the browser, rendering and handlers' own work between calls isn't measured.
  • A call sits under the page view or latest navigation, not the click that made it (browsers lack AsyncContext).
  • Behind a load balancer, a page's spans that reach a server that wasn't in its trace are dropped.
  • tcm traces reads this machine's traces. For a deployed server's, use the studio (traces from production), OTLP, or tcm traces there.
  • A process that ends abruptly loses its last second of spans.

Built from c482a14e