$ TEBAKO_TRACE=/tmp/capture.jsonl metanorma compile site.adoc
GUIDES · 14
Trace a packaged app.
The trace toolkit shows what a packaged application really touches, from three vantage points. Inside the process, an interception bus in the virtual-filesystem layer emits one structured JSON event per decision — every mount, open, dlopen, exec, materialize, and jail verdict — identically on linux, macOS, and windows. At the libc boundary, retrace watches the same process from outside. At the kernel, ptrace, eBPF, or ETW/procmon see what no userland hook can. The tebako trace commands work across all three layers: trace run serves discovery, trace explain serves diagnosis, and trace cover measures coverage.
|
Note
|
This guide is part 1 of the Advanced payloads tutorial. The blog chapter traces a real packaging failure. |
The three questions.
-
Discovery states what an application touches and therefore what its manifest must declare.
tebako trace runcollects a run’s events and drafts a suggested manifest. -
Diagnosis answers which hop failed when a packaged application fails.
tebako trace explainreplays a capture into the hop chain — mount → manifest read → resolve → materialize → OS bind — and names the first red hop. -
Coverage answers whether anything the application does remains not virtualized.
tebako trace coversubtracts the inside stream from an outside capture and reports the escapes.
The interception bus.
Every interception point the driver and the preload shim already implement emits one event per decision onto the bus — a stream of JavaScript Object Notation (JSON) lines, one object per line, robust to a crashed tail. Events name OS-level facts (paths, verdicts, errnos), never language-runtime facts, so every runtime is covered equally. One environment variable arms it:
-
The driver opens the channel before any mount; the preload shim arms the same contract in its constructor, so spawned children re-arm from the inherited environment and append to the same capture. Every event carries
pid/tidso the front-ends can regroup interleaved writers. -
Observability never gates. A trace-channel failure degrades to a loud stderr note; the payload run proceeds and its exit code is unaffected. Disarmed, an emission point is one branch — the bus costs nothing when off.
-
TEBAKO_DEBUG_TFSremains as the human-readable degraded view over the same emission points; the bus is the structured one.
Discovery: tebako trace run.
The verb runs a package under the record policy with the bus armed; when the package exits, the capture is synthesized into a suggested manifest fragment — commented YAML on stdout (or --out), never applied. The process exits with the payload’s own exit code.
$ tebako trace run ./metanorma-1.16.9-arm64-macos --capture compile.jsonl --out draft.yaml -- compile site.adoc
tebako: trace: draft written to draft.yaml
tebako: trace: the capture is compile.jsonl
The draft it writes looks like this:
# Suggested manifest additions for ./metanorma-1.16.9-arm64-macos — `tebako trace run` (spec 25 §4, discovery).
# 214 interception event(s) observed; the capture is compile.jsonl
# REVIEW BEFORE MERGING: a generated suggestion never edits a manifest by itself
# (spec 25 law 7). Flip ro/rw, delete noise, fill every `why`.
# …
needs:
host:
- path: "$HOME/.fontist"
access: ro
why: "TODO — observed: 3 read, 0 write"
# Files the run consumed through a RAW host fd (a native library's own
# stdio/loader read below the interposition — the exec-cache answer):
materialize:
- path: "/tfs/lib/ruby/gems/3.3.0/gems/sassc-2.4.0/ext/Makefile"
why: "TODO — read through a materialized host copy (observed: 2 events, first at 2026-08-22T09:14:07.113420Z)"
# NOTE: closure-covered — nothing to declare: "/tfs/lib/ruby/gems/3.3.0/extensions/arm64-darwin/3.3.0/psych-5.1.2/psych.bundle" (3 deps, all in-image; observed: 1 events, first at 2026-08-22T09:14:02.881104Z)
# NOTE: host executable observed: "/usr/bin/git" — an entrypoint/runtime-dep candidate (observed: 1 events, first at 2026-08-22T09:14:11.502377Z)
The draft extends the jail journal’s needs discovery to the axes a journal cannot see: host reads and writes become needs: grants (strongest-observed-op wins); an in-image file read through a raw host fd becomes a materialize: candidate; an in-image dlopen whose dependency closure resolved entirely in-image earns a closure-covered — nothing to declare note; a spawned host executable earns an entrypoint/runtime-dep note. Floor, store, and exec-cache paths are excluded. Then the human gate: the author reviews the draft, flips ro↔rw where production differs from the observation, deletes noise, and fills every why — a generated suggestion never edits a manifest by itself. Output representative; paths abbreviated.
Diagnosis: tebako trace explain.
The verb replays a capture — a finished file or a live run’s growing one — into the hop chain. The first hop whose verdict is red prints with its evidence; the signature table that maps stream shapes to hops lives in data, seeded from the incident corpus (a lost handoff env, the OS loader refusing a resolved closure, a jail denial, an exec-cache write failure) and extended as incidents teach new shapes.
$ tebako trace explain compile.jsonl
tebako trace explain: compile.jsonl — 87 event(s) replayed (hop chain: mount → manifest read → resolve → materialize → OS bind)
RED hop: mount — env image never mounted (handoff env lost) [signature: env-image-never-mounted]
evidence: no `mount/ok` verdict reached the stream in 87 event(s) — corroborate with the child's prelude-class stderr (spec 25 §5)
note: A mount event with an error verdict does NOT suppress this signature — a failed mount IS the never-mounted case.
$ echo $?
1
A clean capture replays like this:
$ tebako trace explain compile.jsonl
tebako trace explain: compile.jsonl — 214 event(s) replayed (hop chain: mount → manifest read → resolve → materialize → OS bind)
GREEN: no red hop — every hop's verdict is clean in 214 event(s)
Output representative.
Coverage: tebako trace cover.
The inside stream can never prove its own completeness — an escape is invisible from inside by definition. The question of what tebako is not intercepting is answerable only by subtracting the inside stream from an outside capture of the same run. Capture the run once per side, then correlate:
$ tebako trace run ./myapp --capture inside.jsonl --out draft.yaml -- serve ./public
$ # …the same run again under the outside capture of your layer (retrace preload shown)…
$ tebako trace cover --inside inside.jsonl --outside retrace.json --prefix /mnt/tfs
escape /mnt/tfs/secret/keys.pem func=open tid=603 pid=601 class=read
escape /mnt/tfs/tmp/.lock func=openat tid=604 pid=601 class=read
$ echo $?
1
--prefix names the virtualized root as the outside capture spells it; an under-prefix touch the inside stream never saw is an escape. stdout is a machine contract — one escape <path> func=… tid=… pid=… class=… line per escape, or a JSON array with --json — byte-identical with retrace’s retrace-correlate on the shared golden fixtures (the parity is asserted in continuous integration (CI); the format is retrace’s, tebako never authors a competing one). The summary, the per-surface-class coverage percentages, and the producing layer go to stderr:
tebako: trace cover: inside=214 entries, 96 paths; prefix=/mnt/tfs; escapes=2
tebako: trace cover: outside capture layer: libc boundary (retrace preload / inline hooks) — certifies libc-routed escapes only; sub-libc escapes (raw syscall, loader-internal probes) are UNCERTIFIABLE at this layer (spec 25 §6.1; named by --layer, default libc)
tebako: trace cover: coverage by surface class (under-prefix outside touches):
tebako: trace cover: fs: 41/43 covered (95.3%), 2 escapes
The remaining flags scope the correlation: --pid N considers only one process’s outside entries, --window SECS requires timestamps within the window for coverage (0, the default, is pure set-difference), and --exclude-probes drops probe-class escapes (an existence probe on an under-prefix path counts as an escape by default — a name-not-found answer is still information the VFS never served). The escape lines above are the toolkit’s golden fixture, verbatim.
The windows path: import a procmon capture.
On windows the outside capture that works today is Procmon’s comma-separated values (CSV) export — the kernel layer, normalized into the retrace JSON shape by an offline converter in the CLI (in-process, byte-parity with upstream’s procmon2retrace):
PS> tebako trace import procmon .\sassc-run.csv > outside.json
tebako: trace import: entries=1832 bad-rows=0
PS> tebako trace cover --inside inside.jsonl --outside outside.json --prefix C:/pkg/scss --pid 9012 --layer kernel
escape C:/pkg/scss/_a.scss func=QueryOpen tid=0 pid=9012 class=probe
escape C:/pkg/scss/_hidden.scss func=CreateFile tid=0 pid=9012 class=read
That verdict is the golden libsass-importer fixture, reproduced from the CSV end: a native importer probing partials with raw Win32 calls, below every libc hook — exactly the escape class only a kernel-layer capture can certify. One honest limitation: the libc-boundary alternative on windows (retrace’s inline-hook backends with live injection) is a documented absent prerequisite at retrace v2.14.0 — the win-run injector crashes inside the child’s engine boot — so the procmon chain carries windows coverage certification until that lands upstream. A probe leg in tebako’s CI pins the broken signature and fails on drift in either direction.
When to use which layer.
| Layer | Producer | What it can see |
|---|---|---|
Inside the VFS |
the tfs interception bus |
every decision tebako itself made — mounts, dispatches, dlopen closures, jail verdicts. The layer is always available, on every platform, with no setup beyond one environment variable. |
The libc boundary |
retrace (preload on POSIX; inline hooks on windows) |
The layer sees libc-routed escapes: a payload calling open(2) or CreateFileW directly, below tebako’s interposition but through libc. |
The kernel |
retrace ptrace, an eBPF bridge, ETW / procmon (via the converter) |
The layer sees sub-libc escapes: raw syscalls and loader-internal probes that no libc hook can observe. |
A coverage report is only as strong as its outside capture’s layer: a libc-boundary capture certifies libc-routed escapes only, and certifying sub-libc escapes takes a kernel-layer capture. The report names the producing layer — pass --layer kernel when the outside capture is ptrace, eBPF, or procmon.
Exit codes.
| Verb | 0 | 1 | 2 |
|---|---|---|---|
|
the payload’s own exit code — the run’s verdict is the payload’s |
— |
— |
|
no red hop appears; every hop’s verdict is clean |
a red hop is named |
a usage or I/O error occurred |
|
no escapes are found |
escapes are found |
a usage or I/O error occurred |
|
entries are emitted |
the conversion produced zero entries |
a usage or I/O error occurred |
The 0/1 split for explain and cover exists so the verdicts are CI-gateable.
See also: Check a payload · Set a jail policy · Debug & logging