Golden sessions
A golden session replays a recorded model conversation through the real agent runner, tool registry, approval gate, checkpointing and conversation persistence. Only the model is faked. It exists because a green unit suite has missed bugs that a run through the loop found.
Source: packages/adapters/src/llm/golden/ (fixtures, recorder, scrubber, scenario tests) and packages/adapters/src/test-support/golden/ (replay, harness, transcript helpers). Run with
bun test packages/adapters/src/llm/golden. No keys, network or daemon are involved.
How it works
- Fixture: one entry per model call, holding the request summary (message roles, tool names
offered, tool results fed back) and the AI SDK
fullStreamparts the model produced. - Replay (
replay.ts): serves entries in order. The parts run through the realStreamProcessor, so stream events, usage and tool-call parsing are shipping code. A call beyond the fixture, a different set of tools offered, or a different role sequence fails with a diff of expected against actual. A run that makes fewer calls than recorded failsassertExhausted(). - Harness (
harness.ts):createGoldenSessionrunsAgentRunner.runwith scripted tools in the real registry,onCheckpointpersisting to a temp directory, and an in-memory run store for parking. It exposesevents,checkpoints,loadSaved()andresumeParked(). - Scrubbing (
scrub.ts): secrets, credential-shaped tokens, home directory, username, hostname, temp directory, timestamps and random ids become stable placeholders.
Recording a session
Authors record; CI only replays.
JAZZ_GOLDEN_RECORD=packages/adapters/src/llm/golden/fixtures/my-scenario.json \
jazz run "the prompt"
The AI SDK service scrubs each streamed call as it arrives and rewrites the file after every call. Secret-named environment values and the configured provider key are scrubbed by value. Read the file before committing it: scrubbing is a safety net, and a recording made against a real conversation can still hold personal text. Replace such text with neutral wording, then keep the part shapes as recorded. Never commit a recording of a real conversation.
Adding a scenario
- Record or hand-write
fixtures/<name>.json. List every tool the run offers, including the built-in skill tools, in the request summary of each call. - Add a test in
scenarios.test.tsthat opens a session with the fixture and the scripted tools the scenario needs. - Run it. A drift failure names the missing or extra tool or role; fix the fixture only when the change is intended.
What to assert
Assert behavior on the transcript and events using transcript.ts:
- the ordered roles of the transcript
- tool call names and argument keys, not argument values
- each tool result answers a call, and no call is left unanswered
- parked, completed and interrupted states, and which checkpoint phases were taken
- a final assistant answer exists, and usage was reported
Do not snapshot text. Do not regenerate a fixture to make a failing test pass: a refreshed expectation without a semantic assertion blesses the regression. For wording that a producer owns (such as the interrupted-result text), derive it from the producing function instead of copying the string.
Scenarios
| Scenario | Proves |
|---|---|
| plain-answer | One call, usage, a persisted transcript, no tool traffic |
| tool-call-result | A tool runs once and its result reaches the next request |
| approval-park-resume | A mutating call parks unattended, runs once on approve, then answers |
| empty-finish-retry | An empty model finish is retried and then answered |
| interrupted-batch | A kill mid-batch leaves a saved checkpoint with the call closed as interrupted |