Provider contract
Jazz gets provider behavior from the AI SDK. It has no adapter interface of its own below
LLMService, so the obligations a provider package must meet are the ones Jazz’s stream
boundary already depends on. This page states them as the code enforces them today. Each
obligation names its source and its test, and anything Jazz does not enforce is marked
Not enforced.
Source:
stream-processor.ts ·
ai-sdk-service.ts ·
reasoning-replay.ts ·
reasoning-parts.ts ·
tool-call-parts.ts ·
llm-error.ts ·
completion-validation.ts ·
answer-outcome.ts
Tests: provider-stream-contract.test.ts
runs one set of assertions over every case in
provider-stream-fixtures.ts. A
failing assertion names its case. To cover a new provider behavior, add a case to the table.
Where the boundary sits
AISDKService calls streamText and hands result.fullStream to a StreamProcessor. The
processor turns SDK parts into Jazz StreamEvents and one ChatCompletionResponse. Jazz
passes maxRetries: 0 to the SDK (AI_SDK_MAX_RETRIES), so every retry is Jazz’s own and
happens in the executors, never inside the SDK.
Stream event order
The processor emits, in this order:
stream_startfirst, always.- Between them, in arrival order:
thinking_start,thinking_chunk,thinking_complete,text_start,text_chunk,tool_call. usage_update, only when usage was found (see below).complete, last, exactly once.
What the processor enforces:
- Reasoning closes (
thinking_complete) before the first text. Areasoning-endpart, atext-delta, or the end of the stream closes it.thinking_startis also emitted from areasoning-deltathat arrives without areasoning-start. - Empty
text-deltaparts are dropped. tool-input-start,tool-input-deltaandtool-input-endproduce no events. A tool call is emitted once, from the SDK’s assembledtool-callpart.- The
finishpart ends processing. Parts after it are ignored, including a lateerror. - Provider-native tool calls (for example OpenAI web search) are held until their
tool-resultarrives so the call carries its real query, and are flushed atfinishif no result came. - Inline
<think>tags in text are parsed into reasoning only when a reasoning parser is selected for the provider and model (local providers), and only until a separate reasoning channel appears.
Not enforced: that reasoning-start precedes reasoning-delta, that text-start and
text-end exist, or that the SDK emits parts in any other particular order. The processor
reads only the part types listed above.
A stream that ends without a finish part fails the request with Stream completed without finish event. A finish whose reason is error fails it too.
Usage
Usage is read in this order, first hit wins:
totalUsageon thefinishpart, available in-stream.result.usage, awaited for at most 2 seconds (USAGE_SETTLE_TIMEOUT_MS).
When usage is found the processor emits usage_update and then complete, so a consumer sees
usage before completion. When neither source yields usage, no usage_update is emitted and
the response carries no usage. Consumers treat a missing report as zero tokens.
Normalization into TokenUsage:
| Jazz field | AI SDK field | Rule |
|---|---|---|
promptTokens |
inputTokens |
0 when absent; includes cache reads |
completionTokens |
outputTokens |
0 when absent |
totalTokens |
totalTokens |
0 when absent |
cacheReadTokens |
inputTokenDetails.cacheReadTokens |
set only when reported |
cacheWriteTokens |
inputTokenDetails.cacheWriteTokens |
set only when reported |
reasoningTokens |
reasoning-end part, else outputTokenDetails.reasoningTokens |
set only when reported |
Cache reads are part of promptTokens. Cost code relies on this: usage-cost.ts clamps
cacheReadTokens to promptTokens and prices the remainder as uncached input. A provider
package that reports inputTokens excluding cached tokens would be over-discounted. Jazz does
not detect that case.
Tool input schemas
Anthropic requires JSON Schema draft 2020-12 for tool inputs. Jazz converts legacy tuple
schemas from the SDK or MCP (items: [...], with optional additionalItems) to
prefixItems: [...] and items at the Anthropic request boundary. Nested tuples and array
bounds are preserved. Other providers retain their existing schemas, and local argument
validation continues to use the original tool schema.
Source: anthropic-tool-schema.ts.
The wire-format regression in
prompt-cache-breakpoints.test.ts
exercises memory and scratchpad line ranges through the Anthropic SDK serializer.
Tool-call arguments
The SDK assembles streamed argument deltas and parses them. Jazz receives the parsed input
on the tool-call part and serializes it with JSON.stringify into
ToolCall.function.arguments. The result is a JSON string that parses back to the same
object; key order is kept but whitespace is not.
The raw model text survives in one case: when the SDK marks the call invalid, input is the
unparsed string and Jazz keeps it verbatim, with invalidReason describing the failure so the
executor answers the model instead of running the tool.
thought_signature (Google) is the only provider metadata kept on a tool call. When a stored
call is replayed, arguments is parsed back into an object for the SDK.
Reasoning storage and replay
Stored on the response: reasoning holds the accumulated reasoning text for display.
reasoningParts holds the provider’s own reasoning parts from result.response.messages,
each tagged with the provider name and carrying providerOptions verbatim (Anthropic’s
signature, OpenAI’s item id and encrypted content). A part with neither text nor
providerOptions is dropped. The response is awaited for at most 50 ms, and a failure to read
it never fails the completion, so reasoningParts can be missing on a slow response.
Replayed on the next request by resolveReasoningReplay(provider, model):
| Provider | Scope | Text required |
|---|---|---|
anthropic, openai, chatgpt |
current-turn |
no |
moonshotai |
current-turn |
yes |
deepseek, V4 and later ids |
all-turns |
yes |
deepseek, legacy ids |
current-turn |
yes |
| every other provider | none |
no |
current-turn means assistant messages after the last non-ephemeral user message, which is
the in-flight tool loop. A stored part is replayed only when its provider equals the current
provider, so switching providers mid-conversation drops the old provider’s reasoning rather
than sending it to a provider that cannot read it. A malformed stored part is skipped and
logged with a reason, never sent.
Error paths
A request fails through exactly one of two paths, and both end in convertToLLMError:
- Thrown before the stream exists. Setup (model resolution, message conversion, attachment loading) throws, and the service converts it.
- An
errorpart in the stream. The processor throwspart.error. The service catches it, converts it, fails the Effect stream with theLLMError, and rejects the response.
The processor never emits a StreamEvent of type error itself, and it never completes after
failing. process() either returns a response or throws.
Not enforced: a provider that signals failure some third way (for example a finish with
a non-error reason and an empty body) is not an error. It reaches the empty-finish check below.
Classification
convertToLLMError maps the thrown value to one of three tags. The checks run in this order:
| Condition | Result | Retryable |
|---|---|---|
APICallError with a 2xx status (unreadable body) |
LLMRequestError, unreadableResponse |
yes |
Name in the permanent set (AI_UnsupportedFunctionalityError, AI_InvalidArgumentError, AI_InvalidPromptError, AI_MessageConversionError, AI_TypeValidationError, AI_NoSuchModelError, AI_NoSuchToolError, and others) anywhere in the cause chain |
LLMRequestError, permanent |
no |
StreamIdleTimeoutError |
LLMRequestError, streamStalled |
yes |
| 401, or a credential failure | LLMAuthenticationError |
no |
| 403 about billing or plan | LLMRequestError, permanent |
no |
| 429 | LLMRateLimitError |
yes, unless out of credits (permanent) |
| Other 4xx | LLMRequestError with statusCode |
no |
| 5xx | LLMRequestError with statusCode |
yes |
No status (connection, DNS, stream cut off, missing finish) |
LLMRequestError, no statusCode |
yes |
isRetryableLLMError is the single predicate: rate limits that are not permanent, and
LLMRequestErrors that are not permanent and have no status or a 5xx. Authentication errors
never retry. A 4xx with contextOverflow is not retried either; the agent loop reacts to it by
compacting.
Retry loop
The streaming and batch executors share one retry budget and schedule: exponential backoff
from 1 second, jittered, capped at MAX_RETRY_DELAY_SECONDS, replaced by Retry-After when
the provider sends one, and abandoned when that wait exceeds MAX_RETRY_AFTER_MS. A streaming
step has no wall-clock limit: the stream idle timeout ends an attempt that goes silent, so long
reasoning is never cut off. Each non-streaming request, the streaming executor’s fallback
included, is bounded by LLM_TIMEOUT_SECONDS, and the batch executor bounds its whole step,
retries included, by it.
In the streaming executor, after 3 transient streaming failures
(STREAMING_FAILURES_BEFORE_FALLBACK) the remaining attempts use the non-streaming request.
A 429 does not count toward the 3. A stalled stream or an unreadable stream goes to the
fallback at once.
The stream idle watchdog fails a stream that stays silent: 120 seconds between parts by
default (llm.streamIdleTimeoutMs or JAZZ_STREAM_IDLE_TIMEOUT_MS override), and at least
300 seconds before the first content part (FIRST_CONTENT_TIMEOUT_MS).
Empty finish
A completion is an empty finish when it has no visible text, no reasoning text, no tool
calls, no artifacts, no reasoning tokens and zero completion tokens (a missing usage report
counts as zero), and its finish reason is neither content-filter nor length.
Both executors pass every completion through validateCompletion. An empty finish becomes
an LLMRequestError with emptyResponse: true and no status code, so it takes the transient
retry path under the same budget. The rejected response’s request count and usage are
recorded before it is discarded, because it never reaches the agent loop.
After the budget is spent, or when a run ends with an empty answer, judgeAnswer reports the
run as failed with empty_response. A content-filter finish is content_filtered and is
never retried, since the same prompt is withheld the same way.
Options a provider can reject
Jazz decides most option support before the request, from the model’s resolved capabilities, and adjusts the request instead of failing it:
- Tools are dropped when the model does not take tools (
toolsDisabledis set on the response and an info log is written). temperatureis dropped when the model reportssupportsTemperature: false.- A reasoning effort the model does not list is clamped to the nearest supported one, with a one-time warning per provider, model and effort.
These three are silent to the model and the user, and are the opposite of “throw on unsupported options”. Not enforced: failing fast on an option Jazz cannot honor.
When a provider package itself rejects a request with AI_UnsupportedFunctionalityError or
another name in the permanent set, the error is permanent, is not retried, and is reported
as a rejected request. Any other rejection arrives as a status code and follows the table
above.
Adding a provider
Before a provider is wired in, confirm against this page:
- Its usage reports
inputTokensincluding cached input tokens. - Its reasoning parts survive the round trip: add the provider to
resolveReasoningReplayif it needs them back, and add a case to the fixtures table that proves they are stored with the provider’s name. - Its failures arrive as thrown errors or
errorparts, with a status code where the remote gave one.
See Add a model provider for the wiring steps.
Blank or duplicate tool identities are rejected before appending history or executing tools. The shared request retry budget can ask the provider for a valid completion; rejected usage remains failed-attempt spend. Identity repair never receives an extra retry budget.