feature/anthropic messages api - #442
Open
klement wants to merge 8 commits into
Open
Conversation
Serve clients built against Anthropic's Messages API, enabled by adding "Anthropic" to network.api_servers. Two endpoints: /v1/messages and /v1/messages/count_tokens. Requests are translated into the internal chat completion request rather than served by a parallel inference path, so the Anthropic API inherits chat templating, reasoning tag parsing, tool call formats, samplers, multimodal plumbing and context length handling from the existing pipeline, and does not drift from it as that pipeline changes. Request content blocks are modelled as a discriminated union with a permissive fallback so an unsupported block reports its own type instead of a pydantic validation dump. Blocks are joined on a blank line: Anthropic clients routinely split one turn across several text blocks (a system prompt and an environment block, say), which are separate pieces of context rather than a continuing sentence. Thinking blocks replayed by the client become reasoning_content; redacted_thinking is dropped, since it is encrypted by the Anthropic API and carries nothing a local model could use. Sampler fields are forwarded only when the request set them, so model defaults and configured overrides still apply to everything else. thinking.type maps onto the enable_thinking template variable; a budget_tokens has no local equivalent and is ignored, as are cache_control, service_tier, container and mcp_servers. stop_sequence is reported only for a sequence the client actually sent. Prompt templates contribute their own stop strings, and surfacing one of those would name a string the client never asked for, so a template stop is reported as an ordinary end_turn. Errors are returned in the Anthropic envelope. The reshaping lives in a route class rather than an app-level exception handler so it also covers failures raised while solving dependencies, which never reach the endpoint: an invalid API key would otherwise return TabbyAPI's envelope to a client that cannot parse it. Anthropic clients authenticate with x-api-key, which check_api_key already accepts, so pointing an SDK at the server is a base URL change. Support is partial. Streaming, tool use, and images are rejected with a descriptive error rather than silently dropped: quietly discarding a tool_result would delete the tool output from the conversation and produce a confidently wrong answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anthropic's stream is a block-structured event sequence rather than the flat chunk stream the OAI API emits: each event carries an SSE event name, and content arrives inside explicitly opened and closed blocks, ending on message_stop rather than a [DONE] sentinel. The generation pipeline emits reasoning and content as separate deltas on the same chunk, so translating means tracking which block is open and closing it when the channel changes. Blocks are opened lazily and indexed in the order they open, so a response without reasoning has its text at index 0 rather than leaving a hole where a thinking block would have been. Thinking blocks carry an empty signature, matching the non-streaming path: there is nothing to authenticate locally, but SDK response models require the field. The stop reason is now derived by one function taking the raw finish fields, shared by both paths. A stop reason that differs between streaming and non-streaming is a class of bug worth designing out, and the streaming path has only the generation dict to work from, not a response object. message_start has to carry the prompt token count before generation begins, but the pipeline only reports it in the finish chunk. Rather than tokenize twice, validate_context_length now returns the length it already computes and check_context_length passes it back, so the streaming preflight the endpoint performs anyway supplies the count. The finish chunk remains authoritative and is reported in message_delta. Errors reaching a stream that has already committed HTTP 200 are emitted as an error event in the Anthropic envelope, since the status line is long gone. Two hangs are designed out. The collector normally ends the stream with a finish chunk, but a plain return would leave the consumer awaiting a chunk that never arrives, holding the connection and the model slot open, so the queue is raced against the collector task. A client that hangs up mid-stream leaves the collector running, so it is cancelled on the way out. When developer.disable_request_streaming is set, a streaming request is rejected rather than served a non-streaming body: an SSE client would fail in its own parser rather than report what went wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tool definitions are translated into the OAI shape chat templates already render, so the Anthropic API inherits every tool call format the pipeline supports rather than reimplementing detection or parsing per format. tool_choice maps auto and none unchanged, any onto "required", and a named choice onto a forced call. disable_parallel_tool_use is inverted into parallel_tool_calls, and left unset when the request does not mention it so the pipeline's own default applies. Anthropic's server-side tools arrive in the same tools list distinguished by a type field; they run on their infrastructure, so they are rejected by name rather than failing schema validation. A tool_use block replayed by the client becomes an OAI tool call whose arguments are a JSON string, which is what templates render and what format_messages_with_template parses back for the templates wanting a mapping. Tool results need a shape change, not just a rename. Anthropic packs every result for a turn into one user message, while chat templates expect one tool message per result, so a single message fans out. The results lead the turn, which is the order Anthropic requires them in anyway, and any text that followed them becomes the user message after. A turn of nothing but results adds no user message of its own, and an assistant turn that only called tools carries no content. is_error is folded into the result text as an Error prefix. Templates have no concept of a failed call, and the model can only react to the failure if it can read it. On the way out, tool calls become tool_use blocks after the text introducing them. The pipeline hands back arguments as a JSON string, so it is parsed into the object the wire format carries; a string that will not parse into an object surfaces the call with empty input and a warning rather than failing the whole response, since the tool name is the useful part. Streaming emits each tool call as its own block. Tool calls are parsed only once generation has finished, so the arguments arrive whole and are sent as a single input_json_delta, which an accumulating client handles the same as fragments. Any open text block is closed first, so block indices stay well-ordered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Image blocks become the image_url message parts the templating step turns into embeddings, so both APIs reach the image loader the same way: a base64 source becomes the data URL the loader already parses, and a url source passes through. A file source names the Anthropic Files API, which has no local counterpart to resolve an id against, and is rejected. An image is rejected outright unless the loaded model has vision enabled. Templating only builds embeddings for a vision model, so the alternative is dropping the image and asking the model about a picture it was never shown. Message content stays a plain string unless an image is present, keeping the common case the flat string a template renders. When one is, content becomes an ordered part list, and consecutive text blocks are still joined on a blank line so an image between them does not change how the surrounding text reads. Images are also accepted inside a tool_result, which is how a screenshotting tool reports back. An error result carrying only an image keeps its Error marker as a leading text part. A block naming a supported type that still fell through to the permissive fallback failed its own validation, which is a different problem from an unhandled type. Reporting it as unsupported was actively misleading: a malformed image inside a tool_result claimed images were not supported there while the same sentence offered to accept them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/v1/models now carries the Anthropic field names (type, display_name, created_at) beside the OpenAI ones, and the list carries has_more with computed first_id and last_id, so an Anthropic SDK can list models without a second endpoint. The ids are computed rather than assigned at construction because callers build the list empty and append to data afterwards, which would leave them null. The card for the loaded model now reports its context window. Anthropic's model schema calls it max_input_tokens and carries max_tokens beside it, so both are served, computed from max_seq_len rather than stored, which also keeps them null on a card that has no parameters. They are equal because a generation here is bounded by the context minus the prompt and not by an output limit of its own. The same number stays available as parameters.max_seq_len, which is what the OpenAI side of the listing already documented. A listing that reported none of this left a client with no way to learn the number short of running into a context length error. Only the loaded model has parameters to report, so every other directory in the listing stays bare, and the prompt template content is dropped from a listing because kilobytes of Jinja on a card nobody reads it from is not what a listing is for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Recent Anthropic models take operator instructions as a system-role entry in the messages array rather than by editing the top-level system prompt, and Claude Code sends one on every request. The role was restricted to user and assistant, so every Claude Code request failed validation with 422. Accepting the role is not enough on its own. Chat templates generally allow a system turn only in first position, and Qwen's raises "System message must be at the beginning" outright, which would have turned the 422 into a template error. A system message that is not first is therefore carried in a user turn wrapped in <system-reminder>, the same fallback the Anthropic API documents for models without the feature. One that is first, and has no top-level system prompt ahead of it, becomes the system turn. Folding a later one into the leading system prompt was the alternative and is the wrong trade: that prompt sits at the front of the prefix, so rewriting it every turn invalidates the whole prompt cache, which is the cost the mid-conversation channel exists to avoid. Inserting in place leaves the cached prefix ahead of it intact. Verified against Claude Code driving a local Tess-4-27B: a plain turn, and a tool-using turn where it read a file through the Read tool and answered from the result. Both streamed, both with concurrent requests in flight, all 200. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chat completion endpoint lets a request pass arbitrary variables to the chat template, which is how a client reaches model-specific knobs the API surface has no field for. The Anthropic endpoint mapped only the thinking field onto enable_thinking, so anything else was reachable solely through config, leaving the two endpoints asymmetric for no reason other than the Messages API not having somewhere obvious to put it. Accept template_vars, under the same name and chat_template_kwargs alias the chat completion path uses, on both messages and count_tokens: counting has to render the prompt generation would, so the same variables have to apply. Precedence follows the existing chain rather than inventing one. Explicit variables outrank the value derived from thinking, matching how the chat completion path ranks template_vars above its own flat reasoning fields, and both still lose to the model's template_vars_force. The Anthropic SDKs will not send an unknown field, so this is reached through their escape hatch (extra_body) or plain HTTP, which is the usual way to pass a provider extension. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The backend already reports how much of each prompt the prefix cache served, and the metrics endpoint has always used it, but the figure stopped at handle_finish_chunk as far as the APIs were concerned. The Anthropic endpoint hardcoded cache_read_input_tokens to zero and reported the whole prompt as input_tokens, so a client pricing a conversation saw every replayed turn as freshly processed. Over a session that is most of the prompt counted again on every request, which is why an estimate climbs so fast. Carry cached_tokens on UsageStats, filled from the finish chunk and taken from the first choice when aggregating, since one prompt is shared across choices. The Anthropic endpoint then splits it the way Anthropic counts: input_tokens is what was actually processed and cache_read_input_tokens the remainder, which are disjoint rather than additive. cache_creation_input_tokens stays zero. The backend does not distinguish writing to the cache from ordinary prefill, and clients price a cache write above plain input, so a guess there would overstate rather than understate. The streaming path now repeats the whole usage object in message_delta instead of the output count alone. The split is only known once prefill has run, so message_start cannot carry it and reports the full prompt length as its best estimate at that point. This makes a client's token accounting truthful. The rate applied to those tokens is still whatever the client falls back to for an unrecognized model, so an absolute cost figure remains notional. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is your pull request related to a problem? Please describe.
A clear and concise description of what the problem is. You can also link to an existing issue.
Why should this feature be added?
An explanation of why the feature should be added. Please be as specific as possible to help us understand the reasoning.
Examples
Examples of the feature in action and its significance compared to not having the feature.
Additional context
Add any other context or screenshots about the pull request here.