Servable framework refactor - phase 2

[mzegla]

To be added...

[Copilot]

Pull request overview

This PR advances the “servable framework refactor” by unifying unary and streaming response handling around the same delta-based pipeline (OVMSTextStreamer → OutputParser::parseChunk → accumulated deltas), and by consolidating duplicated “legacy servable” synchronization / partial-response logic into shared base utilities. It also migrates output parsers away from bespoke unary parsing toward a single streaming/state-machine model configured via the new ParsingConfig.

Changes:

• Unify unary response generation with streaming by collecting OVMSTextStreamer deltas and serializing from deltas (including multi-sequence/unary support).
• Introduce ParsingConfig and refactor output parsers to be configuration-driven and streaming-first (removing many unary parse() implementations).
• Reduce duplication in legacy servables via LegacyServableExecutionContextBase and shared prepareLegacyPartialResponse, and update/extend unit tests using a shared streamer-based test helper.

Reviewed changes

Copilot reviewed 69 out of 69 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/test/llm/output_parsers/qwen3coder_output_parser_test.cpp	Switch tests to streamer-driven parsing helper.
src/test/llm/output_parsers/qwen3_output_parser_test.cpp	Use streamer-driven unary aggregation; disable a couple of known-gap tests with TODO context.
src/test/llm/output_parsers/phi4_output_parser_test.cpp	Use streamer-driven unary aggregation; adjust/disable behavior-gap test.
src/test/llm/output_parsers/output_parser_test_utils.hpp	New helper to drive token streams through OVMSTextStreamer in tests and aggregate deltas into ParsedOutput.
src/test/llm/output_parsers/mistral_output_parser_test.cpp	Use streamer-driven unary aggregation; disable behavior-gap test; update expectations to unified contract.
src/test/llm/output_parsers/minicpm5_output_parser_test.cpp	Move tests to streamer-driven path; update implicit reasoning scenario to match production template behavior.
src/test/llm/output_parsers/llama3_output_parser_test.cpp	Use streamer-driven unary aggregation.
src/test/llm/output_parsers/hermes3_output_parser_test.cpp	Use streamer-driven unary aggregation; disable behavior-gap test around malformed tool calls.
src/test/llm/output_parsers/gemma4_output_parser_test.cpp	Use streamer-driven unary aggregation; adjust special-token decode expectation when no tools available.
src/test/llm/output_parsers/devstral_output_parser_test.cpp	Use streamer-driven unary aggregation; update expectations for invalid order handling.
src/llm/visual_language_model/legacy/servable.hpp	Migrate legacy VLM servable to shared legacy base context/base class; add legacy finish/usage hooks.
src/llm/visual_language_model/legacy/servable.cpp	Remove bespoke partial response implementation; unify unary serialization to delta-based path.
src/llm/servable.hpp	Add legacy execution context base + shared legacy partial-response function + LegacyServableBase.
src/llm/servable.cpp	Rework unary response serialization to accumulate per-sequence deltas (multi-sequence support) and wire in shared legacy partial-response path.
src/llm/ovms_text_streamer.hpp	Extend streamer to dynamically layer parser-required special-token decode mode over user preference.
src/llm/ovms_text_streamer.cpp	Implement dynamic decode-mode switching and two-step STOP flushing for parser finalization.
src/llm/omni_model/legacy/servable.cpp	Switch unary legacy omni response path to delta-based serialization; adjust special-token configuration logic.
src/llm/language_model/legacy/servable.hpp	Migrate legacy LLM servable to shared legacy base context/base class; add legacy finish/usage hooks.
src/llm/language_model/legacy/servable.cpp	Remove bespoke partial response implementation; unify unary serialization to delta-based path.
src/llm/io_processing/utils.cpp	Improve single-quote handling to treat apostrophes inside words as non-quote delimiters.
src/llm/io_processing/qwen3coder/qwen3coder_tool_parser.hpp	Add ParsingConfig override support; remove bespoke start/end tag overrides.
src/llm/io_processing/qwen3coder/qwen3coder_tool_parser.cpp	Move parser boundary config into BaseOutputParser(ParsingConfig); remove unary parse().
src/llm/io_processing/qwen3/reasoning_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/qwen3/reasoning_parser.cpp	Remove unary parse() implementation (streaming path only).
src/llm/io_processing/phi4/tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/phi4/tool_parser.cpp	Remove unary parse() implementation (streaming path only).
src/llm/io_processing/parsing_config.hpp	New shared configuration struct for phase detection and decode-mode requirements.
src/llm/io_processing/output_parser.hpp	Remove unary parse(); add resetStreamingState and dynamic decode-mode decision API; store parser identity for reset.
src/llm/io_processing/output_parser.cpp	Implement resetStreamingState and needSpecialTokensForCurrentDecode; remove unary parse().
src/llm/io_processing/mistral/tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/mistral/tool_parser.cpp	Remove unary parse(); improve streaming empty-chunk finalization and start-tag handling.
src/llm/io_processing/minicpm5/minicpm5_tool_parser.hpp	Move parser boundary config into ParsingConfig; remove unary parse().
src/llm/io_processing/minicpm5/minicpm5_tool_parser.cpp	Construct via ParsingConfig; remove unary parse().
src/llm/io_processing/minicpm5/minicpm5_reasoning_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/minicpm5/minicpm5_reasoning_parser.cpp	Rework streaming parseChunk to handle end-tag token batching and avoid emitting tag tokens.
src/llm/io_processing/llama3/tool_parser.hpp	Introduce defaultParsingConfig (special-token start boundary); remove unary parse().
src/llm/io_processing/llama3/tool_parser.cpp	Remove unary parse(); improve empty STOP flush handling for delayed arguments.
src/llm/io_processing/lfm2/lfm25_tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/lfm2/lfm25_tool_parser.cpp	Remove unary parse(); handle empty STOP flushes with pending buffered state.
src/llm/io_processing/lfm2/lfm25_reasoning_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/lfm2/lfm25_reasoning_parser.cpp	Remove unary parse(); move token IDs to local constants.
src/llm/io_processing/lfm2/lfm2_utils.cpp	Tighten streaming state handling for tool list bracket vs args parsing.
src/llm/io_processing/lfm2/lfm2_tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse(); add single-call parsing helper.
src/llm/io_processing/lfm2/lfm2_tool_parser.cpp	Remove unary parse(); add parseSingleToolCall and include shared utils.
src/llm/io_processing/hermes3/tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/hermes3/tool_parser.cpp	Remove unary parse(); handle empty STOP flushes with pending buffered state.
src/llm/io_processing/gptoss/tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/gptoss/tool_parser.cpp	Remove unary parse() implementation (streaming path only).
src/llm/io_processing/gptoss/reasoning_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/gptoss/reasoning_parser.cpp	Remove unary parse() implementation (streaming path only).
src/llm/io_processing/gemma4/gemma4_tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse().
src/llm/io_processing/gemma4/gemma4_tool_parser.cpp	Remove unary parse(); improve end-state handling and finalize/cleanup on STOP.
src/llm/io_processing/gemma4/gemma4_reasoning_parser.hpp	Migrate reasoning parser to ParsingConfig override; resolve special token IDs.
src/llm/io_processing/gemma4/gemma4_reasoning_parser.cpp	Remove unary parse() implementation (streaming path only).
src/llm/io_processing/devstral/tool_parser.hpp	Introduce defaultParsingConfig; migrate to ParsingConfig-based construction; remove unary parse(); track argument-emission.
src/llm/io_processing/devstral/tool_parser.cpp	Remove unary parse(); improve empty STOP flush behavior and invalid-order flushing behavior.
src/llm/io_processing/base_output_parser.hpp	Add ParsingConfig storage and special-token ID resolution; remove unary parse() from interface; default tag accessors from config.
src/llm/BUILD	Split OVMSTextStreamer into its own Bazel target and wire it into genai_servables deps.
src/llm/apis/openai_responses.hpp	Update unary serialization interface to delta-based APIs.
src/llm/apis/openai_responses.cpp	Implement delta-based unary serialization and multi-sequence overload for Responses endpoint.
src/llm/apis/openai_completions.hpp	Update unary serialization interface to delta-based APIs.
src/llm/apis/openai_completions.cpp	Implement delta-based unary serialization; rework VLM multi-choice serialization and echo usage accounting.
src/llm/apis/openai_api_handler.hpp	Replace token-based unary parsing helper with delta aggregation; add logprob carrier struct and new serializeUnaryResponse signatures.
src/llm/apis/openai_api_handler.cpp	Implement parsedOutputFromDeltas and new unary serialization overload routing.
src/BUILD	Update test target deps to use new text_streamer library and add test util header.
spelling-whitelist.txt	Whitelist test-specific “thi” substring used in split-tag test.

[Copilot]

Pull request overview

Copilot reviewed 71 out of 71 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/llm/ovms_text_streamer.cpp:68

• OVMSTextStreamer initializes m_current_special_mode based on the parser’s needs, but it never applies that mode to m_additional_detokenization_params in the constructor. If the parser requires special tokens in UNKNOWN phase (e.g. alwaysNeedsSpecialTokens=true) and the user requested skip_special_tokens=true, the streamer will still decode with special tokens skipped because write() won’t see a mode change (newMode == m_current_special_mode). This can break phase detection/parsing for those models.
  src/test/llm/llmnode_test.cpp:2086
• This test now accesses choice["delta"] unconditionally. If a chunk omits the delta member entirely (which is legal in some streaming event variants), RapidJSON’s operator[] on a missing member can assert/crash. Keep the original HasMember("delta") guard (and ensure it’s an object) before checking for content.