refactor: add per-model runtime-adapter contract + DeepSeek/GPT-OSS adapters - #257
Open
Andrewxu313 wants to merge 2 commits into
Open
refactor: add per-model runtime-adapter contract + DeepSeek/GPT-OSS adapters#257Andrewxu313 wants to merge 2 commits into
Andrewxu313 wants to merge 2 commits into
Conversation
Add batchgen/contracts/runtime_adapter.py: frozen RuntimeState (sole worker<->model coupling point) + ModelRuntimeAdapter ABC for the behavioral leaks audited in core_model_purity_audit.md (attention-backend config, position-id computation, per-token KV byte size). Defaults cover the common GQA/non-MLA case; past_kv_byte_size is abstract. Additive only -- generic files do not call it yet (phase A). Mirrors the cuda_graph adapter pattern. Adds GPU-free contract test (6 cases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrewxu313 <Andrewxu313@users.noreply.github.com>
Per-model ModelRuntimeAdapter implementations for the two models on the generic decode path, reproducing the exact legacy inline formulas they will replace: DeepSeek-V3 (full decode position_ids, compressed_kv_dim KV byte size, flash-attn toggle) and GPT-OSS (GQA KV byte size; base defaults otherwise). Exposed via each initializer's get_runtime_behavior_adapter(). Additive only -- decode.py/prefill.py do not call them yet. Equivalence unit tests prove adapter outputs match the original decode.py formulas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrewxu313 <Andrewxu313@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add
batchgen/contracts/runtime_adapter.py: a frozenRuntimeState(the sole worker↔model coupling point) +ModelRuntimeAdapterABC for per-model runtime behaviors that otherwise leak into the core asif "<model>" in model_typebranches (attention-backend config, position-id computation, per-token KV byte size). Adds DeepSeek-V3 and GPT-OSS adapters reproducing the exact legacy formulas, exposed via each initializer'sget_runtime_behavior_adapter(). Additive only — the generic runtime does not call them yet (the call-site swap is a follow-up). Mirrors the existingcuda_graph/adapter.pypattern.Motivation
Foundation for de-leaking per-model behavioral branches out of
batchgen_worker.py. Landing the contract + adapters first (with equivalence tests) lets the worker swap follow safely. Part of the modularization effort.Type of Change
Checklist