Skip to content

Context budgets never reserve limit.output — guaranteed ContextWindowExceeded on engines enforcing prompt+max_tokens <= window #275

Description

@EagleClaw89

Summary

Every context budget in the plugin is computed as a percentage of the whole resolved context limit — limit.output is never subtracted anywhere. On engines that enforce prompt + max_tokens <= max_model_len (vLLM, and most OpenAI-compatible local servers), this makes a hard ContextWindowExceededError unavoidable for any session that genuinely fills up, even when the user's opencode config declares the model window accurately.

Details (verified against 0.32.4 dist bundle)

  • resolveLimit(limit) returns limit.input ?? limit.context, whole. On opencode 1.18.x the input branch is dead code (the binary's LLM.ModelLimits class carries only {context, output} and construction sites drop other keys), so the plugin always budgets against limit.context.
  • Every downstream budget is multiplicative on that value: deriveProtectedTailTokenTarget (usable = contextLimit * threshold / 100), selectPerRunCap, resolveHistoryBudgetTokens, the 95% EMERGENCY_DRAIN band, planEmergencyDrop's ceiling.
  • A grep of the whole bundle for limit.output finds exactly one hit — a finish_reason string comparison. The output reservation opencode will add as max_tokens on the very request being trimmed is invisible to the trimmer.

Concrete failure (measured): model declared limit: {context: 122880, output: 16384} against a vLLM engine with max_model_len = 131072. The plugin allows input to grow to ~122,880; opencode sends max_tokens = 16384 on top; 122880 + 16384 = 139264 > 131072 → hard 400 at the engine. Sessions in our fleet repeatedly died at 93%+ of the declared context. (The overflow-detection/detected_context_limit machinery correctly min()s — it cannot help here because the configured value is the one that overshoots once output is added.)

Suggested fix

Either (or both):

  1. When only limit.context and limit.output are available (opencode ≤ 1.18.x), budget against limit.context − limit.output rather than limit.context.
  2. Add an output_reserve (or similar) config key so operators can state the reservation explicitly where the harness cannot supply limit.input.

The workaround we shipped is to hand-encode the reservation into limit.context (declare context = max_model_len − output − slack), which works but makes the config value mean something different from what its name says, and every operator has to rediscover this the hard way.

Happy to provide the full trace/measurements if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions