update long context demo - #4415
Open
dtrawins wants to merge 1 commit into
Open
Conversation
przepeck
reviewed
Jul 29, 2026
| Parameter `--max_prompt_len` has impact on the latency. It should be adjusted for expected input length to optimize performance. | ||
|
|
||
| | Max prompt length | TTFT (ms) | TPOT (ms) | | ||
| | Max prompt length | 1k input TTFT (ms) | TPOT (ms) | |
Collaborator
There was a problem hiding this comment.
I am not sure why 1k input is here, we are setting max_prompt_len=8000 and still sending 1k tokens only? Is this similar for TPOT? Maybe it would be better to add another column, something like Input length, what do you think?
| - Set `--cache_interval_multiplier=64` to optimize memory usage with prefix caching | ||
| - Default (adaptive): optimizes memory for long context | ||
| - 64: recommended for medium context | ||
| - 8: this minimal value optimize for performance, recommended with short inputs |
Collaborator
There was a problem hiding this comment.
Can we add what does it mean short input? Like <2k or <500 tokens?
mzegla
reviewed
Jul 29, 2026
| **Cache interval multiplier** | ||
| This parameter is dedicated for models with linear attention and prefix caching enabled. It adjusts the allocation size for state blocks internally in openvino.genai backend. For processing long inputs with low memory footprint, it is recommended to increase this parameter from default value 8 to higher like 64. | ||
|
|
||
| **Max number batched tokens** |
Collaborator
There was a problem hiding this comment.
Suggested change
| **Max number batched tokens** | |
| **Max number of batched tokens** |
|
|
||
| **Max number batched tokens** | ||
| This parameter influences behavior of continuous batching algorithm and the size of chunked prompts for batching. It is efficient to use default value of 256 tokens when concurrent processing is expected. When usually one client is connecting to the local model, especially with long prompts, increasing the value might improve first token latency. | ||
| This parameter influences the behavior of the continuous batching algorithm and the size of chunked prompts. The default value of 256 tokens is efficient when concurrent processing is expected. When typically only one client connects to the local model (especially with long prompts), increasing this value can improve first token latency. |
Collaborator
There was a problem hiding this comment.
Personally the connection between this parameter and concurrency is not very clear to me.
| - **TTFT (Time To First Token)**: Latency from sending a request to receiving the first output token (in milliseconds) | ||
| - **TPOT (Time Per Output Token)**: Average time to generate each subsequent token after the first (in milliseconds) | ||
|
|
||
| The command below generates synthetic load with a configurable cached prompt length (5000) and new tokens length (10): |
Collaborator
There was a problem hiding this comment.
Maybe we could explain which parameters exactly configure input and output length
| ### Concurrency Control | ||
| To limit concurrent requests on the server: | ||
| - `--rest_workers`: Controls number of simultaneous connections (default: number of CPU cores) | ||
| - `--max_num_seqs`: Limits concurrency at the model level for finer control |
Collaborator
There was a problem hiding this comment.
Do you confirm it works? Last time I checked I had issues with limiting concurrency using this parameter.
|
|
||
| In a scenario with low concurrency and long context, increase `max_num_batched_tokens` to higher numbers like 4096 or even max model context. | ||
| ### Low-Concurrency / Long-Context Scenarios | ||
| - Increase `--max_num_batched_tokens` to 4096 or even the model's max context length |
Collaborator
There was a problem hiding this comment.
Perhaps we should run plan to run some profiling. Maybe it would be good to change current defaults or make it more adaptive (like keep 256 on CPU, change to 4k or more on GPU?)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary
🧪 Checklist
``