refresh performance and troubleshooting guide - #4416
Conversation
| Lower values can improve prefix-caching efficiency by using smaller chunks, but also increase memory consumption. | ||
| Allowed range: `8-256`. | ||
|
|
||
| - Set `enable_prefix_cache: true` (default) in the graph configuration to reuse KV cache for sequential requests with repeated prompt tokens (for example, chat history). This avoids duplicated prompt evaluation. |
There was a problem hiding this comment.
maybe to be consistent with other params use --enable_prefix_caching true
|
|
||
| - The `--max_num_batched_tokens` parameter influences how prompts are divided and grouped in the prefill phase. | ||
| The default value `256` is efficient for short contexts and high concurrency memory management. | ||
| Increasing this value to `4096`, `8192`, or even the model context limit can improve first-token latency, but may increase KV cache memory consumption. |
There was a problem hiding this comment.
Not sure I get the phrasing right. KV cache memory consumption as a space needed for KV per token is related to processed tokens amount. If we process 1000 tokens in one batch or in 10 batches, 100 tokens each, we need the same amount of memory for the generation. In higher concurrency lower max num batched tokens can indeed work as a bottleneck, so less concurrent requests are processed with big throughput and they get cleaned up before new ones are accepted.
The overall, peak memory consumption is strictly tied to this parameter though. The bigger the batch, the more math is happening, so we need to use more memory to execute single step. That's less impactful than KV cache growth, but might be relevant in some situations.
There was a problem hiding this comment.
yes, that is correct interpretation, with a single client, memory usage is the same. With concurrency it will be higher than using small chunks
| - Newly enabled LLM/VLM models may not always support the continuous batching pipeline, especially in preview or early-access scenarios. | ||
| In that case, switch to the non-continuous-batching pipeline with `--pipeline_type LM`, `--pipeline_type VLM`, or `--pipeline_type OMNI` as appropriate. | ||
| Continuous batching variants are the defaults: `LM_CB`, `VLM_CB`, and `OMNI_CB`. | ||
| If the model loads only after changing the pipeline type, the issue is usually a pipeline-model compatibility mismatch rather than a corrupted model. |
There was a problem hiding this comment.
Maybe worth noting that NPU does not support continuous batching pipelines at all.
Co-authored-by: Trawinski, Dariusz <dariusz.trawinski@intel.com>
| Lower values can improve prefix-caching efficiency by using smaller chunks, but also increase memory consumption. | ||
| Allowed range: `8-256`. | ||
|
|
||
| - Set `--enable_prefix_cache: true` (default) in the graph configuration to reuse KV cache for sequential requests with repeated prompt tokens (for example, chat history). This avoids duplicated prompt evaluation. |
There was a problem hiding this comment.
| - Set `--enable_prefix_cache: true` (default) in the graph configuration to reuse KV cache for sequential requests with repeated prompt tokens (for example, chat history). This avoids duplicated prompt evaluation. | |
| - Set `--enable_prefix_cache true` (default) in the graph configuration to reuse KV cache for sequential requests with repeated prompt tokens (for example, chat history). This avoids duplicated prompt evaluation. |
3 participants
🛠 Summary
🧪 Checklist
``