From a3baea5bc9011d4b43f688e0e882938268209ee8 Mon Sep 17 00:00:00 2001 From: Jazzcort Date: Thu, 13 Aug 2026 16:04:08 -0400 Subject: [PATCH] LCORE-1675: Documentation for conversation compaction Document the conversation compaction feature across OpenAPI spec, configuration guide, architecture overview, and query endpoint docs. Add context_status field ("full"/"summarized") to QueryResponse and StreamingQueryResponse documentation. Create comprehensive user guide at docs/user_doc/conversation_compaction.md with configuration examples, behavior details, and FAQ. --- docs/README.md | 2 + docs/devel_doc/ARCHITECTURE.md | 90 +++++++++++- docs/devel_doc/openapi.md | 5 +- docs/devel_doc/query_endpoint.md | 15 +- docs/index.md | 2 + docs/user_doc/config.md | 45 ++++++ docs/user_doc/conversation_compaction.md | 171 +++++++++++++++++++++++ 7 files changed, 316 insertions(+), 14 deletions(-) create mode 100644 docs/user_doc/conversation_compaction.md diff --git a/docs/README.md b/docs/README.md index b8cfebb23..af86732ab 100644 --- a/docs/README.md +++ b/docs/README.md @@ -29,6 +29,8 @@ See the full documentation at [`../README.md`](../README.md) or browse sub-pages [OKP guide](https://lightspeed-core.github.io/lightspeed-stack/user_doc/okp_guide.html) +[Conversation compaction](https://lightspeed-core.github.io/lightspeed-stack/user_doc/conversation_compaction.html) + [Authentication and Authorization](https://lightspeed-core.github.io/lightspeed-stack/user_doc/auth.html) [User data collection](https://lightspeed-core.github.io/lightspeed-stack/user_doc/user_data_collection.html) diff --git a/docs/devel_doc/ARCHITECTURE.md b/docs/devel_doc/ARCHITECTURE.md index 03bb277ef..0308c7ae9 100644 --- a/docs/devel_doc/ARCHITECTURE.md +++ b/docs/devel_doc/ARCHITECTURE.md @@ -35,7 +35,7 @@ To keep requests on-topic and protect sensitive data, LCore applies **safety shi - **Multi-Provider Support**: Works with multiple LLM providers (Ollama, OpenAI, Watsonx, etc.) - **Enterprise Security**: Authentication, authorization (RBAC), and secure credential management - **Resource Management**: Token-based quota limits and usage tracking -- **Conversation Management**: Multi-turn conversations with history and caching +- **Conversation Management**: Multi-turn conversations with history, caching, and automatic compaction - **RAG Integration**: Retrieval-Augmented Generation for context-aware responses - **Tool Orchestration**: Model Context Protocol (MCP) server integration - **Observability**: Prometheus metrics, structured logging, and health checks @@ -370,6 +370,83 @@ External A2A requests go through LCore's standard authentication system (K8s, RH --- +### 2.11 Conversation Compaction (`utils/compaction.py`, `utils/conversation_compaction.py`) + +**Purpose:** Automatically summarize older conversation turns when the conversation history approaches the LLM's context window limit, preventing HTTP 413 failures and enabling arbitrarily long conversations. + +**Design Philosophy (Option A):** Once compaction triggers, LCore takes ownership of the context sent to the LLM. The `conversation` parameter is dropped from the OGX call (`omit_conversation=True`), and LCore constructs the input explicitly from summaries + recent turns + new query. The full original history remains in OGX for auditing. + +**Architecture:** + +The compaction system is split into two layers: + +1. **Pure Logic Layer** (`utils/compaction.py`) — Side-effect-free functions: + - `partition_conversation()` — Splits conversation items into old and recent chunks using a *degrading guard*: starts with the configured `buffer_turns` and shrinks one pair at a time until the recent chunk fits the token budget + - `summarize_chunk()` — Single LLM call to produce a `ConversationSummary` from older turns + - `recursively_resummarize()` — Folds multiple accumulated summaries into one when they approach the context limit + +2. **Runtime Integration Layer** (`utils/conversation_compaction.py`) — Manages side effects: + - Per-conversation locking (serializes concurrent requests on the same conversation) + - Compaction state loading (cache-preferred with marker fallback) + - Marker persistence (`[lightspeed:compaction-summary]` sentinel in conversation items) + - `CompactionStartedEvent` emission for streaming progress indicators + - `apply_compaction()` (async generator) — Main entry point used by all endpoints + - `store_compacted_turn()` — Appends user query + LLM output when in compacted mode + +**Data Flow:** + +``` +User Query → Estimate Tokens → Exceeds Threshold? + │ + No │ Yes + ↓ │ ↓ + Pass-through Acquire Lock + ↓ + Fetch Conversation Items + ↓ + Load Compaction State + (cache → marker fallback) + ↓ + Partition (old | recent) + ↓ + Summarize Old Chunk (LLM call) + ↓ + Write Marker + Cache Summary + ↓ + Recursive Fold (if needed) + ↓ + Build Explicit Input: + [summaries + recent + query] + ↓ + Set omit_conversation=True + ↓ + Release Lock → Continue to LLM +``` + +**Endpoint Integration:** + +| Endpoint | Mode | Cache | `context_status` | +|---|---|---|---| +| `/v1/query` | Blocking (`apply_compaction_blocking()`) | Yes | Yes (`"full"` / `"summarized"`) | +| `/v1/streaming_query` | Streaming (`apply_compaction()` generator) | Yes | Yes (in `end` event) | +| `/v1/responses` | Blocking | Yes | No (OpenAI-compatible, silent) | +| `/a2a` | Blocking, marker-only (no cache) | No | No (A2A protocol scope) | + +**Configuration:** + +Compaction is controlled by `CompactionConfiguration` in `lightspeed-stack.yaml`: +- `enabled` (default: `false`) — Master switch +- `threshold_ratio` (default: `0.7`) — Fraction of context window that triggers compaction +- `token_floor` (default: `4096`) — Minimum token count before compaction can fire +- `buffer_turns` (default: `4`) — Recent turns kept verbatim +- `buffer_max_ratio` (default: `0.3`) — Max fraction of window for the buffer + +Models must have context windows registered via `inference.context_windows` (a map of model ID to token count). + +**Concurrency:** A per-conversation lock dictionary serializes concurrent compaction requests on the same conversation. Lock entries are reference-counted and cleaned up when the last waiter exits. + +--- + ## 3. Request Processing Pipeline This section illustrates how requests flow through LCore from initial receipt to final response. @@ -400,11 +477,12 @@ Here's how a real query flows through the system: 5. **Model Selection** - Use configured default model (e.g., `meta-llama/Llama-3.1-8B-Instruct`) 6. **Context Building** - Retrieve conversation history, query RAG vector stores for relevant docs, determine available MCP tools 7. **Shield moderation** - LCore-owned direct-run moderation (and agent capabilities where applicable) using shields configured in LCORE config -8. **Llama Stack / agent call** - Send request with system prompt, RAG context, and MCP tools -9. **LLM Processing** - Stack / agent generates response, may invoke MCP tools, returns token counts -10. **Post-Processing** - Generate conversation summary if new -11. **Store Results** - Save to Cache DB, User DB, consume quota, update metrics -12. **Return Response** - Complete LLM response with referenced documents, token usage, and remaining quota +8. **Conversation compaction** - If enabled and estimated tokens exceed the threshold, summarize older turns and rebuild the context (see [Section 2.11](#211-conversation-compaction-utilscompactionpy-utilsconversation_compactionpy)) +9. **OGX / agent call** - Send request with system prompt, RAG context, and MCP tools +10. **LLM Processing** - Stack / agent generates response, may invoke MCP tools, returns token counts +11. **Post-Processing** - Generate conversation summary if new +12. **Store Results** - Save to Cache DB, User DB, consume quota, update metrics +13. **Return Response** - Complete LLM response with referenced documents, token usage, and remaining quota **Key Takeaways:** - RAG enhances responses with relevant documentation diff --git a/docs/devel_doc/openapi.md b/docs/devel_doc/openapi.md index 34e1e9f1c..ed4785fb3 100644 --- a/docs/devel_doc/openapi.md +++ b/docs/devel_doc/openapi.md @@ -2738,7 +2738,7 @@ user's query to a selected Llama Stack LLM and returning the generated response. - mcp_headers: Headers that should be passed to MCP servers. ### Returns: -- QueryResponse: Contains the conversation ID and the LLM-generated response. +- QueryResponse: Contains the conversation ID, the LLM-generated response, and a `context_status` field indicating whether the conversation context is `"full"` or `"summarized"`. ### Raises: - HTTPException: @@ -3021,7 +3021,7 @@ content type text/event-stream. - mcp_headers: Headers that should be passed to MCP servers. ### Returns: -- SSE-formatted events for the query lifecycle. +- SSE-formatted events for the query lifecycle. Includes a `context_status` field (`"full"` or `"summarized"`) in the `end` event payload indicating whether conversation compaction was applied. When compaction is triggered, a `compaction` SSE event is emitted before inference begins. ### Raises: - HTTPException: @@ -8010,6 +8010,7 @@ Attributes: | available_quotas | object | Quota available as measured by all configured quota limiters | | tool_calls | array | List of tool calls made during response generation | | tool_results | array | List of tool results | +| context_status | string | Indicates whether the conversation context sent to the LLM is `"full"` (complete history) or `"summarized"` (older turns were summarized). Only present in QueryResponse and StreamingQueryResponse; omitted from `/v1/responses` (OpenAI-compatible) and `/a2a` responses. | ## QuotaExceededResponse diff --git a/docs/devel_doc/query_endpoint.md b/docs/devel_doc/query_endpoint.md index a49613a9e..e5aded8cd 100644 --- a/docs/devel_doc/query_endpoint.md +++ b/docs/devel_doc/query_endpoint.md @@ -145,6 +145,7 @@ The optional `solr` field configures Solr inline RAG behavior: | `tool_results` | array[object] | `[]` | Tool call results | | `rag_chunks` | array[object] | `[]` | *(Deprecated)* RAG chunks used | | `truncated` | boolean | `false` | *(Deprecated)* Always `false` | +| `context_status` | string | `"full"` | Whether the conversation context is `"full"` (complete history) or `"summarized"` (older turns were summarized via conversation compaction) | **`referenced_documents` items:** @@ -242,7 +243,7 @@ Emitted when the full response is assembled. #### 7. `end` -Emitted last on success. Contains metadata. +Emitted last on success. Contains metadata including `context_status` (`"full"` or `"summarized"`). ```json { @@ -251,7 +252,8 @@ Emitted last on success. Contains metadata. "referenced_documents": [], "truncated": null, "input_tokens": 11, - "output_tokens": 19 + "output_tokens": 19, + "context_status": "full" }, "available_quotas": {"UserQuotaLimiter": 998911} } @@ -327,9 +329,9 @@ Both endpoints share the same pre-processing pipeline: 11. Prepare Responses API parameters (model, system prompt, tools, MCP headers) 12. Extract image attachments separately for multimodal input construction -**`/v1/query` then:** applies conversation compaction (blocking), calls the LLM, generates topic summary, consumes tokens, stores results, returns JSON. +**`/v1/query` then:** applies conversation compaction (blocking), calls the LLM, generates topic summary, consumes tokens, stores results, returns JSON. When compaction is applied, the response includes `context_status: "summarized"`; otherwise `context_status: "full"`. -**`/v1/streaming_query` then:** generates a `request_id`, starts the SSE stream, emits events as the LLM generates tokens, performs post-stream cleanup (topic summary, token consumption, persistence). +**`/v1/streaming_query` then:** generates a `request_id`, starts the SSE stream, applies compaction if needed (emitting a `compaction` SSE event), emits events as the LLM generates tokens, performs post-stream cleanup (topic summary, token consumption, persistence). The `end` event includes `context_status` indicating whether compaction was applied. --- @@ -409,7 +411,8 @@ curl -X POST http://localhost:8090/v1/query \ "tool_calls": [], "tool_results": [], "rag_chunks": [], - "truncated": false + "truncated": false, + "context_status": "full" } ``` @@ -500,7 +503,7 @@ data: {"event": "token", "data": {"id": 2, "token": " an"}} data: {"event": "turn_complete", "data": {"id": 50, "token": "Kubernetes is an open-source..."}} -data: {"event": "end", "data": {"referenced_documents": [], "truncated": null, "input_tokens": 11, "output_tokens": 50}, "available_quotas": {"UserQuotaLimiter": 998950}} +data: {"event": "end", "data": {"referenced_documents": [], "truncated": null, "input_tokens": 11, "output_tokens": 50, "context_status": "full"}, "available_quotas": {"UserQuotaLimiter": 998950}} ``` ### Streaming Query Interrupt diff --git a/docs/index.md b/docs/index.md index c11f2e2b7..ff5227db0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,6 +34,8 @@ product questions using backend LLM services, agents, and RAG databases. [OKP guide](https://lightspeed-core.github.io/lightspeed-stack/user_doc/okp_guide.html) +[Conversation compaction](https://lightspeed-core.github.io/lightspeed-stack/user_doc/conversation_compaction.html) + [Authentication and Authorization](https://lightspeed-core.github.io/lightspeed-stack/user_doc/auth.html) [User data collection](https://lightspeed-core.github.io/lightspeed-stack/user_doc/user_data_collection.html) diff --git a/docs/user_doc/config.md b/docs/user_doc/config.md index bf8d52d2b..433f1d3ee 100644 --- a/docs/user_doc/config.md +++ b/docs/user_doc/config.md @@ -224,6 +224,51 @@ Attributes: | buffer_turns | integer | Number of recent turns to keep verbatim. | | buffer_max_ratio | number | Maximum fraction of context window the buffer zone can occupy, regardless of buffer_turns. | +### How to enable conversation compaction + +Compaction is disabled by default. To enable it, add a `compaction` section to your `lightspeed-stack.yaml` and set `enabled: true`. You must also register context window sizes for the models you use via the `inference.context_windows` map so the compaction trigger can calculate when older turns should be summarized. + +**Minimal configuration:** + +```yaml +inference: + default_provider: openai + default_model: gpt-4o-mini + context_windows: + openai/gpt-4o-mini: 128000 + +compaction: + enabled: true +``` + +**Full configuration with all options:** + +```yaml +inference: + default_provider: openai + default_model: gpt-4o-mini + context_windows: + openai/gpt-4o-mini: 128000 + openai/gpt-4o: 128000 + +compaction: + enabled: true + threshold_ratio: 0.7 # trigger at 70% of context window (default) + token_floor: 4096 # minimum tokens before compaction can fire (default) + buffer_turns: 4 # recent turns kept verbatim (default) + buffer_max_ratio: 0.3 # buffer may use at most 30% of the window (default) +``` + +**Key considerations:** + +- `context_windows` is required. Models absent from this map have no registered window and compaction will not trigger for them. +- `threshold_ratio` controls how aggressively compaction fires. Lower values compact sooner; higher values wait longer (closer to the window limit). +- `buffer_turns` sets how many recent user/assistant turn pairs are kept in full. A degrading guard automatically reduces this if the buffer itself would exceed `buffer_max_ratio` of the window. +- `token_floor` prevents compaction from triggering on very short conversations. +- When compaction is disabled (the default), requests that exceed the context window surface as HTTP 413. + +For a comprehensive explanation of the feature, see the [Conversation Compaction Guide](conversation_compaction.md). + ## Configuration diff --git a/docs/user_doc/conversation_compaction.md b/docs/user_doc/conversation_compaction.md new file mode 100644 index 000000000..19b1822cd --- /dev/null +++ b/docs/user_doc/conversation_compaction.md @@ -0,0 +1,171 @@ +# Conversation Compaction Guide + +## Overview + +Conversation compaction is a feature that automatically summarizes older conversation turns when the conversation history approaches the LLM's context window limit. Instead of failing with an HTTP 413 error when the input becomes too long, compaction condenses earlier parts of the conversation into a summary while preserving recent turns verbatim. This allows long-running conversations to continue seamlessly. + +## How it works + +When a user sends a query, the system estimates the total token count of the conversation history. If the estimated tokens exceed a configurable fraction of the model's context window (the *threshold ratio*), compaction is triggered: + +1. **Partition** -- The conversation is split into two parts: older turns and a recent buffer of the most recent turns. +2. **Summarize** -- The older turns are sent to the LLM with a summarization prompt. The resulting summary is stored as a compaction marker in the conversation. +3. **Rebuild context** -- The LLM receives the summary plus the recent buffer plus the new user query, keeping the total input well within the context window. +4. **Recursive fold** -- If accumulated summaries themselves grow too large, they are recursively re-summarized into a single condensed summary. + +After compaction, the service takes ownership of the context window. The full original conversation history remains stored in OGX for auditing and retrieval, but only the compacted view is sent to the LLM for inference. + +### Affected endpoints + +| Endpoint | Compaction behavior | +|---|---| +| `POST /v1/query` | Blocking compaction before inference. Response includes `context_status`. | +| `POST /v1/streaming_query` | Compaction runs inside the SSE stream. A `compaction` event is emitted before tokens begin. The `end` event includes `context_status`. | +| `POST /v1/responses` | Compaction runs silently (no `context_status` in response). The `/v1/responses` endpoint follows the OpenAI Responses API specification and does not add custom fields. | +| `POST /a2a` | Compaction runs in marker-only mode (no cache, no recursive fold). No `context_status` is surfaced. | + +### The `context_status` field + +The `/v1/query` and `/v1/streaming_query` endpoints include a `context_status` field in their responses: + +| Value | Meaning | +|---|---| +| `"full"` | The complete conversation history was sent to the LLM without summarization. | +| `"summarized"` | Older conversation turns were summarized before sending to the LLM. | + +**Synchronous response example (`/v1/query`):** + +```json +{ + "conversation_id": "123e4567-e89b-12d3-a456-426614174000", + "response": "Here is the answer to your question...", + "context_status": "summarized", + "input_tokens": 1250, + "output_tokens": 200, + "available_quotas": {"UserQuotaLimiter": 998550} +} +``` + +**Streaming `end` event example (`/v1/streaming_query`):** + +```json +{ + "event": "end", + "data": { + "referenced_documents": [], + "input_tokens": 1250, + "output_tokens": 200, + "context_status": "summarized" + }, + "available_quotas": {"UserQuotaLimiter": 998550} +} +``` + +**Streaming `compaction` event** (emitted before inference when compaction triggers): + +```json +{"event": "compaction", "data": {"status": "started", "conversation_id": "123e4567-e89b-12d3-a456-426614174000"}} +``` + +## Configuration + +Compaction is disabled by default. To enable it, add a `compaction` section to your `lightspeed-stack.yaml` configuration file and register context window sizes for your models. + +### Minimal configuration + +```yaml +inference: + default_provider: openai + default_model: gpt-4o-mini + context_windows: + openai/gpt-4o-mini: 128000 + +compaction: + enabled: true +``` + +### Full configuration + +```yaml +inference: + default_provider: openai + default_model: gpt-4o-mini + context_windows: + openai/gpt-4o-mini: 128000 + openai/gpt-4o: 128000 + +compaction: + enabled: true + threshold_ratio: 0.7 + token_floor: 4096 + buffer_turns: 4 + buffer_max_ratio: 0.3 +``` + +### Configuration fields + +| Field | Type | Default | Description | +|---|---|---|---| +| `enabled` | boolean | `false` | Master switch. When `false`, compaction never triggers and all other fields are inert. | +| `threshold_ratio` | float | `0.7` | Trigger compaction when estimated input tokens exceed this fraction of the model's context window. Valid range: 0.0 to 1.0. | +| `token_floor` | integer | `4096` | Minimum estimated token count before compaction can trigger, regardless of `threshold_ratio`. Prevents compaction on very short conversations. | +| `buffer_turns` | integer | `4` | Number of recent user/assistant turn pairs to keep verbatim (not summarized). The runtime applies a *degrading guard*: if these turns exceed the available budget, `buffer_turns` is reduced by one repeatedly until the budget fits, down to zero. | +| `buffer_max_ratio` | float | `0.3` | Hard cap on the fraction of the context window the recent buffer may occupy. Even if `buffer_turns` would fit, the buffer is trimmed if it exceeds this ratio. | + +### Prerequisites + +- **Context windows must be registered.** Add entries to `inference.context_windows` mapping each fully-qualified model identifier (e.g., `"openai/gpt-4o-mini"`) to its context window size in tokens. Models absent from this map have no registered window and compaction will not trigger for them. +- **A conversation cache is recommended.** While compaction works without a cache (using marker-only mode), enabling a conversation cache (PostgreSQL, SQLite, or in-memory) improves performance by caching summaries across requests and enabling recursive fold-up of accumulated summaries. + +## Behavior details + +### Compaction trigger + +Compaction triggers when **all** of the following are true: + +1. `compaction.enabled` is `true` +2. The model has a registered context window in `inference.context_windows` +3. The estimated token count of the conversation exceeds `threshold_ratio × context_window` +4. The estimated token count is at least `token_floor` + +### Degrading guard + +The `buffer_turns` setting specifies a target number of recent turns to preserve. If the selected buffer turns exceed the available budget (the context window minus the summary minus the new query), the system reduces the buffer by one turn pair at a time until the budget fits. In extreme cases, the buffer can shrink to zero turns, meaning only the summary and the current query are sent to the LLM. + +### Marker persistence + +When compaction occurs, a marker message (prefixed with `[lightspeed:compaction-summary]`) is appended to the conversation in OGX. This marker serves as a fallback for reconstructing the compacted state in cache-less deployments or after cache eviction. + +### Per-conversation locking + +Compaction acquires a per-conversation lock to prevent concurrent requests on the same conversation from racing during summarization. If multiple requests arrive simultaneously, they are serialized. The lock is automatically released after processing. + +### Recursive re-summarization + +Over very long conversations, multiple compaction summaries may accumulate. When the total size of cached summaries approaches the context window threshold, they are recursively folded into a single summary using a dedicated re-summarization prompt. This prevents summaries from themselves exceeding the context window. + +## When compaction is disabled + +When compaction is disabled (the default), requests that cause the conversation history to exceed the model's context window will fail with HTTP 413 (Prompt Too Long). Clients must manage conversation length themselves, for example by starting new conversations or deleting old ones. + +## Frequently asked questions + +**Does compaction lose information?** + +Compaction summarizes older turns, so fine-grained details from early in the conversation may be condensed. The full original conversation history remains stored in OGX and is retrievable via the conversations API. The LLM simply receives a summary instead of the full transcript for inference. + +**Does compaction use extra tokens?** + +Yes. The summarization step requires an additional LLM call, which consumes tokens. These tokens are counted against the user's quota. The trade-off is that the conversation can continue instead of failing with HTTP 413. + +**Can I use compaction with all LLM providers?** + +Compaction works with any provider supported by OGX, as long as the model's context window is registered in `inference.context_windows`. + +**How does compaction interact with RAG?** + +RAG context (retrieved documents) is injected at query time and is not affected by compaction. Compaction only summarizes conversation history turns, not RAG-injected content. + +**Why don't `/v1/responses` and `/a2a` return `context_status`?** + +The `/v1/responses` endpoint follows the OpenAI Responses API specification and must not include custom fields to remain compatible. The `/a2a` endpoint uses the A2A protocol specification, which does not define UI indicator fields. Compaction still runs on both endpoints — the status is simply not surfaced in the response.