v1.2.0

This release introduces RL serving with verl, disaggregated ViT serving, a hybrid-cache architecture for linear-attention models, and a multi-level cache system with cache-aware PD. It also delivers broad model-performance improvements—including expanded MTP speculative decoding and optimizations across prefill, attention, MoE, quantization, and distributed inference—together with extensive stability and correctness fixes across scheduling, caching, multimodal processing, APIs, and long-running services.

Highlights

Reinforcement Learning Serving with verl

LightLLM can now serve as the rollout and inference backend for verl-based reinforcement learning workflows. A dedicated RL control plane supports online model-weight updates through distributed communication, tensors, or CUDA IPC, together with cache flushing, request aborts, generation pause/resume, and GPU-memory release/resume. The integration also includes routing-data capture for MoE training. #1298

Disaggregated ViT Serving

Vision encoding can now run as an independent visual_only service on dedicated GPU nodes. LLM servers use a proxy mode to discover registered visual workers through the config server, dispatch image encoding remotely, and retrieve the resulting embeddings through a shared AFS/Redis-backed cache. This separates ViT compute from language-model inference so the two tiers can be deployed and scaled independently. The release also adds configurable ViT attention backends and independent controls for visual and audio modules. #1234 #1191 #1229

Hybrid Radix Cache with Large and Small Pages

A new dual-granularity Hybrid Radix Cache is introduced for hybrid full-attention and linear-attention models such as Qwen3.5 and Qwen3-Next:

• Large pages represent multi-block, reusable prefix checkpoints and keep the full-attention KV data together with the corresponding linear-attention convolution and SSM states.
• Small pages preserve fine-grained or fragmented prefix tails, avoiding coarse page boundaries from limiting cache reuse.
• Small-page entries can be consolidated into large-page checkpoints and integrated with CPU cache/offload, allowing the cache to retain fine-grained matching while managing recurrent states efficiently.

The design addresses the different storage granularities required by token-level KV cache and recurrent linear-attention state, enabling effective prefix caching for models that combine full and linear attention. #1265 #1422

Multi-Level Cache and Cache-Aware PD

The cache hierarchy now spans GPU, CPU, and disk, with support for quantized CPU KV cache, FP8/INT8 KV formats, NUMA-aware CPU page placement, and faster cache initialization. PD disaggregation gains a cache-aware scheduling path, upgraded NIXL KV transfer, Qwen3.5 support, and stronger lifecycle, health-check, disconnect, and multimodal handling. #997 #1098 #1379 #1399 #1412

What's Changed

• v100 triton kernel fix by @hiworldwzj in #1040
• fix tl.where warning by @sufubao in #1041
• force to warmup triton autotune configs in start. by @sufubao in #1043
• fix mtp mem alloc in overlap manner by @shihaobai in #1044
• add stream_options for openai api by @shihaobai in #1046
• [fix]fix fp8 bug when load moe model by @SangChengC in #1045
• vit fa3 api fix by @shihaobai in #1047
• Use environment variable for RMSNORM_WARPS by @WANDY666 in #1027
• Fix the pause issue under extremely aggressive scheduling. by @shihaobai in #1057
• fix deepgemm set_num_sms import by @shihaobai in #1058
• add moe_align_fused by @shihaobai in #1054
• pd with nixl backend by @kingder in #1042
• Fix the incorrect logic when loading Mixtral series model weights. by @flyinglandlord in #1064
• merge q_a_proj and kv_a_proj to reduce the kernel launch overhead by @shihaobai in #1055
• Fix type hint in _create_paged_xfer_handles method by @WANDY666 in #1067
• Startup Optimization & Reliability: Shared Memory and Parallel Model Initialization. by @sufubao in #1051
• reformat pre cache kv by @hiworldwzj in #1066
• [add] add skip image cache and disable_prompt_cache para by @SangChengC in #1061
• [misc] Adding Generic Types for IDE by @sufubao in #1063
• [add]add whisper sdpa by @SangChengC in #1071
• Fix qwen3moe overlap mode by @sufubao in #1070
• Add Support For GPT-OSS Model by @flyinglandlord in #1023
• fix unit test for silu_and_mul kernel. by @hiworldwzj in #1075
• deepseek-MTP eagle, topk=1 by @shihaobai in #1073
• fix gpt_oss import by @shihaobai in #1077
• deepseek tpsp lora rank qkv all gather. by @hiworldwzj in #1078
• bugfix:qwen3 fa3 inferstruct init, add b_prefill_start_loc for init_req_to_token_indexes by @shihaobai in #1081
• fix qwen2vl image process by @SangChengC in #1082
• Cpu KV Cache feature by @hiworldwzj in #997
• Fix Tool Call API & Minor Change by @flyinglandlord in #1080
• fix api_cli & qwen25 parser by @shihaobai in #1085
• tpsp mode support db prefill balance. by @hiworldwzj in #1086
• add time log for debug. by @hiworldwzj in #1087
• Add Lab4AI material in README by @zhhangBian in #1088
• fix: minor updates and fixs for unit_tests to match current code by @WuSiYu in #1083
• static test fix by @shihaobai in #1089
• implement radix_cache node merge function by @Longxmas in #1090
• add profile_demo.py and add synchronize in infer_loop by @hiworldwzj in #1091
• bugfix: Fix precision issue with Triton operator token_att_fwd by @WuSiYu in #1092
• opti-qwen2-vl-pre-process by @SangChengC in #1094
• fix-audio-rpyc by @SangChengC in #1096
• fix grouped_topk tl.sort when numel=1 by @SangChengC in #1101
• moe triton kernel use tma. by @hiworldwzj in #1100
• Awq support and mm refactor by @shihaobai in #1084
• fix mm slicer by @hiworldwzj in #1104
• dockerfile for cuda12.8.0 by @hiworldwzj in #1106
• fix dockerfile by @hiworldwzj in #1107
• fix tool_call_parser deepseekv31 by @shihaobai in #1110
• add chat template by @shihaobai in #1111
• fix fc stream by @shihaobai in #1117
• add fp8_scaled_mm_per_token by @WANDY666 in #1112
• add visual_send_bs args by @SangChengC in #1109
• fix triton_rotary_rope_emb by @SangChengC in #1125
• [Feat] Add structured generation OpenAI API by @flyinglandlord in #1114
• diverse mode fast gen decode kernel. by @hiworldwzj in #1123
• fix pos_emb by @SangChengC in #1126
• support-whisper-longaudio by @SangChengC in #1128
• [bugfix]: qwen2_vl rope_type default by @shihaobai in #1129
• [bugfix]: deepgemm online quant by @shihaobai in #1130
• feat: disk cache v1.0 by @blueswhen in #1098
• 235B H20 configs by @shihaobai in #1132
• cpu kv cache support quanted kv. by @hiworldwzj in #1133
• fix prefill classed mode. by @hiworldwzj in #1134
• fix health req id gen when httpserver worker num > 1 by @hiworldwzj in #1137
• hash 128 and Update doc by @blueswhen in http...

LightLLM v1.1.0 Release!

🎉 Announcing LightLLM v1.1.0: More Efficient, More Powerful!

We are thrilled to introduce LightLLM v1.1.0, featuring major architectural and optimization upgrades for higher performance and broader applicability.

✨ Key Highlights

🚀 CPU-GPU Unified Folding Architecture

• Drastically reduces system-level CPU overhead

⚡ Deep Model Optimizations

• Enhanced support for DeepSeek and Qwen3-MoE
• Integration of DeepEP / DeepGEMM and fused MoE Triton optimizations
• New balanced DP request scheduler.
• Added support for MTP

⚙️ Autotuner for Triton Kernels

• Automatically tunes kernel operators used by the model at service startup

🏆 ACL Outstanding Award: Pre^3

🖼️ Improved Multimodal Inference

• Further optimizations for faster and more reliable multimodal model inference

📖 Learn More

More details can be found in the LightLLM v1.1.0 blog post.

v1.0.1

Highlights

• DeepSeek-R1 Multi-Node H100 Deployment Support
• FlashInfer Integration
• XGrammer Integration

What's Changed

• Benchclient by @shihaobai in #740
• fix pause reqs by @shihaobai in #741
• add RETURN_LIST for tgi_api by @shihaobai in #742
• fix: fix a precision bug in the context_flashattention by @blueswhen in #743
• Improve the accuracy of deepseekv3 by @hiworldwzj in #744
• deepseekv3 bmm noquant and fix moe gemm bug. by @hiworldwzj in #745
• Add Xgrammar Support by @flyinglandlord in #701
• fuse fp8 quant in kv copying and add flashinfer decode mla operator in the attention module by @blueswhen in #737
• fix: add flashinfer-python in the requirements.txt by @blueswhen in #749
• Fix tokens2 by @SangChengC in #748
• Fix Unit-test in PR: Add xgrammar by @flyinglandlord in #750
• add support for multinode tp by @shihaobai in #751

Full Changelog: v1.0.0...v1.0.1

LightLLM v1.0.0 Release!

New Features

• Cross-Process Request Object:

  • Retained and optimized the previous three-process architecture design.
  • Introduced a request object that can be accessed across processes, significantly reducing inter-process communication overhead.

• Folding of scheduling and model inference:

  • Implemented the folding of scheduling and model inference, significantly reducing communication overhead between the scheduler and modelrpc.

• CacheTensorManager:

  • New class to manage the allocation and release of Torch tensors within the framework.
  • Maximizes tensor sharing across layers at runtime and enhances memory sharing between different CUDA graphs.
  • On an 8x80GB H100 machine, using the DeepSeek-v2 model, LightLLM can run 200 CUDA graphs concurrently without out of memory (OOM).

• PD-Disaggregation Prototype

  • Dynamic registration of P and D nodes

• Fastest DeepSeek-R1 performance on H200

  • sglang==0.4.3, vllm==0.7.2, trtllm==0.17.0
  • num_clients = 100. The input length of the test data is 1024, and the output follows a Gaussian distribution with a mean of 128
  • 

For more details, stay tuned to our blog at https://www.light-ai.top/lightllm-blog/. Thanks to outstanding projects like vllm, sglang, and trtllm, LightLLM also leverages some of the high-performance quantization kernels from vllm. We hope to collaborate in driving the growth of the open-source community.