Skip to content

fix(fun-asr-nano): stabilize fp16 vLLM decoding - #3597

Open
LauraGPT wants to merge 1 commit into
mainfrom
codex/fix-vllm-fp16-3595
Open

fix(fun-asr-nano): stabilize fp16 vLLM decoding#3597
LauraGPT wants to merge 1 commit into
mainfrom
codex/fix-vllm-fp16-3595

Conversation

@LauraGPT

Copy link
Copy Markdown
Collaborator

Summary

  • keep Fun-ASR-Nano audio encoder/adaptor compute in float16 when users request dtype="fp16"
  • run the Qwen3 vLLM decoder in bfloat16, avoiding the reproducible repetitive/garbage transcript without increasing decoder weight memory
  • share the same dtype resolution in offline and streaming vLLM constructors
  • replace the old warning that attributed the failure to the audio path with an accurate compatibility warning

Root cause and real regression

On an H100 with FunASR 1.4.11, vLLM 0.10.2, and PyTorch 2.8.0+cu128, the public sample consistently returned a long string of ! with both audio and Qwen3 in fp16. Split tests isolated the failure to the Qwen3 decoder: audio fp16 + Qwen3 bf16 returned 欢迎您来体验达摩院推出的语音识别模型。, while audio bf16 + Qwen3 fp16 still returned garbage.

The exact reported vLLM 0.27.1 / PyTorch 2.13.0 stack requires CUDA 13, while the available H100 host driver supports CUDA 12.8, so that binary could not initialize locally. The fix was therefore validated on the compatible H100 stack above and covered by constructor-level regression tests for both paths.

Verification

  • 53 related unit tests passed
  • python -m py_compile passed for both implementations and the new test
  • git diff --check passed
  • H100 real-audio confirmation on exact head 881c396a2632c0b4336b5041de7197f9526e0951:
    • requested: fp16
    • audio dtype: torch.float16
    • vLLM dtype: torch.bfloat16
    • final: 欢迎您来体验达摩院推出的语音识别模型。

Addresses #3595. Please keep the issue open until the reporter retests the merged/released build.

Signed-off-by: LauraGPT 18321252+LauraGPT@users.noreply.github.com

Signed-off-by: LauraGPT <18321252+LauraGPT@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant