ASR audio tests - #112
Draft
barvhaim wants to merge 5 commits into
Draft
Conversation
barvhaim
force-pushed
the
feature/47-audio-tests
branch
from
July 27, 2026 14:10
46b71ec to
bbfda40
Compare
…lips, clip ceiling Signed-off-by: BAR HAIM <barha@il.ibm.com>
…/x3) CI-runnable serving smoke closing the #47 e2e box. Boots a real composed, audio-enabled GraniteSwitch checkpoint under vLLM and drives text-only, adapter-control-token, and audio x1/x2/x3 requests through one live engine. Markers slow+requires_model+gpu (opt-in via -m); model built through the compose CLI with --enable-audio; synthetic audio keeps it asset-free. Verified green on an A100 with ibm-granite/granite-4.1-3b + granitelib-core-r1.0: 5 passed. Signed-off-by: BAR HAIM <barha@il.ibm.com>
barvhaim
force-pushed
the
feature/47-audio-tests
branch
from
July 27, 2026 14:11
bbfda40 to
51b9be6
Compare
Adds the first correctness test for the "adapters + audio together" #47 box: the RAG answerability adapter judging a query against a document supplied as speech rather than text. The document is delivered as audio by placing a single <|audio|> marker as the document text (documents=[{"text": "<|audio|>"}]); the Granite template renders each document via `doc | tojson`, so the marker lands in the <documents> block exactly where vLLM's ASR processor splices the transcript — the document the adapter reasons over is the transcribed speech. The <|answerability|> control token is inserted by the same template (aLoRA fallback path), so the switch fires the adapter at the generation prompt. Ground truth comes from one real clip (hf-internal-testing/ librispeech_asr_dummy row 0), driving both classes without shipping a fixture: an answerable question whose answer is spoken, and an unanswerable one about content the clip never mentions. Both are distinctive enough to survive ASR word-errors, so the assertion tests the answerability decision, not transcription accuracy (WER is a separate box). FLAC is decoded with soundfile from the raw bytes to avoid requiring torchcodec (datasets>=4). Where test_audio_serving_smoke.py only proves an adapter control token doesn't crash the audio path, this proves the switch produces the correct verdict end-to-end through a live vLLM engine. Verified on an A100 with granite-4.1-3b + granitelib-rag-r1.0: 2 passed. Markers slow+requires_model+gpu; opt-in via pytest -m "slow and requires_model and gpu" Signed-off-by: BAR HAIM <barha@il.ibm.com>
Replace the runtime hf-internal-testing/librispeech_asr_dummy download with a committed 3s 16 kHz WAV (SpeechT5, MIT-licensed) so the test runs offline and carries clean provenance for merge-to-main. The spoken sentence states a location that distil-whisper transcribes cleanly, so the questions key on the location rather than the proper noun. Verdicts verified: 2 passed on A100 (granite-4.1-3b + granitelib-rag-r1.0). Signed-off-by: BAR HAIM <barha@il.ibm.com>
Adds the 'all default adapters route correctly with audio enabled' coverage for issue #47. Composes the full default set (RAG + Core + Guardian) with --enable-audio and asserts the switch maps each adapter_token_ids[i] to index i+1 (pre-control positions stay on base 0), observed via the HF backend's _last_adapter_indices. For granite-4.1-3b, 12 adapters resolve (context_relevance has no 4.1-3b flavor); verified 12/12 route correctly on A100. Loads with ignore_mismatched_sizes=True: the <|audio|> token bumps vocab_size one past the switch's control_to_substitute_lut buffer, which SingleSwitch rebuilds from config, so the freshly-built buffer is kept. Signed-off-by: BAR HAIM <barha@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.