From caf5e166bb83ae3a206fdffe9454e0e2b0fd4bda Mon Sep 17 00:00:00 2001 From: "Alan Ding (Google)" Date: Wed, 12 Aug 2026 12:58:04 -0700 Subject: [PATCH 1/4] Update audioStartTime and audioEndTime to be relative to start of audio stream We want to switch to stream-Relative (0-based) impl. since: - In Web Audio and Media APIs (HTMLMediaElement.currentTime, AudioContext.currentTime, WebCodecs VideoFrame.timestamp), media timelines are always 0-based offsets relative to stream start, not the time origin. - Immune to inter-process jitter since SODA and audio capture run in a separate utility/browser process. Translating stream offsets to the renderer's timeOrigin relies on estimating when IPC AudioStarted() arrived, which introduces IPC latency jitter. Stream-relative offsets are not prune to this and aligns with the raw audio frames. - If SpeechRecognition is used with a pre-recorded MediaStreamTrack, a 0-based stream offset reflects the actual position in the audio track regardless of when the webpage was loaded. - Being relative to performance.timeOrigin doesn't make sense in general for the Web Speech API because it assumes that the audio source is live. Since a SpeechRecognizer can also be created for a prerecorded media stream I think the timestamps on the speech recognition events should be relative to the position in that media stream. See https://crbug.com/542330168 for more details. --- explainers/speech-recognition-result-timestamps.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index c6ccd3b..8e85717 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -17,10 +17,10 @@ We propose extending the `SpeechRecognitionResult` interface to include optional ```webidl partial interface SpeechRecognitionResult { - // Start timestamp of the audio segment in milliseconds (relative to time origin) + // Start timestamp of the audio segment in milliseconds (relative to the start of the audio stream) readonly attribute DOMHighResTimeStamp? audioStartTime; - // End timestamp of the audio segment in milliseconds (relative to time origin) + // End timestamp of the audio segment in milliseconds (relative to the start of the audio stream) readonly attribute DOMHighResTimeStamp? audioEndTime; }; ``` From df41ad5c47bcae2d46f98d9a06b63491d88f4203 Mon Sep 17 00:00:00 2001 From: "Alan Ding (Google)" Date: Wed, 12 Aug 2026 13:33:40 -0700 Subject: [PATCH 2/4] Update speech recognition explainer with timestamp conversion Added a section on converting stream timestamps to document time origin and provided a live translation latency example with code. --- .../speech-recognition-result-timestamps.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index 8e85717..c06978f 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -54,6 +54,63 @@ recognition.onresult = (event) => { recognition.start(); ``` +## Converting Stream Timestamps to Document Time Origin + +`audioStartTime` and `audioEndTime` are defined as media-local offsets in milliseconds relative to the start of the audio stream ($t = 0.0\text{ms}$). + +For real-time applications such as **live translation**, **subtitling overlays**, and **audio-visual sync**, developers often need to map these stream offsets to the document's global timeline (`DOMHighResTimeStamp` / `performance.now()`). + +### Pattern: Capturing the Audio Timeline Origin + +To convert stream-relative timestamps to document time coordinates: +1. Record the baseline timestamp when the `audiostart` event fires (`event.timeStamp` is a `DOMHighResTimeStamp` relative to `timeOrigin`). +2. Add the result's `audioStartTime` and `audioEndTime` offsets to that baseline. + +$$\text{absoluteStartTime} = \text{audioOrigin} + \text{result.audioStartTime}$$ +$$\text{absoluteEndTime} = \text{audioOrigin} + \text{result.audioEndTime}$$ + +--- + +### Measuring Live Translation Latency Example + +In live speech translation workflows, measuring both **Speech-to-Text (STT) latency** and **Machine Translation (MT) end-to-end latency** is essential: + +```javascript +const recognition = new SpeechRecognition(); +recognition.continuous = true; +recognition.interimResults = true; + +let audioOriginTime = 0; + +// 1. Capture the audio stream's time origin on the document timeline +recognition.onaudiostart = (event) => { + audioOriginTime = event.timeStamp; +}; + +recognition.onresult = async (event) => { + const result = event.results[event.resultIndex]; + if (result.audioEndTime === null) return; + + // 2. Convert stream offsets to document time origin coordinates + const absoluteAudioStart = audioOriginTime + result.audioStartTime; + const absoluteAudioEnd = audioOriginTime + result.audioEndTime; + + // 3. Compute ASR recognition latency + const asrLatencyMs = event.timeStamp - absoluteAudioEnd; + + // 4. Perform live translation + const text = result[0].transcript; + const translationStartTime = performance.now(); + const translatedText = await translateService.translate(text, 'es'); + const translationEndTime = performance.now(); + + // 5. Total end-to-end latency from speaker utterance to translated subtitle + const totalE2ELatencyMs = translationEndTime - absoluteAudioEnd; + + console.log(`ASR Processing Time: ${asrLatencyMs.toFixed(1)}ms`); + console.log(`Total Live Translation Delay: ${totalE2ELatencyMs.toFixed(1)}ms`); +}; + ### Security and Privacy Considerations #### Fingerprinting Risk From e15f76585292c286e30a53a2505d6f29da623a24 Mon Sep 17 00:00:00 2001 From: "Alan Ding (Google)" Date: Wed, 12 Aug 2026 15:07:05 -0700 Subject: [PATCH 3/4] Minor update to close live transcription latency measurement example Added separator before security section and closed example section for live transcription measurements. --- explainers/speech-recognition-result-timestamps.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index c06978f..0ebe57c 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -69,8 +69,6 @@ To convert stream-relative timestamps to document time coordinates: $$\text{absoluteStartTime} = \text{audioOrigin} + \text{result.audioStartTime}$$ $$\text{absoluteEndTime} = \text{audioOrigin} + \text{result.audioEndTime}$$ ---- - ### Measuring Live Translation Latency Example In live speech translation workflows, measuring both **Speech-to-Text (STT) latency** and **Machine Translation (MT) end-to-end latency** is essential: @@ -110,7 +108,8 @@ recognition.onresult = async (event) => { console.log(`ASR Processing Time: ${asrLatencyMs.toFixed(1)}ms`); console.log(`Total Live Translation Delay: ${totalE2ELatencyMs.toFixed(1)}ms`); }; - +``` +--- ### Security and Privacy Considerations #### Fingerprinting Risk From 95da8db47f179ef2db12008c03e24f59e72d615e Mon Sep 17 00:00:00 2001 From: "Alan Ding (Google)" Date: Wed, 12 Aug 2026 16:54:35 -0700 Subject: [PATCH 4/4] respond to review comment --- explainers/speech-recognition-result-timestamps.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/explainers/speech-recognition-result-timestamps.md b/explainers/speech-recognition-result-timestamps.md index 0ebe57c..0200cc5 100644 --- a/explainers/speech-recognition-result-timestamps.md +++ b/explainers/speech-recognition-result-timestamps.md @@ -39,7 +39,7 @@ recognition.interimResults = true; recognition.onresult = (event) => { const result = event.results[event.resultIndex]; - if (result.audioEndTime !== null) { + if (result.audioEndTime !== null && result.audioEndTime !== undefined) { // Calculate on-device processing latency const processingLatencyMs = event.timeStamp - result.audioEndTime; @@ -87,7 +87,7 @@ recognition.onaudiostart = (event) => { recognition.onresult = async (event) => { const result = event.results[event.resultIndex]; - if (result.audioEndTime === null) return; + if (result.audioEndTime === null || result.audioEndTime === undefined) return; // 2. Convert stream offsets to document time origin coordinates const absoluteAudioStart = audioOriginTime + result.audioStartTime;