fix(ios): enable voice-processing I/O (echo cancellation) for voiceChat/videoChat session modes - #1216
Closed
felixlabsco wants to merge 1 commit into
Closed
Conversation
… modes AVAudioSession mode voiceChat configures routing and EQ but does not enable echo cancellation — that requires setVoiceProcessingEnabled on the engine's input node. Full-duplex apps (VoIP, realtime voice agents) currently get raw speaker echo in the mic stream, so server-side VAD self-triggers and agents interrupt their own playback. Enable Apple's voice-processing I/O (AEC + noise suppression + AGC) on the system input node whenever the desired session mode is voiceChat or videoChat, before the input connection format is read (voice processing changes the hardware format) and only while the engine is stopped. Modes are synced both ways so leaving voice-chat mode disables it again.
Member
|
Looks solid, need probably a one or two days to wrap my head around it and decide on the approach, but its a good baseline |
Member
|
I applied your approach on top of the #1210, because it was closer to my vision. Thank you for the contribution nevertheless, your work made my work much easier. |
Author
|
Glad to hear it! Love the project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Setting
AudioManager.setAudioSessionOptions({ iosMode: 'voiceChat', ... })configures AVAudioSession routing and EQ, but does not enable echo cancellation — that requiressetVoiceProcessingEnabled:on the engine's input node, which the library never calls.Consequence for any full-duplex app (simultaneous
AudioRecordercapture + playback, e.g. VoIP or realtime voice-agent clients): speaker output loops straight back into the microphone stream. Downstream VAD (local or server-side, e.g. speech-to-speech APIs with barge-in) detects the app's own playback as user speech, so the agent interrupts itself — in my testing this manifested as playback chopped into ~1s fragments as the remote VAD repeatedly fired on the echo. Developers settingvoiceChatmode almost universally expect AEC to be part of the deal, which makes this a silent trap.Fix
When the input node is materialized, sync Apple's voice-processing I/O (AEC + noise suppression + AGC) with the desired session mode: enabled for
voiceChat/videoChat, disabled otherwise. No public API change.Ordering constraints handled:
currentInputConnectionFormatis read — enabling voice processing changes the hardware format;setVoiceProcessingEnabled:fails on a running engine);Testing
Verified on device (iPhone, iOS 26): with
iosCategory: 'playAndRecord',iosMode: 'voiceChat', streaming 16 kHz mic PCM viaAudioRecorder.onAudioReadywhile playing 24 kHz PCM through anAudioBufferQueueSourceNodeon the built-in speaker at full volume:Known inherent side effect of Apple's voice processing (documented behavior, not introduced by this PR beyond enabling the unit): slight output level reduction on the speaker route.
I first shipped this as a
pnpm patchagainst 0.13.2 (enable-only variant) and it's been solid; this PR is the upstreamed, mode-synced version. Happy to rework it as an explicit opt-in option (e.g. aSessionOptionsflag) if you'd rather not tie it to the session mode — mode-implied seemed like the least-surprise default sincevoiceChatwithout AEC is rarely what anyone wants.