Skip to content

fix(ios): enable voice-processing I/O (echo cancellation) for voiceChat/videoChat session modes - #1216

Closed
felixlabsco wants to merge 1 commit into
software-mansion:mainfrom
felixlabsco:ios-voice-processing-for-voicechat-mode
Closed

fix(ios): enable voice-processing I/O (echo cancellation) for voiceChat/videoChat session modes#1216
felixlabsco wants to merge 1 commit into
software-mansion:mainfrom
felixlabsco:ios-voice-processing-for-voicechat-mode

Conversation

@felixlabsco

Copy link
Copy Markdown

Problem

Setting AudioManager.setAudioSessionOptions({ iosMode: 'voiceChat', ... }) configures AVAudioSession routing and EQ, but does not enable echo cancellation — that requires setVoiceProcessingEnabled: on the engine's input node, which the library never calls.

Consequence for any full-duplex app (simultaneous AudioRecorder capture + playback, e.g. VoIP or realtime voice-agent clients): speaker output loops straight back into the microphone stream. Downstream VAD (local or server-side, e.g. speech-to-speech APIs with barge-in) detects the app's own playback as user speech, so the agent interrupts itself — in my testing this manifested as playback chopped into ~1s fragments as the remote VAD repeatedly fired on the echo. Developers setting voiceChat mode almost universally expect AEC to be part of the deal, which makes this a silent trap.

Fix

When the input node is materialized, sync Apple's voice-processing I/O (AEC + noise suppression + AGC) with the desired session mode: enabled for voiceChat/videoChat, disabled otherwise. No public API change.

Ordering constraints handled:

  • runs before currentInputConnectionFormat is read — enabling voice processing changes the hardware format;
  • only toggles while the engine is stopped (setVoiceProcessingEnabled: fails on a running engine);
  • no-ops when the state already matches, and syncs both ways so leaving voice-chat mode restores the raw path.

Testing

Verified on device (iPhone, iOS 26): with iosCategory: 'playAndRecord', iosMode: 'voiceChat', streaming 16 kHz mic PCM via AudioRecorder.onAudioReady while playing 24 kHz PCM through an AudioBufferQueueSourceNode on the built-in speaker at full volume:

  • before: remote VAD fired continuously on echo; playback self-interrupted every ~1s;
  • after: zero echo-triggered VAD events over multi-minute sessions; deliberate talk-over (barge-in) still detected correctly.

Known inherent side effect of Apple's voice processing (documented behavior, not introduced by this PR beyond enabling the unit): slight output level reduction on the speaker route.

I first shipped this as a pnpm patch against 0.13.2 (enable-only variant) and it's been solid; this PR is the upstreamed, mode-synced version. Happy to rework it as an explicit opt-in option (e.g. a SessionOptions flag) if you'd rather not tie it to the session mode — mode-implied seemed like the least-surprise default since voiceChat without AEC is rarely what anyone wants.

… modes

AVAudioSession mode voiceChat configures routing and EQ but does not enable
echo cancellation — that requires setVoiceProcessingEnabled on the engine's
input node. Full-duplex apps (VoIP, realtime voice agents) currently get raw
speaker echo in the mic stream, so server-side VAD self-triggers and agents
interrupt their own playback.

Enable Apple's voice-processing I/O (AEC + noise suppression + AGC) on the
system input node whenever the desired session mode is voiceChat or videoChat,
before the input connection format is read (voice processing changes the
hardware format) and only while the engine is stopped. Modes are synced both
ways so leaving voice-chat mode disables it again.
@mdydek

mdydek commented Aug 11, 2026

Copy link
Copy Markdown
Member

Looks solid, need probably a one or two days to wrap my head around it and decide on the approach, but its a good baseline

@mdydek

mdydek commented Aug 12, 2026

Copy link
Copy Markdown
Member

I applied your approach on top of the #1210, because it was closer to my vision. Thank you for the contribution nevertheless, your work made my work much easier.

@mdydek mdydek closed this Aug 12, 2026
@felixlabsco

Copy link
Copy Markdown
Author

Glad to hear it! Love the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants