feat: carry FaceTime Audio call audio through BlackHole into the voice pipeline - #5373
Merged
Conversation
…e pipeline (#5308) Phase 1 could dial and hang up; nothing could hear the call. This adds the audio path and the session that owns it. A new call-host page (/voice/call-host) runs in a browser tab on the Mac — device permissions and setSinkId need a real browser profile — and bridges the two virtual devices: it reads BlackHole 16ch (what FaceTime plays into), streams 16 kHz mono PCM to the server, and plays each reply back through BlackHole 2ch (what FaceTime hears as its microphone). Every failure is named rather than spun on: a browser missing required APIs lists all of them at once, a missing or misconfigured device is named specifically, an unlabeled device list is reported as a missing microphone permission rather than a missing driver, and a second tab is refused by both a Web Lock and the server's single host slot, so two tabs can never double-answer one call. A phone call has no push-to-talk, so the server decides where a turn ends: energy-based VAD with 700 ms of trailing silence and a 20-second ceiling, driven by sample count rather than a wall clock so a delayed socket flush endpoints exactly where a smooth stream would. Each utterance runs the existing pipeline unchanged — same persona, tools, confirm gate, and TTS settings — and speaking over a reply interrupts it like the widget's barge-in. The session treats the helper's view of the FaceTime window as the source of truth: a probe that fails is unknown, never a hangup. It ends on a remote hangup, 60 seconds of caller silence, the configured maximum call length, or the host tab going away, because a call nobody can hear is worse than no call. Only a text transcript is kept — appended to the daily journal labelled Caller and PortOS, with the configured handle nowhere in it. Setup now offers to install BlackHole through the user's own Homebrew (GPLv3, never bundled, declining is supported), and Check setup verifies each device's label, rate, and channel count instead of always reporting them missing. Closes #5308
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 (#5307) could dial and hang up; nothing could hear the call. This is the audio path and the session that owns it.
/voice/call-host, a browser tab on the Mac (device permissions andsetSinkIdneed a real browser profile). Reads BlackHole 16ch, streams 16 kHz mono PCM to the server, plays each reply back through BlackHole 2ch. Input-level meter and a one-second test tone confirm both directions.callEndpointing.js) — energy VAD, 700 ms trailing silence, 20 s ceiling, 200 ms pre-roll so the first consonant is not clipped. Driven by sample count, not a wall clock, so a delayed socket flush endpoints exactly where a smooth stream would (there is a test for that equivalence).callSession.js) —idle → dialing → connected/listening → speaking → ended, reconciled against the phase-1probe()every 2 s. A failed probe is unknown, never a hangup. Ends on remote hangup, 60 s of caller silence, the configuredmaxCallMinutes(default 15), or the host tab going away.runTurnwith the persona, tools, confirm gate, and TTS settings unchanged; only the audio is re-addressed (voice:tts:audio→voice:call:tts). Barge-in aborts the in-flight turn exactly as the widget does.Caller/PortOS. Call audio is never persisted and the configured handle never appears in the transcript or its metadata.npm run setup:facetimenow offersbrew install blackhole-2ch blackhole-16ch. BlackHole is GPLv3 and is never bundled; declining is supported (dial/hang up keep working).checkSetupnow actually verifies each device's label, rate, and channel count instead of hardcoding "missing", and distinguishes could-not-read from not-installed.NAV_COMMANDSentry (feature-gatedfacetime, route keeps working per the nav-gating convention), a Settings → Voice link to the host page, and amaxCallMinutesfield.Closes #5308. Unblocks #5309, #5310, #5311.
Test plan
server/services/voice/callEndpointing.test.js(new, 8): silence boundary held to the exact frame, pre-roll bounded on a 10 s idle line, 20 s cap, sub-minimum click dropped, identical endpointing for smooth vs. burst arrival, mid-sentence flush, idle line. Synthetic PCM, no sleeps.server/services/voice/callSession.test.js(new, 20): refuses to dial with no host; single host slot with reconnect allowed; reachesconnectedonly via probe; failed dial does not strand the session; probe failure ≠ hangup; remote hangup; silence timer measured from the last caller turn; configured ceiling plus fallback for absent/nonsense values; journal transcript content, the empty-call case, and a failed write still resetting; host detach ends the call while a non-owner's detach is ignored; listener broadcast and a throwing listener.server/services/voice/facetimeBridge.test.js(+4): device check passes/fails on rate and channel count, matches labels case-insensitively, and separates could-not-read from not-installed.client/src/lib/callAudioBridge.test.js(new, 21): capability probe per API and all-at-once; exact label matching (a"BlackHole 1"prefix must not resolve to the 16ch device); permission-vs-driver diagnosis; downmix, 48k→16k resample, clamped Int16 (wrapping would flip a hot sample's sign into a click), level meter, faded test tone.client/src/pages/VoiceCallHost.test.jsx(new, 8): the fail-closed paths end to end — missing APIs, lock held elsewhere, missing device, exactgetUserMediaconstraints with all processing off, serverhost-taken, live state rendering, detach on unmount.server(35,473) andclient(10,250) suites pass; biome lint clean; client build succeeds; API-route and socket-event catalogs regenerated.Real-device validation (a live FaceTime call with BlackHole installed) is a manual step on macOS, per the epic's test-plan note.