Skip to content

feat: carry FaceTime Audio call audio through BlackHole into the voice pipeline - #5373

Merged
atomantic merged 2 commits into
mainfrom
next/issue-5308
Aug 29, 2026
Merged

feat: carry FaceTime Audio call audio through BlackHole into the voice pipeline#5373
atomantic merged 2 commits into
mainfrom
next/issue-5308

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

Phase 1 (#5307) could dial and hang up; nothing could hear the call. This is the audio path and the session that owns it.

  • Call host page/voice/call-host, a browser tab on the Mac (device permissions and setSinkId need a real browser profile). Reads BlackHole 16ch, streams 16 kHz mono PCM to the server, plays each reply back through BlackHole 2ch. Input-level meter and a one-second test tone confirm both directions.
  • Fails closed and says why. A browser missing required APIs lists all of them at once; a missing or misconfigured device is named specifically; an unlabeled device list is reported as a missing microphone permission rather than a missing driver (an ungranted tab enumerates unlabeled devices — "reinstall BlackHole" would send the user to fix the wrong thing); a second tab is refused by a Web Lock and independently by the server's single host slot.
  • Endpointing (callEndpointing.js) — energy VAD, 700 ms trailing silence, 20 s ceiling, 200 ms pre-roll so the first consonant is not clipped. Driven by sample count, not a wall clock, so a delayed socket flush endpoints exactly where a smooth stream would (there is a test for that equivalence).
  • Session (callSession.js) — idle → dialing → connected/listening → speaking → ended, reconciled against the phase-1 probe() every 2 s. A failed probe is unknown, never a hangup. Ends on remote hangup, 60 s of caller silence, the configured maxCallMinutes (default 15), or the host tab going away.
  • Pipeline reuse — each utterance runs the existing runTurn with the persona, tools, confirm gate, and TTS settings unchanged; only the audio is re-addressed (voice:tts:audiovoice:call:tts). Barge-in aborts the in-flight turn exactly as the widget does.
  • Privacy — only a text transcript is kept, appended to the daily journal labelled Caller/PortOS. Call audio is never persisted and the configured handle never appears in the transcript or its metadata.
  • Setupnpm run setup:facetime now offers brew install blackhole-2ch blackhole-16ch. BlackHole is GPLv3 and is never bundled; declining is supported (dial/hang up keep working). checkSetup now actually verifies each device's label, rate, and channel count instead of hardcoding "missing", and distinguishes could-not-read from not-installed.
  • Nav/settingsNAV_COMMANDS entry (feature-gated facetime, route keeps working per the nav-gating convention), a Settings → Voice link to the host page, and a maxCallMinutes field.

Closes #5308. Unblocks #5309, #5310, #5311.

Test plan

  • server/services/voice/callEndpointing.test.js (new, 8): silence boundary held to the exact frame, pre-roll bounded on a 10 s idle line, 20 s cap, sub-minimum click dropped, identical endpointing for smooth vs. burst arrival, mid-sentence flush, idle line. Synthetic PCM, no sleeps.
  • server/services/voice/callSession.test.js (new, 20): refuses to dial with no host; single host slot with reconnect allowed; reaches connected only via probe; failed dial does not strand the session; probe failure ≠ hangup; remote hangup; silence timer measured from the last caller turn; configured ceiling plus fallback for absent/nonsense values; journal transcript content, the empty-call case, and a failed write still resetting; host detach ends the call while a non-owner's detach is ignored; listener broadcast and a throwing listener.
  • server/services/voice/facetimeBridge.test.js (+4): device check passes/fails on rate and channel count, matches labels case-insensitively, and separates could-not-read from not-installed.
  • client/src/lib/callAudioBridge.test.js (new, 21): capability probe per API and all-at-once; exact label matching (a "BlackHole 1" prefix must not resolve to the 16ch device); permission-vs-driver diagnosis; downmix, 48k→16k resample, clamped Int16 (wrapping would flip a hot sample's sign into a click), level meter, faded test tone.
  • client/src/pages/VoiceCallHost.test.jsx (new, 8): the fail-closed paths end to end — missing APIs, lock held elsewhere, missing device, exact getUserMedia constraints with all processing off, server host-taken, live state rendering, detach on unmount.
  • Full server (35,473) and client (10,250) suites pass; biome lint clean; client build succeeds; API-route and socket-event catalogs regenerated.

Real-device validation (a live FaceTime call with BlackHole installed) is a manual step on macOS, per the epic's test-plan note.

…e pipeline (#5308)

Phase 1 could dial and hang up; nothing could hear the call. This adds the
audio path and the session that owns it.

A new call-host page (/voice/call-host) runs in a browser tab on the Mac —
device permissions and setSinkId need a real browser profile — and bridges the
two virtual devices: it reads BlackHole 16ch (what FaceTime plays into),
streams 16 kHz mono PCM to the server, and plays each reply back through
BlackHole 2ch (what FaceTime hears as its microphone). Every failure is named
rather than spun on: a browser missing required APIs lists all of them at once,
a missing or misconfigured device is named specifically, an unlabeled device
list is reported as a missing microphone permission rather than a missing
driver, and a second tab is refused by both a Web Lock and the server's single
host slot, so two tabs can never double-answer one call.

A phone call has no push-to-talk, so the server decides where a turn ends:
energy-based VAD with 700 ms of trailing silence and a 20-second ceiling,
driven by sample count rather than a wall clock so a delayed socket flush
endpoints exactly where a smooth stream would. Each utterance runs the existing
pipeline unchanged — same persona, tools, confirm gate, and TTS settings — and
speaking over a reply interrupts it like the widget's barge-in.

The session treats the helper's view of the FaceTime window as the source of
truth: a probe that fails is unknown, never a hangup. It ends on a remote
hangup, 60 seconds of caller silence, the configured maximum call length, or
the host tab going away, because a call nobody can hear is worse than no call.
Only a text transcript is kept — appended to the daily journal labelled Caller
and PortOS, with the configured handle nowhere in it.

Setup now offers to install BlackHole through the user's own Homebrew (GPLv3,
never bundled, declining is supported), and Check setup verifies each device's
label, rate, and channel count instead of always reporting them missing.

Closes #5308
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[repo-study-facetime-bridge-audio-bridge] FaceTime Audio phase 2: BlackHole audio bridge + call-host page wired into the voice pipeline

1 participant