Veskslop-t/docs/voice-audio-split.md

Split user audio — investigation and plan

Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed stream.

What happens today

The daemon does not see per-user audio. The client mixes every remote user before the audio leaves the browser.

Step	Where	Detail
Playback	Vesktop renderer	Discord mixes the remote tracks. Per-user volume is applied before the mix.
Routing	poc/src/audio.ts	pactl move-sink-input moves the client's playback streams into the VeskslopCapture sink.
Capture	poc/src/capture.ts	ffmpeg records VeskslopCapture.monitor as raw PCM. One stream, all users.
Split	poc/src/vad.ts	The VAD cuts the stream into utterances and writes one WAV each.
Text	poc/src/stt.ts, poc/src/transcript.ts	STT turns each WAV into text when a provider is configured.
Speaker	poc/src/audio-service.ts	VOICE_SPEAKING events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user.

The core bridge (src/renderer/puppet/actions.ts) has no WebRTC hooks today. The only audio call is navigator.mediaDevices.enumerateDevices() for the device list.

Where per-user audio exists

Each remote user arrives in the renderer as a WebRTC MediaStreamTrack. Discord attaches the tracks to an RTCPeerConnection and mixes them in an AudioContext graph. Per-user gain (the volume sliders) is a node on each track's path. The OS output receives only the mix.

Split options

Option	How	Cost
Hook RTCPeerConnection	Wrap ontrack and addEventListener("track") before Discord's code runs, then keep every remote track	Straightforward to hook, but the track carries no user ID
Hook AudioContext	Wrap createMediaStreamSource and createGain. Discord makes one source per user, so the graph shows the per-user split	Better mapping chance, but fragile against client updates
Map by event order	Pair the first chunk after a VOICE_SPEAKING event with that user	Works with heavy traffic, mislabels overlaps
Chromium capture	Use Electron audio capture of webContents	Gives the mix again, not per-user

Recommended path: hook RTCPeerConnection for the tracks, and hook AudioContext only to map a track to a user (the source node creation order matches the track order, and the per-user gain node sits next to it). Keep the event-order pairing as the fallback for unknown tracks.

Transport for per-user PCM

The daemon needs one PCM stream per user. Two candidate shapes:

1. AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge. A 20 ms frame is 640 bytes, about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge traffic, small renderer cost.
2. MediaRecorder per track, Opus chunks every second. Lower traffic and less renderer CPU. The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU.

Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon already runs ffmpeg for resampling. Add a binary frame type later when the volume grows.

Plan

Step	Work
M1	Renderer hook lists remote tracks (voice.tracks action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping.
M2	One track taps to an AudioWorklet, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only.
M3	Map track to user ID (AudioContext hook, or the VOICE_SPEAKING fallback). Per-user VAD and transcript rows with a real speakerId.
M4	The split user audio setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback.

Setting

preferences.audio.splitUserAudio (boolean, default false), shown in Settings → Browser audio as split user audio. When it is off, the mixed capture feeds STT as it does today. When it is on, the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for the browser monitor.

Failure modes to handle

• Discord updates can break the hooks. Gate the feature on a capability probe (voice.tracks answers with a list) and fall back to the mixed stream.
• Tracks appear and disappear during a call. Handle track-added and track-ended events.
• Muted or deafened users can still produce a track with silence. The VAD must drop it.
• The audio thread must never block. Copy in the worklet, send outside it.
• The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as mixed.