# Split user audio — investigation and plan

Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed
stream.

## What happens today

The daemon does not see per-user audio. The client mixes every remote user before the audio leaves
the browser.

| Step | Where | Detail |
| --- | --- | --- |
| Playback | Vesktop renderer | Discord mixes the remote tracks. Per-user volume is applied before the mix. |
| Routing | `poc/src/audio.ts` | `pactl move-sink-input` moves the client's playback streams into the `VeskslopCapture` sink. |
| Capture | `poc/src/capture.ts` | `ffmpeg` records `VeskslopCapture.monitor` as raw PCM. One stream, all users. |
| Split | `poc/src/vad.ts` | The VAD cuts the stream into utterances and writes one WAV each. |
| Text | `poc/src/stt.ts`, `poc/src/transcript.ts` | STT turns each WAV into text when a provider is configured. |
| Speaker | `poc/src/audio-service.ts` | `VOICE_SPEAKING` events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user. |

The core bridge (`src/renderer/puppet/actions.ts`) has no WebRTC hooks today. The only audio call
is `navigator.mediaDevices.enumerateDevices()` for the device list.

## Where per-user audio exists

Each remote user arrives in the renderer as a WebRTC `MediaStreamTrack`. Discord attaches the
tracks to an `RTCPeerConnection` and mixes them in an `AudioContext` graph. Per-user gain (the
volume sliders) is a node on each track's path. The OS output receives only the mix.

## Split options

| Option | How | Cost |
| --- | --- | --- |
| Hook `RTCPeerConnection` | Wrap `ontrack` and `addEventListener("track")` before Discord's code runs, then keep every remote track | Straightforward to hook, but the track carries no user ID |
| Hook `AudioContext` | Wrap `createMediaStreamSource` and `createGain`. Discord makes one source per user, so the graph shows the per-user split | Better mapping chance, but fragile against client updates |
| Map by event order | Pair the first chunk after a `VOICE_SPEAKING` event with that user | Works with heavy traffic, mislabels overlaps |
| Chromium capture | Use Electron audio capture of `webContents` | Gives the mix again, not per-user |

Recommended path: hook `RTCPeerConnection` for the tracks, and hook `AudioContext` only to map a
track to a user (the source node creation order matches the track order, and the per-user gain node
sits next to it). Keep the event-order pairing as the fallback for unknown tracks.

## Transport for per-user PCM

The daemon needs one PCM stream per user. Two candidate shapes:

1. **AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge.** A 20 ms frame is 640 bytes,
   about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge
   traffic, small renderer cost.
2. **MediaRecorder per track, Opus chunks every second.** Lower traffic and less renderer CPU.
   The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU.

Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon
already runs ffmpeg for resampling. Add a binary frame type later when the volume grows.

## Plan

| Step | Work |
| --- | --- |
| M1 | Renderer hook lists remote tracks (`voice.tracks` action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping. |
| M2 | One track taps to an `AudioWorklet`, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only. |
| M3 | Map track to user ID (`AudioContext` hook, or the `VOICE_SPEAKING` fallback). Per-user VAD and transcript rows with a real `speakerId`. |
| M4 | The `split user audio` setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback. |

## Setting

`preferences.audio.splitUserAudio` (boolean, default false), shown in Settings → Browser audio as
**split user audio**. When it is off, the mixed capture feeds STT as it does today. When it is on,
the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for
the browser monitor.

## Failure modes to handle

- Discord updates can break the hooks. Gate the feature on a capability probe (`voice.tracks`
  answers with a list) and fall back to the mixed stream.
- Tracks appear and disappear during a call. Handle `track-added` and `track-ended` events.
- Muted or deafened users can still produce a track with silence. The VAD must drop it.
- The audio thread must never block. Copy in the worklet, send outside it.
- The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as
  mixed.