80 lines
4.6 KiB
Markdown
80 lines
4.6 KiB
Markdown
# Split user audio — investigation and plan
|
|
|
|
Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed
|
|
stream.
|
|
|
|
## What happens today
|
|
|
|
The daemon does not see per-user audio. The client mixes every remote user before the audio leaves
|
|
the browser.
|
|
|
|
| Step | Where | Detail |
|
|
| --- | --- | --- |
|
|
| Playback | Vesktop renderer | Discord mixes the remote tracks. Per-user volume is applied before the mix. |
|
|
| Routing | `poc/src/audio.ts` | `pactl move-sink-input` moves the client's playback streams into the `VeskslopCapture` sink. |
|
|
| Capture | `poc/src/capture.ts` | `ffmpeg` records `VeskslopCapture.monitor` as raw PCM. One stream, all users. |
|
|
| Split | `poc/src/vad.ts` | The VAD cuts the stream into utterances and writes one WAV each. |
|
|
| Text | `poc/src/stt.ts`, `poc/src/transcript.ts` | STT turns each WAV into text when a provider is configured. |
|
|
| Speaker | `poc/src/audio-service.ts` | `VOICE_SPEAKING` events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user. |
|
|
|
|
The core bridge (`src/renderer/puppet/actions.ts`) has no WebRTC hooks today. The only audio call
|
|
is `navigator.mediaDevices.enumerateDevices()` for the device list.
|
|
|
|
## Where per-user audio exists
|
|
|
|
Each remote user arrives in the renderer as a WebRTC `MediaStreamTrack`. Discord attaches the
|
|
tracks to an `RTCPeerConnection` and mixes them in an `AudioContext` graph. Per-user gain (the
|
|
volume sliders) is a node on each track's path. The OS output receives only the mix.
|
|
|
|
## Split options
|
|
|
|
| Option | How | Cost |
|
|
| --- | --- | --- |
|
|
| Hook `RTCPeerConnection` | Wrap `ontrack` and `addEventListener("track")` before Discord's code runs, then keep every remote track | Straightforward to hook, but the track carries no user ID |
|
|
| Hook `AudioContext` | Wrap `createMediaStreamSource` and `createGain`. Discord makes one source per user, so the graph shows the per-user split | Better mapping chance, but fragile against client updates |
|
|
| Map by event order | Pair the first chunk after a `VOICE_SPEAKING` event with that user | Works with heavy traffic, mislabels overlaps |
|
|
| Chromium capture | Use Electron audio capture of `webContents` | Gives the mix again, not per-user |
|
|
|
|
Recommended path: hook `RTCPeerConnection` for the tracks, and hook `AudioContext` only to map a
|
|
track to a user (the source node creation order matches the track order, and the per-user gain node
|
|
sits next to it). Keep the event-order pairing as the fallback for unknown tracks.
|
|
|
|
## Transport for per-user PCM
|
|
|
|
The daemon needs one PCM stream per user. Two candidate shapes:
|
|
|
|
1. **AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge.** A 20 ms frame is 640 bytes,
|
|
about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge
|
|
traffic, small renderer cost.
|
|
2. **MediaRecorder per track, Opus chunks every second.** Lower traffic and less renderer CPU.
|
|
The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU.
|
|
|
|
Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon
|
|
already runs ffmpeg for resampling. Add a binary frame type later when the volume grows.
|
|
|
|
## Plan
|
|
|
|
| Step | Work |
|
|
| --- | --- |
|
|
| M1 | Renderer hook lists remote tracks (`voice.tracks` action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping. |
|
|
| M2 | One track taps to an `AudioWorklet`, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only. |
|
|
| M3 | Map track to user ID (`AudioContext` hook, or the `VOICE_SPEAKING` fallback). Per-user VAD and transcript rows with a real `speakerId`. |
|
|
| M4 | The `split user audio` setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback. |
|
|
|
|
## Setting
|
|
|
|
`preferences.audio.splitUserAudio` (boolean, default false), shown in Settings → Browser audio as
|
|
**split user audio**. When it is off, the mixed capture feeds STT as it does today. When it is on,
|
|
the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for
|
|
the browser monitor.
|
|
|
|
## Failure modes to handle
|
|
|
|
- Discord updates can break the hooks. Gate the feature on a capability probe (`voice.tracks`
|
|
answers with a list) and fall back to the mixed stream.
|
|
- Tracks appear and disappear during a call. Handle `track-added` and `track-ended` events.
|
|
- Muted or deafened users can still produce a track with silence. The VAD must drop it.
|
|
- The audio thread must never block. Copy in the worklet, send outside it.
|
|
- The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as
|
|
mixed.
|