Add TTS volume, a browser preview, and the split-audio plan
This commit is contained in:
Fisto
@@ -11,6 +11,7 @@ This file lists the project documents.
|
||||
| [daemon.md](daemon.md) | The external MCP daemon |
|
||||
| [webui.md](webui.md) | The web control UI and its JSON API |
|
||||
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
|
||||
| [voice-audio-split.md](voice-audio-split.md) | Per-user audio investigation and the split-user STT plan |
|
||||
| [media.md](media.md) | Media archive: command, storage, limits, and Web UI |
|
||||
| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema |
|
||||
| [media-mcp.md](media-mcp.md) | The media MCP tool surface |
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# Split user audio — investigation and plan
|
||||
|
||||
Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed
|
||||
stream.
|
||||
|
||||
## What happens today
|
||||
|
||||
The daemon does not see per-user audio. The client mixes every remote user before the audio leaves
|
||||
the browser.
|
||||
|
||||
| Step | Where | Detail |
|
||||
| --- | --- | --- |
|
||||
| Playback | Vesktop renderer | Discord mixes the remote tracks. Per-user volume is applied before the mix. |
|
||||
| Routing | `poc/src/audio.ts` | `pactl move-sink-input` moves the client's playback streams into the `VeskslopCapture` sink. |
|
||||
| Capture | `poc/src/capture.ts` | `ffmpeg` records `VeskslopCapture.monitor` as raw PCM. One stream, all users. |
|
||||
| Split | `poc/src/vad.ts` | The VAD cuts the stream into utterances and writes one WAV each. |
|
||||
| Text | `poc/src/stt.ts`, `poc/src/transcript.ts` | STT turns each WAV into text when a provider is configured. |
|
||||
| Speaker | `poc/src/audio-service.ts` | `VOICE_SPEAKING` events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user. |
|
||||
|
||||
The core bridge (`src/renderer/puppet/actions.ts`) has no WebRTC hooks today. The only audio call
|
||||
is `navigator.mediaDevices.enumerateDevices()` for the device list.
|
||||
|
||||
## Where per-user audio exists
|
||||
|
||||
Each remote user arrives in the renderer as a WebRTC `MediaStreamTrack`. Discord attaches the
|
||||
tracks to an `RTCPeerConnection` and mixes them in an `AudioContext` graph. Per-user gain (the
|
||||
volume sliders) is a node on each track's path. The OS output receives only the mix.
|
||||
|
||||
## Split options
|
||||
|
||||
| Option | How | Cost |
|
||||
| --- | --- | --- |
|
||||
| Hook `RTCPeerConnection` | Wrap `ontrack` and `addEventListener("track")` before Discord's code runs, then keep every remote track | Straightforward to hook, but the track carries no user ID |
|
||||
| Hook `AudioContext` | Wrap `createMediaStreamSource` and `createGain`. Discord makes one source per user, so the graph shows the per-user split | Better mapping chance, but fragile against client updates |
|
||||
| Map by event order | Pair the first chunk after a `VOICE_SPEAKING` event with that user | Works with heavy traffic, mislabels overlaps |
|
||||
| Chromium capture | Use Electron audio capture of `webContents` | Gives the mix again, not per-user |
|
||||
|
||||
Recommended path: hook `RTCPeerConnection` for the tracks, and hook `AudioContext` only to map a
|
||||
track to a user (the source node creation order matches the track order, and the per-user gain node
|
||||
sits next to it). Keep the event-order pairing as the fallback for unknown tracks.
|
||||
|
||||
## Transport for per-user PCM
|
||||
|
||||
The daemon needs one PCM stream per user. Two candidate shapes:
|
||||
|
||||
1. **AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge.** A 20 ms frame is 640 bytes,
|
||||
about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge
|
||||
traffic, small renderer cost.
|
||||
2. **MediaRecorder per track, Opus chunks every second.** Lower traffic and less renderer CPU.
|
||||
The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU.
|
||||
|
||||
Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon
|
||||
already runs ffmpeg for resampling. Add a binary frame type later when the volume grows.
|
||||
|
||||
## Plan
|
||||
|
||||
| Step | Work |
|
||||
| --- | --- |
|
||||
| M1 | Renderer hook lists remote tracks (`voice.tracks` action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping. |
|
||||
| M2 | One track taps to an `AudioWorklet`, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only. |
|
||||
| M3 | Map track to user ID (`AudioContext` hook, or the `VOICE_SPEAKING` fallback). Per-user VAD and transcript rows with a real `speakerId`. |
|
||||
| M4 | The `split user audio` setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback. |
|
||||
|
||||
## Setting
|
||||
|
||||
`preferences.audio.splitUserAudio` (boolean, default false), shown in Settings → Browser audio as
|
||||
**split user audio**. When it is off, the mixed capture feeds STT as it does today. When it is on,
|
||||
the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for
|
||||
the browser monitor.
|
||||
|
||||
## Failure modes to handle
|
||||
|
||||
- Discord updates can break the hooks. Gate the feature on a capability probe (`voice.tracks`
|
||||
answers with a list) and fall back to the mixed stream.
|
||||
- Tracks appear and disappear during a call. Handle `track-added` and `track-ended` events.
|
||||
- Muted or deafened users can still produce a track with silence. The VAD must drop it.
|
||||
- The audio thread must never block. Copy in the worklet, send outside it.
|
||||
- The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as
|
||||
mixed.
|
||||
@@ -294,6 +294,8 @@ replaceable.
|
||||
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
|
||||
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
|
||||
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?`, `speed?`, `pitch?` | Queue text to speech |
|
||||
| POST | `/api/audio/preview` | the speak fields | Synthesize and return a `url` for browser playback. Never touches the call |
|
||||
| GET | `/api/audio/tts/:name` | none | Serve a preview WAV (`tts-<n>-<n>.wav` only) |
|
||||
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
||||
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
||||
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
||||
|
||||
Reference in New Issue
Block a user