Update docs and README for voice audio, capture, and the web UI
This commit is contained in:
Fisto
+128
-13
@@ -1,6 +1,7 @@
|
||||
# Voice
|
||||
|
||||
Voice state reads exist. Voice actions do not. This document records the plan and the decisions.
|
||||
Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document
|
||||
records the state and the plan.
|
||||
|
||||
## Current
|
||||
|
||||
@@ -9,31 +10,145 @@ Voice state reads exist. Voice actions do not. This document records the plan an
|
||||
| `voice.get` | Implemented. One user's voice channel and its members. Without `guildId` it searches all guilds. |
|
||||
| `voice.overview` | Implemented. Occupied voice channels and members for one guild. |
|
||||
| `VOICE_STATE_UPDATE` events | Normalized and pushed. Mock-verified. Live check pending. |
|
||||
| Join and leave | Not started. |
|
||||
| Audio in and out | Not started. |
|
||||
| `voice.join` | Implemented. Joins muted. Internal lookups verified live. |
|
||||
| `voice.leave` | Implemented. Disconnects the client. |
|
||||
| `voice.move` | Implemented. Moves to another channel. |
|
||||
| Auto-leave | Not started. |
|
||||
| `stay @user` | Not started. |
|
||||
| Capture and transcript | Implemented. VAD segments, WAV per utterance, JSONL store, live events. |
|
||||
| Speech to text | Hook implemented (local command or HTTP API); provider not chosen yet. |
|
||||
| Audio in and out | Output and capture work. Browser mic puppeting is not started. |
|
||||
| Video | Not started. |
|
||||
|
||||
The core reads `VoiceStateStore` only. It never joins a call on its own.
|
||||
|
||||
## Planned actions
|
||||
## Actions
|
||||
|
||||
| Action | Parameters | Result |
|
||||
| --- | --- | --- |
|
||||
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true }` |
|
||||
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true, muted }` |
|
||||
| `voice.leave` | none | `{ left: true }` |
|
||||
| `voice.move` | `channelId` | `{ channelId, moved: true }` |
|
||||
| `voice.move` | `channelId` | `{ channelId, joined: true }` |
|
||||
|
||||
Rules:
|
||||
|
||||
- The channel must be a voice channel. The core validates the snowflake and the channel type.
|
||||
- Join joins the call muted. The bot does not capture or send audio.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue, but
|
||||
the daemon rate limits them separately.
|
||||
- The join implementation goes through Discord's internal channel action, located at runtime. The
|
||||
lookup is unverified until the first live run. This is the same pattern as `reaction.add`.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
|
||||
daemon spaces them 250-750 ms (`pacedVoice`).
|
||||
- The join goes through Discord's internal channel action, located at runtime.
|
||||
`selectVoiceChannel` and `setSelfMute` were verified live on 2026-09-13. The end-to-end join was
|
||||
not run.
|
||||
|
||||
The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.
|
||||
Audio playback is a later feature.
|
||||
|
||||
## Audio
|
||||
|
||||
Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.
|
||||
|
||||
### Devices
|
||||
|
||||
| Device | Kind | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `veskslop_tts` | null sink | Playback target for TTS |
|
||||
| `veskslop_mic` | remap source of `veskslop_tts.monitor` | The client microphone |
|
||||
| `veskslop_capture` | null sink | Client output, recorded by the daemon |
|
||||
|
||||
The host needs a sound server. On Debian and Ubuntu:
|
||||
|
||||
```sh
|
||||
apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng
|
||||
```
|
||||
|
||||
The daemon starts `pulseaudio` automatically when `pactl` cannot reach a server, unless
|
||||
`VESKSLOP_AUDIO_MANAGE_SERVER=0`. If `pactl` is missing, the status reports the install hint. The
|
||||
Web UI Audio section has a **Connect devices** button that rechecks the client device list.
|
||||
|
||||
On startup the daemon creates missing devices with `pactl` and unloads the modules it created on
|
||||
exit. When the client connects, the daemon finds the devices by label and sets them through
|
||||
`voice.setDevices`, unless `VESKSLOP_AUDIO_AUTOCONNECT=0`. Names are configurable with
|
||||
`VESKSLOP_AUDIO_SINK`, `VESKSLOP_AUDIO_SOURCE`, and `VESKSLOP_AUDIO_CAPTURE_SINK`.
|
||||
|
||||
If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon
|
||||
works as before.
|
||||
|
||||
### Playback
|
||||
|
||||
- TTS has two providers:
|
||||
- **local**: `VESKSLOP_TTS_CMD` with `{out}` and `{text_file}` placeholders. Example:
|
||||
`piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}`.
|
||||
- **api**: `VESKSLOP_TTS_API_URL` receives a JSON body `{ text, voice? }` and returns audio
|
||||
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
|
||||
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
|
||||
binary. Lines play one at a time in order.
|
||||
|
||||
### Browser monitor (Client audio)
|
||||
|
||||
The header **Client audio** button streams the call to the browser:
|
||||
|
||||
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
|
||||
the WebSocket at `/audio`.
|
||||
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
|
||||
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
|
||||
|
||||
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
|
||||
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
|
||||
|
||||
### Ducking
|
||||
|
||||
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
||||
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
|
||||
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
|
||||
Yield (stop mid-line) is not implemented.
|
||||
|
||||
### Capture and transcript
|
||||
|
||||
The daemon records the client output and splits it into utterances.
|
||||
|
||||
1. `ffmpeg` reads `veskslop_capture.monitor` as 16 kHz mono PCM (`VESKSLOP_FFMPEG` overrides).
|
||||
2. The VAD opens a segment after `VESKSLOP_VAD_MIN_SPEECH_MS` above `VESKSLOP_VAD_THRESHOLD_DB`,
|
||||
keeps `VESKSLOP_VAD_PRE_ROLL_MS` of audio from before, and closes after
|
||||
`VESKSLOP_VAD_SILENCE_MS` of quiet.
|
||||
3. Each segment becomes a WAV under `<cache>/audio/captures/` and a transcript entry in
|
||||
`VESKSLOP_TRANSCRIPT` (default `<state dir>/transcripts.jsonl`).
|
||||
4. When `VOICE_SPEAKING` covers the segment, the dominant speaker is attached.
|
||||
5. When STT is configured, the segment is transcribed and the text is attached.
|
||||
|
||||
STT providers:
|
||||
|
||||
- **Vosk (default)**: CPU, small and light. Install it and drop a model in `poc/models/`
|
||||
(auto-detected), or set `VESKSLOP_VOSK_MODEL`:
|
||||
```sh
|
||||
pip install vosk
|
||||
curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
|
||||
unzip vosk-model-small-en-us-0.15.zip -d poc/models
|
||||
```
|
||||
The daemon runs `poc/scripts/stt-vosk.py` for each segment. Quality is robotic but light.
|
||||
- **local command**: `VESKSLOP_STT_CMD` with `{wav}`; the command prints text on stdout.
|
||||
- **api**: `VESKSLOP_STT_API_URL` receives WAV bytes and returns text or JSON `{ text }`.
|
||||
`VESKSLOP_STT_API_KEY` adds a bearer token.
|
||||
|
||||
TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences).
|
||||
The local engine defaults to `espeak-ng` when installed; the voice list comes from
|
||||
`espeak-ng --voices`.
|
||||
|
||||
Start and stop with `POST /api/audio/capture`, the Audio settings button, or the MCP tools. Set
|
||||
`VESKSLOP_CAPTURE_AUTO=1` to start capture when the bot joins a call and stop it on leave.
|
||||
|
||||
Transcript events go over the SSE stream: `TRANSCRIPT_SESSION`, `TRANSCRIPT_SEGMENT`,
|
||||
`TRANSCRIPT_TEXT`, `TRANSCRIPT_SESSION_END`, and `TRANSCRIPT_ERROR`. The agent console's
|
||||
Transcript tab shows the live list; `GET /api/transcript` and the `get_transcript` MCP tool read
|
||||
the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.
|
||||
|
||||
### Controls
|
||||
|
||||
- Chat: `speak <text>` and `speak stop` from a trusted user.
|
||||
- MCP: `speak`, `stop_audio`, `get_audio_status`, `start_capture`, `stop_capture`,
|
||||
`list_audio_devices`, `set_audio_devices`, `get_transcript`.
|
||||
- HTTP: `/api/audio`, `/api/audio/speak`, `/api/audio/stop`, `/api/audio/capture`,
|
||||
`/api/audio/devices`.
|
||||
- Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.
|
||||
|
||||
|
||||
## Lifecycle
|
||||
|
||||
@@ -60,8 +175,8 @@ when the puppet cannot see a user's voice channel or the channel members. It sta
|
||||
|
||||
A live run must confirm these points.
|
||||
|
||||
1. The internal `selectVoiceChannel` lookup exists and joins the given channel.
|
||||
2. The internal disconnect action leaves the call.
|
||||
1. `voice.join` joins the given channel end to end. (The `selectVoiceChannel` lookup is verified.)
|
||||
2. `voice.leave` disconnects the call.
|
||||
3. `VOICE_STATE_UPDATE` carries the fields that the core normalizes.
|
||||
4. The join is muted and Discord shows the mute state.
|
||||
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.
|
||||
|
||||
Reference in New Issue
Block a user