183 lines
8.3 KiB
Markdown
183 lines
8.3 KiB
Markdown
# Voice
|
|
|
|
Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document
|
|
records the state and the plan.
|
|
|
|
## Current
|
|
|
|
| Part | Status |
|
|
| --- | --- |
|
|
| `voice.get` | Implemented. One user's voice channel and its members. Without `guildId` it searches all guilds. |
|
|
| `voice.overview` | Implemented. Occupied voice channels and members for one guild. |
|
|
| `VOICE_STATE_UPDATE` events | Normalized and pushed. Mock-verified. Live check pending. |
|
|
| `voice.join` | Implemented. Joins muted. Internal lookups verified live. |
|
|
| `voice.leave` | Implemented. Disconnects the client. |
|
|
| `voice.move` | Implemented. Moves to another channel. |
|
|
| Auto-leave | Not started. |
|
|
| `stay @user` | Not started. |
|
|
| Capture and transcript | Implemented. VAD segments, WAV per utterance, JSONL store, live events. |
|
|
| Speech to text | Hook implemented (local command or HTTP API); provider not chosen yet. |
|
|
| Audio in and out | Output and capture work. Browser mic puppeting is not started. |
|
|
| Video | Not started. |
|
|
|
|
The core reads `VoiceStateStore` only. It never joins a call on its own.
|
|
|
|
## Actions
|
|
|
|
| Action | Parameters | Result |
|
|
| --- | --- | --- |
|
|
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true, muted }` |
|
|
| `voice.leave` | none | `{ left: true }` |
|
|
| `voice.move` | `channelId` | `{ channelId, joined: true }` |
|
|
|
|
Rules:
|
|
|
|
- The channel must be a voice channel. The core validates the snowflake and the channel type.
|
|
- Join joins the call muted. The bot does not capture or send audio.
|
|
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
|
|
daemon spaces them 250-750 ms (`pacedVoice`).
|
|
- The join goes through Discord's internal channel action, located at runtime.
|
|
`selectVoiceChannel` and `setSelfMute` were verified live on 2026-09-13. The end-to-end join was
|
|
not run.
|
|
|
|
The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.
|
|
|
|
## Audio
|
|
|
|
Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.
|
|
|
|
### Devices
|
|
|
|
| Device | Kind | Purpose |
|
|
| --- | --- | --- |
|
|
| `veskslop_tts` | null sink | Playback target for TTS |
|
|
| `veskslop_mic` | remap source of `veskslop_tts.monitor` | The client microphone |
|
|
| `veskslop_capture` | null sink | Client output, recorded by the daemon |
|
|
|
|
The host needs a sound server. On Debian and Ubuntu:
|
|
|
|
```sh
|
|
apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng
|
|
```
|
|
|
|
The daemon starts `pulseaudio` automatically when `pactl` cannot reach a server, unless
|
|
`VESKSLOP_AUDIO_MANAGE_SERVER=0`. If `pactl` is missing, the status reports the install hint. The
|
|
Web UI Audio section has a **Connect devices** button that rechecks the client device list.
|
|
|
|
On startup the daemon creates missing devices with `pactl` and unloads the modules it created on
|
|
exit. When the client connects, the daemon finds the devices by label and sets them through
|
|
`voice.setDevices`, unless `VESKSLOP_AUDIO_AUTOCONNECT=0`. Names are configurable with
|
|
`VESKSLOP_AUDIO_SINK`, `VESKSLOP_AUDIO_SOURCE`, and `VESKSLOP_AUDIO_CAPTURE_SINK`.
|
|
|
|
If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon
|
|
works as before.
|
|
|
|
### Playback
|
|
|
|
- TTS has two providers:
|
|
- **local**: `VESKSLOP_TTS_CMD` with `{out}` and `{text_file}` placeholders. Example:
|
|
`piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}`.
|
|
- **api**: `VESKSLOP_TTS_API_URL` receives a JSON body `{ text, voice? }` and returns audio
|
|
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
|
|
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
|
|
binary. Lines play one at a time in order.
|
|
|
|
### Browser monitor (Client audio)
|
|
|
|
The header **Client audio** button streams the call to the browser:
|
|
|
|
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
|
|
the WebSocket at `/audio`.
|
|
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
|
|
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
|
|
|
|
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
|
|
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
|
|
|
|
### Ducking
|
|
|
|
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
|
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
|
|
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
|
|
Yield (stop mid-line) is not implemented.
|
|
|
|
### Capture and transcript
|
|
|
|
The daemon records the client output and splits it into utterances.
|
|
|
|
1. `ffmpeg` reads `veskslop_capture.monitor` as 16 kHz mono PCM (`VESKSLOP_FFMPEG` overrides).
|
|
2. The VAD opens a segment after `VESKSLOP_VAD_MIN_SPEECH_MS` above `VESKSLOP_VAD_THRESHOLD_DB`,
|
|
keeps `VESKSLOP_VAD_PRE_ROLL_MS` of audio from before, and closes after
|
|
`VESKSLOP_VAD_SILENCE_MS` of quiet.
|
|
3. Each segment becomes a WAV under `<cache>/audio/captures/` and a transcript entry in
|
|
`VESKSLOP_TRANSCRIPT` (default `<state dir>/transcripts.jsonl`).
|
|
4. When `VOICE_SPEAKING` covers the segment, the dominant speaker is attached.
|
|
5. When STT is configured, the segment is transcribed and the text is attached.
|
|
|
|
STT providers:
|
|
|
|
- **Vosk (default)**: CPU, small and light. Install it and drop a model in `poc/models/`
|
|
(auto-detected), or set `VESKSLOP_VOSK_MODEL`:
|
|
```sh
|
|
pip install vosk
|
|
curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
|
|
unzip vosk-model-small-en-us-0.15.zip -d poc/models
|
|
```
|
|
The daemon runs `poc/scripts/stt-vosk.py` for each segment. Quality is robotic but light.
|
|
- **local command**: `VESKSLOP_STT_CMD` with `{wav}`; the command prints text on stdout.
|
|
- **api**: `VESKSLOP_STT_API_URL` receives WAV bytes and returns text or JSON `{ text }`.
|
|
`VESKSLOP_STT_API_KEY` adds a bearer token.
|
|
|
|
TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences).
|
|
The local engine defaults to `espeak-ng` when installed; the voice list comes from
|
|
`espeak-ng --voices`.
|
|
|
|
Start and stop with `POST /api/audio/capture`, the Audio settings button, or the MCP tools. Set
|
|
`VESKSLOP_CAPTURE_AUTO=1` to start capture when the bot joins a call and stop it on leave.
|
|
|
|
Transcript events go over the SSE stream: `TRANSCRIPT_SESSION`, `TRANSCRIPT_SEGMENT`,
|
|
`TRANSCRIPT_TEXT`, `TRANSCRIPT_SESSION_END`, and `TRANSCRIPT_ERROR`. The agent console's
|
|
Transcript tab shows the live list; `GET /api/transcript` and the `get_transcript` MCP tool read
|
|
the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.
|
|
|
|
### Controls
|
|
|
|
- Chat: `speak <text>` and `speak stop` from a trusted user.
|
|
- MCP: `speak`, `stop_audio`, `get_audio_status`, `start_capture`, `stop_capture`,
|
|
`list_audio_devices`, `set_audio_devices`, `get_transcript`.
|
|
- HTTP: `/api/audio`, `/api/audio/speak`, `/api/audio/stop`, `/api/audio/capture`,
|
|
`/api/audio/devices`.
|
|
- Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.
|
|
|
|
|
|
## Lifecycle
|
|
|
|
The bot leaves a call on its own in two cases.
|
|
|
|
1. **Owner leaves.** When the owner leaves the call, the bot waits `voiceLeaveDelaySeconds` and
|
|
then leaves. A rejoin inside the window cancels the leave.
|
|
2. **A stay target leaves.** The `stay @user` command pins the bot to a user. When that user
|
|
leaves, the bot waits the same delay and then leaves.
|
|
|
|
`stay @user` needs a chat command. Chat commands are paused at this time, so the command waits for
|
|
the voice backend. The default delay is proposed as 30 seconds and lives in the config.
|
|
|
|
The bot never leaves while the owner or the stay target is present. A manual `voice.leave` wins
|
|
over both rules.
|
|
|
|
## Collector account
|
|
|
|
The collector account is a stripped-down client with read-only voice membership. The daemon uses it
|
|
when the puppet cannot see a user's voice channel or the channel members. It stays deferred. See
|
|
[TODO.md](../TODO.md).
|
|
|
|
## Verification list
|
|
|
|
A live run must confirm these points.
|
|
|
|
1. `voice.join` joins the given channel end to end. (The `selectVoiceChannel` lookup is verified.)
|
|
2. `voice.leave` disconnects the call.
|
|
3. `VOICE_STATE_UPDATE` carries the fields that the core normalizes.
|
|
4. The join is muted and Discord shows the mute state.
|
|
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.
|