# Voice Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document records the state and the plan. ## Current | Part | Status | | --- | --- | | `voice.get` | Implemented. One user's voice channel and its members. Without `guildId` it searches all guilds. | | `voice.overview` | Implemented. Occupied voice channels and members for one guild. | | `VOICE_STATE_UPDATE` events | Normalized and pushed. Mock-verified. Live check pending. | | `voice.join` | Implemented. Joins muted. Internal lookups verified live. | | `voice.leave` | Implemented. Disconnects the client. | | `voice.move` | Implemented. Moves to another channel. | | Auto-leave | Not started. | | `stay @user` | Not started. | | Capture and transcript | Implemented. VAD segments, WAV per utterance, JSONL store, live events. | | Speech to text | Hook implemented (local command or HTTP API); provider not chosen yet. | | Audio in and out | Output and capture work. Browser mic puppeting is not started. | | Video | Not started. | The core reads `VoiceStateStore` only. It never joins a call on its own. ## Actions | Action | Parameters | Result | | --- | --- | --- | | `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true, muted }` | | `voice.leave` | none | `{ left: true }` | | `voice.move` | `channelId` | `{ channelId, joined: true }` | Rules: - The channel must be a voice channel. The core validates the snowflake and the channel type. - Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions, and a deafened client receives no call audio, which silences the monitor and the transcript. - Voice actions are state changes, not send floods. They stay out of the message pacing queue. The daemon spaces them 250-750 ms (`pacedVoice`). - The join goes through Discord's internal channel action, located at runtime. `selectVoiceChannel` and `setSelfMute` were verified live on 2026-09-13. The end-to-end join was not run. The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again. ## Audio Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring. ### Devices | Device | Kind | Purpose | | --- | --- | --- | | `veskslop_tts` | null sink | Playback target for TTS | | `veskslop_mic` | remap source of `veskslop_tts.monitor` | The client microphone | | `veskslop_capture` | null sink | Client output, recorded by the daemon | Discord's own microphone processing fights a TTS line: **automatic gain control** ramps the volume up over the first words (so the start is quiet or clipped), and echo/noise cancellation gates the first syllable. The core reads and turns those gates off (`voice.processing`; MCP `set_voice_processing`), and every `voice.join` applies it automatically before connecting. Discord stores these per client, not per account, so a user's own app is unaffected. The host needs a sound server. On Debian and Ubuntu: ```sh apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng ``` The daemon starts `pulseaudio` automatically when `pactl` cannot reach a server, unless `VESKSLOP_AUDIO_MANAGE_SERVER=0`. If `pactl` is missing, the status reports the install hint. The Web UI Audio section has a **Connect devices** button that rechecks the client device list. On startup the daemon creates missing devices with `pactl` and unloads the modules it created on exit. When the client connects, the daemon finds the devices by label and sets them through `voice.setDevices`, unless `VESKSLOP_AUDIO_AUTOCONNECT=0`. Names are configurable with `VESKSLOP_AUDIO_SINK`, `VESKSLOP_AUDIO_SOURCE`, and `VESKSLOP_AUDIO_CAPTURE_SINK`. If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon works as before. ### Playback - TTS has two providers: - **local**: `VESKSLOP_TTS_CMD` with `{out}` and `{text_file}` placeholders. Example: `piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}`. - **api**: `VESKSLOP_TTS_API_URL` receives a JSON body `{ text, voice? }` and returns audio bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token. - `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the binary. Lines play one at a time in order. - Each line gets a lead-in of silence before the first sample (`VESKSLOP_TTS_LEAD_MS`, default 150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first syllable is not swallowed. The daemon adds it with `adelay` at playback; the engine's file is unchanged. The status reports it as `ttsLeadMs`. ### Per-user volume Discord stores a playback volume per user (0 to 200, default 100). **0 silences that user**, which silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user at zero. Read it with `GET /api/voice/volume` (defaults to the current call) or the `get_user_volume` MCP tool; the `voice.get` member list and `get_voice_state` carry `volume` too. The response also carries the client's own `outputVolume`, which has the same effect when it is 0. Set it with `POST /api/voice/volume` or `set_user_volume`. The action goes through pacing and is written to the action log as `set-volume`. ### Engines The Speech settings list every configured engine: **espeak-ng**, the command engine (`VESKSLOP_TTS_CMD`, shown as `VESKSLOP_TTS_NAME` or "custom engine"), and **HTTP API** (`VESKSLOP_TTS_API_URL`). Unavailable engines are disabled. Auto picks the command engine first, then espeak-ng, then the API. Selecting an engine switches the voice list to that engine. The language selector is hidden for single-language engines such as the command engine; their voices come from `VESKSLOP_TTS_VOICES`. The TTS panel carries the same engine and voice selects plus the speed and pitch sliders. **Start engine** in the Speech settings warms the selected engine by running one throwaway synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no warm-up. The same action is available as `POST /api/audio/tts/warm` and the MCP `start_tts_engine` tool. ### Voices The local engine is espeak-ng. `GET /api/audio/tts/voices` returns every base voice plus the espeak variants (`m1`–`m7`, `f1`–`f5`, `croak`, `whisper`) as `+`, so each language has roughly 15 voices. For natural voices, point `VESKSLOP_TTS_API_URL` at a network TTS (piper, OpenAI, ElevenLabs) and pick the `api` provider; `VESKSLOP_TTS_CMD` can also call any local engine that accepts `{out}`, `{text_file}`, and `{voice}`. When `VESKSLOP_TTS_CMD` is set, the daemon hides the espeak list and shows only the voices the command declares through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or newlines and use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or `id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (hide the pitch control) and `expression` / `expression=NN` (Flextalk Expression parameter; NN is the registry default). The external-engine contract, including speed, pitch, expression, and voice metadata, is in [TTS-DESIGN.md](../TTS-DESIGN.md). The bundled engine integration lives in [`microslopsam/`](../microslopsam/README.md). ### Speed, pitch, and expression The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x, 1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson) also expose an **Expression** slider (0–100) that changes the voice character; it appears only for those voices and defaults to the engine's registry value until moved. Values are stored in the daemon preferences and travel to the provider per line; the command provider also receives `VESKSLOP_TTS_SPEED`, `VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION`. Absent values mean "engine default". `/api/audio/speak` and the MCP `speak` tool accept `speed`, `pitch`, and `expression` overrides for a single line. ### Flextalk voice profile (MicroslopSam) Voices flagged `expression=NN` are Flextalk voices and open the **Flextalk voice profile** window from the TTS panel: - **Voice character** - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to the default clears the override. - **Presets** - five named slots stored in `tts.profilePresets`; Save and Load copy the whole profile. - **Radio kit** - high-pass (`tts.highpass`, 40-400 Hz, default 40), mono echo (`tts.echo`, 0-100, default 0), and word mode (`tts.words`) with a gap in ms (`tts.wordGapMs`) for the radio/robot character. **VOX prosody** (`tts.vox`) routes the line through the NetTTS-style encoder first - sentence breaks, lead-ins, weighted pauses, and Flextalk vendor tags - for Black Mesa announcement energy; **clean** (`tts.voxClean`) drops the staccato wrapper. Word mode **combines with the presets** and takes precedence over VOX prosody: each word is its own engine instance, so the parser is skipped and the vendor tags go on every word (`\!R` rate, `\!%` pitch). VOX needs `python3` on the host. Both travel to the command provider as `VESKSLOP_TTS_VOX` / `VESKSLOP_TTS_VOX_CLEAN` (or the `{vox}` / `{vox_clean}` placeholders). A **crunch** stage adds the lossy 1990s codec character as its own step **after** the rest of the chain (echo tails included): `tts.crunch` presets `hl90` (8-bit, 11 kHz), `hl90hq` (8-bit, 22 kHz), `telephone` (8 kHz band-limited), `adpcm` (4-bit) and `crush` (custom), with optional `tts.crunchBits` (1-16) and `tts.crunchRate` (4-48 kHz) overrides. The output stays 48 kHz s16le, so only the character changes. Echo gained `tts.echoDelayMs` (10-400) and `tts.echoLevel` (0-0.95) alongside the amount. **Stream word mode** (`tts.streamWords`, on by default) makes the wrapper write the WAV while it synthesizes, so playback starts on the first word: measured first sound 0.5 s instead of 2.4 s on a ten-word line. The wrapper touches `.ready` after the first word and the daemon waits for that before starting playback, so a player never opens the bare WAV header and exits before the words arrive. Streaming skips the loudness measurement (`VESKSLOP_TTS_STREAM_GAIN_DB` compensates if needed) and cannot use VOX prosody, which needs the whole sentence. The Half-Life voices are first-class entries in the voice list: **VOX (Half-Life)** and **HEV suit** are Keith Bell with the corresponding profile preset, and VOX carries its prosody with it (so no separate prosody toggle). Selecting the VOX voice shows its **clean (no stutter)** option in the profile area. The underlying `tts.preset` / `vox` / `voxClean` values remain available to the API and MCP; precedence is preset < profile < expression, so the sliders still win. The wrapper writes the audible profile to both registry hives (plus the preset's Gender/Line Mode and a Default Speaker pointer), and for Flextalk it sends rate/pitch as the sticky `\!R` / `\!%` vendor tags rather than `ITTSAttributes`; attributes only apply to MSTTS voices. Neutral sliders emit neither, so the preset is reproduced exactly. The daemon always exports `VESKSLOP_TTS_VOX` (0 or 1) so an explicit off is honored, and the panel lets a preset, VOX, and word mode be combined. `sh microslopsam/setup/test-wrapper.sh` checks this contract with stubbed wine and engine binaries, no Wine install needed. The preset values also show up in the voice metadata defaults, so the panel's "default" snap points match what the engine gets. - **Test lines** - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines kept in `localStorage`. Each button speaks the line with the current profile, so a change can be auditioned without touching the real queue. The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed profile keys travel as `profile` (with `highpass`, `echo`, `words`, `wordGapMs`, `vox`, and `voxClean`) to the command provider, which exports `VESKSLOP_TTS_PROFILE`, `VESKSLOP_TTS_HIGHPASS`, `VESKSLOP_TTS_ECHO`, `VESKSLOP_TTS_WORDS`, `VESKSLOP_TTS_WORD_GAP_MS`, `VESKSLOP_TTS_VOX`, and `VESKSLOP_TTS_VOX_CLEAN`. The wrapper contract and the registry pitfalls (`wine reg import`; never `echo` for the `.reg` key path) are in [TTS-DESIGN.md](../TTS-DESIGN.md). ### Browser monitor (Client audio) The header **Client audio** button streams the call to the browser: - The daemon moves the client's playback streams into `veskslop_capture` (match on `application.name` / `application.process.binary`, default `vesktop,electron,chromium,discord`, `VESKSLOP_AUDIO_CLIENT_MATCH` overrides). Without the move the capture sink is silent. The adopt pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor or the capture session runs. - The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over the WebSocket at `/audio`. - The browser plays the PCM with the Web Audio API and can send it to a chosen output device. - This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call. - Diagnose with `cd poc && pnpm audio:doctor`. It prints the default sink, the sink inputs and their sinks, the daemon audio status (`clientMatch`, `adopted`, `lastAdoptError`), and eight seconds of `VOICE_SPEAKING` / `VOICE_STATE_UPDATE` events. Run it during an active call. Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling. ### Ducking When another person speaks, the daemon lowers the `veskslop_tts` sink volume to `VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the `VOICE_SPEAKING` event. The TTS panel has a **duck on voice** toggle; it is stored in the daemon preferences and reported as `duckEnabled` by `/api/audio`. Staying quiet while someone else talks is a queue property, not a switch. Yield (stop mid-line) is not implemented. ### Capture and transcript The daemon records the client output and splits it into utterances. 1. `ffmpeg` reads `veskslop_capture.monitor` as 16 kHz mono PCM (`VESKSLOP_FFMPEG` overrides). 2. The VAD opens a segment after `VESKSLOP_VAD_MIN_SPEECH_MS` above `VESKSLOP_VAD_THRESHOLD_DB`, keeps `VESKSLOP_VAD_PRE_ROLL_MS` of audio from before, and closes after `VESKSLOP_VAD_SILENCE_MS` of quiet. 3. Each segment becomes a WAV under `/audio/captures/` and a transcript entry in `VESKSLOP_TRANSCRIPT` (default `/transcripts.jsonl`). 4. When `VOICE_SPEAKING` covers the segment, the dominant speaker is attached. 5. When STT is configured, the segment is transcribed and the text is attached. STT providers: - **Vosk (default)**: CPU, small and light. Install it and drop a model in `poc/models/` (auto-detected), or set `VESKSLOP_VOSK_MODEL`: ```sh pip install vosk curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip unzip vosk-model-small-en-us-0.15.zip -d poc/models ``` The daemon runs `poc/scripts/stt-vosk.py` for each segment. Quality is robotic but light. - **local command**: `VESKSLOP_STT_CMD` with `{wav}`; the command prints text on stdout. - **api**: `VESKSLOP_STT_API_URL` receives WAV bytes and returns text or JSON `{ text }`. `VESKSLOP_STT_API_KEY` adds a bearer token. TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences). The local engine defaults to `espeak-ng` when installed; the voice list comes from `espeak-ng --voices`. Start and stop with `POST /api/audio/capture`, the Audio settings button, or the MCP tools. Set `VESKSLOP_CAPTURE_AUTO=1` to start capture when the bot joins a call and stop it on leave. Transcript events go over the SSE stream: `TRANSCRIPT_SESSION`, `TRANSCRIPT_SEGMENT`, `TRANSCRIPT_TEXT`, `TRANSCRIPT_SESSION_END`, and `TRANSCRIPT_ERROR`. The agent console's Transcript tab shows the live list; `GET /api/transcript` and the `get_transcript` MCP tool read the store. Segments without text still carry the WAV path, so an LLM can transcribe them later. ### Controls - Chat: `speak ` and `speak stop` from a trusted user. - MCP: `speak`, `stop_audio`, `get_audio_status`, `start_capture`, `stop_capture`, `list_audio_devices`, `set_audio_devices`, `get_transcript`. - HTTP: `/api/audio`, `/api/audio/speak`, `/api/audio/stop`, `/api/audio/capture`, `/api/audio/devices`. - Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section. ## Lifecycle The bot leaves a call on its own in two cases. 1. **Owner leaves.** When the owner leaves the call, the bot waits `voiceLeaveDelaySeconds` (default 30, `VESKSLOP_VOICE_LEAVE_DELAY` sets the default, `set_puppet_config` changes it) and then leaves. A rejoin inside the window cancels the leave. The owner is zzib for now. 2. **A stay target leaves.** The `stay @user` command pins the bot to a user. Not implemented yet; the owner rule is the only one active. The bot never leaves while the owner is present. A manual `voice.leave` wins over both rules. ## Collector account The collector account is a stripped-down client with read-only voice membership. The daemon uses it when the puppet cannot see a user's voice channel or the channel members. It stays deferred. See [TODO.md](../TODO.md). ## Verification list A live run must confirm these points. 1. `voice.join` joins the given channel end to end. (The `selectVoiceChannel` lookup is verified.) 2. `voice.leave` disconnects the call. 3. `VOICE_STATE_UPDATE` carries the fields that the core normalizes. 4. The join is muted and Discord shows the mute state. 5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.