Veskslop-t/docs/voice.md

Voice

Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document records the state and the plan.

Current

Part	Status
voice.get	Implemented. One user's voice channel and its members. Without guildId it searches all guilds.
voice.overview	Implemented. Occupied voice channels and members for one guild.
VOICE_STATE_UPDATE events	Normalized and pushed. Mock-verified. Live check pending.
voice.join	Implemented. Joins muted. Internal lookups verified live.
voice.leave	Implemented. Disconnects the client.
voice.move	Implemented. Moves to another channel.
Auto-leave	Not started.
stay @user	Not started.
Capture and transcript	Implemented. VAD segments, WAV per utterance, JSONL store, live events.
Speech to text	Hook implemented (local command or HTTP API); provider not chosen yet.
Audio in and out	Output and capture work. Browser mic puppeting is not started.
Video	Not started.

The core reads VoiceStateStore only. It never joins a call on its own.

Actions

Action	Parameters	Result
voice.join	channelId, optional mute (default true)	{ channelId, joined: true, muted }
voice.leave	none	{ left: true }
voice.move	channelId	{ channelId, joined: true }

Rules:

• The channel must be a voice channel. The core validates the snowflake and the channel type.
• Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions, and a deafened client receives no call audio, which silences the monitor and the transcript.
• Voice actions are state changes, not send floods. They stay out of the message pacing queue. The daemon spaces them 250-750 ms (pacedVoice).
• The join goes through Discord's internal channel action, located at runtime. selectVoiceChannel and setSelfMute were verified live on 2026-09-13. The end-to-end join was not run.

The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.

Audio

Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.

Devices

Device	Kind	Purpose
veskslop_tts	null sink	Playback target for TTS
veskslop_mic	remap source of veskslop_tts.monitor	The client microphone
veskslop_capture	null sink	Client output, recorded by the daemon

The host needs a sound server. On Debian and Ubuntu:

apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng

The daemon starts pulseaudio automatically when pactl cannot reach a server, unless VESKSLOP_AUDIO_MANAGE_SERVER=0. If pactl is missing, the status reports the install hint. The Web UI Audio section has a Connect devices button that rechecks the client device list.

On startup the daemon creates missing devices with pactl and unloads the modules it created on exit. When the client connects, the daemon finds the devices by label and sets them through voice.setDevices, unless VESKSLOP_AUDIO_AUTOCONNECT=0. Names are configurable with VESKSLOP_AUDIO_SINK, VESKSLOP_AUDIO_SOURCE, and VESKSLOP_AUDIO_CAPTURE_SINK.

If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon works as before.

Playback

• TTS has two providers:
  • local: VESKSLOP_TTS_CMD with {out} and {text_file} placeholders. Example: piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}.
  • api: VESKSLOP_TTS_API_URL receives a JSON body { text, voice? } and returns audio bytes. VESKSLOP_TTS_API_KEY adds a bearer token.
• ffplay decodes and plays into veskslop_tts (PULSE_SINK). VESKSLOP_FFPLAY overrides the binary. Lines play one at a time in order.
• Each line gets a lead-in of silence before the first sample (VESKSLOP_TTS_LEAD_MS, default 150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first syllable is not swallowed. The daemon adds it with adelay at playback; the engine's file is unchanged. The status reports it as ttsLeadMs.

Per-user volume

Discord stores a playback volume per user (0 to 200, default 100). 0 silences that user, which silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user at zero. Read it with GET /api/voice/volume (defaults to the current call) or the get_user_volume MCP tool; the voice.get member list and get_voice_state carry volume too. The response also carries the client's own outputVolume, which has the same effect when it is 0.

Set it with POST /api/voice/volume or set_user_volume. The action goes through pacing and is written to the action log as set-volume.

Engines

The Speech settings list every configured engine: espeak-ng, the command engine (VESKSLOP_TTS_CMD, shown as VESKSLOP_TTS_NAME or "custom engine"), and HTTP API (VESKSLOP_TTS_API_URL). Unavailable engines are disabled. Auto picks the command engine first, then espeak-ng, then the API.

Selecting an engine switches the voice list to that engine. The language selector is hidden for single-language engines such as the command engine; their voices come from VESKSLOP_TTS_VOICES. The TTS panel carries the same engine and voice selects plus the speed and pitch sliders.

Start engine in the Speech settings warms the selected engine by running one throwaway synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no warm-up. The same action is available as POST /api/audio/tts/warm and the MCP start_tts_engine tool.

Voices

The local engine is espeak-ng. GET /api/audio/tts/voices returns every base voice plus the espeak variants (m1–m7, f1–f5, croak, whisper) as <voice>+<variant>, so each language has roughly 15 voices. For natural voices, point VESKSLOP_TTS_API_URL at a network TTS (piper, OpenAI, ElevenLabs) and pick the api provider; VESKSLOP_TTS_CMD can also call any local engine that accepts {out}, {text_file}, and {voice}.

When VESKSLOP_TTS_CMD is set, the daemon hides the espeak list and shows only the voices the command declares through VESKSLOP_TTS_VOICES. Entries are separated by commas or newlines and use id, id|Name, id|Name|gender, id|Name|gender|language, or id|Name|gender|language|flags, where flags are +-separated. Flags: no-pitch (hide the pitch control) and expression / expression=NN (Flextalk Expression parameter; NN is the registry default). The external-engine contract, including speed, pitch, expression, and voice metadata, is in TTS-DESIGN.md. The bundled engine integration lives in microslopsam/.

Speed, pitch, and expression

The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x, 1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson) also expose an Expression slider (0–100) that changes the voice character; it appears only for those voices and defaults to the engine's registry value until moved. Values are stored in the daemon preferences and travel to the provider per line; the command provider also receives VESKSLOP_TTS_SPEED, VESKSLOP_TTS_PITCH, and VESKSLOP_TTS_EXPRESSION. Absent values mean "engine default". /api/audio/speak and the MCP speak tool accept speed, pitch, and expression overrides for a single line.

Flextalk voice profile (MicroslopSam)

Voices flagged expression=NN are Flextalk voices and open the Flextalk voice profile window from the TTS panel:

• Voice character - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to the default clears the override.
• Presets - five named slots stored in tts.profilePresets; Save and Load copy the whole profile.
• Radio kit - high-pass (tts.highpass, 40-400 Hz, default 40), mono echo (tts.echo, 0-100, default 0), and word mode (tts.words) with a gap in ms (tts.wordGapMs) for the radio/robot character. VOX prosody (tts.vox) routes the line through the NetTTS-style encoder first - sentence breaks, lead-ins, weighted pauses, and Flextalk vendor tags - for Black Mesa announcement energy; clean (tts.voxClean) drops the staccato wrapper. Word mode combines with the presets and takes precedence over VOX prosody: each word is its own engine instance, so the parser is skipped and the vendor tags go on every word (\!R rate, \!% pitch). VOX needs python3 on the host. Both travel to the command provider as VESKSLOP_TTS_VOX / VESKSLOP_TTS_VOX_CLEAN (or the {vox} / {vox_clean} placeholders). A crunch stage adds the lossy 1990s codec character as its own step after the rest of the chain (echo tails included): tts.crunch presets hl90 (8-bit, 11 kHz), hl90hq (8-bit, 22 kHz), telephone (8 kHz band-limited), adpcm (4-bit) and crush (custom), with optional tts.crunchBits (1-16) and tts.crunchRate (4-48 kHz) overrides. The output stays 48 kHz s16le, so only the character changes. Echo gained tts.echoDelayMs (10-400) and tts.echoLevel (0-0.95) alongside the amount. Stream word mode (tts.streamWords, on by default) makes the wrapper write the WAV while it synthesizes, so playback starts on the first word: measured first sound 0.5 s instead of 2.4 s on a ten-word line. The wrapper touches <file>.ready after the first word and the daemon waits for that before starting playback, so a player never opens the bare WAV header and exits before the words arrive. Streaming skips the loudness measurement (VESKSLOP_TTS_STREAM_GAIN_DB compensates if needed) and cannot use VOX prosody, which needs the whole sentence. The Half-Life voices are first-class entries in the voice list: VOX (Half-Life) and HEV suit are Keith Bell with the corresponding profile preset, and VOX carries its prosody with it (so no separate prosody toggle). Selecting the VOX voice shows its clean (no stutter) option in the profile area. The underlying tts.preset / vox / voxClean values remain available to the API and MCP; precedence is preset < profile < expression, so the sliders still win. The wrapper writes the audible profile to both registry hives (plus the preset's Gender/Line Mode and a Default Speaker pointer), and for Flextalk it sends rate/pitch as the sticky \!R / \!% vendor tags rather than ITTSAttributes; attributes only apply to MSTTS voices. Neutral sliders emit neither, so the preset is reproduced exactly. The daemon always exports VESKSLOP_TTS_VOX (0 or 1) so an explicit off is honored, and the panel lets a preset, VOX, and word mode be combined. sh microslopsam/setup/test-wrapper.sh checks this contract with stubbed wine and engine binaries, no Wine install needed. The preset values also show up in the voice metadata defaults, so the panel's "default" snap points match what the engine gets.
• Test lines - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines kept in localStorage. Each button speaks the line with the current profile, so a change can be auditioned without touching the real queue.

The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed profile keys travel as profile (with highpass, echo, words, wordGapMs, vox, and voxClean) to the command provider, which exports VESKSLOP_TTS_PROFILE, VESKSLOP_TTS_HIGHPASS, VESKSLOP_TTS_ECHO, VESKSLOP_TTS_WORDS, VESKSLOP_TTS_WORD_GAP_MS, VESKSLOP_TTS_VOX, and VESKSLOP_TTS_VOX_CLEAN. The wrapper contract and the registry pitfalls (wine reg import; never echo for the .reg key path) are in TTS-DESIGN.md.

Browser monitor (Client audio)

The header Client audio button streams the call to the browser:

• The daemon moves the client's playback streams into veskslop_capture (match on application.name / application.process.binary, default vesktop,electron,chromium,discord, VESKSLOP_AUDIO_CLIENT_MATCH overrides). Without the move the capture sink is silent. The adopt pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor or the capture session runs.
• The daemon reads veskslop_capture.monitor with ffmpeg and sends raw 48 kHz mono PCM over the WebSocket at /audio.
• The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
• This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
• Diagnose with cd poc && pnpm audio:doctor. It prints the default sink, the sink inputs and their sinks, the daemon audio status (clientMatch, adopted, lastAdoptError), and eight seconds of VOICE_SPEAKING / VOICE_STATE_UPDATE events. Run it during an active call.

Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so the browser mic and TTS share veskslop_mic, a pacat input, and echo handling.

Ducking

When another person speaks, the daemon lowers the veskslop_tts sink volume to VESKSLOP_AUDIO_DUCK percent (default 30) and restores it when they stop. Speaking comes from the VOICE_SPEAKING event. The TTS panel has a duck on voice toggle; it is stored in the daemon preferences and reported as duckEnabled by /api/audio. Staying quiet while someone else talks is a queue property, not a switch. Yield (stop mid-line) is not implemented.

Capture and transcript

The daemon records the client output and splits it into utterances.

1. ffmpeg reads veskslop_capture.monitor as 16 kHz mono PCM (VESKSLOP_FFMPEG overrides).
2. The VAD opens a segment after VESKSLOP_VAD_MIN_SPEECH_MS above VESKSLOP_VAD_THRESHOLD_DB, keeps VESKSLOP_VAD_PRE_ROLL_MS of audio from before, and closes after VESKSLOP_VAD_SILENCE_MS of quiet.
3. Each segment becomes a WAV under <cache>/audio/captures/ and a transcript entry in VESKSLOP_TRANSCRIPT (default <state dir>/transcripts.jsonl).
4. When VOICE_SPEAKING covers the segment, the dominant speaker is attached.
5. When STT is configured, the segment is transcribed and the text is attached.

STT providers:

• Vosk (default): CPU, small and light. Install it and drop a model in poc/models/ (auto-detected), or set VESKSLOP_VOSK_MODEL:

  pip install vosk
  curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
  unzip vosk-model-small-en-us-0.15.zip -d poc/models
  

  The daemon runs poc/scripts/stt-vosk.py for each segment. Quality is robotic but light.
• local command: VESKSLOP_STT_CMD with {wav}; the command prints text on stdout.
• api: VESKSLOP_STT_API_URL receives WAV bytes and returns text or JSON { text }. VESKSLOP_STT_API_KEY adds a bearer token.

TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences). The local engine defaults to espeak-ng when installed; the voice list comes from espeak-ng --voices.

Start and stop with POST /api/audio/capture, the Audio settings button, or the MCP tools. Set VESKSLOP_CAPTURE_AUTO=1 to start capture when the bot joins a call and stop it on leave.

Transcript events go over the SSE stream: TRANSCRIPT_SESSION, TRANSCRIPT_SEGMENT, TRANSCRIPT_TEXT, TRANSCRIPT_SESSION_END, and TRANSCRIPT_ERROR. The agent console's Transcript tab shows the live list; GET /api/transcript and the get_transcript MCP tool read the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.

Controls

• Chat: speak <text> and speak stop from a trusted user.
• MCP: speak, stop_audio, get_audio_status, start_capture, stop_capture, list_audio_devices, set_audio_devices, get_transcript.
• HTTP: /api/audio, /api/audio/speak, /api/audio/stop, /api/audio/capture, /api/audio/devices.
• Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.

Lifecycle

The bot leaves a call on its own in two cases.

1. Owner leaves. When the owner leaves the call, the bot waits voiceLeaveDelaySeconds (default 30, VESKSLOP_VOICE_LEAVE_DELAY sets the default, set_puppet_config changes it) and then leaves. A rejoin inside the window cancels the leave. The owner is zzib for now.
2. A stay target leaves. The stay @user command pins the bot to a user. Not implemented yet; the owner rule is the only one active.

The bot never leaves while the owner is present. A manual voice.leave wins over both rules.

Collector account

The collector account is a stripped-down client with read-only voice membership. The daemon uses it when the puppet cannot see a user's voice channel or the channel members. It stays deferred. See TODO.md.

Verification list

A live run must confirm these points.

1. voice.join joins the given channel end to end. (The selectVoiceChannel lookup is verified.)
2. voice.leave disconnects the call.
3. VOICE_STATE_UPDATE carries the fields that the core normalizes.
4. The join is muted and Discord shows the mute state.
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.