Veskslop-t/docs/voice.md

Voice

Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document records the state and the plan.

Current

Part	Status
voice.get	Implemented. One user's voice channel and its members. Without guildId it searches all guilds.
voice.overview	Implemented. Occupied voice channels and members for one guild.
VOICE_STATE_UPDATE events	Normalized and pushed. Mock-verified. Live check pending.
voice.join	Implemented. Joins muted. Internal lookups verified live.
voice.leave	Implemented. Disconnects the client.
voice.move	Implemented. Moves to another channel.
Auto-leave	Not started.
stay @user	Not started.
Capture and transcript	Implemented. VAD segments, WAV per utterance, JSONL store, live events.
Speech to text	Hook implemented (local command or HTTP API); provider not chosen yet.
Audio in and out	Output and capture work. Browser mic puppeting is not started.
Video	Not started.

The core reads VoiceStateStore only. It never joins a call on its own.

Actions

Action	Parameters	Result
voice.join	channelId, optional mute (default true)	{ channelId, joined: true, muted }
voice.leave	none	{ left: true }
voice.move	channelId	{ channelId, joined: true }

Rules:

• The channel must be a voice channel. The core validates the snowflake and the channel type.
• Join joins the call muted. The bot does not capture or send audio.
• Voice actions are state changes, not send floods. They stay out of the message pacing queue. The daemon spaces them 250-750 ms (pacedVoice).
• The join goes through Discord's internal channel action, located at runtime. selectVoiceChannel and setSelfMute were verified live on 2026-09-13. The end-to-end join was not run.

The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.

Audio

Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.

Devices

Device	Kind	Purpose
veskslop_tts	null sink	Playback target for TTS
veskslop_mic	remap source of veskslop_tts.monitor	The client microphone
veskslop_capture	null sink	Client output, recorded by the daemon

The host needs a sound server. On Debian and Ubuntu:

apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng

The daemon starts pulseaudio automatically when pactl cannot reach a server, unless VESKSLOP_AUDIO_MANAGE_SERVER=0. If pactl is missing, the status reports the install hint. The Web UI Audio section has a Connect devices button that rechecks the client device list.

On startup the daemon creates missing devices with pactl and unloads the modules it created on exit. When the client connects, the daemon finds the devices by label and sets them through voice.setDevices, unless VESKSLOP_AUDIO_AUTOCONNECT=0. Names are configurable with VESKSLOP_AUDIO_SINK, VESKSLOP_AUDIO_SOURCE, and VESKSLOP_AUDIO_CAPTURE_SINK.

If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon works as before.

Playback

• TTS has two providers:
  • local: VESKSLOP_TTS_CMD with {out} and {text_file} placeholders. Example: piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}.
  • api: VESKSLOP_TTS_API_URL receives a JSON body { text, voice? } and returns audio bytes. VESKSLOP_TTS_API_KEY adds a bearer token.
• ffplay decodes and plays into veskslop_tts (PULSE_SINK). VESKSLOP_FFPLAY overrides the binary. Lines play one at a time in order.

Browser monitor (Client audio)

The header Client audio button streams the call to the browser:

• The daemon reads veskslop_capture.monitor with ffmpeg and sends raw 48 kHz mono PCM over the WebSocket at /audio.
• The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
• This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.

Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so the browser mic and TTS share veskslop_mic, a pacat input, and echo handling.

Ducking

When another person speaks, the daemon lowers the veskslop_tts sink volume to VESKSLOP_AUDIO_DUCK percent (default 30) and restores it when they stop. Speaking comes from the VOICE_SPEAKING event. Staying quiet while someone else talks is a queue property, not a switch. Yield (stop mid-line) is not implemented.

Capture and transcript

The daemon records the client output and splits it into utterances.

1. ffmpeg reads veskslop_capture.monitor as 16 kHz mono PCM (VESKSLOP_FFMPEG overrides).
2. The VAD opens a segment after VESKSLOP_VAD_MIN_SPEECH_MS above VESKSLOP_VAD_THRESHOLD_DB, keeps VESKSLOP_VAD_PRE_ROLL_MS of audio from before, and closes after VESKSLOP_VAD_SILENCE_MS of quiet.
3. Each segment becomes a WAV under <cache>/audio/captures/ and a transcript entry in VESKSLOP_TRANSCRIPT (default <state dir>/transcripts.jsonl).
4. When VOICE_SPEAKING covers the segment, the dominant speaker is attached.
5. When STT is configured, the segment is transcribed and the text is attached.

STT providers:

• Vosk (default): CPU, small and light. Install it and drop a model in poc/models/ (auto-detected), or set VESKSLOP_VOSK_MODEL:

  pip install vosk
  curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
  unzip vosk-model-small-en-us-0.15.zip -d poc/models
  

  The daemon runs poc/scripts/stt-vosk.py for each segment. Quality is robotic but light.
• local command: VESKSLOP_STT_CMD with {wav}; the command prints text on stdout.
• api: VESKSLOP_STT_API_URL receives WAV bytes and returns text or JSON { text }. VESKSLOP_STT_API_KEY adds a bearer token.

TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences). The local engine defaults to espeak-ng when installed; the voice list comes from espeak-ng --voices.

Start and stop with POST /api/audio/capture, the Audio settings button, or the MCP tools. Set VESKSLOP_CAPTURE_AUTO=1 to start capture when the bot joins a call and stop it on leave.

Transcript events go over the SSE stream: TRANSCRIPT_SESSION, TRANSCRIPT_SEGMENT, TRANSCRIPT_TEXT, TRANSCRIPT_SESSION_END, and TRANSCRIPT_ERROR. The agent console's Transcript tab shows the live list; GET /api/transcript and the get_transcript MCP tool read the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.

Controls

• Chat: speak <text> and speak stop from a trusted user.
• MCP: speak, stop_audio, get_audio_status, start_capture, stop_capture, list_audio_devices, set_audio_devices, get_transcript.
• HTTP: /api/audio, /api/audio/speak, /api/audio/stop, /api/audio/capture, /api/audio/devices.
• Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.

Lifecycle

The bot leaves a call on its own in two cases.

1. Owner leaves. When the owner leaves the call, the bot waits voiceLeaveDelaySeconds and then leaves. A rejoin inside the window cancels the leave.
2. A stay target leaves. The stay @user command pins the bot to a user. When that user leaves, the bot waits the same delay and then leaves.

stay @user needs a chat command. Chat commands are paused at this time, so the command waits for the voice backend. The default delay is proposed as 30 seconds and lives in the config.

The bot never leaves while the owner or the stay target is present. A manual voice.leave wins over both rules.

Collector account

The collector account is a stripped-down client with read-only voice membership. The daemon uses it when the puppet cannot see a user's voice channel or the channel members. It stays deferred. See TODO.md.

Verification list

A live run must confirm these points.

1. voice.join joins the given channel end to end. (The selectVoiceChannel lookup is verified.)
2. voice.leave disconnects the call.
3. VOICE_STATE_UPDATE carries the fields that the core normalizes.
4. The join is muted and Discord shows the mute state.
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.