Veskslop-t/docs/voice.md

# Voice

Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document
records the state and the plan.

## Current

| Part | Status |
| --- | --- |
| `voice.get` | Implemented. One user's voice channel and its members. Without `guildId` it searches all guilds. |
| `voice.overview` | Implemented. Occupied voice channels and members for one guild. |
| `VOICE_STATE_UPDATE` events | Normalized and pushed. Mock-verified. Live check pending. |
| `voice.join` | Implemented. Joins muted. Internal lookups verified live. |
| `voice.leave` | Implemented. Disconnects the client. |
| `voice.move` | Implemented. Moves to another channel. |
| Auto-leave | Not started. |
| `stay @user` | Not started. |
| Capture and transcript | Implemented. VAD segments, WAV per utterance, JSONL store, live events. |
| Speech to text | Hook implemented (local command or HTTP API); provider not chosen yet. |
| Audio in and out | Output and capture work. Browser mic puppeting is not started. |
| Video | Not started. |

The core reads `VoiceStateStore` only. It never joins a call on its own.

## Actions

| Action | Parameters | Result |
| --- | --- | --- |
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true, muted }` |
| `voice.leave` | none | `{ left: true }` |
| `voice.move` | `channelId` | `{ channelId, joined: true }` |

Rules:

- The channel must be a voice channel. The core validates the snowflake and the channel type.
- Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions,
  and a deafened client receives no call audio, which silences the monitor and the transcript.
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
  daemon spaces them 250-750 ms (`pacedVoice`).
- The join goes through Discord's internal channel action, located at runtime.
  `selectVoiceChannel` and `setSelfMute` were verified live on 2026-09-13. The end-to-end join was
  not run.

The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.

## Audio

Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.

### Devices

| Device | Kind | Purpose |
| --- | --- | --- |
| `veskslop_tts` | null sink | Playback target for TTS |
| `veskslop_mic` | remap source of `veskslop_tts.monitor` | The client microphone |
| `veskslop_capture` | null sink | Client output, recorded by the daemon |

The host needs a sound server. On Debian and Ubuntu:

```sh
apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng
```

The daemon starts `pulseaudio` automatically when `pactl` cannot reach a server, unless
`VESKSLOP_AUDIO_MANAGE_SERVER=0`. If `pactl` is missing, the status reports the install hint. The
Web UI Audio section has a **Connect devices** button that rechecks the client device list.

On startup the daemon creates missing devices with `pactl` and unloads the modules it created on
exit. When the client connects, the daemon finds the devices by label and sets them through
`voice.setDevices`, unless `VESKSLOP_AUDIO_AUTOCONNECT=0`. Names are configurable with
`VESKSLOP_AUDIO_SINK`, `VESKSLOP_AUDIO_SOURCE`, and `VESKSLOP_AUDIO_CAPTURE_SINK`.

If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon
works as before.

### Playback

- TTS has two providers:
  - **local**: `VESKSLOP_TTS_CMD` with `{out}` and `{text_file}` placeholders. Example:
    `piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}`.
  - **api**: `VESKSLOP_TTS_API_URL` receives a JSON body `{ text, voice? }` and returns audio
    bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
  binary. Lines play one at a time in order.
- Each line gets a lead-in of silence before the first sample (`VESKSLOP_TTS_LEAD_MS`, default
  150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first
  syllable is not swallowed. The daemon adds it with `adelay` at playback; the engine's file is
  unchanged. The status reports it as `ttsLeadMs`.

### Per-user volume

Discord stores a playback volume per user (0 to 200, default 100). **0 silences that user**, which
silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user
at zero. Read it with `GET /api/voice/volume` (defaults to the current call) or the
`get_user_volume` MCP tool; the `voice.get` member list and `get_voice_state` carry `volume` too.
The response also carries the client's own `outputVolume`, which has the same effect when it is 0.

Set it with `POST /api/voice/volume` or `set_user_volume`. The action goes through pacing and is
written to the action log as `set-volume`.

### Engines

The Speech settings list every configured engine: **espeak-ng**, the command engine
(`VESKSLOP_TTS_CMD`, shown as `VESKSLOP_TTS_NAME` or "custom engine"), and **HTTP API**
(`VESKSLOP_TTS_API_URL`). Unavailable engines are disabled. Auto picks the command engine first,
then espeak-ng, then the API.

Selecting an engine switches the voice list to that engine. The language selector is hidden for
single-language engines such as the command engine; their voices come from `VESKSLOP_TTS_VOICES`.
The TTS panel carries the same engine and voice selects plus the speed and pitch sliders.

**Start engine** in the Speech settings warms the selected engine by running one throwaway
synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real
line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no
warm-up. The same action is available as `POST /api/audio/tts/warm` and the MCP
`start_tts_engine` tool.

### Voices

The local engine is espeak-ng. `GET /api/audio/tts/voices` returns every base voice plus the
espeak variants (`m1`–`m7`, `f1`–`f5`, `croak`, `whisper`) as `<voice>+<variant>`, so each language
has roughly 15 voices. For natural voices, point `VESKSLOP_TTS_API_URL` at a network TTS (piper,
OpenAI, ElevenLabs) and pick the `api` provider; `VESKSLOP_TTS_CMD` can also call any local engine
that accepts `{out}`, `{text_file}`, and `{voice}`.

When `VESKSLOP_TTS_CMD` is set, the daemon hides the espeak list and shows only the voices the
command declares through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or newlines and
use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
`id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (hide the pitch
control) and `expression` / `expression=NN` (Flextalk Expression parameter; NN is the registry
default). The external-engine contract, including speed, pitch, expression, and voice metadata, is
in [TTS-DESIGN.md](../TTS-DESIGN.md). The bundled engine integration lives in
[`microslopsam/`](../microslopsam/README.md).

### Speed, pitch, and expression

The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x,
1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both
hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson)
also expose an **Expression** slider (0–100) that changes the voice character; it appears only for
those voices and defaults to the engine's registry value until moved. Values are stored in the
daemon preferences and travel to the provider per line; the command provider also receives
`VESKSLOP_TTS_SPEED`, `VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION`. Absent values mean
"engine default". `/api/audio/speak` and the MCP `speak` tool accept `speed`, `pitch`, and
`expression` overrides for a single line.

### Flextalk voice profile (MicroslopSam)

Voices flagged `expression=NN` are Flextalk voices and open the **Flextalk voice profile** window
from the TTS panel:

- **Voice character** - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the
  voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to
  the default clears the override.
- **Presets** - five named slots stored in `tts.profilePresets`; Save and Load copy the whole
  profile.
- **Radio kit** - high-pass (`tts.highpass`, 40-400 Hz, default 40), mono echo (`tts.echo`, 0-100,
  default 0), and word mode (`tts.words`) with a gap in ms (`tts.wordGapMs`) for the radio/robot
  character.
- **Test lines** - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines
  kept in `localStorage`. Each button speaks the line with the current profile, so a change can be
  auditioned without touching the real queue.

The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed
profile keys travel as `profile` (with `highpass`, `echo`, `words`, and `wordGapMs`) to the command
provider, which exports `VESKSLOP_TTS_PROFILE`, `VESKSLOP_TTS_HIGHPASS`, `VESKSLOP_TTS_ECHO`,
`VESKSLOP_TTS_WORDS`, and `VESKSLOP_TTS_WORD_GAP_MS`. The wrapper contract and the registry
pitfalls (`wine reg import`; never `echo` for the `.reg` key path) are in
[TTS-DESIGN.md](../TTS-DESIGN.md).

### Browser monitor (Client audio)

The header **Client audio** button streams the call to the browser:

- The daemon moves the client's playback streams into `veskslop_capture` (match on
  `application.name` / `application.process.binary`, default `vesktop,electron,chromium,discord`,
  `VESKSLOP_AUDIO_CLIENT_MATCH` overrides). Without the move the capture sink is silent. The adopt
  pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor
  or the capture session runs.
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
  the WebSocket at `/audio`.
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
- Diagnose with `cd poc && pnpm audio:doctor`. It prints the default sink, the sink inputs and their
  sinks, the daemon audio status (`clientMatch`, `adopted`, `lastAdoptError`), and eight seconds of
  `VOICE_SPEAKING` / `VOICE_STATE_UPDATE` events. Run it during an active call.

Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.

### Ducking

When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
`VOICE_SPEAKING` event. The TTS panel has a **duck on voice** toggle; it is stored in the daemon
preferences and reported as `duckEnabled` by `/api/audio`. Staying quiet while someone else talks
is a queue property, not a switch. Yield (stop mid-line) is not implemented.

### Capture and transcript

The daemon records the client output and splits it into utterances.

1. `ffmpeg` reads `veskslop_capture.monitor` as 16 kHz mono PCM (`VESKSLOP_FFMPEG` overrides).
2. The VAD opens a segment after `VESKSLOP_VAD_MIN_SPEECH_MS` above `VESKSLOP_VAD_THRESHOLD_DB`,
   keeps `VESKSLOP_VAD_PRE_ROLL_MS` of audio from before, and closes after
   `VESKSLOP_VAD_SILENCE_MS` of quiet.
3. Each segment becomes a WAV under `<cache>/audio/captures/` and a transcript entry in
   `VESKSLOP_TRANSCRIPT` (default `<state dir>/transcripts.jsonl`).
4. When `VOICE_SPEAKING` covers the segment, the dominant speaker is attached.
5. When STT is configured, the segment is transcribed and the text is attached.

STT providers:

- **Vosk (default)**: CPU, small and light. Install it and drop a model in `poc/models/`
  (auto-detected), or set `VESKSLOP_VOSK_MODEL`:
  ```sh
  pip install vosk
  curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
  unzip vosk-model-small-en-us-0.15.zip -d poc/models
  ```
  The daemon runs `poc/scripts/stt-vosk.py` for each segment. Quality is robotic but light.
- **local command**: `VESKSLOP_STT_CMD` with `{wav}`; the command prints text on stdout.
- **api**: `VESKSLOP_STT_API_URL` receives WAV bytes and returns text or JSON `{ text }`.
  `VESKSLOP_STT_API_KEY` adds a bearer token.

TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences).
The local engine defaults to `espeak-ng` when installed; the voice list comes from
`espeak-ng --voices`.

Start and stop with `POST /api/audio/capture`, the Audio settings button, or the MCP tools. Set
`VESKSLOP_CAPTURE_AUTO=1` to start capture when the bot joins a call and stop it on leave.

Transcript events go over the SSE stream: `TRANSCRIPT_SESSION`, `TRANSCRIPT_SEGMENT`,
`TRANSCRIPT_TEXT`, `TRANSCRIPT_SESSION_END`, and `TRANSCRIPT_ERROR`. The agent console's
Transcript tab shows the live list; `GET /api/transcript` and the `get_transcript` MCP tool read
the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.

### Controls

- Chat: `speak <text>` and `speak stop` from a trusted user.
- MCP: `speak`, `stop_audio`, `get_audio_status`, `start_capture`, `stop_capture`,
  `list_audio_devices`, `set_audio_devices`, `get_transcript`.
- HTTP: `/api/audio`, `/api/audio/speak`, `/api/audio/stop`, `/api/audio/capture`,
  `/api/audio/devices`.
- Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.


## Lifecycle

The bot leaves a call on its own in two cases.

1. **Owner leaves.** When the owner leaves the call, the bot waits `voiceLeaveDelaySeconds` (default
   30, `VESKSLOP_VOICE_LEAVE_DELAY` sets the default, `set_puppet_config` changes it) and then
   leaves. A rejoin inside the window cancels the leave. The owner is zzib for now.
2. **A stay target leaves.** The `stay @user` command pins the bot to a user. Not implemented yet;
   the owner rule is the only one active.

The bot never leaves while the owner is present. A manual `voice.leave` wins over both rules.

## Collector account

The collector account is a stripped-down client with read-only voice membership. The daemon uses it
when the puppet cannot see a user's voice channel or the channel members. It stays deferred. See
[TODO.md](../TODO.md).

## Verification list

A live run must confirm these points.

1. `voice.join` joins the given channel end to end. (The `selectVoiceChannel` lookup is verified.)
2. `voice.leave` disconnects the call.
3. `VOICE_STATE_UPDATE` carries the fields that the core normalizes.
4. The join is muted and Discord shows the mute state.
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.