Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls

This commit is contained in:
Fisto
2026-09-15 15:09:23 +02:00
parent eab94132bb
commit 302436f95f
42 changed files with 4675 additions and 180 deletions
+98 -3
View File
@@ -33,7 +33,8 @@ The core reads `VoiceStateStore` only. It never joins a call on its own.
Rules:
- The channel must be a voice channel. The core validates the snowflake and the channel type.
- Join joins the call muted. The bot does not capture or send audio.
- Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions,
and a deafened client receives no call audio, which silences the monitor and the transcript.
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
daemon spaces them 250-750 ms (`pacedVoice`).
- The join goes through Discord's internal channel action, located at runtime.
@@ -81,15 +82,108 @@ works as before.
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
binary. Lines play one at a time in order.
- Each line gets a lead-in of silence before the first sample (`VESKSLOP_TTS_LEAD_MS`, default
150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first
syllable is not swallowed. The daemon adds it with `adelay` at playback; the engine's file is
unchanged. The status reports it as `ttsLeadMs`.
### Per-user volume
Discord stores a playback volume per user (0 to 200, default 100). **0 silences that user**, which
silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user
at zero. Read it with `GET /api/voice/volume` (defaults to the current call) or the
`get_user_volume` MCP tool; the `voice.get` member list and `get_voice_state` carry `volume` too.
The response also carries the client's own `outputVolume`, which has the same effect when it is 0.
Set it with `POST /api/voice/volume` or `set_user_volume`. The action goes through pacing and is
written to the action log as `set-volume`.
### Engines
The Speech settings list every configured engine: **espeak-ng**, the command engine
(`VESKSLOP_TTS_CMD`, shown as `VESKSLOP_TTS_NAME` or "custom engine"), and **HTTP API**
(`VESKSLOP_TTS_API_URL`). Unavailable engines are disabled. Auto picks the command engine first,
then espeak-ng, then the API.
Selecting an engine switches the voice list to that engine. The language selector is hidden for
single-language engines such as the command engine; their voices come from `VESKSLOP_TTS_VOICES`.
The TTS panel carries the same engine and voice selects plus the speed and pitch sliders.
**Start engine** in the Speech settings warms the selected engine by running one throwaway
synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real
line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no
warm-up. The same action is available as `POST /api/audio/tts/warm` and the MCP
`start_tts_engine` tool.
### Voices
The local engine is espeak-ng. `GET /api/audio/tts/voices` returns every base voice plus the
espeak variants (`m1`–`m7`, `f1`–`f5`, `croak`, `whisper`) as `<voice>+<variant>`, so each language
has roughly 15 voices. For natural voices, point `VESKSLOP_TTS_API_URL` at a network TTS (piper,
OpenAI, ElevenLabs) and pick the `api` provider; `VESKSLOP_TTS_CMD` can also call any local engine
that accepts `{out}`, `{text_file}`, and `{voice}`.
When `VESKSLOP_TTS_CMD` is set, the daemon hides the espeak list and shows only the voices the
command declares through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or newlines and
use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
`id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (hide the pitch
control) and `expression` / `expression=NN` (Flextalk Expression parameter; NN is the registry
default). The external-engine contract, including speed, pitch, expression, and voice metadata, is
in [TTS-DESIGN.md](../TTS-DESIGN.md). The bundled engine integration lives in
[`microslopsam/`](../microslopsam/README.md).
### Speed, pitch, and expression
The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x,
1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both
hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson)
also expose an **Expression** slider (0–100) that changes the voice character; it appears only for
those voices and defaults to the engine's registry value until moved. Values are stored in the
daemon preferences and travel to the provider per line; the command provider also receives
`VESKSLOP_TTS_SPEED`, `VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION`. Absent values mean
"engine default". `/api/audio/speak` and the MCP `speak` tool accept `speed`, `pitch`, and
`expression` overrides for a single line.
### Flextalk voice profile (MicroslopSam)
Voices flagged `expression=NN` are Flextalk voices and open the **Flextalk voice profile** window
from the TTS panel:
- **Voice character** - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the
voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to
the default clears the override.
- **Presets** - five named slots stored in `tts.profilePresets`; Save and Load copy the whole
profile.
- **Radio kit** - high-pass (`tts.highpass`, 40-400 Hz, default 40), mono echo (`tts.echo`, 0-100,
default 0), and word mode (`tts.words`) with a gap in ms (`tts.wordGapMs`) for the radio/robot
character.
- **Test lines** - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines
kept in `localStorage`. Each button speaks the line with the current profile, so a change can be
auditioned without touching the real queue.
The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed
profile keys travel as `profile` (with `highpass`, `echo`, `words`, and `wordGapMs`) to the command
provider, which exports `VESKSLOP_TTS_PROFILE`, `VESKSLOP_TTS_HIGHPASS`, `VESKSLOP_TTS_ECHO`,
`VESKSLOP_TTS_WORDS`, and `VESKSLOP_TTS_WORD_GAP_MS`. The wrapper contract and the registry
pitfalls (`wine reg import`; never `echo` for the `.reg` key path) are in
[TTS-DESIGN.md](../TTS-DESIGN.md).
### Browser monitor (Client audio)
The header **Client audio** button streams the call to the browser:
- The daemon moves the client's playback streams into `veskslop_capture` (match on
`application.name` / `application.process.binary`, default `vesktop,electron,chromium,discord`,
`VESKSLOP_AUDIO_CLIENT_MATCH` overrides). Without the move the capture sink is silent. The adopt
pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor
or the capture session runs.
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
the WebSocket at `/audio`.
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
- Diagnose with `cd poc && pnpm audio:doctor`. It prints the default sink, the sink inputs and their
sinks, the daemon audio status (`clientMatch`, `adopted`, `lastAdoptError`), and eight seconds of
`VOICE_SPEAKING` / `VOICE_STATE_UPDATE` events. Run it during an active call.
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
@@ -98,8 +192,9 @@ the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
Yield (stop mid-line) is not implemented.
`VOICE_SPEAKING` event. The TTS panel has a **duck on voice** toggle; it is stored in the daemon
preferences and reported as `duckEnabled` by `/api/audio`. Staying quiet while someone else talks
is a queue property, not a switch. Yield (stop mid-line) is not implemented.
### Capture and transcript