Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls
This commit is contained in:
+98
-3
@@ -33,7 +33,8 @@ The core reads `VoiceStateStore` only. It never joins a call on its own.
|
||||
Rules:
|
||||
|
||||
- The channel must be a voice channel. The core validates the snowflake and the channel type.
|
||||
- Join joins the call muted. The bot does not capture or send audio.
|
||||
- Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions,
|
||||
and a deafened client receives no call audio, which silences the monitor and the transcript.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
|
||||
daemon spaces them 250-750 ms (`pacedVoice`).
|
||||
- The join goes through Discord's internal channel action, located at runtime.
|
||||
@@ -81,15 +82,108 @@ works as before.
|
||||
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
|
||||
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
|
||||
binary. Lines play one at a time in order.
|
||||
- Each line gets a lead-in of silence before the first sample (`VESKSLOP_TTS_LEAD_MS`, default
|
||||
150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first
|
||||
syllable is not swallowed. The daemon adds it with `adelay` at playback; the engine's file is
|
||||
unchanged. The status reports it as `ttsLeadMs`.
|
||||
|
||||
### Per-user volume
|
||||
|
||||
Discord stores a playback volume per user (0 to 200, default 100). **0 silences that user**, which
|
||||
silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user
|
||||
at zero. Read it with `GET /api/voice/volume` (defaults to the current call) or the
|
||||
`get_user_volume` MCP tool; the `voice.get` member list and `get_voice_state` carry `volume` too.
|
||||
The response also carries the client's own `outputVolume`, which has the same effect when it is 0.
|
||||
|
||||
Set it with `POST /api/voice/volume` or `set_user_volume`. The action goes through pacing and is
|
||||
written to the action log as `set-volume`.
|
||||
|
||||
### Engines
|
||||
|
||||
The Speech settings list every configured engine: **espeak-ng**, the command engine
|
||||
(`VESKSLOP_TTS_CMD`, shown as `VESKSLOP_TTS_NAME` or "custom engine"), and **HTTP API**
|
||||
(`VESKSLOP_TTS_API_URL`). Unavailable engines are disabled. Auto picks the command engine first,
|
||||
then espeak-ng, then the API.
|
||||
|
||||
Selecting an engine switches the voice list to that engine. The language selector is hidden for
|
||||
single-language engines such as the command engine; their voices come from `VESKSLOP_TTS_VOICES`.
|
||||
The TTS panel carries the same engine and voice selects plus the speed and pitch sliders.
|
||||
|
||||
**Start engine** in the Speech settings warms the selected engine by running one throwaway
|
||||
synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real
|
||||
line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no
|
||||
warm-up. The same action is available as `POST /api/audio/tts/warm` and the MCP
|
||||
`start_tts_engine` tool.
|
||||
|
||||
### Voices
|
||||
|
||||
The local engine is espeak-ng. `GET /api/audio/tts/voices` returns every base voice plus the
|
||||
espeak variants (`m1`–`m7`, `f1`–`f5`, `croak`, `whisper`) as `<voice>+<variant>`, so each language
|
||||
has roughly 15 voices. For natural voices, point `VESKSLOP_TTS_API_URL` at a network TTS (piper,
|
||||
OpenAI, ElevenLabs) and pick the `api` provider; `VESKSLOP_TTS_CMD` can also call any local engine
|
||||
that accepts `{out}`, `{text_file}`, and `{voice}`.
|
||||
|
||||
When `VESKSLOP_TTS_CMD` is set, the daemon hides the espeak list and shows only the voices the
|
||||
command declares through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or newlines and
|
||||
use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
|
||||
`id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (hide the pitch
|
||||
control) and `expression` / `expression=NN` (Flextalk Expression parameter; NN is the registry
|
||||
default). The external-engine contract, including speed, pitch, expression, and voice metadata, is
|
||||
in [TTS-DESIGN.md](../TTS-DESIGN.md). The bundled engine integration lives in
|
||||
[`microslopsam/`](../microslopsam/README.md).
|
||||
|
||||
### Speed, pitch, and expression
|
||||
|
||||
The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x,
|
||||
1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both
|
||||
hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson)
|
||||
also expose an **Expression** slider (0–100) that changes the voice character; it appears only for
|
||||
those voices and defaults to the engine's registry value until moved. Values are stored in the
|
||||
daemon preferences and travel to the provider per line; the command provider also receives
|
||||
`VESKSLOP_TTS_SPEED`, `VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION`. Absent values mean
|
||||
"engine default". `/api/audio/speak` and the MCP `speak` tool accept `speed`, `pitch`, and
|
||||
`expression` overrides for a single line.
|
||||
|
||||
### Flextalk voice profile (MicroslopSam)
|
||||
|
||||
Voices flagged `expression=NN` are Flextalk voices and open the **Flextalk voice profile** window
|
||||
from the TTS panel:
|
||||
|
||||
- **Voice character** - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the
|
||||
voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to
|
||||
the default clears the override.
|
||||
- **Presets** - five named slots stored in `tts.profilePresets`; Save and Load copy the whole
|
||||
profile.
|
||||
- **Radio kit** - high-pass (`tts.highpass`, 40-400 Hz, default 40), mono echo (`tts.echo`, 0-100,
|
||||
default 0), and word mode (`tts.words`) with a gap in ms (`tts.wordGapMs`) for the radio/robot
|
||||
character.
|
||||
- **Test lines** - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines
|
||||
kept in `localStorage`. Each button speaks the line with the current profile, so a change can be
|
||||
auditioned without touching the real queue.
|
||||
|
||||
The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed
|
||||
profile keys travel as `profile` (with `highpass`, `echo`, `words`, and `wordGapMs`) to the command
|
||||
provider, which exports `VESKSLOP_TTS_PROFILE`, `VESKSLOP_TTS_HIGHPASS`, `VESKSLOP_TTS_ECHO`,
|
||||
`VESKSLOP_TTS_WORDS`, and `VESKSLOP_TTS_WORD_GAP_MS`. The wrapper contract and the registry
|
||||
pitfalls (`wine reg import`; never `echo` for the `.reg` key path) are in
|
||||
[TTS-DESIGN.md](../TTS-DESIGN.md).
|
||||
|
||||
### Browser monitor (Client audio)
|
||||
|
||||
The header **Client audio** button streams the call to the browser:
|
||||
|
||||
- The daemon moves the client's playback streams into `veskslop_capture` (match on
|
||||
`application.name` / `application.process.binary`, default `vesktop,electron,chromium,discord`,
|
||||
`VESKSLOP_AUDIO_CLIENT_MATCH` overrides). Without the move the capture sink is silent. The adopt
|
||||
pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor
|
||||
or the capture session runs.
|
||||
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
|
||||
the WebSocket at `/audio`.
|
||||
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
|
||||
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
|
||||
- Diagnose with `cd poc && pnpm audio:doctor`. It prints the default sink, the sink inputs and their
|
||||
sinks, the daemon audio status (`clientMatch`, `adopted`, `lastAdoptError`), and eight seconds of
|
||||
`VOICE_SPEAKING` / `VOICE_STATE_UPDATE` events. Run it during an active call.
|
||||
|
||||
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
|
||||
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
|
||||
@@ -98,8 +192,9 @@ the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling
|
||||
|
||||
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
||||
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
|
||||
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
|
||||
Yield (stop mid-line) is not implemented.
|
||||
`VOICE_SPEAKING` event. The TTS panel has a **duck on voice** toggle; it is stored in the daemon
|
||||
preferences and reported as `duckEnabled` by `/api/audio`. Staying quiet while someone else talks
|
||||
is a queue property, not a switch. Yield (stop mid-line) is not implemented.
|
||||
|
||||
### Capture and transcript
|
||||
|
||||
|
||||
Reference in New Issue
Block a user