Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls
This commit is contained in:
Fisto
@@ -25,4 +25,6 @@ This file lists the project documents.
|
||||
| [../CURRENT.md](../CURRENT.md) | Live session state |
|
||||
| [../TODO.md](../TODO.md) | Ordered backlog |
|
||||
| [../poc/README.md](../poc/README.md) | Spike and daemon run commands |
|
||||
| [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration |
|
||||
| [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine |
|
||||
| [../README.md](../README.md) | Project introduction |
|
||||
|
||||
+15
-3
@@ -87,13 +87,19 @@ The renderer normalizes client events and pushes them to subscribers.
|
||||
| `VOICE_STATE_UPDATE` | User, channel, guild, session, mute, deaf, self-mute, self-deaf |
|
||||
| `VOICE_SPEAKING` | User, speaking flag, channel |
|
||||
|
||||
Discord's dispatcher only sends reliable speaking-start events. The daemon polls the core's
|
||||
`voice.speaking` action (which reads `ChannelRTCStore.getSpeakingParticipants`) once a second and
|
||||
synthesizes the missing start/stop events, feeding dispatcher hints into the same set. Discord
|
||||
sends `speaking` as a bitfield number, so the core treats `0` as false and a missing flag as a
|
||||
start pulse. A user who mutes or leaves also emits a state update that ends the speaking state in
|
||||
the UI.
|
||||
|
||||
## Validation rules
|
||||
|
||||
The core validates all input. The daemon cannot bypass these rules.
|
||||
|
||||
- A channel ID and a user ID must match the snowflake format. The format is 17 to 20 digits.
|
||||
- `content` must be a string. The content must not be empty after a trim.
|
||||
- `content` must not contain control characters.
|
||||
- `content` must be a string. The content must not be empty after a trim.- `content` must not contain control characters.
|
||||
- `content` must not be longer than 2000 characters.
|
||||
- The target channel must exist. The channel type must accept text. The allowed types are
|
||||
`GUILD_TEXT`, `DM`, `GROUP_DM`, `GUILD_ANNOUNCEMENT`, `ANNOUNCEMENT_THREAD`, `PUBLIC_THREAD`,
|
||||
@@ -145,7 +151,13 @@ The core rate limit is a safety net. It does not replace the daemon pacing.
|
||||
| `DM_UNAVAILABLE` | The client cannot create a DM channel. |
|
||||
| `NOT_SUPPORTED` | The client does not offer the internal action. |
|
||||
| `RATE_LIMITED` | The core rate limit rejected the action. |
|
||||
| `INTERNAL` | An unexpected error occurred. The details go to the local log only. |
|
||||
| `INTERNAL` | An unexpected error occurred. The error message is included (for example a lazy module lookup before the renderer finished loading). |
|
||||
|
||||
A renderer command that is sent while the window is destroyed or crashed is rejected at once with
|
||||
`CORE_UNAVAILABLE`, and a command that gets no answer times out after 15 seconds. The bridge socket
|
||||
survives a renderer restart; the daemon retries background work such as device auto-connect. While
|
||||
the renderer is gone, the bridge logs one warning instead of one stack per command, then stays
|
||||
quiet until a command succeeds again.
|
||||
|
||||
## Hooks into upstream files
|
||||
|
||||
|
||||
+14
-2
@@ -61,11 +61,14 @@ The daemon exposes MCP over Streamable HTTP. The default address is `http://127.
|
||||
| `send_dm` | Send a DM to a user. |
|
||||
| `get_voice_state` | Which voice channel a user is in and who else is there. |
|
||||
| `get_voice_overview` | Occupied voice channels in a guild and their members. |
|
||||
| `get_user_volume` | Per-user playback volume and the client output volume. |
|
||||
| `set_user_volume` | Set a user's playback volume (0 to 200). |
|
||||
| `voice_join` | Join a voice channel, muted by default. |
|
||||
| `voice_leave` | Leave the current voice channel. |
|
||||
| `voice_move` | Move to another voice channel. |
|
||||
| `get_audio_status` | Audio engine, TTS, queue, ducking, and capture state. |
|
||||
| `speak` | Queue text to speech into the virtual microphone. |
|
||||
| `speak` | Queue text to speech into the virtual microphone (`text`, `provider?`, `voice?`, `speed?`, `pitch?`). |
|
||||
| `start_tts_engine` | Warm up the TTS engine so the first line is fast (`engine?`). |
|
||||
| `stop_audio` | Stop speech and clear the queue. |
|
||||
| `start_capture` / `stop_capture` | Record or stop recording the client output sink. |
|
||||
| `list_audio_devices` | The client's input and output audio devices. |
|
||||
@@ -160,10 +163,19 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
|
||||
| `VESKSLOP_AUDIO_AUTOCONNECT` | `1` | Point the client at the virtual devices on connect. |
|
||||
| `VESKSLOP_AUDIO_MANAGE_SERVER` | `1` | Start `pulseaudio` when `pactl` cannot reach a server. |
|
||||
| `VESKSLOP_AUDIO_SINK` / `_SOURCE` / `_CAPTURE_SINK` | `veskslop_tts` / `veskslop_mic` / `veskslop_capture` | Virtual device names. |
|
||||
| `VESKSLOP_AUDIO_CLIENT_MATCH` | `vesktop,electron,chromium,discord` | Stream name patterns moved into the capture sink. |
|
||||
|
||||
The daemon warns at startup when `dist/` is older than the core sources. Rebuild with `pnpm build`
|
||||
and restart the client, or new actions fail with `UNKNOWN_ACTION`.
|
||||
| `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. |
|
||||
| `VESKSLOP_TTS_LEAD_MS` | `150` | Lead-in silence before each TTS line (0 disables, max 2000). |
|
||||
| `VESKSLOP_FFPLAY` / `VESKSLOP_FFMPEG` | `ffplay` / `ffmpeg` | Playback and capture binaries. |
|
||||
| `VESKSLOP_TTS_CMD` | unset | Local TTS command with `{out}` and `{text_file}`. |
|
||||
| `VESKSLOP_TTS_CMD` | unset | Local TTS command with `{out}` and `{text_file}` (also `{voice}`, `{speed}`, `{pitch}`). |
|
||||
| `VESKSLOP_TTS_NAME` | `custom engine` | Display name for the command engine in the web UI. |
|
||||
| `VESKSLOP_TTS_API_URL` / `_API_KEY` / `_PROVIDER` | unset | HTTP TTS provider and bearer token. |
|
||||
| `VESKSLOP_TTS_VOICES` | unset | Voice list for a custom command: `id[|Name[|gender[|language]]]`, comma separated. |
|
||||
| `VESKSLOP_TTS_SPEED` / `_PITCH` | unset | Exported to the command only when a value is set. |
|
||||
| `VESKSLOP_TTS_EXPRESSION` | unset | Flextalk Expression profile value (0-100), exported when set. |
|
||||
| `VESKSLOP_TRANSCRIPT` | `<state dir>/transcripts.jsonl` | Transcript store path. |
|
||||
| `VESKSLOP_PREFERENCES` | `<state dir>/preferences.json` | Web UI preferences (visible guilds, TTS). |
|
||||
| `VESKSLOP_CAPTURE_AUTO` | `0` | Start capture when the bot joins a call. |
|
||||
|
||||
+98
-3
@@ -33,7 +33,8 @@ The core reads `VoiceStateStore` only. It never joins a call on its own.
|
||||
Rules:
|
||||
|
||||
- The channel must be a voice channel. The core validates the snowflake and the channel type.
|
||||
- Join joins the call muted. The bot does not capture or send audio.
|
||||
- Join joins the call muted and clears self-deafen. Discord persists self-deafen across sessions,
|
||||
and a deafened client receives no call audio, which silences the monitor and the transcript.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
|
||||
daemon spaces them 250-750 ms (`pacedVoice`).
|
||||
- The join goes through Discord's internal channel action, located at runtime.
|
||||
@@ -81,15 +82,108 @@ works as before.
|
||||
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
|
||||
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
|
||||
binary. Lines play one at a time in order.
|
||||
- Each line gets a lead-in of silence before the first sample (`VESKSLOP_TTS_LEAD_MS`, default
|
||||
150 ms, maximum 2000). This gives Discord's voice pipeline a moment to settle so the first
|
||||
syllable is not swallowed. The daemon adds it with `adelay` at playback; the engine's file is
|
||||
unchanged. The status reports it as `ttsLeadMs`.
|
||||
|
||||
### Per-user volume
|
||||
|
||||
Discord stores a playback volume per user (0 to 200, default 100). **0 silences that user**, which
|
||||
silences the capture sink even though the stream is uncorked and at 100%: the client mixes the user
|
||||
at zero. Read it with `GET /api/voice/volume` (defaults to the current call) or the
|
||||
`get_user_volume` MCP tool; the `voice.get` member list and `get_voice_state` carry `volume` too.
|
||||
The response also carries the client's own `outputVolume`, which has the same effect when it is 0.
|
||||
|
||||
Set it with `POST /api/voice/volume` or `set_user_volume`. The action goes through pacing and is
|
||||
written to the action log as `set-volume`.
|
||||
|
||||
### Engines
|
||||
|
||||
The Speech settings list every configured engine: **espeak-ng**, the command engine
|
||||
(`VESKSLOP_TTS_CMD`, shown as `VESKSLOP_TTS_NAME` or "custom engine"), and **HTTP API**
|
||||
(`VESKSLOP_TTS_API_URL`). Unavailable engines are disabled. Auto picks the command engine first,
|
||||
then espeak-ng, then the API.
|
||||
|
||||
Selecting an engine switches the voice list to that engine. The language selector is hidden for
|
||||
single-language engines such as the command engine; their voices come from `VESKSLOP_TTS_VOICES`.
|
||||
The TTS panel carries the same engine and voice selects plus the speed and pitch sliders.
|
||||
|
||||
**Start engine** in the Speech settings warms the selected engine by running one throwaway
|
||||
synthesis. For the command engine this starts Xvfb, Wine, and the engine host, so the first real
|
||||
line is fast. The result reports the elapsed milliseconds; espeak-ng and the HTTP API need no
|
||||
warm-up. The same action is available as `POST /api/audio/tts/warm` and the MCP
|
||||
`start_tts_engine` tool.
|
||||
|
||||
### Voices
|
||||
|
||||
The local engine is espeak-ng. `GET /api/audio/tts/voices` returns every base voice plus the
|
||||
espeak variants (`m1`–`m7`, `f1`–`f5`, `croak`, `whisper`) as `<voice>+<variant>`, so each language
|
||||
has roughly 15 voices. For natural voices, point `VESKSLOP_TTS_API_URL` at a network TTS (piper,
|
||||
OpenAI, ElevenLabs) and pick the `api` provider; `VESKSLOP_TTS_CMD` can also call any local engine
|
||||
that accepts `{out}`, `{text_file}`, and `{voice}`.
|
||||
|
||||
When `VESKSLOP_TTS_CMD` is set, the daemon hides the espeak list and shows only the voices the
|
||||
command declares through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or newlines and
|
||||
use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
|
||||
`id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (hide the pitch
|
||||
control) and `expression` / `expression=NN` (Flextalk Expression parameter; NN is the registry
|
||||
default). The external-engine contract, including speed, pitch, expression, and voice metadata, is
|
||||
in [TTS-DESIGN.md](../TTS-DESIGN.md). The bundled engine integration lives in
|
||||
[`microslopsam/`](../microslopsam/README.md).
|
||||
|
||||
### Speed, pitch, and expression
|
||||
|
||||
The TTS panel has a speed slider (0.5x–2.0x, 1.0x is the engine's natural rate) with 0.5x, 0.75x,
|
||||
1x, 1.25x, and 1.5x preset buttons, and a pitch slider (-12 to +12 semitones, 0 is natural). Both
|
||||
hide for voices whose engine does not support them. Flextalk voices (Keith Bell, Kerry Watson)
|
||||
also expose an **Expression** slider (0–100) that changes the voice character; it appears only for
|
||||
those voices and defaults to the engine's registry value until moved. Values are stored in the
|
||||
daemon preferences and travel to the provider per line; the command provider also receives
|
||||
`VESKSLOP_TTS_SPEED`, `VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION`. Absent values mean
|
||||
"engine default". `/api/audio/speak` and the MCP `speak` tool accept `speed`, `pitch`, and
|
||||
`expression` overrides for a single line.
|
||||
|
||||
### Flextalk voice profile (MicroslopSam)
|
||||
|
||||
Voices flagged `expression=NN` are Flextalk voices and open the **Flextalk voice profile** window
|
||||
from the TTS panel:
|
||||
|
||||
- **Voice character** - Pitch, Speaking rate, Prominences, Head size, and Lowering, each showing the
|
||||
voice's engine default and snapping to it within +/-2 units, plus Expression. A value equal to
|
||||
the default clears the override.
|
||||
- **Presets** - five named slots stored in `tts.profilePresets`; Save and Load copy the whole
|
||||
profile.
|
||||
- **Radio kit** - high-pass (`tts.highpass`, 40-400 Hz, default 40), mono echo (`tts.echo`, 0-100,
|
||||
default 0), and word mode (`tts.words`) with a gap in ms (`tts.wordGapMs`) for the radio/robot
|
||||
character.
|
||||
- **Test lines** - Attention, Laboratory, Critical, Health, and Radiation, plus two custom lines
|
||||
kept in `localStorage`. Each button speaks the line with the current profile, so a change can be
|
||||
auditioned without touching the real queue.
|
||||
|
||||
The window is movable and recenters every time it opens, so it cannot be lost off screen. Changed
|
||||
profile keys travel as `profile` (with `highpass`, `echo`, `words`, and `wordGapMs`) to the command
|
||||
provider, which exports `VESKSLOP_TTS_PROFILE`, `VESKSLOP_TTS_HIGHPASS`, `VESKSLOP_TTS_ECHO`,
|
||||
`VESKSLOP_TTS_WORDS`, and `VESKSLOP_TTS_WORD_GAP_MS`. The wrapper contract and the registry
|
||||
pitfalls (`wine reg import`; never `echo` for the `.reg` key path) are in
|
||||
[TTS-DESIGN.md](../TTS-DESIGN.md).
|
||||
|
||||
### Browser monitor (Client audio)
|
||||
|
||||
The header **Client audio** button streams the call to the browser:
|
||||
|
||||
- The daemon moves the client's playback streams into `veskslop_capture` (match on
|
||||
`application.name` / `application.process.binary`, default `vesktop,electron,chromium,discord`,
|
||||
`VESKSLOP_AUDIO_CLIENT_MATCH` overrides). Without the move the capture sink is silent. The adopt
|
||||
pass runs when devices are set up, when the monitor starts, and every 5 seconds while the monitor
|
||||
or the capture session runs.
|
||||
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
|
||||
the WebSocket at `/audio`.
|
||||
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
|
||||
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
|
||||
- Diagnose with `cd poc && pnpm audio:doctor`. It prints the default sink, the sink inputs and their
|
||||
sinks, the daemon audio status (`clientMatch`, `adopted`, `lastAdoptError`), and eight seconds of
|
||||
`VOICE_SPEAKING` / `VOICE_STATE_UPDATE` events. Run it during an active call.
|
||||
|
||||
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
|
||||
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
|
||||
@@ -98,8 +192,9 @@ the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling
|
||||
|
||||
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
||||
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
|
||||
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
|
||||
Yield (stop mid-line) is not implemented.
|
||||
`VOICE_SPEAKING` event. The TTS panel has a **duck on voice** toggle; it is stored in the daemon
|
||||
preferences and reported as `duckEnabled` by `/api/audio`. Staying quiet while someone else talks
|
||||
is a queue property, not a switch. Yield (stop mid-line) is not implemented.
|
||||
|
||||
### Capture and transcript
|
||||
|
||||
|
||||
+27
-7
@@ -64,9 +64,15 @@ With plain HTTP, `navigator.mediaDevices` only exists on localhost; remote brows
|
||||
- Right column: **In voice** (favourited users in a call, grouped by call), **Favourites**, and
|
||||
**Open DMs & groups** as quick access. The guild's own occupied channels live in the left rail.
|
||||
- Speaking users show animated level bars and a bold name (design 2c: audio → video → share marks).
|
||||
The bars scale up when speaking starts and stay on until an explicit stop, a mute, or a channel
|
||||
change. A missing stop event leaves them on by design; they do not decay on their own. The
|
||||
fade-out runs the equalizer inside a shrinking container, so the amplitude drops to nothing
|
||||
instead of the block collapsing at full height. Muted members never show bars.
|
||||
Nicknames come from the guild member store; DMs fall back to the username.
|
||||
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event. The
|
||||
Transcript tab lists captured segments (time, speaker, text, WAV) and updates live.
|
||||
Transcript tab lists captured segments (time, speaker, text, WAV) and updates live. It has
|
||||
**Start**, **Stop**, and **Refresh** buttons; the status line shows `capturing` or `capture off`
|
||||
plus the segment count.
|
||||
- Opening a chat fetches the latest 50 messages from Discord into the client cache. The refresh
|
||||
button by the channel header re-fetches. Live message events read the cache. Edited messages show
|
||||
an `edited` marker and replies show `↳ replying to <name>`.
|
||||
@@ -131,9 +137,19 @@ The Settings button opens a slide-over:
|
||||
- **Client** — a collapsible group with:
|
||||
- **Bot audio** — engine status, TTS provider state, a **Connect devices** button, and the
|
||||
capture toggle. These are the daemon host's devices.
|
||||
- **Speech** — the TTS engine (Auto / Local / API), language, and voice. Voices are grouped by
|
||||
base language and filtered by the chosen language; English, Swedish, German, Danish,
|
||||
Norwegian, Russian, and Cherokee are pinned on top. Stored in the daemon preferences.
|
||||
- **Speech** — the TTS engine (Auto plus every configured engine: espeak-ng, the command engine
|
||||
by its `VESKSLOP_TTS_NAME`, and the HTTP API), language, and voice. **Start engine** warms the
|
||||
selected engine (starts Xvfb, Wine, and the host for the command engine). Selecting an engine
|
||||
switches the voice list to that engine; the language row hides for single-language engines.
|
||||
Voices are grouped by base language and filtered by the chosen language; English, Swedish,
|
||||
German, Danish, Norwegian, Russian, and Cherokee are pinned on top. Stored in the daemon
|
||||
preferences. Speed (0.5x–2.0x with 0.5/0.75/1/1.25/1.5 presets), pitch (-12 to +12
|
||||
semitones), and an Expression slider (0–100, Flextalk voices only) live in the TTS panel and
|
||||
save to the same preferences; controls the selected voice does not support stay hidden. For
|
||||
Flextalk voices the TTS panel also opens the **Flextalk voice profile** window: a movable panel
|
||||
that recenters on each open, with character sliders, five named presets, radio-kit effects
|
||||
(high-pass, echo, word mode), and test lines (Attention, Laboratory, Critical, Health,
|
||||
Radiation, plus two custom) that speak immediately.
|
||||
- **Display** — the timestamp timezone. Defaults to the browser's zone
|
||||
(`Intl.DateTimeFormat().resolvedOptions().timeZone`); pick any IANA zone to override. Stored in
|
||||
`localStorage` and applied to messages, the transcript, the event log, the action log, and the
|
||||
@@ -141,7 +157,9 @@ The Settings button opens a slide-over:
|
||||
- **Browser audio** — puppeting controls: the browser output device for Client audio, a
|
||||
placeholder mic select, and the call-audio state. **Enable browser audio** asks for the
|
||||
microphone permission, which browsers require before they list output devices. Separate from
|
||||
the bot's devices.
|
||||
the bot's devices. Clicking **Client audio** in the header does the same: it asks for the
|
||||
permission, unlocks the audio context in the same click, and then starts listening. A second
|
||||
click stops it.
|
||||
- **Server** — a collapsible group with:
|
||||
- **Core process** — state, last exit, output, Start / Stop / Restart.
|
||||
- **Account login** — token file state, QR login, captcha.
|
||||
@@ -200,6 +218,8 @@ replaceable.
|
||||
| GET | `/api/dms` | `limit?` | Cached DM and group DM channels with recipients |
|
||||
| GET | `/api/voice` | `userId?`, `guildId?` | One user's voice state |
|
||||
| GET | `/api/voice/overview` | `guildId` | Occupied voice channels and members |
|
||||
| GET | `/api/voice/volume` | `userIds?` comma-separated | Per-user volumes plus output volume; defaults to the current call |
|
||||
| POST | `/api/voice/volume` | `userId`, `volume` | Set a user's playback volume (0 to 200) |
|
||||
| POST | `/api/voice/join` | `channelId`, `mute?` | Join a voice channel (muted by default) |
|
||||
| POST | `/api/voice/leave` | none | Leave the current voice channel |
|
||||
| POST | `/api/voice/move` | `channelId` | Move to another voice channel |
|
||||
@@ -207,11 +227,11 @@ replaceable.
|
||||
| POST | `/api/preferences` | `visibleGuilds?`, `tts?` | Update preferences. Stored in `<state dir>/preferences.json`. |
|
||||
| GET | `/api/audio` | none | Audio engine, TTS, queue, ducking, capture state |
|
||||
| GET | `/api/audio/devices` | none | The client's input and output devices |
|
||||
| GET | `/api/audio/tts/voices` | none | Local TTS voices (espeak-ng) |
|
||||
| GET | `/api/audio/tts/voices` | `engine?` | Voices for the selected engine (espeak-ng or `VESKSLOP_TTS_VOICES`) |
|
||||
| POST | `/api/audio/connect` | none | Re-apply the virtual devices to the client |
|
||||
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
|
||||
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
|
||||
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?` | Queue text to speech |
|
||||
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?`, `speed?`, `pitch?` | Queue text to speech |
|
||||
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
||||
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
||||
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
||||
|
||||
Reference in New Issue
Block a user