Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls

This commit is contained in:
Fisto
2026-09-15 15:09:23 +02:00
parent eab94132bb
commit 302436f95f
42 changed files with 4675 additions and 180 deletions
+121 -1
View File
@@ -1,6 +1,6 @@
# CURRENT.md — session state
Last updated: 2026-09-14
Last updated: 2026-09-15
This file tracks the work state. Read it first. Update it before you stop.
@@ -12,6 +12,17 @@ mobile draft at 390px.
## In progress
- Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
open; it carries the character sliders with engine defaults and +/-2 snapping, five named
presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and
test lines (Attention, Laboratory, Critical, Health, Radiation, plus two custom lines in
`localStorage`) that post the current profile straight to `/api/audio/speak`. Fixed the test
buttons reading a stale expression from saved state; they now read the slider like Speak does.
- Root cause of the profile never reaching the engine (2026-09-15): the wrapper wrote the Flextalk
`.reg` with `echo`, and dash's `echo` treats `\2` in `...\Watson\2.0\...` as an escape, which
corrupted the key so `wine reg import` was a silent no-op (it still returns 0). The wrapper now
uses `printf`, validates that every profile value is numeric, and rejects a `.reg` containing
control characters. Documented in TTS-DESIGN.md.
- The Veskslop agent lives in `agent/` (Go 1.25+, Eino v0.9.19, remote llama.cpp over the
OpenAI-compatible API, daemon tools over MCP Streamable HTTP). It is disabled by default
(`VESKSLOP_AGENT_ENABLED=1`), takes tasks on `POST /task`, and logs every tool call. Smoke-tested
@@ -24,6 +35,115 @@ mobile draft at 390px.
390x844. Next: transcript and TTS box in the Call tab, per the 1c design.
- Fixed a pre-existing web UI bug: `loadDms` called the nonexistent `renderDirect`, so the DM rail
only rendered from other paths. It now calls `renderDmRail()`.
- Fixed the speaking marks: the core turned Discord's numeric `speaking: 0` into `true`, so the
equalizer bars never stopped. The bars now fade in on start and decay out on stop (500 ms), with
a 2.5 s idle timeout for missed stops; a mute or channel change ends the state at once. Muted
members show no bars.
- Browser audio: **Client audio** now runs the browser unlock in the click (microphone permission,
AudioContext created and resumed, output sink applied) before it starts listening. Verified
headless: 52 PCM frames played through a running context. The button label no longer wipes the
icon markup.
- Capture routing: nothing moved the client's playback streams into `veskslop_capture`, so the
browser monitor streamed digital silence. The audio manager now adopts matching streams
(`VESKSLOP_AUDIO_CLIENT_MATCH`, default `vesktop,electron,chromium,discord`) on device setup,
monitor start, and every 5 s while listening or capturing.
- LAN audio verified end to end on 2026-09-14: a tone stream on `veskslop_tts` was moved to
`veskslop_capture` (daemon log: `moved stream 81 to veskpop_capture`) and the browser, connected
to `https://192.168.0.178:18801/`, played 256+ PCM frames with a peak of 4096 (a stream already
on the capture sink peaked at 29274).
- Speaking bars are always in the layout now: `scaleY(0)` and transparent when idle, they scale up
with the fade-in at start, run the equalizer animation while speaking, and stay on until an
explicit stop, a mute, or a channel change. Live data showed repeated `speaking: true` pulses and
no stop events inside an 8 s window, so the idle decay is removed: bars stick rather than
disappear. The fade runs the equalizer inside a shrinking container, so the amplitude drops
smoothly instead of the block collapsing at full height. The core treats a missing `speaking`
flag as a start pulse and accepts `user_id`. Core-side changes need a rebuild plus a client
restart.
- Added `poc/scripts/audio-doctor.sh` (`pnpm audio:doctor`): default sink, sink inputs, audio
status with adoption counters, a 5 s `volumedetect` on `veskslop_capture.monitor`, and a live
event peek. Run it in a real call to trace a silent browser monitor.
- Live doctor output (2026-09-14): the `vesktop`/`electron` streams sit on `veskslop_capture`
(sink 40) already, `adopted: 0`, and `VOICE_SPEAKING` events arrive. Client audio routing is
therefore in place. Bot TTS works. Capture was off (`capturing: false`), so the transcript was
empty; Vosk transcribes correctly when invoked with the model env.
- Live call audio works now (the user confirmed TTS is audible and the silent call was partly a
per-user volume set to zero for some members, plus another issue). The `volume` read returns
`null` because this client build exposes no `UserVolumeStore`; the setter reports
`NOT_SUPPORTED`. Volume is not a usable control here.
- External TTS engine support (external project, first target is a command wrapper): the contract
lives in `TTS-DESIGN.md`. Veskslop now has nullable `tts.speed` (0.5-2.0) and `tts.pitch`
(-12..+12) preferences, per-line overrides on `/api/audio/speak` and MCP `speak`, `{speed}`/
`{pitch}` placeholders, `VESKSLOP_TTS_SPEED`/`_PITCH` exports, and speed/pitch sliders in the TTS
panel. A custom command advertises its voices through `VESKSLOP_TTS_VOICES`; the espeak list is
hidden while a custom command is set. Verified with a wrapper that logged argv and env:
preferences and overrides both arrive. Reading the engine's `GET /voices` for the HTTP provider
remains future work.
- The MicroslopSam engine project is now part of the tree in `microslopsam/` (copied from the
separate project; its `build/` output and voice data stay ignored). The wrapper was renamed
`setup/vekslop-tts.sh` -> `setup/veskslop-tts.sh` and every reference and configuration export
was fixed. The engine also corrected the Flextalk pitch finding (REG_DWORD bug), so Keith Bell
and Kerry Watson now support pitch and carry no `no-pitch` flag.
- Implemented the Flextalk Expression profile parameter end to end: `VESKSLOP_TTS_VOICES` marks the
two voices as `expression=59` / `expression=100`; the wrapper writes the registry value before
synthesis; Veskslop has `tts.expression` (0-100 nullable), the `{expression}` placeholder,
`VESKSLOP_TTS_EXPRESSION`, an HTTP/MCP field, and an Expression slider that appears only for
those voices (default shown until moved, null leaves the registry untouched). Verified: 0/59/100
produce distinct audio, the registry reaches 0x14 for 20, and the UI shows 25/100/default states.
- Fixed the null-value coercion in the preference normalizers (`Number(null) === 0` turned a
cleared speed/expression into 0.5/0); unset now stays null.
- Renderer-down handling polished: the bridge logs one warning per outage instead of a stack per
polled command, and the `navigator.languages` request catches the new timeout rejection instead
of raising an unhandled promise warning. (The `exitCode: 9` renderer kills in the live log were
client restarts; the daemon's retry handled the startup race as designed.)
- The TTS panel has speed presets (0.5x, 0.75x, 1x, 1.25x, 1.5x) under the fine-adjust slider, and
the pitch control is hidden entirely for voices that report `supportsPitch: false`. (Flextalk
voices no longer report that; see the pitch correction above.) Verified: preset clicks update the
slider/label and save.
- TTS playback has a lead-in of silence before each line (`VESKSLOP_TTS_LEAD_MS`, default 150 ms,
max 2000, added with `adelay` in `ffplay`; status reports `ttsLeadMs`) so Discord's voice
pipeline is settled before the first syllable. Verified on the sink monitor: first sound at
0.02 s with 0 ms and 0.62 s with 600 ms.
- Speech settings have a **Start engine** button that warms the selected engine
(`POST /api/audio/tts/warm`, MCP `start_tts_engine`): it runs one throwaway synthesis so the
command engine's Xvfb/Wine/host are up. Verified with MicroslopSam: cold 1732 ms, warm 505 ms,
and the UI shows "MicroslopSam ready in 835 ms". The warm text is a full sentence because their
two-pass `loudnorm` fails with `inf` on very short input.
- Startup and crash robustness: `sendRendererCommand` now rejects immediately when the client
renderer is destroyed or crashed, and times out after 15 s, so a dead frame no longer logs
Electron's "Render frame was disposed" stack once per poll. Device auto-connect retries up to
10 times every 2.5 s, because the bridge socket is ready before the renderer's modules load
(the live error was `Reflect.get called on non-object` from a lazy lookup at startup). Verified
with the mock: attempt 1 failed, attempt 2 set the devices.
- TTS engines are selectable: the Speech settings and the TTS panel list espeak-ng, the command
engine (`VESKSLOP_TTS_NAME`, default "custom engine"), and the HTTP API; unavailable entries are
disabled and Auto resolves to the command engine first. Selecting an engine switches the voice
list (`/api/audio/tts/voices?engine=`) and hides the language row for single-language engines.
The TTS panel now has a voice select next to the speed/pitch sliders. Fixed a slider bug where a
null speed preference (`Number(null) === 0`) clamped the slider to 0.5 and then persisted it.
- TTS controls: the TTS panel has a working **duck on voice** toggle (stored in preferences,
reported as `duckEnabled`). Ducking was verified live with the mock: the `veskslop_tts` sink
drops to 30% while TTS plays and someone speaks, and returns to 100% after. The Transcript tab
in the console has **Start** and **Stop** buttons; the status line shows `capturing`/`capture
off`. The espeak-ng voice list now includes the `m1`–`m7`, `f1`–`f5`, `croak`, and `whisper`
variants (`<voice>+<variant>`), so each language has about 15 voices instead of one.
- Speaking state is now authoritative: the daemon polls the core's `voice.speaking` action
(`ChannelRTCStore.getSpeakingParticipants`) every second and synthesizes the missing stop/start
events; dispatcher `VOICE_SPEAKING` hints feed the same set so stale users clear. This fixes the
bars sticking for everyone and the transcript attribution, which credited the first member
because speaker intervals never closed. Verified with the mock: a `speaking: true` event cleared
within one poll.
- The client streams sit uncorked, unmuted, and at 100% on `veskslop_capture`, yet the monitor is
silent. Gateway `VOICE_SPEAKING` events do not prove inbound RTP, and the TTS path only proves
outbound, so the remaining suspect is the client's inbound voice state (stuck deafen or a broken
voice connection), not the sink routing. `/health` now returns `selfId` so the bot's own member
flags can be picked out of the voice member list; the doctor prints it.
- Client output is still silent at the sink (`-91 dB`) although the streams are uncorked, unmuted,
and at 100%, and a headless Chromium tone into the same sink measured `-7.4 dB`. Added per-user
volume reading and writing to find Discord-side silencing: `user.volume` / `user.setVolume` in
the core, `/api/voice/volume` (GET and POST), MCP `get_user_volume` / `set_user_volume`, and
`volume` in every voice member. A volume of 0 (or an output volume of 0) silences a user even
when Pulse shows a healthy stream. `voice.join` also clears persisted self-deafen now.
The doctor prints output and per-user volumes.
- Voice audio is implemented: daemon-managed virtual devices (`veskslop_tts`, `veskslop_mic`,
`veskslop_capture`), TTS providers (local command + HTTP API), queued playback through `ffplay`,
ducking on `VOICE_SPEAKING`, and WAV capture. The daemon starts `pulseaudio` when `pactl` cannot