Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls
This commit is contained in:
+121
-1
@@ -1,6 +1,6 @@
|
||||
# CURRENT.md — session state
|
||||
|
||||
Last updated: 2026-09-14
|
||||
Last updated: 2026-09-15
|
||||
|
||||
This file tracks the work state. Read it first. Update it before you stop.
|
||||
|
||||
@@ -12,6 +12,17 @@ mobile draft at 390px.
|
||||
|
||||
## In progress
|
||||
|
||||
- Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
|
||||
open; it carries the character sliders with engine defaults and +/-2 snapping, five named
|
||||
presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and
|
||||
test lines (Attention, Laboratory, Critical, Health, Radiation, plus two custom lines in
|
||||
`localStorage`) that post the current profile straight to `/api/audio/speak`. Fixed the test
|
||||
buttons reading a stale expression from saved state; they now read the slider like Speak does.
|
||||
- Root cause of the profile never reaching the engine (2026-09-15): the wrapper wrote the Flextalk
|
||||
`.reg` with `echo`, and dash's `echo` treats `\2` in `...\Watson\2.0\...` as an escape, which
|
||||
corrupted the key so `wine reg import` was a silent no-op (it still returns 0). The wrapper now
|
||||
uses `printf`, validates that every profile value is numeric, and rejects a `.reg` containing
|
||||
control characters. Documented in TTS-DESIGN.md.
|
||||
- The Veskslop agent lives in `agent/` (Go 1.25+, Eino v0.9.19, remote llama.cpp over the
|
||||
OpenAI-compatible API, daemon tools over MCP Streamable HTTP). It is disabled by default
|
||||
(`VESKSLOP_AGENT_ENABLED=1`), takes tasks on `POST /task`, and logs every tool call. Smoke-tested
|
||||
@@ -24,6 +35,115 @@ mobile draft at 390px.
|
||||
390x844. Next: transcript and TTS box in the Call tab, per the 1c design.
|
||||
- Fixed a pre-existing web UI bug: `loadDms` called the nonexistent `renderDirect`, so the DM rail
|
||||
only rendered from other paths. It now calls `renderDmRail()`.
|
||||
- Fixed the speaking marks: the core turned Discord's numeric `speaking: 0` into `true`, so the
|
||||
equalizer bars never stopped. The bars now fade in on start and decay out on stop (500 ms), with
|
||||
a 2.5 s idle timeout for missed stops; a mute or channel change ends the state at once. Muted
|
||||
members show no bars.
|
||||
- Browser audio: **Client audio** now runs the browser unlock in the click (microphone permission,
|
||||
AudioContext created and resumed, output sink applied) before it starts listening. Verified
|
||||
headless: 52 PCM frames played through a running context. The button label no longer wipes the
|
||||
icon markup.
|
||||
- Capture routing: nothing moved the client's playback streams into `veskslop_capture`, so the
|
||||
browser monitor streamed digital silence. The audio manager now adopts matching streams
|
||||
(`VESKSLOP_AUDIO_CLIENT_MATCH`, default `vesktop,electron,chromium,discord`) on device setup,
|
||||
monitor start, and every 5 s while listening or capturing.
|
||||
- LAN audio verified end to end on 2026-09-14: a tone stream on `veskslop_tts` was moved to
|
||||
`veskslop_capture` (daemon log: `moved stream 81 to veskpop_capture`) and the browser, connected
|
||||
to `https://192.168.0.178:18801/`, played 256+ PCM frames with a peak of 4096 (a stream already
|
||||
on the capture sink peaked at 29274).
|
||||
- Speaking bars are always in the layout now: `scaleY(0)` and transparent when idle, they scale up
|
||||
with the fade-in at start, run the equalizer animation while speaking, and stay on until an
|
||||
explicit stop, a mute, or a channel change. Live data showed repeated `speaking: true` pulses and
|
||||
no stop events inside an 8 s window, so the idle decay is removed: bars stick rather than
|
||||
disappear. The fade runs the equalizer inside a shrinking container, so the amplitude drops
|
||||
smoothly instead of the block collapsing at full height. The core treats a missing `speaking`
|
||||
flag as a start pulse and accepts `user_id`. Core-side changes need a rebuild plus a client
|
||||
restart.
|
||||
- Added `poc/scripts/audio-doctor.sh` (`pnpm audio:doctor`): default sink, sink inputs, audio
|
||||
status with adoption counters, a 5 s `volumedetect` on `veskslop_capture.monitor`, and a live
|
||||
event peek. Run it in a real call to trace a silent browser monitor.
|
||||
- Live doctor output (2026-09-14): the `vesktop`/`electron` streams sit on `veskslop_capture`
|
||||
(sink 40) already, `adopted: 0`, and `VOICE_SPEAKING` events arrive. Client audio routing is
|
||||
therefore in place. Bot TTS works. Capture was off (`capturing: false`), so the transcript was
|
||||
empty; Vosk transcribes correctly when invoked with the model env.
|
||||
- Live call audio works now (the user confirmed TTS is audible and the silent call was partly a
|
||||
per-user volume set to zero for some members, plus another issue). The `volume` read returns
|
||||
`null` because this client build exposes no `UserVolumeStore`; the setter reports
|
||||
`NOT_SUPPORTED`. Volume is not a usable control here.
|
||||
- External TTS engine support (external project, first target is a command wrapper): the contract
|
||||
lives in `TTS-DESIGN.md`. Veskslop now has nullable `tts.speed` (0.5-2.0) and `tts.pitch`
|
||||
(-12..+12) preferences, per-line overrides on `/api/audio/speak` and MCP `speak`, `{speed}`/
|
||||
`{pitch}` placeholders, `VESKSLOP_TTS_SPEED`/`_PITCH` exports, and speed/pitch sliders in the TTS
|
||||
panel. A custom command advertises its voices through `VESKSLOP_TTS_VOICES`; the espeak list is
|
||||
hidden while a custom command is set. Verified with a wrapper that logged argv and env:
|
||||
preferences and overrides both arrive. Reading the engine's `GET /voices` for the HTTP provider
|
||||
remains future work.
|
||||
- The MicroslopSam engine project is now part of the tree in `microslopsam/` (copied from the
|
||||
separate project; its `build/` output and voice data stay ignored). The wrapper was renamed
|
||||
`setup/vekslop-tts.sh` -> `setup/veskslop-tts.sh` and every reference and configuration export
|
||||
was fixed. The engine also corrected the Flextalk pitch finding (REG_DWORD bug), so Keith Bell
|
||||
and Kerry Watson now support pitch and carry no `no-pitch` flag.
|
||||
- Implemented the Flextalk Expression profile parameter end to end: `VESKSLOP_TTS_VOICES` marks the
|
||||
two voices as `expression=59` / `expression=100`; the wrapper writes the registry value before
|
||||
synthesis; Veskslop has `tts.expression` (0-100 nullable), the `{expression}` placeholder,
|
||||
`VESKSLOP_TTS_EXPRESSION`, an HTTP/MCP field, and an Expression slider that appears only for
|
||||
those voices (default shown until moved, null leaves the registry untouched). Verified: 0/59/100
|
||||
produce distinct audio, the registry reaches 0x14 for 20, and the UI shows 25/100/default states.
|
||||
- Fixed the null-value coercion in the preference normalizers (`Number(null) === 0` turned a
|
||||
cleared speed/expression into 0.5/0); unset now stays null.
|
||||
- Renderer-down handling polished: the bridge logs one warning per outage instead of a stack per
|
||||
polled command, and the `navigator.languages` request catches the new timeout rejection instead
|
||||
of raising an unhandled promise warning. (The `exitCode: 9` renderer kills in the live log were
|
||||
client restarts; the daemon's retry handled the startup race as designed.)
|
||||
- The TTS panel has speed presets (0.5x, 0.75x, 1x, 1.25x, 1.5x) under the fine-adjust slider, and
|
||||
the pitch control is hidden entirely for voices that report `supportsPitch: false`. (Flextalk
|
||||
voices no longer report that; see the pitch correction above.) Verified: preset clicks update the
|
||||
slider/label and save.
|
||||
- TTS playback has a lead-in of silence before each line (`VESKSLOP_TTS_LEAD_MS`, default 150 ms,
|
||||
max 2000, added with `adelay` in `ffplay`; status reports `ttsLeadMs`) so Discord's voice
|
||||
pipeline is settled before the first syllable. Verified on the sink monitor: first sound at
|
||||
0.02 s with 0 ms and 0.62 s with 600 ms.
|
||||
- Speech settings have a **Start engine** button that warms the selected engine
|
||||
(`POST /api/audio/tts/warm`, MCP `start_tts_engine`): it runs one throwaway synthesis so the
|
||||
command engine's Xvfb/Wine/host are up. Verified with MicroslopSam: cold 1732 ms, warm 505 ms,
|
||||
and the UI shows "MicroslopSam ready in 835 ms". The warm text is a full sentence because their
|
||||
two-pass `loudnorm` fails with `inf` on very short input.
|
||||
- Startup and crash robustness: `sendRendererCommand` now rejects immediately when the client
|
||||
renderer is destroyed or crashed, and times out after 15 s, so a dead frame no longer logs
|
||||
Electron's "Render frame was disposed" stack once per poll. Device auto-connect retries up to
|
||||
10 times every 2.5 s, because the bridge socket is ready before the renderer's modules load
|
||||
(the live error was `Reflect.get called on non-object` from a lazy lookup at startup). Verified
|
||||
with the mock: attempt 1 failed, attempt 2 set the devices.
|
||||
- TTS engines are selectable: the Speech settings and the TTS panel list espeak-ng, the command
|
||||
engine (`VESKSLOP_TTS_NAME`, default "custom engine"), and the HTTP API; unavailable entries are
|
||||
disabled and Auto resolves to the command engine first. Selecting an engine switches the voice
|
||||
list (`/api/audio/tts/voices?engine=`) and hides the language row for single-language engines.
|
||||
The TTS panel now has a voice select next to the speed/pitch sliders. Fixed a slider bug where a
|
||||
null speed preference (`Number(null) === 0`) clamped the slider to 0.5 and then persisted it.
|
||||
- TTS controls: the TTS panel has a working **duck on voice** toggle (stored in preferences,
|
||||
reported as `duckEnabled`). Ducking was verified live with the mock: the `veskslop_tts` sink
|
||||
drops to 30% while TTS plays and someone speaks, and returns to 100% after. The Transcript tab
|
||||
in the console has **Start** and **Stop** buttons; the status line shows `capturing`/`capture
|
||||
off`. The espeak-ng voice list now includes the `m1`–`m7`, `f1`–`f5`, `croak`, and `whisper`
|
||||
variants (`<voice>+<variant>`), so each language has about 15 voices instead of one.
|
||||
- Speaking state is now authoritative: the daemon polls the core's `voice.speaking` action
|
||||
(`ChannelRTCStore.getSpeakingParticipants`) every second and synthesizes the missing stop/start
|
||||
events; dispatcher `VOICE_SPEAKING` hints feed the same set so stale users clear. This fixes the
|
||||
bars sticking for everyone and the transcript attribution, which credited the first member
|
||||
because speaker intervals never closed. Verified with the mock: a `speaking: true` event cleared
|
||||
within one poll.
|
||||
- The client streams sit uncorked, unmuted, and at 100% on `veskslop_capture`, yet the monitor is
|
||||
silent. Gateway `VOICE_SPEAKING` events do not prove inbound RTP, and the TTS path only proves
|
||||
outbound, so the remaining suspect is the client's inbound voice state (stuck deafen or a broken
|
||||
voice connection), not the sink routing. `/health` now returns `selfId` so the bot's own member
|
||||
flags can be picked out of the voice member list; the doctor prints it.
|
||||
- Client output is still silent at the sink (`-91 dB`) although the streams are uncorked, unmuted,
|
||||
and at 100%, and a headless Chromium tone into the same sink measured `-7.4 dB`. Added per-user
|
||||
volume reading and writing to find Discord-side silencing: `user.volume` / `user.setVolume` in
|
||||
the core, `/api/voice/volume` (GET and POST), MCP `get_user_volume` / `set_user_volume`, and
|
||||
`volume` in every voice member. A volume of 0 (or an output volume of 0) silences a user even
|
||||
when Pulse shows a healthy stream. `voice.join` also clears persisted self-deafen now.
|
||||
The doctor prints output and per-user volumes.
|
||||
- Voice audio is implemented: daemon-managed virtual devices (`veskslop_tts`, `veskslop_mic`,
|
||||
`veskslop_capture`), TTS providers (local command + HTTP API), queued playback through `ffplay`,
|
||||
ducking on `VOICE_SPEAKING`, and WAV capture. The daemon starts `pulseaudio` when `pactl` cannot
|
||||
|
||||
Reference in New Issue
Block a user