Add MicroslopSam TTS, Flextalk voice profiles, effects, and engine controls
This commit is contained in:
Fisto
@@ -0,0 +1,99 @@
|
||||
# Notes
|
||||
|
||||
## Registry facts
|
||||
|
||||
From `spchapi/SPCHAPI.INF` (`%49000%` = `%25%\speech` = `C:\windows\speech`):
|
||||
|
||||
- `CLSID_TTSEnumerator` `{D67C0280-C743-11cd-80E5-00AA003E4B50}` -> `Speech.dll`
|
||||
- "Audio Destination to File Object" `{D4623720-E4B9-11cf-8D56-00A0C9034A7E}` -> `Speech.dll`
|
||||
- "Audio Destination Object" `{CB96B400-C743-11cd-80E5-00AA003E4B50}` -> `Speech.dll`
|
||||
- Lexicon dialog `{92655FB1-ADF9-11d1-BEB9-006008317CE8}` -> `Speech.dll`
|
||||
- Phoneme converter `{B9F11A96-90E3-11d0-8D77-00A0C9034A7E}` -> `Speech.dll`
|
||||
|
||||
From `msttsl/mstts.inf` (`[TTSReg]`):
|
||||
|
||||
- Engine `MSTTSSyn` = `{E0725551-286F-11d0-8E73-00A0C9083363}` -> `MSTTSSYN.dll`
|
||||
- Engine install dir: `Common Files\microsoft shared\SpeechEngines\TTS`.
|
||||
- Enumerates voices by scanning `*.vce` in its own directory.
|
||||
|
||||
## Flextalk registration (from the installer's `setup.ins`)
|
||||
|
||||
- Engine value: `HKLM\Software\Voice\TextToSpeech\Engine`
|
||||
`"AT&T FlexTalk" = "{FFE78E00-C69C-11CE-AEC1-0000C0E9CB87}"`
|
||||
- Engine subkey (forces mode enumeration): `...\Engine\{FFE78E00-...}` with
|
||||
`Dirty = 1` (REG_DWORD). The runtime rewrites `Count` + `Mode0..N` and
|
||||
clears `Dirty` after enumerating.
|
||||
- `HKCR\CLSID\{FFE78E00-C69C-11CE-AEC1-0000C0E9CB87}\InprocServer32` =
|
||||
`<root>\EngineDLLs\FlexTalk.dll`, `ThreadingModel = Apartment`.
|
||||
Also registered: `{778EAAE2-499F-11CF-B0BA-444553540000}` -> same DLL.
|
||||
- Directories: `HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Directories`
|
||||
`RootDirectory = C:\FlexTalk\`.
|
||||
- Dictionaries: `...\FlexTalk\2.0\Dictionaries`:
|
||||
`StandardDictionary="2 ft_sdict.dll"`, `ExceptionDictionary="0 ft_udict.dll"`,
|
||||
`HomographDictionary="1 ft_hgrap.dll"`, `ProperNameDictionary="3 ft_namsa.dll"`,
|
||||
`LetterToSoundDictionary="4 ft_l2s.dll"`.
|
||||
- Options: `...\FlexTalk\2.0\Options` `Gender=Male`, `SampleRate=11025`.
|
||||
- Speakers (per-voice settings; names come from these values). **All numeric
|
||||
values must be REG_DWORD**: with REG_SZ the engine reads garbage and
|
||||
synthesizes a monotone buzz (pitch fixed at max, pitch control dead).
|
||||
- `{11260610-C84A-11ce-AEC4-0000C0E9CB87}`: Mode Name / Speaker Name
|
||||
"Keith Bell", Pitch 27, Speaking Rate 33, Head Size 46, Expression 59,
|
||||
Lowering 40, Prominences 17, Age 30, Volume 100.
|
||||
- `{000031CB-3699-551F-AED0-6B4893382D57}`: "Kerry Watson", Pitch 65,
|
||||
Speaking Rate 32, Head Size 45, Expression 100, Lowering 40,
|
||||
Prominences 17, Age 30, Volume 100.
|
||||
- Voice data: `EngineData\AE1M08LC.*`, `AE1M11LC.*` (male, 8/11 kHz) and
|
||||
`AE2F08LC.*`, `AE2F11LC.*` (female); `.INV` inventory, `.NDX` index,
|
||||
`.COD` codebook, plus shared `DICT_CI.*`, `phondef`, `AMPRULES`, `ENDINGS`.
|
||||
|
||||
Extraction: `unshield -O -d extracted x installer/flextalk/data1.cab`
|
||||
(`-O` = old compression; the archive.org package has no `data2.cab`, so the
|
||||
final file `WatsonUNInstall.dll` is incomplete).
|
||||
|
||||
## SAPI4 runtime engine discovery (`speech.dll`)
|
||||
|
||||
- `FUN_635747a4` reads `HKLM\Software\Voice\TextToSpeech\Engine` values
|
||||
(value name = engine name, data = CLSID string), validates the GUID with
|
||||
`FUN_635712e4`, then opens a *subkey named by the CLSID string*.
|
||||
- `FUN_635758b1` (TTS mode-list ctor) reads `Dirty`; when non-zero/missing it
|
||||
calls `CoCreateInstance(CLSID, ..., IID @635791c0)` and pulls modes via
|
||||
vtable+0xC into 0xaf0-byte records, writes `Mode%d` (REG_BINARY), `Count`,
|
||||
then clears `Dirty`. When `Dirty=0` it reads the cached `Mode%d` blobs.
|
||||
- Practical rule: to add an engine, register the `Engine` value + the CLSID
|
||||
`InprocServer32` + an engine subkey with `Dirty=1` (a missing subkey also
|
||||
triggers enumeration).
|
||||
|
||||
## Environment gotchas
|
||||
|
||||
- Wine needs a working X display (the runtime creates windows). `xvfb-run -a`
|
||||
proved flaky here; `setup/run.sh` starts a persistent `Xvfb :77` instead.
|
||||
- Wine reports `err:ole:apartment_add_dll couldn't load in-process dll` when
|
||||
the `InprocServer32` path does not exist. Check the file copy first
|
||||
(this bit us: a bad `unshield` run created empty directories).
|
||||
- Generate registry `.reg` files with `printf`, never `echo`: dash's `echo`
|
||||
interprets `\2` in `...\Watson\2.0\...` as 0x02, corrupting the key so that
|
||||
`wine reg import` is a silent no-op (it still returns 0). The wrapper rejects
|
||||
a generated profile containing control characters.
|
||||
- 32-bit engines require a 32-bit prefix (`WINEARCH=win32`).
|
||||
- `sudo` on this machine requires a password; installs are manual.
|
||||
|
||||
## Engine units (measured)
|
||||
|
||||
- speed: MSTTS range 30-450, Flextalk 50-390; the natural rate is about 150 for every
|
||||
voice (duration at 100/200/300 fits a natural value near 140-150).
|
||||
- pitch: engine units, roughly linear in F0 near the natural point (~1.1 Hz per unit).
|
||||
Natural values: about 100 (MSTTS), 106 (Keith Bell), 186 (Kerry Watson); Flextalk max 250.
|
||||
Verified with the corrected DWORD registry: Kerry 0 st = 209 Hz, -6 st = 148 Hz,
|
||||
+6 st = 250 Hz (clamped at max); Keith 0 st = 119 Hz, +6 st = 170 Hz.
|
||||
- The wrapper maps semitones as `natural * 2^(st/12)` and clamps to the range.
|
||||
- volume: available through `ITTSAttributes` (host flag `-V`). Flextalk is linear
|
||||
(0-65535, default 65535; half is about -6 dB). MSTTS is non-linear (0 = silence,
|
||||
4096 about -24 dB, 65535+ full scale); treat it as on/off.
|
||||
- Flextalk per-speaker profile values are applied at voice load; `Expression` was
|
||||
verified audible (0/59/100 produce different waveforms at equal duration). Changing
|
||||
them requires a registry edit and re-selecting the voice.
|
||||
|
||||
## Decompiled corpora (Ghidra)
|
||||
|
||||
- `/mnt/ghidra/decomp/msttssyn.dll/` (786 functions, engine)
|
||||
- `/mnt/ghidra/decomp/speech.dll/` (1030 functions, SAPI4 runtime)
|
||||
@@ -0,0 +1,477 @@
|
||||
# TTS-DESIGN.md — building a TTS engine for Veskslop
|
||||
|
||||
This document is the integration contract for a text-to-speech engine that Veskslop will use. It is
|
||||
written to be self-contained: an engine developer can work from this file alone.
|
||||
|
||||
## How Veskslop uses TTS
|
||||
|
||||
1. A line of text is queued (`POST /api/audio/speak` or the MCP `speak` tool).
|
||||
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
|
||||
3. `ffplay` decodes the file and plays it into the PulseAudio null sink `veskslop_tts`.
|
||||
4. `veskslop_tts.monitor` is exposed as the source `veskslop_mic`.
|
||||
5. The Vesktop client uses `veskslop_mic` as its microphone, so the call hears the speech.
|
||||
6. While TTS plays and another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
||||
30% and restores it after. This is "ducking". The engine does not implement ducking.
|
||||
|
||||
Lines are serialized: one synthesis, then playback, then the next line. There is no streaming
|
||||
playback path and no partial-audio protocol. The engine returns a complete audio object.
|
||||
|
||||
## Integration modes
|
||||
|
||||
### HTTP provider (recommended)
|
||||
|
||||
Set `VESKSLOP_TTS_API_URL`. Select the API provider with `VESKSLOP_TTS_PROVIDER=api` or the
|
||||
**API TTS** choice in the TTS panel. The daemon sends:
|
||||
|
||||
```
|
||||
POST <VESKSLOP_TTS_API_URL>
|
||||
Content-Type: application/json
|
||||
Authorization: Bearer <VESKSLOP_TTS_API_KEY> (only when the key is set)
|
||||
|
||||
{ "text": "Hello there.", "voice": "anna" }
|
||||
```
|
||||
|
||||
The response body must be the audio bytes. `Content-Type` selects the file extension: `audio/mpeg`
|
||||
or `audio/mp3` → `.mp3`, `audio/ogg` → `.ogg`, `audio/wav` or `audio/x-wav` → `.wav`, anything else
|
||||
is treated as `.mp3`. Prefer `audio/wav`.
|
||||
|
||||
A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs
|
||||
it and drops the line.
|
||||
|
||||
Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the
|
||||
per-line process start cost.
|
||||
|
||||
### Local command provider
|
||||
|
||||
Set `VESKSLOP_TTS_CMD`. It takes precedence over the built-in espeak-ng engine. Veskslop replaces
|
||||
these placeholders, splits the string on whitespace, and runs the result directly (no shell):
|
||||
|
||||
| Placeholder | Replaced with |
|
||||
| --- | --- |
|
||||
| `{out}` | Output file path, extension `.wav`. Required. |
|
||||
| `{text_file}` | A UTF-8 text file holding the line. |
|
||||
| `{voice}` | The selected voice ID, or an empty string. |
|
||||
| `{speed}` | The rate multiplier, or an empty string when unset. |
|
||||
| `{pitch}` | The pitch offset in semitones, or an empty string when unset. |
|
||||
| `{expression}` | The Flextalk Expression profile value, or an empty string when unset. |
|
||||
|
||||
The process also receives `VESKSLOP_TTS_VOICE`, `VESKSLOP_TTS_SPEED`,
|
||||
`VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION` when those are set. Write the audio to
|
||||
`{out}` and exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it
|
||||
short.
|
||||
|
||||
Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and
|
||||
shell tricks cannot work. Use a small wrapper script if you need options.
|
||||
|
||||
## Audio output requirements
|
||||
|
||||
| Property | Recommendation | Notes |
|
||||
| --- | --- | --- |
|
||||
| Container | WAV (RIFF) | Any format ffmpeg decodes works, but WAV avoids decoder startup cost. |
|
||||
| Codec | PCM signed 16-bit little-endian | `pcm_s16le`. |
|
||||
| Sample rate | 48000 Hz | 44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample. |
|
||||
| Channels | Mono | Discord voice is effectively mono. Stereo is accepted but adds no value. |
|
||||
| Duration | One line of speech | No intro/outro music, no chimes. |
|
||||
| Peak level | About -3 dBFS | Leave headroom; ducking and Discord gain must not clip. |
|
||||
| Loudness | Consistent across lines | Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume. |
|
||||
| Silence | Trim start to under 100 ms, keep roughly 20-50 ms of padding | Long leading silence delays the first audible word. |
|
||||
| Edges | 5-10 ms fade in and out | Prevents clicks. |
|
||||
| DC offset | None | Center the waveform. |
|
||||
|
||||
Veskslop does not normalize, limit, or re-encode the audio. Playback is `ffplay` at 100% sink
|
||||
volume, so the engine's levels are what Discord hears.
|
||||
|
||||
## Voices
|
||||
|
||||
A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such
|
||||
as `anna`, `sv-anna`, or `en-male-2`. The ID can contain letters, digits, `-`, `_`, and `+`.
|
||||
|
||||
If the engine has multiple voices, expose them so a UI can list them:
|
||||
|
||||
```
|
||||
GET /voices
|
||||
[
|
||||
{
|
||||
"id": "anna",
|
||||
"name": "Anna",
|
||||
"language": "sv",
|
||||
"gender": "female",
|
||||
"speed": { "min": 0.5, "max": 2.0, "default": 1.0 },
|
||||
"pitch": { "min": -12, "max": 12, "default": 0, "unit": "semitones" }
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after
|
||||
Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set
|
||||
the voice through the engine's own configuration and leave the Veskslop voice setting empty.
|
||||
|
||||
## Speed and pitch
|
||||
|
||||
Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only `text`,
|
||||
`voice`, and `provider`. No UI or preference exists for speed or pitch.
|
||||
|
||||
Recommended semantics for the engine:
|
||||
|
||||
- **speed**: a multiplier, `1.0` is normal, useful range `0.5` to `2.0`. Do not change pitch when
|
||||
changing speed unless the user asks for it.
|
||||
- **pitch**: semitones, `0` is the voice's natural pitch, useful range `-12` to `+12`. State the
|
||||
unit in the `/voices` response so clients do not guess.
|
||||
|
||||
Recommended HTTP shape (extras are optional; an engine may ignore them):
|
||||
|
||||
```
|
||||
{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }
|
||||
```
|
||||
|
||||
Recommended command shape: `{speed}` and `{pitch}` are substituted when set, and the same values
|
||||
are exported as `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH`. Read either; the environment form
|
||||
survives placeholder-free commands.
|
||||
|
||||
### How Veskslop should add first-class settings
|
||||
|
||||
Implemented on the Veskslop side (2026-09-15):
|
||||
|
||||
- `tts` preferences carry `speed` (0.5–2.0) and `pitch` (-12–+12, semitones), both nullable.
|
||||
- `/api/audio/speak` and the MCP `speak` tool accept optional `speed` and `pitch` that override
|
||||
the preferences for that line.
|
||||
- The command provider substitutes `{speed}`/`{pitch}`/`{expression}` and exports
|
||||
`VESKSLOP_TTS_SPEED` / `VESKSLOP_TTS_PITCH` / `VESKSLOP_TTS_EXPRESSION` only when the value is set.
|
||||
- The HTTP provider adds `speed`, `pitch`, and `expression` to the request body only when set.
|
||||
- The TTS panel has speed presets, speed/pitch sliders, and an Expression slider that appears only
|
||||
for voices that declare it (`expression` / `expression=NN` flags). Changes save to preferences.
|
||||
- Command mode advertises voices through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or
|
||||
newlines and use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
|
||||
`id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (engine ignores
|
||||
pitch) and `expression` / `expression=NN` (Flextalk profile parameter with its registry default).
|
||||
`/api/audio/tts/voices` reports `supportsPitch` and `supportsExpression` /
|
||||
`expressionDefault`; the TTS panel hides controls the voice does not support. While a custom
|
||||
command is set, the engine's declared voices replace the espeak list. Reading `/voices` for the
|
||||
HTTP provider is still future work.
|
||||
|
||||
Do not encode settings into the voice string (for example `anna+speed1.2`). It breaks voice lists
|
||||
and is hard to validate.
|
||||
|
||||
## Text handling
|
||||
|
||||
- UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
|
||||
- Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
|
||||
- Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine
|
||||
should not fail on an unknown character; skip or verbalize it as it prefers.
|
||||
- Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints
|
||||
for pacing.
|
||||
|
||||
## Errors, limits, and behavior
|
||||
|
||||
- One request = one complete audio object. No chunked playback, no callbacks.
|
||||
- Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget
|
||||
if the engine is slower. A warm HTTP server is the way to meet this.
|
||||
- Requests are serialized by Veskslop, so the engine does not need a request queue, but it should
|
||||
tolerate repeated back-to-back requests.
|
||||
- The daemon never retries a failed line. Make failures visible in the response and stderr.
|
||||
- Do not write to the `veskslop_tts` sink or the `veskslop_mic` source directly. The daemon owns
|
||||
playback routing.
|
||||
|
||||
## Optional: caching
|
||||
|
||||
Veskslop does not cache audio. The engine may cache by `hash(text + voice + speed + pitch)` and
|
||||
return the cached file or bytes. This is useful for repeated lines and for expensive models.
|
||||
|
||||
## Veskslop changes needed
|
||||
|
||||
Status as of 2026-09-15:
|
||||
|
||||
1. `speed` and `pitch` are first-class: preferences, per-request overrides on `/api/audio/speak`
|
||||
and MCP `speak`, env exports and placeholders for the command provider, request fields for the
|
||||
HTTP provider, and two sliders in the TTS panel.
|
||||
2. Command-mode voices come from `VESKSLOP_TTS_VOICES`. Reading the engine's `GET /voices` for the
|
||||
HTTP provider is not implemented yet. Until then, declare voices with the env var or configure
|
||||
the voice inside the wrapper.
|
||||
3. No changes are needed for the audio format; the engine post-processes to the required output.
|
||||
|
||||
Until `/voices` is wired, use defaults configured in the engine.
|
||||
|
||||
## Acceptance checklist
|
||||
|
||||
- [ ] Engine starts and answers on a stable URL or command.
|
||||
- [ ] Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
|
||||
- [ ] First audible word starts within about 500 ms of the request.
|
||||
- [ ] Speech is clean: no clicks, no clipping, no long leading silence.
|
||||
- [ ] Lines are consistent in loudness.
|
||||
- [ ] `voice` accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
|
||||
- [ ] `speed` and `pitch` are optional and documented in `/voices`.
|
||||
- [ ] Errors return a short reason and do not leave the engine wedged.
|
||||
- [ ] No files are written outside the engine's own cache or the `{out}` path.
|
||||
|
||||
---
|
||||
|
||||
## Clarifications from the engine side (MicroslopSam)
|
||||
|
||||
_Added 2026-09-15 by the engine implementer. The contract above stands; these points
|
||||
record what the engine needs to interpret, add, or hide._
|
||||
|
||||
### Output pipeline (WAV requirements)
|
||||
|
||||
- Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices
|
||||
(Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
|
||||
- Raw Flextalk output **clips** (peak 32768, RMS about 22000) and MSTTS is much quieter,
|
||||
so one shared level target is required. The engine post-processes every line:
|
||||
resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and
|
||||
level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
|
||||
- Implemented in `setup/veskslop-tts.sh` with ffmpeg: high-pass at 40 Hz, silence trim at
|
||||
both ends (trim-start, reverse, trim-start, reverse), 8 ms fades, then a **fixed gain**
|
||||
computed from an EBU R128 measurement (`ebur128=peak=true`):
|
||||
`gain = min(-16 LUFS target, -3 dBFS true-peak clamp)`, then 48 kHz mono s16le.
|
||||
- `loudnorm` was rejected: on short lines (under about 3 s) the loudness range is 0 and
|
||||
dynamic mode attenuated a 2.8 s line to -37.9 LUFS instead of about -18 LUFS; two-pass
|
||||
linear mode showed the same result. The fixed gain is steady for one-line utterances,
|
||||
as Veskslop suggested.
|
||||
- Verified: Keith Bell -18.4 LUFS, Kerry Watson -17.9 LUFS, Sam -20.6 LUFS; every file is
|
||||
48 kHz mono s16le at -3.0 dBFS true peak. The WAV handed to Veskslop is compliant; the
|
||||
"Veskslop does not normalize" rule holds.
|
||||
|
||||
### Voice IDs
|
||||
|
||||
- Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS
|
||||
variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but
|
||||
has no installed data for.
|
||||
- Assigned stable IDs: `sam`, `mary`, `mike`, `keith-bell`, `kerry-watson`; variants get
|
||||
`mike-telephone`, `mike-hall`, `mary-stadium`, `mike-space`, and so on. Unknown IDs fall
|
||||
back to `sam`.
|
||||
- `/voices` (HTTP mode) reports `language: "en"`, gender, and the speed/pitch ranges below.
|
||||
|
||||
### Speed and pitch mapping (measured 2026-09-15)
|
||||
|
||||
- SAPI4 uses engine units, not multipliers or semitones. Measured on the dev host with a
|
||||
correctly typed Flextalk install:
|
||||
- MSTTS Sam: speed 30-450, natural about 150; pitch 50-200 with a real audible effect.
|
||||
- Flextalk: speed 50-390, natural about 150 (durations at 100/200/300 fit a natural rate
|
||||
near 140-150); pitch 50-250 works. Measured F0: Kerry 0 st = 209 Hz, -6 st = 148 Hz,
|
||||
+6 st = 250 Hz (clamped); Keith 0 st = 119 Hz, +6 st = 170 Hz.
|
||||
- **Correction to the earlier note**: the previous "Flextalk ignores pitch / natural 300"
|
||||
finding was an artifact of the speaker registry values being stored as `REG_SZ`. The
|
||||
engine requires `REG_DWORD` for numeric speaker values; with strings it synthesizes a
|
||||
monotone buzz at maximum pitch. Fixed in `setup/setup-flextalk.sh`; no `no-pitch` flag
|
||||
is needed.
|
||||
- Mapping implemented in `setup/veskslop-tts.sh`: `speed 1.0` = natural rate 150 for every
|
||||
voice, scaled by the multiplier and clamped to the engine range. `pitch` semitones scale
|
||||
the voice's natural pitch unit by `2^(st/12)` and clamp to the range. Natural units are
|
||||
about 100 (MSTTS), 106 (Keith Bell) and 186 (Kerry Watson). The engines' F0 response is
|
||||
approximate, not semitone-calibrated.
|
||||
- `{setting}` is unused; the wrapper reads `VESKSLOP_TTS_SPEED` and
|
||||
`VESKSLOP_TTS_PITCH` from the environment when Veskslop later exports them.
|
||||
|
||||
### Text handling
|
||||
|
||||
- SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and
|
||||
unmappable characters to spaces. The host passes `TTSDATAFLAG_TAGGED`, so SAPI4 control
|
||||
tags in the text are enabled at the engine level; tagged input is not part of this
|
||||
contract and is untested. Plain text (no SSML) is the contract.
|
||||
- Lines up to 1000 characters are well within engine limits.
|
||||
|
||||
### Other voice adjustments (beyond speed)
|
||||
|
||||
All voices implement SAPI4 `ITTSAttributes`, so **pitch, speed and volume** exist at the
|
||||
API level. The Veskslop contract exposes only speed and pitch; volume is available in the
|
||||
host as `sapi4host -V <value>` and is not sent by Veskslop.
|
||||
|
||||
- **Volume** (measured 2026-09-15):
|
||||
- Flextalk: linear, range 0-65535, default 65535; half (32767) is about -6 dB, 0 is
|
||||
silence.
|
||||
- MSTTS: default is 0xFFFFFFFF; 0 is silence, but the response is not linear (for
|
||||
example 4096 gives about -24 dB while 65535 and larger are full scale). Treat volume
|
||||
as effectively on/off for MSTTS.
|
||||
- **Flextalk per-speaker profile** (applied at voice load, not runtime-adjustable):
|
||||
values live in `HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{GUID}` (see
|
||||
`docs/NOTES.md` for the full list and defaults). `Pitch`/`Speaking Rate` set the natural
|
||||
voice; **Expression** was verified to change the synthesis audibly (Keith Bell at
|
||||
Expression 0 / 59 / 100 yields different waveforms with the same duration). Changing a
|
||||
profile requires a registry edit and re-selecting the voice.
|
||||
- **MSTTS voice character** is fixed by the `.vce`/`.cfg` data; the engine exposes no
|
||||
per-voice profile in the registry. Its effect variants (telephone, in Hall, in Stadium,
|
||||
in Space) are separate voice modes with separate data, exposed as distinct voice IDs
|
||||
(`mike-telephone`, `mary-hall`, ...), not runtime effects.
|
||||
- Not exposed by this integration: SSML, prosody/style tags, or SAPI4 attribute
|
||||
extensions beyond pitch/speed/volume.
|
||||
|
||||
### Flextalk profile parameters (how to implement)
|
||||
|
||||
Each Flextalk speaker has a profile the engine reads when the voice is loaded. It lives in
|
||||
the registry; **every numeric value must be `REG_DWORD`**:
|
||||
|
||||
HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{11260610-C84A-11ce-AEC4-0000C0E9CB87} # Keith Bell
|
||||
HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{000031CB-3699-551F-AED0-6B4893382D57} # Kerry Watson
|
||||
|
||||
| Value (REG_DWORD unless noted) | Keith Bell | Kerry Watson | Measured effect |
|
||||
| --- | --- | --- | --- |
|
||||
| Pitch | 27 | 65 | Natural pitch (F0); the same scale `ITTSAttributes` shifts at runtime |
|
||||
| Speaking Rate | 33 | 32 | Natural speech rate (about 150 wpm equivalent) |
|
||||
| Volume | 100 | 100 | Output gain; 0 = silence |
|
||||
| Expression | 59 | 100 | Audible; changes the waveform at equal duration |
|
||||
| Prominences | 17 | 17 | Audible; strong prosody/emphasis effect (F0 median 122 -> 197 Hz at 100) |
|
||||
| Head Size | 46 | 45 | Audible; brightness/level (peak 20001 -> 32205 at 100) |
|
||||
| Lowering | 40 | 40 | Audible but subtle |
|
||||
| Age | 30 | 30 | No measured effect on synthesis (informational) |
|
||||
| Gender (REG_SZ) | "" | "" | No measured effect on synthesis (informational) |
|
||||
| Mode Name / Speaker Name (REG_SZ) | "Keith Bell" | "Kerry Watson" | Mode name SAPI4 shows for the voice |
|
||||
|
||||
How to apply a change:
|
||||
|
||||
1. `setup/setup-flextalk.sh` writes these defaults, so re-running it restores them.
|
||||
2. Profiles are read at voice load. The host loads the engine per request, so the next
|
||||
synthesis after a registry change uses the new profile; no engine restart is needed.
|
||||
3. Example (lower and slow Keith Bell), with `WINEPREFIX` pointing at the project prefix:
|
||||
|
||||
wine reg add 'HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{11260610-C84A-11ce-AEC4-0000C0E9CB87}' \
|
||||
/v Pitch /t REG_DWORD /d 20 /f
|
||||
wine reg add '...' /v "Speaking Rate" /t REG_DWORD /d 28 /f
|
||||
|
||||
Never use `REG_SZ` for numeric values: the engine reads garbage and synthesizes a
|
||||
monotone buzz at maximum pitch.
|
||||
4. Not exposed by the Veskslop contract (which sends only text/voice/speed/pitch). If
|
||||
profile control is wanted later, the clean place is the wrapper: read an env var such
|
||||
as `VESKSLOP_TTS_PROFILE` (`key=value` list or JSON), write the registry values before
|
||||
synthesis, then restore. That keeps the Veskslop interface unchanged.
|
||||
5. Runtime `ITTSAttributes` pitch/speed/volume layer on top of the profile: the profile
|
||||
defines the natural values, the attributes shift them per request.
|
||||
|
||||
### Transport and latency
|
||||
|
||||
- Installation is one command: `setup/install.sh` (system dependencies with sudo, wine
|
||||
prefix + SAPI4 runtime + MSTTS voices, Flextalk extraction/registration, host build).
|
||||
Re-running is safe; a fresh machine needs nothing else.
|
||||
- Command provider, implemented as `setup/veskslop-tts.sh`:
|
||||
`VESKSLOP_TTS_CMD="<repo>/setup/veskslop-tts.sh {out} {text_file}"` and
|
||||
`VESKSLOP_TTS_VOICES="$(<repo>/setup/veskslop-tts.sh --voices)"`. The wrapper reads the
|
||||
text file, transliterates UTF-8 to CP1252, starts or reuses an Xvfb when `DISPLAY` is
|
||||
unset, calls the engine, then post-processes (trim, fades, measured fixed gain, 48 kHz
|
||||
mono s16le). A stale private Xvfb is retried once.
|
||||
- Measured on the dev host: engine-only synthesis is about 290 ms wall for Sam and about
|
||||
40 ms for Flextalk, including process start; the full wrapper adds the ffmpeg
|
||||
measure/apply passes, about 630-880 ms per short line. That is above the 500 ms target;
|
||||
the known optimization is a per-voice static gain (skip the measurement pass), expected
|
||||
to land around 400-500 ms.
|
||||
- Manual checks: `setup/run.sh -l`, `setup/run.sh -v "Keith Bell" -t "Hello." -o out.wav`
|
||||
(raw engine WAV), and the wrapper above for normalized output.
|
||||
- HTTP provider (later): a persistent process returning `audio/wav` bytes and `/voices`,
|
||||
with optional caching by `hash(text + voice + speed + pitch)`.
|
||||
|
||||
### Quirks the wrapper must hide
|
||||
|
||||
- Flextalk numeric speaker settings must be `REG_DWORD`; `REG_SZ` values make the engine
|
||||
synthesize a buzz. `setup/setup-flextalk.sh` writes them correctly.
|
||||
- The mode list includes TruVoice names with no installed voice data; filter those out of
|
||||
`/voices` (or mark them `unavailable`).
|
||||
- Wine needs a working X display; the project helper starts a persistent Xvfb.
|
||||
|
||||
### Licensing
|
||||
|
||||
- The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a
|
||||
local integration; do not redistribute the DLLs or voice data with Veskslop.
|
||||
|
||||
### Extra acceptance items for the engine
|
||||
|
||||
- [x] Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
|
||||
- [x] Leading silence under 100 ms.
|
||||
- [x] Voice ID fallback works; TruVoice entries filtered.
|
||||
- [x] Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).
|
||||
|
||||
## Responses from the Veskslop side
|
||||
|
||||
_Added 2026-09-15._
|
||||
|
||||
- **Env vars and placeholders**: `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH` are exported only
|
||||
when a value is set; `{speed}` and `{pitch}` substitute the same values. Reading the environment
|
||||
is the safer option because it works even when the command has no placeholders.
|
||||
- **Defaults**: a missing value means "engine default". Leave the sliders at 1.00x and 0 st in
|
||||
Veskslop to keep the fields out of the request entirely.
|
||||
- **`loudnorm` caution**: single-pass `loudnorm` can pump or over-amplify noise on short lines.
|
||||
Set the target explicitly (`loudnorm=I=-16:TP=-1.5:LRA=11`) and verify a few lines with
|
||||
`ebur128`. A fixed gain that hits about -3 dBFS peak with a consistent RMS is often steadier for
|
||||
one-line utterances. Ducking lowers the sink to 30%, so keep peaks conservative and loudness
|
||||
even between lines.
|
||||
- **Display**: the daemon does not export `DISPLAY` to TTS children. The wrapper must start or
|
||||
find its own Xvfb and set `DISPLAY` itself, as the project helper already does. Do not rely on
|
||||
the client's display.
|
||||
- **Engine selection**: the Speech settings and the TTS panel list every configured engine. The
|
||||
command engine appears under `VESKSLOP_TTS_NAME` (default "custom engine") and is selectable
|
||||
directly; Auto uses the command engine first. Selecting it hides the language row and shows the
|
||||
`VESKSLOP_TTS_VOICES` list.
|
||||
- **Start engine**: the Speech settings have a **Start engine** button (`POST /api/audio/tts/warm`,
|
||||
MCP `start_tts_engine`) that runs one throwaway synthesis to warm the command engine (Xvfb, Wine,
|
||||
host). Keep the wrapper able to handle a short sentence; the warm-up uses "The quick brown fox
|
||||
jumps over the lazy dog." and reports the elapsed milliseconds. Two-pass `loudnorm` must not fail
|
||||
on that input.
|
||||
- **Lead-in silence**: Veskslop prepends a short silence at playback time (`VESKSLOP_TTS_LEAD_MS`,
|
||||
default 150 ms) so Discord's voice pipeline is ready before the first syllable. It is added with
|
||||
`adelay` in the player, not in the audio file. The engine needs no change and can keep its
|
||||
5–10 ms edge fades.
|
||||
- **Voice declarations**: list only usable IDs in `VESKSLOP_TTS_VOICES`. Veskslop hides the
|
||||
espeak list while a custom command is set, so a missing declaration means an empty voice list,
|
||||
not a fallback. The list accepts commas or newlines. Optional flags: `no-pitch` (hide the pitch
|
||||
control) and `expression` / `expression=NN` (show the Expression slider and use NN as its
|
||||
default).
|
||||
- **Flextalk pitch**: works now that the speaker registry values are `REG_DWORD`; no flag is
|
||||
needed and Veskslop shows pitch for Keith Bell and Kerry Watson.
|
||||
- **Expression profile**: implemented for the Flextalk voices. Veskslop stores `tts.expression`
|
||||
(0-100, nullable), sends it per line (`{expression}` / `VESKSLOP_TTS_EXPRESSION` / the HTTP body),
|
||||
and shows the slider only for voices flagged `expression=NN`. A null value leaves the registry
|
||||
untouched, so the voice keeps its engine default.
|
||||
- **Licensing**: keep the engine DLLs and voice data local and out of any repository; Veskslop
|
||||
never bundles or redistributes them.
|
||||
|
||||
## Current integration path (Veskslop side, 2026-09-15)
|
||||
|
||||
Everything below is implemented. The engine stays unchanged; the wrapper and the daemon do the work.
|
||||
|
||||
### Voice metadata
|
||||
|
||||
`VESKSLOP_TTS_VOICES` entries may carry flags after the language field:
|
||||
|
||||
id|Name|gender|language|flags
|
||||
|
||||
Flags are `+`-separated:
|
||||
|
||||
- `no-pitch` - hide the pitch control for this voice.
|
||||
- `expression=NN` - Flextalk profile voice; NN is the default Expression value.
|
||||
- `defaults=pitch:27:rate:33:prominences:17:head_size:46:lowering:40:expression:59` -
|
||||
per-voice engine defaults. The UI shows "default" at these values and snaps to them
|
||||
within +/-2 units.
|
||||
|
||||
### Wrapper contract (`microslopsam/setup/veskslop-tts.sh`)
|
||||
|
||||
Inputs: `{out}`, `{text_file}`, optional positional voice/speed/pitch/expression, and:
|
||||
|
||||
| Env | Meaning |
|
||||
| --- | --- |
|
||||
| `VESKSLOP_TTS_PROFILE` | `pitch:27:...`/`key=value,...` overrides; the wrapper writes the full profile (defaults + overrides) to the Wine registry with `wine reg import` before every synthesis. |
|
||||
| `VESKSLOP_TTS_HIGHPASS` | High-pass Hz, 40-400 (default 40). |
|
||||
| `VESKSLOP_TTS_ECHO` | Mono echo 0-100 (`aecho`), default 0. |
|
||||
| `VESKSLOP_TTS_WORDS` / `VESKSLOP_TTS_WORD_GAP_MS` | Word-by-word synthesis with silence gaps (Flextalk only). |
|
||||
|
||||
Placeholders `{profile}`, `{highpass}`, `{echo}`, `{words}`, `{word_gap_ms}` mirror the env vars.
|
||||
`wine regedit` fails headlessly but returns 0: use `wine reg import`. A missing profile value
|
||||
must always write the engine default so no stale registry state survives.
|
||||
|
||||
### Daemon and UI path
|
||||
|
||||
1. Preferences (`tts`): `profile` (changed keys only), `profilePresets` (5 named slots),
|
||||
`highpass`, `echo`, `words`, `wordGapMs`, plus `speed`, `pitch`, `expression`, `voice`.
|
||||
2. `/api/audio/speak` and MCP `speak` accept all of them per line; the request overrides the
|
||||
preferences and empty fields are omitted so the engine default applies.
|
||||
3. The TTS panel opens the **Flextalk voice profile** window (movable, recentered on open) with
|
||||
profile sliders, named presets, radio-kit controls, and test lines (Attention, Laboratory,
|
||||
Critical, Health, Radiation, plus two persisted custom lines). Test buttons post the same
|
||||
options as `speak`, so they audition the current profile.
|
||||
|
||||
### Known pitfalls
|
||||
|
||||
- The daemon runs from `tsx`: restart it after protocol, preference, or metadata changes.
|
||||
- Static web files can be cached; hard-reload the browser after UI changes.
|
||||
- `Number(null)` is 0: preference normalizers must treat null/undefined/"" as unset.
|
||||
- The renderer log spam on a dead client is throttled in the bridge; background polls fail with
|
||||
`CORE_UNAVAILABLE` and retry.
|
||||
- Generate the Flextalk profile `.reg` with `printf`, never `echo`: dash's `echo` interprets
|
||||
`\2` in `...\Watson\2.0\...` as an escape (0x02), so the key is corrupted and
|
||||
`wine reg import` becomes a silent no-op (it still returns 0). The wrapper now rejects a
|
||||
generated profile that contains control characters. This was the 2026-09-15 profile bug.
|
||||
Reference in New Issue
Block a user