diff --git a/README.md b/README.md index 34a79d4..7760a30 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,7 @@ Options: `-v` voice, `-t` text, `-o` output WAV, `-p` pitch, `-s` speed. - `setup/inf2reg.py` — INF AddReg -> `.reg` converter - `setup/run.sh` — run helper (prefix + Xvfb) - `docs/NOTES.md` — registry/CLSID facts, runtime RE notes, gotchas +- `docs/TTS-DESIGN.md` — integration contract with Veskslop plus engine clarifications ## Known issues diff --git a/docs/TTS-DESIGN.md b/docs/TTS-DESIGN.md new file mode 100644 index 0000000..a2811c3 --- /dev/null +++ b/docs/TTS-DESIGN.md @@ -0,0 +1,259 @@ +# TTS-DESIGN.md — building a TTS engine for Veskslop + +This document is the integration contract for a text-to-speech engine that Veskslop will use. It is +written to be self-contained: an engine developer can work from this file alone. + +## How Veskslop uses TTS + +1. A line of text is queued (`POST /api/audio/speak` or the MCP `speak` tool). +2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode). +3. `ffplay` decodes the file and plays it into the PulseAudio null sink `veskslop_tts`. +4. `veskslop_tts.monitor` is exposed as the source `veskslop_mic`. +5. The Vesktop client uses `veskslop_mic` as its microphone, so the call hears the speech. +6. While TTS plays and another person speaks, the daemon lowers the `veskslop_tts` sink volume to + 30% and restores it after. This is "ducking". The engine does not implement ducking. + +Lines are serialized: one synthesis, then playback, then the next line. There is no streaming +playback path and no partial-audio protocol. The engine returns a complete audio object. + +## Integration modes + +### HTTP provider (recommended) + +Set `VESKSLOP_TTS_API_URL`. Select the API provider with `VESKSLOP_TTS_PROVIDER=api` or the +**API TTS** choice in the TTS panel. The daemon sends: + +``` +POST +Content-Type: application/json +Authorization: Bearer (only when the key is set) + +{ "text": "Hello there.", "voice": "anna" } +``` + +The response body must be the audio bytes. `Content-Type` selects the file extension: `audio/mpeg` +or `audio/mp3` → `.mp3`, `audio/ogg` → `.ogg`, `audio/wav` or `audio/x-wav` → `.wav`, anything else +is treated as `.mp3`. Prefer `audio/wav`. + +A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs +it and drops the line. + +Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the +per-line process start cost. + +### Local command provider + +Set `VESKSLOP_TTS_CMD`. It takes precedence over the built-in espeak-ng engine. Veskslop replaces +these placeholders, splits the string on whitespace, and runs the result directly (no shell): + +| Placeholder | Replaced with | +| --- | --- | +| `{out}` | Output file path, extension `.wav`. Required. | +| `{text_file}` | A UTF-8 text file holding the line. | +| `{voice}` | The selected voice ID, or an empty string. | +| `{setting}` | Reserved for speed/pitch once Veskslop grows settings support. | + +The process also receives `VESKSLOP_TTS_VOICE` in the environment. Write the audio to `{out}` and +exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it short. + +Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and +shell tricks cannot work. Use a small wrapper script if you need options. + +## Audio output requirements + +| Property | Recommendation | Notes | +| --- | --- | --- | +| Container | WAV (RIFF) | Any format ffmpeg decodes works, but WAV avoids decoder startup cost. | +| Codec | PCM signed 16-bit little-endian | `pcm_s16le`. | +| Sample rate | 48000 Hz | 44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample. | +| Channels | Mono | Discord voice is effectively mono. Stereo is accepted but adds no value. | +| Duration | One line of speech | No intro/outro music, no chimes. | +| Peak level | About -3 dBFS | Leave headroom; ducking and Discord gain must not clip. | +| Loudness | Consistent across lines | Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume. | +| Silence | Trim start to under 100 ms, keep roughly 20-50 ms of padding | Long leading silence delays the first audible word. | +| Edges | 5-10 ms fade in and out | Prevents clicks. | +| DC offset | None | Center the waveform. | + +Veskslop does not normalize, limit, or re-encode the audio. Playback is `ffplay` at 100% sink +volume, so the engine's levels are what Discord hears. + +## Voices + +A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such +as `anna`, `sv-anna`, or `en-male-2`. The ID can contain letters, digits, `-`, `_`, and `+`. + +If the engine has multiple voices, expose them so a UI can list them: + +``` +GET /voices +[ + { + "id": "anna", + "name": "Anna", + "language": "sv", + "gender": "female", + "speed": { "min": 0.5, "max": 2.0, "default": 1.0 }, + "pitch": { "min": -12, "max": 12, "default": 0, "unit": "semitones" } + } +] +``` + +Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after +Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set +the voice through the engine's own configuration and leave the Veskslop voice setting empty. + +## Speed and pitch + +Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only `text`, +`voice`, and `provider`. No UI or preference exists for speed or pitch. + +Recommended semantics for the engine: + +- **speed**: a multiplier, `1.0` is normal, useful range `0.5` to `2.0`. Do not change pitch when + changing speed unless the user asks for it. +- **pitch**: semitones, `0` is the voice's natural pitch, useful range `-12` to `+12`. State the + unit in the `/voices` response so clients do not guess. + +Recommended HTTP shape (extras are optional; an engine may ignore them): + +``` +{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 } +``` + +Recommended command shape: `{setting}` is reserved. Until Veskslop passes it, read defaults from +the engine's own config file or environment. + +### How Veskslop should add first-class settings + +A future change would add `speed` and `pitch` to `tts` preferences, pass them to the provider +alongside `voice`, and add two controls to the TTS panel. For the command provider, `{speed}` and +`{pitch}` placeholders (or one `{settings_file}` JSON file) would be substituted the same way as +`{voice}`. Design the engine API so it works with defaults when these fields are absent. + +Do not encode settings into the voice string (for example `anna+speed1.2`). It breaks voice lists +and is hard to validate. + +## Text handling + +- UTF-8 input. Lines arrive trimmed and are at most 1000 characters. +- Do not parse or require SSML. Plain text is the contract. SSML support can be additive. +- Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine + should not fail on an unknown character; skip or verbalize it as it prefers. +- Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints + for pacing. + +## Errors, limits, and behavior + +- One request = one complete audio object. No chunked playback, no callbacks. +- Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget + if the engine is slower. A warm HTTP server is the way to meet this. +- Requests are serialized by Veskslop, so the engine does not need a request queue, but it should + tolerate repeated back-to-back requests. +- The daemon never retries a failed line. Make failures visible in the response and stderr. +- Do not write to the `veskslop_tts` sink or the `veskslop_mic` source directly. The daemon owns + playback routing. + +## Optional: caching + +Veskslop does not cache audio. The engine may cache by `hash(text + voice + speed + pitch)` and +return the cached file or bytes. This is useful for repeated lines and for expensive models. + +## Veskslop changes needed + +If the engine wants first-class voice lists, speed, or pitch in the Veskslop UI, Veskslop must: + +1. Read the engine's `/voices` endpoint and merge it into `/api/audio/tts/voices`. +2. Add `speed` and `pitch` to the `tts` preferences and to the `speak` request. +3. Add controls to the TTS panel and pass the values through to the provider. + +Until those exist, use defaults configured in the engine. + +## Acceptance checklist + +- [ ] Engine starts and answers on a stable URL or command. +- [ ] Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS. +- [ ] First audible word starts within about 500 ms of the request. +- [ ] Speech is clean: no clicks, no clipping, no long leading silence. +- [ ] Lines are consistent in loudness. +- [ ] `voice` accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing. +- [ ] `speed` and `pitch` are optional and documented in `/voices`. +- [ ] Errors return a short reason and do not leave the engine wedged. +- [ ] No files are written outside the engine's own cache or the `{out}` path. + +--- + +## Clarifications from the engine side (MicroslopSam) + +_Added 2026-09-15 by the engine implementer. The contract above stands; these points +record what the engine needs to interpret, add, or hide._ + +### Output pipeline (WAV requirements) + +- Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices + (Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz. +- Raw Flextalk output **clips** (peak 32768, RMS about 22000) and MSTTS is much quieter, + so one shared level target is required. The engine post-processes every line: + resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and + level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS). +- Proposed implementation, engine-side, with ffmpeg (already installed on the dev host): + `ffmpeg -i raw.wav -af "highpass=f=40,,loudnorm=..." -ar 48000 -ac 1 out.wav`. + The WAV handed to Veskslop is then compliant; the "Veskslop does not normalize" rule holds. + +### Voice IDs + +- Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS + variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but + has no installed data for. +- Assigned stable IDs: `sam`, `mary`, `mike`, `keith-bell`, `kerry-watson`; variants get + `mike-telephone`, `mike-hall`, `mary-stadium`, `mike-space`, and so on. Unknown IDs fall + back to `sam`. +- `/voices` (HTTP mode) reports `language: "en"`, gender, and the speed/pitch ranges below. + +### Speed and pitch mapping + +- SAPI4 uses engine units, not multipliers or semitones. Observed values: + - MSTTS Sam: speed 30-450 (default 150), pitch 50-200 (default 100). + - Flextalk: speed 50-390, pitch 50-250; the engine reports the **defaults as the + maximum**, so the wrapper uses the voice profile from the registry as the natural + values (Keith Bell: pitch 27, rate 33; Kerry Watson: pitch 65, rate 32). +- Mapping (approximate, stated in `/voices`): `speed 1.0` = the voice's natural rate; + 0.5-2.0 scales that rate and clamps to the engine range. `pitch` ±12 semitones scales the + engine pitch range by about ±25% around the natural value. +- Command mode: `{setting}` is unused for now. The wrapper reads `VESKSLOP_TTS_SPEED` and + `VESKSLOP_TTS_PITCH` from the environment when Veskslop later exports them. + +### Text handling + +- SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and + unmappable characters to spaces. Engine tag parsing is disabled, so input stays plain + text (no SSML), matching the contract. +- Lines up to 1000 characters are well within engine limits. + +### Transport and latency + +- Command provider (first target): a wrapper script reads `{text_file}`, calls the engine, + post-processes, writes `{out}`, and exits non-zero with one short stderr line on failure. +- Measured on the dev host (warm Wine): Sam line about 288 ms wall, Flextalk line about + 41 ms wall, including process start, synthesis, and file write. Both are inside the + 500 ms budget. +- HTTP provider (later): a persistent process returning `audio/wav` bytes and `/voices`, + with optional caching by `hash(text + voice + speed + pitch)`. + +### Quirks the wrapper must hide + +- Flextalk reports default pitch/speed as maximum through `ITTSAttributes`. +- The mode list includes TruVoice names with no installed voice data; filter those out of + `/voices` (or mark them `unavailable`). +- Wine needs a working X display; the project helper starts a persistent Xvfb. + +### Licensing + +- The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a + local integration; do not redistribute the DLLs or voice data with Veskslop. + +### Extra acceptance items for the engine + +- [ ] Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present. +- [ ] Leading silence under 100 ms. +- [ ] Voice ID fallback works; TruVoice entries filtered. +- [ ] Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).