MicroslopSam/docs/TTS-DESIGN.md

TTS-DESIGN.md — building a TTS engine for Veskslop

This document is the integration contract for a text-to-speech engine that Veskslop will use. It is written to be self-contained: an engine developer can work from this file alone.

How Veskslop uses TTS

1. A line of text is queued (POST /api/audio/speak or the MCP speak tool).
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
3. ffplay decodes the file and plays it into the PulseAudio null sink veskslop_tts.
4. veskslop_tts.monitor is exposed as the source veskslop_mic.
5. The Vesktop client uses veskslop_mic as its microphone, so the call hears the speech.
6. While TTS plays and another person speaks, the daemon lowers the veskslop_tts sink volume to 30% and restores it after. This is "ducking". The engine does not implement ducking.

Lines are serialized: one synthesis, then playback, then the next line. There is no streaming playback path and no partial-audio protocol. The engine returns a complete audio object.

Integration modes

HTTP provider (recommended)

Set VESKSLOP_TTS_API_URL. Select the API provider with VESKSLOP_TTS_PROVIDER=api or the API TTS choice in the TTS panel. The daemon sends:

POST <VESKSLOP_TTS_API_URL>
Content-Type: application/json
Authorization: Bearer <VESKSLOP_TTS_API_KEY>   (only when the key is set)

{ "text": "Hello there.", "voice": "anna" }

The response body must be the audio bytes. Content-Type selects the file extension: audio/mpeg or audio/mp3 → .mp3, audio/ogg → .ogg, audio/wav or audio/x-wav → .wav, anything else is treated as .mp3. Prefer audio/wav.

A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs it and drops the line.

Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the per-line process start cost.

Local command provider

Set VESKSLOP_TTS_CMD. It takes precedence over the built-in espeak-ng engine. Veskslop replaces these placeholders, splits the string on whitespace, and runs the result directly (no shell):

Placeholder	Replaced with
{out}	Output file path, extension .wav. Required.
{text_file}	A UTF-8 text file holding the line.
{voice}	The selected voice ID, or an empty string.
{speed}	The rate multiplier, or an empty string when unset.
{pitch}	The pitch offset in semitones, or an empty string when unset.

The process also receives VESKSLOP_TTS_VOICE, and VESKSLOP_TTS_SPEED / VESKSLOP_TTS_PITCH when those are set. Write the audio to {out} and exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it short.

Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and shell tricks cannot work. Use a small wrapper script if you need options.

Audio output requirements

Property	Recommendation	Notes
Container	WAV (RIFF)	Any format ffmpeg decodes works, but WAV avoids decoder startup cost.
Codec	PCM signed 16-bit little-endian	pcm_s16le.
Sample rate	48000 Hz	44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample.
Channels	Mono	Discord voice is effectively mono. Stereo is accepted but adds no value.
Duration	One line of speech	No intro/outro music, no chimes.
Peak level	About -3 dBFS	Leave headroom; ducking and Discord gain must not clip.
Loudness	Consistent across lines	Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume.
Silence	Trim start to under 100 ms, keep roughly 20-50 ms of padding	Long leading silence delays the first audible word.
Edges	5-10 ms fade in and out	Prevents clicks.
DC offset	None	Center the waveform.

Veskslop does not normalize, limit, or re-encode the audio. Playback is ffplay at 100% sink volume, so the engine's levels are what Discord hears.

Voices

A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such as anna, sv-anna, or en-male-2. The ID can contain letters, digits, -, _, and +.

If the engine has multiple voices, expose them so a UI can list them:

GET /voices
[
  {
    "id": "anna",
    "name": "Anna",
    "language": "sv",
    "gender": "female",
    "speed":  { "min": 0.5, "max": 2.0, "default": 1.0 },
    "pitch":  { "min": -12,  "max": 12,  "default": 0, "unit": "semitones" }
  }
]

Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set the voice through the engine's own configuration and leave the Veskslop voice setting empty.

Speed and pitch

Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only text, voice, and provider. No UI or preference exists for speed or pitch.

Recommended semantics for the engine:

• speed: a multiplier, 1.0 is normal, useful range 0.5 to 2.0. Do not change pitch when changing speed unless the user asks for it.
• pitch: semitones, 0 is the voice's natural pitch, useful range -12 to +12. State the unit in the /voices response so clients do not guess.

Recommended HTTP shape (extras are optional; an engine may ignore them):

{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }

Recommended command shape: {speed} and {pitch} are substituted when set, and the same values are exported as VESKSLOP_TTS_SPEED and VESKSLOP_TTS_PITCH. Read either; the environment form survives placeholder-free commands.

How Veskslop should add first-class settings

Implemented on the Veskslop side (2026-09-15):

• tts preferences carry speed (0.5–2.0) and pitch (-12–+12, semitones), both nullable.
• /api/audio/speak and the MCP speak tool accept optional speed and pitch that override the preferences for that line.
• The command provider substitutes {speed}/{pitch} and exports VESKSLOP_TTS_SPEED / VESKSLOP_TTS_PITCH only when the value is set.
• The HTTP provider adds speed and pitch to the request body only when set.
• The TTS panel has speed and pitch sliders; changes are saved to the daemon preferences.
• Command mode advertises voices through VESKSLOP_TTS_VOICES. Entries are separated by commas or newlines and use id, id|Name, id|Name|gender, id|Name|gender|language, or id|Name|gender|language|flags, where flags are +-separated. The no-pitch flag marks a voice that ignores pitch; /api/audio/tts/voices then reports supportsPitch: false and the TTS panel disables the pitch slider. While a custom command is set, the engine's declared voices replace the espeak list, so no misleading espeak entries appear. Reading /voices for the HTTP provider is still future work.

Do not encode settings into the voice string (for example anna+speed1.2). It breaks voice lists and is hard to validate.

Text handling

• UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
• Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
• Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine should not fail on an unknown character; skip or verbalize it as it prefers.
• Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints for pacing.

Errors, limits, and behavior

• One request = one complete audio object. No chunked playback, no callbacks.
• Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget if the engine is slower. A warm HTTP server is the way to meet this.
• Requests are serialized by Veskslop, so the engine does not need a request queue, but it should tolerate repeated back-to-back requests.
• The daemon never retries a failed line. Make failures visible in the response and stderr.
• Do not write to the veskslop_tts sink or the veskslop_mic source directly. The daemon owns playback routing.

Optional: caching

Veskslop does not cache audio. The engine may cache by hash(text + voice + speed + pitch) and return the cached file or bytes. This is useful for repeated lines and for expensive models.

Veskslop changes needed

Status as of 2026-09-15:

1. speed and pitch are first-class: preferences, per-request overrides on /api/audio/speak and MCP speak, env exports and placeholders for the command provider, request fields for the HTTP provider, and two sliders in the TTS panel.
2. Command-mode voices come from VESKSLOP_TTS_VOICES. Reading the engine's GET /voices for the HTTP provider is not implemented yet. Until then, declare voices with the env var or configure the voice inside the wrapper.
3. No changes are needed for the audio format; the engine post-processes to the required output.

Until /voices is wired, use defaults configured in the engine.

Acceptance checklist

• Engine starts and answers on a stable URL or command.
• Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
• First audible word starts within about 500 ms of the request.
• Speech is clean: no clicks, no clipping, no long leading silence.
• Lines are consistent in loudness.
• voice accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
• speed and pitch are optional and documented in /voices.
• Errors return a short reason and do not leave the engine wedged.
• No files are written outside the engine's own cache or the {out} path.

---

Clarifications from the engine side (MicroslopSam)

Added 2026-09-15 by the engine implementer. The contract above stands; these points record what the engine needs to interpret, add, or hide.

Output pipeline (WAV requirements)

• Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices (Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
• Raw Flextalk output clips (peak 32768, RMS about 22000) and MSTTS is much quieter, so one shared level target is required. The engine post-processes every line: resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
• Proposed implementation, engine-side, with ffmpeg (already installed on the dev host): ffmpeg -i raw.wav -af "highpass=f=40,<trim><fade>,loudnorm=..." -ar 48000 -ac 1 out.wav. The WAV handed to Veskslop is then compliant; the "Veskslop does not normalize" rule holds.

Voice IDs

• Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but has no installed data for.
• Assigned stable IDs: sam, mary, mike, keith-bell, kerry-watson; variants get mike-telephone, mike-hall, mary-stadium, mike-space, and so on. Unknown IDs fall back to sam.
• /voices (HTTP mode) reports language: "en", gender, and the speed/pitch ranges below.

Speed and pitch mapping (measured 2026-09-15)

• SAPI4 uses engine units, not multipliers or semitones. Measured on the dev host:
  • MSTTS Sam: speed 30-450, natural about 150; pitch 50-200 with a real audible effect (F0 about 134 Hz at pitch 50, about 199 Hz at pitch 200 on the test phrase).
  • Flextalk: speed 50-390, natural about 300 (output with no setting is byte-identical to -s 300); pitch has no effect (byte-identical output at pitch 100/150/200), so the wrapper ignores pitch for Keith Bell and Kerry Watson.
• Mapping implemented in setup/vekslop-tts.sh: speed 1.0 = the voice's natural rate (150 MSTTS, 300 Flextalk), scaled by the multiplier and clamped to the engine range. pitch for MSTTS maps ±12 semitones to engine units 100 ± 6.25·st, clamped to 50-200. Pitch is a no-op for Flextalk voices; the wrapper's --voices output marks keith-bell and kerry-watson with the no-pitch flag so the UI disables the slider.
• {setting} is unused; the wrapper reads VESKSLOP_TTS_SPEED and VESKSLOP_TTS_PITCH from the environment when Veskslop later exports them.

Text handling

• SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and unmappable characters to spaces. Engine tag parsing is disabled, so input stays plain text (no SSML), matching the contract.
• Lines up to 1000 characters are well within engine limits.

Transport and latency

• Command provider, implemented as setup/vekslop-tts.sh: VESKSLOP_TTS_CMD="<repo>/setup/vekslop-tts.sh {out} {text_file}" and VESKSLOP_TTS_VOICES="$(<repo>/setup/vekslop-tts.sh --voices)". The wrapper reads the text file, transliterates UTF-8 to CP1252, starts or reuses an Xvfb when DISPLAY is unset, calls the engine, then post-processes (trim, fades, two-pass loudnorm, 48 kHz mono s16le). Verified: -17.7 LUFS integrated, -3.0 dBFS true peak.
• Measured on the dev host (warm Wine): Sam line about 288 ms wall, Flextalk line about 41 ms wall, including process start, synthesis, and file write. Both are inside the 500 ms budget.
• HTTP provider (later): a persistent process returning audio/wav bytes and /voices, with optional caching by hash(text + voice + speed + pitch).

Quirks the wrapper must hide

• Flextalk numeric speaker settings must be REG_DWORD; REG_SZ values make the engine synthesize a buzz. setup/setup-flextalk.sh writes them correctly.
• The mode list includes TruVoice names with no installed voice data; filter those out of /voices (or mark them unavailable).
• Wine needs a working X display; the project helper starts a persistent Xvfb.

Licensing

• The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a local integration; do not redistribute the DLLs or voice data with Veskslop.

Extra acceptance items for the engine

• Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
• Leading silence under 100 ms.
• Voice ID fallback works; TruVoice entries filtered.
• Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).

Responses from the Veskslop side

Added 2026-09-15.

• Env vars and placeholders: VESKSLOP_TTS_SPEED and VESKSLOP_TTS_PITCH are exported only when a value is set; {speed} and {pitch} substitute the same values. Reading the environment is the safer option because it works even when the command has no placeholders.
• Defaults: a missing value means "engine default". Leave the sliders at 1.00x and 0 st in Veskslop to keep the fields out of the request entirely.
• loudnorm caution: single-pass loudnorm can pump or over-amplify noise on short lines. Set the target explicitly (loudnorm=I=-16:TP=-1.5:LRA=11) and verify a few lines with ebur128. A fixed gain that hits about -3 dBFS peak with a consistent RMS is often steadier for one-line utterances. Ducking lowers the sink to 30%, so keep peaks conservative and loudness even between lines.
• Display: the daemon does not export DISPLAY to TTS children. The wrapper must start or find its own Xvfb and set DISPLAY itself, as the project helper already does. Do not rely on the client's display.
• Engine selection: the Speech settings and the TTS panel list every configured engine. The command engine appears under VESKSLOP_TTS_NAME (default "custom engine") and is selectable directly; Auto uses the command engine first. Selecting it hides the language row and shows the VESKSLOP_TTS_VOICES list.
• Start engine: the Speech settings have a Start engine button (POST /api/audio/tts/warm, MCP start_tts_engine) that runs one throwaway synthesis to warm the command engine (Xvfb, Wine, host). Keep the wrapper able to handle a short sentence; the warm-up uses "The quick brown fox jumps over the lazy dog." and reports the elapsed milliseconds. Two-pass loudnorm must not fail on that input.
• Lead-in silence: Veskslop prepends a short silence at playback time (VESKSLOP_TTS_LEAD_MS, default 150 ms) so Discord's voice pipeline is ready before the first syllable. It is added with adelay in the player, not in the audio file. The engine needs no change and can keep its 5–10 ms edge fades.
• Voice declarations: list only usable IDs in VESKSLOP_TTS_VOICES. Veskslop hides the espeak list while a custom command is set, so a missing declaration means an empty voice list, not a fallback. The list accepts commas or newlines, and the optional no-pitch flag tells the UI to grey out the pitch slider for that voice.
• Flextalk pitch: confirmed that pitch is a no-op for Keith Bell and Kerry Watson. Mark those entries no-pitch so users are not offered a control that does nothing. Speed applies to every voice and needs no flag. Engine correction (2026-09-15): the no-op was caused by the speaker registry values being REG_SZ; with REG_DWORD pitch works and the voices are declared without the flag. See the speed/pitch section.
• Licensing: keep the engine DLLs and voice data local and out of any repository; Veskslop never bundles or redistributes them.