MicroslopSam/docs/TTS-DESIGN.md

TTS-DESIGN.md — building a TTS engine for Veskslop

This document is the integration contract for a text-to-speech engine that Veskslop will use. It is written to be self-contained: an engine developer can work from this file alone.

How Veskslop uses TTS

1. A line of text is queued (POST /api/audio/speak or the MCP speak tool).
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
3. ffplay decodes the file and plays it into the PulseAudio null sink veskslop_tts.
4. veskslop_tts.monitor is exposed as the source veskslop_mic.
5. The Vesktop client uses veskslop_mic as its microphone, so the call hears the speech.
6. While TTS plays and another person speaks, the daemon lowers the veskslop_tts sink volume to 30% and restores it after. This is "ducking". The engine does not implement ducking.

Lines are serialized: one synthesis, then playback, then the next line. There is no streaming playback path and no partial-audio protocol. The engine returns a complete audio object.

Integration modes

HTTP provider (recommended)

Set VESKSLOP_TTS_API_URL. Select the API provider with VESKSLOP_TTS_PROVIDER=api or the API TTS choice in the TTS panel. The daemon sends:

POST <VESKSLOP_TTS_API_URL>
Content-Type: application/json
Authorization: Bearer <VESKSLOP_TTS_API_KEY>   (only when the key is set)

{ "text": "Hello there.", "voice": "anna" }

The response body must be the audio bytes. Content-Type selects the file extension: audio/mpeg or audio/mp3 → .mp3, audio/ogg → .ogg, audio/wav or audio/x-wav → .wav, anything else is treated as .mp3. Prefer audio/wav.

A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs it and drops the line.

Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the per-line process start cost.

Local command provider

Set VESKSLOP_TTS_CMD. It takes precedence over the built-in espeak-ng engine. Veskslop replaces these placeholders, splits the string on whitespace, and runs the result directly (no shell):

Placeholder	Replaced with
{out}	Output file path, extension .wav. Required.
{text_file}	A UTF-8 text file holding the line.
{voice}	The selected voice ID, or an empty string.
{setting}	Reserved for speed/pitch once Veskslop grows settings support.

The process also receives VESKSLOP_TTS_VOICE in the environment. Write the audio to {out} and exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it short.

Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and shell tricks cannot work. Use a small wrapper script if you need options.

Audio output requirements

Property	Recommendation	Notes
Container	WAV (RIFF)	Any format ffmpeg decodes works, but WAV avoids decoder startup cost.
Codec	PCM signed 16-bit little-endian	pcm_s16le.
Sample rate	48000 Hz	44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample.
Channels	Mono	Discord voice is effectively mono. Stereo is accepted but adds no value.
Duration	One line of speech	No intro/outro music, no chimes.
Peak level	About -3 dBFS	Leave headroom; ducking and Discord gain must not clip.
Loudness	Consistent across lines	Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume.
Silence	Trim start to under 100 ms, keep roughly 20-50 ms of padding	Long leading silence delays the first audible word.
Edges	5-10 ms fade in and out	Prevents clicks.
DC offset	None	Center the waveform.

Veskslop does not normalize, limit, or re-encode the audio. Playback is ffplay at 100% sink volume, so the engine's levels are what Discord hears.

Voices

A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such as anna, sv-anna, or en-male-2. The ID can contain letters, digits, -, _, and +.

If the engine has multiple voices, expose them so a UI can list them:

GET /voices
[
  {
    "id": "anna",
    "name": "Anna",
    "language": "sv",
    "gender": "female",
    "speed":  { "min": 0.5, "max": 2.0, "default": 1.0 },
    "pitch":  { "min": -12,  "max": 12,  "default": 0, "unit": "semitones" }
  }
]

Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set the voice through the engine's own configuration and leave the Veskslop voice setting empty.

Speed and pitch

Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only text, voice, and provider. No UI or preference exists for speed or pitch.

Recommended semantics for the engine:

• speed: a multiplier, 1.0 is normal, useful range 0.5 to 2.0. Do not change pitch when changing speed unless the user asks for it.
• pitch: semitones, 0 is the voice's natural pitch, useful range -12 to +12. State the unit in the /voices response so clients do not guess.

Recommended HTTP shape (extras are optional; an engine may ignore them):

{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }

Recommended command shape: {setting} is reserved. Until Veskslop passes it, read defaults from the engine's own config file or environment.

How Veskslop should add first-class settings

A future change would add speed and pitch to tts preferences, pass them to the provider alongside voice, and add two controls to the TTS panel. For the command provider, {speed} and {pitch} placeholders (or one {settings_file} JSON file) would be substituted the same way as {voice}. Design the engine API so it works with defaults when these fields are absent.

Do not encode settings into the voice string (for example anna+speed1.2). It breaks voice lists and is hard to validate.

Text handling

• UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
• Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
• Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine should not fail on an unknown character; skip or verbalize it as it prefers.
• Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints for pacing.

Errors, limits, and behavior

• One request = one complete audio object. No chunked playback, no callbacks.
• Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget if the engine is slower. A warm HTTP server is the way to meet this.
• Requests are serialized by Veskslop, so the engine does not need a request queue, but it should tolerate repeated back-to-back requests.
• The daemon never retries a failed line. Make failures visible in the response and stderr.
• Do not write to the veskslop_tts sink or the veskslop_mic source directly. The daemon owns playback routing.

Optional: caching

Veskslop does not cache audio. The engine may cache by hash(text + voice + speed + pitch) and return the cached file or bytes. This is useful for repeated lines and for expensive models.

Veskslop changes needed

If the engine wants first-class voice lists, speed, or pitch in the Veskslop UI, Veskslop must:

1. Read the engine's /voices endpoint and merge it into /api/audio/tts/voices.
2. Add speed and pitch to the tts preferences and to the speak request.
3. Add controls to the TTS panel and pass the values through to the provider.

Until those exist, use defaults configured in the engine.

Acceptance checklist

• Engine starts and answers on a stable URL or command.
• Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
• First audible word starts within about 500 ms of the request.
• Speech is clean: no clicks, no clipping, no long leading silence.
• Lines are consistent in loudness.
• voice accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
• speed and pitch are optional and documented in /voices.
• Errors return a short reason and do not leave the engine wedged.
• No files are written outside the engine's own cache or the {out} path.

---

Clarifications from the engine side (MicroslopSam)

Added 2026-09-15 by the engine implementer. The contract above stands; these points record what the engine needs to interpret, add, or hide.

Output pipeline (WAV requirements)

• Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices (Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
• Raw Flextalk output clips (peak 32768, RMS about 22000) and MSTTS is much quieter, so one shared level target is required. The engine post-processes every line: resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
• Proposed implementation, engine-side, with ffmpeg (already installed on the dev host): ffmpeg -i raw.wav -af "highpass=f=40,<trim><fade>,loudnorm=..." -ar 48000 -ac 1 out.wav. The WAV handed to Veskslop is then compliant; the "Veskslop does not normalize" rule holds.

Voice IDs

• Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but has no installed data for.
• Assigned stable IDs: sam, mary, mike, keith-bell, kerry-watson; variants get mike-telephone, mike-hall, mary-stadium, mike-space, and so on. Unknown IDs fall back to sam.
• /voices (HTTP mode) reports language: "en", gender, and the speed/pitch ranges below.

Speed and pitch mapping

• SAPI4 uses engine units, not multipliers or semitones. Observed values:
  • MSTTS Sam: speed 30-450 (default 150), pitch 50-200 (default 100).
  • Flextalk: speed 50-390, pitch 50-250; the engine reports the defaults as the maximum, so the wrapper uses the voice profile from the registry as the natural values (Keith Bell: pitch 27, rate 33; Kerry Watson: pitch 65, rate 32).
• Mapping (approximate, stated in /voices): speed 1.0 = the voice's natural rate; 0.5-2.0 scales that rate and clamps to the engine range. pitch ±12 semitones scales the engine pitch range by about ±25% around the natural value.
• Command mode: {setting} is unused for now. The wrapper reads VESKSLOP_TTS_SPEED and VESKSLOP_TTS_PITCH from the environment when Veskslop later exports them.

Text handling

• SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and unmappable characters to spaces. Engine tag parsing is disabled, so input stays plain text (no SSML), matching the contract.
• Lines up to 1000 characters are well within engine limits.

Transport and latency

• Command provider (first target): a wrapper script reads {text_file}, calls the engine, post-processes, writes {out}, and exits non-zero with one short stderr line on failure.
• Measured on the dev host (warm Wine): Sam line about 288 ms wall, Flextalk line about 41 ms wall, including process start, synthesis, and file write. Both are inside the 500 ms budget.
• HTTP provider (later): a persistent process returning audio/wav bytes and /voices, with optional caching by hash(text + voice + speed + pitch).

Quirks the wrapper must hide

• Flextalk reports default pitch/speed as maximum through ITTSAttributes.
• The mode list includes TruVoice names with no installed voice data; filter those out of /voices (or mark them unavailable).
• Wine needs a working X display; the project helper starts a persistent Xvfb.

Licensing

• The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a local integration; do not redistribute the DLLs or voice data with Veskslop.

Extra acceptance items for the engine

• Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
• Leading silence under 100 ms.
• Voice ID fallback works; TruVoice entries filtered.
• Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).