MicroslopSam/docs/TTS-DESIGN.md

# TTS-DESIGN.md — building a TTS engine for Veskslop

This document is the integration contract for a text-to-speech engine that Veskslop will use. It is
written to be self-contained: an engine developer can work from this file alone.

## How Veskslop uses TTS

1. A line of text is queued (`POST /api/audio/speak` or the MCP `speak` tool).
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
3. `ffplay` decodes the file and plays it into the PulseAudio null sink `veskslop_tts`.
4. `veskslop_tts.monitor` is exposed as the source `veskslop_mic`.
5. The Vesktop client uses `veskslop_mic` as its microphone, so the call hears the speech.
6. While TTS plays and another person speaks, the daemon lowers the `veskslop_tts` sink volume to
   30% and restores it after. This is "ducking". The engine does not implement ducking.

Lines are serialized: one synthesis, then playback, then the next line. There is no streaming
playback path and no partial-audio protocol. The engine returns a complete audio object.

## Integration modes

### HTTP provider (recommended)

Set `VESKSLOP_TTS_API_URL`. Select the API provider with `VESKSLOP_TTS_PROVIDER=api` or the
**API TTS** choice in the TTS panel. The daemon sends:

```
POST <VESKSLOP_TTS_API_URL>
Content-Type: application/json
Authorization: Bearer <VESKSLOP_TTS_API_KEY>   (only when the key is set)

{ "text": "Hello there.", "voice": "anna" }
```

The response body must be the audio bytes. `Content-Type` selects the file extension: `audio/mpeg`
or `audio/mp3` → `.mp3`, `audio/ogg` → `.ogg`, `audio/wav` or `audio/x-wav` → `.wav`, anything else
is treated as `.mp3`. Prefer `audio/wav`.

A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs
it and drops the line.

Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the
per-line process start cost.

### Local command provider

Set `VESKSLOP_TTS_CMD`. It takes precedence over the built-in espeak-ng engine. Veskslop replaces
these placeholders, splits the string on whitespace, and runs the result directly (no shell):

| Placeholder | Replaced with |
| --- | --- |
| `{out}` | Output file path, extension `.wav`. Required. |
| `{text_file}` | A UTF-8 text file holding the line. |
| `{voice}` | The selected voice ID, or an empty string. |
| `{speed}` | The rate multiplier, or an empty string when unset. |
| `{pitch}` | The pitch offset in semitones, or an empty string when unset. |

The process also receives `VESKSLOP_TTS_VOICE`, and `VESKSLOP_TTS_SPEED` / `VESKSLOP_TTS_PITCH`
when those are set. Write the audio to `{out}` and exit 0. Non-zero exits are errors; stderr is
captured into the log message, so keep it short.

Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and
shell tricks cannot work. Use a small wrapper script if you need options.

## Audio output requirements

| Property | Recommendation | Notes |
| --- | --- | --- |
| Container | WAV (RIFF) | Any format ffmpeg decodes works, but WAV avoids decoder startup cost. |
| Codec | PCM signed 16-bit little-endian | `pcm_s16le`. |
| Sample rate | 48000 Hz | 44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample. |
| Channels | Mono | Discord voice is effectively mono. Stereo is accepted but adds no value. |
| Duration | One line of speech | No intro/outro music, no chimes. |
| Peak level | About -3 dBFS | Leave headroom; ducking and Discord gain must not clip. |
| Loudness | Consistent across lines | Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume. |
| Silence | Trim start to under 100 ms, keep roughly 20-50 ms of padding | Long leading silence delays the first audible word. |
| Edges | 5-10 ms fade in and out | Prevents clicks. |
| DC offset | None | Center the waveform. |

Veskslop does not normalize, limit, or re-encode the audio. Playback is `ffplay` at 100% sink
volume, so the engine's levels are what Discord hears.

## Voices

A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such
as `anna`, `sv-anna`, or `en-male-2`. The ID can contain letters, digits, `-`, `_`, and `+`.

If the engine has multiple voices, expose them so a UI can list them:

```
GET /voices
[
  {
    "id": "anna",
    "name": "Anna",
    "language": "sv",
    "gender": "female",
    "speed":  { "min": 0.5, "max": 2.0, "default": 1.0 },
    "pitch":  { "min": -12,  "max": 12,  "default": 0, "unit": "semitones" }
  }
]
```

Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after
Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set
the voice through the engine's own configuration and leave the Veskslop voice setting empty.

## Speed and pitch

Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only `text`,
`voice`, and `provider`. No UI or preference exists for speed or pitch.

Recommended semantics for the engine:

- **speed**: a multiplier, `1.0` is normal, useful range `0.5` to `2.0`. Do not change pitch when
  changing speed unless the user asks for it.
- **pitch**: semitones, `0` is the voice's natural pitch, useful range `-12` to `+12`. State the
  unit in the `/voices` response so clients do not guess.

Recommended HTTP shape (extras are optional; an engine may ignore them):

```
{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }
```

Recommended command shape: `{speed}` and `{pitch}` are substituted when set, and the same values
are exported as `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH`. Read either; the environment form
survives placeholder-free commands.

### How Veskslop should add first-class settings

Implemented on the Veskslop side (2026-09-15):

- `tts` preferences carry `speed` (0.5–2.0) and `pitch` (-12–+12, semitones), both nullable.
- `/api/audio/speak` and the MCP `speak` tool accept optional `speed` and `pitch` that override
  the preferences for that line.
- The command provider substitutes `{speed}`/`{pitch}` and exports `VESKSLOP_TTS_SPEED` /
  `VESKSLOP_TTS_PITCH` only when the value is set.
- The HTTP provider adds `speed` and `pitch` to the request body only when set.
- The TTS panel has speed and pitch sliders; changes are saved to the daemon preferences.
- Command mode advertises voices through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or
  newlines and use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
  `id|Name|gender|language|flags`, where flags are `+`-separated. The `no-pitch` flag marks a voice
  that ignores pitch; `/api/audio/tts/voices` then reports `supportsPitch: false` and the TTS panel
  disables the pitch slider. While a custom command is set, the engine's declared voices replace
  the espeak list, so no misleading espeak entries appear. Reading `/voices` for the HTTP provider
  is still future work.

Do not encode settings into the voice string (for example `anna+speed1.2`). It breaks voice lists
and is hard to validate.

## Text handling

- UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
- Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
- Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine
  should not fail on an unknown character; skip or verbalize it as it prefers.
- Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints
  for pacing.

## Errors, limits, and behavior

- One request = one complete audio object. No chunked playback, no callbacks.
- Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget
  if the engine is slower. A warm HTTP server is the way to meet this.
- Requests are serialized by Veskslop, so the engine does not need a request queue, but it should
  tolerate repeated back-to-back requests.
- The daemon never retries a failed line. Make failures visible in the response and stderr.
- Do not write to the `veskslop_tts` sink or the `veskslop_mic` source directly. The daemon owns
  playback routing.

## Optional: caching

Veskslop does not cache audio. The engine may cache by `hash(text + voice + speed + pitch)` and
return the cached file or bytes. This is useful for repeated lines and for expensive models.

## Veskslop changes needed

Status as of 2026-09-15:

1. `speed` and `pitch` are first-class: preferences, per-request overrides on `/api/audio/speak`
   and MCP `speak`, env exports and placeholders for the command provider, request fields for the
   HTTP provider, and two sliders in the TTS panel.
2. Command-mode voices come from `VESKSLOP_TTS_VOICES`. Reading the engine's `GET /voices` for the
   HTTP provider is not implemented yet. Until then, declare voices with the env var or configure
   the voice inside the wrapper.
3. No changes are needed for the audio format; the engine post-processes to the required output.

Until `/voices` is wired, use defaults configured in the engine.

## Acceptance checklist

- [ ] Engine starts and answers on a stable URL or command.
- [ ] Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
- [ ] First audible word starts within about 500 ms of the request.
- [ ] Speech is clean: no clicks, no clipping, no long leading silence.
- [ ] Lines are consistent in loudness.
- [ ] `voice` accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
- [ ] `speed` and `pitch` are optional and documented in `/voices`.
- [ ] Errors return a short reason and do not leave the engine wedged.
- [ ] No files are written outside the engine's own cache or the `{out}` path.

---

## Clarifications from the engine side (MicroslopSam)

_Added 2026-09-15 by the engine implementer. The contract above stands; these points
record what the engine needs to interpret, add, or hide._

### Output pipeline (WAV requirements)

- Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices
  (Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
- Raw Flextalk output **clips** (peak 32768, RMS about 22000) and MSTTS is much quieter,
  so one shared level target is required. The engine post-processes every line:
  resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and
  level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
- Proposed implementation, engine-side, with ffmpeg (already installed on the dev host):
  `ffmpeg -i raw.wav -af "highpass=f=40,<trim><fade>,loudnorm=..." -ar 48000 -ac 1 out.wav`.
  The WAV handed to Veskslop is then compliant; the "Veskslop does not normalize" rule holds.

### Voice IDs

- Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS
  variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but
  has no installed data for.
- Assigned stable IDs: `sam`, `mary`, `mike`, `keith-bell`, `kerry-watson`; variants get
  `mike-telephone`, `mike-hall`, `mary-stadium`, `mike-space`, and so on. Unknown IDs fall
  back to `sam`.
- `/voices` (HTTP mode) reports `language: "en"`, gender, and the speed/pitch ranges below.

### Speed and pitch mapping (measured 2026-09-15)

- SAPI4 uses engine units, not multipliers or semitones. Measured on the dev host:
  - MSTTS Sam: speed 30-450, natural about 150; pitch 50-200 with a real audible effect
    (F0 about 134 Hz at pitch 50, about 199 Hz at pitch 200 on the test phrase).
  - Flextalk: speed 50-390, natural about 300 (output with no setting is byte-identical to
    `-s 300`); **pitch has no effect** (byte-identical output at pitch 100/150/200), so the
    wrapper ignores `pitch` for Keith Bell and Kerry Watson.
- Mapping implemented in `setup/vekslop-tts.sh`: `speed 1.0` = the voice's natural rate
  (150 MSTTS, 300 Flextalk), scaled by the multiplier and clamped to the engine range.
  `pitch` for MSTTS maps ±12 semitones to engine units `100 ± 6.25·st`, clamped to 50-200.
  Pitch is a no-op for Flextalk voices; the wrapper's `--voices` output marks
  `keith-bell` and `kerry-watson` with the `no-pitch` flag so the UI disables the slider.
- `{setting}` is unused; the wrapper reads `VESKSLOP_TTS_SPEED` and
  `VESKSLOP_TTS_PITCH` from the environment when Veskslop later exports them.

### Text handling

- SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and
  unmappable characters to spaces. Engine tag parsing is disabled, so input stays plain
  text (no SSML), matching the contract.
- Lines up to 1000 characters are well within engine limits.

### Transport and latency

- Command provider, implemented as `setup/vekslop-tts.sh`:
  `VESKSLOP_TTS_CMD="<repo>/setup/vekslop-tts.sh {out} {text_file}"` and
  `VESKSLOP_TTS_VOICES="$(<repo>/setup/vekslop-tts.sh --voices)"`. The wrapper reads the
  text file, transliterates UTF-8 to CP1252, starts or reuses an Xvfb when `DISPLAY` is
  unset, calls the engine, then post-processes (trim, fades, two-pass `loudnorm`, 48 kHz
  mono s16le). Verified: -17.7 LUFS integrated, -3.0 dBFS true peak.
- Measured on the dev host (warm Wine): Sam line about 288 ms wall, Flextalk line about
  41 ms wall, including process start, synthesis, and file write. Both are inside the
  500 ms budget.
- HTTP provider (later): a persistent process returning `audio/wav` bytes and `/voices`,
  with optional caching by `hash(text + voice + speed + pitch)`.

### Quirks the wrapper must hide

- Flextalk reports default pitch/speed as maximum through `ITTSAttributes`.
- The mode list includes TruVoice names with no installed voice data; filter those out of
  `/voices` (or mark them `unavailable`).
- Wine needs a working X display; the project helper starts a persistent Xvfb.

### Licensing

- The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a
  local integration; do not redistribute the DLLs or voice data with Veskslop.

### Extra acceptance items for the engine

- [x] Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
- [x] Leading silence under 100 ms.
- [x] Voice ID fallback works; TruVoice entries filtered.
- [x] Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).

## Responses from the Veskslop side

_Added 2026-09-15._

- **Env vars and placeholders**: `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH` are exported only
  when a value is set; `{speed}` and `{pitch}` substitute the same values. Reading the environment
  is the safer option because it works even when the command has no placeholders.
- **Defaults**: a missing value means "engine default". Leave the sliders at 1.00x and 0 st in
  Veskslop to keep the fields out of the request entirely.
- **`loudnorm` caution**: single-pass `loudnorm` can pump or over-amplify noise on short lines.
  Set the target explicitly (`loudnorm=I=-16:TP=-1.5:LRA=11`) and verify a few lines with
  `ebur128`. A fixed gain that hits about -3 dBFS peak with a consistent RMS is often steadier for
  one-line utterances. Ducking lowers the sink to 30%, so keep peaks conservative and loudness
  even between lines.
- **Display**: the daemon does not export `DISPLAY` to TTS children. The wrapper must start or
  find its own Xvfb and set `DISPLAY` itself, as the project helper already does. Do not rely on
  the client's display.
- **Voice declarations**: list only usable IDs in `VESKSLOP_TTS_VOICES`. Veskslop hides the
  espeak list while a custom command is set, so a missing declaration means an empty voice list,
  not a fallback. The list accepts commas or newlines, and the optional `no-pitch` flag tells the
  UI to grey out the pitch slider for that voice.
- **Flextalk pitch**: confirmed that pitch is a no-op for Keith Bell and Kerry Watson. Mark those
  entries `no-pitch` so users are not offered a control that does nothing. Speed applies to every
  voice and needs no flag.
- **Licensing**: keep the engine DLLs and voice data local and out of any repository; Veskslop
  never bundles or redistributes them.