MicroslopSam/docs/TTS-DESIGN.md

# TTS-DESIGN.md — building a TTS engine for Veskslop

This document is the integration contract for a text-to-speech engine that Veskslop will use. It is
written to be self-contained: an engine developer can work from this file alone.

## How Veskslop uses TTS

1. A line of text is queued (`POST /api/audio/speak` or the MCP `speak` tool).
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
3. `ffplay` decodes the file and plays it into the PulseAudio null sink `veskslop_tts`.
4. `veskslop_tts.monitor` is exposed as the source `veskslop_mic`.
5. The Vesktop client uses `veskslop_mic` as its microphone, so the call hears the speech.
6. While TTS plays and another person speaks, the daemon lowers the `veskslop_tts` sink volume to
   30% and restores it after. This is "ducking". The engine does not implement ducking.

Lines are serialized: one synthesis, then playback, then the next line. There is no streaming
playback path and no partial-audio protocol. The engine returns a complete audio object.

## Integration modes

### HTTP provider (recommended)

Set `VESKSLOP_TTS_API_URL`. Select the API provider with `VESKSLOP_TTS_PROVIDER=api` or the
**API TTS** choice in the TTS panel. The daemon sends:

```
POST <VESKSLOP_TTS_API_URL>
Content-Type: application/json
Authorization: Bearer <VESKSLOP_TTS_API_KEY>   (only when the key is set)

{ "text": "Hello there.", "voice": "anna" }
```

The response body must be the audio bytes. `Content-Type` selects the file extension: `audio/mpeg`
or `audio/mp3` → `.mp3`, `audio/ogg` → `.ogg`, `audio/wav` or `audio/x-wav` → `.wav`, anything else
is treated as `.mp3`. Prefer `audio/wav`.

A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs
it and drops the line.

Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the
per-line process start cost.

### Local command provider

Set `VESKSLOP_TTS_CMD`. It takes precedence over the built-in espeak-ng engine. Veskslop replaces
these placeholders, splits the string on whitespace, and runs the result directly (no shell):

| Placeholder | Replaced with |
| --- | --- |
| `{out}` | Output file path, extension `.wav`. Required. |
| `{text_file}` | A UTF-8 text file holding the line. |
| `{voice}` | The selected voice ID, or an empty string. |
| `{speed}` | The rate multiplier, or an empty string when unset. |
| `{pitch}` | The pitch offset in semitones, or an empty string when unset. |
| `{expression}` | The Flextalk Expression profile value, or an empty string when unset. |

The process also receives `VESKSLOP_TTS_VOICE`, `VESKSLOP_TTS_SPEED`,
`VESKSLOP_TTS_PITCH`, and `VESKSLOP_TTS_EXPRESSION` when those are set. Write the audio to
`{out}` and exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it
short.

Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and
shell tricks cannot work. Use a small wrapper script if you need options.

## Audio output requirements

| Property | Recommendation | Notes |
| --- | --- | --- |
| Container | WAV (RIFF) | Any format ffmpeg decodes works, but WAV avoids decoder startup cost. |
| Codec | PCM signed 16-bit little-endian | `pcm_s16le`. |
| Sample rate | 48000 Hz | 44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample. |
| Channels | Mono | Discord voice is effectively mono. Stereo is accepted but adds no value. |
| Duration | One line of speech | No intro/outro music, no chimes. |
| Peak level | About -3 dBFS | Leave headroom; ducking and Discord gain must not clip. |
| Loudness | Consistent across lines | Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume. |
| Silence | Trim start to under 100 ms, keep roughly 20-50 ms of padding | Long leading silence delays the first audible word. |
| Edges | 5-10 ms fade in and out | Prevents clicks. |
| DC offset | None | Center the waveform. |

Veskslop does not normalize, limit, or re-encode the audio. Playback is `ffplay` at 100% sink
volume, so the engine's levels are what Discord hears.

## Voices

A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such
as `anna`, `sv-anna`, or `en-male-2`. The ID can contain letters, digits, `-`, `_`, and `+`.

If the engine has multiple voices, expose them so a UI can list them:

```
GET /voices
[
  {
    "id": "anna",
    "name": "Anna",
    "language": "sv",
    "gender": "female",
    "speed":  { "min": 0.5, "max": 2.0, "default": 1.0 },
    "pitch":  { "min": -12,  "max": 12,  "default": 0, "unit": "semitones" }
  }
]
```

Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after
Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set
the voice through the engine's own configuration and leave the Veskslop voice setting empty.

## Speed and pitch

Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only `text`,
`voice`, and `provider`. No UI or preference exists for speed or pitch.

Recommended semantics for the engine:

- **speed**: a multiplier, `1.0` is normal, useful range `0.5` to `2.0`. Do not change pitch when
  changing speed unless the user asks for it.
- **pitch**: semitones, `0` is the voice's natural pitch, useful range `-12` to `+12`. State the
  unit in the `/voices` response so clients do not guess.

Recommended HTTP shape (extras are optional; an engine may ignore them):

```
{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }
```

Recommended command shape: `{speed}` and `{pitch}` are substituted when set, and the same values
are exported as `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH`. Read either; the environment form
survives placeholder-free commands.

### How Veskslop should add first-class settings

Implemented on the Veskslop side (2026-09-15):

- `tts` preferences carry `speed` (0.5–2.0) and `pitch` (-12–+12, semitones), both nullable.
- `/api/audio/speak` and the MCP `speak` tool accept optional `speed` and `pitch` that override
  the preferences for that line.
- The command provider substitutes `{speed}`/`{pitch}`/`{expression}` and exports
  `VESKSLOP_TTS_SPEED` / `VESKSLOP_TTS_PITCH` / `VESKSLOP_TTS_EXPRESSION` only when the value is set.
- The HTTP provider adds `speed`, `pitch`, and `expression` to the request body only when set.
- The TTS panel has speed presets, speed/pitch sliders, and an Expression slider that appears only
  for voices that declare it (`expression` / `expression=NN` flags). Changes save to preferences.
- Command mode advertises voices through `VESKSLOP_TTS_VOICES`. Entries are separated by commas or
  newlines and use `id`, `id|Name`, `id|Name|gender`, `id|Name|gender|language`, or
  `id|Name|gender|language|flags`, where flags are `+`-separated. Flags: `no-pitch` (engine ignores
  pitch) and `expression` / `expression=NN` (Flextalk profile parameter with its registry default).
  `/api/audio/tts/voices` reports `supportsPitch` and `supportsExpression` /
  `expressionDefault`; the TTS panel hides controls the voice does not support. While a custom
  command is set, the engine's declared voices replace the espeak list. Reading `/voices` for the
  HTTP provider is still future work.

Do not encode settings into the voice string (for example `anna+speed1.2`). It breaks voice lists
and is hard to validate.

## Text handling

- UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
- Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
- Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine
  should not fail on an unknown character; skip or verbalize it as it prefers.
- Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints
  for pacing.

## Errors, limits, and behavior

- One request = one complete audio object. No chunked playback, no callbacks.
- Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget
  if the engine is slower. A warm HTTP server is the way to meet this.
- Requests are serialized by Veskslop, so the engine does not need a request queue, but it should
  tolerate repeated back-to-back requests.
- The daemon never retries a failed line. Make failures visible in the response and stderr.
- Do not write to the `veskslop_tts` sink or the `veskslop_mic` source directly. The daemon owns
  playback routing.

## Optional: caching

Veskslop does not cache audio. The engine may cache by `hash(text + voice + speed + pitch)` and
return the cached file or bytes. This is useful for repeated lines and for expensive models.

## Veskslop changes needed

Status as of 2026-09-15:

1. `speed` and `pitch` are first-class: preferences, per-request overrides on `/api/audio/speak`
   and MCP `speak`, env exports and placeholders for the command provider, request fields for the
   HTTP provider, and two sliders in the TTS panel.
2. Command-mode voices come from `VESKSLOP_TTS_VOICES`. Reading the engine's `GET /voices` for the
   HTTP provider is not implemented yet. Until then, declare voices with the env var or configure
   the voice inside the wrapper.
3. No changes are needed for the audio format; the engine post-processes to the required output.

Until `/voices` is wired, use defaults configured in the engine.

## Acceptance checklist

- [ ] Engine starts and answers on a stable URL or command.
- [ ] Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
- [ ] First audible word starts within about 500 ms of the request.
- [ ] Speech is clean: no clicks, no clipping, no long leading silence.
- [ ] Lines are consistent in loudness.
- [ ] `voice` accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
- [ ] `speed` and `pitch` are optional and documented in `/voices`.
- [ ] Errors return a short reason and do not leave the engine wedged.
- [ ] No files are written outside the engine's own cache or the `{out}` path.

---

## Clarifications from the engine side (MicroslopSam)

_Added 2026-09-15 by the engine implementer. The contract above stands; these points
record what the engine needs to interpret, add, or hide._

### Output pipeline (WAV requirements)

- Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices
  (Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
- Raw Flextalk output **clips** (peak 32768, RMS about 22000) and MSTTS is much quieter,
  so one shared level target is required. The engine post-processes every line:
  resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and
  level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
- Implemented in `setup/vekslop-tts.sh` with ffmpeg: high-pass at 40 Hz, silence trim at
  both ends (trim-start, reverse, trim-start, reverse), 8 ms fades, then a **fixed gain**
  computed from an EBU R128 measurement (`ebur128=peak=true`):
  `gain = min(-16 LUFS target, -3 dBFS true-peak clamp)`, then 48 kHz mono s16le.
- `loudnorm` was rejected: on short lines (under about 3 s) the loudness range is 0 and
  dynamic mode attenuated a 2.8 s line to -37.9 LUFS instead of about -18 LUFS; two-pass
  linear mode showed the same result. The fixed gain is steady for one-line utterances,
  as Veskslop suggested.
- Verified: Keith Bell -18.4 LUFS, Kerry Watson -17.9 LUFS, Sam -20.6 LUFS; every file is
  48 kHz mono s16le at -3.0 dBFS true peak. The WAV handed to Veskslop is compliant; the
  "Veskslop does not normalize" rule holds.

### Voice IDs

- Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS
  variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but
  has no installed data for.
- Assigned stable IDs: `sam`, `mary`, `mike`, `keith-bell`, `kerry-watson`; variants get
  `mike-telephone`, `mike-hall`, `mary-stadium`, `mike-space`, and so on. Unknown IDs fall
  back to `sam`.
- `/voices` (HTTP mode) reports `language: "en"`, gender, and the speed/pitch ranges below.

### Speed and pitch mapping (measured 2026-09-15)

- SAPI4 uses engine units, not multipliers or semitones. Measured on the dev host with a
  correctly typed Flextalk install:
  - MSTTS Sam: speed 30-450, natural about 150; pitch 50-200 with a real audible effect.
  - Flextalk: speed 50-390, natural about 150 (durations at 100/200/300 fit a natural rate
    near 140-150); pitch 50-250 works. Measured F0: Kerry 0 st = 209 Hz, -6 st = 148 Hz,
    +6 st = 250 Hz (clamped); Keith 0 st = 119 Hz, +6 st = 170 Hz.
- **Correction to the earlier note**: the previous "Flextalk ignores pitch / natural 300"
  finding was an artifact of the speaker registry values being stored as `REG_SZ`. The
  engine requires `REG_DWORD` for numeric speaker values; with strings it synthesizes a
  monotone buzz at maximum pitch. Fixed in `setup/setup-flextalk.sh`; no `no-pitch` flag
  is needed.
- Mapping implemented in `setup/vekslop-tts.sh`: `speed 1.0` = natural rate 150 for every
  voice, scaled by the multiplier and clamped to the engine range. `pitch` semitones scale
  the voice's natural pitch unit by `2^(st/12)` and clamp to the range. Natural units are
  about 100 (MSTTS), 106 (Keith Bell) and 186 (Kerry Watson). The engines' F0 response is
  approximate, not semitone-calibrated.
- `{setting}` is unused; the wrapper reads `VESKSLOP_TTS_SPEED` and
  `VESKSLOP_TTS_PITCH` from the environment when Veskslop later exports them.

### Text handling

- SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and
  unmappable characters to spaces. The host passes `TTSDATAFLAG_TAGGED`, so SAPI4 control
  tags in the text are enabled at the engine level; tagged input is not part of this
  contract and is untested. Plain text (no SSML) is the contract.
- Lines up to 1000 characters are well within engine limits.

### Other voice adjustments (beyond speed)

All voices implement SAPI4 `ITTSAttributes`, so **pitch, speed and volume** exist at the
API level. The Veskslop contract exposes only speed and pitch; volume is available in the
host as `sapi4host -V <value>` and is not sent by Veskslop.

- **Volume** (measured 2026-09-15):
  - Flextalk: linear, range 0-65535, default 65535; half (32767) is about -6 dB, 0 is
    silence.
  - MSTTS: default is 0xFFFFFFFF; 0 is silence, but the response is not linear (for
    example 4096 gives about -24 dB while 65535 and larger are full scale). Treat volume
    as effectively on/off for MSTTS.
- **Flextalk per-speaker profile** (applied at voice load, not runtime-adjustable):
  values live in `HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{GUID}` (see
  `docs/NOTES.md` for the full list and defaults). `Pitch`/`Speaking Rate` set the natural
  voice; **Expression** was verified to change the synthesis audibly (Keith Bell at
  Expression 0 / 59 / 100 yields different waveforms with the same duration). Changing a
  profile requires a registry edit and re-selecting the voice.
- **MSTTS voice character** is fixed by the `.vce`/`.cfg` data; the engine exposes no
  per-voice profile in the registry. Its effect variants (telephone, in Hall, in Stadium,
  in Space) are separate voice modes with separate data, exposed as distinct voice IDs
  (`mike-telephone`, `mary-hall`, ...), not runtime effects.
- Not exposed by this integration: SSML, prosody/style tags, or SAPI4 attribute
  extensions beyond pitch/speed/volume.

### Flextalk profile parameters (how to implement)

Each Flextalk speaker has a profile the engine reads when the voice is loaded. It lives in
the registry; **every numeric value must be `REG_DWORD`**:

    HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{11260610-C84A-11ce-AEC4-0000C0E9CB87}   # Keith Bell
    HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{000031CB-3699-551F-AED0-6B4893382D57}   # Kerry Watson

| Value (REG_DWORD unless noted) | Keith Bell | Kerry Watson | Measured effect |
| --- | --- | --- | --- |
| Pitch | 27 | 65 | Natural pitch (F0); the same scale `ITTSAttributes` shifts at runtime |
| Speaking Rate | 33 | 32 | Natural speech rate (about 150 wpm equivalent) |
| Volume | 100 | 100 | Output gain; 0 = silence |
| Expression | 59 | 100 | Audible; changes the waveform at equal duration |
| Prominences | 17 | 17 | Audible; strong prosody/emphasis effect (F0 median 122 -> 197 Hz at 100) |
| Head Size | 46 | 45 | Audible; brightness/level (peak 20001 -> 32205 at 100) |
| Lowering | 40 | 40 | Audible but subtle |
| Age | 30 | 30 | No measured effect on synthesis (informational) |
| Gender (REG_SZ) | "" | "" | No measured effect on synthesis (informational) |
| Mode Name / Speaker Name (REG_SZ) | "Keith Bell" | "Kerry Watson" | Mode name SAPI4 shows for the voice |

How to apply a change:

1. `setup/setup-flextalk.sh` writes these defaults, so re-running it restores them.
2. Profiles are read at voice load. The host loads the engine per request, so the next
   synthesis after a registry change uses the new profile; no engine restart is needed.
3. Example (lower and slow Keith Bell), with `WINEPREFIX` pointing at the project prefix:

       wine reg add 'HKLM\Software\AT&T\Watson\2.0\FlexTalk\2.0\Speakers\{11260610-C84A-11ce-AEC4-0000C0E9CB87}' \
           /v Pitch /t REG_DWORD /d 20 /f
       wine reg add '...' /v "Speaking Rate" /t REG_DWORD /d 28 /f

   Never use `REG_SZ` for numeric values: the engine reads garbage and synthesizes a
   monotone buzz at maximum pitch.
4. Not exposed by the Veskslop contract (which sends only text/voice/speed/pitch). If
   profile control is wanted later, the clean place is the wrapper: read an env var such
   as `VESKSLOP_TTS_PROFILE` (`key=value` list or JSON), write the registry values before
   synthesis, then restore. That keeps the Veskslop interface unchanged.
5. Runtime `ITTSAttributes` pitch/speed/volume layer on top of the profile: the profile
   defines the natural values, the attributes shift them per request.

### Transport and latency

- Installation is one command: `setup/install.sh` (system dependencies with sudo, wine
  prefix + SAPI4 runtime + MSTTS voices, Flextalk extraction/registration, host build).
  Re-running is safe; a fresh machine needs nothing else.
- Command provider, implemented as `setup/vekslop-tts.sh`:
  `VESKSLOP_TTS_CMD="<repo>/setup/vekslop-tts.sh {out} {text_file}"` and
  `VESKSLOP_TTS_VOICES="$(<repo>/setup/vekslop-tts.sh --voices)"`. The wrapper reads the
  text file, transliterates UTF-8 to CP1252, starts or reuses an Xvfb when `DISPLAY` is
  unset, calls the engine, then post-processes (trim, fades, measured fixed gain, 48 kHz
  mono s16le). A stale private Xvfb is retried once.
- Measured on the dev host: engine-only synthesis is about 290 ms wall for Sam and about
  40 ms for Flextalk, including process start; the full wrapper adds the ffmpeg
  measure/apply passes, about 630-880 ms per short line. That is above the 500 ms target;
  the known optimization is a per-voice static gain (skip the measurement pass), expected
  to land around 400-500 ms.
- Manual checks: `setup/run.sh -l`, `setup/run.sh -v "Keith Bell" -t "Hello." -o out.wav`
  (raw engine WAV), and the wrapper above for normalized output.
- HTTP provider (later): a persistent process returning `audio/wav` bytes and `/voices`,
  with optional caching by `hash(text + voice + speed + pitch)`.

### Quirks the wrapper must hide

- Flextalk numeric speaker settings must be `REG_DWORD`; `REG_SZ` values make the engine
  synthesize a buzz. `setup/setup-flextalk.sh` writes them correctly.
- The mode list includes TruVoice names with no installed voice data; filter those out of
  `/voices` (or mark them `unavailable`).
- Wine needs a working X display; the project helper starts a persistent Xvfb.

### Licensing

- The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a
  local integration; do not redistribute the DLLs or voice data with Veskslop.

### Extra acceptance items for the engine

- [x] Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
- [x] Leading silence under 100 ms.
- [x] Voice ID fallback works; TruVoice entries filtered.
- [x] Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).

## Responses from the Veskslop side

_Added 2026-09-15._

- **Env vars and placeholders**: `VESKSLOP_TTS_SPEED` and `VESKSLOP_TTS_PITCH` are exported only
  when a value is set; `{speed}` and `{pitch}` substitute the same values. Reading the environment
  is the safer option because it works even when the command has no placeholders.
- **Defaults**: a missing value means "engine default". Leave the sliders at 1.00x and 0 st in
  Veskslop to keep the fields out of the request entirely.
- **`loudnorm` caution**: single-pass `loudnorm` can pump or over-amplify noise on short lines.
  Set the target explicitly (`loudnorm=I=-16:TP=-1.5:LRA=11`) and verify a few lines with
  `ebur128`. A fixed gain that hits about -3 dBFS peak with a consistent RMS is often steadier for
  one-line utterances. Ducking lowers the sink to 30%, so keep peaks conservative and loudness
  even between lines.
- **Display**: the daemon does not export `DISPLAY` to TTS children. The wrapper must start or
  find its own Xvfb and set `DISPLAY` itself, as the project helper already does. Do not rely on
  the client's display.
- **Engine selection**: the Speech settings and the TTS panel list every configured engine. The
  command engine appears under `VESKSLOP_TTS_NAME` (default "custom engine") and is selectable
  directly; Auto uses the command engine first. Selecting it hides the language row and shows the
  `VESKSLOP_TTS_VOICES` list.
- **Start engine**: the Speech settings have a **Start engine** button (`POST /api/audio/tts/warm`,
  MCP `start_tts_engine`) that runs one throwaway synthesis to warm the command engine (Xvfb, Wine,
  host). Keep the wrapper able to handle a short sentence; the warm-up uses "The quick brown fox
  jumps over the lazy dog." and reports the elapsed milliseconds. Two-pass `loudnorm` must not fail
  on that input.
- **Lead-in silence**: Veskslop prepends a short silence at playback time (`VESKSLOP_TTS_LEAD_MS`,
  default 150 ms) so Discord's voice pipeline is ready before the first syllable. It is added with
  `adelay` in the player, not in the audio file. The engine needs no change and can keep its
  5–10 ms edge fades.
- **Voice declarations**: list only usable IDs in `VESKSLOP_TTS_VOICES`. Veskslop hides the
  espeak list while a custom command is set, so a missing declaration means an empty voice list,
  not a fallback. The list accepts commas or newlines. Optional flags: `no-pitch` (hide the pitch
  control) and `expression` / `expression=NN` (show the Expression slider and use NN as its
  default).
- **Flextalk pitch**: works now that the speaker registry values are `REG_DWORD`; no flag is
  needed and Veskslop shows pitch for Keith Bell and Kerry Watson.
- **Expression profile**: implemented for the Flextalk voices. Veskslop stores `tts.expression`
  (0-100, nullable), sends it per line (`{expression}` / `VESKSLOP_TTS_EXPRESSION` / the HTTP body),
  and shows the slider only for voices flagged `expression=NN`. A null value leaves the registry
  untouched, so the voice keeps its engine default.
- **Licensing**: keep the engine DLLs and voice data local and out of any repository; Veskslop
  never bundles or redistributes them.