260 lines
12 KiB
Markdown
260 lines
12 KiB
Markdown
# TTS-DESIGN.md — building a TTS engine for Veskslop
|
|
|
|
This document is the integration contract for a text-to-speech engine that Veskslop will use. It is
|
|
written to be self-contained: an engine developer can work from this file alone.
|
|
|
|
## How Veskslop uses TTS
|
|
|
|
1. A line of text is queued (`POST /api/audio/speak` or the MCP `speak` tool).
|
|
2. The daemon asks the engine for an audio file (command mode) or audio bytes (HTTP mode).
|
|
3. `ffplay` decodes the file and plays it into the PulseAudio null sink `veskslop_tts`.
|
|
4. `veskslop_tts.monitor` is exposed as the source `veskslop_mic`.
|
|
5. The Vesktop client uses `veskslop_mic` as its microphone, so the call hears the speech.
|
|
6. While TTS plays and another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
|
30% and restores it after. This is "ducking". The engine does not implement ducking.
|
|
|
|
Lines are serialized: one synthesis, then playback, then the next line. There is no streaming
|
|
playback path and no partial-audio protocol. The engine returns a complete audio object.
|
|
|
|
## Integration modes
|
|
|
|
### HTTP provider (recommended)
|
|
|
|
Set `VESKSLOP_TTS_API_URL`. Select the API provider with `VESKSLOP_TTS_PROVIDER=api` or the
|
|
**API TTS** choice in the TTS panel. The daemon sends:
|
|
|
|
```
|
|
POST <VESKSLOP_TTS_API_URL>
|
|
Content-Type: application/json
|
|
Authorization: Bearer <VESKSLOP_TTS_API_KEY> (only when the key is set)
|
|
|
|
{ "text": "Hello there.", "voice": "anna" }
|
|
```
|
|
|
|
The response body must be the audio bytes. `Content-Type` selects the file extension: `audio/mpeg`
|
|
or `audio/mp3` → `.mp3`, `audio/ogg` → `.ogg`, `audio/wav` or `audio/x-wav` → `.wav`, anything else
|
|
is treated as `.mp3`. Prefer `audio/wav`.
|
|
|
|
A non-2xx response is an error. Return a short JSON or text body with the reason; the daemon logs
|
|
it and drops the line.
|
|
|
|
Use the HTTP mode for any engine with a model load time. It keeps the model warm and avoids the
|
|
per-line process start cost.
|
|
|
|
### Local command provider
|
|
|
|
Set `VESKSLOP_TTS_CMD`. It takes precedence over the built-in espeak-ng engine. Veskslop replaces
|
|
these placeholders, splits the string on whitespace, and runs the result directly (no shell):
|
|
|
|
| Placeholder | Replaced with |
|
|
| --- | --- |
|
|
| `{out}` | Output file path, extension `.wav`. Required. |
|
|
| `{text_file}` | A UTF-8 text file holding the line. |
|
|
| `{voice}` | The selected voice ID, or an empty string. |
|
|
| `{setting}` | Reserved for speed/pitch once Veskslop grows settings support. |
|
|
|
|
The process also receives `VESKSLOP_TTS_VOICE` in the environment. Write the audio to `{out}` and
|
|
exit 0. Non-zero exits are errors; stderr is captured into the log message, so keep it short.
|
|
|
|
Command mode is best for fast, stateless engines. It has no shell quoting: paths with spaces and
|
|
shell tricks cannot work. Use a small wrapper script if you need options.
|
|
|
|
## Audio output requirements
|
|
|
|
| Property | Recommendation | Notes |
|
|
| --- | --- | --- |
|
|
| Container | WAV (RIFF) | Any format ffmpeg decodes works, but WAV avoids decoder startup cost. |
|
|
| Codec | PCM signed 16-bit little-endian | `pcm_s16le`. |
|
|
| Sample rate | 48000 Hz | 44100 Hz is fine. Discord re-encodes to 48 kHz Opus; a 48 kHz source avoids one resample. |
|
|
| Channels | Mono | Discord voice is effectively mono. Stereo is accepted but adds no value. |
|
|
| Duration | One line of speech | No intro/outro music, no chimes. |
|
|
| Peak level | About -3 dBFS | Leave headroom; ducking and Discord gain must not clip. |
|
|
| Loudness | Consistent across lines | Aim for a stable integrated loudness (roughly -16 to -20 LUFS) so lines do not jump in volume. |
|
|
| Silence | Trim start to under 100 ms, keep roughly 20-50 ms of padding | Long leading silence delays the first audible word. |
|
|
| Edges | 5-10 ms fade in and out | Prevents clicks. |
|
|
| DC offset | None | Center the waveform. |
|
|
|
|
Veskslop does not normalize, limit, or re-encode the audio. Playback is `ffplay` at 100% sink
|
|
volume, so the engine's levels are what Discord hears.
|
|
|
|
## Voices
|
|
|
|
A voice ID is an opaque string to Veskslop. Prefer short, stable, lowercase ASCII identifiers such
|
|
as `anna`, `sv-anna`, or `en-male-2`. The ID can contain letters, digits, `-`, `_`, and `+`.
|
|
|
|
If the engine has multiple voices, expose them so a UI can list them:
|
|
|
|
```
|
|
GET /voices
|
|
[
|
|
{
|
|
"id": "anna",
|
|
"name": "Anna",
|
|
"language": "sv",
|
|
"gender": "female",
|
|
"speed": { "min": 0.5, "max": 2.0, "default": 1.0 },
|
|
"pitch": { "min": -12, "max": 12, "default": 0, "unit": "semitones" }
|
|
}
|
|
]
|
|
```
|
|
|
|
Note: Veskslop currently lists voices from espeak-ng only. A custom engine's voices appear after
|
|
Veskslop adds support for reading this endpoint (see "Veskslop changes needed"). Until then, set
|
|
the voice through the engine's own configuration and leave the Veskslop voice setting empty.
|
|
|
|
## Speed and pitch
|
|
|
|
Speed and pitch are engine settings, not Veskslop concepts today. Veskslop passes only `text`,
|
|
`voice`, and `provider`. No UI or preference exists for speed or pitch.
|
|
|
|
Recommended semantics for the engine:
|
|
|
|
- **speed**: a multiplier, `1.0` is normal, useful range `0.5` to `2.0`. Do not change pitch when
|
|
changing speed unless the user asks for it.
|
|
- **pitch**: semitones, `0` is the voice's natural pitch, useful range `-12` to `+12`. State the
|
|
unit in the `/voices` response so clients do not guess.
|
|
|
|
Recommended HTTP shape (extras are optional; an engine may ignore them):
|
|
|
|
```
|
|
{ "text": "...", "voice": "anna", "speed": 1.15, "pitch": -2 }
|
|
```
|
|
|
|
Recommended command shape: `{setting}` is reserved. Until Veskslop passes it, read defaults from
|
|
the engine's own config file or environment.
|
|
|
|
### How Veskslop should add first-class settings
|
|
|
|
A future change would add `speed` and `pitch` to `tts` preferences, pass them to the provider
|
|
alongside `voice`, and add two controls to the TTS panel. For the command provider, `{speed}` and
|
|
`{pitch}` placeholders (or one `{settings_file}` JSON file) would be substituted the same way as
|
|
`{voice}`. Design the engine API so it works with defaults when these fields are absent.
|
|
|
|
Do not encode settings into the voice string (for example `anna+speed1.2`). It breaks voice lists
|
|
and is hard to validate.
|
|
|
|
## Text handling
|
|
|
|
- UTF-8 input. Lines arrive trimmed and are at most 1000 characters.
|
|
- Do not parse or require SSML. Plain text is the contract. SSML support can be additive.
|
|
- Emoji, URLs, markdown, and code blocks can appear. Veskslop does not sanitize them. The engine
|
|
should not fail on an unknown character; skip or verbalize it as it prefers.
|
|
- Keep punctuation-based prosody. A period, comma, question mark, and newline are the only hints
|
|
for pacing.
|
|
|
|
## Errors, limits, and behavior
|
|
|
|
- One request = one complete audio object. No chunked playback, no callbacks.
|
|
- Synthesis blocks the queue. Target under about 500 ms for a short line; state a realistic budget
|
|
if the engine is slower. A warm HTTP server is the way to meet this.
|
|
- Requests are serialized by Veskslop, so the engine does not need a request queue, but it should
|
|
tolerate repeated back-to-back requests.
|
|
- The daemon never retries a failed line. Make failures visible in the response and stderr.
|
|
- Do not write to the `veskslop_tts` sink or the `veskslop_mic` source directly. The daemon owns
|
|
playback routing.
|
|
|
|
## Optional: caching
|
|
|
|
Veskslop does not cache audio. The engine may cache by `hash(text + voice + speed + pitch)` and
|
|
return the cached file or bytes. This is useful for repeated lines and for expensive models.
|
|
|
|
## Veskslop changes needed
|
|
|
|
If the engine wants first-class voice lists, speed, or pitch in the Veskslop UI, Veskslop must:
|
|
|
|
1. Read the engine's `/voices` endpoint and merge it into `/api/audio/tts/voices`.
|
|
2. Add `speed` and `pitch` to the `tts` preferences and to the `speak` request.
|
|
3. Add controls to the TTS panel and pass the values through to the provider.
|
|
|
|
Until those exist, use defaults configured in the engine.
|
|
|
|
## Acceptance checklist
|
|
|
|
- [ ] Engine starts and answers on a stable URL or command.
|
|
- [ ] Output is mono PCM WAV at 48 kHz (or 44.1 kHz), peak near -3 dBFS.
|
|
- [ ] First audible word starts within about 500 ms of the request.
|
|
- [ ] Speech is clean: no clicks, no clipping, no long leading silence.
|
|
- [ ] Lines are consistent in loudness.
|
|
- [ ] `voice` accepts stable ASCII IDs; unknown IDs fall back to a default instead of failing.
|
|
- [ ] `speed` and `pitch` are optional and documented in `/voices`.
|
|
- [ ] Errors return a short reason and do not leave the engine wedged.
|
|
- [ ] No files are written outside the engine's own cache or the `{out}` path.
|
|
|
|
---
|
|
|
|
## Clarifications from the engine side (MicroslopSam)
|
|
|
|
_Added 2026-09-15 by the engine implementer. The contract above stands; these points
|
|
record what the engine needs to interpret, add, or hide._
|
|
|
|
### Output pipeline (WAV requirements)
|
|
|
|
- Raw SAPI4 output is mono PCM s16le WAV but not at the target rate: the MSTTS voices
|
|
(Sam/Mary/Mike) render at 22050 Hz, the Flextalk voices at 11025 Hz.
|
|
- Raw Flextalk output **clips** (peak 32768, RMS about 22000) and MSTTS is much quieter,
|
|
so one shared level target is required. The engine post-processes every line:
|
|
resample to 48000 Hz, trim leading silence (<100 ms), 5-10 ms fades, DC removal, and
|
|
level normalization (peak about -3 dBFS, integrated loudness -16 to -20 LUFS).
|
|
- Proposed implementation, engine-side, with ffmpeg (already installed on the dev host):
|
|
`ffmpeg -i raw.wav -af "highpass=f=40,<trim><fade>,loudnorm=..." -ar 48000 -ac 1 out.wav`.
|
|
The WAV handed to Veskslop is then compliant; the "Veskslop does not normalize" rule holds.
|
|
|
|
### Voice IDs
|
|
|
|
- Engine voice names are "Sam", "Mary", "Mike", "Keith Bell", "Kerry Watson", plus MSTTS
|
|
variants (telephone, hall, stadium, space) and TruVoice names that the engine lists but
|
|
has no installed data for.
|
|
- Assigned stable IDs: `sam`, `mary`, `mike`, `keith-bell`, `kerry-watson`; variants get
|
|
`mike-telephone`, `mike-hall`, `mary-stadium`, `mike-space`, and so on. Unknown IDs fall
|
|
back to `sam`.
|
|
- `/voices` (HTTP mode) reports `language: "en"`, gender, and the speed/pitch ranges below.
|
|
|
|
### Speed and pitch mapping
|
|
|
|
- SAPI4 uses engine units, not multipliers or semitones. Observed values:
|
|
- MSTTS Sam: speed 30-450 (default 150), pitch 50-200 (default 100).
|
|
- Flextalk: speed 50-390, pitch 50-250; the engine reports the **defaults as the
|
|
maximum**, so the wrapper uses the voice profile from the registry as the natural
|
|
values (Keith Bell: pitch 27, rate 33; Kerry Watson: pitch 65, rate 32).
|
|
- Mapping (approximate, stated in `/voices`): `speed 1.0` = the voice's natural rate;
|
|
0.5-2.0 scales that rate and clamps to the engine range. `pitch` ±12 semitones scales the
|
|
engine pitch range by about ±25% around the natural value.
|
|
- Command mode: `{setting}` is unused for now. The wrapper reads `VESKSLOP_TTS_SPEED` and
|
|
`VESKSLOP_TTS_PITCH` from the environment when Veskslop later exports them.
|
|
|
|
### Text handling
|
|
|
|
- SAPI4 is ANSI/CP1252. The wrapper converts UTF-8 to CP1252 and maps emoji and
|
|
unmappable characters to spaces. Engine tag parsing is disabled, so input stays plain
|
|
text (no SSML), matching the contract.
|
|
- Lines up to 1000 characters are well within engine limits.
|
|
|
|
### Transport and latency
|
|
|
|
- Command provider (first target): a wrapper script reads `{text_file}`, calls the engine,
|
|
post-processes, writes `{out}`, and exits non-zero with one short stderr line on failure.
|
|
- Measured on the dev host (warm Wine): Sam line about 288 ms wall, Flextalk line about
|
|
41 ms wall, including process start, synthesis, and file write. Both are inside the
|
|
500 ms budget.
|
|
- HTTP provider (later): a persistent process returning `audio/wav` bytes and `/voices`,
|
|
with optional caching by `hash(text + voice + speed + pitch)`.
|
|
|
|
### Quirks the wrapper must hide
|
|
|
|
- Flextalk reports default pitch/speed as maximum through `ITTSAttributes`.
|
|
- The mode list includes TruVoice names with no installed voice data; filter those out of
|
|
`/voices` (or mark them `unavailable`).
|
|
- Wine needs a working X display; the project helper starts a persistent Xvfb.
|
|
|
|
### Licensing
|
|
|
|
- The engines and voice data are proprietary Microsoft/AT&T/L&H artifacts. Keep this a
|
|
local integration; do not redistribute the DLLs or voice data with Veskslop.
|
|
|
|
### Extra acceptance items for the engine
|
|
|
|
- [ ] Post-processed WAV is 48 kHz mono s16le, peak about -3 dBFS, no clipping, fades present.
|
|
- [ ] Leading silence under 100 ms.
|
|
- [ ] Voice ID fallback works; TruVoice entries filtered.
|
|
- [ ] Latency recorded (Sam about 288 ms, Flextalk about 41 ms on the dev host).
|