diff --git a/CURRENT.md b/CURRENT.md index 5bc848f..1e8c463 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -40,6 +40,20 @@ mobile draft at 390px. - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: `previewed: 2`; thumbs served over `/api/media///thumb` (200). + - [x] C77 Split user audio investigation. [docs/voice-audio-split.md](docs/voice-audio-split.md) + records that the daemon sees only the mixed client output (`VeskslopCapture.monitor` → VAD → + STT) and that per-user tracks exist only in the renderer as WebRTC `MediaStreamTrack`s. The + core bridge has no WebRTC hooks today. The plan: hook `RTCPeerConnection` for tracks, hook + `AudioContext` for track-to-user mapping, stream 16 kHz PCM frames to the daemon, add + `preferences.audio.splitUserAudio` (default off), and keep the mixed path as the fallback. Not + implemented. + - [x] C76 TTS volume and browser preview. `preferences.tts.volume` (0-200, null = natural) + rides the speak options; the daemon applies it at play time with an ffmpeg `volume` filter + (ffplay's own `-volume` stops at 100). The TTS panel gained a volume slider and a **Preview** + button that synthesizes through `POST /api/audio/preview` and plays the returned + `/api/audio/tts/` WAV in the browser. Preview never queues, ducks, or touches the + virtual mic. Verified live: a 1.6 s WAV served as `audio/wav`, and the volume preference + round-trips. - [x] C75 Wakeword routing and the reply-only agent path. Commands now need the wakeword plus a space at the start of the message: default `!s`, so `!s whitelist list`. `!space`, `hey!s cmd`, and bare commands no longer trigger. Settings → Commands picks a preset (`!s !x diff --git a/docs/INDEX.md b/docs/INDEX.md index d8db315..8fd21de 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -11,6 +11,7 @@ This file lists the project documents. | [daemon.md](daemon.md) | The external MCP daemon | | [webui.md](webui.md) | The web control UI and its JSON API | | [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle | +| [voice-audio-split.md](voice-audio-split.md) | Per-user audio investigation and the split-user STT plan | | [media.md](media.md) | Media archive: command, storage, limits, and Web UI | | [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema | | [media-mcp.md](media-mcp.md) | The media MCP tool surface | diff --git a/docs/voice-audio-split.md b/docs/voice-audio-split.md new file mode 100644 index 0000000..901e1d3 --- /dev/null +++ b/docs/voice-audio-split.md @@ -0,0 +1,79 @@ +# Split user audio — investigation and plan + +Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed +stream. + +## What happens today + +The daemon does not see per-user audio. The client mixes every remote user before the audio leaves +the browser. + +| Step | Where | Detail | +| --- | --- | --- | +| Playback | Vesktop renderer | Discord mixes the remote tracks. Per-user volume is applied before the mix. | +| Routing | `poc/src/audio.ts` | `pactl move-sink-input` moves the client's playback streams into the `VeskslopCapture` sink. | +| Capture | `poc/src/capture.ts` | `ffmpeg` records `VeskslopCapture.monitor` as raw PCM. One stream, all users. | +| Split | `poc/src/vad.ts` | The VAD cuts the stream into utterances and writes one WAV each. | +| Text | `poc/src/stt.ts`, `poc/src/transcript.ts` | STT turns each WAV into text when a provider is configured. | +| Speaker | `poc/src/audio-service.ts` | `VOICE_SPEAKING` events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user. | + +The core bridge (`src/renderer/puppet/actions.ts`) has no WebRTC hooks today. The only audio call +is `navigator.mediaDevices.enumerateDevices()` for the device list. + +## Where per-user audio exists + +Each remote user arrives in the renderer as a WebRTC `MediaStreamTrack`. Discord attaches the +tracks to an `RTCPeerConnection` and mixes them in an `AudioContext` graph. Per-user gain (the +volume sliders) is a node on each track's path. The OS output receives only the mix. + +## Split options + +| Option | How | Cost | +| --- | --- | --- | +| Hook `RTCPeerConnection` | Wrap `ontrack` and `addEventListener("track")` before Discord's code runs, then keep every remote track | Straightforward to hook, but the track carries no user ID | +| Hook `AudioContext` | Wrap `createMediaStreamSource` and `createGain`. Discord makes one source per user, so the graph shows the per-user split | Better mapping chance, but fragile against client updates | +| Map by event order | Pair the first chunk after a `VOICE_SPEAKING` event with that user | Works with heavy traffic, mislabels overlaps | +| Chromium capture | Use Electron audio capture of `webContents` | Gives the mix again, not per-user | + +Recommended path: hook `RTCPeerConnection` for the tracks, and hook `AudioContext` only to map a +track to a user (the source node creation order matches the track order, and the per-user gain node +sits next to it). Keep the event-order pairing as the fallback for unknown tracks. + +## Transport for per-user PCM + +The daemon needs one PCM stream per user. Two candidate shapes: + +1. **AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge.** A 20 ms frame is 640 bytes, + about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge + traffic, small renderer cost. +2. **MediaRecorder per track, Opus chunks every second.** Lower traffic and less renderer CPU. + The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU. + +Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon +already runs ffmpeg for resampling. Add a binary frame type later when the volume grows. + +## Plan + +| Step | Work | +| --- | --- | +| M1 | Renderer hook lists remote tracks (`voice.tracks` action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping. | +| M2 | One track taps to an `AudioWorklet`, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only. | +| M3 | Map track to user ID (`AudioContext` hook, or the `VOICE_SPEAKING` fallback). Per-user VAD and transcript rows with a real `speakerId`. | +| M4 | The `split user audio` setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback. | + +## Setting + +`preferences.audio.splitUserAudio` (boolean, default false), shown in Settings → Browser audio as +**split user audio**. When it is off, the mixed capture feeds STT as it does today. When it is on, +the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for +the browser monitor. + +## Failure modes to handle + +- Discord updates can break the hooks. Gate the feature on a capability probe (`voice.tracks` + answers with a list) and fall back to the mixed stream. +- Tracks appear and disappear during a call. Handle `track-added` and `track-ended` events. +- Muted or deafened users can still produce a track with silence. The VAD must drop it. +- The audio thread must never block. Copy in the worklet, send outside it. +- The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as + mixed. diff --git a/docs/webui.md b/docs/webui.md index 800dfa8..4057f5d 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -294,6 +294,8 @@ replaceable. | WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call | | POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices | | POST | `/api/audio/speak` | `text`, `provider?`, `voice?`, `speed?`, `pitch?` | Queue text to speech | +| POST | `/api/audio/preview` | the speak fields | Synthesize and return a `url` for browser playback. Never touches the call | +| GET | `/api/audio/tts/:name` | none | Serve a preview WAV (`tts--.wav` only) | | POST | `/api/audio/stop` | none | Stop speech and clear the queue | | POST | `/api/audio/capture` | `start` | Start or stop capture | | GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments | diff --git a/poc/public/app.js b/poc/public/app.js index 2acb678..96b4e1b 100644 --- a/poc/public/app.js +++ b/poc/public/app.js @@ -2029,9 +2029,11 @@ async function warmEngine() { function fillSpeechSliders(tts) { const speed = typeof tts?.speed === "number" ? tts.speed : 1; const pitch = typeof tts?.pitch === "number" ? tts.pitch : 0; + const volume = typeof tts?.volume === "number" ? tts.volume : 100; $("#tts-speed").value = Number.isFinite(speed) ? speed : 1; $("#tts-pitch").value = Number.isFinite(pitch) ? pitch : 0; + $("#tts-volume").value = Number.isFinite(volume) ? volume : 100; syncSpeechSliderLabels(); } @@ -2042,6 +2044,7 @@ function syncSpeechSliderLabels() { $("#tts-speed-value").textContent = `${speed.toFixed(2)}x`; $("#tts-pitch-value").textContent = pitchSlider.disabled ? "n/a" : `${pitch > 0 ? "+" : ""}${pitch} st`; + $("#tts-volume-value").textContent = `${Number($("#tts-volume").value)}%`; const expressionSlider = $("#tts-expression"); if (expressionSlider) { @@ -2177,6 +2180,7 @@ function effectOptions() { profile: profileFromControls(), highpass: Number($("#tts-highpass").value), echo: Number($("#tts-echo").value), + volume: Number($("#tts-volume").value), words: $("#tts-words").checked, wordGapMs: Number($("#tts-word-gap").value), // VOX and the Half-Life profile are part of the `vox`/`hev` voices now. @@ -2463,6 +2467,40 @@ async function speakText() { } } +async function previewSpeech() { + const text = $("#tts-input").value.trim(); + if (!text) return; + + const button = $("#tts-preview"); + button.disabled = true; + + try { + const provider = $("#tts-provider").value; + const speed = Number($("#tts-speed").value); + const pitch = Number($("#tts-pitch").value); + const expressionEl = $("#tts-expression"); + const expression = + expressionEl.dataset.dirty === "1" ? Number(expressionEl.value) : state.speech.expression ?? undefined; + + const result = await VK.api.post("/api/audio/preview", { + text, + provider, + speed, + pitch, + ...(typeof expression === "number" ? { expression } : {}), + ...effectOptions() + }); + + const audio = new Audio(result.url); + audio.volume = Math.min(1, Math.max(0, Number($("#tts-volume").value) / 100)); + await audio.play(); + } catch (error) { + toast(error.message, "error"); + } finally { + button.disabled = false; + } +} + /* Transcript */ function transcriptLine(segment) { @@ -2880,11 +2918,12 @@ function init() { }); $("#tts-expression").addEventListener("change", () => void saveSpeech()); buildProfileSliders(); - for (const id of ["tts-highpass", "tts-echo", "tts-word-gap"]) { + for (const id of ["tts-highpass", "tts-echo", "tts-word-gap", "tts-volume"]) { $(`#${id}`).addEventListener("input", () => { $("#tts-highpass-value").textContent = `${$("#tts-highpass").value}Hz`; $("#tts-echo-value").textContent = Number($("#tts-echo").value) > 0 ? $("#tts-echo").value : "off"; $("#tts-word-gap-value").textContent = `${$("#tts-word-gap").value}ms`; + $("#tts-volume-value").textContent = `${Number($("#tts-volume").value)}%`; }); $(`#${id}`).addEventListener("change", () => void saveSpeech()); } @@ -3065,6 +3104,7 @@ function init() { $("#tts-close").addEventListener("click", () => { $("#tts-panel").hidden = true; }); + $("#tts-preview").addEventListener("click", () => void previewSpeech()); $("#tts-speak").addEventListener("click", () => void speakText()); $("#tts-stop").addEventListener("click", async () => { try { diff --git a/poc/public/index.html b/poc/public/index.html index 73690b6..b256f1e 100644 --- a/poc/public/index.html +++ b/poc/public/index.html @@ -170,6 +170,11 @@ 0 st + @@ -179,6 +184,7 @@
idle +
diff --git a/poc/src/audio-service.ts b/poc/src/audio-service.ts index 9e9f4fe..3d8414d 100644 --- a/poc/src/audio-service.ts +++ b/poc/src/audio-service.ts @@ -156,7 +156,7 @@ export class AudioService { * grows. The wrapper touches `.ready` after the first word, which is the reliable * signal; a size threshold and a timeout cover providers that do not mark readiness. */ - private async playWhenReady(file: string): Promise { + private async playWhenReady(file: string, volume?: number): Promise { for (let attempt = 0; attempt < 400; attempt++) { try { if (statSync(`${file}.ready`).size >= 0) break; @@ -172,7 +172,7 @@ export class AudioService { } try { - await this.manager.play(file); + await this.manager.play(file, volume); } finally { try { unlinkSync(`${file}.ready`); @@ -182,6 +182,23 @@ export class AudioService { } } + /** The TTS output directory, for serving preview files. */ + get ttsDirectory(): string { + return this.tts.directory; + } + + /** + * Synthesize a line for browser playback. This never touches the call: + * no queue, no ducking, no virtual microphone, no voice-processing bypass. + */ + async preview(text: string, options: TtsOptions = {}): Promise { + const trimmed = text.trim(); + if (!trimmed) throw new Error("text is required"); + if (trimmed.length > 1000) throw new Error("text is too long (max 1000 characters)"); + + return this.tts.synthesize(trimmed, { ...options, stream: false, onOutputPath: undefined }); + } + stop(): void { this.queue.length = 0; this.manager.stopPlayback(); @@ -402,7 +419,7 @@ export class AudioService { options.onOutputPath = path => { streaming = (async () => { await this.applyDuck(this.shouldDuck()); - await this.playWhenReady(path); + await this.playWhenReady(path, item.options.volume); })(); }; } @@ -413,7 +430,7 @@ export class AudioService { await pending; } else { await this.applyDuck(this.shouldDuck()); - await this.manager.play(file); + await this.manager.play(file, item.options.volume); } } catch (error) { console.error("[audio] speak failed:", (error as Error).message); diff --git a/poc/src/audio.ts b/poc/src/audio.ts index 8f69e76..1191520 100644 --- a/poc/src/audio.ts +++ b/poc/src/audio.ts @@ -276,7 +276,7 @@ export class AudioManager { this.modules = []; } - async play(file: string): Promise { + async play(file: string, volume?: number): Promise { if (!this.available) throw new AudioUnavailableError(this.error ?? "audio is not available"); const binary = process.env.VESKSLOP_FFPLAY ?? "ffplay"; @@ -284,7 +284,17 @@ export class AudioManager { // Lead-in silence: give Discord's voice pipeline a moment to settle // before the first syllable, so the attack is not swallowed. const args = ["-nodisp", "-autoexit", "-loglevel", "error"]; - if (LEAD_MS > 0) args.push("-af", `adelay=${LEAD_MS}:all=1`); + const filters: string[] = []; + if (LEAD_MS > 0) filters.push(`adelay=${LEAD_MS}:all=1`); + + // Volume percent, 0-200. 100 keeps the level as synthesized. ffplay's own + // -volume stops at 100, so use the volume filter and its dB scale. + if (typeof volume === "number" && Number.isFinite(volume) && volume !== 100) { + const gain = volume > 0 ? `${(20 * Math.log10(volume / 100)).toFixed(2)}dB` : "0"; + filters.push(`volume=${gain}`); + } + + if (filters.length) args.push("-af", filters.join(",")); args.push(file); await new Promise((resolve, reject) => { diff --git a/poc/src/preferences.ts b/poc/src/preferences.ts index 0676dd3..a124524 100644 --- a/poc/src/preferences.ts +++ b/poc/src/preferences.ts @@ -14,6 +14,8 @@ export interface TtsPreferences { provider: "local" | "espeak" | "command" | "api" | null; language: string | null; voice: string | null; + /** Playback volume percent, 0-200. Null uses the natural level. */ + volume: number | null; /** Speech-rate multiplier, 1.0 is the engine's natural rate. Null uses the engine default. */ speed: number | null; /** Pitch offset in semitones, 0 is the engine's natural pitch. Null uses the engine default. */ @@ -176,6 +178,7 @@ const DEFAULTS: WebPreferences = { provider: null, language: null, voice: null, + volume: null, speed: null, pitch: null, expression: null, @@ -416,6 +419,7 @@ export class PreferencesStore { provider: normalizeProvider(patch.tts?.provider), language: patch.tts?.language ? String(patch.tts.language).slice(0, 16) : null, voice: patch.tts?.voice ? String(patch.tts.voice).slice(0, 64) : null, + volume: normalizeRange(patch.tts?.volume, 0, 200), speed: normalizeSpeed(patch.tts?.speed), pitch: normalizePitch(patch.tts?.pitch), expression: normalizeExpression(patch.tts?.expression), @@ -500,6 +504,7 @@ export class PreferencesStore { provider: normalizeProvider(provider), language: raw?.tts?.language ? String(raw.tts.language).slice(0, 16) : null, voice: raw?.tts?.voice ? String(raw.tts.voice).slice(0, 64) : null, + volume: normalizeRange(raw?.tts?.volume, 0, 200), speed: normalizeSpeed(raw?.tts?.speed), pitch: normalizePitch(raw?.tts?.pitch), expression: normalizeExpression(raw?.tts?.expression), diff --git a/poc/src/tts.ts b/poc/src/tts.ts index 0bc317a..a6899d7 100644 --- a/poc/src/tts.ts +++ b/poc/src/tts.ts @@ -18,6 +18,8 @@ export type TtsProvider = "local" | "espeak" | "command" | "api"; export interface TtsOptions { provider?: TtsProvider; voice?: string; + /** Playback volume percent, 0-200. The daemon applies it at play time. */ + volume?: number; /** Speech-rate multiplier; the engine maps it to its own units. */ speed?: number; /** Pitch offset in semitones; the engine maps it to its own units. */ diff --git a/poc/src/webui.ts b/poc/src/webui.ts index 51436d9..61d45dd 100644 --- a/poc/src/webui.ts +++ b/poc/src/webui.ts @@ -14,7 +14,7 @@ import { existsSync, readFileSync } from "node:fs"; import { Readable } from "node:stream"; import http from "node:http"; import https from "node:https"; -import { dirname } from "node:path"; +import { basename, dirname, join } from "node:path"; import { fileURLToPath } from "node:url"; import QRCode from "qrcode"; import { WebSocketServer } from "ws"; @@ -25,11 +25,11 @@ import { CoreError, type CoreClient } from "./core-client.js"; import type { CoreProcess } from "./core-process.js"; import { MediaLimitError, type MediaService } from "./media.js"; import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js"; -import type { PreferencesStore, WebPreferences } from "./preferences.js"; +import type { PreferencesStore, TtsPreferences, WebPreferences } from "./preferences.js"; import type { Puppet } from "./puppet.js"; import { ensureTls } from "./tls.js"; import type { TranscriptStore } from "./transcript.js"; -import type { TtsProvider, TtsVoice } from "./tts.js"; +import type { TtsOptions, TtsProvider, TtsVoice } from "./tts.js"; const PINNED_LANGUAGES = ["en", "sv", "de", "da", "nb", "no", "ru", "chr"]; const LANGUAGE_NAMES: Record = { @@ -231,6 +231,77 @@ function parseMediaFilters(query: Record) { }; } + +/** + * Turn a speak/preview request body into TTS options. Missing fields fall back + * to the stored preferences. + */ +function buildSpeechOptions(body: any, tts: TtsPreferences, voice: string | undefined): TtsOptions { + const requested = body?.provider; + const provider = ["api", "local", "espeak", "command"].includes(requested) + ? requested + : tts.provider; + // Number(null) is 0: null, undefined, and "" all mean "unset". + const optional = (value: unknown): number | undefined => { + if (value === undefined || value === null || value === "") return undefined; + const number = Number(value); + return Number.isFinite(number) ? number : undefined; + }; + + const speed = optional(body?.speed) ?? tts.speed ?? undefined; + const pitch = optional(body?.pitch) ?? tts.pitch ?? undefined; + const expression = optional(body?.expression) ?? tts.expression ?? undefined; + const profile = body?.profile && typeof body.profile === "object" ? body.profile : tts.profile; + const highpass = optional(body?.highpass) ?? tts.highpass ?? undefined; + const echo = optional(body?.echo) ?? tts.echo ?? undefined; + const words = body?.words !== undefined ? body.words === true : tts.words; + const wordGapMs = optional(body?.wordGapMs) ?? tts.wordGapMs ?? undefined; + // The Half-Life VOX voice carries its prosody with it. + const vox = + (body?.vox !== undefined ? body.vox === true : tts.vox) || voice === "vox"; + const voxClean = body?.voxClean !== undefined ? body.voxClean === true : tts.voxClean; + const preset = + body?.preset === "vox" || body?.preset === "hev" + ? body.preset + : body?.preset === null || body?.preset === "" + ? null + : tts.preset; + const echoDelayMs = optional(body?.echoDelayMs) ?? tts.echoDelayMs ?? undefined; + const echoLevel = optional(body?.echoLevel) ?? tts.echoLevel ?? undefined; + const crunch = + ["hl90", "hl90hq", "telephone", "adpcm", "crush"].includes(String(body?.crunch ?? "")) + ? (String(body.crunch) as "hl90" | "hl90hq" | "telephone" | "adpcm" | "crush") + : body?.crunch === null || body?.crunch === "" || body?.crunch === "off" + ? undefined + : tts.crunch ?? undefined; + const crunchBits = optional(body?.crunchBits) ?? tts.crunchBits ?? undefined; + const crunchRate = optional(body?.crunchRate) ?? tts.crunchRate ?? undefined; + const streamWords = + body?.streamWords !== undefined ? body.streamWords !== false : tts.streamWords; + + return { + provider: provider ?? undefined, + voice, + ...(Number.isFinite(speed) ? { speed } : {}), + ...(Number.isFinite(pitch) ? { pitch } : {}), + ...(Number.isFinite(expression) ? { expression } : {}), + ...(profile && Object.keys(profile).length ? { profile } : {}), + ...(Number.isFinite(highpass) ? { highpass } : {}), + ...(Number.isFinite(echo) ? { echo } : {}), + ...(words ? { words: true } : {}), + ...(Number.isFinite(wordGapMs) ? { wordGapMs } : {}), + ...(vox ? { vox: true } : {}), + ...(voxClean ? { voxClean: true } : {}), + ...(preset ? { preset } : {}), + ...(Number.isFinite(echoDelayMs) ? { echoDelayMs } : {}), + ...(Number.isFinite(echoLevel) ? { echoLevel } : {}), + ...(crunch ? { crunch } : {}), + ...(Number.isFinite(crunchBits) ? { crunchBits } : {}), + ...(Number.isFinite(crunchRate) ? { crunchRate } : {}), + stream: streamWords && words === true + } as TtsOptions; +} + export function startWebUi(deps: WebUiDeps): WebUiHandle { const app = express(); app.use(express.json({ limit: "64kb" })); @@ -508,6 +579,7 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { provider: tts?.provider ?? null, language: tts?.language ?? null, voice: tts?.voice ?? null, + volume: tts?.volume ?? null, speed: tts?.speed ?? null, pitch: tts?.pitch ?? null, expression: tts?.expression ?? null, @@ -794,80 +866,46 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { const text = String(req.body?.text ?? ""); const prefs = deps.preferences.get(); - const requested = req.body?.provider; - const provider = ["api", "local", "espeak", "command"].includes(requested) - ? requested - : prefs.tts.provider; const voice = req.body?.voice ? String(req.body.voice) : prefs.tts.voice ?? undefined; - // Number(null) is 0: null, undefined, and "" all mean "unset". - const optional = (value: unknown): number | undefined => { - if (value === undefined || value === null || value === "") return undefined; - const number = Number(value); - return Number.isFinite(number) ? number : undefined; - }; + const options = buildSpeechOptions(req.body ?? {}, prefs.tts, voice); - const speed = optional(req.body?.speed) ?? prefs.tts.speed ?? undefined; - const pitch = optional(req.body?.pitch) ?? prefs.tts.pitch ?? undefined; - const expression = optional(req.body?.expression) ?? prefs.tts.expression ?? undefined; - const profile = req.body?.profile && typeof req.body.profile === "object" ? req.body.profile : prefs.tts.profile; - const highpass = optional(req.body?.highpass) ?? prefs.tts.highpass ?? undefined; - const echo = optional(req.body?.echo) ?? prefs.tts.echo ?? undefined; - const words = req.body?.words !== undefined ? req.body.words === true : prefs.tts.words; - const wordGapMs = optional(req.body?.wordGapMs) ?? prefs.tts.wordGapMs ?? undefined; - // The Half-Life VOX voice carries its prosody with it. - const vox = - (req.body?.vox !== undefined ? req.body.vox === true : prefs.tts.vox) || voice === "vox"; - const voxClean = req.body?.voxClean !== undefined ? req.body.voxClean === true : prefs.tts.voxClean; - const preset = - req.body?.preset === "vox" || req.body?.preset === "hev" - ? req.body.preset - : req.body?.preset === null || req.body?.preset === "" - ? null - : prefs.tts.preset; - const echoDelayMs = optional(req.body?.echoDelayMs) ?? prefs.tts.echoDelayMs ?? undefined; - const echoLevel = optional(req.body?.echoLevel) ?? prefs.tts.echoLevel ?? undefined; - const crunch = - ["hl90", "hl90hq", "telephone", "adpcm", "crush"].includes(String(req.body?.crunch ?? "")) - ? (String(req.body.crunch) as "hl90" | "hl90hq" | "telephone" | "adpcm" | "crush") - : req.body?.crunch === null || req.body?.crunch === "" || req.body?.crunch === "off" - ? undefined - : prefs.tts.crunch ?? undefined; - const crunchBits = optional(req.body?.crunchBits) ?? prefs.tts.crunchBits ?? undefined; - const crunchRate = optional(req.body?.crunchRate) ?? prefs.tts.crunchRate ?? undefined; - const streamWords = - req.body?.streamWords !== undefined ? req.body.streamWords !== false : prefs.tts.streamWords; - - const result = await deps.audio.speak(text, { - provider: provider ?? undefined, - voice, - ...(Number.isFinite(speed) ? { speed } : {}), - ...(Number.isFinite(pitch) ? { pitch } : {}), - ...(Number.isFinite(expression) ? { expression } : {}), - ...(profile && Object.keys(profile).length ? { profile } : {}), - ...(Number.isFinite(highpass) ? { highpass } : {}), - ...(Number.isFinite(echo) ? { echo } : {}), - ...(words ? { words: true } : {}), - ...(Number.isFinite(wordGapMs) ? { wordGapMs } : {}), - ...(vox ? { vox: true } : {}), - ...(voxClean ? { voxClean: true } : {}), - ...(preset ? { preset } : {}), - ...(Number.isFinite(echoDelayMs) ? { echoDelayMs } : {}), - ...(Number.isFinite(echoLevel) ? { echoLevel } : {}), - ...(crunch ? { crunch } : {}), - ...(Number.isFinite(crunchBits) ? { crunchBits } : {}), - ...(Number.isFinite(crunchRate) ? { crunchRate } : {}), - stream: streamWords && words === true - }); + const result = await deps.audio.speak(text, options); deps.getPuppet()?.appendAction({ type: "audio", action: "speak", - provider: provider ?? deps.audio.status().tts.defaultProvider, + provider: options.provider ?? deps.audio.status().tts.defaultProvider, chars: text.trim().length }); res.json({ ok: true, result }); }) ); + app.post( + "/api/audio/preview", + wrap(async (req, res) => { + const text = String(req.body?.text ?? "").trim(); + const prefs = deps.preferences.get(); + const voice = req.body?.voice ? String(req.body.voice) : prefs.tts.voice ?? undefined; + const file = await deps.audio.preview(text, buildSpeechOptions(req.body ?? {}, prefs.tts, voice)); + + res.json({ ok: true, result: { url: `/api/audio/tts/${encodeURIComponent(basename(file))}` } }); + }) + ); + + // Preview files only: the name pattern blocks path traversal. + app.get( + "/api/audio/tts/:name", + wrap(async (req, res) => { + const name = String(req.params.name); + if (!/^tts-[0-9]+-[0-9]+\.wav$/.test(name)) throw new HttpError(400, "invalid audio name"); + + const file = join(deps.audio.ttsDirectory, name); + if (!existsSync(file)) throw new HttpError(404, "audio not found"); + + res.type("audio/wav").sendFile(file); + }) + ); + app.post( "/api/audio/stop", wrap(async (_req, res) => {