Add TTS volume, a browser preview, and the split-audio plan
This commit is contained in:
+14
@@ -40,6 +40,20 @@ mobile draft at 390px.
|
|||||||
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
||||||
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
||||||
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
||||||
|
- [x] C77 Split user audio investigation. [docs/voice-audio-split.md](docs/voice-audio-split.md)
|
||||||
|
records that the daemon sees only the mixed client output (`VeskslopCapture.monitor` → VAD →
|
||||||
|
STT) and that per-user tracks exist only in the renderer as WebRTC `MediaStreamTrack`s. The
|
||||||
|
core bridge has no WebRTC hooks today. The plan: hook `RTCPeerConnection` for tracks, hook
|
||||||
|
`AudioContext` for track-to-user mapping, stream 16 kHz PCM frames to the daemon, add
|
||||||
|
`preferences.audio.splitUserAudio` (default off), and keep the mixed path as the fallback. Not
|
||||||
|
implemented.
|
||||||
|
- [x] C76 TTS volume and browser preview. `preferences.tts.volume` (0-200, null = natural)
|
||||||
|
rides the speak options; the daemon applies it at play time with an ffmpeg `volume` filter
|
||||||
|
(ffplay's own `-volume` stops at 100). The TTS panel gained a volume slider and a **Preview**
|
||||||
|
button that synthesizes through `POST /api/audio/preview` and plays the returned
|
||||||
|
`/api/audio/tts/<name>` WAV in the browser. Preview never queues, ducks, or touches the
|
||||||
|
virtual mic. Verified live: a 1.6 s WAV served as `audio/wav`, and the volume preference
|
||||||
|
round-trips.
|
||||||
- [x] C75 Wakeword routing and the reply-only agent path. Commands now need the wakeword plus a
|
- [x] C75 Wakeword routing and the reply-only agent path. Commands now need the wakeword plus a
|
||||||
space at the start of the message: default `!s`, so `!s whitelist list`. `!space`,
|
space at the start of the message: default `!s`, so `!s whitelist list`. `!space`,
|
||||||
`hey!s cmd`, and bare commands no longer trigger. Settings → Commands picks a preset (`!s !x
|
`hey!s cmd`, and bare commands no longer trigger. Settings → Commands picks a preset (`!s !x
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ This file lists the project documents.
|
|||||||
| [daemon.md](daemon.md) | The external MCP daemon |
|
| [daemon.md](daemon.md) | The external MCP daemon |
|
||||||
| [webui.md](webui.md) | The web control UI and its JSON API |
|
| [webui.md](webui.md) | The web control UI and its JSON API |
|
||||||
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
|
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
|
||||||
|
| [voice-audio-split.md](voice-audio-split.md) | Per-user audio investigation and the split-user STT plan |
|
||||||
| [media.md](media.md) | Media archive: command, storage, limits, and Web UI |
|
| [media.md](media.md) | Media archive: command, storage, limits, and Web UI |
|
||||||
| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema |
|
| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema |
|
||||||
| [media-mcp.md](media-mcp.md) | The media MCP tool surface |
|
| [media-mcp.md](media-mcp.md) | The media MCP tool surface |
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
# Split user audio — investigation and plan
|
||||||
|
|
||||||
|
Status: investigated 2026-09-22. Not implemented. Until this lands, STT runs on the full mixed
|
||||||
|
stream.
|
||||||
|
|
||||||
|
## What happens today
|
||||||
|
|
||||||
|
The daemon does not see per-user audio. The client mixes every remote user before the audio leaves
|
||||||
|
the browser.
|
||||||
|
|
||||||
|
| Step | Where | Detail |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Playback | Vesktop renderer | Discord mixes the remote tracks. Per-user volume is applied before the mix. |
|
||||||
|
| Routing | `poc/src/audio.ts` | `pactl move-sink-input` moves the client's playback streams into the `VeskslopCapture` sink. |
|
||||||
|
| Capture | `poc/src/capture.ts` | `ffmpeg` records `VeskslopCapture.monitor` as raw PCM. One stream, all users. |
|
||||||
|
| Split | `poc/src/vad.ts` | The VAD cuts the stream into utterances and writes one WAV each. |
|
||||||
|
| Text | `poc/src/stt.ts`, `poc/src/transcript.ts` | STT turns each WAV into text when a provider is configured. |
|
||||||
|
| Speaker | `poc/src/audio-service.ts` | `VOICE_SPEAKING` events give a best-effort speaker per segment. Overlaps and short segments can name the wrong user. |
|
||||||
|
|
||||||
|
The core bridge (`src/renderer/puppet/actions.ts`) has no WebRTC hooks today. The only audio call
|
||||||
|
is `navigator.mediaDevices.enumerateDevices()` for the device list.
|
||||||
|
|
||||||
|
## Where per-user audio exists
|
||||||
|
|
||||||
|
Each remote user arrives in the renderer as a WebRTC `MediaStreamTrack`. Discord attaches the
|
||||||
|
tracks to an `RTCPeerConnection` and mixes them in an `AudioContext` graph. Per-user gain (the
|
||||||
|
volume sliders) is a node on each track's path. The OS output receives only the mix.
|
||||||
|
|
||||||
|
## Split options
|
||||||
|
|
||||||
|
| Option | How | Cost |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Hook `RTCPeerConnection` | Wrap `ontrack` and `addEventListener("track")` before Discord's code runs, then keep every remote track | Straightforward to hook, but the track carries no user ID |
|
||||||
|
| Hook `AudioContext` | Wrap `createMediaStreamSource` and `createGain`. Discord makes one source per user, so the graph shows the per-user split | Better mapping chance, but fragile against client updates |
|
||||||
|
| Map by event order | Pair the first chunk after a `VOICE_SPEAKING` event with that user | Works with heavy traffic, mislabels overlaps |
|
||||||
|
| Chromium capture | Use Electron audio capture of `webContents` | Gives the mix again, not per-user |
|
||||||
|
|
||||||
|
Recommended path: hook `RTCPeerConnection` for the tracks, and hook `AudioContext` only to map a
|
||||||
|
track to a user (the source node creation order matches the track order, and the per-user gain node
|
||||||
|
sits next to it). Keep the event-order pairing as the fallback for unknown tracks.
|
||||||
|
|
||||||
|
## Transport for per-user PCM
|
||||||
|
|
||||||
|
The daemon needs one PCM stream per user. Two candidate shapes:
|
||||||
|
|
||||||
|
1. **AudioWorklet, 16 kHz mono s16le, base64 frames over the bridge.** A 20 ms frame is 640 bytes,
|
||||||
|
about 1.3 KB in base64. Five users cost about 320 KB per minute. Low latency, more bridge
|
||||||
|
traffic, small renderer cost.
|
||||||
|
2. **MediaRecorder per track, Opus chunks every second.** Lower traffic and less renderer CPU.
|
||||||
|
The daemon decodes each chunk with ffmpeg before the VAD. Higher latency, more daemon CPU.
|
||||||
|
|
||||||
|
Recommendation: option 1. The bridge is a WebSocket with JSON frames already, and the daemon
|
||||||
|
already runs ffmpeg for resampling. Add a binary frame type later when the volume grows.
|
||||||
|
|
||||||
|
## Plan
|
||||||
|
|
||||||
|
| Step | Work |
|
||||||
|
| --- | --- |
|
||||||
|
| M1 | Renderer hook lists remote tracks (`voice.tracks` action: track ID, stream ID, label). No audio yet. This proves the hook and the mapping. |
|
||||||
|
| M2 | One track taps to an `AudioWorklet`, streams 16 kHz mono frames to the daemon, and the daemon writes a WAV. Loopback test only. |
|
||||||
|
| M3 | Map track to user ID (`AudioContext` hook, or the `VOICE_SPEAKING` fallback). Per-user VAD and transcript rows with a real `speakerId`. |
|
||||||
|
| M4 | The `split user audio` setting, per-user stop/start on join and leave, mute and deafen handling, and the mixed stream as the fallback. |
|
||||||
|
|
||||||
|
## Setting
|
||||||
|
|
||||||
|
`preferences.audio.splitUserAudio` (boolean, default false), shown in Settings → Browser audio as
|
||||||
|
**split user audio**. When it is off, the mixed capture feeds STT as it does today. When it is on,
|
||||||
|
the daemon asks the core for the tracks, STT runs per user, and the mixed capture stays only for
|
||||||
|
the browser monitor.
|
||||||
|
|
||||||
|
## Failure modes to handle
|
||||||
|
|
||||||
|
- Discord updates can break the hooks. Gate the feature on a capability probe (`voice.tracks`
|
||||||
|
answers with a list) and fall back to the mixed stream.
|
||||||
|
- Tracks appear and disappear during a call. Handle `track-added` and `track-ended` events.
|
||||||
|
- Muted or deafened users can still produce a track with silence. The VAD must drop it.
|
||||||
|
- The audio thread must never block. Copy in the worklet, send outside it.
|
||||||
|
- The feature needs a CPU budget per user. Cap the number of split tracks and report the rest as
|
||||||
|
mixed.
|
||||||
@@ -294,6 +294,8 @@ replaceable.
|
|||||||
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
|
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
|
||||||
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
|
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
|
||||||
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?`, `speed?`, `pitch?` | Queue text to speech |
|
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?`, `speed?`, `pitch?` | Queue text to speech |
|
||||||
|
| POST | `/api/audio/preview` | the speak fields | Synthesize and return a `url` for browser playback. Never touches the call |
|
||||||
|
| GET | `/api/audio/tts/:name` | none | Serve a preview WAV (`tts-<n>-<n>.wav` only) |
|
||||||
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
||||||
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
||||||
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
||||||
|
|||||||
+41
-1
@@ -2029,9 +2029,11 @@ async function warmEngine() {
|
|||||||
function fillSpeechSliders(tts) {
|
function fillSpeechSliders(tts) {
|
||||||
const speed = typeof tts?.speed === "number" ? tts.speed : 1;
|
const speed = typeof tts?.speed === "number" ? tts.speed : 1;
|
||||||
const pitch = typeof tts?.pitch === "number" ? tts.pitch : 0;
|
const pitch = typeof tts?.pitch === "number" ? tts.pitch : 0;
|
||||||
|
const volume = typeof tts?.volume === "number" ? tts.volume : 100;
|
||||||
|
|
||||||
$("#tts-speed").value = Number.isFinite(speed) ? speed : 1;
|
$("#tts-speed").value = Number.isFinite(speed) ? speed : 1;
|
||||||
$("#tts-pitch").value = Number.isFinite(pitch) ? pitch : 0;
|
$("#tts-pitch").value = Number.isFinite(pitch) ? pitch : 0;
|
||||||
|
$("#tts-volume").value = Number.isFinite(volume) ? volume : 100;
|
||||||
syncSpeechSliderLabels();
|
syncSpeechSliderLabels();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2042,6 +2044,7 @@ function syncSpeechSliderLabels() {
|
|||||||
|
|
||||||
$("#tts-speed-value").textContent = `${speed.toFixed(2)}x`;
|
$("#tts-speed-value").textContent = `${speed.toFixed(2)}x`;
|
||||||
$("#tts-pitch-value").textContent = pitchSlider.disabled ? "n/a" : `${pitch > 0 ? "+" : ""}${pitch} st`;
|
$("#tts-pitch-value").textContent = pitchSlider.disabled ? "n/a" : `${pitch > 0 ? "+" : ""}${pitch} st`;
|
||||||
|
$("#tts-volume-value").textContent = `${Number($("#tts-volume").value)}%`;
|
||||||
|
|
||||||
const expressionSlider = $("#tts-expression");
|
const expressionSlider = $("#tts-expression");
|
||||||
if (expressionSlider) {
|
if (expressionSlider) {
|
||||||
@@ -2177,6 +2180,7 @@ function effectOptions() {
|
|||||||
profile: profileFromControls(),
|
profile: profileFromControls(),
|
||||||
highpass: Number($("#tts-highpass").value),
|
highpass: Number($("#tts-highpass").value),
|
||||||
echo: Number($("#tts-echo").value),
|
echo: Number($("#tts-echo").value),
|
||||||
|
volume: Number($("#tts-volume").value),
|
||||||
words: $("#tts-words").checked,
|
words: $("#tts-words").checked,
|
||||||
wordGapMs: Number($("#tts-word-gap").value),
|
wordGapMs: Number($("#tts-word-gap").value),
|
||||||
// VOX and the Half-Life profile are part of the `vox`/`hev` voices now.
|
// VOX and the Half-Life profile are part of the `vox`/`hev` voices now.
|
||||||
@@ -2463,6 +2467,40 @@ async function speakText() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function previewSpeech() {
|
||||||
|
const text = $("#tts-input").value.trim();
|
||||||
|
if (!text) return;
|
||||||
|
|
||||||
|
const button = $("#tts-preview");
|
||||||
|
button.disabled = true;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const provider = $("#tts-provider").value;
|
||||||
|
const speed = Number($("#tts-speed").value);
|
||||||
|
const pitch = Number($("#tts-pitch").value);
|
||||||
|
const expressionEl = $("#tts-expression");
|
||||||
|
const expression =
|
||||||
|
expressionEl.dataset.dirty === "1" ? Number(expressionEl.value) : state.speech.expression ?? undefined;
|
||||||
|
|
||||||
|
const result = await VK.api.post("/api/audio/preview", {
|
||||||
|
text,
|
||||||
|
provider,
|
||||||
|
speed,
|
||||||
|
pitch,
|
||||||
|
...(typeof expression === "number" ? { expression } : {}),
|
||||||
|
...effectOptions()
|
||||||
|
});
|
||||||
|
|
||||||
|
const audio = new Audio(result.url);
|
||||||
|
audio.volume = Math.min(1, Math.max(0, Number($("#tts-volume").value) / 100));
|
||||||
|
await audio.play();
|
||||||
|
} catch (error) {
|
||||||
|
toast(error.message, "error");
|
||||||
|
} finally {
|
||||||
|
button.disabled = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/* Transcript */
|
/* Transcript */
|
||||||
|
|
||||||
function transcriptLine(segment) {
|
function transcriptLine(segment) {
|
||||||
@@ -2880,11 +2918,12 @@ function init() {
|
|||||||
});
|
});
|
||||||
$("#tts-expression").addEventListener("change", () => void saveSpeech());
|
$("#tts-expression").addEventListener("change", () => void saveSpeech());
|
||||||
buildProfileSliders();
|
buildProfileSliders();
|
||||||
for (const id of ["tts-highpass", "tts-echo", "tts-word-gap"]) {
|
for (const id of ["tts-highpass", "tts-echo", "tts-word-gap", "tts-volume"]) {
|
||||||
$(`#${id}`).addEventListener("input", () => {
|
$(`#${id}`).addEventListener("input", () => {
|
||||||
$("#tts-highpass-value").textContent = `${$("#tts-highpass").value}Hz`;
|
$("#tts-highpass-value").textContent = `${$("#tts-highpass").value}Hz`;
|
||||||
$("#tts-echo-value").textContent = Number($("#tts-echo").value) > 0 ? $("#tts-echo").value : "off";
|
$("#tts-echo-value").textContent = Number($("#tts-echo").value) > 0 ? $("#tts-echo").value : "off";
|
||||||
$("#tts-word-gap-value").textContent = `${$("#tts-word-gap").value}ms`;
|
$("#tts-word-gap-value").textContent = `${$("#tts-word-gap").value}ms`;
|
||||||
|
$("#tts-volume-value").textContent = `${Number($("#tts-volume").value)}%`;
|
||||||
});
|
});
|
||||||
$(`#${id}`).addEventListener("change", () => void saveSpeech());
|
$(`#${id}`).addEventListener("change", () => void saveSpeech());
|
||||||
}
|
}
|
||||||
@@ -3065,6 +3104,7 @@ function init() {
|
|||||||
$("#tts-close").addEventListener("click", () => {
|
$("#tts-close").addEventListener("click", () => {
|
||||||
$("#tts-panel").hidden = true;
|
$("#tts-panel").hidden = true;
|
||||||
});
|
});
|
||||||
|
$("#tts-preview").addEventListener("click", () => void previewSpeech());
|
||||||
$("#tts-speak").addEventListener("click", () => void speakText());
|
$("#tts-speak").addEventListener("click", () => void speakText());
|
||||||
$("#tts-stop").addEventListener("click", async () => {
|
$("#tts-stop").addEventListener("click", async () => {
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -170,6 +170,11 @@
|
|||||||
<input id="tts-pitch" type="range" min="-12" max="12" step="1" value="0" />
|
<input id="tts-pitch" type="range" min="-12" max="12" step="1" value="0" />
|
||||||
<span id="tts-pitch-value" class="vk-mono vk-dim">0 st</span>
|
<span id="tts-pitch-value" class="vk-mono vk-dim">0 st</span>
|
||||||
</label>
|
</label>
|
||||||
|
<label class="vk-tts-slider" title="Playback volume for speech, 100% is the level as synthesized">
|
||||||
|
<span class="vk-micro-label">volume</span>
|
||||||
|
<input id="tts-volume" type="range" min="0" max="200" step="5" value="100" />
|
||||||
|
<span id="tts-volume-value" class="vk-mono vk-dim">100%</span>
|
||||||
|
</label>
|
||||||
<button id="profile-open" class="vk-chip vk-chip-off" type="button" title="Flextalk voice profile">
|
<button id="profile-open" class="vk-chip vk-chip-off" type="button" title="Flextalk voice profile">
|
||||||
voice profile…
|
voice profile…
|
||||||
</button>
|
</button>
|
||||||
@@ -179,6 +184,7 @@
|
|||||||
<div class="vk-tts-meta">
|
<div class="vk-tts-meta">
|
||||||
<span class="vk-mono vk-dim" id="tts-status">idle</span>
|
<span class="vk-mono vk-dim" id="tts-status">idle</span>
|
||||||
<button id="tts-stop" class="btn btn-secondary vk-small-btn" type="button">Stop</button>
|
<button id="tts-stop" class="btn btn-secondary vk-small-btn" type="button">Stop</button>
|
||||||
|
<button id="tts-preview" class="btn btn-secondary vk-small-btn" type="button" title="Play this line in the browser, not in the call">Preview</button>
|
||||||
<button id="tts-speak" class="btn btn-primary vk-small-btn" type="button">Speak</button>
|
<button id="tts-speak" class="btn btn-primary vk-small-btn" type="button">Speak</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -156,7 +156,7 @@ export class AudioService {
|
|||||||
* grows. The wrapper touches `<file>.ready` after the first word, which is the reliable
|
* grows. The wrapper touches `<file>.ready` after the first word, which is the reliable
|
||||||
* signal; a size threshold and a timeout cover providers that do not mark readiness.
|
* signal; a size threshold and a timeout cover providers that do not mark readiness.
|
||||||
*/
|
*/
|
||||||
private async playWhenReady(file: string): Promise<void> {
|
private async playWhenReady(file: string, volume?: number): Promise<void> {
|
||||||
for (let attempt = 0; attempt < 400; attempt++) {
|
for (let attempt = 0; attempt < 400; attempt++) {
|
||||||
try {
|
try {
|
||||||
if (statSync(`${file}.ready`).size >= 0) break;
|
if (statSync(`${file}.ready`).size >= 0) break;
|
||||||
@@ -172,7 +172,7 @@ export class AudioService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
try {
|
try {
|
||||||
await this.manager.play(file);
|
await this.manager.play(file, volume);
|
||||||
} finally {
|
} finally {
|
||||||
try {
|
try {
|
||||||
unlinkSync(`${file}.ready`);
|
unlinkSync(`${file}.ready`);
|
||||||
@@ -182,6 +182,23 @@ export class AudioService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** The TTS output directory, for serving preview files. */
|
||||||
|
get ttsDirectory(): string {
|
||||||
|
return this.tts.directory;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Synthesize a line for browser playback. This never touches the call:
|
||||||
|
* no queue, no ducking, no virtual microphone, no voice-processing bypass.
|
||||||
|
*/
|
||||||
|
async preview(text: string, options: TtsOptions = {}): Promise<string> {
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (!trimmed) throw new Error("text is required");
|
||||||
|
if (trimmed.length > 1000) throw new Error("text is too long (max 1000 characters)");
|
||||||
|
|
||||||
|
return this.tts.synthesize(trimmed, { ...options, stream: false, onOutputPath: undefined });
|
||||||
|
}
|
||||||
|
|
||||||
stop(): void {
|
stop(): void {
|
||||||
this.queue.length = 0;
|
this.queue.length = 0;
|
||||||
this.manager.stopPlayback();
|
this.manager.stopPlayback();
|
||||||
@@ -402,7 +419,7 @@ export class AudioService {
|
|||||||
options.onOutputPath = path => {
|
options.onOutputPath = path => {
|
||||||
streaming = (async () => {
|
streaming = (async () => {
|
||||||
await this.applyDuck(this.shouldDuck());
|
await this.applyDuck(this.shouldDuck());
|
||||||
await this.playWhenReady(path);
|
await this.playWhenReady(path, item.options.volume);
|
||||||
})();
|
})();
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -413,7 +430,7 @@ export class AudioService {
|
|||||||
await pending;
|
await pending;
|
||||||
} else {
|
} else {
|
||||||
await this.applyDuck(this.shouldDuck());
|
await this.applyDuck(this.shouldDuck());
|
||||||
await this.manager.play(file);
|
await this.manager.play(file, item.options.volume);
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
console.error("[audio] speak failed:", (error as Error).message);
|
console.error("[audio] speak failed:", (error as Error).message);
|
||||||
|
|||||||
+12
-2
@@ -276,7 +276,7 @@ export class AudioManager {
|
|||||||
this.modules = [];
|
this.modules = [];
|
||||||
}
|
}
|
||||||
|
|
||||||
async play(file: string): Promise<void> {
|
async play(file: string, volume?: number): Promise<void> {
|
||||||
if (!this.available) throw new AudioUnavailableError(this.error ?? "audio is not available");
|
if (!this.available) throw new AudioUnavailableError(this.error ?? "audio is not available");
|
||||||
|
|
||||||
const binary = process.env.VESKSLOP_FFPLAY ?? "ffplay";
|
const binary = process.env.VESKSLOP_FFPLAY ?? "ffplay";
|
||||||
@@ -284,7 +284,17 @@ export class AudioManager {
|
|||||||
// Lead-in silence: give Discord's voice pipeline a moment to settle
|
// Lead-in silence: give Discord's voice pipeline a moment to settle
|
||||||
// before the first syllable, so the attack is not swallowed.
|
// before the first syllable, so the attack is not swallowed.
|
||||||
const args = ["-nodisp", "-autoexit", "-loglevel", "error"];
|
const args = ["-nodisp", "-autoexit", "-loglevel", "error"];
|
||||||
if (LEAD_MS > 0) args.push("-af", `adelay=${LEAD_MS}:all=1`);
|
const filters: string[] = [];
|
||||||
|
if (LEAD_MS > 0) filters.push(`adelay=${LEAD_MS}:all=1`);
|
||||||
|
|
||||||
|
// Volume percent, 0-200. 100 keeps the level as synthesized. ffplay's own
|
||||||
|
// -volume stops at 100, so use the volume filter and its dB scale.
|
||||||
|
if (typeof volume === "number" && Number.isFinite(volume) && volume !== 100) {
|
||||||
|
const gain = volume > 0 ? `${(20 * Math.log10(volume / 100)).toFixed(2)}dB` : "0";
|
||||||
|
filters.push(`volume=${gain}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (filters.length) args.push("-af", filters.join(","));
|
||||||
args.push(file);
|
args.push(file);
|
||||||
|
|
||||||
await new Promise<void>((resolve, reject) => {
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
|||||||
@@ -14,6 +14,8 @@ export interface TtsPreferences {
|
|||||||
provider: "local" | "espeak" | "command" | "api" | null;
|
provider: "local" | "espeak" | "command" | "api" | null;
|
||||||
language: string | null;
|
language: string | null;
|
||||||
voice: string | null;
|
voice: string | null;
|
||||||
|
/** Playback volume percent, 0-200. Null uses the natural level. */
|
||||||
|
volume: number | null;
|
||||||
/** Speech-rate multiplier, 1.0 is the engine's natural rate. Null uses the engine default. */
|
/** Speech-rate multiplier, 1.0 is the engine's natural rate. Null uses the engine default. */
|
||||||
speed: number | null;
|
speed: number | null;
|
||||||
/** Pitch offset in semitones, 0 is the engine's natural pitch. Null uses the engine default. */
|
/** Pitch offset in semitones, 0 is the engine's natural pitch. Null uses the engine default. */
|
||||||
@@ -176,6 +178,7 @@ const DEFAULTS: WebPreferences = {
|
|||||||
provider: null,
|
provider: null,
|
||||||
language: null,
|
language: null,
|
||||||
voice: null,
|
voice: null,
|
||||||
|
volume: null,
|
||||||
speed: null,
|
speed: null,
|
||||||
pitch: null,
|
pitch: null,
|
||||||
expression: null,
|
expression: null,
|
||||||
@@ -416,6 +419,7 @@ export class PreferencesStore {
|
|||||||
provider: normalizeProvider(patch.tts?.provider),
|
provider: normalizeProvider(patch.tts?.provider),
|
||||||
language: patch.tts?.language ? String(patch.tts.language).slice(0, 16) : null,
|
language: patch.tts?.language ? String(patch.tts.language).slice(0, 16) : null,
|
||||||
voice: patch.tts?.voice ? String(patch.tts.voice).slice(0, 64) : null,
|
voice: patch.tts?.voice ? String(patch.tts.voice).slice(0, 64) : null,
|
||||||
|
volume: normalizeRange(patch.tts?.volume, 0, 200),
|
||||||
speed: normalizeSpeed(patch.tts?.speed),
|
speed: normalizeSpeed(patch.tts?.speed),
|
||||||
pitch: normalizePitch(patch.tts?.pitch),
|
pitch: normalizePitch(patch.tts?.pitch),
|
||||||
expression: normalizeExpression(patch.tts?.expression),
|
expression: normalizeExpression(patch.tts?.expression),
|
||||||
@@ -500,6 +504,7 @@ export class PreferencesStore {
|
|||||||
provider: normalizeProvider(provider),
|
provider: normalizeProvider(provider),
|
||||||
language: raw?.tts?.language ? String(raw.tts.language).slice(0, 16) : null,
|
language: raw?.tts?.language ? String(raw.tts.language).slice(0, 16) : null,
|
||||||
voice: raw?.tts?.voice ? String(raw.tts.voice).slice(0, 64) : null,
|
voice: raw?.tts?.voice ? String(raw.tts.voice).slice(0, 64) : null,
|
||||||
|
volume: normalizeRange(raw?.tts?.volume, 0, 200),
|
||||||
speed: normalizeSpeed(raw?.tts?.speed),
|
speed: normalizeSpeed(raw?.tts?.speed),
|
||||||
pitch: normalizePitch(raw?.tts?.pitch),
|
pitch: normalizePitch(raw?.tts?.pitch),
|
||||||
expression: normalizeExpression(raw?.tts?.expression),
|
expression: normalizeExpression(raw?.tts?.expression),
|
||||||
|
|||||||
@@ -18,6 +18,8 @@ export type TtsProvider = "local" | "espeak" | "command" | "api";
|
|||||||
export interface TtsOptions {
|
export interface TtsOptions {
|
||||||
provider?: TtsProvider;
|
provider?: TtsProvider;
|
||||||
voice?: string;
|
voice?: string;
|
||||||
|
/** Playback volume percent, 0-200. The daemon applies it at play time. */
|
||||||
|
volume?: number;
|
||||||
/** Speech-rate multiplier; the engine maps it to its own units. */
|
/** Speech-rate multiplier; the engine maps it to its own units. */
|
||||||
speed?: number;
|
speed?: number;
|
||||||
/** Pitch offset in semitones; the engine maps it to its own units. */
|
/** Pitch offset in semitones; the engine maps it to its own units. */
|
||||||
|
|||||||
+104
-66
@@ -14,7 +14,7 @@ import { existsSync, readFileSync } from "node:fs";
|
|||||||
import { Readable } from "node:stream";
|
import { Readable } from "node:stream";
|
||||||
import http from "node:http";
|
import http from "node:http";
|
||||||
import https from "node:https";
|
import https from "node:https";
|
||||||
import { dirname } from "node:path";
|
import { basename, dirname, join } from "node:path";
|
||||||
import { fileURLToPath } from "node:url";
|
import { fileURLToPath } from "node:url";
|
||||||
import QRCode from "qrcode";
|
import QRCode from "qrcode";
|
||||||
import { WebSocketServer } from "ws";
|
import { WebSocketServer } from "ws";
|
||||||
@@ -25,11 +25,11 @@ import { CoreError, type CoreClient } from "./core-client.js";
|
|||||||
import type { CoreProcess } from "./core-process.js";
|
import type { CoreProcess } from "./core-process.js";
|
||||||
import { MediaLimitError, type MediaService } from "./media.js";
|
import { MediaLimitError, type MediaService } from "./media.js";
|
||||||
import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js";
|
import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js";
|
||||||
import type { PreferencesStore, WebPreferences } from "./preferences.js";
|
import type { PreferencesStore, TtsPreferences, WebPreferences } from "./preferences.js";
|
||||||
import type { Puppet } from "./puppet.js";
|
import type { Puppet } from "./puppet.js";
|
||||||
import { ensureTls } from "./tls.js";
|
import { ensureTls } from "./tls.js";
|
||||||
import type { TranscriptStore } from "./transcript.js";
|
import type { TranscriptStore } from "./transcript.js";
|
||||||
import type { TtsProvider, TtsVoice } from "./tts.js";
|
import type { TtsOptions, TtsProvider, TtsVoice } from "./tts.js";
|
||||||
|
|
||||||
const PINNED_LANGUAGES = ["en", "sv", "de", "da", "nb", "no", "ru", "chr"];
|
const PINNED_LANGUAGES = ["en", "sv", "de", "da", "nb", "no", "ru", "chr"];
|
||||||
const LANGUAGE_NAMES: Record<string, string> = {
|
const LANGUAGE_NAMES: Record<string, string> = {
|
||||||
@@ -231,6 +231,77 @@ function parseMediaFilters(query: Record<string, unknown>) {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Turn a speak/preview request body into TTS options. Missing fields fall back
|
||||||
|
* to the stored preferences.
|
||||||
|
*/
|
||||||
|
function buildSpeechOptions(body: any, tts: TtsPreferences, voice: string | undefined): TtsOptions {
|
||||||
|
const requested = body?.provider;
|
||||||
|
const provider = ["api", "local", "espeak", "command"].includes(requested)
|
||||||
|
? requested
|
||||||
|
: tts.provider;
|
||||||
|
// Number(null) is 0: null, undefined, and "" all mean "unset".
|
||||||
|
const optional = (value: unknown): number | undefined => {
|
||||||
|
if (value === undefined || value === null || value === "") return undefined;
|
||||||
|
const number = Number(value);
|
||||||
|
return Number.isFinite(number) ? number : undefined;
|
||||||
|
};
|
||||||
|
|
||||||
|
const speed = optional(body?.speed) ?? tts.speed ?? undefined;
|
||||||
|
const pitch = optional(body?.pitch) ?? tts.pitch ?? undefined;
|
||||||
|
const expression = optional(body?.expression) ?? tts.expression ?? undefined;
|
||||||
|
const profile = body?.profile && typeof body.profile === "object" ? body.profile : tts.profile;
|
||||||
|
const highpass = optional(body?.highpass) ?? tts.highpass ?? undefined;
|
||||||
|
const echo = optional(body?.echo) ?? tts.echo ?? undefined;
|
||||||
|
const words = body?.words !== undefined ? body.words === true : tts.words;
|
||||||
|
const wordGapMs = optional(body?.wordGapMs) ?? tts.wordGapMs ?? undefined;
|
||||||
|
// The Half-Life VOX voice carries its prosody with it.
|
||||||
|
const vox =
|
||||||
|
(body?.vox !== undefined ? body.vox === true : tts.vox) || voice === "vox";
|
||||||
|
const voxClean = body?.voxClean !== undefined ? body.voxClean === true : tts.voxClean;
|
||||||
|
const preset =
|
||||||
|
body?.preset === "vox" || body?.preset === "hev"
|
||||||
|
? body.preset
|
||||||
|
: body?.preset === null || body?.preset === ""
|
||||||
|
? null
|
||||||
|
: tts.preset;
|
||||||
|
const echoDelayMs = optional(body?.echoDelayMs) ?? tts.echoDelayMs ?? undefined;
|
||||||
|
const echoLevel = optional(body?.echoLevel) ?? tts.echoLevel ?? undefined;
|
||||||
|
const crunch =
|
||||||
|
["hl90", "hl90hq", "telephone", "adpcm", "crush"].includes(String(body?.crunch ?? ""))
|
||||||
|
? (String(body.crunch) as "hl90" | "hl90hq" | "telephone" | "adpcm" | "crush")
|
||||||
|
: body?.crunch === null || body?.crunch === "" || body?.crunch === "off"
|
||||||
|
? undefined
|
||||||
|
: tts.crunch ?? undefined;
|
||||||
|
const crunchBits = optional(body?.crunchBits) ?? tts.crunchBits ?? undefined;
|
||||||
|
const crunchRate = optional(body?.crunchRate) ?? tts.crunchRate ?? undefined;
|
||||||
|
const streamWords =
|
||||||
|
body?.streamWords !== undefined ? body.streamWords !== false : tts.streamWords;
|
||||||
|
|
||||||
|
return {
|
||||||
|
provider: provider ?? undefined,
|
||||||
|
voice,
|
||||||
|
...(Number.isFinite(speed) ? { speed } : {}),
|
||||||
|
...(Number.isFinite(pitch) ? { pitch } : {}),
|
||||||
|
...(Number.isFinite(expression) ? { expression } : {}),
|
||||||
|
...(profile && Object.keys(profile).length ? { profile } : {}),
|
||||||
|
...(Number.isFinite(highpass) ? { highpass } : {}),
|
||||||
|
...(Number.isFinite(echo) ? { echo } : {}),
|
||||||
|
...(words ? { words: true } : {}),
|
||||||
|
...(Number.isFinite(wordGapMs) ? { wordGapMs } : {}),
|
||||||
|
...(vox ? { vox: true } : {}),
|
||||||
|
...(voxClean ? { voxClean: true } : {}),
|
||||||
|
...(preset ? { preset } : {}),
|
||||||
|
...(Number.isFinite(echoDelayMs) ? { echoDelayMs } : {}),
|
||||||
|
...(Number.isFinite(echoLevel) ? { echoLevel } : {}),
|
||||||
|
...(crunch ? { crunch } : {}),
|
||||||
|
...(Number.isFinite(crunchBits) ? { crunchBits } : {}),
|
||||||
|
...(Number.isFinite(crunchRate) ? { crunchRate } : {}),
|
||||||
|
stream: streamWords && words === true
|
||||||
|
} as TtsOptions;
|
||||||
|
}
|
||||||
|
|
||||||
export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
||||||
const app = express();
|
const app = express();
|
||||||
app.use(express.json({ limit: "64kb" }));
|
app.use(express.json({ limit: "64kb" }));
|
||||||
@@ -508,6 +579,7 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
|||||||
provider: tts?.provider ?? null,
|
provider: tts?.provider ?? null,
|
||||||
language: tts?.language ?? null,
|
language: tts?.language ?? null,
|
||||||
voice: tts?.voice ?? null,
|
voice: tts?.voice ?? null,
|
||||||
|
volume: tts?.volume ?? null,
|
||||||
speed: tts?.speed ?? null,
|
speed: tts?.speed ?? null,
|
||||||
pitch: tts?.pitch ?? null,
|
pitch: tts?.pitch ?? null,
|
||||||
expression: tts?.expression ?? null,
|
expression: tts?.expression ?? null,
|
||||||
@@ -794,80 +866,46 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
|||||||
const text = String(req.body?.text ?? "");
|
const text = String(req.body?.text ?? "");
|
||||||
const prefs = deps.preferences.get();
|
const prefs = deps.preferences.get();
|
||||||
|
|
||||||
const requested = req.body?.provider;
|
|
||||||
const provider = ["api", "local", "espeak", "command"].includes(requested)
|
|
||||||
? requested
|
|
||||||
: prefs.tts.provider;
|
|
||||||
const voice = req.body?.voice ? String(req.body.voice) : prefs.tts.voice ?? undefined;
|
const voice = req.body?.voice ? String(req.body.voice) : prefs.tts.voice ?? undefined;
|
||||||
// Number(null) is 0: null, undefined, and "" all mean "unset".
|
const options = buildSpeechOptions(req.body ?? {}, prefs.tts, voice);
|
||||||
const optional = (value: unknown): number | undefined => {
|
|
||||||
if (value === undefined || value === null || value === "") return undefined;
|
|
||||||
const number = Number(value);
|
|
||||||
return Number.isFinite(number) ? number : undefined;
|
|
||||||
};
|
|
||||||
|
|
||||||
const speed = optional(req.body?.speed) ?? prefs.tts.speed ?? undefined;
|
const result = await deps.audio.speak(text, options);
|
||||||
const pitch = optional(req.body?.pitch) ?? prefs.tts.pitch ?? undefined;
|
|
||||||
const expression = optional(req.body?.expression) ?? prefs.tts.expression ?? undefined;
|
|
||||||
const profile = req.body?.profile && typeof req.body.profile === "object" ? req.body.profile : prefs.tts.profile;
|
|
||||||
const highpass = optional(req.body?.highpass) ?? prefs.tts.highpass ?? undefined;
|
|
||||||
const echo = optional(req.body?.echo) ?? prefs.tts.echo ?? undefined;
|
|
||||||
const words = req.body?.words !== undefined ? req.body.words === true : prefs.tts.words;
|
|
||||||
const wordGapMs = optional(req.body?.wordGapMs) ?? prefs.tts.wordGapMs ?? undefined;
|
|
||||||
// The Half-Life VOX voice carries its prosody with it.
|
|
||||||
const vox =
|
|
||||||
(req.body?.vox !== undefined ? req.body.vox === true : prefs.tts.vox) || voice === "vox";
|
|
||||||
const voxClean = req.body?.voxClean !== undefined ? req.body.voxClean === true : prefs.tts.voxClean;
|
|
||||||
const preset =
|
|
||||||
req.body?.preset === "vox" || req.body?.preset === "hev"
|
|
||||||
? req.body.preset
|
|
||||||
: req.body?.preset === null || req.body?.preset === ""
|
|
||||||
? null
|
|
||||||
: prefs.tts.preset;
|
|
||||||
const echoDelayMs = optional(req.body?.echoDelayMs) ?? prefs.tts.echoDelayMs ?? undefined;
|
|
||||||
const echoLevel = optional(req.body?.echoLevel) ?? prefs.tts.echoLevel ?? undefined;
|
|
||||||
const crunch =
|
|
||||||
["hl90", "hl90hq", "telephone", "adpcm", "crush"].includes(String(req.body?.crunch ?? ""))
|
|
||||||
? (String(req.body.crunch) as "hl90" | "hl90hq" | "telephone" | "adpcm" | "crush")
|
|
||||||
: req.body?.crunch === null || req.body?.crunch === "" || req.body?.crunch === "off"
|
|
||||||
? undefined
|
|
||||||
: prefs.tts.crunch ?? undefined;
|
|
||||||
const crunchBits = optional(req.body?.crunchBits) ?? prefs.tts.crunchBits ?? undefined;
|
|
||||||
const crunchRate = optional(req.body?.crunchRate) ?? prefs.tts.crunchRate ?? undefined;
|
|
||||||
const streamWords =
|
|
||||||
req.body?.streamWords !== undefined ? req.body.streamWords !== false : prefs.tts.streamWords;
|
|
||||||
|
|
||||||
const result = await deps.audio.speak(text, {
|
|
||||||
provider: provider ?? undefined,
|
|
||||||
voice,
|
|
||||||
...(Number.isFinite(speed) ? { speed } : {}),
|
|
||||||
...(Number.isFinite(pitch) ? { pitch } : {}),
|
|
||||||
...(Number.isFinite(expression) ? { expression } : {}),
|
|
||||||
...(profile && Object.keys(profile).length ? { profile } : {}),
|
|
||||||
...(Number.isFinite(highpass) ? { highpass } : {}),
|
|
||||||
...(Number.isFinite(echo) ? { echo } : {}),
|
|
||||||
...(words ? { words: true } : {}),
|
|
||||||
...(Number.isFinite(wordGapMs) ? { wordGapMs } : {}),
|
|
||||||
...(vox ? { vox: true } : {}),
|
|
||||||
...(voxClean ? { voxClean: true } : {}),
|
|
||||||
...(preset ? { preset } : {}),
|
|
||||||
...(Number.isFinite(echoDelayMs) ? { echoDelayMs } : {}),
|
|
||||||
...(Number.isFinite(echoLevel) ? { echoLevel } : {}),
|
|
||||||
...(crunch ? { crunch } : {}),
|
|
||||||
...(Number.isFinite(crunchBits) ? { crunchBits } : {}),
|
|
||||||
...(Number.isFinite(crunchRate) ? { crunchRate } : {}),
|
|
||||||
stream: streamWords && words === true
|
|
||||||
});
|
|
||||||
deps.getPuppet()?.appendAction({
|
deps.getPuppet()?.appendAction({
|
||||||
type: "audio",
|
type: "audio",
|
||||||
action: "speak",
|
action: "speak",
|
||||||
provider: provider ?? deps.audio.status().tts.defaultProvider,
|
provider: options.provider ?? deps.audio.status().tts.defaultProvider,
|
||||||
chars: text.trim().length
|
chars: text.trim().length
|
||||||
});
|
});
|
||||||
res.json({ ok: true, result });
|
res.json({ ok: true, result });
|
||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
|
||||||
|
app.post(
|
||||||
|
"/api/audio/preview",
|
||||||
|
wrap(async (req, res) => {
|
||||||
|
const text = String(req.body?.text ?? "").trim();
|
||||||
|
const prefs = deps.preferences.get();
|
||||||
|
const voice = req.body?.voice ? String(req.body.voice) : prefs.tts.voice ?? undefined;
|
||||||
|
const file = await deps.audio.preview(text, buildSpeechOptions(req.body ?? {}, prefs.tts, voice));
|
||||||
|
|
||||||
|
res.json({ ok: true, result: { url: `/api/audio/tts/${encodeURIComponent(basename(file))}` } });
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
// Preview files only: the name pattern blocks path traversal.
|
||||||
|
app.get(
|
||||||
|
"/api/audio/tts/:name",
|
||||||
|
wrap(async (req, res) => {
|
||||||
|
const name = String(req.params.name);
|
||||||
|
if (!/^tts-[0-9]+-[0-9]+\.wav$/.test(name)) throw new HttpError(400, "invalid audio name");
|
||||||
|
|
||||||
|
const file = join(deps.audio.ttsDirectory, name);
|
||||||
|
if (!existsSync(file)) throw new HttpError(404, "audio not found");
|
||||||
|
|
||||||
|
res.type("audio/wav").sendFile(file);
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
app.post(
|
app.post(
|
||||||
"/api/audio/stop",
|
"/api/audio/stop",
|
||||||
wrap(async (_req, res) => {
|
wrap(async (_req, res) => {
|
||||||
|
|||||||
Reference in New Issue
Block a user