zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs and README for voice audio, capture, and the web UI

Browse Source
This commit is contained in:
Fisto
2026-09-14 11:19:11 +02:00
parent c003d9532f
commit c7aad7547f
10 changed files with 356 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CURRENT.md
+59 -7
View File
@@ -6,12 +6,39 @@ This file tracks the work state. Read it first. Update it before you stop.
## Current focus (broad)
The text layer is done. The web UI follows the Nocturne design handoff. The voice backend is the
next feature; live verification and a full QR approval are pending. The LLM attachment and MCP
event delivery are parked.
The text layer is done. The web UI follows the Nocturne design handoff. Voice actions are in
progress: join/leave/move work; auto-leave and `stay @user` are next. Live verification and a full
QR approval are pending. The LLM attachment and MCP event delivery are parked.
## In progress
- Voice actions are started: `voice.join` (muted), `voice.leave`, and `voice.move` are implemented
in the core, wired through the daemon, Web UI, and MCP, and mock-verified. The internal
`selectVoiceChannel` and `setSelfMute` lookups were probed live on the real client.
- Auto-leave and `stay @user` are the next voice step.
- Voice audio is implemented: daemon-managed virtual devices (`veskslop_tts`, `veskslop_mic`,
`veskslop_capture`), TTS providers (local command + HTTP API), queued playback through `ffplay`,
ducking on `VOICE_SPEAKING`, and WAV capture. The daemon starts `pulseaudio` when `pactl` cannot
reach a server and reports install hints. Verified live on 2026-09-13 after installing
`pulseaudio`/`ffmpeg`: engine ok, devices created, PCM streamed.
- The Web UI serves self-signed HTTPS (`openssl`, stored under the state dir) so browser media
APIs have a secure context. `WEBUI_TLS=0` disables it.
- Capture and transcript are implemented: ffmpeg reads the client output, a VAD cuts it into
utterances, each becomes a WAV plus a transcript entry with the dominant speaker, and a
pluggable STT hook (local command or HTTP API) fills the text. Live events feed the console
Transcript tab; `get_transcript` and `/api/transcript` read the store. `VESKSLOP_CAPTURE_AUTO=1`
records calls automatically. Verified end to end with real PulseAudio on 2026-09-13.
- STT uses Vosk (CPU, light): `poc/scripts/stt-vosk.py` plus a small model in `poc/models/`
(auto-detected). Verified with a TTS→capture→Vosk round trip. Other providers remain available
via `VESKSLOP_STT_CMD`/`_API_URL`.
- TTS defaults to `espeak-ng` when installed; engine and voice live in Settings → Client → Speech
(daemon preferences, so all browsers share them).
- Preferences moved to `<state dir>/preferences.json` and no longer depend on the core, so guild
visibility saves even while the client is down. API responses send `Cache-Control: no-store`.
- Browser audio has an **Enable browser audio** button that requests the microphone permission,
which browsers require before listing output devices.
- Browser microphone puppeting (talking through the bot) is next; it needs a mix sink, a `pacat`
input, and echo handling.
- The web UI follows the Nocturne design handoff (`poc/WebUIMockup.zip`). The 7a shell is
implemented and rendered headless for review: header, call strip, three-column body, composer,
and the agent console.
@@ -20,7 +47,6 @@ event delivery are parked.
auto-Xvfb, the bridge connects, and stop exits cleanly.
- Web QR login is implemented. The session reached `pending` against the real gateway; a full
phone approval is pending.
- Voice actions are designed in [docs/voice.md](docs/voice.md). The backend is not started.
## Done
@@ -39,6 +65,24 @@ event delivery are parked.
- `user.search`: cache and guild-member lookup. `voice.get` searches all guilds without a `guildId`.
- `voice.overview`: occupied voice channels and members for a guild.
- `VOICE_STATE_UPDATE` events: normalized and pushed. Mock-verified.
- Voice actions: `voice.join` (muted), `voice.leave`, and `voice.move` in the core, with daemon
routes (`/api/voice/*`), MCP tools, and Web UI wiring. Voice actions are paced 250-750 ms and
written to the action log. The internal lookups were probed on the live client.
- Web UI voice rail: every voice channel is clickable. Clicking joins (muted); clicking the active
channel leaves. Verified with the stateful mock.
- Web UI startup: a **Start server** button appears left of Takeover while the bridge is down and
hides when it connects. State reloads retry while the client boots, so guilds appear without a
manual refresh. Core requests now time out after `VESKSLOP_REQUEST_TIMEOUT_MS` instead of
hanging forever on a missing reply.
- Settings: "Visible guilds" stays top level; Audio sits under a collapsible "Client" group;
Core process, Account login, and Action log sit under a collapsible "Server" group.
- Web UI timestamps use the browser timezone by default; Settings → Client → Display can override
it with any IANA zone. The choice is per browser (`localStorage`).
- Web UI preferences: visible guilds persist server-side in the daemon state file
(`/api/preferences`), so every browser shares them. The Settings panel has an explicit **Save**
button; selections filter the UI live and persist only on Save.
- DM list: recipients resolve from IDs through the user store, so names and avatars load.
- Guild menu: fixed the light default button background.
- Web UI: Nocturne design tokens in `poc/public/nocturne.css`, the 7a debug shell, live guild and
channel rail, messages, DMs, voice membership, agent console with a live event tab, and a
Settings slide-over for core process, login, visible guilds, and the action log. Roadmap
@@ -59,11 +103,19 @@ event delivery are parked.
is written on approval and a running managed core restarts with the new token.
- Unit tests cover the pacing policy. `pnpm lint` and `pnpm testTypes` pass at the root.
- Web UI navigation: Guild/DM scope tabs, a DM rail with unread badges (purple/orange, grey while
Takeover is off), last-message times, favourite stars, and a People list of users seen in voice.
Favourited users get presence and call tracking, grouped by call in **In voice**. Speaking shows
level bars and bold names (2c marks for mute/deaf/video/share). The body dims while the bridge
is down; the Start button becomes Refresh when the daemon is unreachable.
- TTS settings are split into language and voice, grouped by base language with the requested
languages pinned on top.
## Next (in order)
1. Voice backend: `voice.join`, `voice.leave`, `voice.move`, auto-leave, and the `stay @user` command.
2. Live verification: `user.search`, `voice.overview`, `VOICE_STATE_UPDATE`, and a full web QR login
approval on the real client.
1. Voice auto-leave (`voiceLeaveDelaySeconds`) and the `stay @user` command.
2. Live verification: `voice.join`/`voice.leave` end to end, `user.search`, `voice.overview`,
`VOICE_STATE_UPDATE`, and a full web QR login approval on the real client.
3. Notification parsing: edits, deletes, reactions, and message references.
4. Reply-to-puppet counts as a trigger.
5. Per-user rate limit and command cooldowns.