zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs and README for voice audio, capture, and the web UI

Browse Source
This commit is contained in:
Fisto
2026-09-14 11:19:11 +02:00
parent c003d9532f
commit c7aad7547f
10 changed files with 356 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/webui.md
+81 -15
View File
@@ -32,6 +32,19 @@ only. Do not expose the port to the public internet. See [daemon.md](daemon.md).
The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
## TLS
The UI serves HTTPS with a self-signed certificate so browser media APIs have a secure context.
The daemon generates the certificate with `openssl` on first run (localhost plus the host's LAN
addresses) and stores it under `<state dir>/tls/`. The browser shows a warning once; accept it.
| Setting | Default | Purpose |
| --- | --- | --- |
| `WEBUI_TLS` | `1` | Set to `0` to serve plain HTTP. |
| `WEBUI_TLS_KEY` / `WEBUI_TLS_CERT` | unset | Use an existing certificate instead. |
With plain HTTP, `navigator.mediaDevices` only exists on localhost; remote browsers must use HTTPS.
## Page
- Plain HTML, CSS, and JavaScript in `poc/public/`. No build step and no frontend framework.
@@ -39,12 +52,21 @@ The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
in `poc/WebUIMockup.zip`. Implemented screen: **7a**, the docked three-column debug shell.
- Layout: 50px header, conditional call strip, `236px | 1fr | 306px` body, and the agent console
bar (35px collapsed, resizable drawer expanded).
- Header: wordmark, guild picker, health indicators, Takeover switch, Settings.
- Left rail: guild switcher, text channels, voice channels with inline participants, direct
messages, and the account footer.
- Center: channel header, message list, composer, and the TTS panel.
- Right column: In voice, Favourites, Open DMs & groups.
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event.
- Header: wordmark, **Guild / DM** scope tabs, guild picker, health indicators, **Client audio**,
Takeover switch, Settings. While the bridge is down a **Start server** button appears with a
pulsing bloom; if the whole daemon is unreachable it becomes **Refresh** and reloads the page.
The body dims and stops taking input while the bridge is down.
- Left rail (Guild tab): guild switcher, text channels, voice channels with inline participants,
and **People** — users recently seen in a voice channel (click opens their cached DM). Clicking a
voice channel joins it (muted); clicking the active channel leaves.
- Left rail (DM tab): every cached DM and group with unread badge, avatar, name, last-message time,
and a favourite star.
- Right column: **In voice** (favourited users in a call, grouped by call), **Favourites**, and
**Open DMs & groups** as quick access. The guild's own occupied channels live in the left rail.
- Speaking users show animated level bars and a bold name (design 2c: audio → video → share marks).
Nicknames come from the guild member store; DMs fall back to the username.
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event. The
Transcript tab lists captured segments (time, speaker, text, WAV) and updates live.
- Opening a chat fetches the latest 50 messages from Discord into the client cache. The refresh
button by the channel header re-fetches. Live message events read the cache.
- Attachments render from their Discord CDN links. Images display inline; other files are link
@@ -53,9 +75,23 @@ The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
- Composer and TTS drafts persist in `localStorage` per channel. They survive re-renders, the
bridge connecting, and a page reload. A successful send clears the channel draft.
- When the bridge connects after the page loaded, the page reloads state without touching drafts.
The reload retries while the client boots, and the health poll keeps retrying if the first
attempt failed.
- Roadmap gaps are shown, not hidden: TTS, transcript, tool-call tracing, favourites, and voice
join/monitor appear as marked placeholders.
## DMs and favourites
- The **DM** scope tab shows all cached DMs in the rail and the conversation in the centre.
- Unread badges sit left of the avatar: purple for unread, orange for mentions while Takeover is
on; while Takeover is off they are grey, meaning the bot saw them and you did not.
- The hollow star on a row adds it to favourites, stored in the daemon preferences. Starring a 1:1
DM favourites that user; starring a group DM favourites the channel.
- Favourited users get presence tracking. If they are in a call, they appear under **In voice**,
grouped per call. The right column's **Favourites** shows their status when they are not.
- The last-message time comes from the channel's last message snowflake: a time today, `yesterday`,
or a short date.
## Takeover
Takeover is a read-only switch. Off, the composer is locked (a `read-only` tag sits next to the
@@ -65,13 +101,30 @@ only; the API itself does not enforce it.
## Settings
The Settings button opens a slide-over with the server-side controls:
The Settings button opens a slide-over:
- **Core process** — state, last exit, output, Start / Stop / Restart.
- **Account login** — token file state, QR login, captcha.
- **Visible guilds** — checkboxes for the guild switcher and picker. Stored in `localStorage`;
"Show all" clears the filter.
- **Action log** — the persistent JSONL log, newest first.
- **Visible guilds** — top level. Checkboxes filter the guild switcher and picker live. **Save**
writes the selection to the daemon preferences (`/api/preferences`); **Hide all** and **Show all**
select none or every guild. All visible is stored as `null`, none as `[]`. All browsers share
the saved state, and Save works while the client is disconnected.
- **Client** — a collapsible group with:
- **Bot audio** — engine status, TTS provider state, a **Connect devices** button, and the
capture toggle. These are the daemon host's devices.
- **Speech** — the TTS engine (Auto / Local / API), language, and voice. Voices are grouped by
base language and filtered by the chosen language; English, Swedish, German, Danish,
Norwegian, Russian, and Cherokee are pinned on top. Stored in the daemon preferences.
- **Display** — the timestamp timezone. Defaults to the browser's zone
(`Intl.DateTimeFormat().resolvedOptions().timeZone`); pick any IANA zone to override. Stored in
`localStorage` and applied to messages, the transcript, the event log, the action log, and the
core status.
- **Browser audio** — puppeting controls: the browser output device for Client audio, a
placeholder mic select, and the call-audio state. **Enable browser audio** asks for the
microphone permission, which browsers require before they list output devices. Separate from
the bot's devices.
- **Server** — a collapsible group with:
- **Core process** — state, last exit, output, Start / Stop / Restart.
- **Account login** — token file state, QR login, captcha.
- **Action log** — the persistent JSONL log, newest first.
## Core process
@@ -126,8 +179,21 @@ replaceable.
| GET | `/api/dms` | `limit?` | Cached DM and group DM channels with recipients |
| GET | `/api/voice` | `userId?`, `guildId?` | One user's voice state |
| GET | `/api/voice/overview` | `guildId` | Occupied voice channels and members |
| POST | `/api/voice/join` | `channelId` | Reserved. Returns `501` |
| POST | `/api/voice/leave` | none | Reserved. Returns `501` |
| POST | `/api/voice/join` | `channelId`, `mute?` | Join a voice channel (muted by default) |
| POST | `/api/voice/leave` | none | Leave the current voice channel |
| POST | `/api/voice/move` | `channelId` | Move to another voice channel |
| GET | `/api/preferences` | none | Web UI preferences (`visibleGuilds`, `tts`) |
| POST | `/api/preferences` | `visibleGuilds?`, `tts?` | Update preferences. Stored in `<state dir>/preferences.json`. |
| GET | `/api/audio` | none | Audio engine, TTS, queue, ducking, capture state |
| GET | `/api/audio/devices` | none | The client's input and output devices |
| GET | `/api/audio/tts/voices` | none | Local TTS voices (espeak-ng) |
| POST | `/api/audio/connect` | none | Re-apply the virtual devices to the client |
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?` | Queue text to speech |
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
| POST | `/api/audio/capture` | `start` | Start or stop capture |
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
| GET | `/api/logs` | `limit` | The persistent action log |
| GET | `/api/events` | none | Server-sent events |
@@ -140,7 +206,7 @@ gap. See [daemon.md](daemon.md).
## Not started
- Web UI authentication and user accounts.
- Voice join and leave. The UI elements exist and call the reserved routes.
- Voice auto-leave and the `stay @user` command.
- TTS playback, transcript capture, and MCP tool-call tracing.
- Favourites, presence, member counts, and message editing/deletion/reactions.
- Pagination of channel history.