Update docs and README for voice audio, capture, and the web UI
This commit is contained in:
@@ -37,6 +37,7 @@ mention the puppet first.
|
||||
| `whitelist list` | Show the owner and the trusted users with their tiers. |
|
||||
| `whitelist add @user [user\|trusted\|admin]` | Add one or more users. The default tier is `user`. |
|
||||
| `whitelist remove @user` | Remove one or more users from the trust list. |
|
||||
| `speak <text>` | Speak text into the bot's current voice call. `speak stop` stops it. |
|
||||
| `poll ...` | Start a reaction poll. Paused at this time. |
|
||||
| `stay @user` | Planned with the voice backend. Pin the bot to a voice user. See [voice.md](voice.md). |
|
||||
|
||||
@@ -178,6 +179,9 @@ event. The log rotates at the size limit.
|
||||
| `continuation` | Follow-up message ID and the parent entry ID |
|
||||
| `command` | Command text, requester, result |
|
||||
| `reply` | Command, mode, target, result, error |
|
||||
| `voice` | Voice action, channel, mute |
|
||||
| `audio` | Audio action (speak, stop, devices, capture), provider, size |
|
||||
| `manual` | Web UI send: mode, channel or user, content length |
|
||||
| `error` | Scope and message |
|
||||
|
||||
Secret-looking strings are redacted before the write. Discord token shapes and `mfa.` strings
|
||||
|
||||
@@ -59,6 +59,11 @@ Event line:
|
||||
| `state.get` | none | The current user, the guilds, and the channels. Cache-only. Includes user avatars and guild icons. |
|
||||
| `voice.get` | optional `userId`, optional `guildId` | The voice channel and its members, or `channelId: null`. Without `guildId` it searches all guilds. |
|
||||
| `voice.overview` | `guildId` | The occupied voice channels in the guild and their members |
|
||||
| `voice.join` | `channelId`, optional `mute` (default true) | Joins the voice channel. Sets self-mute first. |
|
||||
| `voice.leave` | none | Leaves the current voice channel. |
|
||||
| `voice.move` | `channelId` | Moves the client to another voice channel. |
|
||||
| `voice.devices` | none | The client's input and output audio devices. |
|
||||
| `voice.setDevices` | optional `inputDeviceId`, optional `outputDeviceId` | Sets the client devices. |
|
||||
| `user.search` | `query`, optional `guildId`, optional `limit` | Users from the client cache. Guild members when `guildId` is set. |
|
||||
| `dm.list` | optional `limit` | Cached DM and group DM channels with recipients, icons, and avatars. Cache-only. |
|
||||
| `message.send` | `channelId`, `content` | `{ channelId, messageId, sent: true }` |
|
||||
@@ -77,6 +82,7 @@ The renderer normalizes client events and pushes them to subscribers.
|
||||
| `MESSAGE_REACTION_ADD` | Message ID, channel, guild, user, emoji |
|
||||
| `MESSAGE_REACTION_REMOVE` | Same fields as add |
|
||||
| `VOICE_STATE_UPDATE` | User, channel, guild, session, mute, deaf, self-mute, self-deaf |
|
||||
| `VOICE_SPEAKING` | User, speaking flag, channel |
|
||||
|
||||
## Validation rules
|
||||
|
||||
@@ -94,6 +100,12 @@ The core validates all input. The daemon cannot bypass these rules.
|
||||
to walk history.
|
||||
- `voice.get` accepts snowflakes only. It reads the client cache. It never joins a call.
|
||||
- `voice.overview` requires a guild snowflake. It reads the client cache.
|
||||
- `voice.join` and `voice.move` require a channel snowflake and a voice channel type (`2` or `13`).
|
||||
The join sets self-mute when the action is available. The internal lookups
|
||||
(`selectVoiceChannel`, `setSelfMute`) were verified live on 2026-09-13.
|
||||
- `voice.leave` takes no parameters. It disconnects the client.
|
||||
- `voice.setDevices` takes short device ID strings. The device list comes from
|
||||
`navigator.mediaDevices.enumerateDevices()`.
|
||||
- `user.search` takes a query of 1 to 100 characters and a limit of 1 to 25. It reads the client
|
||||
cache only. An exact snowflake query returns that user when cached.
|
||||
- `dm.list` takes a limit of 1 to 50. It reads `ChannelStore.getSortedPrivateChannels()` only.
|
||||
@@ -126,6 +138,7 @@ The core rate limit is a safety net. It does not replace the daemon pacing.
|
||||
| `NOT_SENDABLE` | The channel type cannot receive messages. |
|
||||
| `SEND_FAILED` | Discord rejected the message. |
|
||||
| `REACTION_FAILED` | Discord rejected the reaction. |
|
||||
| `VOICE_FAILED` | Discord rejected the voice action. |
|
||||
| `DM_UNAVAILABLE` | The client cannot create a DM channel. |
|
||||
| `NOT_SUPPORTED` | The client does not offer the internal action. |
|
||||
| `RATE_LIMITED` | The core rate limit rejected the action. |
|
||||
|
||||
@@ -61,6 +61,16 @@ The daemon exposes MCP over Streamable HTTP. The default address is `http://127.
|
||||
| `send_dm` | Send a DM to a user. |
|
||||
| `get_voice_state` | Which voice channel a user is in and who else is there. |
|
||||
| `get_voice_overview` | Occupied voice channels in a guild and their members. |
|
||||
| `voice_join` | Join a voice channel, muted by default. |
|
||||
| `voice_leave` | Leave the current voice channel. |
|
||||
| `voice_move` | Move to another voice channel. |
|
||||
| `get_audio_status` | Audio engine, TTS, queue, ducking, and capture state. |
|
||||
| `speak` | Queue text to speech into the virtual microphone. |
|
||||
| `stop_audio` | Stop speech and clear the queue. |
|
||||
| `start_capture` / `stop_capture` | Record or stop recording the client output sink. |
|
||||
| `list_audio_devices` | The client's input and output audio devices. |
|
||||
| `set_audio_devices` | Set the client's input and output devices. |
|
||||
| `get_transcript` | Read captured call segments with speaker, WAV path, and text. |
|
||||
| `search_users` | Search the client user cache by username, global name, or nickname. |
|
||||
| `list_dms` | List cached DM and group DM channels with recipients. |
|
||||
| `get_message_log` | Read the trigger log. |
|
||||
@@ -135,6 +145,7 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
|
||||
| Setting | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `VESKSLOP_SOCKET` | `$XDG_RUNTIME_DIR/vesktop-puppet.sock` | Core bridge socket |
|
||||
| `VESKSLOP_REQUEST_TIMEOUT_MS` | `30000` | How long a core request waits before it fails. |
|
||||
| `VESKSLOP_STATE_FILE` | `poc/.veskslop-state.json` | Owner, trust list, and config |
|
||||
| `VESKSLOP_ACTION_LOG` | `<state dir>/actions.jsonl` | Persistent action log |
|
||||
| `VESKSLOP_ACTION_LOG_MAX_BYTES` | 5 MiB | Action log rotation limit |
|
||||
@@ -142,9 +153,23 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
|
||||
| `VESKSLOP_CACHE_MAX_BYTES` | 50 MiB | Context cache size limit |
|
||||
| `MCP_HOST` / `MCP_PORT` | `127.0.0.1` / `8788` | MCP listen address |
|
||||
| `WEBUI_HOST` / `WEBUI_PORT` | `0.0.0.0` / `8789` | Web UI listen address. No authentication. |
|
||||
| `WEBUI_TLS` / `WEBUI_TLS_KEY` / `WEBUI_TLS_CERT` | `1` / unset | Self-signed HTTPS, or an existing certificate. |
|
||||
| `VESKSLOP_CORE_CMD` | unset | Full command that starts the client. Overrides the default. |
|
||||
| `VESKSLOP_CORE_ARGS` | `--no-sandbox --disable-gpu --disable-dev-shm-usage` | Flags for the default Electron command. |
|
||||
| `VESKSLOP_XVFB` | unset | Set to `0` to disable automatic Xvfb start on Linux. |
|
||||
| `VESKSLOP_AUDIO_AUTOCONNECT` | `1` | Point the client at the virtual devices on connect. |
|
||||
| `VESKSLOP_AUDIO_MANAGE_SERVER` | `1` | Start `pulseaudio` when `pactl` cannot reach a server. |
|
||||
| `VESKSLOP_AUDIO_SINK` / `_SOURCE` / `_CAPTURE_SINK` | `veskslop_tts` / `veskslop_mic` / `veskslop_capture` | Virtual device names. |
|
||||
| `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. |
|
||||
| `VESKSLOP_FFPLAY` / `VESKSLOP_FFMPEG` | `ffplay` / `ffmpeg` | Playback and capture binaries. |
|
||||
| `VESKSLOP_TTS_CMD` | unset | Local TTS command with `{out}` and `{text_file}`. |
|
||||
| `VESKSLOP_TTS_API_URL` / `_API_KEY` / `_PROVIDER` | unset | HTTP TTS provider and bearer token. |
|
||||
| `VESKSLOP_TRANSCRIPT` | `<state dir>/transcripts.jsonl` | Transcript store path. |
|
||||
| `VESKSLOP_PREFERENCES` | `<state dir>/preferences.json` | Web UI preferences (visible guilds, TTS). |
|
||||
| `VESKSLOP_CAPTURE_AUTO` | `0` | Start capture when the bot joins a call. |
|
||||
| `VESKSLOP_VAD_THRESHOLD_DB` / `_MIN_SPEECH_MS` / `_SILENCE_MS` / `_PRE_ROLL_MS` / `_MAX_SEGMENT_MS` | `-35` / `120` / `600` / `300` / `30000` | VAD tuning. |
|
||||
| `VESKSLOP_STT_CMD` / `_API_URL` / `_API_KEY` | unset | Speech-to-text provider. |
|
||||
| `VESKSLOP_VOSK_MODEL` / `_VOSK_DIR` | `poc/models/vosk-model*` | Local Vosk model, auto-detected when missing. |
|
||||
|
||||
## How to add an MCP tool
|
||||
|
||||
|
||||
+3
-1
@@ -6,12 +6,14 @@ This document describes the test setup, the run commands, and the known gaps.
|
||||
|
||||
The pure daemon logic has unit tests. The tests cover the ring buffers, the mention helpers, the
|
||||
command parser, the poll parser, the trust helpers, the context assembly, the redaction, and the
|
||||
pacing policy (`planSend`). The cache tests cover merge, dedupe, trimming, and eviction.
|
||||
pacing policy (`planSend`). The cache tests cover merge, dedupe, trimming, and eviction. The audio
|
||||
tests cover the VAD, the WAV writer, the transcript store, and a local STT command.
|
||||
|
||||
```sh
|
||||
cd poc
|
||||
pnpm test:comm
|
||||
pnpm test:cache
|
||||
pnpm test:audio
|
||||
```
|
||||
|
||||
The files are `poc/src/test-communication.ts` and `poc/src/test-cache.ts`. The tests do not need
|
||||
|
||||
+128
-13
@@ -1,6 +1,7 @@
|
||||
# Voice
|
||||
|
||||
Voice state reads exist. Voice actions do not. This document records the plan and the decisions.
|
||||
Voice state reads and the basic actions exist. Auto-leave, audio, and video do not. This document
|
||||
records the state and the plan.
|
||||
|
||||
## Current
|
||||
|
||||
@@ -9,31 +10,145 @@ Voice state reads exist. Voice actions do not. This document records the plan an
|
||||
| `voice.get` | Implemented. One user's voice channel and its members. Without `guildId` it searches all guilds. |
|
||||
| `voice.overview` | Implemented. Occupied voice channels and members for one guild. |
|
||||
| `VOICE_STATE_UPDATE` events | Normalized and pushed. Mock-verified. Live check pending. |
|
||||
| Join and leave | Not started. |
|
||||
| Audio in and out | Not started. |
|
||||
| `voice.join` | Implemented. Joins muted. Internal lookups verified live. |
|
||||
| `voice.leave` | Implemented. Disconnects the client. |
|
||||
| `voice.move` | Implemented. Moves to another channel. |
|
||||
| Auto-leave | Not started. |
|
||||
| `stay @user` | Not started. |
|
||||
| Capture and transcript | Implemented. VAD segments, WAV per utterance, JSONL store, live events. |
|
||||
| Speech to text | Hook implemented (local command or HTTP API); provider not chosen yet. |
|
||||
| Audio in and out | Output and capture work. Browser mic puppeting is not started. |
|
||||
| Video | Not started. |
|
||||
|
||||
The core reads `VoiceStateStore` only. It never joins a call on its own.
|
||||
|
||||
## Planned actions
|
||||
## Actions
|
||||
|
||||
| Action | Parameters | Result |
|
||||
| --- | --- | --- |
|
||||
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true }` |
|
||||
| `voice.join` | `channelId`, optional `mute` (default true) | `{ channelId, joined: true, muted }` |
|
||||
| `voice.leave` | none | `{ left: true }` |
|
||||
| `voice.move` | `channelId` | `{ channelId, moved: true }` |
|
||||
| `voice.move` | `channelId` | `{ channelId, joined: true }` |
|
||||
|
||||
Rules:
|
||||
|
||||
- The channel must be a voice channel. The core validates the snowflake and the channel type.
|
||||
- Join joins the call muted. The bot does not capture or send audio.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue, but
|
||||
the daemon rate limits them separately.
|
||||
- The join implementation goes through Discord's internal channel action, located at runtime. The
|
||||
lookup is unverified until the first live run. This is the same pattern as `reaction.add`.
|
||||
- Voice actions are state changes, not send floods. They stay out of the message pacing queue. The
|
||||
daemon spaces them 250-750 ms (`pacedVoice`).
|
||||
- The join goes through Discord's internal channel action, located at runtime.
|
||||
`selectVoiceChannel` and `setSelfMute` were verified live on 2026-09-13. The end-to-end join was
|
||||
not run.
|
||||
|
||||
The plan for later audio: the bot unmutes only while it plays an audio stream, then mutes again.
|
||||
Audio playback is a later feature.
|
||||
|
||||
## Audio
|
||||
|
||||
Voice audio uses the system audio stack, not Discord internals. The daemon owns the wiring.
|
||||
|
||||
### Devices
|
||||
|
||||
| Device | Kind | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `veskslop_tts` | null sink | Playback target for TTS |
|
||||
| `veskslop_mic` | remap source of `veskslop_tts.monitor` | The client microphone |
|
||||
| `veskslop_capture` | null sink | Client output, recorded by the daemon |
|
||||
|
||||
The host needs a sound server. On Debian and Ubuntu:
|
||||
|
||||
```sh
|
||||
apt install pulseaudio pulseaudio-utils ffmpeg espeak-ng
|
||||
```
|
||||
|
||||
The daemon starts `pulseaudio` automatically when `pactl` cannot reach a server, unless
|
||||
`VESKSLOP_AUDIO_MANAGE_SERVER=0`. If `pactl` is missing, the status reports the install hint. The
|
||||
Web UI Audio section has a **Connect devices** button that rechecks the client device list.
|
||||
|
||||
On startup the daemon creates missing devices with `pactl` and unloads the modules it created on
|
||||
exit. When the client connects, the daemon finds the devices by label and sets them through
|
||||
`voice.setDevices`, unless `VESKSLOP_AUDIO_AUTOCONNECT=0`. Names are configurable with
|
||||
`VESKSLOP_AUDIO_SINK`, `VESKSLOP_AUDIO_SOURCE`, and `VESKSLOP_AUDIO_CAPTURE_SINK`.
|
||||
|
||||
If the stack is unavailable, playback/speak calls return the reason and the rest of the daemon
|
||||
works as before.
|
||||
|
||||
### Playback
|
||||
|
||||
- TTS has two providers:
|
||||
- **local**: `VESKSLOP_TTS_CMD` with `{out}` and `{text_file}` placeholders. Example:
|
||||
`piper --model sv_SE-nst-medium.onnx --output_file {out} -f {text_file}`.
|
||||
- **api**: `VESKSLOP_TTS_API_URL` receives a JSON body `{ text, voice? }` and returns audio
|
||||
bytes. `VESKSLOP_TTS_API_KEY` adds a bearer token.
|
||||
- `ffplay` decodes and plays into `veskslop_tts` (`PULSE_SINK`). `VESKSLOP_FFPLAY` overrides the
|
||||
binary. Lines play one at a time in order.
|
||||
|
||||
### Browser monitor (Client audio)
|
||||
|
||||
The header **Client audio** button streams the call to the browser:
|
||||
|
||||
- The daemon reads `veskslop_capture.monitor` with `ffmpeg` and sends raw 48 kHz mono PCM over
|
||||
the WebSocket at `/audio`.
|
||||
- The browser plays the PCM with the Web Audio API and can send it to a chosen output device.
|
||||
- This is separate from the bot's microphone and TTS. Nothing the browser plays affects the call.
|
||||
|
||||
Browser microphone puppeting (talking through the bot) is not implemented. It needs a mix sink so
|
||||
the browser mic and TTS share `veskslop_mic`, a `pacat` input, and echo handling.
|
||||
|
||||
### Ducking
|
||||
|
||||
When another person speaks, the daemon lowers the `veskslop_tts` sink volume to
|
||||
`VESKSLOP_AUDIO_DUCK` percent (default 30) and restores it when they stop. Speaking comes from the
|
||||
`VOICE_SPEAKING` event. Staying quiet while someone else talks is a queue property, not a switch.
|
||||
Yield (stop mid-line) is not implemented.
|
||||
|
||||
### Capture and transcript
|
||||
|
||||
The daemon records the client output and splits it into utterances.
|
||||
|
||||
1. `ffmpeg` reads `veskslop_capture.monitor` as 16 kHz mono PCM (`VESKSLOP_FFMPEG` overrides).
|
||||
2. The VAD opens a segment after `VESKSLOP_VAD_MIN_SPEECH_MS` above `VESKSLOP_VAD_THRESHOLD_DB`,
|
||||
keeps `VESKSLOP_VAD_PRE_ROLL_MS` of audio from before, and closes after
|
||||
`VESKSLOP_VAD_SILENCE_MS` of quiet.
|
||||
3. Each segment becomes a WAV under `<cache>/audio/captures/` and a transcript entry in
|
||||
`VESKSLOP_TRANSCRIPT` (default `<state dir>/transcripts.jsonl`).
|
||||
4. When `VOICE_SPEAKING` covers the segment, the dominant speaker is attached.
|
||||
5. When STT is configured, the segment is transcribed and the text is attached.
|
||||
|
||||
STT providers:
|
||||
|
||||
- **Vosk (default)**: CPU, small and light. Install it and drop a model in `poc/models/`
|
||||
(auto-detected), or set `VESKSLOP_VOSK_MODEL`:
|
||||
```sh
|
||||
pip install vosk
|
||||
curl -LO https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip
|
||||
unzip vosk-model-small-en-us-0.15.zip -d poc/models
|
||||
```
|
||||
The daemon runs `poc/scripts/stt-vosk.py` for each segment. Quality is robotic but light.
|
||||
- **local command**: `VESKSLOP_STT_CMD` with `{wav}`; the command prints text on stdout.
|
||||
- **api**: `VESKSLOP_STT_API_URL` receives WAV bytes and returns text or JSON `{ text }`.
|
||||
`VESKSLOP_STT_API_KEY` adds a bearer token.
|
||||
|
||||
TTS engine and voice are chosen in Settings → Client → Speech (stored in the daemon preferences).
|
||||
The local engine defaults to `espeak-ng` when installed; the voice list comes from
|
||||
`espeak-ng --voices`.
|
||||
|
||||
Start and stop with `POST /api/audio/capture`, the Audio settings button, or the MCP tools. Set
|
||||
`VESKSLOP_CAPTURE_AUTO=1` to start capture when the bot joins a call and stop it on leave.
|
||||
|
||||
Transcript events go over the SSE stream: `TRANSCRIPT_SESSION`, `TRANSCRIPT_SEGMENT`,
|
||||
`TRANSCRIPT_TEXT`, `TRANSCRIPT_SESSION_END`, and `TRANSCRIPT_ERROR`. The agent console's
|
||||
Transcript tab shows the live list; `GET /api/transcript` and the `get_transcript` MCP tool read
|
||||
the store. Segments without text still carry the WAV path, so an LLM can transcribe them later.
|
||||
|
||||
### Controls
|
||||
|
||||
- Chat: `speak <text>` and `speak stop` from a trusted user.
|
||||
- MCP: `speak`, `stop_audio`, `get_audio_status`, `start_capture`, `stop_capture`,
|
||||
`list_audio_devices`, `set_audio_devices`, `get_transcript`.
|
||||
- HTTP: `/api/audio`, `/api/audio/speak`, `/api/audio/stop`, `/api/audio/capture`,
|
||||
`/api/audio/devices`.
|
||||
- Web UI: the TTS panel (provider, Speak, Stop) and the Audio settings section.
|
||||
|
||||
|
||||
## Lifecycle
|
||||
|
||||
@@ -60,8 +175,8 @@ when the puppet cannot see a user's voice channel or the channel members. It sta
|
||||
|
||||
A live run must confirm these points.
|
||||
|
||||
1. The internal `selectVoiceChannel` lookup exists and joins the given channel.
|
||||
2. The internal disconnect action leaves the call.
|
||||
1. `voice.join` joins the given channel end to end. (The `selectVoiceChannel` lookup is verified.)
|
||||
2. `voice.leave` disconnects the call.
|
||||
3. `VOICE_STATE_UPDATE` carries the fields that the core normalizes.
|
||||
4. The join is muted and Discord shows the mute state.
|
||||
5. The auto-leave timer works across a client restart. The timer is daemon state, not core state.
|
||||
|
||||
+81
-15
@@ -32,6 +32,19 @@ only. Do not expose the port to the public internet. See [daemon.md](daemon.md).
|
||||
|
||||
The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
|
||||
|
||||
## TLS
|
||||
|
||||
The UI serves HTTPS with a self-signed certificate so browser media APIs have a secure context.
|
||||
The daemon generates the certificate with `openssl` on first run (localhost plus the host's LAN
|
||||
addresses) and stores it under `<state dir>/tls/`. The browser shows a warning once; accept it.
|
||||
|
||||
| Setting | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `WEBUI_TLS` | `1` | Set to `0` to serve plain HTTP. |
|
||||
| `WEBUI_TLS_KEY` / `WEBUI_TLS_CERT` | unset | Use an existing certificate instead. |
|
||||
|
||||
With plain HTTP, `navigator.mediaDevices` only exists on localhost; remote browsers must use HTTPS.
|
||||
|
||||
## Page
|
||||
|
||||
- Plain HTML, CSS, and JavaScript in `poc/public/`. No build step and no frontend framework.
|
||||
@@ -39,12 +52,21 @@ The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
|
||||
in `poc/WebUIMockup.zip`. Implemented screen: **7a**, the docked three-column debug shell.
|
||||
- Layout: 50px header, conditional call strip, `236px | 1fr | 306px` body, and the agent console
|
||||
bar (35px collapsed, resizable drawer expanded).
|
||||
- Header: wordmark, guild picker, health indicators, Takeover switch, Settings.
|
||||
- Left rail: guild switcher, text channels, voice channels with inline participants, direct
|
||||
messages, and the account footer.
|
||||
- Center: channel header, message list, composer, and the TTS panel.
|
||||
- Right column: In voice, Favourites, Open DMs & groups.
|
||||
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event.
|
||||
- Header: wordmark, **Guild / DM** scope tabs, guild picker, health indicators, **Client audio**,
|
||||
Takeover switch, Settings. While the bridge is down a **Start server** button appears with a
|
||||
pulsing bloom; if the whole daemon is unreachable it becomes **Refresh** and reloads the page.
|
||||
The body dims and stops taking input while the bridge is down.
|
||||
- Left rail (Guild tab): guild switcher, text channels, voice channels with inline participants,
|
||||
and **People** — users recently seen in a voice channel (click opens their cached DM). Clicking a
|
||||
voice channel joins it (muted); clicking the active channel leaves.
|
||||
- Left rail (DM tab): every cached DM and group with unread badge, avatar, name, last-message time,
|
||||
and a favourite star.
|
||||
- Right column: **In voice** (favourited users in a call, grouped by call), **Favourites**, and
|
||||
**Open DMs & groups** as quick access. The guild's own occupied channels live in the left rail.
|
||||
- Speaking users show animated level bars and a bold name (design 2c: audio → video → share marks).
|
||||
Nicknames come from the guild member store; DMs fall back to the username.
|
||||
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event. The
|
||||
Transcript tab lists captured segments (time, speaker, text, WAV) and updates live.
|
||||
- Opening a chat fetches the latest 50 messages from Discord into the client cache. The refresh
|
||||
button by the channel header re-fetches. Live message events read the cache.
|
||||
- Attachments render from their Discord CDN links. Images display inline; other files are link
|
||||
@@ -53,9 +75,23 @@ The MCP server stays on `127.0.0.1:8788`. Only the UI binds outside localhost.
|
||||
- Composer and TTS drafts persist in `localStorage` per channel. They survive re-renders, the
|
||||
bridge connecting, and a page reload. A successful send clears the channel draft.
|
||||
- When the bridge connects after the page loaded, the page reloads state without touching drafts.
|
||||
The reload retries while the client boots, and the health poll keeps retrying if the first
|
||||
attempt failed.
|
||||
- Roadmap gaps are shown, not hidden: TTS, transcript, tool-call tracing, favourites, and voice
|
||||
join/monitor appear as marked placeholders.
|
||||
|
||||
## DMs and favourites
|
||||
|
||||
- The **DM** scope tab shows all cached DMs in the rail and the conversation in the centre.
|
||||
- Unread badges sit left of the avatar: purple for unread, orange for mentions while Takeover is
|
||||
on; while Takeover is off they are grey, meaning the bot saw them and you did not.
|
||||
- The hollow star on a row adds it to favourites, stored in the daemon preferences. Starring a 1:1
|
||||
DM favourites that user; starring a group DM favourites the channel.
|
||||
- Favourited users get presence tracking. If they are in a call, they appear under **In voice**,
|
||||
grouped per call. The right column's **Favourites** shows their status when they are not.
|
||||
- The last-message time comes from the channel's last message snowflake: a time today, `yesterday`,
|
||||
or a short date.
|
||||
|
||||
## Takeover
|
||||
|
||||
Takeover is a read-only switch. Off, the composer is locked (a `read-only` tag sits next to the
|
||||
@@ -65,13 +101,30 @@ only; the API itself does not enforce it.
|
||||
|
||||
## Settings
|
||||
|
||||
The Settings button opens a slide-over with the server-side controls:
|
||||
The Settings button opens a slide-over:
|
||||
|
||||
- **Core process** — state, last exit, output, Start / Stop / Restart.
|
||||
- **Account login** — token file state, QR login, captcha.
|
||||
- **Visible guilds** — checkboxes for the guild switcher and picker. Stored in `localStorage`;
|
||||
"Show all" clears the filter.
|
||||
- **Action log** — the persistent JSONL log, newest first.
|
||||
- **Visible guilds** — top level. Checkboxes filter the guild switcher and picker live. **Save**
|
||||
writes the selection to the daemon preferences (`/api/preferences`); **Hide all** and **Show all**
|
||||
select none or every guild. All visible is stored as `null`, none as `[]`. All browsers share
|
||||
the saved state, and Save works while the client is disconnected.
|
||||
- **Client** — a collapsible group with:
|
||||
- **Bot audio** — engine status, TTS provider state, a **Connect devices** button, and the
|
||||
capture toggle. These are the daemon host's devices.
|
||||
- **Speech** — the TTS engine (Auto / Local / API), language, and voice. Voices are grouped by
|
||||
base language and filtered by the chosen language; English, Swedish, German, Danish,
|
||||
Norwegian, Russian, and Cherokee are pinned on top. Stored in the daemon preferences.
|
||||
- **Display** — the timestamp timezone. Defaults to the browser's zone
|
||||
(`Intl.DateTimeFormat().resolvedOptions().timeZone`); pick any IANA zone to override. Stored in
|
||||
`localStorage` and applied to messages, the transcript, the event log, the action log, and the
|
||||
core status.
|
||||
- **Browser audio** — puppeting controls: the browser output device for Client audio, a
|
||||
placeholder mic select, and the call-audio state. **Enable browser audio** asks for the
|
||||
microphone permission, which browsers require before they list output devices. Separate from
|
||||
the bot's devices.
|
||||
- **Server** — a collapsible group with:
|
||||
- **Core process** — state, last exit, output, Start / Stop / Restart.
|
||||
- **Account login** — token file state, QR login, captcha.
|
||||
- **Action log** — the persistent JSONL log, newest first.
|
||||
|
||||
## Core process
|
||||
|
||||
@@ -126,8 +179,21 @@ replaceable.
|
||||
| GET | `/api/dms` | `limit?` | Cached DM and group DM channels with recipients |
|
||||
| GET | `/api/voice` | `userId?`, `guildId?` | One user's voice state |
|
||||
| GET | `/api/voice/overview` | `guildId` | Occupied voice channels and members |
|
||||
| POST | `/api/voice/join` | `channelId` | Reserved. Returns `501` |
|
||||
| POST | `/api/voice/leave` | none | Reserved. Returns `501` |
|
||||
| POST | `/api/voice/join` | `channelId`, `mute?` | Join a voice channel (muted by default) |
|
||||
| POST | `/api/voice/leave` | none | Leave the current voice channel |
|
||||
| POST | `/api/voice/move` | `channelId` | Move to another voice channel |
|
||||
| GET | `/api/preferences` | none | Web UI preferences (`visibleGuilds`, `tts`) |
|
||||
| POST | `/api/preferences` | `visibleGuilds?`, `tts?` | Update preferences. Stored in `<state dir>/preferences.json`. |
|
||||
| GET | `/api/audio` | none | Audio engine, TTS, queue, ducking, capture state |
|
||||
| GET | `/api/audio/devices` | none | The client's input and output devices |
|
||||
| GET | `/api/audio/tts/voices` | none | Local TTS voices (espeak-ng) |
|
||||
| POST | `/api/audio/connect` | none | Re-apply the virtual devices to the client |
|
||||
| WS | `/audio` | `{"type":"listen","on":bool}` | Raw 48 kHz mono PCM of the call |
|
||||
| POST | `/api/audio/devices` | `inputDeviceId?`, `outputDeviceId?` | Set the client's devices |
|
||||
| POST | `/api/audio/speak` | `text`, `provider?`, `voice?` | Queue text to speech |
|
||||
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
||||
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
||||
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
||||
| GET | `/api/logs` | `limit` | The persistent action log |
|
||||
| GET | `/api/events` | none | Server-sent events |
|
||||
|
||||
@@ -140,7 +206,7 @@ gap. See [daemon.md](daemon.md).
|
||||
## Not started
|
||||
|
||||
- Web UI authentication and user accounts.
|
||||
- Voice join and leave. The UI elements exist and call the reserved routes.
|
||||
- Voice auto-leave and the `stay @user` command.
|
||||
- TTS playback, transcript capture, and MCP tool-call tracing.
|
||||
- Favourites, presence, member counts, and message editing/deletion/reactions.
|
||||
- Pagination of channel history.
|
||||
|
||||
Reference in New Issue
Block a user