zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the agent scaffold, mobile web UI, voice auto-leave, and text notifications

Browse Source
This commit is contained in:
Fisto
2026-09-14 16:38:14 +02:00
parent c7aad7547f
commit eab94132bb
27 changed files with 1521 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/INDEX.md
+1
View File
@@ -11,6 +11,7 @@ This file lists the project documents.
| [daemon.md](daemon.md) | The external MCP daemon |
| [webui.md](webui.md) | The web control UI and its JSON API |
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
| [agent.md](agent.md) | The local LLM harness and remote MCP agent access |
| [communication.md](communication.md) | Trust model, commands, and the trigger log |
| [auth.md](auth.md) | QR login and token persistence |
| [testing.md](testing.md) | Tests and the live test setup |
docs/agent.md
+91
View File
@@ -0,0 +1,91 @@
# The Veskslop agent
The agent is the local LLM harness. It lets a remote `llama.cpp` instance act through the daemon.
It is separate from the core and from the daemon, and it is optional.
## Two consumers
1. **The local harness (veskslop agent).** A small Go process on the same host as the daemon. It
talks to a remote OpenAI-compatible `llama.cpp` endpoint and to the daemon over MCP. It owns
free-form messages. It is the only consumer that runs an LLM loop.
2. **Remote agents.** External systems connect to the daemon's MCP endpoint and use it as a sensor
and notifier. Example: a task runner wants to reboot a switch, calls `get_voice_state` for the
owner, sees the owner is in a call, and reschedules. Remote agents do not run an LLM here and do
not receive every message. They query state and send notifications.
## Routing: commands versus free-form
The daemon keeps the fixed commands (`whitelist`, `speak`) and the `poll` parser. A trusted
free-form message becomes a task for the local harness. Commands never go to the LLM; this avoids
double replies and keeps privileged actions out of model reach.
## Local harness design
| Piece | Choice |
| --- | --- |
| Language | Go |
| Agent framework | Eino (`github.com/cloudwego/eino`), ReAct agent |
| Model | `eino-ext/components/model/openai` against a remote `llama.cpp` `/v1` endpoint |
| Tools | `eino-ext/components/tool/mcp/officialmcp` over the official MCP Go SDK, Streamable HTTP |
| Tool surface | The daemon's MCP tools. Nothing else. |
| Entry | `agent/` |
| Kill switch | `VESKSLOP_AGENT_ENABLED=1` is required. Default is off. |
| Auth | `VESKSLOP_AGENT_TOKEN` bearer token on the task endpoint. |
| Budget | `VESKSLOP_AGENT_MAX_STEPS` (default 16). Per-task wall clock: 5 minutes. |
The harness receives tasks on `POST /task`. The daemon will push them (not implemented yet; the
scaffold accepts manual pushes). The task body:
```json
{
"id": "trigger-id",
"guildId": "…",
"channelId": "…",
"author": "…",
"content": "the trigger text",
"sensitive": false,
"context": [{ "author": "…", "content": "…", "timestamp": "…" }]
}
```
The model replies through the daemon's own send tools, so pacing, the rate limit, and the action
log apply as for any other sender. The harness never calls Discord directly.
## Safety rules
- Tool results and message context are untrusted. The system prompt says so explicitly.
- The harness is stateless per task. Context comes from the daemon each time.
- The model cannot run fixed commands. It only has MCP tools.
- Planned, not implemented: a tool allowlist, per-conversation serialization with
cancel-and-replace, delivery IDs for dedupe, and `RATE_LIMITED` backoff beyond one retry.
## Remote agent path
The daemon already exposes its tools over MCP (Streamable HTTP). Remote clients can:
- read state (`get_voice_state`, `get_voice_overview`, `get_user_status`, `list_dms`, …)
- notify (`send_dm`, `send_message`, `speak`)
Planned work: a small notification tool set for external systems (`notify_owner`), a documented
LAN or Tailscale bind, and optional shared-token auth on the MCP listener. The listener currently
binds localhost.
## Status
Scaffold implemented in `agent/` and smoke-tested on 2026-09-14: it loaded 28 tools from the
daemon, ran one ReAct tool round against a fake OpenAI-compatible endpoint, and produced the final
message. Daemon push, tests, and the remote notification tools are pending.
## Run the scaffold
```sh
cd agent
VESKSLOP_AGENT_ENABLED=1 \
VESKSLOP_AGENT_LLAMA_URL=http://llama-host:8080/v1 \
VESKSLOP_AGENT_MODEL=your-model \
VESKSLOP_AGENT_MCP_URL=http://127.0.0.1:8788/mcp \
go run . --task "Who is in the call right now?"
```
Build a binary with `go build -o veskstop-agent .`. The Go toolchain is not part of the root
`pnpm` workspace; run `go build` and `go vet` in `agent/` directly.
docs/architecture.md
+9 -3
View File
@@ -7,13 +7,15 @@ real Vesktop client, so the traffic comes from the official client.
## Parts
The system has five parts:
The system has six parts:
1. **Vesktop core** — the real Vesktop client. It runs headless. It exposes a small capability layer.
2. **Core bridge** — the capability layer. It accepts validated actions and pushes events.
3. **Veskslop daemon** — the external process. It holds all logic. It exposes MCP.
4. **Web UI** — a small browser control surface served by the daemon. See [webui.md](webui.md).
5. **MCP clients** — agents and tools. They call the daemon.
5. **Veskslop agent** — the optional local Go harness. It runs an LLM tool loop against a remote
`llama.cpp` and the daemon's MCP tools. See [agent.md](agent.md).
6. **MCP clients** — remote agents and tools. They call the daemon.
```
MCP clients (agents) Web browser (owner)
@@ -62,7 +64,11 @@ when the action list changes.
| Web UI and JSON API | `poc/src/webui.ts`, `poc/public/` | Nocturne 7a debug shell + roadmap placeholders |
| User search | `src/renderer/puppet/actions.ts` (`user.search`) | Implemented |
| Voice state reads | `src/renderer/puppet/actions.ts`, `src/renderer/puppet/events.ts` | Implemented |
| Voice actions, audio, video | Not started | Not started |
| Voice actions | `src/renderer/puppet/actions.ts`, `poc/src/puppet.ts` | Implemented, mock-verified |
| Voice audio (TTS, capture, STT) | `poc/src/audio*.ts`, `poc/src/tts.ts`, `poc/src/capture.ts`, `poc/src/stt.ts` | Implemented, verified live |
| Voice video | — | Not started |
| Local agent harness | `agent/` | Scaffold implemented and smoke-tested. Daemon push pending. |
| Mobile web UI | `poc/public/` | First draft at 390px with four tabs |
## Data flow
docs/communication.md
+4
View File
@@ -11,6 +11,7 @@ A message is a trigger when both conditions are true:
- One of these conditions is true:
- The message is a one-to-one DM to the puppet.
- The message mentions the puppet in a guild channel or a group DM.
- The message replies to a message the puppet sent.
All other messages are ignored. Untrusted messages do not enter the log at this time. Bots and the
puppet's own messages are ignored.
@@ -181,6 +182,9 @@ event. The log rotates at the size limit.
| `reply` | Command, mode, target, result, error |
| `voice` | Voice action, channel, mute |
| `audio` | Audio action (speak, stop, devices, capture), provider, size |
| `message-edit` | Message ID, channel, author, content length, whether the cache had it |
| `message-delete` | Message ID, channel, whether the cache had it |
| `reaction` | Add or remove, message, channel, user, emoji (trusted users only) |
| `manual` | Web UI send: mode, channel or user, content length |
| `error` | Scope and message |
docs/core-bridge.md
+5 -2
View File
@@ -67,7 +67,7 @@ Event line:
| `user.search` | `query`, optional `guildId`, optional `limit` | Users from the client cache. Guild members when `guildId` is set. |
| `dm.list` | optional `limit` | Cached DM and group DM channels with recipients, icons, and avatars. Cache-only. |
| `message.send` | `channelId`, `content` | `{ channelId, messageId, sent: true }` |
| `message.read` | `channelId`, `limit` | `{ cached, messages }`. Cache-only. Messages include the author avatar and attachment CDN links. |
| `message.read` | `channelId`, `limit` | `{ cached, messages }`. Cache-only. Messages include the author avatar, attachment CDN links, `editedAt`, and `replyTo`. |
| `message.fetch` | `channelId`, `limit`, optional `before` or `after` | `{ fetched, messages }`. Fetches from Discord. Same message shape. |
| `reaction.add` | `channelId`, `messageId`, `emoji` | `{ added: true }` |
| `dm.send` | `userId`, `content` | `{ userId, channelId, sent: true }` |
@@ -78,7 +78,10 @@ The renderer normalizes client events and pushes them to subscribers.
| Event | Payload |
| --- | --- |
| `MESSAGE_CREATE` | Message ID, channel, guild, channel type, author, content, mentions, timestamp |
| `MESSAGE_CREATE` | Message ID, channel, guild, channel type, author, content, mentions, `replyTo`, `editedAt`, timestamp |
| `MESSAGE_UPDATE` | Message ID, channel, guild, author, content, `editedAt` |
| `MESSAGE_DELETE` | Message ID, channel, guild |
| `MESSAGE_DELETE_BULK` | Message IDs, channel, guild |
| `MESSAGE_REACTION_ADD` | Message ID, channel, guild, user, emoji |
| `MESSAGE_REACTION_REMOVE` | Same fields as add |
| `VOICE_STATE_UPDATE` | User, channel, guild, session, mute, deaf, self-mute, self-deaf |
docs/voice.md
+6 -9
View File
@@ -154,16 +154,13 @@ the store. Segments without text still carry the WAV path, so an LLM can transcr
The bot leaves a call on its own in two cases.
1. **Owner leaves.** When the owner leaves the call, the bot waits `voiceLeaveDelaySeconds` and
then leaves. A rejoin inside the window cancels the leave.
2. **A stay target leaves.** The `stay @user` command pins the bot to a user. When that user
leaves, the bot waits the same delay and then leaves.
1. **Owner leaves.** When the owner leaves the call, the bot waits `voiceLeaveDelaySeconds` (default
30, `VESKSLOP_VOICE_LEAVE_DELAY` sets the default, `set_puppet_config` changes it) and then
leaves. A rejoin inside the window cancels the leave. The owner is zzib for now.
2. **A stay target leaves.** The `stay @user` command pins the bot to a user. Not implemented yet;
the owner rule is the only one active.
`stay @user` needs a chat command. Chat commands are paused at this time, so the command waits for
the voice backend. The default delay is proposed as 30 seconds and lives in the config.
The bot never leaves while the owner or the stay target is present. A manual `voice.leave` wins
over both rules.
The bot never leaves while the owner is present. A manual `voice.leave` wins over both rules.
## Collector account
docs/webui.md
+22 -1
View File
@@ -68,7 +68,8 @@ With plain HTTP, `navigator.mediaDevices` only exists on localhost; remote brows
- Console: Tool calls, Events (live SSE), Transcript. The ticker shows the newest event. The
Transcript tab lists captured segments (time, speaker, text, WAV) and updates live.
- Opening a chat fetches the latest 50 messages from Discord into the client cache. The refresh
button by the channel header re-fetches. Live message events read the cache.
button by the channel header re-fetches. Live message events read the cache. Edited messages show
an `edited` marker and replies show `↳ replying to <name>`.
- Attachments render from their Discord CDN links. Images display inline; other files are link
chips. The daemon never downloads or stores attachment bytes.
- Avatars and guild icons load from the Discord CDN with a monogram fallback.
@@ -80,6 +81,26 @@ With plain HTTP, `navigator.mediaDevices` only exists on localhost; remote brows
- Roadmap gaps are shown, not hidden: TTS, transcript, tool-call tracing, favourites, and voice
join/monitor appear as marked placeholders.
## Mobile
At 768px and below the shell becomes the 1c phone layout. One pane shows at a time behind a
four-tab bar (Users, Chat, Call, Activity), with 44px hit targets and the call bar pinned above
the content.
- **Users** is the left rail: guild switcher, channels, voice, and people.
- **Chat** is the centre column: messages, composer, TTS panel.
- **Call** is the live call: the whole right column, with the current call's members listed first
(not only favourites).
- **Activity** is the full agent console (Tool calls, Events, Transcript).
- The Chat tab carries the total DM unread count. The header compresses to one line: scope tabs,
guild picker, health dot and event rate, icon-only Client audio and Settings, and the Takeover
switch.
- The selected tab persists in `localStorage`. Picking a channel from Users switches to Chat;
picking a voice channel switches to Call.
The first draft renders headless at 390x844. Remaining gaps against 1c: the transcript and the TTS
box live in Chat, not Call, and the connection stats are not chips yet.
## DMs and favourites
- The **DM** scope tab shows all cached DMs in the rail and the conversation in the centre.