Add the agent scaffold, mobile web UI, voice auto-leave, and text notifications

This commit is contained in:
Fisto
2026-09-14 16:38:14 +02:00
parent c7aad7547f
commit eab94132bb
27 changed files with 1521 additions and 73 deletions
+37 -15
View File
@@ -1,21 +1,29 @@
# CURRENT.md — session state
Last updated: 2026-09-13
Last updated: 2026-09-14
This file tracks the work state. Read it first. Update it before you stop.
## Current focus (broad)
The text layer is done. The web UI follows the Nocturne design handoff. Voice actions are in
progress: join/leave/move work; auto-leave and `stay @user` are next. Live verification and a full
QR approval are pending. The LLM attachment and MCP event delivery are parked.
The text layer is done. Voice actions and audio work. The local LLM harness (Veskslop agent, Go +
Eino) is scaffolded and smoke-tested; the daemon-side task push is next. The web UI has a first
mobile draft at 390px.
## In progress
- Voice actions are started: `voice.join` (muted), `voice.leave`, and `voice.move` are implemented
in the core, wired through the daemon, Web UI, and MCP, and mock-verified. The internal
`selectVoiceChannel` and `setSelfMute` lookups were probed live on the real client.
- Auto-leave and `stay @user` are the next voice step.
- The Veskslop agent lives in `agent/` (Go 1.25+, Eino v0.9.19, remote llama.cpp over the
OpenAI-compatible API, daemon tools over MCP Streamable HTTP). It is disabled by default
(`VESKSLOP_AGENT_ENABLED=1`), takes tasks on `POST /task`, and logs every tool call. Smoke-tested
on 2026-09-14: 28 tools loaded, one ReAct round, final message returned. The daemon does not push
tasks to it yet. Plan and env vars: [docs/agent.md](docs/agent.md).
- Remote agents use the daemon's MCP endpoint directly (query state, send notifications). A
notification tool set and a documented LAN/Tailscale bind are planned.
- Mobile web UI first draft: four tabs (Users, Chat, Call, Activity), one pane at a time, 44px
targets, tab badge, call bar pinned, Activity becomes the full console. Verified headless at
390x844. Next: transcript and TTS box in the Call tab, per the 1c design.
- Fixed a pre-existing web UI bug: `loadDms` called the nonexistent `renderDirect`, so the DM rail
only rendered from other paths. It now calls `renderDmRail()`.
- Voice audio is implemented: daemon-managed virtual devices (`veskslop_tts`, `veskslop_mic`,
`veskslop_capture`), TTS providers (local command + HTTP API), queued playback through `ffplay`,
ducking on `VOICE_SPEAKING`, and WAV capture. The daemon starts `pulseaudio` when `pactl` cannot
@@ -111,20 +119,34 @@ QR approval are pending. The LLM attachment and MCP event delivery are parked.
- TTS settings are split into language and voice, grouped by base language with the requested
languages pinned on top.
- Text notifications are parsed: edits update the context cache, deletes drop from it, trusted
reactions are logged, and replies are kept in the request context. A reply to a puppet message
counts as a trigger without a mention. The web UI marks edited messages and replies.
- Voice auto-leave is implemented: when the owner (zzib) leaves the bot's call, the bot leaves
after `voiceLeaveDelaySeconds` (default 30). A rejoin cancels it. Verified with the mock,
including cancellation. `stay @user` is deferred.
- Audio verified end to end on 2026-09-14: TTS played into the virtual mic (recording
`veskslop_mic` peaked at -4.1 dB) and the browser monitor streamed PCM over the TLS WebSocket.
- Next: per-user limits and the collector account. The agent attachment (remote llama.cpp, MCP
transport, event delivery) waits for a design discussion; the draft prompt has open issues to
settle first.
## Next (in order)
1. Voice auto-leave (`voiceLeaveDelaySeconds`) and the `stay @user` command.
2. Live verification: `voice.join`/`voice.leave` end to end, `user.search`, `voice.overview`,
1. Live verification: `voice.join`/`voice.leave` end to end, `user.search`, `voice.overview`,
`VOICE_STATE_UPDATE`, and a full web QR login approval on the real client.
3. Notification parsing: edits, deletes, reactions, and message references.
4. Reply-to-puppet counts as a trigger.
5. Per-user rate limit and command cooldowns.
6. Collector account: read-only modules for voice membership fallback.
2. Agent: daemon task push for free-form triggers (commands stay local), tool allowlist,
per-conversation serialization, and tests against a fake model.
3. Collector account: read-only modules for voice membership fallback.
4. Per-user rate limit and command cooldowns.
5. Mobile web UI: transcript and TTS in the Call tab; verify on a phone.
## Open discussions (need the user)
1. **Web UI auth.** Accepted: no auth, LAN or Tailscale only. Revisit before any wider exposure.
2. **MCP event delivery.** Parked with the LLM attachment. Options stay in `TODO.md`.
2. **MCP event delivery.** Decided for the local harness: the daemon pushes tasks to the agent's
`POST /task`. The transport details (retry, dedupe IDs, queue) are still open. Remote agents
poll or call tools directly; they do not receive pushes.
3. **Untrusted logging.** Untrusted interactions are ignored at this time. Confirm whether to log
untrusted DMs and mentions again.
4. **Privacy pass.** Only token-shaped and `mfa.` strings are redacted, and only in the persistent