zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
515af01e84eef29dded4d0dfd7be7952c6f575b0
Veskslop-t/CURRENT.md
T

554 lines
44 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CURRENT.md — session state
Last updated: 2026-09-16
This file tracks the work state. Read it first. Update it before you stop.
## Current focus (broad)
The text layer is done. Voice actions and audio work. The local LLM harness (Veskslop agent, Go +
Eino) is scaffolded and smoke-tested; the daemon-side task push is next. The web UI has a first
mobile draft at 390px.
## In progress
- Media archive project, branch `media-scrape-project` on `veskslop-t`. Design approved 2026-09-16
(docs: [media.md](docs/media.md), [media-scraper.md](docs/media-scraper.md),
[slopbot.md](docs/slopbot.md), [media-mcp.md](docs/media-mcp.md)). Decisions: scrape the channel
the command is sent in; slopbot (discord.js bot core) for guild channels and the puppet core
bridge for group DMs; one self-contained SQLite + media dir per scrape under `media-scrape/db/`
(gitignored); links indexed by default, downloads optional, third-party embeds never downloaded;
30 GiB default limit with a 90% warning and 5% no-delete overrun, usage bar and clean button in
Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C8. The slopbot scaffold
(config loader, invite helper) and its discord.js dependency were added 2026-09-16. C5 smoke
test: `/api/media/status` and `/api/media` answer on a daemon with no core.
Checkpoints, push after each:
- [x] C1 design docs, AGENTS/INDEX pointers, `.gitignore`
- [x] C2 core message shape: embeds, forwards, attachment metadata
- [x] C3 media-scrape store and puppet walker, tests
- [x] C4 slopbot backend, tests
- [x] C5 daemon wiring: `scrape media` command, MediaService, `/api/media/*`
- [x] C6 Media and Bot tabs in the Web UI
- [x] C7a storage limits, retention, warning/confirm flow, storage and clean API
- [x] C7b Settings group: limit field with info hover, usage bar, clean modal
- [x] C8 MCP service hooks (`scrape_media`, scrape reads, storage, clean; reads in `run_commands`)
- [x] C10 downloader: attachment and sticker bytes, hash-addressed, serial, size-capped, retries;
embeds excluded in SQL; local files served at `/api/media/<scrape>/<item>/file` and preferred
by the Media tab. Live run 2026-09-16: scrape `...-20260916083401` fetched the 784 KB
attachment; storage counted actual bytes.
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
- [x] C46 Sort and exclusion filters. The Advanced drawer gained a sort select (newest, oldest,
user, size, length; global over the merged URL-dedupe stream) and a **Do not include** group
with nested collapsibles: sources (Twitch, YouTube, Discord, Tenor, Giphy, embeds, attachments,
stickers), file types, size (less/greater than with B/KB/MB/GB units), and length (seconds).
Store, `/api/media` + `/api/media/stats` (shared parser), and MCP `query_media` all carry the
new params; size and length ranges drop rows with unknown values, and the ext matching now
coalesces NULL content types so a `NOT (...)` cannot swallow rows. Also hid the media filter bar
and drawer on the Bot tab. Unit tests for every exclusion, all five sorts (nulls last), and
smoke checks for the hidden bar on the Bot tab.
- [x] C45 Streaming media grid. The grid rebuilt every card on each appended page, which made
filtered browsing flash, and the `Load more` trigger only worked once the grid could already
scroll, so short filtered pages dead-stopped at ~11 cards. Now pages append incrementally
(existing images are never rebuilt), a stream buffer capped at 1.5 viewport heights keeps the
scroll alive while the next page loads, auto-fill cascades while the buffer is on screen, the
buffer note reads `Loading more…` / `Scroll for more` / `End of results`, and an elastic nudge
replaces the hard stop at the bottom. Smoke test: first load reaches 120 cards via auto-fill,
scrolling appends 120 → 440 with no broken images.
- [x] C44 One traffic pipeline and split ETAs. Walks and backfill batches now share a
media-traffic lock: a walk holds it for its whole run, a batch for one batch, so the pacing
settings bound total traffic instead of each action. Starting then immediately stopping a
scrape now fetches nothing (tested). Backfill ETAs split into `etaStartMs` (waiting behind
earlier jobs), `etaRunMs` (own work), and `etaTotalMs`; the Bot tab shows only the total and
puts `starts in X · runs for Y` in the tooltip, same for the queue totals. Tests: lock
ordering, stop semantics, ETA ordering.
- [x] C43 Bot-tab backfill control room and media-tab strip rules. The engine gained
`previewBatch`/`rateLimitStatus`/`updateOptions`; a new `BackfillQueue` groups preview work per
chat (paused and resumed scrapes of a chat combine), runs one job at a time in batches, and
reports pending/cached/failed per kind, throughput, measured and settings-based ETA, and the
live cooldown. `MediaService` drives it and exposes `GET /api/media/backfill` plus start/pause;
`/api/preferences` now carries `media.tunables` (Backfilling / Scraping / Timers, persisted and
applied live). The Bot tab shows totals with a Bot/Puppet split, per-chat job cards with
progress and Start/Pause, **Start backfill** / **Pause all**, the red `API timed out, increase
timers` banner, and the three settings groups. The media strip shows running plus one paused
scrape, dismisses a paused one on Stop without stopping it, turns red while limited, and shows
a session-only `N scrapes completed` note. The old `VESKSLOP_MEDIA_*` tuning env vars are gone.
New unit tests cover batching, the cooldown, live retuning, and queue grouping/finish/pause;
the Playwright smoke test covers the Bot tab. Also fixed a real stall the queue exposed:
`pendingPreviews` never returned thumbnail-less embeds, so they sat pending forever and stalled
the job; the store now includes them and the resolver handles Discord CDN unfurls through the
proxy frame, with a content-type guard so the og:image fallback never reads media bytes. Live:
daemon restarted, pending falling (6018 → ~5900), cached rising, dead CDN unfurls closed as 404,
jobs queue smallest-first, settings round-trip verified (21.4 → 37.5 rows/min at 30 s cooldown).
- [x] C42 Preview attempts cap. The restart log showed the backfill retrying dead hosts (Gfycat
DNS is gone, Streamable answers 400) forever as "transient". Rows now carry
`preview_attempts`; after `VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` (3) the row is marked
`unreachable after N attempt(s)` and leaves the pending queue. Unit-tested.
- [x] C41 Restart keeps its logs. A button restart spawned the replacement with `stdio: ignore`,
so anything after the restart vanished. The replacement now appends to `daemon.log` beside the
token file (`VESKSLOP_DAEMON_LOG` overrides); documented in [docs/daemon.md](docs/daemon.md).
- [x] C40 Stale scrape states. Two `general_chut` scrapes showed `running` because the daemons
that owned them were killed mid-job; the state file kept the last written value. On daemon
start the engine now reconciles `running`/`queued` scrapes to `paused`, and History has a
per-scrape **Delete** button (clean with a single ID) next to Resume. Unit-tested.
- [x] C39 File-type filter and the 4chan default. The Advanced drawer gained extension chips
(video mp4/webm/mov/mkv, image png/jpg/gif/webp/avif, audio mp3/ogg/wav/m4a/flac) matched on
URL or content type, plus a default-on **Hide 4chan links (usually dead)** toggle. Backend,
API (`ext`, `excludeHosts`) and MCP support; live: video count 7696 → 7383 with the default
exclusion, `ext=mp4`/`webm` return only those files. Also confirmed the mr-game-memorial group
has 924 attachment videos with 772 previews cached and 0 preview errors — the rest are just
pending in the 2,140-item backfill queue.
- [x] C38 Filter layout swap. The common filters (filetype chips, source kind, server/DM
origin, multi-select People popover) now sit in the always-visible bar; the Advanced search
drawer holds aspect, resolution, channel, date range, cached-only, and the card-size slider.
Apply/Reset stay in the bar. Fixed a dropped `defaultFilters` helper found by the Playwright
smoke test, which passes on the new layout.
- [x] C37 Advanced search drawer, adaptive paging, and zoom. Filters moved into an **Advanced
search** drawer with Apply/Reset (no auto-update): filetype chips, source kind, origin
(server/DM), multi-select People, and an Advanced section (aspect, resolution, channel, dates,
**cachedOnly**). A **Card size** slider scales tiles from 100 px (2/3 of current) to one image
per row; page size = full rows about 1.5 screens tall and follows zoom/resize. Infinite scroll
preloads the next page's JSON and previews. Backend gained `authorIds`, `origin`, `cachedOnly`
on `/api/media`, `/api/media/stats`, and MCP `query_media`. Verified: `origin=dm` and
`cachedOnly=1` return the expected rows.
- [x] C36 Daemon restart, preview diagnostics, browse-time caching, and rate-limit pauses.
Settings → Server has a **Restart daemon** button (`POST /api/daemon/restart`: re-execs the
daemon, keeps the client running). The preview pass logs every failure with scrape/item/kind/
provider/reason plus a per-pass summary, and `GET /api/media/preview-failures` lists them.
Browsing no longer fetches Discord at all: the grid uses cached stills only (placeholders for
missing ones), GIF hover plays the cached animated GIF (kept as `anim_hash`, served from
`/anim`), and streaming only starts on explicit play. `429`s from previews/downloads raise a
cooldown (`VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS`, default 60 s, or `Retry-After`), leaving rows
pending and pausing both walker and previews.
- [x] C35 Walk/preview cycling. A live scrape now alternates: walk 5 pages, preview up to one
batch (25, with the cooldown), walk on. Tiles fill in during a long walk instead of only at the
end, and preview traffic is spread through the walk. Configurable with
`VESKSLOP_MEDIA_WALK_CYCLE_PAGES` (0 defers previews to the end). The final pass still drains
whatever remains.
- [x] C34 Resume from cursor. A paused scrape can continue where it stopped instead of
re-walking the top: `POST /api/media/scrape/resume`, MCP `resume_media_scrape`, and a Resume
button in History. Previews never needed the walk — backfill fetches them straight from the
stored URLs; the walk is only for discovery. Live: the paused `gernal_chut` scrape resumed at
cursor `365256427247239181`, fetched its final page, and entered the preview pass.
- [x] C33 Puppet history pagination. The core's `message.fetch` read the client store, which
keeps only the newest messages plus the channel's first — every page was 10 messages and the
walk stopped after one page (a 10-message, 4-media scrape of an ancient group with thousands
of messages). It now fetches through the client's REST layer first (contiguous 100-message
pages, `transport: "rest"`), store as fallback. Rebuilt and restarted the managed client;
live re-run of channel `591000745164406784`: 21+ pages / 2,100+ messages and climbing, with
hundreds of images/videos being indexed.
- [x] C32 Group-chat clarity and puppet refresh. Group DM scrapes already worked (live-verified
against the running daemon: `zıbb & fred` and a 1:1 DM, backend `puppet`, done, previews
cached); the Group chat tab had shown an empty list while the core was offline. It now says so
explicitly and offers Retry. Expired attachment URLs in DM/group-DM scrapes refresh through
the core bridge (`message.fetch` with `before: id+1`, which returns exactly the target).
- [x] C31 Batched previews. The preview pass now runs in batches (default 25) with a cooldown
(default 60 s), configurable via `VESKSLOP_MEDIA_PREVIEW_BATCH` and
`VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`; batching counts rows, not successes. Live log:
`preview batch of 25 done, pausing 60000 ms`.
- [x] C30 Preview completeness. Audio attachments and voice messages get an ffmpeg waveform
strip; oversized animated GIFs fall back to an ffmpeg-read first frame instead of `too large`;
transient network failures stay pending for the next backfill instead of becoming permanent.
Live: a scrape with 58 audio items backfilled 13 waveforms in 40 s, serving `image/webp`.
Remaining non-cached: Lottie stickers/custom emoji (not indexed as media) and embeds whose
pages refuse bots (marked PRIVATE/GONE by their 403/404).
- [x] C29 Attachment video previews. Discord's image proxy renders a frame for video files
(`?width=320&height=320&format=webp`), so `previewUrlFor` now resizes attachment mp4/webm in
links mode too and `pendingPreviews` includes non-embed videos. Live: paused-scrape videos
backfilled to `image/webp` thumbs, pending queue drained to 0.
- [x] C28 Media grid usability. Infinite scroll (240 px threshold) with a Load more fallback,
`previewError` exposed so tiles show PRIVATE (403) or GONE (404/410) with the reason in the
tooltip, and the scrape strip lists only queued/running jobs — paused scrapes move to History.
- [x] C27 Non-media stream guard. Rows awaiting backfill fell back to first-frame `<video>` with
a provider page URL (YouTube embed, redirectors), and the stream route piped the HTML through
(`text/html` errors). Now only real video files (mp4/webm/m4v) are framed, the stream route
returns `415` for non-media upstreams, the viewer's still-frame is gated the same way, and
"Open in new tab" prefers `play_url`. Live: YouTube item → 415 JSON; backfill reached
184/271 previews during the run.
- [x] C26 YouTube preview root cause. Dedupe picked the newest row per URL; a paused scrape
(`gernal_chut`, stopped mid-walk) had no previews yet and hid an older scrape that did. Fixes:
dedupe prefers the richer row (downloaded > preview > newer), paused walks still run the
download/preview passes for collected pages, daemon start backfills previews for scrapes that
paused early, and preview gaps are now 200-600 ms. Live: backfill cached 24 previews in the
paused scrape within 45 s and a YouTube item then served `thumbUrl`.
- [x] C25 Live scrape progress. The media tab polls `/api/media/scrapes` every 4 s while open,
so the running row counts update; pages show `X/Yp (~Z%)` from `estimatedMessages` when the
search total is available (some channels deny the bot search with Missing Access, so it stays
optional). Live: pages/messages incremented 2→10 during a walk on `gernal_chut`.
- [x] C24 Embeddable items open straight into their player: clicking a YouTube/Vimeo/Twitch/
Streamable card shows the iframe immediately, no extra play press. Non-embeddable videos keep
the still + play icon.
- [x] C23 Grid and viewer source fixes. The grid now prefers the cached still (or local file) for
every kind, and only falls back to a first-frame video, served through the same-origin stream
endpoint instead of hotlinked CDN/redirector URLs (fixes fxtwitter/YouTube previews that ORB
blocked while the cached thumb sat unused). The viewer shows a muted first frame for videos
without a still, and stream refreshes the signed URL only when the stored one fails (4xx). Live:
stream returned 206 video/webm for a thumbnail-less video; YouTube rows keep their cached thumb.
- [x] C22 Media grid polish: per-kind status row
(`(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V`), GIF tiles with a translucent
GIF label that play the animated original on hover (cached previews are static ffmpeg frames),
Tenor oEmbed preview resolution, and first-frame/inline playback for embeds that carry
Discord's `videoUrl` (Tenor/gifv now classify as video with `play_url`). `remotePreviewUrl`
exposes the animated source to the UI. Live: stats per kind correct; Tenor embed stored with
`play_url`.
- [x] C21 Playback of non-downloaded and expired media. `play_url` stores the embed's direct
video (gifv/Tenor mp4); a new `/api/media/<scrape>/<item>/stream` route serves the local file
or re-fetches the source message through slopbot to refresh expired signed CDN URLs and streams
it with Range support. The viewer and **View full** use it, so Discord attachment videos play
without a prior download and old links no longer dead-end. Live: 2.5 MB webm streamed 200 with
a refreshed URL; play_url populated for a media.discordapp.net gifv mp4 and provider
redirectors.
- [x] C20 Cross-scrape dedupe and viewer polish. `queryMedia` and `stats` deduplicate by URL
(newest scrape wins), so overlapping scrapes no longer duplicate cards or counters (live: 17
items / 17 unique). Viewer: big play icon top-right; play embeds a provider player or a direct
mp4/webm/m4v, otherwise opens the external warning; **View full** opens the embedded
media/player in a new tab while the link button opens the original page; duration is stored
(Discord attachment duration, ffprobe for downloads, 0 normalized to unknown); favicon added.
- [x] C19 External embed previews and MIME fixes. Embeds with no thumbnail in Discord's data are
resolved via provider oEmbed/fxtwitter and an `og:image` fallback; preview bytes keep their
real extension (`preview_ext`), so JPEG thumbnails serve as `image/jpeg` instead of mislabeled
WebP (the browser "Content-Type image/webp is not supported" errors). The grid no longer uses
embed page URLs as media, and first-frame `<video>` is limited to playable direct files
(mp4/webm/m4v) with an on-error placeholder. Live: fxtwitter/vxReddit/TikTok previews cached;
7 jpg + 4 webp thumbs served with the right types.
- [x] C18 Video previews and the status row. Uploaded videos have no Discord thumbnails: the
grid renders a first-frame `<video>` for them, and downloads generate a poster frame with
ffmpeg into `thumbs/` (live: both video attachments in the test channel got posters). The
status row is now `(Images/Videos) | Downloaded A/B | Cached C/B (excl Downl) | Loaded L/B`
with B = total for the active filters.
- [x] C17 Media grid and viewer. Embed videos now render their cached thumbnail with a play
icon; a Metadata dropdown picks banner fields (User, Type, Date, Source, Resolution, Length);
nothing shows until Load media; the status row reads `downloaded A/B | cached C/D (excludes
downloaded) | N loaded` from `/api/media/stats`; clicking a tile opens a viewer with
wide-below / vertical-right metadata, full link with an external-site warning, View full, and
an embedded player for YouTube/Vimeo/Twitch/Streamable when possible.
- [x] C16 Message counting and date-bounded scrapes. Discord search (`total_results`, one guild
request, slopbot only) powers `POST /api/media/count` and MCP `count_media_messages`; the
confirm dialog has From/To dates plus a **Check Message Count** button; the walk honors
`afterId`/`beforeId`; `estimatedMessages`, `fromDate`, and `toDate` land in the scrape meta and
History. Live: count 17 all / 17 after 2026-09-16 / 0 before, DM returns a reason, and a
from-dated scrape stored 17 messages with `estimatedMessages: 17`.
- [x] C15 Scrape UX: custom download destination (absolute path per scrape; files there,
previews local, embed thumbnails copied to both, custom files not served), confirmation popup
before Links/Download, New scrape/History right-aligned with New scrape furthest right, modal
X and backdrop close, active-only progress strip, and a Bot error feed with a red unseen badge
that clears on open plus a Clear button. Live run `...-20260916095029`: 6 files and 7 embed
thumbnails in `/tmp/opencode/media-dest/<id>/`, 11 local previews, 0 local files, `localUrl`
null; error record and clear verified over the API.
- [x] C14 Links-mode embed thumbnails, item metadata, filters, and the scrape popup. Metadata:
poster (`author_id`/`author_name`), `posted_at`, `width`/`height`, `aspect_ratio`, with an
`ffprobe` fallback after downloads. Filters: poster, aspect preset (vertical/square/4:3/16:9/
21:9/other), resolution preset (SD/HD/FHD/QHD/4K+/unknown, by shortest side), and a
posted-date range. UI: scrape popup with Server and Group-chat trees (server -> channels, each
with Links/Download), History popup (time, duration, pages/messages, media, bytes, files,
previews), live progress pulse, full-window view. Live run `...-20260916094120`: 11 previews
in links mode (embed thumbnails included), poster zzib, a 9:16 short stored as 0.563, and the
16:9 filter matched.
- [x] C13 Embed thumbnails in download mode: embed rows cache their stored thumbnail URL;
the full third-party media is never fetched. Live run `...-20260916091458` on the test
channel: 6 files downloaded, 11 previews cached (YouTube, Reddit, Tenor, TikTok, X). Also
fixed the state transition: the job stays `running` through download and preview passes so
clients do not see `done` early.
- [x] C12 Media tab filters and full-window layout: kind chips, source filter
(attachment/sticker/embed), source-channel filter, and the view now fills the window below the
header and call strip. API/MCP gained `source` and `channelId` on `query_media`. Smoke-tested
over `/api/media?source=` and `?channelId=`.
- [~] C9 live verification. Slopbot path verified live on 2026-09-16: login Slopbot#5775, guild
`177931418704281600`. Test channel links scrape: 17 messages, 4 images + 13 videos, 0 errors;
download scrape fetched the 784 KB attachment. Large channel `gernal_chut`: 11 pages, 1088
messages, 96 images + 1 GIF + 203 videos, 0 errors (no 429 at the current pacing). Puppet path
verified live on a group DM (`zıbb & fred`) and a 1:1 DM: backend `puppet`, done, 0 errors.
The puppet field check is done too: the group DM returned attachments plus mapped embeds
(YouTube, Instagram), and the large slopbot channel produced YouTube, Gfycat, Imgur, Steam,
and Twitch embeds alongside attachments. Still owed: a stress walk on the second guild
(`559346566754729995`, channel `559346566754729997`).
- Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
open; it carries the character sliders with engine defaults and +/-2 snapping, five named
presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and
test lines (Attention, Laboratory, Critical, Health, Radiation, plus two custom lines in
`localStorage`) that post the current profile straight to `/api/audio/speak`. Fixed the test
buttons reading a stale expression from saved state; they now read the slider like Speak does.
- Root cause of the profile never reaching the engine (2026-09-15): the wrapper wrote the Flextalk
`.reg` with `echo`, and dash's `echo` treats `\2` in `...\Watson\2.0\...` as an escape, which
corrupted the key so `wine reg import` was a silent no-op (it still returns 0). The wrapper now
uses `printf`, validates that every profile value is numeric, and rejects a `.reg` containing
control characters. Documented in TTS-DESIGN.md.
- The Veskslop agent lives in `agent/` (Go 1.25+, Eino v0.9.19, remote llama.cpp over the
OpenAI-compatible API, daemon tools over MCP Streamable HTTP). It is disabled by default
(`VESKSLOP_AGENT_ENABLED=1`), takes tasks on `POST /task`, and logs every tool call. Smoke-tested
on 2026-09-14: 28 tools loaded, one ReAct round, final message returned. The daemon does not push
tasks to it yet. Plan and env vars: [docs/agent.md](docs/agent.md).
- Remote agents use the daemon's MCP endpoint directly (query state, send notifications). A
notification tool set and a documented LAN/Tailscale bind are planned.
- Mobile web UI first draft: four tabs (Users, Chat, Call, Activity), one pane at a time, 44px
targets, tab badge, call bar pinned, Activity becomes the full console. Verified headless at
390x844. Next: transcript and TTS box in the Call tab, per the 1c design.
- Fixed a pre-existing web UI bug: `loadDms` called the nonexistent `renderDirect`, so the DM rail
only rendered from other paths. It now calls `renderDmRail()`.
- Fixed the speaking marks: the core turned Discord's numeric `speaking: 0` into `true`, so the
equalizer bars never stopped. The bars now fade in on start and decay out on stop (500 ms), with
a 2.5 s idle timeout for missed stops; a mute or channel change ends the state at once. Muted
members show no bars.
- Browser audio: **Client audio** now runs the browser unlock in the click (microphone permission,
AudioContext created and resumed, output sink applied) before it starts listening. Verified
headless: 52 PCM frames played through a running context. The button label no longer wipes the
icon markup.
- Capture routing: nothing moved the client's playback streams into `veskslop_capture`, so the
browser monitor streamed digital silence. The audio manager now adopts matching streams
(`VESKSLOP_AUDIO_CLIENT_MATCH`, default `vesktop,electron,chromium,discord`) on device setup,
monitor start, and every 5 s while listening or capturing.
- LAN audio verified end to end on 2026-09-14: a tone stream on `veskslop_tts` was moved to
`veskslop_capture` (daemon log: `moved stream 81 to veskpop_capture`) and the browser, connected
to `https://192.168.0.178:18801/`, played 256+ PCM frames with a peak of 4096 (a stream already
on the capture sink peaked at 29274).
- Speaking bars are always in the layout now: `scaleY(0)` and transparent when idle, they scale up
with the fade-in at start, run the equalizer animation while speaking, and stay on until an
explicit stop, a mute, or a channel change. Live data showed repeated `speaking: true` pulses and
no stop events inside an 8 s window, so the idle decay is removed: bars stick rather than
disappear. The fade runs the equalizer inside a shrinking container, so the amplitude drops
smoothly instead of the block collapsing at full height. The core treats a missing `speaking`
flag as a start pulse and accepts `user_id`. Core-side changes need a rebuild plus a client
restart.
- Added `poc/scripts/audio-doctor.sh` (`pnpm audio:doctor`): default sink, sink inputs, audio
status with adoption counters, a 5 s `volumedetect` on `veskslop_capture.monitor`, and a live
event peek. Run it in a real call to trace a silent browser monitor.
- Live doctor output (2026-09-14): the `vesktop`/`electron` streams sit on `veskslop_capture`
(sink 40) already, `adopted: 0`, and `VOICE_SPEAKING` events arrive. Client audio routing is
therefore in place. Bot TTS works. Capture was off (`capturing: false`), so the transcript was
empty; Vosk transcribes correctly when invoked with the model env.
- Live call audio works now (the user confirmed TTS is audible and the silent call was partly a
per-user volume set to zero for some members, plus another issue). The `volume` read returns
`null` because this client build exposes no `UserVolumeStore`; the setter reports
`NOT_SUPPORTED`. Volume is not a usable control here.
- External TTS engine support (external project, first target is a command wrapper): the contract
lives in `TTS-DESIGN.md`. Veskslop now has nullable `tts.speed` (0.5-2.0) and `tts.pitch`
(-12..+12) preferences, per-line overrides on `/api/audio/speak` and MCP `speak`, `{speed}`/
`{pitch}` placeholders, `VESKSLOP_TTS_SPEED`/`_PITCH` exports, and speed/pitch sliders in the TTS
panel. A custom command advertises its voices through `VESKSLOP_TTS_VOICES`; the espeak list is
hidden while a custom command is set. Verified with a wrapper that logged argv and env:
preferences and overrides both arrive. Reading the engine's `GET /voices` for the HTTP provider
remains future work.
- The MicroslopSam engine project is now part of the tree in `microslopsam/` (copied from the
separate project; its `build/` output and voice data stay ignored). The wrapper was renamed
`setup/vekslop-tts.sh` -> `setup/veskslop-tts.sh` and every reference and configuration export
was fixed. The engine also corrected the Flextalk pitch finding (REG_DWORD bug), so Keith Bell
and Kerry Watson now support pitch and carry no `no-pitch` flag.
- Implemented the Flextalk Expression profile parameter end to end: `VESKSLOP_TTS_VOICES` marks the
two voices as `expression=59` / `expression=100`; the wrapper writes the registry value before
synthesis; Veskslop has `tts.expression` (0-100 nullable), the `{expression}` placeholder,
`VESKSLOP_TTS_EXPRESSION`, an HTTP/MCP field, and an Expression slider that appears only for
those voices (default shown until moved, null leaves the registry untouched). Verified: 0/59/100
produce distinct audio, the registry reaches 0x14 for 20, and the UI shows 25/100/default states.
- Fixed the null-value coercion in the preference normalizers (`Number(null) === 0` turned a
cleared speed/expression into 0.5/0); unset now stays null.
- Renderer-down handling polished: the bridge logs one warning per outage instead of a stack per
polled command, and the `navigator.languages` request catches the new timeout rejection instead
of raising an unhandled promise warning. (The `exitCode: 9` renderer kills in the live log were
client restarts; the daemon's retry handled the startup race as designed.)
- The TTS panel has speed presets (0.5x, 0.75x, 1x, 1.25x, 1.5x) under the fine-adjust slider, and
the pitch control is hidden entirely for voices that report `supportsPitch: false`. (Flextalk
voices no longer report that; see the pitch correction above.) Verified: preset clicks update the
slider/label and save.
- TTS playback has a lead-in of silence before each line (`VESKSLOP_TTS_LEAD_MS`, default 150 ms,
max 2000, added with `adelay` in `ffplay`; status reports `ttsLeadMs`) so Discord's voice
pipeline is settled before the first syllable. Verified on the sink monitor: first sound at
0.02 s with 0 ms and 0.62 s with 600 ms.
- Speech settings have a **Start engine** button that warms the selected engine
(`POST /api/audio/tts/warm`, MCP `start_tts_engine`): it runs one throwaway synthesis so the
command engine's Xvfb/Wine/host are up. Verified with MicroslopSam: cold 1732 ms, warm 505 ms,
and the UI shows "MicroslopSam ready in 835 ms". The warm text is a full sentence because their
two-pass `loudnorm` fails with `inf` on very short input.
- Startup and crash robustness: `sendRendererCommand` now rejects immediately when the client
renderer is destroyed or crashed, and times out after 15 s, so a dead frame no longer logs
Electron's "Render frame was disposed" stack once per poll. Device auto-connect retries up to
10 times every 2.5 s, because the bridge socket is ready before the renderer's modules load
(the live error was `Reflect.get called on non-object` from a lazy lookup at startup). Verified
with the mock: attempt 1 failed, attempt 2 set the devices.
- TTS engines are selectable: the Speech settings and the TTS panel list espeak-ng, the command
engine (`VESKSLOP_TTS_NAME`, default "custom engine"), and the HTTP API; unavailable entries are
disabled and Auto resolves to the command engine first. Selecting an engine switches the voice
list (`/api/audio/tts/voices?engine=`) and hides the language row for single-language engines.
The TTS panel now has a voice select next to the speed/pitch sliders. Fixed a slider bug where a
null speed preference (`Number(null) === 0`) clamped the slider to 0.5 and then persisted it.
- TTS controls: the TTS panel has a working **duck on voice** toggle (stored in preferences,
reported as `duckEnabled`). Ducking was verified live with the mock: the `veskslop_tts` sink
drops to 30% while TTS plays and someone speaks, and returns to 100% after. The Transcript tab
in the console has **Start** and **Stop** buttons; the status line shows `capturing`/`capture
off`. The espeak-ng voice list now includes the `m1`–`m7`, `f1`–`f5`, `croak`, and `whisper`
variants (`<voice>+<variant>`), so each language has about 15 voices instead of one.
- Speaking state is now authoritative: the daemon polls the core's `voice.speaking` action
(`ChannelRTCStore.getSpeakingParticipants`) every second and synthesizes the missing stop/start
events; dispatcher `VOICE_SPEAKING` hints feed the same set so stale users clear. This fixes the
bars sticking for everyone and the transcript attribution, which credited the first member
because speaker intervals never closed. Verified with the mock: a `speaking: true` event cleared
within one poll.
- The client streams sit uncorked, unmuted, and at 100% on `veskslop_capture`, yet the monitor is
silent. Gateway `VOICE_SPEAKING` events do not prove inbound RTP, and the TTS path only proves
outbound, so the remaining suspect is the client's inbound voice state (stuck deafen or a broken
voice connection), not the sink routing. `/health` now returns `selfId` so the bot's own member
flags can be picked out of the voice member list; the doctor prints it.
- Client output is still silent at the sink (`-91 dB`) although the streams are uncorked, unmuted,
and at 100%, and a headless Chromium tone into the same sink measured `-7.4 dB`. Added per-user
volume reading and writing to find Discord-side silencing: `user.volume` / `user.setVolume` in
the core, `/api/voice/volume` (GET and POST), MCP `get_user_volume` / `set_user_volume`, and
`volume` in every voice member. A volume of 0 (or an output volume of 0) silences a user even
when Pulse shows a healthy stream. `voice.join` also clears persisted self-deafen now.
The doctor prints output and per-user volumes.
- Voice audio is implemented: daemon-managed virtual devices (`veskslop_tts`, `veskslop_mic`,
`veskslop_capture`), TTS providers (local command + HTTP API), queued playback through `ffplay`,
ducking on `VOICE_SPEAKING`, and WAV capture. The daemon starts `pulseaudio` when `pactl` cannot
reach a server and reports install hints. Verified live on 2026-09-13 after installing
`pulseaudio`/`ffmpeg`: engine ok, devices created, PCM streamed.
- The Web UI serves self-signed HTTPS (`openssl`, stored under the state dir) so browser media
APIs have a secure context. `WEBUI_TLS=0` disables it.
- Capture and transcript are implemented: ffmpeg reads the client output, a VAD cuts it into
utterances, each becomes a WAV plus a transcript entry with the dominant speaker, and a
pluggable STT hook (local command or HTTP API) fills the text. Live events feed the console
Transcript tab; `get_transcript` and `/api/transcript` read the store. `VESKSLOP_CAPTURE_AUTO=1`
records calls automatically. Verified end to end with real PulseAudio on 2026-09-13.
- STT uses Vosk (CPU, light): `poc/scripts/stt-vosk.py` plus a small model in `poc/models/`
(auto-detected). Verified with a TTS→capture→Vosk round trip. Other providers remain available
via `VESKSLOP_STT_CMD`/`_API_URL`.
- TTS defaults to `espeak-ng` when installed; engine and voice live in Settings → Client → Speech
(daemon preferences, so all browsers share them).
- Preferences moved to `<state dir>/preferences.json` and no longer depend on the core, so guild
visibility saves even while the client is down. API responses send `Cache-Control: no-store`.
- Browser audio has an **Enable browser audio** button that requests the microphone permission,
which browsers require before listing output devices.
- Browser microphone puppeting (talking through the bot) is next; it needs a mix sink, a `pacat`
input, and echo handling.
- The web UI follows the Nocturne design handoff (`poc/WebUIMockup.zip`). The 7a shell is
implemented and rendered headless for review: header, call strip, three-column body, composer,
and the agent console.
- Takeover is a client-side read-only gate. Off, the composer is locked. Manual sends are logged.
- Core process control is implemented and verified live: the daemon starts the client under
auto-Xvfb, the bridge connects, and stop exits cleanly.
- Web QR login is implemented. The session reached `pending` against the real gateway; a full
phone approval is pending.
## Done
- Validated core bridge: protocol, actions, validation, structured errors, hard outbound rate limit.
- Auth persistence: pinned `DATA_DIR`, token file, preload seeding, `VESKSLOP_HEADLESS`.
- Daemon on the core-bridge transport. CDP stays as a development fallback.
- Trusted-only triggers: DMs and direct mentions. Trust tiers: user, trusted, admin.
- Context: 15 messages or 4000 characters, time stamps, file names only, incremental fetch, tmp
cache with a size limit, fetch cooldown.
- Continuation window: a rushed follow-up joins the request context.
- Response policy: origin channel or DM. Sensitive commands default to DM.
- Persistent action log: triggers, commands, replies, errors, rotation, and secret redaction.
- `poll` is paused (`POLL_ENABLED = false`). The parser and its tests stay.
- Human send pacing: first send of a burst waits 30-200 ms, later sends 500-2800 ms, and the burst
resets after a random 5-20 s of quiet. Reads take 100-300 ms and do not reset the burst.
- `user.search`: cache and guild-member lookup. `voice.get` searches all guilds without a `guildId`.
- `voice.overview`: occupied voice channels and members for a guild.
- `VOICE_STATE_UPDATE` events: normalized and pushed. Mock-verified.
- Voice actions: `voice.join` (muted), `voice.leave`, and `voice.move` in the core, with daemon
routes (`/api/voice/*`), MCP tools, and Web UI wiring. Voice actions are paced 250-750 ms and
written to the action log. The internal lookups were probed on the live client.
- Web UI voice rail: every voice channel is clickable. Clicking joins (muted); clicking the active
channel leaves. Verified with the stateful mock.
- Web UI startup: a **Start server** button appears left of Takeover while the bridge is down and
hides when it connects. State reloads retry while the client boots, so guilds appear without a
manual refresh. Core requests now time out after `VESKSLOP_REQUEST_TIMEOUT_MS` instead of
hanging forever on a missing reply.
- Settings: "Visible guilds" stays top level; Audio sits under a collapsible "Client" group;
Core process, Account login, and Action log sit under a collapsible "Server" group.
- Web UI timestamps use the browser timezone by default; Settings → Client → Display can override
it with any IANA zone. The choice is per browser (`localStorage`).
- Web UI preferences: visible guilds persist server-side in the daemon state file
(`/api/preferences`), so every browser shares them. The Settings panel has an explicit **Save**
button; selections filter the UI live and persist only on Save.
- DM list: recipients resolve from IDs through the user store, so names and avatars load.
- Guild menu: fixed the light default button background.
- Web UI: Nocturne design tokens in `poc/public/nocturne.css`, the 7a debug shell, live guild and
channel rail, messages, DMs, voice membership, agent console with a live event tab, and a
Settings slide-over for core process, login, visible guilds, and the action log. Roadmap
surfaces show as placeholders.
- Web UI data: opening a chat fetches the latest 50 messages into the client cache; attachments
hotlink Discord CDN URLs and are never stored; avatars and guild icons load from the CDN with a
monogram fallback; the page reloads itself when the bridge connects.
- Web UI drafts: composer and TTS text persist per channel in `localStorage` and survive
re-renders, reconnects, and page reloads.
- Visible guilds: the Settings panel filters the guild switcher and picker.
- Takeover: client-side read-only switch with `localStorage` persistence; manual sends are written
to the action log with `type: "manual"`.
- `dm.list`: cached DM and group DM channels, exposed over MCP as `list_dms`.
- Core process control: the daemon starts, stops, and restarts the client, reconnects the bridge
after a restart, auto-starts Xvfb when `DISPLAY` is missing, and stops what it started on exit.
Override with `VESKSLOP_CORE_CMD`/`ARGS`; disable Xvfb with `VESKSLOP_XVFB=0`.
- Web QR login: the Settings panel runs the remote-auth session and renders the QR. The token file
is written on approval and a running managed core restarts with the new token.
- Unit tests cover the pacing policy. `pnpm lint` and `pnpm testTypes` pass at the root.
- Web UI navigation: Guild/DM scope tabs, a DM rail with unread badges (purple/orange, grey while
Takeover is off), last-message times, favourite stars, and a People list of users seen in voice.
Favourited users get presence and call tracking, grouped by call in **In voice**. Speaking shows
level bars and bold names (2c marks for mute/deaf/video/share). The body dims while the bridge
is down; the Start button becomes Refresh when the daemon is unreachable.
- TTS settings are split into language and voice, grouped by base language with the requested
languages pinned on top.
- Text notifications are parsed: edits update the context cache, deletes drop from it, trusted
reactions are logged, and replies are kept in the request context. A reply to a puppet message
counts as a trigger without a mention. The web UI marks edited messages and replies.
- Voice auto-leave is implemented: when the owner (zzib) leaves the bot's call, the bot leaves
after `voiceLeaveDelaySeconds` (default 30). A rejoin cancels it. Verified with the mock,
including cancellation. `stay @user` is deferred.
- Audio verified end to end on 2026-09-14: TTS played into the virtual mic (recording
`veskslop_mic` peaked at -4.1 dB) and the browser monitor streamed PCM over the TLS WebSocket.
- Next: per-user limits and the collector account. The agent attachment (remote llama.cpp, MCP
transport, event delivery) waits for a design discussion; the draft prompt has open issues to
settle first.
## Next (in order)
1. Live verification: `voice.join`/`voice.leave` end to end, `user.search`, `voice.overview`,
`VOICE_STATE_UPDATE`, and a full web QR login approval on the real client.
2. Agent: daemon task push for free-form triggers (commands stay local), tool allowlist,
per-conversation serialization, and tests against a fake model.
3. Collector account: read-only modules for voice membership fallback.
4. Per-user rate limit and command cooldowns.
5. Mobile web UI: transcript and TTS in the Call tab; verify on a phone.
## Open discussions (need the user)
1. **Web UI auth.** Accepted: no auth, LAN or Tailscale only. Revisit before any wider exposure.
2. **MCP event delivery.** Decided for the local harness: the daemon pushes tasks to the agent's
`POST /task`. The transport details (retry, dedupe IDs, queue) are still open. Remote agents
poll or call tools directly; they do not receive pushes.
3. **Untrusted logging.** Untrusted interactions are ignored at this time. Confirm whether to log
untrusted DMs and mentions again.
4. **Privacy pass.** Only token-shaped and `mfa.` strings are redacted, and only in the persistent
action log. A fuller policy (retention, cache clearing, message redaction) is deferred.
5. **Live checks.** `reaction.add`, `user.search`, `voice.overview`, and the voice state event
normalization are mock-verified only. The QR login session reached `pending` and `cancel`
against the real gateway; the approval path needs the phone. Confirm on the first real-client
run.
## Blocked / needs input
- Slopbot: the bot is invited to the test guild and the intent is enabled. The token lives in
`poc/.env` (gitignored, mode 600) and will be replaced when the project goes to prod. The daemon
must be restarted to pick it up.
- The original repo `zib/Veskslop` has a broken server-side git setup. Fetches and clones fail with
`bad pack header` because a script (`/home/git/.sys_health_s3`) runs on the Veskslop read path.
The admin must fix it. Use `zib/Veskslop-t` (remote `veskslop-t`) until then. That remote has no
stored credential; use the one-off token URL in `GITEA.md`. Rechecked 2026-09-16: the original
still fails with `bad pack header`.
- `main` on the original server is `68fc8bb` (zib, "Upload files to docs/images"). It diverged from
the local `main` (`24447f5`). Reconcile after the server is fixed.
## Where to look
- Backlog: `TODO.md`
- Design documents: `docs/INDEX.md`
- Design handoff: `poc/WebUIMockup.zip` (screen 7a is implemented)
- Agent rules: `AGENTS.md`
- Daemon code: `poc/src/`
- Web UI code: `poc/src/webui.ts`, `poc/public/`
- Core bridge code: `src/shared/puppetProtocol.ts`, `src/renderer/puppet/`,
`src/main/puppetBridge.ts`
Reference in New Issue View Git Blame Copy Permalink