# CURRENT.md — session state Last updated: 2026-09-16 This file tracks the work state. Read it first. Update it before you stop. ## Current focus (broad) The text layer is done. Voice actions and audio work. The local LLM harness (Veskslop agent, Go + Eino) is scaffolded and smoke-tested; the daemon-side task push is next. The web UI has a first mobile draft at 390px. ## In progress - Media archive project, branch `media-scrape-project` on `veskslop-t`. Design approved 2026-09-16 (docs: [media.md](docs/media.md), [media-scraper.md](docs/media-scraper.md), [slopbot.md](docs/slopbot.md), [media-mcp.md](docs/media-mcp.md)). Decisions: scrape the channel the command is sent in; slopbot (discord.js bot core) for guild channels and the puppet core bridge for group DMs; one self-contained SQLite + media dir per scrape under `media-scrape/db/` (gitignored); links indexed by default, downloads optional, third-party embeds never downloaded; 30 GiB default limit with a 90% warning and 5% no-delete overrun, usage bar and clean button in Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C8. The slopbot scaffold (config loader, invite helper) and its discord.js dependency were added 2026-09-16. C5 smoke test: `/api/media/status` and `/api/media` answer on a daemon with no core. Checkpoints, push after each: - [x] C1 design docs, AGENTS/INDEX pointers, `.gitignore` - [x] C2 core message shape: embeds, forwards, attachment metadata - [x] C3 media-scrape store and puppet walker, tests - [x] C4 slopbot backend, tests - [x] C5 daemon wiring: `scrape media` command, MediaService, `/api/media/*` - [x] C6 Media and Bot tabs in the Web UI - [x] C7a storage limits, retention, warning/confirm flow, storage and clean API - [x] C7b Settings group: limit field with info hover, usage bar, clean modal - [x] C8 MCP service hooks (`scrape_media`, scrape reads, storage, clean; reads in `run_commands`) - [x] C10 downloader: attachment and sticker bytes, hash-addressed, serial, size-capped, retries; embeds excluded in SQL; local files served at `/api/media///file` and preferred by the Media tab. Live run 2026-09-16: scrape `...-20260916083401` fetched the 784 KB attachment; storage counted actual bytes. - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: `previewed: 2`; thumbs served over `/api/media///thumb` (200). - [x] C62 VOX prosody controls. The last open Veskslop-side item from the MicroslopSam design: `tts.vox` and `tts.voxClean` preferences (per-line overrides on `/api/audio/speak` and MCP `speak`), `VESKSLOP_TTS_VOX` / `VESKSLOP_TTS_VOX_CLEAN` exports plus `{vox}` / `{vox_clean}` placeholders for the command provider, and a Radio-kit row in the TTS panel with the checkboxes and a help tooltip. VOX disables word mode in the panel; the wrapper keeps its documented precedence. No other MicroslopSam work is outstanding: the engine branch is already an ancestor of main, and the remaining contract note (HTTP `/voices` discovery) is not needed for the command-provider path in use. - [x] C61 Media browsing performance. The grid's merge step rescanned every scrape database per request (about 300 ms for a page, 250 ms for stats, both refetched on every scroll append). `MediaService` now keeps a candidate cache keyed by a scrape revision plus the normalized filter shape (paging and empties dropped): one scan per filter set, then ~5 ms per page, and stats share the scan. The client skips stats when the filters are unchanged (60 s TTL), updates only the pager instead of rebuilding the toolbar on page loads, and renders tiles with `content-visibility`/`decoding=async`/low fetch priority. Measured: 302 ms first page, 5 ms later pages; smoke test streamed 2,480 tiles with no console errors. - [x] C60 Incremental re-scrapes and archive cleanup. A new scrape of a channel that already has a completed walk now starts after that walk's newest message (the best completed scrape by message count), so it only covers the newer gap instead of re-fetching history; the previous newest is stored as `afterId` and shown as the from-date. Live-verified: a re-scrape of `559354230444654605` started after `999789325183090769` from the 2026-09-17 scrape. Deleted the four pre-pagination-fix test scrapes (superseded by full walks) and the truncated DM scrape with no complete counterpart; `pnpm verify:scrapes` now reports 10 complete, 0 superseded, 0 incomplete. - [x] C59 Scrape names and the completeness check. Scrape listings (History, Clean media, Bot-tab job/lane cards) now lead with the first two words of the channel or group-chat name plus the scrape id. New `poc/scripts/verify-scrapes.ts` (`pnpm verify:scrapes`) checks every index: done state, no lastError, messages non-zero, messages vs the Discord search total, and the oldest message against the source's snowflake creation time (60-day grace). Scrapes a later full walk supersedes are reported as `old`, not failures. Result: 10 complete, 4 superseded, 1 genuinely incomplete — the early DM test scrape `805255536223715370-20260916215657` (10 messages, pagination stopped before the fix, never re-scraped). - [x] C58 Titles from Discord's embed cache. Gone YouTube videos have no oEmbed answer, but Discord keeps the embed title (and, with the rebuilt core bundle, the author/channel). The title lane now asks the source message's embed first and only then the provider oEmbed; the core maps `authorName` from `embed.author.name`, extraction stores it as `title_author`, and the viewer shows a prominent title/channel heading plus "Video unavailable (removed or private)" when a YouTube row has neither. The 783 title errors were reset and the lane backfilled them: the reported example (message 703788394337009754) now reads "Gun Test: Firing Empty Shell's Handheld XM556 Microgun". The channel name populates after the Vesktop client restarts on the new bundle (the client is externally started). - [x] C57 Page mode. The media grid has an Infinite/Page flippy toggle in the bar. Page mode renders one `pageSize` slice via `offset = (page-1)*pageSize`, shows Prev / editable page number / total / Next, persists the page in localStorage, and scrolls to the next page one at a time with a 2 s cooldown after each load. Same filter + page is stable while the archive is unchanged. Infinite mode keeps the stream buffer; prefetch, buffer fill, and the buffer are disabled in page mode. Page mode pre-loads the next page's JSON and preview images, so jumping one page renders instantly. - [x] C56 Recovery completed. First run after the C55 fix recovered 1,117 of 1,240 Discord-CDN previews (90%); the last 123 were confirmed genuinely gone (refresh + direct fetch and `/stream` both fail) and are marked `unavailable after recovery` so later runs skip them. The recovery query now only covers HTTP errors on refreshable URLs, so proxy `no preview source` rows are not swept. Archive: 26,725 cached previews, 7 pending, 1,621 terminal provider failures, and the recovery task reports 0 to retry. - [x] C55 Recovery fix. The first recovery run parked live files because refreshed embed URLs land in `play_url` while preview resolution kept using the stale `url`; the engine now resolves embed previews from `play_url` first and re-resolves from the refreshed URL on retry, and both refresh branches canonicalize `url` to the direct media link. The task also counts rows that park as suspects as `deferred` (not gone), tries each row once per run instead of looping, and initializes the slopbot so guild scrapes refresh through the bot. The 192 suspects from the broken run were converted back to unverified so they are retried. - [x] C54 Message links, suspect retries, and preview recovery. Items now carry `messageUrl` (from `media_sources` + scrape guild), the viewer shows/copies it instead of the expiring CDN link and opens a fresh stream for View full, YouTube items offer an **Open on YouTube** fallback, and the metadata gained Channel/Scrape/Title (title includes the channel kept from oEmbed, `title_author`). Preview and download HTTP failures on Discord hosts first refresh the URL from the source message and retry; if that fails the row parks as a **suspect** with one delayed retry (50 s, runs after the batch; a batch two-thirds suspect doubles the cooldown, 60 s floor). Still-failing rows are **unverified** and skipped by normal passes; the Bot tab shows suspect/unverified counts and a segmented progress bar (cached blue, pending gray, failed red). **Recover Discord previews** is a paused one-shot task that re-fetches every Discord-CDN row closed with an HTTP error across all scrapes (mixed bot/puppet via the existing refresh path) and lands in the task history when done. All 1,208 Discord 404 rows sampled 15/15 recoverable. - [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated `TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause (hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no media-traffic lock because it only calls YouTube's oEmbed. New `titleBatch` on the engine, per-scrape `titlePending/titled/titleFailed` counters, `/api/media/titles` + start/pause routes, and a Bot-tab card with progress, throughput, ETA, and Start/Pause. The lane auto-starts on boot; with nothing pending it exits immediately. Preview backfill goes back to previews only. Unit tests for `titleBatch` and a lane catch-up run; smoke covers the card. - [x] C52 Wildcard title search. `*` now maps to a LIKE wildcard in both title modes: a leading `*` drops the word anchor (`*car` finds *scar* and *discard*), in-token stars match between letters (`c*r*`), and a trailing star matches the plain prefix. `%` and `_` are escaped with an `ESCAPE` clause so they only match themselves. Unit tests cover prefix vs wildcard vs in-word matches; the drawer hint documents the syntax. - [x] C51 YouTube Shorts normalization. Shorts links broke the embedded player and View full because the viewer needs `watch?v=`. Extraction now rewrites `youtube.com/shorts/` (and `m.`) to the watch form, the viewer also accepts `/shorts/` defensively, and a one-time `PRAGMA user_version` migration rewrites existing rows. The live archive had 45 shorts rows; one collided with an already indexed watch URL and folded into it (preview fields merged, message links moved) — the migration is idempotent and unit-tested, including the collision. 45 → 0 shorts rows, and the user's example `uAf18d7mrnU` is now `watch?v=uAf18d7mrnU` with `embed/uAf18d7mrnU` as the play URL. - [x] C50 Title backfill in the queue. YouTube-like rows (provider or URL matching youtube/youtu.be) without a title are now a second work type in the backfill queue: resolved through the provider oEmbed, stored as `title`/`title_key`, with `title_attempts` and `title_error` so deleted videos close out after the tries setting. Titles and previews share a batch (up to half for titles while both remain); `previewCounts().titlePending`, `BackfillMember.pendingTitles`, and the job totals carry the work, and the Bot tab member line shows `+ N titles`. Unit tests cover selection, resolution, and the give-up cap. Live: the user's queue (batch 40, 30 s, auto off) started once to verify — 133 titles resolved in two minutes, dead videos counting toward the cap. - [x] C49 Searchable video titles. Embed rows now store Discord's `title` plus a normalized `title_key` (lowercase, punctuation collapsed). The Advanced drawer has a title field with an **Exact match (quotes)** toggle: default is order-free multi-word prefix search ("Ma Ca Ju" finds *Man who jumps over car*), exact mode is one phrase; rows without a title drop out while the query is set. Store, `/api/media`, `/api/media/stats`, and MCP carry `title`/`titleExact`; the API result and card tooltip expose the title. Unit tests cover prefixes, mid-word misses, exact phrases, and punctuation folding; smoke checks the field and toggle. Note: titles only exist for rows scraped from now on — pre-existing rows need the backfill queued in TODO.md. - [x] C48 Expanded providers and the Must include mirror. The list grew to 19 entries from the user's picks: reddit, twitter, tiktok, instagram, streamable, facebook, gyazo, a merged `giphy,tenor` (one checkbox, two substrings), 2ch.hk (stand-in for 4chan; the CDN hosts stay behind the default URL exclusion), funnyjunk, and internet archive. The drawer gained a **Must include** block mirroring **Do not include**: provider/source-kind checkboxes (`includeProviders`, `includeSources`), file-type chips, and the size/length bounds (moved here since they are inclusions, not exclusions). Store, `/api/media`, `/api/media/stats`, and MCP all carry the two include params. Live-verified: `includeProviders=youtube` returns only YouTube rows, `giphy,tenor` includes/excludes the same rows complementarily; smoke checks both groups against the API list (22 = 19 + 3 kinds). - [x] C47 Provider list from the dataset. Aggregated the `provider` column over all 11 scraper databases (distinct URLs): discord 13631 (folding cdn.discordapp.com and media.discordapp.net), youtube 4840, tenor 352, twitch 196, twitter 111, steam 83, medal 59, sketchfab 31, gfycat 30, imgur 23. Those ten now live in `PROVIDER_FILTERS` (media-scrape), are served in `/api/media/status` as `providerFilters`, and the Advanced drawer renders the source checkboxes from that list (4chan stays out; the default URL host exclusion covers it). Unit test for the list shape and a smoke check that the checkbox count matches the API list (13 = 10 + 3 kinds). - [x] C46 Sort and exclusion filters. The Advanced drawer gained a sort select (newest, oldest, user, size, length; global over the merged URL-dedupe stream) and a **Do not include** group with nested collapsibles: sources (Twitch, YouTube, Discord, Tenor, Giphy, embeds, attachments, stickers), file types, size (less/greater than with B/KB/MB/GB units), and length (seconds). Store, `/api/media` + `/api/media/stats` (shared parser), and MCP `query_media` all carry the new params; size and length ranges drop rows with unknown values, and the ext matching now coalesces NULL content types so a `NOT (...)` cannot swallow rows. Also hid the media filter bar and drawer on the Bot tab. Unit tests for every exclusion, all five sorts (nulls last), and smoke checks for the hidden bar on the Bot tab. - [x] C45 Streaming media grid. The grid rebuilt every card on each appended page, which made filtered browsing flash, and the `Load more` trigger only worked once the grid could already scroll, so short filtered pages dead-stopped at ~11 cards. Now pages append incrementally (existing images are never rebuilt), a stream buffer capped at 1.5 viewport heights keeps the scroll alive while the next page loads, auto-fill cascades while the buffer is on screen, the buffer note reads `Loading more…` / `Scroll for more` / `End of results`, and an elastic nudge replaces the hard stop at the bottom. Smoke test: first load reaches 120 cards via auto-fill, scrolling appends 120 → 440 with no broken images. - [x] C44 One traffic pipeline and split ETAs. Walks and backfill batches now share a media-traffic lock: a walk holds it for its whole run, a batch for one batch, so the pacing settings bound total traffic instead of each action. Starting then immediately stopping a scrape now fetches nothing (tested). Backfill ETAs split into `etaStartMs` (waiting behind earlier jobs), `etaRunMs` (own work), and `etaTotalMs`; the Bot tab shows only the total and puts `starts in X · runs for Y` in the tooltip, same for the queue totals. Tests: lock ordering, stop semantics, ETA ordering. - [x] C43 Bot-tab backfill control room and media-tab strip rules. The engine gained `previewBatch`/`rateLimitStatus`/`updateOptions`; a new `BackfillQueue` groups preview work per chat (paused and resumed scrapes of a chat combine), runs one job at a time in batches, and reports pending/cached/failed per kind, throughput, measured and settings-based ETA, and the live cooldown. `MediaService` drives it and exposes `GET /api/media/backfill` plus start/pause; `/api/preferences` now carries `media.tunables` (Backfilling / Scraping / Timers, persisted and applied live). The Bot tab shows totals with a Bot/Puppet split, per-chat job cards with progress and Start/Pause, **Start backfill** / **Pause all**, the red `API timed out, increase timers` banner, and the three settings groups. The media strip shows running plus one paused scrape, dismisses a paused one on Stop without stopping it, turns red while limited, and shows a session-only `N scrapes completed` note. The old `VESKSLOP_MEDIA_*` tuning env vars are gone. New unit tests cover batching, the cooldown, live retuning, and queue grouping/finish/pause; the Playwright smoke test covers the Bot tab. Also fixed a real stall the queue exposed: `pendingPreviews` never returned thumbnail-less embeds, so they sat pending forever and stalled the job; the store now includes them and the resolver handles Discord CDN unfurls through the proxy frame, with a content-type guard so the og:image fallback never reads media bytes. Live: daemon restarted, pending falling (6018 → ~5900), cached rising, dead CDN unfurls closed as 404, jobs queue smallest-first, settings round-trip verified (21.4 → 37.5 rows/min at 30 s cooldown). - [x] C42 Preview attempts cap. The restart log showed the backfill retrying dead hosts (Gfycat DNS is gone, Streamable answers 400) forever as "transient". Rows now carry `preview_attempts`; after `VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` (3) the row is marked `unreachable after N attempt(s)` and leaves the pending queue. Unit-tested. - [x] C41 Restart keeps its logs. A button restart spawned the replacement with `stdio: ignore`, so anything after the restart vanished. The replacement now appends to `daemon.log` beside the token file (`VESKSLOP_DAEMON_LOG` overrides); documented in [docs/daemon.md](docs/daemon.md). - [x] C40 Stale scrape states. Two `general_chut` scrapes showed `running` because the daemons that owned them were killed mid-job; the state file kept the last written value. On daemon start the engine now reconciles `running`/`queued` scrapes to `paused`, and History has a per-scrape **Delete** button (clean with a single ID) next to Resume. Unit-tested. - [x] C39 File-type filter and the 4chan default. The Advanced drawer gained extension chips (video mp4/webm/mov/mkv, image png/jpg/gif/webp/avif, audio mp3/ogg/wav/m4a/flac) matched on URL or content type, plus a default-on **Hide 4chan links (usually dead)** toggle. Backend, API (`ext`, `excludeHosts`) and MCP support; live: video count 7696 → 7383 with the default exclusion, `ext=mp4`/`webm` return only those files. Also confirmed the mr-game-memorial group has 924 attachment videos with 772 previews cached and 0 preview errors — the rest are just pending in the 2,140-item backfill queue. - [x] C38 Filter layout swap. The common filters (filetype chips, source kind, server/DM origin, multi-select People popover) now sit in the always-visible bar; the Advanced search drawer holds aspect, resolution, channel, date range, cached-only, and the card-size slider. Apply/Reset stay in the bar. Fixed a dropped `defaultFilters` helper found by the Playwright smoke test, which passes on the new layout. - [x] C37 Advanced search drawer, adaptive paging, and zoom. Filters moved into an **Advanced search** drawer with Apply/Reset (no auto-update): filetype chips, source kind, origin (server/DM), multi-select People, and an Advanced section (aspect, resolution, channel, dates, **cachedOnly**). A **Card size** slider scales tiles from 100 px (2/3 of current) to one image per row; page size = full rows about 1.5 screens tall and follows zoom/resize. Infinite scroll preloads the next page's JSON and previews. Backend gained `authorIds`, `origin`, `cachedOnly` on `/api/media`, `/api/media/stats`, and MCP `query_media`. Verified: `origin=dm` and `cachedOnly=1` return the expected rows. - [x] C36 Daemon restart, preview diagnostics, browse-time caching, and rate-limit pauses. Settings → Server has a **Restart daemon** button (`POST /api/daemon/restart`: re-execs the daemon, keeps the client running). The preview pass logs every failure with scrape/item/kind/ provider/reason plus a per-pass summary, and `GET /api/media/preview-failures` lists them. Browsing no longer fetches Discord at all: the grid uses cached stills only (placeholders for missing ones), GIF hover plays the cached animated GIF (kept as `anim_hash`, served from `/anim`), and streaming only starts on explicit play. `429`s from previews/downloads raise a cooldown (`VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS`, default 60 s, or `Retry-After`), leaving rows pending and pausing both walker and previews. - [x] C35 Walk/preview cycling. A live scrape now alternates: walk 5 pages, preview up to one batch (25, with the cooldown), walk on. Tiles fill in during a long walk instead of only at the end, and preview traffic is spread through the walk. Configurable with `VESKSLOP_MEDIA_WALK_CYCLE_PAGES` (0 defers previews to the end). The final pass still drains whatever remains. - [x] C34 Resume from cursor. A paused scrape can continue where it stopped instead of re-walking the top: `POST /api/media/scrape/resume`, MCP `resume_media_scrape`, and a Resume button in History. Previews never needed the walk — backfill fetches them straight from the stored URLs; the walk is only for discovery. Live: the paused `gernal_chut` scrape resumed at cursor `365256427247239181`, fetched its final page, and entered the preview pass. - [x] C33 Puppet history pagination. The core's `message.fetch` read the client store, which keeps only the newest messages plus the channel's first — every page was 10 messages and the walk stopped after one page (a 10-message, 4-media scrape of an ancient group with thousands of messages). It now fetches through the client's REST layer first (contiguous 100-message pages, `transport: "rest"`), store as fallback. Rebuilt and restarted the managed client; live re-run of channel `591000745164406784`: 21+ pages / 2,100+ messages and climbing, with hundreds of images/videos being indexed. - [x] C32 Group-chat clarity and puppet refresh. Group DM scrapes already worked (live-verified against the running daemon: `zıbb & fred` and a 1:1 DM, backend `puppet`, done, previews cached); the Group chat tab had shown an empty list while the core was offline. It now says so explicitly and offers Retry. Expired attachment URLs in DM/group-DM scrapes refresh through the core bridge (`message.fetch` with `before: id+1`, which returns exactly the target). - [x] C31 Batched previews. The preview pass now runs in batches (default 25) with a cooldown (default 60 s), configurable via `VESKSLOP_MEDIA_PREVIEW_BATCH` and `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`; batching counts rows, not successes. Live log: `preview batch of 25 done, pausing 60000 ms`. - [x] C30 Preview completeness. Audio attachments and voice messages get an ffmpeg waveform strip; oversized animated GIFs fall back to an ffmpeg-read first frame instead of `too large`; transient network failures stay pending for the next backfill instead of becoming permanent. Live: a scrape with 58 audio items backfilled 13 waveforms in 40 s, serving `image/webp`. Remaining non-cached: Lottie stickers/custom emoji (not indexed as media) and embeds whose pages refuse bots (marked PRIVATE/GONE by their 403/404). - [x] C29 Attachment video previews. Discord's image proxy renders a frame for video files (`?width=320&height=320&format=webp`), so `previewUrlFor` now resizes attachment mp4/webm in links mode too and `pendingPreviews` includes non-embed videos. Live: paused-scrape videos backfilled to `image/webp` thumbs, pending queue drained to 0. - [x] C28 Media grid usability. Infinite scroll (240 px threshold) with a Load more fallback, `previewError` exposed so tiles show PRIVATE (403) or GONE (404/410) with the reason in the tooltip, and the scrape strip lists only queued/running jobs — paused scrapes move to History. - [x] C27 Non-media stream guard. Rows awaiting backfill fell back to first-frame `