Add the media scrape plan (review draft)

CURRENT.md

@@ -1,6 +1,6 @@
 # CURRENT.md — session state
 
-Last updated: 2026-09-15
+Last updated: 2026-09-16
 
 This file tracks the work state. Read it first. Update it before you stop.
 
@@ -12,6 +12,11 @@ mobile draft at 390px.
 
 ## In progress
 
+- Media scrape project, branch `media-scrape-project` on `veskslop-t`. Plan drafted at
+  [media-scrape/PLAN.md](media-scrape/PLAN.md) and awaiting owner review; no code yet. The command
+  adds `scrape media` to archive a channel's history over time with DiscordChatExporter-style rate
+  limiting, a SQLite store, and a new Media tab in the Web UI. Open questions in the plan: scope,
+  storage location, bytes vs. links, and tab placement.
 - Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
   open; it carries the character sliders with engine defaults and +/-2 snapping, five named
   presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and
@@ -280,7 +285,9 @@ mobile draft at 390px.
 
 - The original repo `zib/Veskslop` has a broken server-side git setup. Fetches and clones fail with
   `bad pack header` because a script (`/home/git/.sys_health_s3`) runs on the Veskslop read path.
-  The admin must fix it. Use `zib/Veskslop-t` (remote `veskslop-t`) until then.
+  The admin must fix it. Use `zib/Veskslop-t` (remote `veskslop-t`) until then. That remote has no
+  stored credential; use the one-off token URL in `GITEA.md`. Rechecked 2026-09-16: the original
+  still fails with `bad pack header`.
 - `main` on the original server is `68fc8bb` (zib, "Upload files to docs/images"). It diverged from
   the local `main` (`24447f5`). Reconcile after the server is fixed.
 

docs/INDEX.md

@@ -25,6 +25,7 @@ This file lists the project documents.
 | [../CURRENT.md](../CURRENT.md) | Live session state |
 | [../TODO.md](../TODO.md) | Ordered backlog |
 | [../poc/README.md](../poc/README.md) | Spike and daemon run commands |
+| [../media-scrape/PLAN.md](../media-scrape/PLAN.md) | Media scrape and archive plan (review draft) |
 | [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration |
 | [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine |
 | [../README.md](../README.md) | Project introduction |

media-scrape/PLAN.md

@@ -0,0 +1,191 @@
+# Media scrape — design plan
+
+Status: **review draft**. Nothing is implemented until the owner answers the open questions.
+
+## Goal
+
+Let a trusted user ask the puppet to archive the media history of a channel:
+
+```
+@puppet scrape media            # the channel the command is sent in
+@puppet scrape media #channel   # a specific guild channel or thread
+@puppet scrape media status
+@puppet scrape media stop
+```
+
+The daemon walks the channel history over time and records every image, GIF, video (file or
+embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses
+the store with type filters and a link back to the source message.
+
+Constraints:
+
+- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge
+  (`message.fetch`), so the traffic still comes from the official client.
+- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a
+  time, and a configurable rate budget.
+- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the
+  Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside
+  `mapMessage` in `src/renderer/puppet/actions.ts`.
+
+## What existing scrapers do (research, 2026-09-16)
+
+| Project | Stars | License | Approach | What we take |
+| --- | --- | --- | --- | --- |
+| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. |
+| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. |
+| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. |
+| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. |
+| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. |
+
+### Techniques we adopt from DiscordChatExporter
+
+DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement:
+
+- On `429`, wait `Retry-After` + 1s, up to 8 attempts.
+- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds).
+- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out
+  the reset (plus 1s buffer, capped at 60s).
+- A per-token-kind preference: user tokens are treated more carefully than bot tokens.
+
+The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord
+client already retries internally, so the daemon adds its own conservative pacing and blind
+exponential backoff on failures. If live testing shows it needs the headers, the fetch result can
+carry them later.
+
+## Account choice: user account, not a bot
+
+| | Puppet (user account) | Bot account |
+| --- | --- | --- |
+| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. |
+| Guild channels | Every channel the account can see | Only channels the bot is invited to |
+| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions |
+| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets |
+| Auth path | Existing core bridge | A second token and a second login path |
+
+The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only
+path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g.
+`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it.
+
+## Architecture
+
+```
+media-scrape/               new folder, self-contained
+  PLAN.md                   this file
+  src/                      job engine, extractor, store, routes, MCP tools
+  test/                     unit tests against the mock core
+poc/src/webui.ts            one mount: mediaRoutes(deps) under /api/media
+poc/src/control-bridge.ts   MCP tools: scrape_media, get_media, media_status
+poc/public/                 Media tab (HTML/CSS/JS, Nocturne tokens)
+src/renderer/puppet/        mapMessage gains embeds, snapshots, size, attachment id, flags
+```
+
+The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not
+import daemon internals and can be tested alone.
+
+No new core action is needed. `message.fetch` already walks history with `before`/`after` and a
+limit of 1-100. The only core change is the message shape in `mapMessage`:
+
+- `attachments[].id`, `.size`, `.durationSecs`
+- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl`
+- forwarded message snapshots (`messageSnapshots`)
+- `stickerItems[]`
+- `voiceMessage` flag
+
+Validation and the error model stay unchanged.
+
+## Scraper engine
+
+- One job per channel, one running job at a time, queue the rest.
+- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is
+  the oldest message ID seen. Stop when a page returns fewer than 100 items.
+- Persist the cursor and the counters after every page, so a daemon restart resumes instead of
+  restarting.
+- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`),
+  configurable. Separate from the 100-300ms read gap and from the send pacing.
+- A rolling budget, default conservative and configurable: max pages per hour and per day. The job
+  pauses when either is hit and resumes after the window.
+- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the
+  job after repeated failures instead of retrying forever.
+- All outbound sends from commands still go through the normal paced queue. Scrape pages never
+  send.
+- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web
+  UI (`pages`, `messages`, `media`, `errors`, state).
+
+## Media classification
+
+| Kind | Sources |
+| --- | --- |
+| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers |
+| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds |
+| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video |
+| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) |
+
+- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are
+  signed and expire in about a day.
+- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when
+  available. The tab links out or embeds the provider player.
+- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages.
+- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap
+  (default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means
+  the URL expired; the job re-fetches the message later to refresh it.
+
+## Storage
+
+- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency.
+  If that proves awkward, fall back to `better-sqlite3`.
+- Location: `<state dir>/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names),
+  `thumbs/` (optional, `ffmpeg` from the audio pipeline).
+- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`.
+- Indexes on `(kind, timestamp)`, `channel_id`, `hash`.
+- Quota and retention settings on the same file as the daemon preferences.
+
+## Web UI: the Media tab
+
+- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity;
+  see open questions).
+- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on
+  filename, provider, and message content.
+- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original
+  URL, dimensions or duration, and the download button.
+- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern.
+- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`,
+  `GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`,
+  `GET /api/media/status`.
+- No authentication, same as the rest of the UI. LAN or Tailscale only.
+
+## Safety and guardrails
+
+- The command follows the existing trust rules: trusted users only, DM or direct mention, logged.
+- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`.
+- Content served from disk only when the sniffed type matches the extension allowlist.
+- Files are stored by hash only. Original filenames are kept in the database, never used as paths.
+- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command.
+- Respect the project test rules: build and test against the mock core; live-test in a private
+  test guild; the only live DM target stays zzib.
+- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one
+  the owner can lose.
+
+## Phases
+
+1. **Review.** Owner answers the open questions below.
+2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`,
+   `pnpm build`.
+3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core.
+4. **Web UI.** Media tab and API routes.
+5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune
+   the budgets, then a real channel.
+6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention
+   policy.
+
+## Open questions for the owner
+
+1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and
+   a whole-guild mode?
+2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)?
+3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside
+   the code in `media-scrape/`?
+4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only?
+5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the
+   mobile fifth tab be?
+6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred
+   privacy pass.