From 677afc747f842acedb86ffd0f3f33747ee39830e Mon Sep 17 00:00:00 2001 From: Fisto Date: Wed, 16 Sep 2026 07:00:40 +0200 Subject: [PATCH] Add the media scrape plan (review draft) --- CURRENT.md | 11 ++- docs/INDEX.md | 1 + media-scrape/PLAN.md | 191 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 201 insertions(+), 2 deletions(-) create mode 100644 media-scrape/PLAN.md diff --git a/CURRENT.md b/CURRENT.md index bf8d31c..879a228 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -1,6 +1,6 @@ # CURRENT.md — session state -Last updated: 2026-09-15 +Last updated: 2026-09-16 This file tracks the work state. Read it first. Update it before you stop. @@ -12,6 +12,11 @@ mobile draft at 390px. ## In progress +- Media scrape project, branch `media-scrape-project` on `veskslop-t`. Plan drafted at + [media-scrape/PLAN.md](media-scrape/PLAN.md) and awaiting owner review; no code yet. The command + adds `scrape media` to archive a channel's history over time with DiscordChatExporter-style rate + limiting, a SQLite store, and a new Media tab in the Web UI. Open questions in the plan: scope, + storage location, bytes vs. links, and tab placement. - Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each open; it carries the character sliders with engine defaults and +/-2 snapping, five named presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and @@ -280,7 +285,9 @@ mobile draft at 390px. - The original repo `zib/Veskslop` has a broken server-side git setup. Fetches and clones fail with `bad pack header` because a script (`/home/git/.sys_health_s3`) runs on the Veskslop read path. - The admin must fix it. Use `zib/Veskslop-t` (remote `veskslop-t`) until then. + The admin must fix it. Use `zib/Veskslop-t` (remote `veskslop-t`) until then. That remote has no + stored credential; use the one-off token URL in `GITEA.md`. Rechecked 2026-09-16: the original + still fails with `bad pack header`. - `main` on the original server is `68fc8bb` (zib, "Upload files to docs/images"). It diverged from the local `main` (`24447f5`). Reconcile after the server is fixed. diff --git a/docs/INDEX.md b/docs/INDEX.md index 32a84d9..ce549bb 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -25,6 +25,7 @@ This file lists the project documents. | [../CURRENT.md](../CURRENT.md) | Live session state | | [../TODO.md](../TODO.md) | Ordered backlog | | [../poc/README.md](../poc/README.md) | Spike and daemon run commands | +| [../media-scrape/PLAN.md](../media-scrape/PLAN.md) | Media scrape and archive plan (review draft) | | [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration | | [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine | | [../README.md](../README.md) | Project introduction | diff --git a/media-scrape/PLAN.md b/media-scrape/PLAN.md new file mode 100644 index 0000000..7563ff5 --- /dev/null +++ b/media-scrape/PLAN.md @@ -0,0 +1,191 @@ +# Media scrape — design plan + +Status: **review draft**. Nothing is implemented until the owner answers the open questions. + +## Goal + +Let a trusted user ask the puppet to archive the media history of a channel: + +``` +@puppet scrape media # the channel the command is sent in +@puppet scrape media #channel # a specific guild channel or thread +@puppet scrape media status +@puppet scrape media stop +``` + +The daemon walks the channel history over time and records every image, GIF, video (file or +embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses +the store with type filters and a link back to the source message. + +Constraints: + +- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge + (`message.fetch`), so the traffic still comes from the official client. +- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a + time, and a configurable rate budget. +- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the + Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside + `mapMessage` in `src/renderer/puppet/actions.ts`. + +## What existing scrapers do (research, 2026-09-16) + +| Project | Stars | License | Approach | What we take | +| --- | --- | --- | --- | --- | +| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. | +| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. | +| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. | +| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. | +| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. | + +### Techniques we adopt from DiscordChatExporter + +DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement: + +- On `429`, wait `Retry-After` + 1s, up to 8 attempts. +- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds). +- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out + the reset (plus 1s buffer, capped at 60s). +- A per-token-kind preference: user tokens are treated more carefully than bot tokens. + +The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord +client already retries internally, so the daemon adds its own conservative pacing and blind +exponential backoff on failures. If live testing shows it needs the headers, the fetch result can +carry them later. + +## Account choice: user account, not a bot + +| | Puppet (user account) | Bot account | +| --- | --- | --- | +| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. | +| Guild channels | Every channel the account can see | Only channels the bot is invited to | +| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions | +| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets | +| Auth path | Existing core bridge | A second token and a second login path | + +The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only +path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g. +`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it. + +## Architecture + +``` +media-scrape/ new folder, self-contained + PLAN.md this file + src/ job engine, extractor, store, routes, MCP tools + test/ unit tests against the mock core +poc/src/webui.ts one mount: mediaRoutes(deps) under /api/media +poc/src/control-bridge.ts MCP tools: scrape_media, get_media, media_status +poc/public/ Media tab (HTML/CSS/JS, Nocturne tokens) +src/renderer/puppet/ mapMessage gains embeds, snapshots, size, attachment id, flags +``` + +The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not +import daemon internals and can be tested alone. + +No new core action is needed. `message.fetch` already walks history with `before`/`after` and a +limit of 1-100. The only core change is the message shape in `mapMessage`: + +- `attachments[].id`, `.size`, `.durationSecs` +- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl` +- forwarded message snapshots (`messageSnapshots`) +- `stickerItems[]` +- `voiceMessage` flag + +Validation and the error model stay unchanged. + +## Scraper engine + +- One job per channel, one running job at a time, queue the rest. +- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is + the oldest message ID seen. Stop when a page returns fewer than 100 items. +- Persist the cursor and the counters after every page, so a daemon restart resumes instead of + restarting. +- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`), + configurable. Separate from the 100-300ms read gap and from the send pacing. +- A rolling budget, default conservative and configurable: max pages per hour and per day. The job + pauses when either is hit and resumes after the window. +- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the + job after repeated failures instead of retrying forever. +- All outbound sends from commands still go through the normal paced queue. Scrape pages never + send. +- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web + UI (`pages`, `messages`, `media`, `errors`, state). + +## Media classification + +| Kind | Sources | +| --- | --- | +| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers | +| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds | +| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video | +| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) | + +- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are + signed and expire in about a day. +- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when + available. The tab links out or embeds the provider player. +- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages. +- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap + (default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means + the URL expired; the job re-fetches the message later to refresh it. + +## Storage + +- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency. + If that proves awkward, fall back to `better-sqlite3`. +- Location: `/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names), + `thumbs/` (optional, `ffmpeg` from the audio pipeline). +- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`. +- Indexes on `(kind, timestamp)`, `channel_id`, `hash`. +- Quota and retention settings on the same file as the daemon preferences. + +## Web UI: the Media tab + +- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity; + see open questions). +- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on + filename, provider, and message content. +- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original + URL, dimensions or duration, and the download button. +- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern. +- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`, + `GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`, + `GET /api/media/status`. +- No authentication, same as the rest of the UI. LAN or Tailscale only. + +## Safety and guardrails + +- The command follows the existing trust rules: trusted users only, DM or direct mention, logged. +- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`. +- Content served from disk only when the sniffed type matches the extension allowlist. +- Files are stored by hash only. Original filenames are kept in the database, never used as paths. +- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command. +- Respect the project test rules: build and test against the mock core; live-test in a private + test guild; the only live DM target stays zzib. +- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one + the owner can lose. + +## Phases + +1. **Review.** Owner answers the open questions below. +2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`, + `pnpm build`. +3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core. +4. **Web UI.** Media tab and API routes. +5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune + the budgets, then a real channel. +6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention + policy. + +## Open questions for the owner + +1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and + a whole-guild mode? +2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)? +3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside + the code in `media-scrape/`? +4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only? +5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the + mobile fifth tab be? +6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred + privacy pass.