# Media scrape — design plan

Status: **review draft**. Nothing is implemented until the owner answers the open questions.

## Goal

Let a trusted user ask the puppet to archive the media history of a channel:

```
@puppet scrape media            # the channel the command is sent in
@puppet scrape media #channel   # a specific guild channel or thread
@puppet scrape media status
@puppet scrape media stop
```

The daemon walks the channel history over time and records every image, GIF, video (file or
embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses
the store with type filters and a link back to the source message.

Constraints:

- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge
  (`message.fetch`), so the traffic still comes from the official client.
- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a
  time, and a configurable rate budget.
- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the
  Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside
  `mapMessage` in `src/renderer/puppet/actions.ts`.

## What existing scrapers do (research, 2026-09-16)

| Project | Stars | License | Approach | What we take |
| --- | --- | --- | --- | --- |
| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. |
| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. |
| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. |
| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. |
| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. |

### Techniques we adopt from DiscordChatExporter

DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement:

- On `429`, wait `Retry-After` + 1s, up to 8 attempts.
- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds).
- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out
  the reset (plus 1s buffer, capped at 60s).
- A per-token-kind preference: user tokens are treated more carefully than bot tokens.

The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord
client already retries internally, so the daemon adds its own conservative pacing and blind
exponential backoff on failures. If live testing shows it needs the headers, the fetch result can
carry them later.

## Account choice: user account, not a bot

| | Puppet (user account) | Bot account |
| --- | --- | --- |
| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. |
| Guild channels | Every channel the account can see | Only channels the bot is invited to |
| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions |
| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets |
| Auth path | Existing core bridge | A second token and a second login path |

The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only
path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g.
`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it.

## Architecture

```
media-scrape/               new folder, self-contained
  PLAN.md                   this file
  src/                      job engine, extractor, store, routes, MCP tools
  test/                     unit tests against the mock core
poc/src/webui.ts            one mount: mediaRoutes(deps) under /api/media
poc/src/control-bridge.ts   MCP tools: scrape_media, get_media, media_status
poc/public/                 Media tab (HTML/CSS/JS, Nocturne tokens)
src/renderer/puppet/        mapMessage gains embeds, snapshots, size, attachment id, flags
```

The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not
import daemon internals and can be tested alone.

No new core action is needed. `message.fetch` already walks history with `before`/`after` and a
limit of 1-100. The only core change is the message shape in `mapMessage`:

- `attachments[].id`, `.size`, `.durationSecs`
- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl`
- forwarded message snapshots (`messageSnapshots`)
- `stickerItems[]`
- `voiceMessage` flag

Validation and the error model stay unchanged.

## Scraper engine

- One job per channel, one running job at a time, queue the rest.
- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is
  the oldest message ID seen. Stop when a page returns fewer than 100 items.
- Persist the cursor and the counters after every page, so a daemon restart resumes instead of
  restarting.
- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`),
  configurable. Separate from the 100-300ms read gap and from the send pacing.
- A rolling budget, default conservative and configurable: max pages per hour and per day. The job
  pauses when either is hit and resumes after the window.
- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the
  job after repeated failures instead of retrying forever.
- All outbound sends from commands still go through the normal paced queue. Scrape pages never
  send.
- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web
  UI (`pages`, `messages`, `media`, `errors`, state).

## Media classification

| Kind | Sources |
| --- | --- |
| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers |
| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds |
| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video |
| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) |

- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are
  signed and expire in about a day.
- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when
  available. The tab links out or embeds the provider player.
- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages.
- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap
  (default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means
  the URL expired; the job re-fetches the message later to refresh it.

## Storage

- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency.
  If that proves awkward, fall back to `better-sqlite3`.
- Location: `<state dir>/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names),
  `thumbs/` (optional, `ffmpeg` from the audio pipeline).
- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`.
- Indexes on `(kind, timestamp)`, `channel_id`, `hash`.
- Quota and retention settings on the same file as the daemon preferences.

## Web UI: the Media tab

- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity;
  see open questions).
- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on
  filename, provider, and message content.
- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original
  URL, dimensions or duration, and the download button.
- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern.
- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`,
  `GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`,
  `GET /api/media/status`.
- No authentication, same as the rest of the UI. LAN or Tailscale only.

## Safety and guardrails

- The command follows the existing trust rules: trusted users only, DM or direct mention, logged.
- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`.
- Content served from disk only when the sniffed type matches the extension allowlist.
- Files are stored by hash only. Original filenames are kept in the database, never used as paths.
- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command.
- Respect the project test rules: build and test against the mock core; live-test in a private
  test guild; the only live DM target stays zzib.
- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one
  the owner can lose.

## Phases

1. **Review.** Owner answers the open questions below.
2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`,
   `pnpm build`.
3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core.
4. **Web UI.** Media tab and API routes.
5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune
   the budgets, then a real channel.
6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention
   policy.

## Open questions for the owner

1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and
   a whole-guild mode?
2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)?
3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside
   the code in `media-scrape/`?
4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only?
5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the
   mobile fifth tab be?
6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred
   privacy pass.