Add the media scrape plan (review draft)
This commit is contained in:
Fisto
@@ -0,0 +1,191 @@
|
||||
# Media scrape — design plan
|
||||
|
||||
Status: **review draft**. Nothing is implemented until the owner answers the open questions.
|
||||
|
||||
## Goal
|
||||
|
||||
Let a trusted user ask the puppet to archive the media history of a channel:
|
||||
|
||||
```
|
||||
@puppet scrape media # the channel the command is sent in
|
||||
@puppet scrape media #channel # a specific guild channel or thread
|
||||
@puppet scrape media status
|
||||
@puppet scrape media stop
|
||||
```
|
||||
|
||||
The daemon walks the channel history over time and records every image, GIF, video (file or
|
||||
embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses
|
||||
the store with type filters and a link back to the source message.
|
||||
|
||||
Constraints:
|
||||
|
||||
- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge
|
||||
(`message.fetch`), so the traffic still comes from the official client.
|
||||
- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a
|
||||
time, and a configurable rate budget.
|
||||
- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the
|
||||
Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside
|
||||
`mapMessage` in `src/renderer/puppet/actions.ts`.
|
||||
|
||||
## What existing scrapers do (research, 2026-09-16)
|
||||
|
||||
| Project | Stars | License | Approach | What we take |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. |
|
||||
| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. |
|
||||
| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. |
|
||||
| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. |
|
||||
| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. |
|
||||
|
||||
### Techniques we adopt from DiscordChatExporter
|
||||
|
||||
DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement:
|
||||
|
||||
- On `429`, wait `Retry-After` + 1s, up to 8 attempts.
|
||||
- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds).
|
||||
- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out
|
||||
the reset (plus 1s buffer, capped at 60s).
|
||||
- A per-token-kind preference: user tokens are treated more carefully than bot tokens.
|
||||
|
||||
The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord
|
||||
client already retries internally, so the daemon adds its own conservative pacing and blind
|
||||
exponential backoff on failures. If live testing shows it needs the headers, the fetch result can
|
||||
carry them later.
|
||||
|
||||
## Account choice: user account, not a bot
|
||||
|
||||
| | Puppet (user account) | Bot account |
|
||||
| --- | --- | --- |
|
||||
| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. |
|
||||
| Guild channels | Every channel the account can see | Only channels the bot is invited to |
|
||||
| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions |
|
||||
| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets |
|
||||
| Auth path | Existing core bridge | A second token and a second login path |
|
||||
|
||||
The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only
|
||||
path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g.
|
||||
`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
media-scrape/ new folder, self-contained
|
||||
PLAN.md this file
|
||||
src/ job engine, extractor, store, routes, MCP tools
|
||||
test/ unit tests against the mock core
|
||||
poc/src/webui.ts one mount: mediaRoutes(deps) under /api/media
|
||||
poc/src/control-bridge.ts MCP tools: scrape_media, get_media, media_status
|
||||
poc/public/ Media tab (HTML/CSS/JS, Nocturne tokens)
|
||||
src/renderer/puppet/ mapMessage gains embeds, snapshots, size, attachment id, flags
|
||||
```
|
||||
|
||||
The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not
|
||||
import daemon internals and can be tested alone.
|
||||
|
||||
No new core action is needed. `message.fetch` already walks history with `before`/`after` and a
|
||||
limit of 1-100. The only core change is the message shape in `mapMessage`:
|
||||
|
||||
- `attachments[].id`, `.size`, `.durationSecs`
|
||||
- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl`
|
||||
- forwarded message snapshots (`messageSnapshots`)
|
||||
- `stickerItems[]`
|
||||
- `voiceMessage` flag
|
||||
|
||||
Validation and the error model stay unchanged.
|
||||
|
||||
## Scraper engine
|
||||
|
||||
- One job per channel, one running job at a time, queue the rest.
|
||||
- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is
|
||||
the oldest message ID seen. Stop when a page returns fewer than 100 items.
|
||||
- Persist the cursor and the counters after every page, so a daemon restart resumes instead of
|
||||
restarting.
|
||||
- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`),
|
||||
configurable. Separate from the 100-300ms read gap and from the send pacing.
|
||||
- A rolling budget, default conservative and configurable: max pages per hour and per day. The job
|
||||
pauses when either is hit and resumes after the window.
|
||||
- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the
|
||||
job after repeated failures instead of retrying forever.
|
||||
- All outbound sends from commands still go through the normal paced queue. Scrape pages never
|
||||
send.
|
||||
- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web
|
||||
UI (`pages`, `messages`, `media`, `errors`, state).
|
||||
|
||||
## Media classification
|
||||
|
||||
| Kind | Sources |
|
||||
| --- | --- |
|
||||
| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers |
|
||||
| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds |
|
||||
| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video |
|
||||
| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) |
|
||||
|
||||
- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are
|
||||
signed and expire in about a day.
|
||||
- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when
|
||||
available. The tab links out or embeds the provider player.
|
||||
- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages.
|
||||
- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap
|
||||
(default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means
|
||||
the URL expired; the job re-fetches the message later to refresh it.
|
||||
|
||||
## Storage
|
||||
|
||||
- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency.
|
||||
If that proves awkward, fall back to `better-sqlite3`.
|
||||
- Location: `<state dir>/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names),
|
||||
`thumbs/` (optional, `ffmpeg` from the audio pipeline).
|
||||
- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`.
|
||||
- Indexes on `(kind, timestamp)`, `channel_id`, `hash`.
|
||||
- Quota and retention settings on the same file as the daemon preferences.
|
||||
|
||||
## Web UI: the Media tab
|
||||
|
||||
- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity;
|
||||
see open questions).
|
||||
- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on
|
||||
filename, provider, and message content.
|
||||
- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original
|
||||
URL, dimensions or duration, and the download button.
|
||||
- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern.
|
||||
- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`,
|
||||
`GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`,
|
||||
`GET /api/media/status`.
|
||||
- No authentication, same as the rest of the UI. LAN or Tailscale only.
|
||||
|
||||
## Safety and guardrails
|
||||
|
||||
- The command follows the existing trust rules: trusted users only, DM or direct mention, logged.
|
||||
- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`.
|
||||
- Content served from disk only when the sniffed type matches the extension allowlist.
|
||||
- Files are stored by hash only. Original filenames are kept in the database, never used as paths.
|
||||
- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command.
|
||||
- Respect the project test rules: build and test against the mock core; live-test in a private
|
||||
test guild; the only live DM target stays zzib.
|
||||
- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one
|
||||
the owner can lose.
|
||||
|
||||
## Phases
|
||||
|
||||
1. **Review.** Owner answers the open questions below.
|
||||
2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`,
|
||||
`pnpm build`.
|
||||
3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core.
|
||||
4. **Web UI.** Media tab and API routes.
|
||||
5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune
|
||||
the budgets, then a real channel.
|
||||
6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention
|
||||
policy.
|
||||
|
||||
## Open questions for the owner
|
||||
|
||||
1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and
|
||||
a whole-guild mode?
|
||||
2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)?
|
||||
3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside
|
||||
the code in `media-scrape/`?
|
||||
4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only?
|
||||
5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the
|
||||
mobile fifth tab be?
|
||||
6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred
|
||||
privacy pass.
|
||||
Reference in New Issue
Block a user