From b59b9db4616ba85b2e51306937d2daf8e5824f79 Mon Sep 17 00:00:00 2001 From: Fisto Date: Wed, 16 Sep 2026 07:47:28 +0200 Subject: [PATCH] Add media archive and slopbot design docs --- .gitignore | 2 + AGENTS.md | 9 ++ CURRENT.md | 23 +++-- docs/INDEX.md | 6 +- docs/media-mcp.md | 66 ++++++++++++++ docs/media-scraper.md | 114 ++++++++++++++++++++++++ docs/media.md | 136 +++++++++++++++++++++++++++++ docs/slopbot.md | 71 +++++++++++++++ media-scrape/PLAN.md | 191 ----------------------------------------- media-scrape/README.md | 32 +++++++ 10 files changed, 453 insertions(+), 197 deletions(-) create mode 100644 docs/media-mcp.md create mode 100644 docs/media-scraper.md create mode 100644 docs/media.md create mode 100644 docs/slopbot.md delete mode 100644 media-scrape/PLAN.md create mode 100644 media-scrape/README.md diff --git a/.gitignore b/.gitignore index 7a10a1a..28903b9 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,5 @@ node_modules .pnpm-store/ # Ops notes with credentials; never commit. GITEA.md +# Media archive runtime data: scrape databases, downloads, previews. +media-scrape/db/ diff --git a/AGENTS.md b/AGENTS.md index fc1144b..27ffba5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,10 @@ before you stop. - [daemon.md](docs/daemon.md) — the external MCP daemon - [webui.md](docs/webui.md) — the web control UI and its JSON API - [voice.md](docs/voice.md) — voice state reads, planned actions, and lifecycle +- [media.md](docs/media.md) — media archive: command, storage, limits, and Web UI +- [media-scraper.md](docs/media-scraper.md) — the scrape engine: backends, walk, pacing, schema +- [media-mcp.md](docs/media-mcp.md) — the media MCP tool surface +- [slopbot.md](docs/slopbot.md) — the dedicated bot core for media scraping - [communication.md](docs/communication.md) — trust model, commands, trigger log - [auth.md](docs/auth.md) — QR login and token persistence - [testing.md](docs/testing.md) — tests and live test setup @@ -42,6 +46,7 @@ before you stop. - All outbound actions go through the paced queue. - Triggers come from trusted users only: one-to-one DMs and direct mentions. - Test with throwaway accounts and private test guilds only. +- Never commit `media-scrape/db/`, scrape databases, downloaded media, or the slopbot token. - Live send tests: only DMs to user **zzib** (`177117296140091393`). - Run `pnpm testTypes` and `pnpm lint` before you report done. Run `pnpm build` to compile. @@ -69,6 +74,8 @@ before you stop. | Voice video | Not started. Plan in [docs/voice.md](docs/voice.md). | | Notification parsing (edits, deletes, reactions, replies) | Implemented | | Reply-to-puppet as a trigger | Implemented | +| Media archive (`scrape media`, Media/Bot tabs) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). | +| Slopbot (dedicated bot core for scraping) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). | | Auth persistence in the core | Implemented and verified live | | CDP transport | Kept as a development fallback | @@ -82,6 +89,8 @@ See [CURRENT.md](CURRENT.md) and [TODO.md](TODO.md) for the live list. The main 3. Per-user rate limit and command cooldowns. 4. Collector account with read-only voice membership. 5. Remote agent MCP path: notification tools and a documented LAN/Tailscale bind. +6. Media archive (`media-scrape/`, slopbot): work the checkpoints in [CURRENT.md](CURRENT.md) and + push the branch after each one. Paused: the `poll` command and the setup command flow. Open decisions wait in [CURRENT.md](CURRENT.md) → Open discussions. diff --git a/CURRENT.md b/CURRENT.md index 879a228..4788ca2 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -12,11 +12,24 @@ mobile draft at 390px. ## In progress -- Media scrape project, branch `media-scrape-project` on `veskslop-t`. Plan drafted at - [media-scrape/PLAN.md](media-scrape/PLAN.md) and awaiting owner review; no code yet. The command - adds `scrape media` to archive a channel's history over time with DiscordChatExporter-style rate - limiting, a SQLite store, and a new Media tab in the Web UI. Open questions in the plan: scope, - storage location, bytes vs. links, and tab placement. +- Media archive project, branch `media-scrape-project` on `veskslop-t`. Design approved 2026-09-16 + (docs: [media.md](docs/media.md), [media-scraper.md](docs/media-scraper.md), + [slopbot.md](docs/slopbot.md), [media-mcp.md](docs/media-mcp.md)). Decisions: scrape the channel + the command is sent in; slopbot (discord.js bot core) for guild channels and the puppet core + bridge for group DMs; one self-contained SQLite + media dir per scrape under `media-scrape/db/` + (gitignored); links indexed by default, downloads optional, third-party embeds never downloaded; + 30 GiB default limit with a 90% warning and 5% no-delete overrun, usage bar and clean button in + Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C7. + + Checkpoints, push after each: + - [x] C1 design docs, AGENTS/INDEX pointers, `.gitignore` + - [ ] C2 core message shape: embeds, forwards, attachment metadata + - [ ] C3 media-scrape store and puppet walker, tests + - [ ] C4 slopbot backend, tests + - [ ] C5 Web UI Media and Bot tabs, API routes + - [ ] C6 storage limit, warning flow, clean modal, settings + - [ ] C7 MCP service hooks + - [ ] C8 live verification, tuning, docs - Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each open; it carries the character sliders with engine defaults and +/-2 snapping, five named presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and diff --git a/docs/INDEX.md b/docs/INDEX.md index ce549bb..9c72101 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -11,6 +11,10 @@ This file lists the project documents. | [daemon.md](daemon.md) | The external MCP daemon | | [webui.md](webui.md) | The web control UI and its JSON API | | [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle | +| [media.md](media.md) | Media archive: command, storage, limits, and Web UI | +| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema | +| [media-mcp.md](media-mcp.md) | The media MCP tool surface | +| [slopbot.md](slopbot.md) | The dedicated bot core for media scraping | | [agent.md](agent.md) | The local LLM harness and remote MCP agent access | | [communication.md](communication.md) | Trust model, commands, and the trigger log | | [auth.md](auth.md) | QR login and token persistence | @@ -25,7 +29,7 @@ This file lists the project documents. | [../CURRENT.md](../CURRENT.md) | Live session state | | [../TODO.md](../TODO.md) | Ordered backlog | | [../poc/README.md](../poc/README.md) | Spike and daemon run commands | -| [../media-scrape/PLAN.md](../media-scrape/PLAN.md) | Media scrape and archive plan (review draft) | +| [../media-scrape/README.md](../media-scrape/README.md) | Media scrape module layout | | [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration | | [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine | | [../README.md](../README.md) | Project introduction | diff --git a/docs/media-mcp.md b/docs/media-mcp.md new file mode 100644 index 0000000..0ab188e --- /dev/null +++ b/docs/media-mcp.md @@ -0,0 +1,66 @@ +# Media MCP surface + +The media archive is designed for agent access. The daemon exposes one service, and the MCP tools +are thin wrappers around it. The same service backs the Web UI routes, so the two clients cannot +drift. + +Status: hooks designed, implementation at checkpoint C7 in [CURRENT.md](../CURRENT.md). + +## Service interface + +```ts +interface MediaService { + startScrape(input: { channelId: string; mode: "links" | "download"; backend?: "auto" | "slopbot" | "puppet" }): Promise; + getScrape(scrapeId: string): Promise; + listScrapes(): Promise; + stopScrape(scrapeId: string): Promise; + deleteScrape(scrapeId: string): Promise; + queryMedia(query: MediaQuery): Promise<{ items: MediaItem[]; nextCursor: string | null }>; + getMediaItem(id: string): Promise; + storageStatus(): Promise; +} +``` + +`MediaQuery`: `kind`, `scrapeId`, `channelId`, `guildId`, `authorId`, `after`, `before`, `text`, +`limit`, `cursor`. + +`StorageStatus`: `usedBytes`, `limitBytes`, `warnPercent`, `overrunPercent`, `byKind`, +`scrapes: [{ id, bytes, state, createdAt }]`. + +## Tools + +| Tool | Access | Purpose | +| --- | --- | --- | +| `scrape_media` | trusted | Start a scrape (`channelId`, `mode`) | +| `list_media_scrapes` | trusted | Every scrape with state, size, and counts | +| `get_media_scrape` | trusted | One scrape, live counters and cursor | +| `stop_media_scrape` | trusted | Stop the active scrape | +| `query_media` | trusted | Filter and page through media items | +| `get_media_item` | trusted | One item with sources and local path | +| `get_media_storage` | trusted | Usage against the limit, per-kind breakdown | +| `delete_media_scrape` | admin | Delete a scrape and its files | + +Read tools are cache-only against `index.db` and the scrape registry. They never call Discord. +`scrape_media` and `stop_media_scrape` drive the same job queue as the chat command, so pacing and +the action log apply. + +## Rules + +- Every tool goes through the existing trust tiers. `delete_media_scrape` and `mode: "download"` + need `admin`. +- Tool calls are written to the action log like any other daemon action. +- The storage limit is enforced for the agent exactly as for the chat command: near the limit, + `scrape_media` returns the warning and needs a second call with `confirm: true`. +- Media bytes are never returned inline. Tools return metadata, the original URL, and the local + path; an agent that wants the bytes reads the file through the daemon host. + +## Hooks to leave in place before C7 + +1. `MediaService` is constructed once in the daemon and passed to both the Web UI routes and the + MCP registration. No globals. +2. `ScrapeJob` and `MediaItem` are plain JSON-serializable objects, the same shape the Web UI + consumes. +3. The job queue exposes `start`, `stop`, `list`, and `events` so MCP and the UI can subscribe to + the same stream. +4. Storage accounting is a service method, not UI code, so the MCP limit check uses the same + numbers as the Settings bar. diff --git a/docs/media-scraper.md b/docs/media-scraper.md new file mode 100644 index 0000000..bcb11f9 --- /dev/null +++ b/docs/media-scraper.md @@ -0,0 +1,114 @@ +# Media scraper + +The scrape engine walks a channel's history and fills a per-scrape database. It is backend +agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs. +See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md). + +## Backends + +Both backends implement one interface: + +```ts +interface ScrapeBackend { + fetchPage(channelId: string, before: string | null, limit: number): + Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>; +} +``` + +| Backend | Implementation | +| --- | --- | +| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client | +| `puppet` | The core action `message.fetch` over the bridge | + +The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the +guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same +backend only. + +## Walk + +1. The command resolves the channel and creates a scrape directory. +2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID + seen. +3. Extract media from each message and write the page to the database in one transaction. +4. Persist `scrape.json` (cursor, counters, state) after every page. +5. Stop when a page returns fewer than `limit` messages or the oldest message is reached. +6. Resume after a daemon restart from the stored cursor. + +Rules: + +- One running scrape at a time. Further requests queue. +- `stop` marks the job and lets the current page finish. +- Every page is an outbound read. Sends from commands keep their own pacing. + +## Pacing and backoff + +The engine is deliberately slower than normal reads. All values are configurable; defaults are +starting points, not Discord facts. + +| Setting | Default | +| --- | --- | +| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 | +| `media.pagesPerHour` | 300 | +| `media.pagesPerDay` | 3000 | +| `media.maxRetries` | 8 | + +- A random gap in the page window between pages. +- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window + resets. +- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1` + seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping. +- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend + uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js + handle buckets. + +## Classification + +The extractor maps one message to zero or more media items and returns their kind, source URL, +provider, dimensions, duration when known, and size when known. + +- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio. +- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with + `downloaded = false`. Image and gif embeds become cached previews only. +- Forwarded message snapshots are treated as messages with their own media. +- Stickers become images or gifs. +- Voice messages (`voiceMessage` flag) become audio. +- Third-party embeds are never downloaded, by rule. + +## Database schema + +One SQLite file per scrape (`index.db`, `node:sqlite`). + +| Table | Columns | +| --- | --- | +| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` | +| `media` | `id`, `kind`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `duration_ms`, `downloaded`, `created_at` | +| `media_sources` | `media_id`, `message_id` | +| `downloads` | `media_id`, `state`, `attempts`, `error` | + +Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.json`, and +`file_hash` for dedupe. The database is an index; the media folder holds bytes. + +## Downloads and previews + +- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap + (`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors. +- `4xx` marks the URL expired. The item is re-fetched with its message later. +- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash + in `thumbs/`. A preview failure never fails the scrape. + +## Job states + +`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`. + +Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab +and the MCP `get_media_scrape` tool read the same fields. + +## Tests + +- Unit tests run against the mock core and a fake slopbot REST. No live Discord. +- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff. +- The live run is a private test guild first, then a real channel. + +## Checkpoints + +Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4. diff --git a/docs/media.md b/docs/media.md new file mode 100644 index 0000000..78e6989 --- /dev/null +++ b/docs/media.md @@ -0,0 +1,136 @@ +# Media archive + +Status: design approved 2026-09-16. Implementation in progress. Checkpoints live in +[CURRENT.md](../CURRENT.md). + +The media archive saves the media history of a channel. A trusted user runs `scrape media` in the +channel. The daemon walks the history through a scrape backend and indexes every image, GIF, +video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses +the archive, **Bot** controls slopbot and the storage settings. + +The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three +points: the Web UI route mount, the MCP tool registration, and the message shape in the core. + +## Command + +``` +@puppet scrape media # index links (default) +@puppet scrape media download # also download attachment bytes +@puppet scrape media status +@puppet scrape media stop +``` + +- The target is the channel the command is sent in. Group DMs and guild channels work. +- The command needs the `trusted` tier. `stop` and `download` needs `admin`. +- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns + into the confirmation (see Retention). +- The reply follows the response policy. The job runs in the background. + +## Backends + +The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one. + +| | slopbot (bot account) | puppet (user account) | +| --- | --- | --- | +| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. | +| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. | +| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) | +| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff | +| Requires the Electron client | No | Yes | + +See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md). + +## Storage + +Runtime data lives under `media-scrape/db/`, ignored by git and configurable with +`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained: + +``` +media-scrape/db/ + / # - + scrape.json # channel, guild, mode, state, cursor, counters + index.db # SQLite metadata index + media/ # downloaded bytes, only in download mode + . + thumbs/ # cached Discord-hosted image previews + .webp +``` + +- The database is an index. It holds the scrape state, the messages seen, the media items, and the + link between a media item and its source messages. Source channel, author, and timestamp are + indexed so filters can be added later without a re-scrape. +- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original + name is metadata. +- Deleting a scrape is deleting its directory. Nothing is shared between scrapes. + +## Index versus download + +- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is + downloaded. +- `download` fetches attachment bytes into `media/`. Deduplicated by hash. +- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other + embed stays a link with metadata. +- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN + resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count + toward the storage limit. +- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is + refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry. + +## Media classification + +| Kind | Sources | +| --- | --- | +| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers | +| `gif` | gif attachments; gifv, Tenor, Giphy embeds | +| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video | +| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages | + +## Retention and limits + +The storage limit covers everything under the DB root: databases, downloads, and previews. + +| Setting | Default | Meaning | +| --- | --- | --- | +| `media.limitBytes` | 30 GiB | Combined size limit | +| `media.warnPercent` | 90 | Warn threshold | +| `media.overrunPercent` | 5 | No-delete allowance above the limit | + +Behavior: + +1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the + usage. Re-running the same command accepts. The confirmation lives for a short window. +2. At or above 100% the oldest completed scrapes are marked `pending-delete`. +3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits. +4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field) + explains this. + +**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states +that deletion is irreversible. The delete button is disabled for a 5 second countdown. + +## Web UI + +- **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text + search. Grid with lazy loading; click opens a lightbox with the source message link, the + original URL, and a download button. A scrape-status strip shows progress with Start and Stop. +- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar. + The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the + database overhead. +- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and + **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage + bar, and the clean button. +- Mobile is out of scope for now. + +## MCP + +The agent can query the archive and start scrapes with MCP tools. The hooks are designed in +[media-mcp.md](media-mcp.md) and implemented at checkpoint C7. + +## Security + +- The command follows the existing trust rules and is written to the action log. +- The slopbot token lives in the state dir or the environment, never in the repo. See + [slopbot.md](slopbot.md). +- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are + hashes, never user input. +- The Web UI has no authentication. LAN or Tailscale only, as before. +- This automates a user account. The account should be one the owner can afford to lose. diff --git a/docs/slopbot.md b/docs/slopbot.md new file mode 100644 index 0000000..6ade421 --- /dev/null +++ b/docs/slopbot.md @@ -0,0 +1,71 @@ +# Slopbot + +Slopbot is the dedicated bot account for media scraping. It is separate from the puppet and from +the Vesktop core. It uses the official Discord bot API through +[discord.js](https://github.com/discordjs/discord.js) (Apache-2.0), the external core the feature +is built on. + +## Why a bot + +For guild channels, a bot scrapes better than a user account: + +- The official bot REST API has documented, per-route rate limits. discord.js queues by bucket and + honors `429` automatically. +- No Electron client, no Xvfb, no renderer round-trip. The walk continues while the puppet is + offline. +- The bot token is a separate credential with its own lifecycle. + +Slopbot cannot scrape user DMs or group DMs. Discord does not expose them to bots. The puppet +backend covers those, and covers guild channels where slopbot is not present. This split is a +Discord fact, not a design choice. + +Slopbot is a real bot account. It is not a selfbot: the locked decision against selfbot protocol +libraries stands. + +## Setup + +1. Create an application in the Discord Developer Portal and add a bot. +2. Enable the **Message Content** privileged intent. Without it Discord blanks `content`, + `attachments`, and `embeds` in both gateway events and REST history responses. Unverified apps + in under 100 servers can enable it in the portal. +3. Invite the bot with the `bot` and `applications.commands` scopes and the minimum permissions: + View Channels, Read Message History, Send Messages (for status replies), Embed Links. +4. Store the token in the daemon state file `slopbot.json` (mode 600) or `VESKSLOP_SLOPBOT_TOKEN`. + Never commit it. The action log redacts token-shaped strings as before. + +Slopbot needs no slash commands: it is a scraping sensor, not a chat citizen. It never replies in +channels. + +## Configuration + +| Setting | Default | Purpose | +| --- | --- | --- | +| `VESKSLOP_SLOPBOT_ENABLED` | `0` | Kill switch, off by default | +| `VESKSLOP_SLOPBOT_TOKEN` / `slopbot.json` | unset | Bot token | +| `VESKSLOP_SLOPBOT_CLIENT_ID` | unset | For the invite URL helper | + +## Code + +- `media-scrape/slopbot/` holds the bot core: client lifecycle, the `ScrapeBackend` implementation, + and the invite helper. +- discord.js is a dependency of the daemon package (`poc/package.json`), the same way the MCP SDK + is. The code stays in `media-scrape/`; the daemon only mounts it. +- The Bot tab shows connection state, the guilds slopbot is in, the active scrape, and the queue. + It also renders the invite URL from the client ID. + +## Rate limits + +discord.js handles per-route buckets and retries `429` and `5xx` internally. The scrape engine adds +its own budgets on top (see [media-scraper.md](media-scraper.md)): page gap, hourly and daily caps. +Slopbot runs with the faster bot budget; the puppet backend uses the slower user budget. + +## Security + +- The bot token is a credential. It is redacted from logs, never committed, and loaded from the + state dir or the environment. +- The bot gets minimum permissions and only the guilds the owner invites it to. +- A guild channel the bot cannot see is a normal fallback to the puppet, not an error. + +## Status + +Design approved 2026-09-16. Implementation at checkpoint C4 in [CURRENT.md](../CURRENT.md). diff --git a/media-scrape/PLAN.md b/media-scrape/PLAN.md deleted file mode 100644 index 7563ff5..0000000 --- a/media-scrape/PLAN.md +++ /dev/null @@ -1,191 +0,0 @@ -# Media scrape — design plan - -Status: **review draft**. Nothing is implemented until the owner answers the open questions. - -## Goal - -Let a trusted user ask the puppet to archive the media history of a channel: - -``` -@puppet scrape media # the channel the command is sent in -@puppet scrape media #channel # a specific guild channel or thread -@puppet scrape media status -@puppet scrape media stop -``` - -The daemon walks the channel history over time and records every image, GIF, video (file or -embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses -the store with type filters and a link back to the source message. - -Constraints: - -- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge - (`message.fetch`), so the traffic still comes from the official client. -- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a - time, and a configurable rate budget. -- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the - Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside - `mapMessage` in `src/renderer/puppet/actions.ts`. - -## What existing scrapers do (research, 2026-09-16) - -| Project | Stars | License | Approach | What we take | -| --- | --- | --- | --- | --- | -| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. | -| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. | -| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. | -| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. | -| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. | - -### Techniques we adopt from DiscordChatExporter - -DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement: - -- On `429`, wait `Retry-After` + 1s, up to 8 attempts. -- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds). -- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out - the reset (plus 1s buffer, capped at 60s). -- A per-token-kind preference: user tokens are treated more carefully than bot tokens. - -The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord -client already retries internally, so the daemon adds its own conservative pacing and blind -exponential backoff on failures. If live testing shows it needs the headers, the fetch result can -carry them later. - -## Account choice: user account, not a bot - -| | Puppet (user account) | Bot account | -| --- | --- | --- | -| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. | -| Guild channels | Every channel the account can see | Only channels the bot is invited to | -| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions | -| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets | -| Auth path | Existing core bridge | A second token and a second login path | - -The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only -path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g. -`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it. - -## Architecture - -``` -media-scrape/ new folder, self-contained - PLAN.md this file - src/ job engine, extractor, store, routes, MCP tools - test/ unit tests against the mock core -poc/src/webui.ts one mount: mediaRoutes(deps) under /api/media -poc/src/control-bridge.ts MCP tools: scrape_media, get_media, media_status -poc/public/ Media tab (HTML/CSS/JS, Nocturne tokens) -src/renderer/puppet/ mapMessage gains embeds, snapshots, size, attachment id, flags -``` - -The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not -import daemon internals and can be tested alone. - -No new core action is needed. `message.fetch` already walks history with `before`/`after` and a -limit of 1-100. The only core change is the message shape in `mapMessage`: - -- `attachments[].id`, `.size`, `.durationSecs` -- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl` -- forwarded message snapshots (`messageSnapshots`) -- `stickerItems[]` -- `voiceMessage` flag - -Validation and the error model stay unchanged. - -## Scraper engine - -- One job per channel, one running job at a time, queue the rest. -- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is - the oldest message ID seen. Stop when a page returns fewer than 100 items. -- Persist the cursor and the counters after every page, so a daemon restart resumes instead of - restarting. -- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`), - configurable. Separate from the 100-300ms read gap and from the send pacing. -- A rolling budget, default conservative and configurable: max pages per hour and per day. The job - pauses when either is hit and resumes after the window. -- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the - job after repeated failures instead of retrying forever. -- All outbound sends from commands still go through the normal paced queue. Scrape pages never - send. -- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web - UI (`pages`, `messages`, `media`, `errors`, state). - -## Media classification - -| Kind | Sources | -| --- | --- | -| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers | -| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds | -| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video | -| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) | - -- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are - signed and expire in about a day. -- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when - available. The tab links out or embeds the provider player. -- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages. -- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap - (default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means - the URL expired; the job re-fetches the message later to refresh it. - -## Storage - -- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency. - If that proves awkward, fall back to `better-sqlite3`. -- Location: `/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names), - `thumbs/` (optional, `ffmpeg` from the audio pipeline). -- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`. -- Indexes on `(kind, timestamp)`, `channel_id`, `hash`. -- Quota and retention settings on the same file as the daemon preferences. - -## Web UI: the Media tab - -- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity; - see open questions). -- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on - filename, provider, and message content. -- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original - URL, dimensions or duration, and the download button. -- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern. -- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`, - `GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`, - `GET /api/media/status`. -- No authentication, same as the rest of the UI. LAN or Tailscale only. - -## Safety and guardrails - -- The command follows the existing trust rules: trusted users only, DM or direct mention, logged. -- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`. -- Content served from disk only when the sniffed type matches the extension allowlist. -- Files are stored by hash only. Original filenames are kept in the database, never used as paths. -- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command. -- Respect the project test rules: build and test against the mock core; live-test in a private - test guild; the only live DM target stays zzib. -- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one - the owner can lose. - -## Phases - -1. **Review.** Owner answers the open questions below. -2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`, - `pnpm build`. -3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core. -4. **Web UI.** Media tab and API routes. -5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune - the budgets, then a real channel. -6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention - policy. - -## Open questions for the owner - -1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and - a whole-guild mode? -2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)? -3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside - the code in `media-scrape/`? -4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only? -5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the - mobile fifth tab be? -6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred - privacy pass. diff --git a/media-scrape/README.md b/media-scrape/README.md new file mode 100644 index 0000000..32b1f78 --- /dev/null +++ b/media-scrape/README.md @@ -0,0 +1,32 @@ +# media-scrape + +The media archive feature. Contained here on purpose: the daemon mounts it, but the logic does not +spread through `poc/src/`. + +## Layout + +``` +media-scrape/ + README.md this file + src/ scrape engine, store, extractor, Web UI routes, MCP tools + slopbot/ the bot core (discord.js backend) + test/ unit tests against the mock core and a fake bot REST + db/ runtime data, ignored by git +``` + +`db/` holds one directory per scrape: `scrape.json`, `index.db`, `media/`, `thumbs/`. It is +configurable with `VESKSLOP_MEDIA_DB_DIR` and never committed. See +[../docs/media.md](../docs/media.md). + +The daemon imports this folder. discord.js is declared in `poc/package.json`, like the MCP SDK. + +## Docs + +- [../docs/media.md](../docs/media.md) — the feature, storage, limits, Web UI +- [../docs/media-scraper.md](../docs/media-scraper.md) — the walk, pacing, schema +- [../docs/slopbot.md](../docs/slopbot.md) — the bot core and its setup +- [../docs/media-mcp.md](../docs/media-mcp.md) — the MCP surface + +## Status + +Design approved 2026-09-16. Implementation checkpoints live in [../CURRENT.md](../CURRENT.md).