zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add media archive and slopbot design docs

Browse Source
This commit is contained in:
Fisto
2026-09-16 07:47:28 +02:00
parent 677afc747f
commit b59b9db461
10 changed files with 453 additions and 197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+2
View File
@@ -6,3 +6,5 @@ node_modules
.pnpm-store/
# Ops notes with credentials; never commit.
GITEA.md
# Media archive runtime data: scrape databases, downloads, previews.
media-scrape/db/
AGENTS.md
+9
View File
@@ -16,6 +16,10 @@ before you stop.
- [daemon.md](docs/daemon.md) — the external MCP daemon
- [webui.md](docs/webui.md) — the web control UI and its JSON API
- [voice.md](docs/voice.md) — voice state reads, planned actions, and lifecycle
- [media.md](docs/media.md) — media archive: command, storage, limits, and Web UI
- [media-scraper.md](docs/media-scraper.md) — the scrape engine: backends, walk, pacing, schema
- [media-mcp.md](docs/media-mcp.md) — the media MCP tool surface
- [slopbot.md](docs/slopbot.md) — the dedicated bot core for media scraping
- [communication.md](docs/communication.md) — trust model, commands, trigger log
- [auth.md](docs/auth.md) — QR login and token persistence
- [testing.md](docs/testing.md) — tests and live test setup
@@ -42,6 +46,7 @@ before you stop.
- All outbound actions go through the paced queue.
- Triggers come from trusted users only: one-to-one DMs and direct mentions.
- Test with throwaway accounts and private test guilds only.
- Never commit `media-scrape/db/`, scrape databases, downloaded media, or the slopbot token.
- Live send tests: only DMs to user **zzib** (`177117296140091393`).
- Run `pnpm testTypes` and `pnpm lint` before you report done. Run `pnpm build` to compile.
@@ -69,6 +74,8 @@ before you stop.
| Voice video | Not started. Plan in [docs/voice.md](docs/voice.md). |
| Notification parsing (edits, deletes, reactions, replies) | Implemented |
| Reply-to-puppet as a trigger | Implemented |
| Media archive (`scrape media`, Media/Bot tabs) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). |
| Slopbot (dedicated bot core for scraping) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). |
| Auth persistence in the core | Implemented and verified live |
| CDP transport | Kept as a development fallback |
@@ -82,6 +89,8 @@ See [CURRENT.md](CURRENT.md) and [TODO.md](TODO.md) for the live list. The main
3. Per-user rate limit and command cooldowns.
4. Collector account with read-only voice membership.
5. Remote agent MCP path: notification tools and a documented LAN/Tailscale bind.
6. Media archive (`media-scrape/`, slopbot): work the checkpoints in [CURRENT.md](CURRENT.md) and
push the branch after each one.
Paused: the `poll` command and the setup command flow. Open decisions wait in
[CURRENT.md](CURRENT.md) → Open discussions.
CURRENT.md
+18 -5
View File
@@ -12,11 +12,24 @@ mobile draft at 390px.
## In progress
- Media scrape project, branch `media-scrape-project` on `veskslop-t`. Plan drafted at
[media-scrape/PLAN.md](media-scrape/PLAN.md) and awaiting owner review; no code yet. The command
adds `scrape media` to archive a channel's history over time with DiscordChatExporter-style rate
limiting, a SQLite store, and a new Media tab in the Web UI. Open questions in the plan: scope,
storage location, bytes vs. links, and tab placement.
- Media archive project, branch `media-scrape-project` on `veskslop-t`. Design approved 2026-09-16
(docs: [media.md](docs/media.md), [media-scraper.md](docs/media-scraper.md),
[slopbot.md](docs/slopbot.md), [media-mcp.md](docs/media-mcp.md)). Decisions: scrape the channel
the command is sent in; slopbot (discord.js bot core) for guild channels and the puppet core
bridge for group DMs; one self-contained SQLite + media dir per scrape under `media-scrape/db/`
(gitignored); links indexed by default, downloads optional, third-party embeds never downloaded;
30 GiB default limit with a 90% warning and 5% no-delete overrun, usage bar and clean button in
Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C7.
Checkpoints, push after each:
- [x] C1 design docs, AGENTS/INDEX pointers, `.gitignore`
- [ ] C2 core message shape: embeds, forwards, attachment metadata
- [ ] C3 media-scrape store and puppet walker, tests
- [ ] C4 slopbot backend, tests
- [ ] C5 Web UI Media and Bot tabs, API routes
- [ ] C6 storage limit, warning flow, clean modal, settings
- [ ] C7 MCP service hooks
- [ ] C8 live verification, tuning, docs
- Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
open; it carries the character sliders with engine defaults and +/-2 snapping, five named
presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and
docs/INDEX.md
+5 -1
View File
@@ -11,6 +11,10 @@ This file lists the project documents.
| [daemon.md](daemon.md) | The external MCP daemon |
| [webui.md](webui.md) | The web control UI and its JSON API |
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
| [media.md](media.md) | Media archive: command, storage, limits, and Web UI |
| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema |
| [media-mcp.md](media-mcp.md) | The media MCP tool surface |
| [slopbot.md](slopbot.md) | The dedicated bot core for media scraping |
| [agent.md](agent.md) | The local LLM harness and remote MCP agent access |
| [communication.md](communication.md) | Trust model, commands, and the trigger log |
| [auth.md](auth.md) | QR login and token persistence |
@@ -25,7 +29,7 @@ This file lists the project documents.
| [../CURRENT.md](../CURRENT.md) | Live session state |
| [../TODO.md](../TODO.md) | Ordered backlog |
| [../poc/README.md](../poc/README.md) | Spike and daemon run commands |
| [../media-scrape/PLAN.md](../media-scrape/PLAN.md) | Media scrape and archive plan (review draft) |
| [../media-scrape/README.md](../media-scrape/README.md) | Media scrape module layout |
| [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration |
| [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine |
| [../README.md](../README.md) | Project introduction |
docs/media-mcp.md
+66
View File
@@ -0,0 +1,66 @@
# Media MCP surface
The media archive is designed for agent access. The daemon exposes one service, and the MCP tools
are thin wrappers around it. The same service backs the Web UI routes, so the two clients cannot
drift.
Status: hooks designed, implementation at checkpoint C7 in [CURRENT.md](../CURRENT.md).
## Service interface
```ts
interface MediaService {
startScrape(input: { channelId: string; mode: "links" | "download"; backend?: "auto" | "slopbot" | "puppet" }): Promise<ScrapeJob>;
getScrape(scrapeId: string): Promise<ScrapeJob | null>;
listScrapes(): Promise<ScrapeJob[]>;
stopScrape(scrapeId: string): Promise<ScrapeJob>;
deleteScrape(scrapeId: string): Promise<void>;
queryMedia(query: MediaQuery): Promise<{ items: MediaItem[]; nextCursor: string | null }>;
getMediaItem(id: string): Promise<MediaItem | null>;
storageStatus(): Promise<StorageStatus>;
}
```
`MediaQuery`: `kind`, `scrapeId`, `channelId`, `guildId`, `authorId`, `after`, `before`, `text`,
`limit`, `cursor`.
`StorageStatus`: `usedBytes`, `limitBytes`, `warnPercent`, `overrunPercent`, `byKind`,
`scrapes: [{ id, bytes, state, createdAt }]`.
## Tools
| Tool | Access | Purpose |
| --- | --- | --- |
| `scrape_media` | trusted | Start a scrape (`channelId`, `mode`) |
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
| `stop_media_scrape` | trusted | Stop the active scrape |
| `query_media` | trusted | Filter and page through media items |
| `get_media_item` | trusted | One item with sources and local path |
| `get_media_storage` | trusted | Usage against the limit, per-kind breakdown |
| `delete_media_scrape` | admin | Delete a scrape and its files |
Read tools are cache-only against `index.db` and the scrape registry. They never call Discord.
`scrape_media` and `stop_media_scrape` drive the same job queue as the chat command, so pacing and
the action log apply.
## Rules
- Every tool goes through the existing trust tiers. `delete_media_scrape` and `mode: "download"`
need `admin`.
- Tool calls are written to the action log like any other daemon action.
- The storage limit is enforced for the agent exactly as for the chat command: near the limit,
`scrape_media` returns the warning and needs a second call with `confirm: true`.
- Media bytes are never returned inline. Tools return metadata, the original URL, and the local
path; an agent that wants the bytes reads the file through the daemon host.
## Hooks to leave in place before C7
1. `MediaService` is constructed once in the daemon and passed to both the Web UI routes and the
MCP registration. No globals.
2. `ScrapeJob` and `MediaItem` are plain JSON-serializable objects, the same shape the Web UI
consumes.
3. The job queue exposes `start`, `stop`, `list`, and `events` so MCP and the UI can subscribe to
the same stream.
4. Storage accounting is a service method, not UI code, so the MCP limit check uses the same
numbers as the Settings bar.
docs/media-scraper.md
+114
View File
@@ -0,0 +1,114 @@
# Media scraper
The scrape engine walks a channel's history and fills a per-scrape database. It is backend
agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs.
See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md).
## Backends
Both backends implement one interface:
```ts
interface ScrapeBackend {
fetchPage(channelId: string, before: string | null, limit: number):
Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}
```
| Backend | Implementation |
| --- | --- |
| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client |
| `puppet` | The core action `message.fetch` over the bridge |
The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the
guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same
backend only.
## Walk
1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
seen.
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.
Rules:
- One running scrape at a time. Further requests queue.
- `stop` marks the job and lets the current page finish.
- Every page is an outbound read. Sends from commands keep their own pacing.
## Pacing and backoff
The engine is deliberately slower than normal reads. All values are configurable; defaults are
starting points, not Discord facts.
| Setting | Default |
| --- | --- |
| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 |
| `media.pagesPerHour` | 300 |
| `media.pagesPerDay` | 3000 |
| `media.maxRetries` | 8 |
- A random gap in the page window between pages.
- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window
resets.
- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1`
seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping.
- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend
uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js
handle buckets.
## Classification
The extractor maps one message to zero or more media items and returns their kind, source URL,
provider, dimensions, duration when known, and size when known.
- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio.
- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with
`downloaded = false`. Image and gif embeds become cached previews only.
- Forwarded message snapshots are treated as messages with their own media.
- Stickers become images or gifs.
- Voice messages (`voiceMessage` flag) become audio.
- Third-party embeds are never downloaded, by rule.
## Database schema
One SQLite file per scrape (`index.db`, `node:sqlite`).
| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `duration_ms`, `downloaded`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.json`, and
`file_hash` for dedupe. The database is an index; the media folder holds bytes.
## Downloads and previews
- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
in `thumbs/`. A preview failure never fails the scrape.
## Job states
`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.
Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab
and the MCP `get_media_scrape` tool read the same fields.
## Tests
- Unit tests run against the mock core and a fake slopbot REST. No live Discord.
- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
- The live run is a private test guild first, then a real channel.
## Checkpoints
Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4.
docs/media.md
+136
View File
@@ -0,0 +1,136 @@
# Media archive
Status: design approved 2026-09-16. Implementation in progress. Checkpoints live in
[CURRENT.md](../CURRENT.md).
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
the archive, **Bot** controls slopbot and the storage settings.
The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.
## Command
```
@puppet scrape media # index links (default)
@puppet scrape media download # also download attachment bytes
@puppet scrape media status
@puppet scrape media stop
```
- The target is the channel the command is sent in. Group DMs and guild channels work.
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
into the confirmation (see Retention).
- The reply follows the response policy. The job runs in the background.
## Backends
The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.
| | slopbot (bot account) | puppet (user account) |
| --- | --- | --- |
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. |
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
| Requires the Electron client | No | Yes |
See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).
## Storage
Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:
```
media-scrape/db/
<scrape-id>/ # <channel-id>-<YYYYMMDD-HHMMSS>
scrape.json # channel, guild, mode, state, cursor, counters
index.db # SQLite metadata index
media/ # downloaded bytes, only in download mode
<sha256>.<ext>
thumbs/ # cached Discord-hosted image previews
<sha256>.webp
```
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
link between a media item and its source messages. Source channel, author, and timestamp are
indexed so filters can be added later without a re-scrape.
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
name is metadata.
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.
## Index versus download
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
downloaded.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count
toward the storage limit.
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.
## Media classification
| Kind | Sources |
| --- | --- |
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |
## Retention and limits
The storage limit covers everything under the DB root: databases, downloads, and previews.
| Setting | Default | Meaning |
| --- | --- | --- |
| `media.limitBytes` | 30 GiB | Combined size limit |
| `media.warnPercent` | 90 | Warn threshold |
| `media.overrunPercent` | 5 | No-delete allowance above the limit |
Behavior:
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
usage. Re-running the same command accepts. The confirmation lives for a short window.
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
explains this.
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
that deletion is irreversible. The delete button is disabled for a 5 second countdown.
## Web UI
- **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text
search. Grid with lazy loading; click opens a lightbox with the source message link, the
original URL, and a download button. A scrape-status strip shows progress with Start and Stop.
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
database overhead.
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
**Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
bar, and the clean button.
- Mobile is out of scope for now.
## MCP
The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
[media-mcp.md](media-mcp.md) and implemented at checkpoint C7.
## Security
- The command follows the existing trust rules and is written to the action log.
- The slopbot token lives in the state dir or the environment, never in the repo. See
[slopbot.md](slopbot.md).
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
hashes, never user input.
- The Web UI has no authentication. LAN or Tailscale only, as before.
- This automates a user account. The account should be one the owner can afford to lose.
docs/slopbot.md
+71
View File
@@ -0,0 +1,71 @@
# Slopbot
Slopbot is the dedicated bot account for media scraping. It is separate from the puppet and from
the Vesktop core. It uses the official Discord bot API through
[discord.js](https://github.com/discordjs/discord.js) (Apache-2.0), the external core the feature
is built on.
## Why a bot
For guild channels, a bot scrapes better than a user account:
- The official bot REST API has documented, per-route rate limits. discord.js queues by bucket and
honors `429` automatically.
- No Electron client, no Xvfb, no renderer round-trip. The walk continues while the puppet is
offline.
- The bot token is a separate credential with its own lifecycle.
Slopbot cannot scrape user DMs or group DMs. Discord does not expose them to bots. The puppet
backend covers those, and covers guild channels where slopbot is not present. This split is a
Discord fact, not a design choice.
Slopbot is a real bot account. It is not a selfbot: the locked decision against selfbot protocol
libraries stands.
## Setup
1. Create an application in the Discord Developer Portal and add a bot.
2. Enable the **Message Content** privileged intent. Without it Discord blanks `content`,
`attachments`, and `embeds` in both gateway events and REST history responses. Unverified apps
in under 100 servers can enable it in the portal.
3. Invite the bot with the `bot` and `applications.commands` scopes and the minimum permissions:
View Channels, Read Message History, Send Messages (for status replies), Embed Links.
4. Store the token in the daemon state file `slopbot.json` (mode 600) or `VESKSLOP_SLOPBOT_TOKEN`.
Never commit it. The action log redacts token-shaped strings as before.
Slopbot needs no slash commands: it is a scraping sensor, not a chat citizen. It never replies in
channels.
## Configuration
| Setting | Default | Purpose |
| --- | --- | --- |
| `VESKSLOP_SLOPBOT_ENABLED` | `0` | Kill switch, off by default |
| `VESKSLOP_SLOPBOT_TOKEN` / `slopbot.json` | unset | Bot token |
| `VESKSLOP_SLOPBOT_CLIENT_ID` | unset | For the invite URL helper |
## Code
- `media-scrape/slopbot/` holds the bot core: client lifecycle, the `ScrapeBackend` implementation,
and the invite helper.
- discord.js is a dependency of the daemon package (`poc/package.json`), the same way the MCP SDK
is. The code stays in `media-scrape/`; the daemon only mounts it.
- The Bot tab shows connection state, the guilds slopbot is in, the active scrape, and the queue.
It also renders the invite URL from the client ID.
## Rate limits
discord.js handles per-route buckets and retries `429` and `5xx` internally. The scrape engine adds
its own budgets on top (see [media-scraper.md](media-scraper.md)): page gap, hourly and daily caps.
Slopbot runs with the faster bot budget; the puppet backend uses the slower user budget.
## Security
- The bot token is a credential. It is redacted from logs, never committed, and loaded from the
state dir or the environment.
- The bot gets minimum permissions and only the guilds the owner invites it to.
- A guild channel the bot cannot see is a normal fallback to the puppet, not an error.
## Status
Design approved 2026-09-16. Implementation at checkpoint C4 in [CURRENT.md](../CURRENT.md).
media-scrape/PLAN.md
-191
View File
@@ -1,191 +0,0 @@
# Media scrape — design plan
Status: **review draft**. Nothing is implemented until the owner answers the open questions.
## Goal
Let a trusted user ask the puppet to archive the media history of a channel:
```
@puppet scrape media # the channel the command is sent in
@puppet scrape media #channel # a specific guild channel or thread
@puppet scrape media status
@puppet scrape media stop
```
The daemon walks the channel history over time and records every image, GIF, video (file or
embedded link), and audio attachment into a local store. A new **Media** tab in the Web UI browses
the store with type filters and a link back to the source message.
Constraints:
- The puppet is a full user account, not a bot. The walk uses the existing validated core bridge
(`message.fetch`), so the traffic still comes from the official client.
- The walk must not look like spam: conservative pacing, backoff, a resumable cursor, one job at a
time, and a configurable rate budget.
- The feature lives in its own folder, `media-scrape/`, and touches the daemon at two points: the
Web UI route mount and the MCP tool registration. The Vesktop core patch grows only inside
`mapMessage` in `src/renderer/puppet/actions.ts`.
## What existing scrapers do (research, 2026-09-16)
| Project | Stars | License | Approach | What we take |
| --- | --- | --- | --- | --- |
| [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) | 12k | MIT | Exports channels with a user **or** bot token over the REST API. | The rate-limit playbook: advisory headers, `Retry-After` handling, retry policy. |
| [Discord History Tracker](https://github.com/chylex/Discord-History-Tracker) | 584 | MIT | Does not call the API at all. A script runs inside the logged-in Discord client and reads its own store, with a configurable delay. | Validates the core-bridge approach: drive the official client, not raw HTTP. |
| [Discrub](https://github.com/pratherbytecraft/discrub) | 538 | Source-visible, no SPDX | Browser extension that uses the logged-in session to search, export, and purge. | The UI pattern (filter, browse, jump to source). Not the code. |
| [Discord-Media-Scraper](https://github.com/valentinemackenzie/Discord-Media-Scraper) | 2 | Apache-2.0 | Batch downloads media from servers, DMs, and forums with a self-bot. | Feature checklist only. |
| selfbot libraries (eris/discord.js forks) | — | — | Stale protocol emulation. | Nothing. Forbidden by the locked decisions. |
### Techniques we adopt from DiscordChatExporter
DCE is the reference for safe history walking. Its `DiscordClient` and `Http` files implement:
- On `429`, wait `Retry-After` + 1s, up to 8 attempts.
- On `408` and all `5xx`, retry with exponential backoff (`2^attempt + 1` seconds).
- Read `X-RateLimit-Remaining` and `X-RateLimit-Reset-After`; when the bucket is empty, wait out
the reset (plus 1s buffer, capped at 60s).
- A per-token-kind preference: user tokens are treated more carefully than bot tokens.
The daemon cannot see Discord's HTTP headers through the JSON-line core bridge. The core's Discord
client already retries internally, so the daemon adds its own conservative pacing and blind
exponential backoff on failures. If live testing shows it needs the headers, the fetch result can
carry them later.
## Account choice: user account, not a bot
| | Puppet (user account) | Bot account |
| --- | --- | --- |
| DMs and group DMs | Yes | **No.** Discord does not give bots user DMs or group DMs. |
| Guild channels | Every channel the account can see | Only channels the bot is invited to |
| Setup | The account is already in the call/guild | Needs an app, a token, and an invite with permissions |
| Rate limits | Undocumented, stricter, abuse-prone | Documented per-route buckets |
| Auth path | Existing core bridge | A second token and a second login path |
The primary case is a group chat, and bots cannot scrape group DMs. The user account is the only
path that covers it. A dedicated bot core stays an optional later phase (guild channels only, e.g.
`discord.js`, MIT, as a separate process that reads the daemon's store). The plan does not need it.
## Architecture
```
media-scrape/ new folder, self-contained
PLAN.md this file
src/ job engine, extractor, store, routes, MCP tools
test/ unit tests against the mock core
poc/src/webui.ts one mount: mediaRoutes(deps) under /api/media
poc/src/control-bridge.ts MCP tools: scrape_media, get_media, media_status
poc/public/ Media tab (HTML/CSS/JS, Nocturne tokens)
src/renderer/puppet/ mapMessage gains embeds, snapshots, size, attachment id, flags
```
The module gets a small injected interface (fetch page, logger, state dir, pacing) so it does not
import daemon internals and can be tested alone.
No new core action is needed. `message.fetch` already walks history with `before`/`after` and a
limit of 1-100. The only core change is the message shape in `mapMessage`:
- `attachments[].id`, `.size`, `.durationSecs`
- `embeds[]`: `type`, `url`, `provider`, `videoUrl`, `thumbnailUrl`
- forwarded message snapshots (`messageSnapshots`)
- `stickerItems[]`
- `voiceMessage` flag
Validation and the error model stay unchanged.
## Scraper engine
- One job per channel, one running job at a time, queue the rest.
- Walk newest to oldest: `message.fetch({ channelId, limit: 100, before: cursor })`; the cursor is
the oldest message ID seen. Stop when a page returns fewer than 100 items.
- Persist the cursor and the counters after every page, so a daemon restart resumes instead of
restarting.
- A gap between pages, default 1.2-3.5s random (`VESKSLOP_SCRAPE_PAGE_MIN_MS` / `_MAX_MS`),
configurable. Separate from the 100-300ms read gap and from the send pacing.
- A rolling budget, default conservative and configurable: max pages per hour and per day. The job
pauses when either is hit and resumes after the window.
- Backoff on failure: `Retry-After` when the error carries it; exponential otherwise; pause the
job after repeated failures instead of retrying forever.
- All outbound sends from commands still go through the normal paced queue. Scrape pages never
send.
- The job writes to the persistent action log (`scrape` entries) and reports progress to the Web
UI (`pages`, `messages`, `media`, `errors`, state).
## Media classification
| Kind | Sources |
| --- | --- |
| `image` | `png`, `jpg`, `jpeg`, `webp`, `bmp`, `avif` attachments; image embeds; stickers |
| `gif` | `gif` attachments, `gifv` / Tenor / Giphy embeds |
| `video` | `mp4`, `webm`, `mov`, `mkv` attachments; YouTube, Vimeo, Twitch clips, Streamable, and other video embeds; forwarded video |
| `audio` | `mp3`, `m4a`, `ogg`, `wav`, `flac` attachments; Discord voice messages (ogg) |
- Attachment bytes are downloaded (content-addressed), because Discord's attachment URLs are
signed and expire in about a day.
- Embedded video links are indexed, not downloaded: URL, provider, title or thumbnail when
available. The tab links out or embeds the provider player.
- Dedupe by SHA-256 of the bytes and by canonical URL. One media row can have many source messages.
- Downloads run in their own serial queue, one file at a time, with a 1-3s gap, a size cap
(default 50 MiB, configurable), a timeout, and 2 retries on network or 5xx errors. `4xx` means
the URL expired; the job re-fetches the message later to refresh it.
## Storage
- `node:sqlite` (`DatabaseSync`) is available in the local Node 22.23 without a new dependency.
If that proves awkward, fall back to `better-sqlite3`.
- Location: `<state dir>/media/` (`VESKSLOP_MEDIA_DIR`): `media.db`, `files/` (sha256 names),
`thumbs/` (optional, `ffmpeg` from the audio pipeline).
- Tables: `jobs`, `channels`, `messages`, `media`, `media_sources`, `download_queue`.
- Indexes on `(kind, timestamp)`, `channel_id`, `hash`.
- Quota and retention settings on the same file as the daemon preferences.
## Web UI: the Media tab
- New scope beside **Guild / DM** on desktop; on mobile a fifth tab (or an entry from Activity;
see open questions).
- Filters: All, Images, GIFs, Videos, Audio; source channel and guild; date range; text search on
filename, provider, and message content.
- Grid with lazy loading and a lightbox. The lightbox shows the source message link, the original
URL, dimensions or duration, and the download button.
- A job strip shows state, progress, and Start / Stop, matching the Transcript tab pattern.
- Routes: `GET /api/media`, `GET /api/media/:id`, `GET /api/media/:id/file`,
`GET /api/media/:id/thumb`, `POST /api/media/scrape`, `POST /api/media/scrape/stop`,
`GET /api/media/status`.
- No authentication, same as the rest of the UI. LAN or Tailscale only.
## Safety and guardrails
- The command follows the existing trust rules: trusted users only, DM or direct mention, logged.
- The Media command needs the `trusted` tier; `stop` and guild-wide scopes need `admin`.
- Content served from disk only when the sniffed type matches the extension allowlist.
- Files are stored by hash only. Original filenames are kept in the database, never used as paths.
- The scraping feature is opt-in per channel. Nothing is walked until someone runs the command.
- Respect the project test rules: build and test against the mock core; live-test in a private
test guild; the only live DM target stays zzib.
- Wiki and docs repeat the ToS caveat: this automates a user account. The account should be one
the owner can lose.
## Phases
1. **Review.** Owner answers the open questions below.
2. **Core shape.** Extend `mapMessage`, add fixtures, run `pnpm testTypes`, `pnpm lint`,
`pnpm build`.
3. **Engine.** `media-scrape/` walker, store, downloader, unit tests against the mock core.
4. **Web UI.** Media tab and API routes.
5. **Live run.** Private test guild, small channel first, measure page timing and any 429s, tune
the budgets, then a real channel.
6. **Optional later.** Live ingestion of new messages, thumbnails, the bot-core path, retention
policy.
## Open questions for the owner
1. **Scope.** Should `scrape media` take only the current channel, or also `#channel`, threads, and
a whole-guild mode?
2. **Bot or not.** Confirm: user account only for now, bot core deferred (guild channels only)?
3. **Storage.** Is SQLite under the daemon state dir acceptable, or does the database belong beside
the code in `media-scrape/`?
4. **Bytes or links.** Download attachment bytes (recommended; URLs expire), or index links only?
5. **Tab placement.** New top-level Media tab, or a pane inside an existing tab? What should the
mobile fifth tab be?
6. **Retention.** Any size cap or expiry policy for the archive? This interacts with the deferred
privacy pass.
media-scrape/README.md
+32
View File
@@ -0,0 +1,32 @@
# media-scrape
The media archive feature. Contained here on purpose: the daemon mounts it, but the logic does not
spread through `poc/src/`.
## Layout
```
media-scrape/
README.md this file
src/ scrape engine, store, extractor, Web UI routes, MCP tools
slopbot/ the bot core (discord.js backend)
test/ unit tests against the mock core and a fake bot REST
db/ runtime data, ignored by git
```
`db/` holds one directory per scrape: `scrape.json`, `index.db`, `media/`, `thumbs/`. It is
configurable with `VESKSLOP_MEDIA_DB_DIR` and never committed. See
[../docs/media.md](../docs/media.md).
The daemon imports this folder. discord.js is declared in `poc/package.json`, like the MCP SDK.
## Docs
- [../docs/media.md](../docs/media.md) — the feature, storage, limits, Web UI
- [../docs/media-scraper.md](../docs/media-scraper.md) — the walk, pacing, schema
- [../docs/slopbot.md](../docs/slopbot.md) — the bot core and its setup
- [../docs/media-mcp.md](../docs/media-mcp.md) — the MCP surface
## Status
Design approved 2026-09-16. Implementation checkpoints live in [../CURRENT.md](../CURRENT.md).