zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add media archive and slopbot design docs

Browse Source
This commit is contained in:
Fisto
2026-09-16 07:47:28 +02:00
parent 677afc747f
commit b59b9db461
10 changed files with 453 additions and 197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/INDEX.md
+5 -1
View File
@@ -11,6 +11,10 @@ This file lists the project documents.
| [daemon.md](daemon.md) | The external MCP daemon |
| [webui.md](webui.md) | The web control UI and its JSON API |
| [voice.md](voice.md) | Voice state reads, planned actions, and lifecycle |
| [media.md](media.md) | Media archive: command, storage, limits, and Web UI |
| [media-scraper.md](media-scraper.md) | The scrape engine: backends, walk, pacing, schema |
| [media-mcp.md](media-mcp.md) | The media MCP tool surface |
| [slopbot.md](slopbot.md) | The dedicated bot core for media scraping |
| [agent.md](agent.md) | The local LLM harness and remote MCP agent access |
| [communication.md](communication.md) | Trust model, commands, and the trigger log |
| [auth.md](auth.md) | QR login and token persistence |
@@ -25,7 +29,7 @@ This file lists the project documents.
| [../CURRENT.md](../CURRENT.md) | Live session state |
| [../TODO.md](../TODO.md) | Ordered backlog |
| [../poc/README.md](../poc/README.md) | Spike and daemon run commands |
| [../media-scrape/PLAN.md](../media-scrape/PLAN.md) | Media scrape and archive plan (review draft) |
| [../media-scrape/README.md](../media-scrape/README.md) | Media scrape module layout |
| [../microslopsam/README.md](../microslopsam/README.md) | Bundled MicroslopSam SAPI4 TTS engine integration |
| [../TTS-DESIGN.md](../TTS-DESIGN.md) | Integration contract for an external TTS engine |
| [../README.md](../README.md) | Project introduction |
docs/media-mcp.md
+66
View File
@@ -0,0 +1,66 @@
# Media MCP surface
The media archive is designed for agent access. The daemon exposes one service, and the MCP tools
are thin wrappers around it. The same service backs the Web UI routes, so the two clients cannot
drift.
Status: hooks designed, implementation at checkpoint C7 in [CURRENT.md](../CURRENT.md).
## Service interface
```ts
interface MediaService {
startScrape(input: { channelId: string; mode: "links" | "download"; backend?: "auto" | "slopbot" | "puppet" }): Promise<ScrapeJob>;
getScrape(scrapeId: string): Promise<ScrapeJob | null>;
listScrapes(): Promise<ScrapeJob[]>;
stopScrape(scrapeId: string): Promise<ScrapeJob>;
deleteScrape(scrapeId: string): Promise<void>;
queryMedia(query: MediaQuery): Promise<{ items: MediaItem[]; nextCursor: string | null }>;
getMediaItem(id: string): Promise<MediaItem | null>;
storageStatus(): Promise<StorageStatus>;
}
```
`MediaQuery`: `kind`, `scrapeId`, `channelId`, `guildId`, `authorId`, `after`, `before`, `text`,
`limit`, `cursor`.
`StorageStatus`: `usedBytes`, `limitBytes`, `warnPercent`, `overrunPercent`, `byKind`,
`scrapes: [{ id, bytes, state, createdAt }]`.
## Tools
| Tool | Access | Purpose |
| --- | --- | --- |
| `scrape_media` | trusted | Start a scrape (`channelId`, `mode`) |
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
| `stop_media_scrape` | trusted | Stop the active scrape |
| `query_media` | trusted | Filter and page through media items |
| `get_media_item` | trusted | One item with sources and local path |
| `get_media_storage` | trusted | Usage against the limit, per-kind breakdown |
| `delete_media_scrape` | admin | Delete a scrape and its files |
Read tools are cache-only against `index.db` and the scrape registry. They never call Discord.
`scrape_media` and `stop_media_scrape` drive the same job queue as the chat command, so pacing and
the action log apply.
## Rules
- Every tool goes through the existing trust tiers. `delete_media_scrape` and `mode: "download"`
need `admin`.
- Tool calls are written to the action log like any other daemon action.
- The storage limit is enforced for the agent exactly as for the chat command: near the limit,
`scrape_media` returns the warning and needs a second call with `confirm: true`.
- Media bytes are never returned inline. Tools return metadata, the original URL, and the local
path; an agent that wants the bytes reads the file through the daemon host.
## Hooks to leave in place before C7
1. `MediaService` is constructed once in the daemon and passed to both the Web UI routes and the
MCP registration. No globals.
2. `ScrapeJob` and `MediaItem` are plain JSON-serializable objects, the same shape the Web UI
consumes.
3. The job queue exposes `start`, `stop`, `list`, and `events` so MCP and the UI can subscribe to
the same stream.
4. Storage accounting is a service method, not UI code, so the MCP limit check uses the same
numbers as the Settings bar.
docs/media-scraper.md
+114
View File
@@ -0,0 +1,114 @@
# Media scraper
The scrape engine walks a channel's history and fills a per-scrape database. It is backend
agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs.
See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md).
## Backends
Both backends implement one interface:
```ts
interface ScrapeBackend {
fetchPage(channelId: string, before: string | null, limit: number):
Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}
```
| Backend | Implementation |
| --- | --- |
| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client |
| `puppet` | The core action `message.fetch` over the bridge |
The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the
guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same
backend only.
## Walk
1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
seen.
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.
Rules:
- One running scrape at a time. Further requests queue.
- `stop` marks the job and lets the current page finish.
- Every page is an outbound read. Sends from commands keep their own pacing.
## Pacing and backoff
The engine is deliberately slower than normal reads. All values are configurable; defaults are
starting points, not Discord facts.
| Setting | Default |
| --- | --- |
| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 |
| `media.pagesPerHour` | 300 |
| `media.pagesPerDay` | 3000 |
| `media.maxRetries` | 8 |
- A random gap in the page window between pages.
- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window
resets.
- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1`
seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping.
- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend
uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js
handle buckets.
## Classification
The extractor maps one message to zero or more media items and returns their kind, source URL,
provider, dimensions, duration when known, and size when known.
- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio.
- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with
`downloaded = false`. Image and gif embeds become cached previews only.
- Forwarded message snapshots are treated as messages with their own media.
- Stickers become images or gifs.
- Voice messages (`voiceMessage` flag) become audio.
- Third-party embeds are never downloaded, by rule.
## Database schema
One SQLite file per scrape (`index.db`, `node:sqlite`).
| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `duration_ms`, `downloaded`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.json`, and
`file_hash` for dedupe. The database is an index; the media folder holds bytes.
## Downloads and previews
- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
in `thumbs/`. A preview failure never fails the scrape.
## Job states
`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.
Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab
and the MCP `get_media_scrape` tool read the same fields.
## Tests
- Unit tests run against the mock core and a fake slopbot REST. No live Discord.
- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
- The live run is a private test guild first, then a real channel.
## Checkpoints
Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4.
docs/media.md
+136
View File
@@ -0,0 +1,136 @@
# Media archive
Status: design approved 2026-09-16. Implementation in progress. Checkpoints live in
[CURRENT.md](../CURRENT.md).
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
the archive, **Bot** controls slopbot and the storage settings.
The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.
## Command
```
@puppet scrape media # index links (default)
@puppet scrape media download # also download attachment bytes
@puppet scrape media status
@puppet scrape media stop
```
- The target is the channel the command is sent in. Group DMs and guild channels work.
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
into the confirmation (see Retention).
- The reply follows the response policy. The job runs in the background.
## Backends
The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.
| | slopbot (bot account) | puppet (user account) |
| --- | --- | --- |
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. |
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
| Requires the Electron client | No | Yes |
See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).
## Storage
Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:
```
media-scrape/db/
<scrape-id>/ # <channel-id>-<YYYYMMDD-HHMMSS>
scrape.json # channel, guild, mode, state, cursor, counters
index.db # SQLite metadata index
media/ # downloaded bytes, only in download mode
<sha256>.<ext>
thumbs/ # cached Discord-hosted image previews
<sha256>.webp
```
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
link between a media item and its source messages. Source channel, author, and timestamp are
indexed so filters can be added later without a re-scrape.
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
name is metadata.
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.
## Index versus download
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
downloaded.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count
toward the storage limit.
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.
## Media classification
| Kind | Sources |
| --- | --- |
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |
## Retention and limits
The storage limit covers everything under the DB root: databases, downloads, and previews.
| Setting | Default | Meaning |
| --- | --- | --- |
| `media.limitBytes` | 30 GiB | Combined size limit |
| `media.warnPercent` | 90 | Warn threshold |
| `media.overrunPercent` | 5 | No-delete allowance above the limit |
Behavior:
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
usage. Re-running the same command accepts. The confirmation lives for a short window.
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
explains this.
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
that deletion is irreversible. The delete button is disabled for a 5 second countdown.
## Web UI
- **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text
search. Grid with lazy loading; click opens a lightbox with the source message link, the
original URL, and a download button. A scrape-status strip shows progress with Start and Stop.
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
database overhead.
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
**Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
bar, and the clean button.
- Mobile is out of scope for now.
## MCP
The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
[media-mcp.md](media-mcp.md) and implemented at checkpoint C7.
## Security
- The command follows the existing trust rules and is written to the action log.
- The slopbot token lives in the state dir or the environment, never in the repo. See
[slopbot.md](slopbot.md).
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
hashes, never user input.
- The Web UI has no authentication. LAN or Tailscale only, as before.
- This automates a user account. The account should be one the owner can afford to lose.
docs/slopbot.md
+71
View File
@@ -0,0 +1,71 @@
# Slopbot
Slopbot is the dedicated bot account for media scraping. It is separate from the puppet and from
the Vesktop core. It uses the official Discord bot API through
[discord.js](https://github.com/discordjs/discord.js) (Apache-2.0), the external core the feature
is built on.
## Why a bot
For guild channels, a bot scrapes better than a user account:
- The official bot REST API has documented, per-route rate limits. discord.js queues by bucket and
honors `429` automatically.
- No Electron client, no Xvfb, no renderer round-trip. The walk continues while the puppet is
offline.
- The bot token is a separate credential with its own lifecycle.
Slopbot cannot scrape user DMs or group DMs. Discord does not expose them to bots. The puppet
backend covers those, and covers guild channels where slopbot is not present. This split is a
Discord fact, not a design choice.
Slopbot is a real bot account. It is not a selfbot: the locked decision against selfbot protocol
libraries stands.
## Setup
1. Create an application in the Discord Developer Portal and add a bot.
2. Enable the **Message Content** privileged intent. Without it Discord blanks `content`,
`attachments`, and `embeds` in both gateway events and REST history responses. Unverified apps
in under 100 servers can enable it in the portal.
3. Invite the bot with the `bot` and `applications.commands` scopes and the minimum permissions:
View Channels, Read Message History, Send Messages (for status replies), Embed Links.
4. Store the token in the daemon state file `slopbot.json` (mode 600) or `VESKSLOP_SLOPBOT_TOKEN`.
Never commit it. The action log redacts token-shaped strings as before.
Slopbot needs no slash commands: it is a scraping sensor, not a chat citizen. It never replies in
channels.
## Configuration
| Setting | Default | Purpose |
| --- | --- | --- |
| `VESKSLOP_SLOPBOT_ENABLED` | `0` | Kill switch, off by default |
| `VESKSLOP_SLOPBOT_TOKEN` / `slopbot.json` | unset | Bot token |
| `VESKSLOP_SLOPBOT_CLIENT_ID` | unset | For the invite URL helper |
## Code
- `media-scrape/slopbot/` holds the bot core: client lifecycle, the `ScrapeBackend` implementation,
and the invite helper.
- discord.js is a dependency of the daemon package (`poc/package.json`), the same way the MCP SDK
is. The code stays in `media-scrape/`; the daemon only mounts it.
- The Bot tab shows connection state, the guilds slopbot is in, the active scrape, and the queue.
It also renders the invite URL from the client ID.
## Rate limits
discord.js handles per-route buckets and retries `429` and `5xx` internally. The scrape engine adds
its own budgets on top (see [media-scraper.md](media-scraper.md)): page gap, hourly and daily caps.
Slopbot runs with the faster bot budget; the puppet backend uses the slower user budget.
## Security
- The bot token is a credential. It is redacted from logs, never committed, and loaded from the
state dir or the environment.
- The bot gets minimum permissions and only the guilds the owner invites it to.
- A guild channel the bot cannot see is a normal fallback to the puppet, not an error.
## Status
Design approved 2026-09-16. Implementation at checkpoint C4 in [CURRENT.md](../CURRENT.md).