Add media archive and slopbot design docs
This commit is contained in:
Fisto
+136
@@ -0,0 +1,136 @@
|
||||
# Media archive
|
||||
|
||||
Status: design approved 2026-09-16. Implementation in progress. Checkpoints live in
|
||||
[CURRENT.md](../CURRENT.md).
|
||||
|
||||
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
|
||||
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
|
||||
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
|
||||
the archive, **Bot** controls slopbot and the storage settings.
|
||||
|
||||
The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
|
||||
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.
|
||||
|
||||
## Command
|
||||
|
||||
```
|
||||
@puppet scrape media # index links (default)
|
||||
@puppet scrape media download # also download attachment bytes
|
||||
@puppet scrape media status
|
||||
@puppet scrape media stop
|
||||
```
|
||||
|
||||
- The target is the channel the command is sent in. Group DMs and guild channels work.
|
||||
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
|
||||
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
|
||||
into the confirmation (see Retention).
|
||||
- The reply follows the response policy. The job runs in the background.
|
||||
|
||||
## Backends
|
||||
|
||||
The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.
|
||||
|
||||
| | slopbot (bot account) | puppet (user account) |
|
||||
| --- | --- | --- |
|
||||
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
|
||||
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. |
|
||||
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
|
||||
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
|
||||
| Requires the Electron client | No | Yes |
|
||||
|
||||
See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).
|
||||
|
||||
## Storage
|
||||
|
||||
Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
|
||||
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:
|
||||
|
||||
```
|
||||
media-scrape/db/
|
||||
<scrape-id>/ # <channel-id>-<YYYYMMDD-HHMMSS>
|
||||
scrape.json # channel, guild, mode, state, cursor, counters
|
||||
index.db # SQLite metadata index
|
||||
media/ # downloaded bytes, only in download mode
|
||||
<sha256>.<ext>
|
||||
thumbs/ # cached Discord-hosted image previews
|
||||
<sha256>.webp
|
||||
```
|
||||
|
||||
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
|
||||
link between a media item and its source messages. Source channel, author, and timestamp are
|
||||
indexed so filters can be added later without a re-scrape.
|
||||
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
|
||||
name is metadata.
|
||||
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.
|
||||
|
||||
## Index versus download
|
||||
|
||||
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
|
||||
downloaded.
|
||||
- `download` fetches attachment bytes into `media/`. Deduplicated by hash.
|
||||
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
|
||||
embed stays a link with metadata.
|
||||
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
|
||||
resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count
|
||||
toward the storage limit.
|
||||
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
|
||||
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.
|
||||
|
||||
## Media classification
|
||||
|
||||
| Kind | Sources |
|
||||
| --- | --- |
|
||||
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
|
||||
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
|
||||
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
|
||||
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |
|
||||
|
||||
## Retention and limits
|
||||
|
||||
The storage limit covers everything under the DB root: databases, downloads, and previews.
|
||||
|
||||
| Setting | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `media.limitBytes` | 30 GiB | Combined size limit |
|
||||
| `media.warnPercent` | 90 | Warn threshold |
|
||||
| `media.overrunPercent` | 5 | No-delete allowance above the limit |
|
||||
|
||||
Behavior:
|
||||
|
||||
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
|
||||
usage. Re-running the same command accepts. The confirmation lives for a short window.
|
||||
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
|
||||
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
|
||||
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
|
||||
explains this.
|
||||
|
||||
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
|
||||
that deletion is irreversible. The delete button is disabled for a 5 second countdown.
|
||||
|
||||
## Web UI
|
||||
|
||||
- **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text
|
||||
search. Grid with lazy loading; click opens a lightbox with the source message link, the
|
||||
original URL, and a download button. A scrape-status strip shows progress with Start and Stop.
|
||||
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
|
||||
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
|
||||
database overhead.
|
||||
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
|
||||
**Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
|
||||
bar, and the clean button.
|
||||
- Mobile is out of scope for now.
|
||||
|
||||
## MCP
|
||||
|
||||
The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
|
||||
[media-mcp.md](media-mcp.md) and implemented at checkpoint C7.
|
||||
|
||||
## Security
|
||||
|
||||
- The command follows the existing trust rules and is written to the action log.
|
||||
- The slopbot token lives in the state dir or the environment, never in the repo. See
|
||||
[slopbot.md](slopbot.md).
|
||||
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
|
||||
hashes, never user input.
|
||||
- The Web UI has no authentication. LAN or Tailscale only, as before.
|
||||
- This automates a user account. The account should be one the owner can afford to lose.
|
||||
Reference in New Issue
Block a user