# Media archive Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command, the `/api/media/*` routes, storage limits with retention, the Settings group, and the MCP tools are implemented (C3-C8). The Media and Bot tabs are C6. The slopbot path is live-verified: the test channel produced 17 messages and 4 images + 13 videos with no errors. The puppet path and a `download` run are still owed (C9). Checkpoints live in [CURRENT.md](../CURRENT.md). The media archive saves the media history of a channel. A trusted user runs `scrape media` in the channel. The daemon walks the history through a scrape backend and indexes every image, GIF, video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses the archive, **Bot** controls slopbot and the storage settings. The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three points: the Web UI route mount, the MCP tool registration, and the message shape in the core. ## Command ``` @puppet scrape media # index links (default) @puppet scrape media download # also download attachment bytes @puppet scrape media status @puppet scrape media stop ``` - The target is the channel the command is sent in. Group DMs and guild channels work. - The command needs the `trusted` tier. `stop` and `download` needs `admin`. - `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns into the confirmation (see Retention). - The reply follows the response policy. The job runs in the background. ## Backends The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one. | | slopbot (bot account) | puppet (user account) | | --- | --- | --- | | Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. | | Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. | | Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) | | Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff | | Requires the Electron client | No | Yes | See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md). ## Storage Runtime data lives under `media-scrape/db/`, ignored by git and configurable with `VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained: ``` media-scrape/db/ / # - scrape.json # channel, guild, mode, state, cursor, counters index.db # SQLite metadata index media/ # downloaded bytes, only in download mode . thumbs/ # cached Discord-hosted image previews .webp ``` - The database is an index. It holds the scrape state, the messages seen, the media items, and the link between a media item and its source messages. Source channel, author, and timestamp are indexed so filters can be added later without a re-scrape. - Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes, `better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install. - The file name in `media/` is the SHA-256 of the bytes, never the original name. The original name is metadata. - Deleting a scrape is deleting its directory. Nothing is shared between scrapes. ## Index versus download - `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is downloaded. - `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet** (checkpoint C10): the command accepts the mode and the schema has the columns, but the download queue is still owed. - Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other embed stays a link with metadata. - Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count toward the storage limit. - Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry. ## Media classification | Kind | Sources | | --- | --- | | `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers | | `gif` | gif attachments; gifv, Tenor, Giphy embeds | | `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video | | `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages | ## Retention and limits The storage limit covers everything under the DB root: databases, downloads, and previews. | Setting | Default | Meaning | | --- | --- | --- | | `media.limitBytes` | 30 GiB | Combined size limit | | `media.warnPercent` | 90 | Warn threshold | | `media.overrunPercent` | 5 | No-delete allowance above the limit | Behavior: 1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true` on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`). 2. At or above 100% the oldest completed scrapes are marked `pending-delete`. 3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits. 4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field) explains this. The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`, `failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the scrape directory and forgets it in the engine. **Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states that deletion is irreversible. The delete button is disabled for a 5 second countdown. ## Web UI - **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text search. Grid with lazy loading; click opens a lightbox with the source message link, the original URL, and a download button. A scrape-status strip shows progress with Start and Stop. - **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar. The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the database overhead. - Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second countdown. - Mobile is out of scope for now. ## MCP The agent can query the archive and start scrapes with MCP tools. The hooks are designed in [media-mcp.md](media-mcp.md) and implemented at checkpoint C8. ## Security - The command follows the existing trust rules and is written to the action log. - The slopbot token lives in the state dir or the environment, never in the repo. See [slopbot.md](slopbot.md). - Media is served from disk only when the sniffed type matches the extension allowlist. Paths are hashes, never user input. - The Web UI has no authentication. LAN or Tailscale only, as before. - This automates a user account. The account should be one the owner can afford to lose.