# Media archive Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command, the `/api/media/*` routes, and storage limits with retention are implemented (C3-C7a). The Media and Bot tabs are C6; the Settings group is C7b. Checkpoints live in [CURRENT.md](../CURRENT.md). The media archive saves the media history of a channel. A trusted user runs `scrape media` in the channel. The daemon walks the history through a scrape backend and indexes every image, GIF, video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses the archive, **Bot** controls slopbot and the storage settings. The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three points: the Web UI route mount, the MCP tool registration, and the message shape in the core. ## Command ``` @puppet scrape media # index links (default) @puppet scrape media download # also download attachment bytes @puppet scrape media status @puppet scrape media stop ``` - The target is the channel the command is sent in. Group DMs and guild channels work. - The command needs the `trusted` tier. `stop` and `download` needs `admin`. - `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns into the confirmation (see Retention). - The reply follows the response policy. The job runs in the background. ## Backends The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one. | | slopbot (bot account) | puppet (user account) | | --- | --- | --- | | Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. | | Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. | | Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) | | Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff | | Requires the Electron client | No | Yes | See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md). ## Storage Runtime data lives under `media-scrape/db/`, ignored by git and configurable with `VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained: ``` media-scrape/db/ / # - scrape.json # channel, guild, mode, state, cursor, counters index.db # SQLite metadata index media/ # downloaded bytes, only in download mode . thumbs/ # cached Discord-hosted image previews .webp ``` - The database is an index. It holds the scrape state, the messages seen, the media items, and the link between a media item and its source messages. Source channel, author, and timestamp are indexed so filters can be added later without a re-scrape. - Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes, `better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install. - The file name in `media/` is the SHA-256 of the bytes, never the original name. The original name is metadata. - Deleting a scrape is deleting its directory. Nothing is shared between scrapes. ## Index versus download - `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is downloaded. - `download` fetches attachment bytes into `media/`. Deduplicated by hash. - Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other embed stays a link with metadata. - Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count toward the storage limit. - Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry. ## Media classification | Kind | Sources | | --- | --- | | `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers | | `gif` | gif attachments; gifv, Tenor, Giphy embeds | | `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video | | `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages | ## Retention and limits The storage limit covers everything under the DB root: databases, downloads, and previews. | Setting | Default | Meaning | | --- | --- | --- | | `media.limitBytes` | 30 GiB | Combined size limit | | `media.warnPercent` | 90 | Warn threshold | | `media.overrunPercent` | 5 | No-delete allowance above the limit | Behavior: 1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true` on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`). 2. At or above 100% the oldest completed scrapes are marked `pending-delete`. 3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits. 4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field) explains this. The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`, `failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the scrape directory and forgets it in the engine. **Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states that deletion is irreversible. The delete button is disabled for a 5 second countdown. ## Web UI - **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), source channel, date, and text search. Grid with lazy loading; click opens a lightbox with the source message link, the original URL, and a download button. A scrape-status strip shows progress with Start and Stop. - **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar. The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the database overhead. - Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage bar, and the clean button. - Mobile is out of scope for now. ## MCP The agent can query the archive and start scrapes with MCP tools. The hooks are designed in [media-mcp.md](media-mcp.md) and implemented at checkpoint C8. ## Security - The command follows the existing trust rules and is written to the action log. - The slopbot token lives in the state dir or the environment, never in the repo. See [slopbot.md](slopbot.md). - Media is served from disk only when the sniffed type matches the extension allowlist. Paths are hashes, never user input. - The Web UI has no authentication. LAN or Tailscale only, as before. - This automates a user account. The account should be one the owner can afford to lose.