# Media archive

Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
the `/api/media/*` routes, storage limits with retention, the Settings group, the MCP tools, and
the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6. Live-verified:

- slopbot on the test channel: 17 messages (4 images, 13 videos), 0 errors.
- slopbot on a larger channel: 11 pages, 1088 messages (96 images, 1 GIF, 203 videos), 0 errors.
- puppet on a group DM and a 1:1 DM: done, 0 errors, backend `puppet`.

Preview caching is implemented and live-verified (2 WebP thumbs served over the thumb route).
Remaining: a 429 observation on a very large walk. Checkpoints live in [CURRENT.md](../CURRENT.md).

The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
the archive, **Bot** controls slopbot and the storage settings.

The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.

## Command

```
@puppet scrape media            # index links (default)
@puppet scrape media download   # also download attachment bytes
@puppet scrape media status
@puppet scrape media stop
```

- The target is the channel the command is sent in. Group DMs and guild channels work.
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
  into the confirmation (see Retention).
- The reply follows the response policy. The job runs in the background.

## Backends

The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.

| | slopbot (bot account) | puppet (user account) |
| --- | --- | --- |
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. |
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
| Requires the Electron client | No | Yes |

See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).

## Storage

Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:

```
media-scrape/db/
  <scrape-id>/                 # <channel-id>-<YYYYMMDD-HHMMSS>
    scrape.json                # channel, guild, mode, state, cursor, counters
    index.db                   # SQLite metadata index
    media/                     # downloaded bytes, only in download mode
      <sha256>.<ext>
    thumbs/                    # cached Discord-hosted image previews
      <sha256>.webp
```

- The database is an index. It holds the scrape state, the messages seen, the media items, and the
  link between a media item and its source messages. Source channel, author, and timestamp are
  indexed so filters can be added later without a re-scrape.
- Scrapes stay self-contained, but the API and UI **deduplicate by URL across scrapes**: one card
  per URL, the newest scrape wins; stats count unique URLs too. Re-scraping a channel does not
  duplicate anything in the grid or the counters. (Files are still stored per scrape, so the
  on-disk size doubles with overlapping scrapes; a shared content store is a possible later step.)
- Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes,
  `better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install.
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
  name is metadata.
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.

## Index versus download

- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
  downloaded.
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
  are never download targets. The Media tab prefers the local file when one exists, so the archive
  survives CDN URL expiry.
- A custom download destination sends the bytes to `<dir>/<scrape-id>/media/` instead. Previews
  stay local, and embed thumbnails are copied to `<dir>/<scrape-id>/thumbnails/` so the download
  has a poster. Files in a custom destination are not served or linked by the Media tab; the user
  owns that path.
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
  other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
- Previews are cached in `thumbs/` after every walk, in both modes: Discord-hosted images, GIFs,
  and video files (the proxy renders a frame, resized to 320px WebP) plus third-party embed
  thumbnails (YouTube, X, TikTok, and so on). The
  stored bytes keep their real type (`preview_ext`), so JPEG provider thumbnails are served as
  `image/jpeg`, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved
  through the provider's oEmbed/fxtwitter API and finally the page's `og:image`. Previews count
  toward the storage limit.
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
  a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
  labeled placeholder, and a failed image falls back to that placeholder.
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
  `posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
  filled in with `ffprobe` after the file lands.
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
  refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.

## Media classification

| Kind | Sources |
| --- | --- |
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |

## Retention and limits

The storage limit covers everything under the DB root: databases, downloads, and previews.

| Setting | Default | Meaning |
| --- | --- | --- |
| `media.limitBytes` | 30 GiB | Combined size limit |
| `media.warnPercent` | 90 | Warn threshold |
| `media.overrunPercent` | 5 | No-delete allowance above the limit |

Behavior:

1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
   usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true`
   on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`).
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
   explains this.

The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`,
`failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the
scrape directory and forgets it in the engine.

**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
that deletion is irreversible. The delete button is disabled for a 5 second countdown.

## Web UI

- **Media** tab: filter chips (All, Images, GIFs, Videos, Audio), and filters for source (any,
  attachments, stickers, embeds), source channel, poster, aspect ratio (Vertical, 1:1, 4:3, 16:9,
  21:9, Other), resolution (SD, HD, FHD, QHD, 4K+, Unknown), and a posted-date range (from/to).
  Nothing renders until **Load media** is pressed. A **Metadata** dropdown toggles which fields
  show in the small banner under each tile (User, Type, Date, Source, Resolution, Length). Video
  tiles carry a translucent play icon top-right. Discord serves no thumbnails for uploaded videos,
  so a video without a cached preview renders a first-frame `<video>` in the grid, and downloaded
  videos get an ffmpeg poster cached in `thumbs/`. Clicking a tile opens a viewer: wide media
  shows the metadata banner below, vertical media shows it to the right, and videos carry a large
  play icon top-right. An embeddable item (YouTube, Vimeo, Twitch, Streamable) opens straight into
  its player; other videos play a direct mp4/webm/m4v, and when neither is possible the play
  action opens the external-site warning, since the page is the only way to watch. Playback and **View full** go through
  `  /api/media/<scrape>/<item>/stream`, which serves the local file or re-fetches the source message
  for a fresh signed URL, so expired CDN links still play. The grid prefers the cached still; a
  video without one gets its first frame through the same-origin stream endpoint, which avoids
  cross-origin blocking and URL expiry. Only real video-file URLs are framed; provider pages
  (YouTube embeds, redirector URLs) are never, and the stream route answers `415` when the
  upstream is not media — so a pending preview shows a placeholder instead of a browser error. Embed `videoUrl`s (gifv/Tenor mp4s) are stored as
  `play_url` and used through the stream. The full link is a separate button that opens the
  original page with the warning. Length comes from Discord's attachment duration and, for
  downloads, an ffprobe probe. GIF tiles carry a translucent **GIF** label top-left; the cached
  preview is a static frame and the animated GIF plays while hovering. The status row reads
  `(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V` — image counts include GIFs, and
  each slot is the count for the active filters. A scrape strip shows live progress with Stop; it
  polls every 4 seconds while the tab is open, lists only queued/running jobs (paused ones live in
  History), and shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely
  (a **Load more** button appears as a fallback). Media whose source answers `403` shows a
  **PRIVATE** tile and `404`/`410` shows **GONE**, with the reason in the tooltip. Some channels deny the bot the search endpoint (`Missing Access`) even though history
  reads work, so the total is optional.
  **New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
  channels; each row has Links and Download). Every scrape button asks for confirmation; the
  dialog takes optional From/To dates and a **Check Message Count** button (Discord search total,
  slopbot guild channels only), and Download can target a custom absolute destination. **History** lists each scrape with its start
  time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
  backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
  below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
  that disappears when the tab is opened and a Clear button.
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
  The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
  database overhead.
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
  **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
  bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second
  countdown.
- Mobile is out of scope for now.

## MCP

The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
[media-mcp.md](media-mcp.md) and implemented at checkpoint C8.

## Security

- The command follows the existing trust rules and is written to the action log.
- The slopbot token lives in the state dir or the environment, never in the repo. See
  [slopbot.md](slopbot.md).
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
  hashes, never user input.
- The Web UI has no authentication. LAN or Tailscale only, as before.
- This automates a user account. The account should be one the owner can afford to lose.