209 lines
13 KiB
Markdown
209 lines
13 KiB
Markdown
# Media archive
|
|
|
|
Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
|
|
the `/api/media/*` routes, storage limits with retention, the Settings group, the MCP tools, and
|
|
the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6. Live-verified:
|
|
|
|
- slopbot on the test channel: 17 messages (4 images, 13 videos), 0 errors.
|
|
- slopbot on a larger channel: 11 pages, 1088 messages (96 images, 1 GIF, 203 videos), 0 errors.
|
|
- puppet on a group DM and a 1:1 DM: done, 0 errors, backend `puppet`.
|
|
|
|
Preview caching is implemented and live-verified (2 WebP thumbs served over the thumb route).
|
|
Remaining: a 429 observation on a very large walk. Checkpoints live in [CURRENT.md](../CURRENT.md).
|
|
|
|
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
|
|
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
|
|
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
|
|
the archive, **Bot** controls slopbot and the storage settings.
|
|
|
|
The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
|
|
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.
|
|
|
|
## Command
|
|
|
|
```
|
|
@puppet scrape media # index links (default)
|
|
@puppet scrape media download # also download attachment bytes
|
|
@puppet scrape media status
|
|
@puppet scrape media stop
|
|
```
|
|
|
|
- The target is the channel the command is sent in. Group DMs and guild channels work.
|
|
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
|
|
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
|
|
into the confirmation (see Retention).
|
|
- The reply follows the response policy. The job runs in the background.
|
|
|
|
## Backends
|
|
|
|
The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.
|
|
|
|
| | slopbot (bot account) | puppet (user account) |
|
|
| --- | --- | --- |
|
|
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
|
|
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. Expired attachment URLs refresh through the core bridge (`message.fetch` around the ID). |
|
|
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
|
|
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
|
|
| Requires the Electron client | No | Yes |
|
|
|
|
See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).
|
|
|
|
## Storage
|
|
|
|
Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
|
|
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:
|
|
|
|
```
|
|
media-scrape/db/
|
|
<scrape-id>/ # <channel-id>-<YYYYMMDD-HHMMSS>
|
|
scrape.json # channel, guild, mode, state, cursor, counters
|
|
index.db # SQLite metadata index
|
|
media/ # downloaded bytes, only in download mode
|
|
<sha256>.<ext>
|
|
thumbs/ # cached Discord-hosted image previews
|
|
<sha256>.webp
|
|
```
|
|
|
|
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
|
|
link between a media item and its source messages. Source channel, author, and timestamp are
|
|
indexed so filters can be added later without a re-scrape.
|
|
- Scrapes stay self-contained, but the API and UI **deduplicate by URL across scrapes**: one card
|
|
per URL, the newest scrape wins; stats count unique URLs too. Re-scraping a channel does not
|
|
duplicate anything in the grid or the counters. (Files are still stored per scrape, so the
|
|
on-disk size doubles with overlapping scrapes; a shared content store is a possible later step.)
|
|
- Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes,
|
|
`better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install.
|
|
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
|
|
name is metadata.
|
|
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.
|
|
|
|
## Index versus download
|
|
|
|
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
|
|
downloaded.
|
|
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
|
|
are never download targets. The Media tab prefers the local file when one exists, so the archive
|
|
survives CDN URL expiry.
|
|
- A custom download destination sends the bytes to `<dir>/<scrape-id>/media/` instead. Previews
|
|
stay local, and embed thumbnails are copied to `<dir>/<scrape-id>/thumbnails/` so the download
|
|
has a poster. Files in a custom destination are not served or linked by the Media tab; the user
|
|
owns that path.
|
|
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
|
|
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
|
|
- Previews are cached in `thumbs/` after every walk, in both modes: Discord-hosted images, GIFs,
|
|
and video files (the proxy renders a frame, resized to 320px WebP), third-party embed thumbnails
|
|
(YouTube, X, TikTok, and so on), and audio, which gets a waveform strip so audio tiles are not
|
|
blank. The
|
|
stored bytes keep their real type (`preview_ext`), so JPEG provider thumbnails are served as
|
|
`image/jpeg`, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved
|
|
through the provider's oEmbed/fxtwitter API and finally the page's `og:image`. Previews count
|
|
toward the storage limit.
|
|
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
|
|
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
|
|
labeled placeholder, and a failed image falls back to that placeholder.
|
|
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
|
|
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
|
|
filled in with `ffprobe` after the file lands.
|
|
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
|
|
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.
|
|
|
|
## Media classification
|
|
|
|
| Kind | Sources |
|
|
| --- | --- |
|
|
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
|
|
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
|
|
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
|
|
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |
|
|
|
|
## Retention and limits
|
|
|
|
The storage limit covers everything under the DB root: databases, downloads, and previews.
|
|
|
|
| Setting | Default | Meaning |
|
|
| --- | --- | --- |
|
|
| `media.limitBytes` | 30 GiB | Combined size limit |
|
|
| `media.warnPercent` | 90 | Warn threshold |
|
|
| `media.overrunPercent` | 5 | No-delete allowance above the limit |
|
|
|
|
Behavior:
|
|
|
|
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
|
|
usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true`
|
|
on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`).
|
|
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
|
|
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
|
|
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
|
|
explains this.
|
|
|
|
The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`,
|
|
`failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the
|
|
scrape directory and forgets it in the engine.
|
|
|
|
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
|
|
that deletion is irreversible. The delete button is disabled for a 5 second countdown.
|
|
|
|
## Web UI
|
|
|
|
- **Media** tab: the bar carries the common filters — kind chips, source kind, server/DM origin,
|
|
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
|
|
source channel, posted-date range, file-type chips, cached-only, and a default-on hide for 4chan
|
|
links (they 404 almost always), with **Apply filters** / **Reset** in the bar.
|
|
Nothing renders until **Load media** is pressed. A **Metadata** dropdown toggles which fields
|
|
show in the small banner under each tile (User, Type, Date, Source, Resolution, Length). Video
|
|
tiles carry a translucent play icon top-right. Discord serves no thumbnails for uploaded videos,
|
|
so a video without a cached preview renders a first-frame `<video>` in the grid, and downloaded
|
|
videos get an ffmpeg poster cached in `thumbs/`. Clicking a tile opens a viewer: wide media
|
|
shows the metadata banner below, vertical media shows it to the right, and videos carry a large
|
|
play icon top-right. An embeddable item (YouTube, Vimeo, Twitch, Streamable) opens straight into
|
|
its player; other videos play a direct mp4/webm/m4v, and when neither is possible the play
|
|
action opens the external-site warning, since the page is the only way to watch. Playback and **View full** go through
|
|
` /api/media/<scrape>/<item>/stream`, which serves the local file or re-fetches the source message
|
|
for a fresh signed URL, so expired CDN links still play. The grid prefers the cached still; a
|
|
video without one gets its first frame through the same-origin stream endpoint, which avoids
|
|
cross-origin blocking and URL expiry. Only real video-file URLs are framed; provider pages
|
|
(YouTube embeds, redirector URLs) are never, and the stream route answers `415` when the
|
|
upstream is not media — so a pending preview shows a placeholder instead of a browser error. Embed `videoUrl`s (gifv/Tenor mp4s) are stored as
|
|
`play_url` and used through the stream. The full link is a separate button that opens the
|
|
original page with the warning. Length comes from Discord's attachment duration and, for
|
|
downloads, an ffprobe probe. GIF tiles carry a translucent **GIF** label top-left; the cached
|
|
preview is a static frame and the animated GIF plays while hovering. The status row reads
|
|
`(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V` — image counts include GIFs, and
|
|
each slot is the count for the active filters. A scrape strip shows live progress with Stop; it
|
|
polls every 4 seconds while the tab is open, lists only queued/running jobs (paused ones live in
|
|
History), and shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely
|
|
(a **Load more** button appears as a fallback). Media whose source answers `403` shows a
|
|
**PRIVATE** tile and `404`/`410` shows **GONE**, with the reason in the tooltip. Some channels deny the bot the search endpoint (`Missing Access`) even though history
|
|
reads work, so the total is optional.
|
|
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
|
|
channels; each row has Links and Download). Every scrape button asks for confirmation; the
|
|
dialog takes optional From/To dates and a **Check Message Count** button (Discord search total,
|
|
slopbot guild channels only), and Download can target a custom absolute destination. **History** lists each scrape with its start
|
|
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
|
|
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
|
|
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
|
|
that disappears when the tab is opened and a Clear button.
|
|
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
|
|
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
|
|
database overhead.
|
|
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
|
|
**Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
|
|
bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second
|
|
countdown.
|
|
- Mobile is out of scope for now.
|
|
|
|
## MCP
|
|
|
|
The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
|
|
[media-mcp.md](media-mcp.md) and implemented at checkpoint C8.
|
|
|
|
## Security
|
|
|
|
- The command follows the existing trust rules and is written to the action log.
|
|
- The slopbot token lives in the state dir or the environment, never in the repo. See
|
|
[slopbot.md](slopbot.md).
|
|
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
|
|
hashes, never user input.
|
|
- The Web UI has no authentication. LAN or Tailscale only, as before.
|
|
- This automates a user account. The account should be one the owner can afford to lose.
|