229 lines
15 KiB
Markdown
229 lines
15 KiB
Markdown
# Media archive
|
|
|
|
Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
|
|
the `/api/media/*` routes, storage limits with retention, the Settings group, the MCP tools, and
|
|
the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6. Live-verified:
|
|
|
|
- slopbot on the test channel: 17 messages (4 images, 13 videos), 0 errors.
|
|
- slopbot on a larger channel: 11 pages, 1088 messages (96 images, 1 GIF, 203 videos), 0 errors.
|
|
- puppet on a group DM and a 1:1 DM: done, 0 errors, backend `puppet`.
|
|
|
|
Preview caching is implemented and live-verified (2 WebP thumbs served over the thumb route).
|
|
Remaining: a 429 observation on a very large walk. Checkpoints live in [CURRENT.md](../CURRENT.md).
|
|
|
|
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
|
|
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
|
|
video, and audio item into a per-scrape database. Two new Web UI tabs expose it: **Media** browses
|
|
the archive, **Bot** controls slopbot and the storage settings.
|
|
|
|
The feature lives in [`media-scrape/`](../media-scrape/README.md) and touches the daemon at three
|
|
points: the Web UI route mount, the MCP tool registration, and the message shape in the core.
|
|
|
|
## Command
|
|
|
|
```
|
|
@puppet scrape media # index links (default)
|
|
@puppet scrape media download # also download attachment bytes
|
|
@puppet scrape media status
|
|
@puppet scrape media stop
|
|
```
|
|
|
|
- The target is the channel the command is sent in. Group DMs and guild channels work.
|
|
- The command needs the `trusted` tier. `stop` and `download` needs `admin`.
|
|
- `scrape media` with no argument re-runs the last scrape state; near the storage limit it turns
|
|
into the confirmation (see Retention).
|
|
- The reply follows the response policy. The job runs in the background.
|
|
|
|
## Backends
|
|
|
|
The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.
|
|
|
|
| | slopbot (bot account) | puppet (user account) |
|
|
| --- | --- | --- |
|
|
| Guild channels | Yes. Preferred. | Fallback when slopbot is not in the guild. |
|
|
| Group DMs and DMs | **No.** Discord bots cannot see them. | Yes. The only option. Expired attachment URLs refresh through the core bridge (`message.fetch` around the ID). |
|
|
| Transport | Official bot REST via discord.js | The validated core bridge (`message.fetch`) |
|
|
| Rate limits | Documented, per-route buckets, handled by discord.js | Undocumented; daemon pacing and backoff |
|
|
| Requires the Electron client | No | Yes |
|
|
|
|
See [slopbot.md](slopbot.md) and [media-scraper.md](media-scraper.md).
|
|
|
|
## Storage
|
|
|
|
Runtime data lives under `media-scrape/db/`, ignored by git and configurable with
|
|
`VESKSLOP_MEDIA_DB_DIR`. Each scrape is self-contained:
|
|
|
|
```
|
|
media-scrape/db/
|
|
<scrape-id>/ # <channel-id>-<YYYYMMDD-HHMMSS>
|
|
scrape.json # channel, guild, mode, state, cursor, counters
|
|
index.db # SQLite metadata index
|
|
media/ # downloaded bytes, only in download mode
|
|
<sha256>.<ext>
|
|
thumbs/ # cached Discord-hosted image previews
|
|
<sha256>.webp
|
|
```
|
|
|
|
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
|
|
link between a media item and its source messages. Source channel, author, and timestamp are
|
|
indexed so filters can be added later without a re-scrape.
|
|
- Scrapes stay self-contained, but the API and UI **deduplicate by URL across scrapes**: one card
|
|
per URL, the newest scrape wins; stats count unique URLs too. Re-scraping a channel does not
|
|
duplicate anything in the grid or the counters. (Files are still stored per scrape, so the
|
|
on-disk size doubles with overlapping scrapes; a shared content store is a possible later step.)
|
|
- Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes,
|
|
`better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install.
|
|
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
|
|
name is metadata.
|
|
- Deleting a scrape is deleting its directory. Nothing is shared between scrapes.
|
|
|
|
## Index versus download
|
|
|
|
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
|
|
downloaded.
|
|
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
|
|
are never download targets. The Media tab prefers the local file when one exists, so the archive
|
|
survives CDN URL expiry.
|
|
- A custom download destination sends the bytes to `<dir>/<scrape-id>/media/` instead. Previews
|
|
stay local, and embed thumbnails are copied to `<dir>/<scrape-id>/thumbnails/` so the download
|
|
has a poster. Files in a custom destination are not served or linked by the Media tab; the user
|
|
owns that path.
|
|
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
|
|
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
|
|
- Previews are cached in `thumbs/` after every walk, in both modes: Discord-hosted images, GIFs,
|
|
and video files (the proxy renders a frame, resized to 320px WebP), third-party embed thumbnails
|
|
(YouTube, X, TikTok, and so on), and audio, which gets a waveform strip so audio tiles are not
|
|
blank. The
|
|
stored bytes keep their real type (`preview_ext`), so JPEG provider thumbnails are served as
|
|
`image/jpeg`, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved
|
|
through the provider's oEmbed/fxtwitter API and finally the page's `og:image`. Previews count
|
|
toward the storage limit.
|
|
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
|
|
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
|
|
labeled placeholder, and a failed image falls back to that placeholder.
|
|
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
|
|
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
|
|
filled in with `ffprobe` after the file lands.
|
|
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
|
|
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.
|
|
|
|
## Media classification
|
|
|
|
| Kind | Sources |
|
|
| --- | --- |
|
|
| `image` | png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers |
|
|
| `gif` | gif attachments; gifv, Tenor, Giphy embeds |
|
|
| `video` | mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video |
|
|
| `audio` | mp3, m4a, ogg, wav, flac attachments; Discord voice messages |
|
|
|
|
## Retention and limits
|
|
|
|
The storage limit covers everything under the DB root: databases, downloads, and previews.
|
|
|
|
| Setting | Default | Meaning |
|
|
| --- | --- | --- |
|
|
| `media.limitBytes` | 30 GiB | Combined size limit |
|
|
| `media.warnPercent` | 90 | Warn threshold |
|
|
| `media.overrunPercent` | 5 | No-delete allowance above the limit |
|
|
|
|
Behavior:
|
|
|
|
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
|
|
usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true`
|
|
on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`).
|
|
2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
|
|
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
|
|
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
|
|
explains this.
|
|
|
|
The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`,
|
|
`failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the
|
|
scrape directory and forgets it in the engine.
|
|
|
|
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
|
|
that deletion is irreversible. The delete button is disabled for a 5 second countdown.
|
|
|
|
## Web UI
|
|
|
|
- **Media** tab: the bar carries the common filters — kind chips, source kind, server/DM origin,
|
|
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
|
|
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
|
|
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
|
|
include** and **Must include** groups with nested collapsibles for sources (the common providers
|
|
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
|
|
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
|
|
rows with no known value. The bar and drawer show only on the Media tab.
|
|
Nothing renders until **Load media** is pressed. A **Metadata** dropdown toggles which fields
|
|
show in the small banner under each tile (User, Type, Date, Source, Resolution, Length). Video
|
|
tiles carry a translucent play icon top-right. Discord serves no thumbnails for uploaded videos,
|
|
so a video without a cached preview renders a first-frame `<video>` in the grid, and downloaded
|
|
videos get an ffmpeg poster cached in `thumbs/`. Clicking a tile opens a viewer: wide media
|
|
shows the metadata banner below, vertical media shows it to the right, and videos carry a large
|
|
play icon top-right. An embeddable item (YouTube, Vimeo, Twitch, Streamable) opens straight into
|
|
its player; other videos play a direct mp4/webm/m4v, and when neither is possible the play
|
|
action opens the external-site warning, since the page is the only way to watch. Playback and **View full** go through
|
|
` /api/media/<scrape>/<item>/stream`, which serves the local file or re-fetches the source message
|
|
for a fresh signed URL, so expired CDN links still play. The grid prefers the cached still; a
|
|
video without one gets its first frame through the same-origin stream endpoint, which avoids
|
|
cross-origin blocking and URL expiry. Only real video-file URLs are framed; provider pages
|
|
(YouTube embeds, redirector URLs) are never, and the stream route answers `415` when the
|
|
upstream is not media — so a pending preview shows a placeholder instead of a browser error. Embed `videoUrl`s (gifv/Tenor mp4s) are stored as
|
|
`play_url` and used through the stream. The full link is a separate button that opens the
|
|
original page with the warning. Length comes from Discord's attachment duration and, for
|
|
downloads, an ffprobe probe. GIF tiles carry a translucent **GIF** label top-left; the cached
|
|
preview is a static frame and the animated GIF plays while hovering. The status row reads
|
|
`(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V` — image counts include GIFs, and
|
|
each slot is the count for the active filters. A scrape strip shows live progress and polls every
|
|
4 seconds while the tab is open. It shows every queued/running job plus at most one paused scrape
|
|
(the newest not dismissed); **Stop** on a paused scrape only hides it here, so a new paused scrape
|
|
can take its place while the hidden one stays resumable. The strip text turns red and reads
|
|
`API limited, retry in Ns` while Discord throttles media traffic, and a translucent
|
|
`N scrapes completed` line appears when scrapes finish — it lives until the tab is reloaded. Each
|
|
row shows `X/Yp (~Z%)` when the search total is known. The grid streams: appending a page inserts
|
|
only the new cards, so loaded images never rebuild or flash, and a stream **buffer** below the last
|
|
row keeps the scroll alive while the next page is in flight. The buffer is capped at 1.5 viewport
|
|
heights, shrinks as the filtered results run out, and says `Loading more…` / `Scroll for more` /
|
|
`End of results`; auto-fill cascades pages while the buffer is still in view, so a filter that
|
|
returns few items does not dead-stop at one page. At the very bottom while a page loads, an
|
|
elastic nudge replaces the hard stop. Media whose source answers `403` shows a **PRIVATE** tile
|
|
and `404`/`410` shows **GONE**, with the reason in the tooltip. Some channels deny the bot the search endpoint (`Missing Access`) even though history
|
|
reads work, so the total is optional.
|
|
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
|
|
channels; each row has Links and Download). Every scrape button asks for confirmation; the
|
|
dialog takes optional From/To dates and a **Check Message Count** button (Discord search total,
|
|
slopbot guild channels only), and Download can target a custom absolute destination. **History** lists each scrape with its start
|
|
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
|
|
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
|
|
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
|
|
that disappears when the tab is opened and a Clear button.
|
|
- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
|
|
**Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
|
|
current settings, and a Bot/Puppet split of pending/cached/failed. One job card per chat lists its
|
|
member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
|
|
counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
|
|
carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
|
|
while a Discord cooldown holds the queue. **Settings** holds three collapsible groups —
|
|
Backfilling, Scraping, Timers — each with a Save button; values persist in `preferences.json` and
|
|
apply to live jobs immediately. The **Errors** list keeps the bot and scrape errors with the red
|
|
badge and Clear button.
|
|
- Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
|
|
**Media** if the bot path is removed. It holds the storage limit field with the info tooltip, the
|
|
usage bar (segmented by image, GIF, video, audio, and other), and the clean button with its
|
|
5 second countdown. Pacing and batching knobs live in the Bot tab, not here.
|
|
- Mobile is out of scope for now.
|
|
|
|
## MCP
|
|
|
|
The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
|
|
[media-mcp.md](media-mcp.md) and implemented at checkpoint C8.
|
|
|
|
## Security
|
|
|
|
- The command follows the existing trust rules and is written to the action log.
|
|
- The slopbot token lives in the state dir or the environment, never in the repo. See
|
|
[slopbot.md](slopbot.md).
|
|
- Media is served from disk only when the sniffed type matches the extension allowlist. Paths are
|
|
hashes, never user input.
|
|
- The Web UI has no authentication. LAN or Tailscale only, as before.
|
|
- This automates a user account. The account should be one the owner can afford to lose.
|