Media archive

Status: design approved 2026-09-16. The engine, the slopbot backend, the scrape media command, the /api/media/* routes, storage limits with retention, the Settings group, the MCP tools, and the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6. Live-verified:

slopbot on the test channel: 17 messages (4 images, 13 videos), 0 errors.
slopbot on a larger channel: 11 pages, 1088 messages (96 images, 1 GIF, 203 videos), 0 errors.
puppet on a group DM and a 1:1 DM: done, 0 errors, backend puppet.

Preview caching is implemented and live-verified (2 WebP thumbs served over the thumb route). Remaining: a 429 observation on a very large walk. Checkpoints live in CURRENT.md.

The media archive saves the media history of a channel. A trusted user runs scrape media in the channel. The daemon walks the history through a scrape backend and indexes every image, GIF, video, and audio item into a per-scrape database. Two new Web UI tabs expose it: Media browses the archive, Bot controls slopbot and the storage settings.

The feature lives in media-scrape/ and touches the daemon at three points: the Web UI route mount, the MCP tool registration, and the message shape in the core.

Command

@puppet scrape media            # index links (default)
@puppet scrape media download   # also download attachment bytes
@puppet scrape media status
@puppet scrape media stop

The target is the channel the command is sent in. Group DMs and guild channels work.
The command needs the trusted tier. stop and download needs admin.
scrape media with no argument re-runs the last scrape state; near the storage limit it turns into the confirmation (see Retention).
The reply follows the response policy. The job runs in the background.

Backends

The walk runs through one of two backends. The daemon picks automatically; a scrape can pin one.

	slopbot (bot account)	puppet (user account)
Guild channels	Yes. Preferred.	Fallback when slopbot is not in the guild.
Group DMs and DMs	No. Discord bots cannot see them.	Yes. The only option. Expired attachment URLs refresh through the core bridge (`message.fetch` around the ID).
Transport	Official bot REST via discord.js	The validated core bridge (`message.fetch`)
Rate limits	Documented, per-route buckets, handled by discord.js	Undocumented; daemon pacing and backoff
Requires the Electron client	No	Yes

See slopbot.md and media-scraper.md.

Storage

Runtime data lives under media-scrape/db/, ignored by git and configurable with VESKSLOP_MEDIA_DB_DIR. Each scrape is self-contained:

media-scrape/db/
  <scrape-id>/                 # <channel-id>-<YYYYMMDD-HHMMSS>
    scrape.json                # channel, guild, mode, state, cursor, counters
    index.db                   # SQLite metadata index
    media/                     # downloaded bytes, only in download mode
      <sha256>.<ext>
    thumbs/                    # cached Discord-hosted image previews
      <sha256>.webp

The database is an index. It holds the scrape state, the messages seen, the media items, and the link between a media item and its source messages. Source channel, author, and timestamp are indexed so filters can be added later without a re-scrape.
Scrapes stay self-contained, but the API and UI deduplicate by URL across scrapes: one card per URL, the newest scrape wins; stats count unique URLs too. Re-scraping a channel does not duplicate anything in the grid or the counters. (Files are still stored per scrape, so the on-disk size doubles with overlapping scrapes; a shared content store is a possible later step.)
Storage uses node:sqlite (experimental in Node 22, prints a warning). If its API changes, better-sqlite3 is the fallback; it needs a native build, so that switch is an owner install.
The file name in media/ is the SHA-256 of the bytes, never the original name. The original name is metadata.
Deleting a scrape is deleting its directory. Nothing is shared between scrapes.

Index versus download

links is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is downloaded.
download fetches attachment bytes into media/, one file at a time, hash-addressed. Embed rows are never download targets. The Media tab prefers the local file when one exists, so the archive survives CDN URL expiry.
A custom download destination sends the bytes to <dir>/<scrape-id>/media/ instead. Previews stay local, and embed thumbnails are copied to <dir>/<scrape-id>/thumbnails/ so the download has a poster. Files in a custom destination are not served or linked by the Media tab; the user owns that path.
Third-party embeds are never downloaded as media: YouTube, Instagram, X, TikTok, and every other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
Previews are cached in thumbs/ after every walk, in both modes: Discord-hosted images, GIFs, and video files (the proxy renders a frame, resized to 320px WebP), third-party embed thumbnails (YouTube, X, TikTok, and so on), and audio, which gets a waveform strip so audio tiles are not blank. The stored bytes keep their real type (preview_ext), so JPEG provider thumbnails are served as image/jpeg, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved through the provider's oEmbed/fxtwitter API and finally the page's og:image. Previews count toward the storage limit.
The grid never renders an embed's page URL as media. A direct video file without a preview shows a first-frame <video> only for playable containers (mp4, webm, m4v); anything else shows a labeled placeholder, and a failed image falls back to that placeholder.
Every item records who posted it, when, and its dimensions: author_id, author_name, posted_at, width, height, and aspect_ratio. In download mode a missing resolution is filled in with ffprobe after the file lands.
Discord attachment URLs are signed and expire in about a day. In links mode an expired URL is refreshed by re-fetching its message; in download mode the bytes are caught before expiry.

Media classification

Kind	Sources
`image`	png, jpg, jpeg, webp, bmp, avif attachments; image embeds; stickers
`gif`	gif attachments; gifv, Tenor, Giphy embeds
`video`	mp4, webm, mov, mkv attachments; YouTube, Vimeo, Twitch clip, Streamable embeds; forwarded video
`audio`	mp3, m4a, ogg, wav, flac attachments; Discord voice messages

Retention and limits

The storage limit covers everything under the DB root: databases, downloads, and previews.

Setting	Default	Meaning
`media.limitBytes`	30 GiB	Combined size limit
`media.warnPercent`	90	Warn threshold
`media.overrunPercent`	5	No-delete allowance above the limit

Behavior:

At or above the warn threshold, a new scrape media command is refused and the reply shows the usage. Re-running the same command within 2 minutes accepts. The API mirror is confirm: true on POST /api/media/scrape (otherwise 409 MEDIA_LIMIT).
At or above 100% the oldest completed scrapes are marked pending-delete.
Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
No data is deleted inside the 5% overrun band. The hover info (small i beside the limit field) explains this.

The planning is a pure function (media-scrape/src/retention.ts) with tests. Only done, failed, and paused scrapes are ever marked; a running scrape is untouched. Deletion removes the scrape directory and forgets it in the engine.

Clean media in Settings lists exactly what will be deleted, with sizes and counts, and states that deletion is irreversible. The delete button is disabled for a 5 second countdown.

Web UI

Media tab: the bar carries the common filters — kind chips, source kind, server/DM origin, and the multi-select People popover. The Advanced search drawer has aspect ratio, resolution, source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a Do not include and Must include groups with nested collapsibles, a title search for embed titles (multi-word prefix matching, with an Exact match (quotes) toggle) for sources (the common providers in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units), and length (seconds), with Apply filters / Reset in the bar. Size and length ranges drop rows with no known value. The bar and drawer show only on the Media tab. Nothing renders until Load media is pressed. A Metadata dropdown toggles which fields show in the small banner under each tile (User, Type, Date, Source, Resolution, Length). Video tiles carry a translucent play icon top-right. Discord serves no thumbnails for uploaded videos, so a video without a cached preview renders a first-frame <video> in the grid, and downloaded videos get an ffmpeg poster cached in thumbs/. Clicking a tile opens a viewer: wide media shows the metadata banner below, vertical media shows it to the right, and videos carry a large play icon top-right. An embeddable item (YouTube, Vimeo, Twitch, Streamable) opens straight into its player; other videos play a direct mp4/webm/m4v, and when neither is possible the play action opens the external-site warning, since the page is the only way to watch. Playback and View full go through /api/media/<scrape>/<item>/stream, which serves the local file or re-fetches the source message for a fresh signed URL, so expired CDN links still play. The grid prefers the cached still; a video without one gets its first frame through the same-origin stream endpoint, which avoids cross-origin blocking and URL expiry. Only real video-file URLs are framed; provider pages (YouTube embeds, redirector URLs) are never, and the stream route answers 415 when the upstream is not media — so a pending preview shows a placeholder instead of a browser error. Embed videoUrls (gifv/Tenor mp4s) are stored as play_url and used through the stream. The full link is a separate button that opens the original page with the warning. Length comes from Discord's attachment duration and, for downloads, an ffprobe probe. GIF tiles carry a translucent GIF label top-left; the cached preview is a static frame and the animated GIF plays while hovering. The status row reads (Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V — image counts include GIFs, and each slot is the count for the active filters. A scrape strip shows live progress and polls every 4 seconds while the tab is open. It shows every queued/running job plus at most one paused scrape (the newest not dismissed); Stop on a paused scrape only hides it here, so a new paused scrape can take its place while the hidden one stays resumable. The strip text turns red and reads API limited, retry in Ns while Discord throttles media traffic, and a translucent N scrapes completed line appears when scrapes finish — it lives until the tab is reloaded. Each row shows X/Yp (~Z%) when the search total is known. The grid streams: appending a page inserts only the new cards, so loaded images never rebuild or flash, and a stream buffer below the last row keeps the scroll alive while the next page is in flight. The buffer is capped at 1.5 viewport heights, shrinks as the filtered results run out, and says Loading more… / Scroll for more / End of results; auto-fill cascades pages while the buffer is still in view, so a filter that returns few items does not dead-stop at one page. At the very bottom while a page loads, an elastic nudge replaces the hard stop. Media whose source answers 403 shows a PRIVATE tile and 404/410 shows GONE, with the reason in the tooltip. Some channels deny the bot the search endpoint (Missing Access) even though history reads work, so the total is optional. New scrape… (rightmost) opens a popup with Server and Group chat trees (servers expand to channels; each row has Links and Download). Every scrape button asks for confirmation; the dialog takes optional From/To dates and a Check Message Count button (Discord search total, slopbot guild channels only), and Download can target a custom absolute destination. History lists each scrape with its start time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge that disappears when the tab is opened and a Clear button.
Bot tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables. Backfill shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing YouTube titles; each member line shows its share as + N titles. One job card per chat lists its member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending counts, a cached/total progress bar, a measured ETA, and Start/Pause; the section header carries Start backfill and Pause all. A red banner reads API timed out, increase timers while a Discord cooldown holds the queue. Settings holds three collapsible groups — Backfilling, Scraping, Timers — each with a Save button; values persist in preferences.json and apply to live jobs immediately. The Errors list keeps the bot and scrape errors with the red badge and Clear button.
Settings gains one collapsible group. Its name is Bot while the bot path is active, and Media if the bot path is removed. It holds the storage limit field with the info tooltip, the usage bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second countdown. Pacing and batching knobs live in the Bot tab, not here.
Mobile is out of scope for now.

MCP

The agent can query the archive and start scrapes with MCP tools. The hooks are designed in media-mcp.md and implemented at checkpoint C8.

Security

The command follows the existing trust rules and is written to the action log.
The slopbot token lives in the state dir or the environment, never in the repo. See slopbot.md.
Media is served from disk only when the sniffed type matches the extension allowlist. Paths are hashes, never user input.
The Web UI has no authentication. LAN or Tailscale only, as before.
This automates a user account. The account should be one the owner can afford to lose.

15 KiB Raw Blame History