Veskslop-t/docs/media-scraper.md

Media scraper

The scrape engine walks a channel's history and fills a per-scrape database. It is backend agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs. See slopbot.md and core-bridge.md.

Backends

Both backends implement one interface:

interface ScrapeBackend {
    fetchPage(channelId: string, before: string | null, limit: number):
        Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}

Backend	Implementation
slopbot	discord.js REST (channels.messages.fetch), bucket-aware client
puppet	The core action message.fetch over the bridge: the client's REST layer paginates (100/page, contiguous), with the client store as fallback. Expired URLs refresh with before: id+1, which returns exactly the target message

The daemon picks slopbot when the channel is a guild channel and slopbot is configured and in the guild; otherwise puppet. A scrape records the backend it used. The walk can resume on the same backend only.

Walk

1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch limit messages before the cursor. The cursor is the oldest message ID seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the boundary is crossed (afterId/beforeId in the meta).
3. Extract media from each message and write the page to the database in one transaction.
4. Persist scrape.json (cursor, counters, state) after every page.
5. Stop when a page returns fewer than limit messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.

Rules:

• One running scrape at a time. Further requests queue.
• stop marks the job and lets the current page finish.
• resume continues a paused scrape from its stored cursor, so an interrupted walk does not fetch the pages it already has again. The History popup has Resume and Delete buttons and the API a POST /api/media/scrape/resume route.
• On daemon start, scrapes still marked running or queued on disk are stale (their job died with the previous daemon). They are reconciled to paused so the UI tells the truth and Resume can continue them.
• Every page is an outbound read. Sends from commands keep their own pacing.

Pacing and backoff

The engine is deliberately slower than normal reads. All values are configurable; defaults are starting points, not Discord facts.

Setting	Default
media.pageMinMs / media.pageMaxMs	1200 / 3500
media.pagesPerHour	300
media.pagesPerDay	3000
media.maxRetries	8

• A random gap in the page window between pages.
• A rolling budget per hour and per day. At the cap the job pauses and resumes when the window resets.
• On failure: use the backend's retryAfterMs when present, else exponential 2^attempt + 1 seconds, matching DiscordChatExporter. After maxRetries the job pauses instead of looping.
• DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js handle buckets.

Classification

The extractor maps one message to zero or more media items and returns their kind, source URL, provider, dimensions, duration when known, and size when known.

• Attachments from attachments[]: extension and contentType decide image, gif, video, audio.
• Embeds from embeds[]: type and provider decide. video embeds become kind video with downloaded = false. Image and gif embeds become cached previews only.
• Forwarded message snapshots are treated as messages with their own media.
• Stickers become images or gifs.
• Voice messages (voiceMessage flag) become audio.
• Third-party embeds are never downloaded, by rule.

Database schema

One SQLite file per scrape (index.db, node:sqlite).

Table	Columns
messages	id, author_id, author_name, timestamp, content
media	id, kind, source, author_id, author_name, posted_at, url, provider, title, title_key, title_error, file_hash, preview_hash, content_type, size, width, height, aspect_ratio, duration_secs, downloaded, file_path, created_at
media_sources	media_id, message_id
downloads	media_id, state, attempts, error

scrape.json also records fromDate/toDate, and estimatedMessages from Discord search when the slopbot backend could ask for it (one request; null otherwise). Indexes: kind, posted_at, and media_sources(message_id). The database is an index; the media folder holds bytes. Every item carries its poster (author_id, author_name), the post time (posted_at), and its dimensions (width, height, aspect_ratio). youtube.com/shorts/<id> links are stored as watch?v=<id> (extraction and a one-time user_version migration), because the embedded player and View full need the watch form; a shorts row that collides with an already indexed watch URL folds into it (preview/file fields merge, message links move over). Embed rows also store Discord's title (YouTube and friends) plus title_key, the lowercased, punctuation-collapsed form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds Man who jumps over car), one phrase with the exact-match toggle, and rows without a title drop out while a title query is set. * is a wildcard anywhere in a token: *car matches inside words (scar, discard), c*r* matches between letters, car* behaves like the plain prefix. Literal % and _ are escaped, so they only match themselves. Filter presets live in media-scrape/src/filter-presets.ts: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is filled with ffprobe after a download when VESKSLOP_FFPROBE is available.

Downloads and previews

Downloads run after a finished walk when the scrape used mode: "download":

• A serial queue, one file at a time, 1-3 s apart, with a size cap (the Scraping settings, default 50 MiB) and 2 retries on network or 5xx errors.
• 4xx from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
• The file name is the SHA-256 of the bytes under media/. file_path, file_hash, and file_bytes are written to the row, so the run resumes and dedupes by content.
• Only attachment and sticker rows are targets. Embed rows are excluded in SQL (source != 'embed').
• With a custom destination, files go to <destination>/<scrape-id>/media/ and the row stores the absolute path; without one they stay in <scrape>/media/. Embed previews are copied to <destination>/<scrape-id>/thumbnails/ while the local copy stays for the Media tab.
• Downloaded videos get a poster frame (ffmpeg -ss 0.5, 480px WebP) cached in thumbs/, because Discord serves no thumbnails for uploaded video files.
• The ffprobe pass fills missing dimensions and duration_secs after a download. Attachment durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown.
• Progress counters: downloaded, downloadErrors.

Previews cycle with the walk: after every Preview every N walk pages setting (default 5) pages the engine previews up to one batch (the Backfilling batch size, default 25), then keeps walking, so tiles fill in during a long walk instead of only at the end. Set the cycle to 0 to defer all previews to the end.

Previews run after every walk:

• In both modes, Discord-hosted image, GIF, and video URLs (cdn.discordapp.com, media.discordapp.net) are transformed to the CDN resizer (media.discordapp.net?...width=320&height=320&format=webp). The proxy renders a frame for video files, so attachment mp4/webm get a cached poster in links mode without downloading the file. Download mode additionally keeps an ffmpeg poster.
• In both modes, embed rows also get their stored preview_url (YouTube, X, TikTok thumbnails, and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from pendingDownloads.
• Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first: provider oEmbed/fxtwitter API, then the page's og:image/twitter:image (512 KiB read cap, 10 s timeout, one attempt). The page read never consumes media bytes: a non-HTML content type stops it. An embed that unfurls a pasted Discord CDN attachment link resolves through the Discord proxy instead, so it gets the same still (and video frame) as a plain attachment. Every thumbnail-less embed enters preview passes; one with no resolvable image is closed as no preview source.
• Animated GIF previews are stored as a static first frame (ffmpeg, 480px WebP); the original animated GIF URL stays in preview_url for hover playback. Tenor embeds without a thumbnail are resolved through Tenor's oEmbed. An animated GIF over the size cap is framed by ffmpeg reading the stream directly, so it is not lost to too large.
• Audio attachments and voice messages get a waveform strip (ffmpeg -filter_complex showwavespic) as their preview, rendered by ffmpeg from the URL.
• Transient network failures are left pending for the next backfill pass; only skips (4xx, too large, no source) are marked permanent.
• One attempt per row, no retries; a permanent failure (http 404, http 403, too large) marks preview_error and never fails the scrape.
• Stored by hash in thumbs/<sha256>.<ext>, where the extension follows the response type (preview_ext, default webp); preview_hash and preview_bytes land on the row, and the counter is previewed. Previews count toward the storage limit.
• The job stays running through the download and preview passes, so a client never sees done on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
• The backfill queue (media-scrape/src/backfill.ts) fills in scrapes that paused or finished before their previews were cached, and resolves missing titles. Titles: YouTube-like rows (provider or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved through the provider oEmbed and stored as title + title_key; failures count toward title_attempts and close out as title_error after the tries-per-item setting, so deleted videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while both kinds of work remain. One job per chat groups every paused, done, or failed scrape of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The Bot tab drives them: Start backfill, per-job Start/Pause, and Pause all. Jobs with the least pending work start first, so small chats clear instead of waiting behind a long one; a job that is started but waiting for the queue reads queued. On daemon boot jobs start automatically when the Backfilling setting Start automatically on daemon boot is on.
• Transient fetch failures keep the row pending and count up preview_attempts. After the Tries before giving up setting (default 3) the row is marked unreachable after N attempt(s), so dead hosts (Gfycat, Streamable) stop burning batches forever.
• Preview gaps and batching live in the Bot tab settings (Backfilling): gap 200-600 ms, batches of 25 with a 60 s cooldown between them, because bursts of thumbnail fetches trigger Discord and provider rate limits. A batch counts rows, not successes.

Rate limits

Discord and the provider APIs throttle bursts. The engine detects limiting and holds all media traffic:

• A 429 from a preview or download fetch pauses the pass; the row stays pending (never marked as a permanent error) and is retried later.
• A walk page that fails with a 429 or a rate-limit message registers a cooldown, and every page and preview waits it out before continuing.
• The pause is the Timers setting Rate-limit cooldown (default 60 s), or the server's Retry-After when it is longer. Detection is logged as rate limit detected; pausing media traffic until …. The Bot tab banner reads API timed out, increase timers while media traffic is held.

Browsing cache

Browsing never calls Discord. The grid renders cached previews or local files only; videos without a cached still show a labeled placeholder, and playback streams only when the user presses play. GIF hover plays the animated GIF cached in thumbs/ (the original is kept alongside the static frame and referenced by anim_hash), never the provider, so scrolling cannot spam anyone.

Bot tab settings

The Bot tab owns the tunables; they persist in preferences.json and apply to live jobs at once.

Group	Setting	Default
Backfilling	Batch size (rows)	25
Backfilling	Cooldown between batches (ms)	60000
Backfilling	Tries before giving up	3
Backfilling	Preview gap min / max (ms)	200 / 600
Backfilling	Max preview size (MiB)	5
Backfilling	Start automatically on daemon boot	on
Scraping	Preview every N walk pages	5
Scraping	Max file size (MiB)	50
Scraping	Download gap min / max (ms)	1000 / 3000
Scraping	Page fetch retries	2
Timers	Walk page gap min / max (ms)	1200 / 3500
Timers	Rate-limit cooldown (ms)	60000

The old VESKSLOP_MEDIA_* tuning variables are gone; the values above were their defaults.

Job states

queued, running, paused, stopping, done, failed, pending-delete, deleting, deleted.

Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab and the MCP get_media_scrape tool read the same fields.

Tests

• Unit tests run against the mock core and a fake slopbot REST. No live Discord.
• Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
• The live run is a private test guild first, then a real channel.

Checkpoints

Implementation goals live in CURRENT.md. The engine is C3; slopbot is C4; daemon wiring (command, service, routes) is C5.