# Media scraper The scrape engine walks a channel's history and fills a per-scrape database. It is backend agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs. See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md). ## Backends Both backends implement one interface: ```ts interface ScrapeBackend { fetchPage(channelId: string, before: string | null, limit: number): Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>; } ``` | Backend | Implementation | | --- | --- | | `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client | | `puppet` | The core action `message.fetch` over the bridge: the client's REST layer paginates (100/page, contiguous), with the client store as fallback. Expired URLs refresh with `before: id+1`, which returns exactly the target message | The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same backend only. ## Walk 1. The command resolves the channel and creates a scrape directory. 2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the boundary is crossed (`afterId`/`beforeId` in the meta). 3. Extract media from each message and write the page to the database in one transaction. 4. Persist `scrape.json` (cursor, counters, state) after every page. 5. Stop when a page returns fewer than `limit` messages or the oldest message is reached. 6. Resume after a daemon restart from the stored cursor. Rules: - One running scrape at a time. Further requests queue. - `stop` marks the job and lets the current page finish. - `resume` continues a paused scrape from its stored cursor, so an interrupted walk does not fetch the pages it already has again. The History popup has a **Resume** button and the API a `POST /api/media/scrape/resume` route. - Every page is an outbound read. Sends from commands keep their own pacing. ## Pacing and backoff The engine is deliberately slower than normal reads. All values are configurable; defaults are starting points, not Discord facts. | Setting | Default | | --- | --- | | `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 | | `media.pagesPerHour` | 300 | | `media.pagesPerDay` | 3000 | | `media.maxRetries` | 8 | - A random gap in the page window between pages. - A rolling budget per hour and per day. At the cap the job pauses and resumes when the window resets. - On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1` seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping. - DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js handle buckets. ## Classification The extractor maps one message to zero or more media items and returns their kind, source URL, provider, dimensions, duration when known, and size when known. - Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio. - Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with `downloaded = false`. Image and gif embeds become cached previews only. - Forwarded message snapshots are treated as messages with their own media. - Stickers become images or gifs. - Voice messages (`voiceMessage` flag) become audio. - Third-party embeds are never downloaded, by rule. ## Database schema One SQLite file per scrape (`index.db`, `node:sqlite`). | Table | Columns | | --- | --- | | `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` | | `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` | | `media_sources` | `media_id`, `message_id` | | `downloads` | `media_id`, `state`, `attempts`, `error` | `scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time (`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in `media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available. ## Downloads and previews Downloads run after a finished walk when the scrape used `mode: "download"`: - A serial queue, one file at a time, 1-3 s apart, with a size cap (`VESKSLOP_MEDIA_MAX_FILE_BYTES`, default 50 MiB) and 2 retries on network or 5xx errors. - `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried. - The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and `file_bytes` are written to the row, so the run resumes and dedupes by content. - Only attachment and sticker rows are targets. Embed rows are excluded in SQL (`source != 'embed'`). - With a custom destination, files go to `//media/` and the row stores the absolute path; without one they stay in `/media/`. Embed previews are copied to `//thumbnails/` while the local copy stays for the Media tab. - Downloaded videos get a poster frame (`ffmpeg -ss 0.5`, 480px WebP) cached in `thumbs/`, because Discord serves no thumbnails for uploaded video files. - The ffprobe pass fills missing dimensions and `duration_secs` after a download. Attachment durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown. - Progress counters: `downloaded`, `downloadErrors`. Previews run after every walk: - In both modes, Discord-hosted image, GIF, and **video** URLs (`cdn.discordapp.com`, `media.discordapp.net`) are transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`). The proxy renders a frame for video files, so attachment mp4/webm get a cached poster in links mode without downloading the file. Download mode additionally keeps an ffmpeg poster. - In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails, and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from `pendingDownloads`. - Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first: provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap, 10 s timeout, one attempt). - Animated GIF previews are stored as a static first frame (`ffmpeg`, 480px WebP); the original animated GIF URL stays in `preview_url` for hover playback. Tenor embeds without a thumbnail are resolved through Tenor's oEmbed. An animated GIF over the size cap is framed by ffmpeg reading the stream directly, so it is not lost to `too large`. - Audio attachments and voice messages get a waveform strip (`ffmpeg -filter_complex showwavespic`) as their preview, rendered by ffmpeg from the URL. - Transient network failures are left pending for the next backfill pass; only skips (4xx, too large, no source) are marked permanent. - One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape. - Stored by hash in `thumbs/.`, where the extension follows the response type (`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the counter is `previewed`. Previews count toward the storage limit. - The job stays `running` through the download and preview passes, so a client never sees `done` on a half-archived scrape. A paused walk still gets these passes for the pages it collected. - At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their previews were cached, so a stopped scrape does not leave blank tiles. - Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches (`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of thumbnail fetches trigger Discord and provider rate limits. A batch counts rows, not successes. ## Job states `queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`. Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab and the MCP `get_media_scrape` tool read the same fields. ## Tests - Unit tests run against the mock core and a fake slopbot REST. No live Discord. - Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff. - The live run is a private test guild first, then a real channel. ## Checkpoints Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4; daemon wiring (command, service, routes) is C5.