Veskslop-t/docs/media-scraper.md

# Media scraper

The scrape engine walks a channel's history and fills a per-scrape database. It is backend
agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs.
See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md).

## Backends

Both backends implement one interface:

```ts
interface ScrapeBackend {
    fetchPage(channelId: string, before: string | null, limit: number):
        Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}
```

| Backend | Implementation |
| --- | --- |
| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client |
| `puppet` | The core action `message.fetch` over the bridge: the client's REST layer paginates (100/page, contiguous), with the client store as fallback. Expired URLs refresh with `before: id+1`, which returns exactly the target message |

The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the
guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same
backend only.

## Walk

1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
   seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
   boundary is crossed (`afterId`/`beforeId` in the meta).
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.

Rules:

- One running scrape at a time. Further requests queue.
- `stop` marks the job and lets the current page finish.
- `resume` continues a paused scrape from its stored cursor, so an interrupted walk does not fetch
  the pages it already has again. The History popup has **Resume** and **Delete** buttons and the
  API a `POST /api/media/scrape/resume` route.
- On daemon start, scrapes still marked `running` or `queued` on disk are stale (their job died
  with the previous daemon). They are reconciled to `paused` so the UI tells the truth and Resume
  can continue them.
- Every page is an outbound read. Sends from commands keep their own pacing.

## Pacing and backoff

The engine is deliberately slower than normal reads. All values are configurable; defaults are
starting points, not Discord facts.

| Setting | Default |
| --- | --- |
| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 |
| `media.pagesPerHour` | 300 |
| `media.pagesPerDay` | 3000 |
| `media.maxRetries` | 8 |

- A random gap in the page window between pages.
- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window
  resets.
- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1`
  seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping.
- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend
  uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js
  handle buckets.

## Classification

The extractor maps one message to zero or more media items and returns their kind, source URL,
provider, dimensions, duration when known, and size when known.

- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio.
- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with
  `downloaded = false`. Image and gif embeds become cached previews only.
- Forwarded message snapshots are treated as messages with their own media.
- Stickers become images or gifs.
- Voice messages (`voiceMessage` flag) become audio.
- Third-party embeds are never downloaded, by rule.

## Database schema

One SQLite file per scrape (`index.db`, `node:sqlite`).

| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `title_error`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |

`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
`media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Embed rows also store
Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed
form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps
over car*), one phrase with the exact-match toggle, and rows without a title drop out while a
title query is set. Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.

## Downloads and previews

Downloads run after a finished walk when the scrape used `mode: "download"`:

- A serial queue, one file at a time, 1-3 s apart, with a size cap
  (the **Scraping** settings, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
- The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and
  `file_bytes` are written to the row, so the run resumes and dedupes by content.
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
  (`source != 'embed'`).
- With a custom destination, files go to `<destination>/<scrape-id>/media/` and the row stores the
  absolute path; without one they stay in `<scrape>/media/`. Embed previews are copied to
  `<destination>/<scrape-id>/thumbnails/` while the local copy stays for the Media tab.
- Downloaded videos get a poster frame (`ffmpeg -ss 0.5`, 480px WebP) cached in `thumbs/`, because
  Discord serves no thumbnails for uploaded video files.
- The ffprobe pass fills missing dimensions and `duration_secs` after a download. Attachment
  durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown.
- Progress counters: `downloaded`, `downloadErrors`.

Previews cycle with the walk: after every **Preview every N walk pages** setting (default 5) pages
the engine previews up to one batch (the **Backfilling** batch size, default 25), then keeps walking,
so tiles fill in during a long walk instead of only at the end. Set the cycle to `0` to defer all
previews to the end.

Previews run after every walk:

- In both modes, Discord-hosted image, GIF, and **video** URLs (`cdn.discordapp.com`,
  `media.discordapp.net`) are transformed to the CDN resizer
  (`media.discordapp.net?...width=320&height=320&format=webp`). The proxy renders a frame for
  video files, so attachment mp4/webm get a cached poster in links mode without downloading the
  file. Download mode additionally keeps an ffmpeg poster.
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
  and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
  `pendingDownloads`.
- Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first:
  provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap,
  10 s timeout, one attempt). The page read never consumes media bytes: a non-HTML content type
  stops it. An embed that unfurls a pasted Discord CDN attachment link resolves through the Discord
  proxy instead, so it gets the same still (and video frame) as a plain attachment. Every
  thumbnail-less embed enters preview passes; one with no resolvable image is closed as
  `no preview source`.
- Animated GIF previews are stored as a static first frame (`ffmpeg`, 480px WebP); the original
  animated GIF URL stays in `preview_url` for hover playback. Tenor embeds without a thumbnail are
  resolved through Tenor's oEmbed. An animated GIF over the size cap is framed by ffmpeg reading
  the stream directly, so it is not lost to `too large`.
- Audio attachments and voice messages get a waveform strip
  (`ffmpeg -filter_complex showwavespic`) as their preview, rendered by ffmpeg from the URL.
- Transient network failures are left pending for the next backfill pass; only skips (4xx, too
  large, no source) are marked permanent.
- One attempt per row, no retries; a permanent failure (`http 404`, `http 403`, `too large`) marks
  `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
  (`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
  counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
  on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
  before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
  (`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
  through the provider oEmbed and stored as `title` + `title_key`; failures count toward
  `title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
  videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
  both kinds of work remain. One job per chat groups every paused, done, or failed scrape
  of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
  batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
  Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
  least pending work start first, so small chats clear instead of waiting behind a long one; a job
  that is started but waiting for the queue reads `queued`. On daemon boot jobs start automatically
  when the Backfilling setting **Start automatically on daemon boot** is on.
- Transient fetch failures keep the row pending and count up `preview_attempts`. After the
  **Tries before giving up** setting (default 3) the row is marked `unreachable after N attempt(s)`,
  so dead hosts (Gfycat, Streamable) stop burning batches forever.
- Preview gaps and batching live in the Bot tab settings (**Backfilling**): gap 200-600 ms, batches
  of 25 with a 60 s cooldown between them, because bursts of thumbnail fetches trigger Discord and
  provider rate limits. A batch counts rows, not successes.

## Rate limits

Discord and the provider APIs throttle bursts. The engine detects limiting and holds all media
traffic:

- A `429` from a preview or download fetch pauses the pass; the row stays pending (never marked as a
  permanent error) and is retried later.
- A walk page that fails with a `429` or a rate-limit message registers a cooldown, and every page
  and preview waits it out before continuing.
- The pause is the **Timers** setting **Rate-limit cooldown** (default 60 s), or the server's
  `Retry-After` when it is longer. Detection is logged as `rate limit detected; pausing media traffic
  until …`. The Bot tab banner reads `API timed out, increase timers` while media traffic is held.

## Browsing cache

Browsing never calls Discord. The grid renders cached previews or local files only; videos without
a cached still show a labeled placeholder, and playback streams only when the user presses play.
GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.

## Bot tab settings

The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.

| Group | Setting | Default |
| --- | --- | --- |
| Backfilling | Batch size (rows) | 25 |
| Backfilling | Cooldown between batches (ms) | 60000 |
| Backfilling | Tries before giving up | 3 |
| Backfilling | Preview gap min / max (ms) | 200 / 600 |
| Backfilling | Max preview size (MiB) | 5 |
| Backfilling | Start automatically on daemon boot | on |
| Scraping | Preview every N walk pages | 5 |
| Scraping | Max file size (MiB) | 50 |
| Scraping | Download gap min / max (ms) | 1000 / 3000 |
| Scraping | Page fetch retries | 2 |
| Timers | Walk page gap min / max (ms) | 1200 / 3500 |
| Timers | Rate-limit cooldown (ms) | 60000 |

The old `VESKSLOP_MEDIA_*` tuning variables are gone; the values above were their defaults.

## Job states

`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.

Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab
and the MCP `get_media_scrape` tool read the same fields.

## Tests

- Unit tests run against the mock core and a fake slopbot REST. No live Discord.
- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
- The live run is a private test guild first, then a real channel.

## Checkpoints

Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4; daemon
wiring (command, service, routes) is C5.