Veskslop-t/docs/media-scraper.md

# Media scraper

The scrape engine walks a channel's history and fills a per-scrape database. It is backend
agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs.
See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md).

## Backends

Both backends implement one interface:

```ts
interface ScrapeBackend {
    fetchPage(channelId: string, before: string | null, limit: number):
        Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}
```

| Backend | Implementation |
| --- | --- |
| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client |
| `puppet` | The core action `message.fetch` over the bridge: the client's REST layer paginates (100/page, contiguous), with the client store as fallback. Expired URLs refresh with `before: id+1`, which returns exactly the target message |

The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the
guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same
backend only.

## Walk

1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
   seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
   boundary is crossed (`afterId`/`beforeId` in the meta).
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.

Rules:

- One running scrape at a time. Further requests queue.
- `stop` marks the job and lets the current page finish.
- `resume` continues a paused scrape from its stored cursor, so an interrupted walk does not fetch
  the pages it already has again. The History popup has **Resume** and **Delete** buttons and the
  API a `POST /api/media/scrape/resume` route.
- On daemon start, scrapes still marked `running` or `queued` on disk are stale (their job died
  with the previous daemon). They are reconciled to `paused` so the UI tells the truth and Resume
  can continue them.
- Every page is an outbound read. Sends from commands keep their own pacing.

## Pacing and backoff

The engine is deliberately slower than normal reads. All values are configurable; defaults are
starting points, not Discord facts.

| Setting | Default |
| --- | --- |
| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 |
| `media.pagesPerHour` | 300 |
| `media.pagesPerDay` | 3000 |
| `media.maxRetries` | 8 |

- A random gap in the page window between pages.
- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window
  resets.
- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1`
  seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping.
- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend
  uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js
  handle buckets.

## Classification

The extractor maps one message to zero or more media items and returns their kind, source URL,
provider, dimensions, duration when known, and size when known.

- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio.
- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with
  `downloaded = false`. Image and gif embeds become cached previews only.
- Forwarded message snapshots are treated as messages with their own media.
- Stickers become images or gifs.
- Voice messages (`voiceMessage` flag) become audio.
- Third-party embeds are never downloaded, by rule.

## Database schema

One SQLite file per scrape (`index.db`, `node:sqlite`).

| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `title_error`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |

`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
`media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). `youtube.com/shorts/<id>`
links are stored as `watch?v=<id>` (extraction and a one-time `user_version` migration), because the
embedded player and View full need the watch form; a shorts row that collides with an already
indexed watch URL folds into it (preview/file fields merge, message links move over). Embed rows also store
Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed
form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps
over car*), one phrase with the exact-match toggle, and rows without a title drop out while a
title query is set. `*` is a wildcard anywhere in a token: `*car` matches inside words (*scar*,
*discard*), `c*r*` matches between letters, `car*` behaves like the plain prefix. Literal `%` and
`_` are escaped, so they only match themselves. Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.

## Downloads and previews

Downloads run after a finished walk when the scrape used `mode: "download"`:

- A serial queue, one file at a time, 1-3 s apart, with a size cap
  (the **Scraping** settings, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
- The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and
  `file_bytes` are written to the row, so the run resumes and dedupes by content.
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
  (`source != 'embed'`).
- With a custom destination, files go to `<destination>/<scrape-id>/media/` and the row stores the
  absolute path; without one they stay in `<scrape>/media/`. Embed previews are copied to
  `<destination>/<scrape-id>/thumbnails/` while the local copy stays for the Media tab.
- Downloaded videos get a poster frame (`ffmpeg -ss 0.5`, 480px WebP) cached in `thumbs/`, because
  Discord serves no thumbnails for uploaded video files.
- The ffprobe pass fills missing dimensions and `duration_secs` after a download. Attachment
  durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown.
- Progress counters: `downloaded`, `downloadErrors`.

Previews cycle with the walk: after every **Preview every N walk pages** setting (default 5) pages
the engine previews up to one batch (the **Backfilling** batch size, default 25), then keeps walking,
so tiles fill in during a long walk instead of only at the end. Set the cycle to `0` to defer all
previews to the end.

Previews run after every walk:

- In both modes, Discord-hosted image, GIF, and **video** URLs (`cdn.discordapp.com`,
  `media.discordapp.net`) are transformed to the CDN resizer
  (`media.discordapp.net?...width=320&height=320&format=webp`). The proxy renders a frame for
  video files, so attachment mp4/webm get a cached poster in links mode without downloading the
  file. Download mode additionally keeps an ffmpeg poster.
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
  and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
  `pendingDownloads`.
- Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first:
  provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap,
  10 s timeout, one attempt). The page read never consumes media bytes: a non-HTML content type
  stops it. An embed that unfurls a pasted Discord CDN attachment link resolves through the Discord
  proxy instead, so it gets the same still (and video frame) as a plain attachment. Every
  thumbnail-less embed enters preview passes; one with no resolvable image is closed as
  `no preview source`.
- Animated GIF previews are stored as a static first frame (`ffmpeg`, 480px WebP); the original
  animated GIF URL stays in `preview_url` for hover playback. Tenor embeds without a thumbnail are
  resolved through Tenor's oEmbed. An animated GIF over the size cap is framed by ffmpeg reading
  the stream directly, so it is not lost to `too large`.
- Audio attachments and voice messages get a waveform strip
  (`ffmpeg -filter_complex showwavespic`) as their preview, rendered by ffmpeg from the URL.
- Transient network failures are left pending for the next backfill pass; only skips (4xx, too
  large, no source) are marked permanent.
- One attempt per row, no retries; a permanent failure (`http 404`, `http 403`, `too large`) marks
  `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
  (`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
  counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
  on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
  before their previews were cached. It holds preview work only.
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
  indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
  job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
  media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
  beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
  youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
  stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
  throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
  of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
  batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
  Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
  least pending work start first, so small chats clear instead of waiting behind a long one; a job
  that is started but waiting for the queue reads `queued`. On daemon boot jobs start automatically
  when the Backfilling setting **Start automatically on daemon boot** is on.
- Transient fetch failures keep the row pending and count up `preview_attempts`. After the
  **Tries before giving up** setting (default 3) the row is marked `unreachable after N attempt(s)`,
  so dead hosts (Gfycat, Streamable) stop burning batches forever.
- Preview gaps and batching live in the Bot tab settings (**Backfilling**): gap 200-600 ms, batches
  of 25 with a 60 s cooldown between them, because bursts of thumbnail fetches trigger Discord and
  provider rate limits. A batch counts rows, not successes.

## Browsing performance

Every grid request used to rescan every scrape database to merge URLs. `MediaService` now keeps a
short-lived candidate cache: the key is a revision of the scrape metadata (id, state, update time,
preview count) plus the normalized filter shape — page limits, offsets, and empty values are
dropped, so every page of one filter set shares a single scan and later pages return in
milliseconds. The 15 s TTL and the revision key cover new pages, finished scrapes, and freshly
cached previews. The client only refetches stats when the filters change (or after a minute), and
grid cards use lazy async decoding with CSS `content-visibility` so a long stream does not pay
layout and paint for off-screen tiles.

## Rate limits

Discord and the provider APIs throttle bursts. The engine detects limiting and holds all media
traffic:

- A `429` from a preview or download fetch pauses the pass; the row stays pending (never marked as a
  permanent error) and is retried later.
- A walk page that fails with a `429` or a rate-limit message registers a cooldown, and every page
  and preview waits it out before continuing.
- The pause is the **Timers** setting **Rate-limit cooldown** (default 60 s), or the server's
  `Retry-After` when it is longer. Detection is logged as `rate limit detected; pausing media traffic
  until …`. The Bot tab banner reads `API timed out, increase timers` while media traffic is held.

## Browsing cache

Browsing never calls Discord. The grid renders cached previews or local files only; videos without
a cached still show a labeled placeholder, and playback streams only when the user presses play.
GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.

## Failure policy and recovery

The archive never treats a transient HTTP answer as a dead link on the first try.

- A preview or download that fails with an HTTP status on Discord's CDN is retried once with a
  **refreshed URL** from the source message (the signed link expires in about a day). Only if the
  refreshed fetch fails is the row parked.
- Refreshing an embed canonicalizes both links: `play_url` gets the provider's direct media URL and
  `url` falls back to it when Discord's `embed.url` is missing. Preview resolution for embeds uses
  `play_url` first, so a refreshed file is never fetched through an expired `url`. This is what the
  recovery task relies on.
- Parked rows become **suspects**: they get one delayed retry (about 50 s). The retry runs after
  the batch, not immediately. If a batch is two-thirds suspect failures, the cooldown doubles
  (floor 60 s) and the retries run before the next batch.
- A suspect whose delayed retry fails again is left **unverified**: it is skipped by later passes,
  so a finished backfill never churns over genuinely unavailable media. The Bot tab shows suspects
  and unverified counts per member, and the progress bar fills blue for cached, gray for pending,
  and red for failed rows.
- **Recover Discord previews** is a one-shot task in the Bot tab: it walks every scrape's
  Discord-CDN rows closed with HTTP errors, refreshes each URL through the message (slopbot REST or
  the puppet, whichever the scrape used), and fetches the preview again. It starts paused; when it
  finishes it moves to the task history. The code stays for reuse.

## Bot tab settings

The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.

| Group | Setting | Default |
| --- | --- | --- |
| Backfilling | Batch size (rows) | 25 |
| Backfilling | Cooldown between batches (ms) | 60000 |
| Backfilling | Tries before giving up | 3 |
| Backfilling | Preview gap min / max (ms) | 200 / 600 |
| Backfilling | Max preview size (MiB) | 5 |
| Backfilling | Start automatically on daemon boot | on |
| Scraping | Preview every N walk pages | 5 |
| Scraping | Max file size (MiB) | 50 |
| Scraping | Download gap min / max (ms) | 1000 / 3000 |
| Scraping | Page fetch retries | 2 |
| Timers | Walk page gap min / max (ms) | 1200 / 3500 |
| Timers | Rate-limit cooldown (ms) | 60000 |

The old `VESKSLOP_MEDIA_*` tuning variables are gone; the values above were their defaults.

## Job states

`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.

Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab
and the MCP `get_media_scrape` tool read the same fields.

## Tests

- Unit tests run against the mock core and a fake slopbot REST. No live Discord.
- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
- The live run is a private test guild first, then a real channel.

## Checkpoints

Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4; daemon
wiring (command, service, routes) is C5.