Veskslop-t/docs/media-scraper.md

Media scraper

The scrape engine walks a channel's history and fills a per-scrape database. It is backend agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs. See slopbot.md and core-bridge.md.

Backends

Both backends implement one interface:

interface ScrapeBackend {
    fetchPage(channelId: string, before: string | null, limit: number):
        Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
}

Backend	Implementation
slopbot	discord.js REST (channels.messages.fetch), bucket-aware client
puppet	The core action message.fetch over the bridge

The daemon picks slopbot when the channel is a guild channel and slopbot is configured and in the guild; otherwise puppet. A scrape records the backend it used. The walk can resume on the same backend only.

Walk

1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch limit messages before the cursor. The cursor is the oldest message ID seen.
3. Extract media from each message and write the page to the database in one transaction.
4. Persist scrape.json (cursor, counters, state) after every page.
5. Stop when a page returns fewer than limit messages or the oldest message is reached.
6. Resume after a daemon restart from the stored cursor.

Rules:

• One running scrape at a time. Further requests queue.
• stop marks the job and lets the current page finish.
• Every page is an outbound read. Sends from commands keep their own pacing.

Pacing and backoff

The engine is deliberately slower than normal reads. All values are configurable; defaults are starting points, not Discord facts.

Setting	Default
media.pageMinMs / media.pageMaxMs	1200 / 3500
media.pagesPerHour	300
media.pagesPerDay	3000
media.maxRetries	8

• A random gap in the page window between pages.
• A rolling budget per hour and per day. At the cap the job pauses and resumes when the window resets.
• On failure: use the backend's retryAfterMs when present, else exponential 2^attempt + 1 seconds, matching DiscordChatExporter. After maxRetries the job pauses instead of looping.
• DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js handle buckets.

Classification

The extractor maps one message to zero or more media items and returns their kind, source URL, provider, dimensions, duration when known, and size when known.

• Attachments from attachments[]: extension and contentType decide image, gif, video, audio.
• Embeds from embeds[]: type and provider decide. video embeds become kind video with downloaded = false. Image and gif embeds become cached previews only.
• Forwarded message snapshots are treated as messages with their own media.
• Stickers become images or gifs.
• Voice messages (voiceMessage flag) become audio.
• Third-party embeds are never downloaded, by rule.

Database schema

One SQLite file per scrape (index.db, node:sqlite).

Table	Columns
messages	id, author_id, author_name, timestamp, content
media	id, kind, url, provider, file_hash, preview_hash, content_type, size, width, height, duration_ms, downloaded, created_at
media_sources	media_id, message_id
downloads	media_id, state, attempts, error

Indexes: (kind, timestamp) via media_sources, channel metadata in scrape.json, and file_hash for dedupe. The database is an index; the media folder holds bytes.

Downloads and previews

• downloads is a serial queue, one file at a time, 1-3 s apart, with a size cap (media.maxFileBytes, default 50 MiB) and 2 retries on network or 5xx errors.
• 4xx marks the URL expired. The item is re-fetched with its message later.
• Previews use the Discord CDN resizer (?width=320&height=320&format=webp) and are stored by hash in thumbs/. A preview failure never fails the scrape.

Job states

queued, running, paused, stopping, done, failed, pending-delete, deleting, deleted.

Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab and the MCP get_media_scrape tool read the same fields.

Tests

• Unit tests run against the mock core and a fake slopbot REST. No live Discord.
• Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
• The live run is a private test guild first, then a real channel.

Checkpoints

Implementation goals live in CURRENT.md. The engine is C3; slopbot is C4.