121 lines
5.0 KiB
Markdown
121 lines
5.0 KiB
Markdown
# Media scraper
|
|
|
|
The scrape engine walks a channel's history and fills a per-scrape database. It is backend
|
|
agnostic: slopbot for guild channels, the puppet's core bridge for everything including group DMs.
|
|
See [slopbot.md](slopbot.md) and [core-bridge.md](core-bridge.md).
|
|
|
|
## Backends
|
|
|
|
Both backends implement one interface:
|
|
|
|
```ts
|
|
interface ScrapeBackend {
|
|
fetchPage(channelId: string, before: string | null, limit: number):
|
|
Promise<{ messages: RawMessage[]; rateLimited?: { retryAfterMs: number } }>;
|
|
}
|
|
```
|
|
|
|
| Backend | Implementation |
|
|
| --- | --- |
|
|
| `slopbot` | discord.js REST (`channels.messages.fetch`), bucket-aware client |
|
|
| `puppet` | The core action `message.fetch` over the bridge |
|
|
|
|
The daemon picks `slopbot` when the channel is a guild channel and slopbot is configured and in the
|
|
guild; otherwise `puppet`. A scrape records the backend it used. The walk can resume on the same
|
|
backend only.
|
|
|
|
## Walk
|
|
|
|
1. The command resolves the channel and creates a scrape directory.
|
|
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
|
|
seen.
|
|
3. Extract media from each message and write the page to the database in one transaction.
|
|
4. Persist `scrape.json` (cursor, counters, state) after every page.
|
|
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
|
|
6. Resume after a daemon restart from the stored cursor.
|
|
|
|
Rules:
|
|
|
|
- One running scrape at a time. Further requests queue.
|
|
- `stop` marks the job and lets the current page finish.
|
|
- Every page is an outbound read. Sends from commands keep their own pacing.
|
|
|
|
## Pacing and backoff
|
|
|
|
The engine is deliberately slower than normal reads. All values are configurable; defaults are
|
|
starting points, not Discord facts.
|
|
|
|
| Setting | Default |
|
|
| --- | --- |
|
|
| `media.pageMinMs` / `media.pageMaxMs` | 1200 / 3500 |
|
|
| `media.pagesPerHour` | 300 |
|
|
| `media.pagesPerDay` | 3000 |
|
|
| `media.maxRetries` | 8 |
|
|
|
|
- A random gap in the page window between pages.
|
|
- A rolling budget per hour and per day. At the cap the job pauses and resumes when the window
|
|
resets.
|
|
- On failure: use the backend's `retryAfterMs` when present, else exponential `2^attempt + 1`
|
|
seconds, matching DiscordChatExporter. After `maxRetries` the job pauses instead of looping.
|
|
- DiscordChatExporter treats user-token limits more carefully than bot limits. The puppet backend
|
|
uses the slower schedule and the tighter budgets; slopbot can run faster and lets discord.js
|
|
handle buckets.
|
|
|
|
## Classification
|
|
|
|
The extractor maps one message to zero or more media items and returns their kind, source URL,
|
|
provider, dimensions, duration when known, and size when known.
|
|
|
|
- Attachments from `attachments[]`: extension and `contentType` decide image, gif, video, audio.
|
|
- Embeds from `embeds[]`: `type` and `provider` decide. `video` embeds become kind `video` with
|
|
`downloaded = false`. Image and gif embeds become cached previews only.
|
|
- Forwarded message snapshots are treated as messages with their own media.
|
|
- Stickers become images or gifs.
|
|
- Voice messages (`voiceMessage` flag) become audio.
|
|
- Third-party embeds are never downloaded, by rule.
|
|
|
|
## Database schema
|
|
|
|
One SQLite file per scrape (`index.db`, `node:sqlite`).
|
|
|
|
| Table | Columns |
|
|
| --- | --- |
|
|
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
|
|
| `media` | `id`, `kind`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `duration_ms`, `downloaded`, `created_at` |
|
|
| `media_sources` | `media_id`, `message_id` |
|
|
| `downloads` | `media_id`, `state`, `attempts`, `error` |
|
|
|
|
Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.json`, and
|
|
`file_hash` for dedupe. The database is an index; the media folder holds bytes.
|
|
|
|
## Downloads and previews
|
|
|
|
**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
|
|
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
|
|
|
|
The plan:
|
|
|
|
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
|
|
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
|
|
- `4xx` marks the URL expired. The item is re-fetched with its message later.
|
|
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
|
|
in `thumbs/`. A preview failure never fails the scrape.
|
|
|
|
## Job states
|
|
|
|
`queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.
|
|
|
|
Counters: pages, messages, media by kind, downloaded bytes, preview bytes, errors. The Media tab
|
|
and the MCP `get_media_scrape` tool read the same fields.
|
|
|
|
## Tests
|
|
|
|
- Unit tests run against the mock core and a fake slopbot REST. No live Discord.
|
|
- Fixtures cover each media kind, forwards, voice messages, expired URLs, and 429 backoff.
|
|
- The live run is a private test guild first, then a real channel.
|
|
|
|
## Checkpoints
|
|
|
|
Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4; daemon
|
|
wiring (command, service, routes) is C5.
|