Add message counting and date-bounded scrapes

This commit is contained in:
Fisto
2026-09-16 12:06:34 +02:00
parent f14f067413
commit fae754fba0
21 changed files with 410 additions and 22 deletions
+5 -2
View File
@@ -28,7 +28,8 @@ backend only.
1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
seen.
seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
boundary is crossed (`afterId`/`beforeId` in the meta).
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
@@ -85,7 +86,9 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media
`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
`media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and