Add message counting and date-bounded scrapes
This commit is contained in:
@@ -28,7 +28,8 @@ backend only.
|
||||
|
||||
1. The command resolves the channel and creates a scrape directory.
|
||||
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
|
||||
seen.
|
||||
seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
|
||||
boundary is crossed (`afterId`/`beforeId` in the meta).
|
||||
3. Extract media from each message and write the page to the database in one transaction.
|
||||
4. Persist `scrape.json` (cursor, counters, state) after every page.
|
||||
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
|
||||
@@ -85,7 +86,9 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
|
||||
| `media_sources` | `media_id`, `message_id` |
|
||||
| `downloads` | `media_id`, `state`, `attempts`, `error` |
|
||||
|
||||
Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media
|
||||
`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
|
||||
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
|
||||
`media_sources(message_id)`. The database is an index; the media
|
||||
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
|
||||
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
|
||||
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
|
||||
|
||||
Reference in New Issue
Block a user