Add message counting and date-bounded scrapes

This commit is contained in:
Fisto
2026-09-16 12:06:34 +02:00
parent f14f067413
commit fae754fba0
21 changed files with 410 additions and 22 deletions
+2 -1
View File
@@ -78,7 +78,8 @@ The daemon exposes MCP over Streamable HTTP. The default address is `http://127.
| `list_dms` | List cached DM and group DM channels with recipients. |
| `get_message_log` | Read the trigger log. |
| `get_action_log` | Read the persistent action log. |
| `scrape_media` | Start a media scrape of a channel (`channelId`, `guildId?`, `mode?`, `confirm?`). |
| `scrape_media` | Start a media scrape of a channel (`channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?`, `after?`, `before?`). |
| `count_media_messages` | Discord search total for a channel or date range (slopbot guild channels only). |
| `list_media_scrapes` | Media scrapes with state, pages, messages, and counts. |
| `get_media_scrape` | One scrape by ID. |
| `stop_media_scrape` | Stop a scrape; it pauses after the current page. |
+2 -1
View File
@@ -35,7 +35,8 @@ preset keys from `filter-presets.ts`; `after`/`before` are ISO times on `posted_
| Tool | Access | Purpose |
| --- | --- | --- |
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`) |
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`, `after?`, `before?`) |
| `count_media_messages` | trusted | Discord search total for a channel or range (slopbot guild channels only) |
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
| `stop_media_scrape` | trusted | Stop the active scrape |
+5 -2
View File
@@ -28,7 +28,8 @@ backend only.
1. The command resolves the channel and creates a scrape directory.
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
seen.
seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
boundary is crossed (`afterId`/`beforeId` in the meta).
3. Extract media from each message and write the page to the database in one transaction.
4. Persist `scrape.json` (cursor, counters, state) after every page.
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
@@ -85,7 +86,9 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media
`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
`media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
+3 -2
View File
@@ -139,8 +139,9 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
Grid with lazy loading; previews come from the cache, then the local file, then the CDN; each
tile's tooltip shows the poster and posted time. A scrape strip shows live progress with Stop.
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
channels; each row has Links and Download). Every scrape button asks for confirmation, and
Download can target a custom absolute destination. **History** lists each scrape with its start
channels; each row has Links and Download). Every scrape button asks for confirmation; the
dialog takes optional From/To dates and a **Check Message Count** button (Discord search total,
slopbot guild channels only), and Download can target a custom absolute destination. **History** lists each scrape with its start
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
+4
View File
@@ -186,6 +186,10 @@ and fetches one history page without printing the token:
cd poc
VESKSLOP_SLOPBOT_ENABLED=1 VESKSLOP_SLOPBOT_STATE=/path/to/slopbot.json \
pnpm test:slopbot:live <channelId> [guildId]
# Search total for a channel or range, without scraping:
VESKSLOP_SLOPBOT_ENABLED=1 VESKSLOP_SLOPBOT_STATE=/path/to/slopbot.json \
pnpm test:slopbot:count <channelId> <guildId> [afterIso] [beforeIso]
```
## Sanctioned live target
+2 -1
View File
@@ -264,7 +264,8 @@ replaceable.
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local). Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?`, `after?`, `before?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local); `after`/`before` are ISO dates bounding the walk. Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
| POST | `/api/media/count` | `channelId`, `guildId?`, `after?`, `before?` | Discord search total for the channel or range. Returns `{ count }` or `{ count: null, reason }` (slopbot guild channels only) |
| GET | `/api/media/errors` | none | Bot and scrape errors, newest first |
| POST | `/api/media/errors/clear` | none | Clear the error list |
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |