Add message counting and date-bounded scrapes
This commit is contained in:
+2
-1
@@ -78,7 +78,8 @@ The daemon exposes MCP over Streamable HTTP. The default address is `http://127.
|
||||
| `list_dms` | List cached DM and group DM channels with recipients. |
|
||||
| `get_message_log` | Read the trigger log. |
|
||||
| `get_action_log` | Read the persistent action log. |
|
||||
| `scrape_media` | Start a media scrape of a channel (`channelId`, `guildId?`, `mode?`, `confirm?`). |
|
||||
| `scrape_media` | Start a media scrape of a channel (`channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?`, `after?`, `before?`). |
|
||||
| `count_media_messages` | Discord search total for a channel or date range (slopbot guild channels only). |
|
||||
| `list_media_scrapes` | Media scrapes with state, pages, messages, and counts. |
|
||||
| `get_media_scrape` | One scrape by ID. |
|
||||
| `stop_media_scrape` | Stop a scrape; it pauses after the current page. |
|
||||
|
||||
+2
-1
@@ -35,7 +35,8 @@ preset keys from `filter-presets.ts`; `after`/`before` are ISO times on `posted_
|
||||
|
||||
| Tool | Access | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`) |
|
||||
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`, `after?`, `before?`) |
|
||||
| `count_media_messages` | trusted | Discord search total for a channel or range (slopbot guild channels only) |
|
||||
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
|
||||
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
|
||||
| `stop_media_scrape` | trusted | Stop the active scrape |
|
||||
|
||||
@@ -28,7 +28,8 @@ backend only.
|
||||
|
||||
1. The command resolves the channel and creates a scrape directory.
|
||||
2. Newest to oldest: fetch `limit` messages before the cursor. The cursor is the oldest message ID
|
||||
seen.
|
||||
seen. A to-date sets the starting cursor; a from-date trims each page and ends the walk when the
|
||||
boundary is crossed (`afterId`/`beforeId` in the meta).
|
||||
3. Extract media from each message and write the page to the database in one transaction.
|
||||
4. Persist `scrape.json` (cursor, counters, state) after every page.
|
||||
5. Stop when a page returns fewer than `limit` messages or the oldest message is reached.
|
||||
@@ -85,7 +86,9 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
|
||||
| `media_sources` | `media_id`, `message_id` |
|
||||
| `downloads` | `media_id`, `state`, `attempts`, `error` |
|
||||
|
||||
Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media
|
||||
`scrape.json` also records `fromDate`/`toDate`, and `estimatedMessages` from Discord search when the
|
||||
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
|
||||
`media_sources(message_id)`. The database is an index; the media
|
||||
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
|
||||
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
|
||||
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
|
||||
|
||||
+3
-2
@@ -139,8 +139,9 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
|
||||
Grid with lazy loading; previews come from the cache, then the local file, then the CDN; each
|
||||
tile's tooltip shows the poster and posted time. A scrape strip shows live progress with Stop.
|
||||
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
|
||||
channels; each row has Links and Download). Every scrape button asks for confirmation, and
|
||||
Download can target a custom absolute destination. **History** lists each scrape with its start
|
||||
channels; each row has Links and Download). Every scrape button asks for confirmation; the
|
||||
dialog takes optional From/To dates and a **Check Message Count** button (Discord search total,
|
||||
slopbot guild channels only), and Download can target a custom absolute destination. **History** lists each scrape with its start
|
||||
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
|
||||
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
|
||||
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
|
||||
|
||||
@@ -186,6 +186,10 @@ and fetches one history page without printing the token:
|
||||
cd poc
|
||||
VESKSLOP_SLOPBOT_ENABLED=1 VESKSLOP_SLOPBOT_STATE=/path/to/slopbot.json \
|
||||
pnpm test:slopbot:live <channelId> [guildId]
|
||||
|
||||
# Search total for a channel or range, without scraping:
|
||||
VESKSLOP_SLOPBOT_ENABLED=1 VESKSLOP_SLOPBOT_STATE=/path/to/slopbot.json \
|
||||
pnpm test:slopbot:count <channelId> <guildId> [afterIso] [beforeIso]
|
||||
```
|
||||
|
||||
## Sanctioned live target
|
||||
|
||||
+2
-1
@@ -264,7 +264,8 @@ replaceable.
|
||||
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
|
||||
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
||||
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
||||
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local). Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
|
||||
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?`, `after?`, `before?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local); `after`/`before` are ISO dates bounding the walk. Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
|
||||
| POST | `/api/media/count` | `channelId`, `guildId?`, `after?`, `before?` | Discord search total for the channel or range. Returns `{ count }` or `{ count: null, reason }` (slopbot guild channels only) |
|
||||
| GET | `/api/media/errors` | none | Bot and scrape errors, newest first |
|
||||
| POST | `/api/media/errors/clear` | none | Clear the error list |
|
||||
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
|
||||
|
||||
Reference in New Issue
Block a user