Add custom download destinations, confirmations, and the bot error feed

This commit is contained in:
Fisto
2026-09-16 11:53:59 +02:00
parent 973b9cff92
commit 42fd52b96b
14 changed files with 471 additions and 62 deletions
+1 -1
View File
@@ -35,7 +35,7 @@ preset keys from `filter-presets.ts`; `after`/`before` are ISO times on `posted_
| Tool | Access | Purpose |
| --- | --- | --- |
| `scrape_media` | trusted | Start a scrape (`channelId`, `mode`) |
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`) |
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
| `stop_media_scrape` | trusted | Stop the active scrape |
+3
View File
@@ -103,6 +103,9 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
`file_bytes` are written to the row, so the run resumes and dedupes by content.
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
(`source != 'embed'`).
- With a custom destination, files go to `<destination>/<scrape-id>/media/` and the row stores the
absolute path; without one they stay in `<scrape>/media/`. Embed previews are copied to
`<destination>/<scrape-id>/thumbnails/` while the local copy stays for the Media tab.
- Progress counters: `downloaded`, `downloadErrors`.
Previews run after every walk:
+11 -4
View File
@@ -80,6 +80,10 @@ media-scrape/db/
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
are never download targets. The Media tab prefers the local file when one exists, so the archive
survives CDN URL expiry.
- A custom download destination sends the bytes to `<dir>/<scrape-id>/media/` instead. Previews
stay local, and embed thumbnails are copied to `<dir>/<scrape-id>/thumbnails/` so the download
has a poster. Files in a custom destination are not served or linked by the Media tab; the user
owns that path.
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk, in both modes: Discord
@@ -134,10 +138,13 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
21:9, Other), resolution (SD, HD, FHD, QHD, 4K+, Unknown), and a posted-date range (from/to).
Grid with lazy loading; previews come from the cache, then the local file, then the CDN; each
tile's tooltip shows the poster and posted time. A scrape strip shows live progress with Stop.
**New scrape…** opens a popup with Server and Group chat trees (servers expand to channels; each
row has Links and Download), and **History** lists each scrape with its start time, duration,
pages, messages, media counts, bytes, files, and previews. The view fills the window below the
header and the call strip.
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
channels; each row has Links and Download). Every scrape button asks for confirmation, and
Download can target a custom absolute destination. **History** lists each scrape with its start
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
that disappears when the tab is opened and a Clear button.
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
database overhead.
+3 -1
View File
@@ -263,7 +263,9 @@ replaceable.
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`. Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local). Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
| GET | `/api/media/errors` | none | Bot and scrape errors, newest first |
| POST | `/api/media/errors/clear` | none | Clear the error list |
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
| POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent |
| GET | `/api/logs` | `limit` | The persistent action log |