Add the media downloader with hash storage and local file serving

This commit is contained in:
Fisto
2026-09-16 10:35:55 +02:00
parent 3ea72f587d
commit f7664256d1
15 changed files with 431 additions and 28 deletions
+3
View File
@@ -172,6 +172,9 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
| `VESKSLOP_AUDIO_MANAGE_SERVER` | `1` | Start `pulseaudio` when `pactl` cannot reach a server. |
| `VESKSLOP_AUDIO_SINK` / `_SOURCE` / `_CAPTURE_SINK` | `veskslop_tts` / `veskslop_mic` / `veskslop_capture` | Virtual device names. |
| `VESKSLOP_AUDIO_CLIENT_MATCH` | `vesktop,electron,chromium,discord` | Stream name patterns moved into the capture sink. |
| `VESKSLOP_MEDIA_DB_DIR` | `<repo>/media-scrape/db` | Media archive root: per-scrape databases, downloads, previews. |
| `VESKSLOP_MEDIA_MAX_FILE_BYTES` | `52428800` | Download size cap (50 MiB). Larger files are skipped. |
| `VESKSLOP_MEDIA_DOWNLOAD_MIN_MS` / `_MAX_MS` | `1000` / `3000` | Gap between downloaded files. |
The daemon warns at startup when `dist/` is older than the core sources. Rebuild with `pnpm build`
and restart the client, or new actions fail with `UNKNOWN_ACTION`.
+11 -8
View File
@@ -90,16 +90,19 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.
## Downloads and previews
**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
Downloads run after a finished walk when the scrape used `mode: "download"`:
The plan:
- A serial queue, one file at a time, 1-3 s apart, with a size cap
(`VESKSLOP_MEDIA_MAX_FILE_BYTES`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
- The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and
`file_bytes` are written to the row, so the run resumes and dedupes by content.
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
(`source != 'embed'`).
- Progress counters: `downloaded`, `downloadErrors`.
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
in `thumbs/`. A preview failure never fails the scrape.
Previews are **planned (C11)**: the Discord CDN resizer (`?width=320&height=320&format=webp`) into
`thumbs/`, stored by hash, failure never fails the scrape.
## Job states
+11 -7
View File
@@ -1,10 +1,12 @@
# Media archive
Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
the `/api/media/*` routes, storage limits with retention, the Settings group, and the MCP tools are
implemented (C3-C8). The Media and Bot tabs are C6. The slopbot path is live-verified: the test
channel produced 17 messages and 4 images + 13 videos with no errors. The puppet path and a
`download` run are still owed (C9). Checkpoints live in [CURRENT.md](../CURRENT.md).
the `/api/media/*` routes, storage limits with retention, the Settings group, the MCP tools, and
the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6. Live-verified on
the test channel: links mode produced 17 messages (4 images, 13 videos) with no errors, and
download mode fetched the attachment (784 KB) and served it locally. Remaining: preview caching
(C11), the puppet `message.fetch` field check, and a 429 observation. Checkpoints live in
[CURRENT.md](../CURRENT.md).
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
@@ -72,9 +74,11 @@ media-scrape/db/
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
downloaded.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet**
(checkpoint C10): the command accepts the mode and the schema has the columns, but the download
queue is still owed.
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
are never download targets. The Media tab prefers the local file when one exists, so the archive
survives CDN URL expiry.
- Cached image previews are still planned (C11). Until then, the Media tab hotlinks Discord CDN URLs
in `links` mode.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
+2 -1
View File
@@ -249,7 +249,8 @@ replaceable.
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
| POST | `/api/audio/capture` | `start` | Start or stop capture |
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
| GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first |
| GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first; `localUrl` when a file was downloaded |
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
| GET | `/api/media/scrapes` | none | Every scrape with state and counters |
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |