zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Discord-hosted preview caching (C11)

Browse Source
This commit is contained in:
Fisto
2026-09-16 11:09:51 +02:00
parent b94031d816
commit 2573dd4d63
14 changed files with 242 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/daemon.md
+1
View File
@@ -174,6 +174,7 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
| `VESKSLOP_AUDIO_CLIENT_MATCH` | `vesktop,electron,chromium,discord` | Stream name patterns moved into the capture sink. |
| `VESKSLOP_MEDIA_DB_DIR` | `<repo>/media-scrape/db` | Media archive root: per-scrape databases, downloads, previews. |
| `VESKSLOP_MEDIA_MAX_FILE_BYTES` | `52428800` | Download size cap (50 MiB). Larger files are skipped. |
| `VESKSLOP_MEDIA_PREVIEW_MAX_BYTES` | `5242880` | Preview size cap (5 MiB). Larger previews are skipped. |
| `VESKSLOP_MEDIA_DOWNLOAD_MIN_MS` / `_MAX_MS` | `1000` / `3000` | Gap between downloaded files. |
| `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. |
| `VESKSLOP_TTS_LEAD_MS` | `150` | Lead-in silence before each TTS line (0 disables, max 2000). |
docs/media-scraper.md
+8 -2
View File
@@ -101,8 +101,14 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
(`source != 'embed'`).
- Progress counters: `downloaded`, `downloadErrors`.
Previews are **planned (C11)**: the Discord CDN resizer (`?width=320&height=320&format=webp`) into
`thumbs/`, stored by hash, failure never fails the scrape.
Previews run after every walk, in both modes:
- Only Discord-hosted image and GIF URLs (`cdn.discordapp.com`, `media.discordapp.net`) are
transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`).
Third-party embeds are skipped.
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.
## Job states
docs/media.md
+5 -4
View File
@@ -8,8 +8,8 @@ the byte downloader are implemented (C3-C8, C10). The Media and Bot tabs are C6.
- slopbot on a larger channel: 11 pages, 1088 messages (96 images, 1 GIF, 203 videos), 0 errors.
- puppet on a group DM and a 1:1 DM: done, 0 errors, backend `puppet`.
Remaining: preview caching (C11) and a 429 observation on a very large walk. Checkpoints live in
[CURRENT.md](../CURRENT.md).
Preview caching is implemented and live-verified (2 WebP thumbs served over the thumb route).
Remaining: a 429 observation on a very large walk. Checkpoints live in [CURRENT.md](../CURRENT.md).
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
@@ -80,8 +80,9 @@ media-scrape/db/
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
are never download targets. The Media tab prefers the local file when one exists, so the archive
survives CDN URL expiry.
- Cached image previews are still planned (C11). Until then, the Media tab hotlinks Discord CDN URLs
in `links` mode.
- Discord-hosted image and GIF previews are cached small (WebP, 320px) in `thumbs/` after every
walk, in both modes, so the grid does not depend on expiring CDN links. Third-party embed
thumbnails are never fetched.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
docs/webui.md
+1
View File
@@ -251,6 +251,7 @@ replaceable.
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
| GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first; `localUrl` when a file was downloaded |
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
| GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached WebP preview |
| GET | `/api/media/scrapes` | none | Every scrape with state and counters |
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |