zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cache embed thumbnails in download mode

Browse Source
This commit is contained in:
Fisto
2026-09-16 11:17:37 +02:00
parent 902646224f
commit 1fe204b91e
7 changed files with 161 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+8 -4
View File
@@ -101,14 +101,18 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
(`source != 'embed'`).
- Progress counters: `downloaded`, `downloadErrors`.
Previews run after every walk, in both modes:
Previews run after every walk:
- Only Discord-hosted image and GIF URLs (`cdn.discordapp.com`, `media.discordapp.net`) are
transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`).
Third-party embeds are skipped.
- In both modes, Discord-hosted image and GIF URLs (`cdn.discordapp.com`, `media.discordapp.net`)
are transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`).
- In `download` mode only, embed rows also get their stored `preview_url` (YouTube, X, TikTok
thumbnails, and so on) cached. The embed's full media URL is never fetched; embed rows are
excluded from `pendingDownloads`.
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape.
## Job states
docs/media.md
+6 -8
View File
@@ -80,14 +80,12 @@ media-scrape/db/
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
are never download targets. The Media tab prefers the local file when one exists, so the archive
survives CDN URL expiry.
- Discord-hosted image and GIF previews are cached small (WebP, 320px) in `thumbs/` after every
walk, in both modes, so the grid does not depend on expiring CDN links. Third-party embed
thumbnails are never fetched.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
resizer) in both modes, so the Media grid does not depend on expiring CDN links. Previews count
toward the storage limit.
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
other embed stays a link. In `download` mode their thumbnail is cached (below); the video itself
is never fetched.
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk. In both modes this covers
Discord-hosted images and GIFs; in `download` mode it also covers embed thumbnails, so the grid
does not depend on expiring CDN links. Previews count toward the storage limit.
- Discord attachment URLs are signed and expire in about a day. In `links` mode an expired URL is
refreshed by re-fetching its message; in `download` mode the bytes are caught before expiry.