zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Discord-hosted preview caching (C11)

Browse Source
This commit is contained in:
Fisto
2026-09-16 11:09:51 +02:00
parent b94031d816
commit 2573dd4d63
14 changed files with 242 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+8 -2
View File
@@ -101,8 +101,14 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
(`source != 'embed'`).
- Progress counters: `downloaded`, `downloadErrors`.
Previews are **planned (C11)**: the Discord CDN resizer (`?width=320&height=320&format=webp`) into
`thumbs/`, stored by hash, failure never fails the scrape.
Previews run after every walk, in both modes:
- Only Discord-hosted image and GIF URLs (`cdn.discordapp.com`, `media.discordapp.net`) are
transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`).
Third-party embeds are skipped.
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.
## Job states