Resolve external embed previews and keep their real MIME types

This commit is contained in:
Fisto
2026-09-16 12:52:44 +02:00
parent 7fa6118b6f
commit 1b5fcfed2f
11 changed files with 313 additions and 41 deletions
+6 -2
View File
@@ -120,9 +120,13 @@ Previews run after every walk:
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
`pendingDownloads`.
- Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first:
provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap,
10 s timeout, one attempt).
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape.