Resolve external embed previews and keep their real MIME types

This commit is contained in:
Fisto
2026-09-16 12:52:44 +02:00
parent 7fa6118b6f
commit 1b5fcfed2f
11 changed files with 313 additions and 41 deletions
+6 -2
View File
@@ -120,9 +120,13 @@ Previews run after every walk:
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
`pendingDownloads`.
- Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first:
provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap,
10 s timeout, one attempt).
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape.
+9 -3
View File
@@ -86,9 +86,15 @@ media-scrape/db/
owns that path.
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk, in both modes: Discord
hosted images and GIFs, and third-party embed thumbnails (YouTube, X, TikTok, and so on), so the
grid does not depend on expiring CDN links. Previews count toward the storage limit.
- Previews are cached in `thumbs/` after every walk, in both modes: Discord-hosted images and GIFs
(resized to 320px WebP) and third-party embed thumbnails (YouTube, X, TikTok, and so on). The
stored bytes keep their real type (`preview_ext`), so JPEG provider thumbnails are served as
`image/jpeg`, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved
through the provider's oEmbed/fxtwitter API and finally the page's `og:image`. Previews count
toward the storage limit.
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
labeled placeholder, and a failed image falls back to that placeholder.
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
filled in with `ffprobe` after the file lands.