Resolve external embed previews and keep their real MIME types
This commit is contained in:
@@ -120,9 +120,13 @@ Previews run after every walk:
|
||||
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
|
||||
and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
|
||||
`pendingDownloads`.
|
||||
- Embeds without a thumbnail in Discord's data (fxtwitter, vxReddit, ...) are resolved first:
|
||||
provider oEmbed/fxtwitter API, then the page's `og:image`/`twitter:image` (512 KiB read cap,
|
||||
10 s timeout, one attempt).
|
||||
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
|
||||
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
|
||||
the counter is `previewed`. Previews count toward the storage limit.
|
||||
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
|
||||
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
|
||||
counter is `previewed`. Previews count toward the storage limit.
|
||||
- The job stays `running` through the download and preview passes, so a client never sees `done`
|
||||
on a half-archived scrape.
|
||||
|
||||
|
||||
+9
-3
@@ -86,9 +86,15 @@ media-scrape/db/
|
||||
owns that path.
|
||||
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
|
||||
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
|
||||
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk, in both modes: Discord
|
||||
hosted images and GIFs, and third-party embed thumbnails (YouTube, X, TikTok, and so on), so the
|
||||
grid does not depend on expiring CDN links. Previews count toward the storage limit.
|
||||
- Previews are cached in `thumbs/` after every walk, in both modes: Discord-hosted images and GIFs
|
||||
(resized to 320px WebP) and third-party embed thumbnails (YouTube, X, TikTok, and so on). The
|
||||
stored bytes keep their real type (`preview_ext`), so JPEG provider thumbnails are served as
|
||||
`image/jpeg`, not mislabeled WebP. Embeds without a thumbnail in Discord's data are resolved
|
||||
through the provider's oEmbed/fxtwitter API and finally the page's `og:image`. Previews count
|
||||
toward the storage limit.
|
||||
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
|
||||
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
|
||||
labeled placeholder, and a failed image falls back to that placeholder.
|
||||
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
|
||||
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
|
||||
filled in with `ffprobe` after the file lands.
|
||||
|
||||
Reference in New Issue
Block a user