diff --git a/CURRENT.md b/CURRENT.md index 9aa7920..ec8527e 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -40,6 +40,19 @@ mobile draft at 390px. - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: `previewed: 2`; thumbs served over `/api/media///thumb` (200). + - [x] C54 Message links, suspect retries, and preview recovery. Items now carry `messageUrl` + (from `media_sources` + scrape guild), the viewer shows/copies it instead of the expiring CDN + link and opens a fresh stream for View full, YouTube items offer an **Open on YouTube** + fallback, and the metadata gained Channel/Scrape/Title (title includes the channel kept from + oEmbed, `title_author`). Preview and download HTTP failures on Discord hosts first refresh the + URL from the source message and retry; if that fails the row parks as a **suspect** with one + delayed retry (50 s, runs after the batch; a batch two-thirds suspect doubles the cooldown, + 60 s floor). Still-failing rows are **unverified** and skipped by normal passes; the Bot tab + shows suspect/unverified counts and a segmented progress bar (cached blue, pending gray, + failed red). **Recover Discord previews** is a paused one-shot task that re-fetches every + Discord-CDN row closed with an HTTP error across all scrapes (mixed bot/puppet via the + existing refresh path) and lands in the task history when done. All 1,208 Discord 404 rows + sampled 15/15 recoverable. - [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated `TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause (hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no diff --git a/docs/media-scraper.md b/docs/media-scraper.md index b5e6d7e..15415d5 100644 --- a/docs/media-scraper.md +++ b/docs/media-scraper.md @@ -210,6 +210,25 @@ a cached still show a labeled placeholder, and playback streams only when the us GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone. +## Failure policy and recovery + +The archive never treats a transient HTTP answer as a dead link on the first try. + +- A preview or download that fails with an HTTP status on Discord's CDN is retried once with a + **refreshed URL** from the source message (the signed link expires in about a day). Only if the + refreshed fetch fails is the row parked. +- Parked rows become **suspects**: they get one delayed retry (about 50 s). The retry runs after + the batch, not immediately. If a batch is two-thirds suspect failures, the cooldown doubles + (floor 60 s) and the retries run before the next batch. +- A suspect whose delayed retry fails again is left **unverified**: it is skipped by later passes, + so a finished backfill never churns over genuinely unavailable media. The Bot tab shows suspects + and unverified counts per member, and the progress bar fills blue for cached, gray for pending, + and red for failed rows. +- **Recover Discord previews** is a one-shot task in the Bot tab: it walks every scrape's + Discord-CDN rows closed with HTTP errors, refreshes each URL through the message (slopbot REST or + the puppet, whichever the scrape used), and fetches the preview again. It starts paused; when it + finishes it moves to the task history. The code stays for reuse. + ## Bot tab settings The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once. diff --git a/docs/media.md b/docs/media.md index 5510cfe..433ca22 100644 --- a/docs/media.md +++ b/docs/media.md @@ -101,6 +101,9 @@ media-scrape/db/ - The grid never renders an embed's page URL as media. A direct video file without a preview shows a first-frame `