Cap transient preview retries and close dead hosts

This commit is contained in:
Fisto
2026-09-17 08:44:55 +02:00
parent 445c6ca2fb
commit e52623f1b2
6 changed files with 104 additions and 7 deletions
+5 -1
View File
@@ -147,7 +147,8 @@ Previews run after every walk:
(`ffmpeg -filter_complex showwavespic`) as their preview, rendered by ffmpeg from the URL.
- Transient network failures are left pending for the next backfill pass; only skips (4xx, too
large, no source) are marked permanent.
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- One attempt per row, no retries; a permanent failure (`http 404`, `http 403`, `too large`) marks
`preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
counter is `previewed`. Previews count toward the storage limit.
@@ -155,6 +156,9 @@ Previews run after every walk:
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their
previews were cached, so a stopped scrape does not leave blank tiles.
- Transient fetch failures keep the row pending and count up `preview_attempts`. After
`VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` tries (default 3) the row is marked
`unreachable after N attempt(s)`, so dead hosts (Gfycat, Streamable) stop burning batches forever.
- Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than
file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches
(`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of