Cap transient preview retries and close dead hosts
This commit is contained in:
@@ -147,7 +147,8 @@ Previews run after every walk:
|
||||
(`ffmpeg -filter_complex showwavespic`) as their preview, rendered by ffmpeg from the URL.
|
||||
- Transient network failures are left pending for the next backfill pass; only skips (4xx, too
|
||||
large, no source) are marked permanent.
|
||||
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
|
||||
- One attempt per row, no retries; a permanent failure (`http 404`, `http 403`, `too large`) marks
|
||||
`preview_error` and never fails the scrape.
|
||||
- Stored by hash in `thumbs/<sha256>.<ext>`, where the extension follows the response type
|
||||
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
|
||||
counter is `previewed`. Previews count toward the storage limit.
|
||||
@@ -155,6 +156,9 @@ Previews run after every walk:
|
||||
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
|
||||
- At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their
|
||||
previews were cached, so a stopped scrape does not leave blank tiles.
|
||||
- Transient fetch failures keep the row pending and count up `preview_attempts`. After
|
||||
`VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` tries (default 3) the row is marked
|
||||
`unreachable after N attempt(s)`, so dead hosts (Gfycat, Streamable) stop burning batches forever.
|
||||
- Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than
|
||||
file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches
|
||||
(`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of
|
||||
|
||||
Reference in New Issue
Block a user