zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Backfill previews for paused scrapes and prefer richer dedupe rows

Browse Source
This commit is contained in:
Fisto
2026-09-16 14:02:11 +02:00
parent 2309393e3c
commit 96fe17b4c9
8 changed files with 259 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+5 -1
View File
@@ -133,7 +133,11 @@ Previews run after every walk:
(`preview_ext`, default webp); `preview_hash` and `preview_bytes` land on the row, and the
counter is `previewed`. Previews count toward the storage limit.
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape.
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their
previews were cached, so a stopped scrape does not leave blank tiles.
- Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than
file downloads.
## Job states