Fetch previews in batches with a rate-limit cooldown

2026-09-16 23:51:16 +02:00
parent 8f30ec14ad
commit 6c500f0cc5
7 changed files with 78 additions and 1 deletions
@@ -177,6 +177,8 @@ Verified live on 2026-09-13: the web UI started the client under auto-Xvfb, the
 | `VESKSLOP_MEDIA_MAX_FILE_BYTES` | `52428800` | Download size cap (50 MiB). Larger files are skipped. |
 | `VESKSLOP_MEDIA_PREVIEW_MAX_BYTES` | `5242880` | Preview size cap (5 MiB). Larger previews are skipped. |
 | `VESKSLOP_MEDIA_PREVIEW_MIN_MS` / `_MAX_MS` | `200` / `600` | Gap between preview fetches. |
+| `VESKSLOP_MEDIA_PREVIEW_BATCH` | `25` | Previews per batch before the cooldown. |
+| `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS` | `60000` | Cooldown between preview batches. `0` disables batching. |
 | `VESKSLOP_FFPROBE` | `ffprobe` | Probe binary used to fill missing dimensions after a download. |
 | `VESKSLOP_MEDIA_DOWNLOAD_MIN_MS` / `_MAX_MS` | `1000` / `3000` | Gap between downloaded files. |
 | `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. |
@@ -145,7 +145,9 @@ Previews run after every walk:
 - At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their
  previews were cached, so a stopped scrape does not leave blank tiles.
 - Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than
-  file downloads.
+  file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches
+  (`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of
+  thumbnail fetches trigger Discord and provider rate limits. A batch counts rows, not successes.

 ## Job states