Add the Bot-tab backfill control room and media strip rules

docs/daemon.md

@@ -183,15 +183,8 @@ daemon, so a hard kill never leaves ghost jobs.
 | `VESKSLOP_AUDIO_SINK` / `_SOURCE` / `_CAPTURE_SINK` | `veskslop_tts` / `veskslop_mic` / `veskslop_capture` | Virtual device names. |
 | `VESKSLOP_AUDIO_CLIENT_MATCH` | `vesktop,electron,chromium,discord` | Stream name patterns moved into the capture sink. |
 | `VESKSLOP_MEDIA_DB_DIR` | `<repo>/media-scrape/db` | Media archive root: per-scrape databases, downloads, previews. |
-| `VESKSLOP_MEDIA_MAX_FILE_BYTES` | `52428800` | Download size cap (50 MiB). Larger files are skipped. |
-| `VESKSLOP_MEDIA_PREVIEW_MAX_BYTES` | `5242880` | Preview size cap (5 MiB). Larger previews are skipped. |
-| `VESKSLOP_MEDIA_PREVIEW_MIN_MS` / `_MAX_MS` | `200` / `600` | Gap between preview fetches. |
-| `VESKSLOP_MEDIA_PREVIEW_BATCH` | `25` | Previews per batch before the cooldown. |
-| `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS` | `60000` | Cooldown between preview batches. `0` disables batching. |
-| `VESKSLOP_MEDIA_WALK_CYCLE_PAGES` | `5` | Walk pages between preview cycles on a live scrape. `0` defers previews to the end. |
-| `VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS` | `60000` | Minimum pause after a `429` before media traffic resumes. |
+| Pacing and limit knobs | Bot tab settings | Batch size, gaps, retries, and cooldowns live in the Bot tab (Backfilling / Scraping / Timers) and persist in `preferences.json`. |
 | `VESKSLOP_FFPROBE` | `ffprobe` | Probe binary used to fill missing dimensions after a download. |
-| `VESKSLOP_MEDIA_DOWNLOAD_MIN_MS` / `_MAX_MS` | `1000` / `3000` | Gap between downloaded files. |
 | `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. |
 | `VESKSLOP_TTS_LEAD_MS` | `150` | Lead-in silence before each TTS line (0 disables, max 2000). |
 | `VESKSLOP_FFPLAY` / `VESKSLOP_FFMPEG` | `ffplay` / `ffmpeg` | Playback and capture binaries. |

docs/media-scraper.md

@@ -106,7 +106,7 @@ filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
 Downloads run after a finished walk when the scrape used `mode: "download"`:
 
 - A serial queue, one file at a time, 1-3 s apart, with a size cap
-  (`VESKSLOP_MEDIA_MAX_FILE_BYTES`, default 50 MiB) and 2 retries on network or 5xx errors.
+  (the **Scraping** settings, default 50 MiB) and 2 retries on network or 5xx errors.
 - `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
 - The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and
   `file_bytes` are written to the row, so the run resumes and dedupes by content.
@@ -121,8 +121,8 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
   durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown.
 - Progress counters: `downloaded`, `downloadErrors`.
 
-Previews cycle with the walk: after every `VESKSLOP_MEDIA_WALK_CYCLE_PAGES` pages (default 5) the
-engine previews up to one batch (`VESKSLOP_MEDIA_PREVIEW_BATCH`, default 25), then keeps walking,
+Previews cycle with the walk: after every **Preview every N walk pages** setting (default 5) pages
+the engine previews up to one batch (the **Backfilling** batch size, default 25), then keeps walking,
 so tiles fill in during a long walk instead of only at the end. Set the cycle to `0` to defer all
 previews to the end.
 
@@ -154,15 +154,19 @@ Previews run after every walk:
   counter is `previewed`. Previews count toward the storage limit.
 - The job stays `running` through the download and preview passes, so a client never sees `done`
   on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
-- At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their
-  previews were cached, so a stopped scrape does not leave blank tiles.
-- Transient fetch failures keep the row pending and count up `preview_attempts`. After
-  `VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` tries (default 3) the row is marked
-  `unreachable after N attempt(s)`, so dead hosts (Gfycat, Streamable) stop burning batches forever.
-- Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than
-  file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches
-  (`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of
-  thumbnail fetches trigger Discord and provider rate limits. A batch counts rows, not successes.
+- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
+  before their previews were cached. One job per chat groups every paused, done, or failed scrape
+  of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
+  batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
+  Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. On daemon boot
+  jobs start automatically when the Backfilling setting **Start automatically on daemon boot** is
+  on.
+- Transient fetch failures keep the row pending and count up `preview_attempts`. After the
+  **Tries before giving up** setting (default 3) the row is marked `unreachable after N attempt(s)`,
+  so dead hosts (Gfycat, Streamable) stop burning batches forever.
+- Preview gaps and batching live in the Bot tab settings (**Backfilling**): gap 200-600 ms, batches
+  of 25 with a 60 s cooldown between them, because bursts of thumbnail fetches trigger Discord and
+  provider rate limits. A batch counts rows, not successes.
 
 ## Rate limits
 
@@ -173,8 +177,9 @@ traffic:
   permanent error) and is retried later.
 - A walk page that fails with a `429` or a rate-limit message registers a cooldown, and every page
   and preview waits it out before continuing.
-- The pause is `VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS` (default 60 s), or the server's `Retry-After`
-  when it is longer. Detection is logged as `rate limit detected; pausing media traffic until …`.
+- The pause is the **Timers** setting **Rate-limit cooldown** (default 60 s), or the server's
+  `Retry-After` when it is longer. Detection is logged as `rate limit detected; pausing media traffic
+  until …`. The Bot tab banner reads `API timed out, increase timers` while media traffic is held.
 
 ## Browsing cache
 
@@ -183,6 +188,27 @@ a cached still show a labeled placeholder, and playback streams only when the us
 GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
 frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.
 
+## Bot tab settings
+
+The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.
+
+| Group | Setting | Default |
+| --- | --- | --- |
+| Backfilling | Batch size (rows) | 25 |
+| Backfilling | Cooldown between batches (ms) | 60000 |
+| Backfilling | Tries before giving up | 3 |
+| Backfilling | Preview gap min / max (ms) | 200 / 600 |
+| Backfilling | Max preview size (MiB) | 5 |
+| Backfilling | Start automatically on daemon boot | on |
+| Scraping | Preview every N walk pages | 5 |
+| Scraping | Max file size (MiB) | 50 |
+| Scraping | Download gap min / max (ms) | 1000 / 3000 |
+| Scraping | Page fetch retries | 2 |
+| Timers | Walk page gap min / max (ms) | 1200 / 3500 |
+| Timers | Rate-limit cooldown (ms) | 60000 |
+
+The old `VESKSLOP_MEDIA_*` tuning variables are gone; the values above were their defaults.
+
 ## Job states
 
 `queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`.

docs/media.md

@@ -169,9 +169,13 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
   downloads, an ffprobe probe. GIF tiles carry a translucent **GIF** label top-left; the cached
   preview is a static frame and the animated GIF plays while hovering. The status row reads
   `(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V` — image counts include GIFs, and
-  each slot is the count for the active filters. A scrape strip shows live progress with Stop; it
-  polls every 4 seconds while the tab is open, lists only queued/running jobs (paused ones live in
-  History), and shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely
+  each slot is the count for the active filters. A scrape strip shows live progress and polls every
+  4 seconds while the tab is open. It shows every queued/running job plus at most one paused scrape
+  (the newest not dismissed); **Stop** on a paused scrape only hides it here, so a new paused scrape
+  can take its place while the hidden one stays resumable. The strip text turns red and reads
+  `API limited, retry in Ns` while Discord throttles media traffic, and a translucent
+  `N scrapes completed` line appears when scrapes finish — it lives until the tab is reloaded. Each
+  row shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely
   (a **Load more** button appears as a fallback). Media whose source answers `403` shows a
   **PRIVATE** tile and `404`/`410` shows **GONE**, with the reason in the tooltip. Some channels deny the bot the search endpoint (`Missing Access`) even though history
   reads work, so the total is optional.
@@ -183,13 +187,20 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
   backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
   below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
   that disappears when the tab is opened and a Clear button.
-- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
-  The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
-  database overhead.
+- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
+  **Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
+  current settings, and a Bot/Puppet split of pending/cached/failed. One job card per chat lists its
+  member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
+  counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
+  carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
+  while a Discord cooldown holds the queue. **Settings** holds three collapsible groups —
+  Backfilling, Scraping, Timers — each with a Save button; values persist in `preferences.json` and
+  apply to live jobs immediately. The **Errors** list keeps the bot and scrape errors with the red
+  badge and Clear button.
 - Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and
-  **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage
-  bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second
-  countdown.
+  **Media** if the bot path is removed. It holds the storage limit field with the info tooltip, the
+  usage bar (segmented by image, GIF, video, audio, and other), and the clean button with its
+  5 second countdown. Pacing and batching knobs live in the Bot tab, not here.
 - Mobile is out of scope for now.
 
 ## MCP

docs/testing.md

@@ -189,6 +189,8 @@ VESKSLOP_UI_URL=https://192.168.0.178:8789/ pnpm test:ui
 ```
 
 It opens the Media tab, loads tiles, verifies no broken images, applies the cached-only filter,
+checks the zoom slider, opens the Bot tab and asserts the backfill totals, the Bot/Puppet split and
+the three settings groups render,
 checks that the zoom slider changes the grid density, presses Reset, and fails on any console
 error. A screenshot lands in `/tmp/opencode/ui-smoke.png`.
 

docs/webui.md

@@ -142,10 +142,19 @@ box live in Chat, not Call, and the connection stats are not chips yet.
   **Links** and **Download** buttons, and group chats always use the puppet. **History** lists
   each scrape with start time, duration, pages, messages, media counts, bytes, files, and previews,
   and offers **Resume** for paused ones and **Delete** for anything not running.
-  The scrape strip shows live progress with a pulse and **Stop**. Images and GIFs render from the
-  cached preview when one exists, then the local file, then the CDN URL; videos and audio are link
-  cards with the poster and posted time in the tooltip.
-- Bot shows the slopbot connection state and the scrape queue.
+  The scrape strip shows live progress with a pulse and **Stop**: every running/queued scrape plus at
+  most one paused scrape, whose **Stop** only dismisses it from the strip (it stays resumable and the
+  next paused scrape can appear). The line turns red with `API limited, retry in Ns` while Discord
+  throttles, and a translucent `N scrapes completed` note appears as scrapes finish and clears when
+  the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local
+  file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
+  tooltip.
+- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has
+  totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
+  per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
+  **Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
+  three settings groups (Backfilling, Scraping, Timers) save to `preferences.json` and retune live
+  jobs. The error list sits below with the Clear button.
 - The views are self-contained: `poc/public/media.js` and `media.css`, over the `/api/media/*`
   routes in the table below. See [media.md](media.md).
 
@@ -273,7 +282,10 @@ replaceable.
 | GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) |
 | GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` |
 | GET | `/api/media/scrapes` | none | Every scrape with state and counters |
-| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
+| GET | `/api/media/status` | none | Slopbot connection state, the scrape list, and `limitedUntil` while a Discord cooldown holds media traffic |
+| GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, ETA, totals with a Bot/Puppet split, the live settings, and `limitedUntil` |
+| POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job |
+| POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch |
 | GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
 | GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
 | GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
@@ -285,6 +297,7 @@ replaceable.
 | GET | `/api/media/:scrapeId/:itemId/anim` | none | The cached animated GIF for hover playback |
 | POST | `/api/daemon/restart` | none | Replace the daemon process with a fresh copy (the client keeps running) |
 | POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
+| POST | `/api/preferences` | `visibleGuilds?`, `tts?`, `audio?`, `favourites?`, `media?` | Also accepts `media.tunables` with `backfill`, `scrape`, and `timers` groups; changes apply to live jobs immediately |
 | POST | `/api/media/scrape/resume` | `id` | Continue a paused scrape from its cursor |
 | POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent |
 | GET | `/api/logs` | `limit` | The persistent action log |