diff --git a/AGENTS.md b/AGENTS.md index 5bb16b3..24abc2f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -77,8 +77,8 @@ before you stop. | Voice video | Not started. Plan in [docs/voice.md](docs/voice.md). | | Notification parsing (edits, deletes, reactions, replies) | Implemented | | Reply-to-puppet as a trigger | Implemented | -| Media archive (`scrape media`, Media/Bot tabs) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). | -| Slopbot (dedicated bot core for scraping) | Design approved 2026-09-16. Checkpoints in [CURRENT.md](CURRENT.md). | +| Media archive (`scrape media`, Media/Bot tabs) | Implemented: walk, download, previews, per-chat backfill queue, filters, tunables. Checkpoints in [CURRENT.md](CURRENT.md). | +| Slopbot (dedicated bot core for scraping) | Implemented and live-verified. Checkpoints in [CURRENT.md](CURRENT.md). | | Auth persistence in the core | Implemented and verified live | | CDP transport | Kept as a development fallback | diff --git a/CURRENT.md b/CURRENT.md index 79eb4c6..5ff8828 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -40,6 +40,20 @@ mobile draft at 390px. - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: `previewed: 2`; thumbs served over `/api/media///thumb` (200). + - [x] C43 Bot-tab backfill control room and media-tab strip rules. The engine gained + `previewBatch`/`rateLimitStatus`/`updateOptions`; a new `BackfillQueue` groups preview work per + chat (paused and resumed scrapes of a chat combine), runs one job at a time in batches, and + reports pending/cached/failed per kind, throughput, measured and settings-based ETA, and the + live cooldown. `MediaService` drives it and exposes `GET /api/media/backfill` plus start/pause; + `/api/preferences` now carries `media.tunables` (Backfilling / Scraping / Timers, persisted and + applied live). The Bot tab shows totals with a Bot/Puppet split, per-chat job cards with + progress and Start/Pause, **Start backfill** / **Pause all**, the red `API timed out, increase + timers` banner, and the three settings groups. The media strip shows running plus one paused + scrape, dismisses a paused one on Stop without stopping it, turns red while limited, and shows + a session-only `N scrapes completed` note. The old `VESKSLOP_MEDIA_*` tuning env vars are gone. + New unit tests cover batching, the cooldown, live retuning, and queue grouping/finish/pause; + the Playwright smoke test covers the Bot tab. Live: daemon restarted, 6018 pending / 13321 + cached / 652 failed, ETA 4h 7m at 21.4 rows/min, settings round-trip verified. - [x] C42 Preview attempts cap. The restart log showed the backfill retrying dead hosts (Gfycat DNS is gone, Streamable answers 400) forever as "transient". Rows now carry `preview_attempts`; after `VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` (3) the row is marked diff --git a/docs/daemon.md b/docs/daemon.md index 1a51cbd..9fa3d3f 100644 --- a/docs/daemon.md +++ b/docs/daemon.md @@ -183,15 +183,8 @@ daemon, so a hard kill never leaves ghost jobs. | `VESKSLOP_AUDIO_SINK` / `_SOURCE` / `_CAPTURE_SINK` | `veskslop_tts` / `veskslop_mic` / `veskslop_capture` | Virtual device names. | | `VESKSLOP_AUDIO_CLIENT_MATCH` | `vesktop,electron,chromium,discord` | Stream name patterns moved into the capture sink. | | `VESKSLOP_MEDIA_DB_DIR` | `/media-scrape/db` | Media archive root: per-scrape databases, downloads, previews. | -| `VESKSLOP_MEDIA_MAX_FILE_BYTES` | `52428800` | Download size cap (50 MiB). Larger files are skipped. | -| `VESKSLOP_MEDIA_PREVIEW_MAX_BYTES` | `5242880` | Preview size cap (5 MiB). Larger previews are skipped. | -| `VESKSLOP_MEDIA_PREVIEW_MIN_MS` / `_MAX_MS` | `200` / `600` | Gap between preview fetches. | -| `VESKSLOP_MEDIA_PREVIEW_BATCH` | `25` | Previews per batch before the cooldown. | -| `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS` | `60000` | Cooldown between preview batches. `0` disables batching. | -| `VESKSLOP_MEDIA_WALK_CYCLE_PAGES` | `5` | Walk pages between preview cycles on a live scrape. `0` defers previews to the end. | -| `VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS` | `60000` | Minimum pause after a `429` before media traffic resumes. | +| Pacing and limit knobs | Bot tab settings | Batch size, gaps, retries, and cooldowns live in the Bot tab (Backfilling / Scraping / Timers) and persist in `preferences.json`. | | `VESKSLOP_FFPROBE` | `ffprobe` | Probe binary used to fill missing dimensions after a download. | -| `VESKSLOP_MEDIA_DOWNLOAD_MIN_MS` / `_MAX_MS` | `1000` / `3000` | Gap between downloaded files. | | `VESKSLOP_AUDIO_DUCK` | `30` | Sink volume percent while others speak. | | `VESKSLOP_TTS_LEAD_MS` | `150` | Lead-in silence before each TTS line (0 disables, max 2000). | | `VESKSLOP_FFPLAY` / `VESKSLOP_FFMPEG` | `ffplay` / `ffmpeg` | Playback and capture binaries. | diff --git a/docs/media-scraper.md b/docs/media-scraper.md index e7a332b..c44a929 100644 --- a/docs/media-scraper.md +++ b/docs/media-scraper.md @@ -106,7 +106,7 @@ filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available. Downloads run after a finished walk when the scrape used `mode: "download"`: - A serial queue, one file at a time, 1-3 s apart, with a size cap - (`VESKSLOP_MEDIA_MAX_FILE_BYTES`, default 50 MiB) and 2 retries on network or 5xx errors. + (the **Scraping** settings, default 50 MiB) and 2 retries on network or 5xx errors. - `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried. - The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and `file_bytes` are written to the row, so the run resumes and dedupes by content. @@ -121,8 +121,8 @@ Downloads run after a finished walk when the scrape used `mode: "download"`: durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown. - Progress counters: `downloaded`, `downloadErrors`. -Previews cycle with the walk: after every `VESKSLOP_MEDIA_WALK_CYCLE_PAGES` pages (default 5) the -engine previews up to one batch (`VESKSLOP_MEDIA_PREVIEW_BATCH`, default 25), then keeps walking, +Previews cycle with the walk: after every **Preview every N walk pages** setting (default 5) pages +the engine previews up to one batch (the **Backfilling** batch size, default 25), then keeps walking, so tiles fill in during a long walk instead of only at the end. Set the cycle to `0` to defer all previews to the end. @@ -154,15 +154,19 @@ Previews run after every walk: counter is `previewed`. Previews count toward the storage limit. - The job stays `running` through the download and preview passes, so a client never sees `done` on a half-archived scrape. A paused walk still gets these passes for the pages it collected. -- At daemon start, `backfillPreviews` runs the same pass over scrapes that paused before their - previews were cached, so a stopped scrape does not leave blank tiles. -- Transient fetch failures keep the row pending and count up `preview_attempts`. After - `VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS` tries (default 3) the row is marked - `unreachable after N attempt(s)`, so dead hosts (Gfycat, Streamable) stop burning batches forever. -- Preview gaps default to 200-600 ms (`VESKSLOP_MEDIA_PREVIEW_MIN_MS`/`_MAX_MS`), much faster than - file downloads. Previews are fetched in **batches of 25 with a 60 s cooldown** between batches - (`VESKSLOP_MEDIA_PREVIEW_BATCH`, `VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS`), because bursts of - thumbnail fetches trigger Discord and provider rate limits. A batch counts rows, not successes. +- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished + before their previews were cached. One job per chat groups every paused, done, or failed scrape + of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in + batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The + Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. On daemon boot + jobs start automatically when the Backfilling setting **Start automatically on daemon boot** is + on. +- Transient fetch failures keep the row pending and count up `preview_attempts`. After the + **Tries before giving up** setting (default 3) the row is marked `unreachable after N attempt(s)`, + so dead hosts (Gfycat, Streamable) stop burning batches forever. +- Preview gaps and batching live in the Bot tab settings (**Backfilling**): gap 200-600 ms, batches + of 25 with a 60 s cooldown between them, because bursts of thumbnail fetches trigger Discord and + provider rate limits. A batch counts rows, not successes. ## Rate limits @@ -173,8 +177,9 @@ traffic: permanent error) and is retried later. - A walk page that fails with a `429` or a rate-limit message registers a cooldown, and every page and preview waits it out before continuing. -- The pause is `VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS` (default 60 s), or the server's `Retry-After` - when it is longer. Detection is logged as `rate limit detected; pausing media traffic until …`. +- The pause is the **Timers** setting **Rate-limit cooldown** (default 60 s), or the server's + `Retry-After` when it is longer. Detection is logged as `rate limit detected; pausing media traffic + until …`. The Bot tab banner reads `API timed out, increase timers` while media traffic is held. ## Browsing cache @@ -183,6 +188,27 @@ a cached still show a labeled placeholder, and playback streams only when the us GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone. +## Bot tab settings + +The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once. + +| Group | Setting | Default | +| --- | --- | --- | +| Backfilling | Batch size (rows) | 25 | +| Backfilling | Cooldown between batches (ms) | 60000 | +| Backfilling | Tries before giving up | 3 | +| Backfilling | Preview gap min / max (ms) | 200 / 600 | +| Backfilling | Max preview size (MiB) | 5 | +| Backfilling | Start automatically on daemon boot | on | +| Scraping | Preview every N walk pages | 5 | +| Scraping | Max file size (MiB) | 50 | +| Scraping | Download gap min / max (ms) | 1000 / 3000 | +| Scraping | Page fetch retries | 2 | +| Timers | Walk page gap min / max (ms) | 1200 / 3500 | +| Timers | Rate-limit cooldown (ms) | 60000 | + +The old `VESKSLOP_MEDIA_*` tuning variables are gone; the values above were their defaults. + ## Job states `queued`, `running`, `paused`, `stopping`, `done`, `failed`, `pending-delete`, `deleting`, `deleted`. diff --git a/docs/media.md b/docs/media.md index 5551fb9..bbb3a6c 100644 --- a/docs/media.md +++ b/docs/media.md @@ -169,9 +169,13 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun downloads, an ffprobe probe. GIF tiles carry a translucent **GIF** label top-left; the cached preview is a static frame and the animated GIF plays while hovering. The status row reads `(Images/Videos) | Downloaded I/V | Cached I/V | Loaded I/V` — image counts include GIFs, and - each slot is the count for the active filters. A scrape strip shows live progress with Stop; it - polls every 4 seconds while the tab is open, lists only queued/running jobs (paused ones live in - History), and shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely + each slot is the count for the active filters. A scrape strip shows live progress and polls every + 4 seconds while the tab is open. It shows every queued/running job plus at most one paused scrape + (the newest not dismissed); **Stop** on a paused scrape only hides it here, so a new paused scrape + can take its place while the hidden one stays resumable. The strip text turns red and reads + `API limited, retry in Ns` while Discord throttles media traffic, and a translucent + `N scrapes completed` line appears when scrapes finish — it lives until the tab is reloaded. Each + row shows `X/Yp (~Z%)` when the search total is known. The grid scrolls infinitely (a **Load more** button appears as a fallback). Media whose source answers `403` shows a **PRIVATE** tile and `404`/`410` shows **GONE**, with the reason in the tooltip. Some channels deny the bot the search endpoint (`Missing Access`) even though history reads work, so the total is optional. @@ -183,13 +187,20 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge that disappears when the tab is opened and a Clear button. -- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar. - The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the - database overhead. +- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables. + **Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the + current settings, and a Bot/Puppet split of pending/cached/failed. One job card per chat lists its + member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending + counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header + carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers` + while a Discord cooldown holds the queue. **Settings** holds three collapsible groups — + Backfilling, Scraping, Timers — each with a Save button; values persist in `preferences.json` and + apply to live jobs immediately. The **Errors** list keeps the bot and scrape errors with the red + badge and Clear button. - Settings gains one collapsible group. Its name is **Bot** while the bot path is active, and - **Media** if the bot path is removed. It holds the limit field with the info tooltip, the usage - bar (segmented by image, GIF, video, audio, and other), and the clean button with its 5 second - countdown. + **Media** if the bot path is removed. It holds the storage limit field with the info tooltip, the + usage bar (segmented by image, GIF, video, audio, and other), and the clean button with its + 5 second countdown. Pacing and batching knobs live in the Bot tab, not here. - Mobile is out of scope for now. ## MCP diff --git a/docs/testing.md b/docs/testing.md index b3536c5..b9e8162 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -189,6 +189,8 @@ VESKSLOP_UI_URL=https://192.168.0.178:8789/ pnpm test:ui ``` It opens the Media tab, loads tiles, verifies no broken images, applies the cached-only filter, +checks the zoom slider, opens the Bot tab and asserts the backfill totals, the Bot/Puppet split and +the three settings groups render, checks that the zoom slider changes the grid density, presses Reset, and fails on any console error. A screenshot lands in `/tmp/opencode/ui-smoke.png`. diff --git a/docs/webui.md b/docs/webui.md index ed1a3d0..77ca7ea 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -142,10 +142,19 @@ box live in Chat, not Call, and the connection stats are not chips yet. **Links** and **Download** buttons, and group chats always use the puppet. **History** lists each scrape with start time, duration, pages, messages, media counts, bytes, files, and previews, and offers **Resume** for paused ones and **Delete** for anything not running. - The scrape strip shows live progress with a pulse and **Stop**. Images and GIFs render from the - cached preview when one exists, then the local file, then the CDN URL; videos and audio are link - cards with the poster and posted time in the tooltip. -- Bot shows the slopbot connection state and the scrape queue. + The scrape strip shows live progress with a pulse and **Stop**: every running/queued scrape plus at + most one paused scrape, whose **Stop** only dismisses it from the strip (it stays resumable and the + next paused scrape can appear). The line turns red with `API limited, retry in Ns` while Discord + throttles, and a translucent `N scrapes completed` note appears as scrapes finish and clears when + the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local + file, then the CDN URL; videos and audio are link cards with the poster and posted time in the + tooltip. +- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has + totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes, + per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and + **Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The + three settings groups (Backfilling, Scraping, Timers) save to `preferences.json` and retune live + jobs. The error list sits below with the Clear button. - The views are self-contained: `poc/public/media.js` and `media.css`, over the `/api/media/*` routes in the table below. See [media.md](media.md). @@ -273,7 +282,10 @@ replaceable. | GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) | | GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` | | GET | `/api/media/scrapes` | none | Every scrape with state and counters | -| GET | `/api/media/status` | none | Slopbot connection state and the scrape list | +| GET | `/api/media/status` | none | Slopbot connection state, the scrape list, and `limitedUntil` while a Discord cooldown holds media traffic | +| GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, ETA, totals with a Bot/Puppet split, the live settings, and `limitedUntil` | +| POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job | +| POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch | | GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts | | GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` | | GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over | @@ -285,6 +297,7 @@ replaceable. | GET | `/api/media/:scrapeId/:itemId/anim` | none | The cached animated GIF for hover playback | | POST | `/api/daemon/restart` | none | Replace the daemon process with a fresh copy (the client keeps running) | | POST | `/api/media/scrape/stop` | `id` | Stop a media scrape | +| POST | `/api/preferences` | `visibleGuilds?`, `tts?`, `audio?`, `favourites?`, `media?` | Also accepts `media.tunables` with `backfill`, `scrape`, and `timers` groups; changes apply to live jobs immediately | | POST | `/api/media/scrape/resume` | `id` | Continue a paused scrape from its cursor | | POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent | | GET | `/api/logs` | `limit` | The persistent action log | diff --git a/poc/public/media.css b/poc/public/media.css index 361f138..e351340 100644 --- a/poc/public/media.css +++ b/poc/public/media.css @@ -541,3 +541,202 @@ color: #fff; font: 600 9px/1 ui-monospace, Menlo, monospace; } + +/* Bot tab: backfill jobs, stats, and settings. */ + +.vk-media-bot { + display: flex; + flex-direction: column; + gap: 12px; + flex: 1; + min-height: 0; + overflow-y: auto; +} + +.vk-media-bot[hidden] { + display: none; +} + +.vk-bot-limited { + padding: 8px 10px; + border-radius: 6px; + font: 500 11px/1.4 ui-monospace, Menlo, monospace; + color: #ffb0b0; + background: color-mix(in srgb, #ff5555 14%, transparent); + border: 1px solid color-mix(in srgb, #ff5555 35%, transparent); +} + +.vk-bot-limited[hidden] { + display: none; +} + +.vk-bot-section { + border: 1px solid var(--color-divider); + border-radius: 6px; + padding: 8px 10px; + background: var(--color-surface); +} + +.vk-bot-head { + display: flex; + align-items: center; + justify-content: space-between; + gap: 8px; + margin-bottom: 6px; +} + +.vk-bot-title { + font-weight: 600; + font-size: 12px; +} + +.vk-bot-actions { + display: flex; + gap: 6px; +} + +.vk-bot-summary { + display: flex; + flex-direction: column; + gap: 2px; + margin-bottom: 8px; +} + +.vk-bot-summary-line { + font: 400 11px/1.5 ui-monospace, Menlo, monospace; + color: var(--color-neutral-300); +} + +.vk-bot-jobs { + display: flex; + flex-direction: column; + gap: 6px; +} + +.vk-bot-empty { + font: 400 11px/1.5 ui-monospace, Menlo, monospace; + color: var(--color-neutral-500); +} + +.vk-bot-job { + border: 1px solid var(--color-neutral-800); + border-radius: 5px; + padding: 6px 8px; + display: flex; + flex-direction: column; + gap: 4px; +} + +.vk-bot-job-running { + border-color: color-mix(in srgb, var(--color-accent, #4f8cff) 45%, transparent); +} + +.vk-bot-job-head { + display: flex; + align-items: center; + gap: 8px; + flex-wrap: wrap; +} + +.vk-bot-job-title { + flex: 1; + font-weight: 600; + font-size: 11px; +} + +.vk-bot-job-meta { + font: 400 11px/1.4 ui-monospace, Menlo, monospace; + color: var(--color-neutral-400); +} + +.vk-bot-state { + font: 600 10px/1.4 ui-monospace, Menlo, monospace; + text-transform: uppercase; + padding: 0 5px; + border-radius: 3px; + background: var(--color-neutral-800); + color: var(--color-neutral-300); +} + +.vk-bot-state-running { + background: color-mix(in srgb, var(--color-accent, #4f8cff) 25%, transparent); + color: var(--color-accent-300, #9fc1ff); +} + +.vk-bot-state-paused { + background: color-mix(in srgb, #ffaa33 20%, transparent); + color: #ffc880; +} + +.vk-bot-state-done { + background: color-mix(in srgb, #44cc88 20%, transparent); + color: #8fe0b8; +} + +.vk-bot-job-member { + font: 400 10px/1.5 ui-monospace, Menlo, monospace; + color: var(--color-neutral-400); + word-break: break-all; +} + +.vk-bot-progress { + height: 4px; + border-radius: 2px; + background: var(--color-neutral-800); + overflow: hidden; +} + +.vk-bot-progress-fill { + height: 100%; + background: var(--color-accent-400, #4f8cff); +} + +.vk-bot-job-actions { + display: flex; + gap: 6px; +} + +.vk-bot-settings { + display: flex; + flex-direction: column; + gap: 6px; +} + +.vk-bot-group-title { + cursor: pointer; + font: 600 11px/1.6 ui-monospace, Menlo, monospace; + color: var(--color-neutral-300); +} + +.vk-bot-group-body { + display: flex; + flex-direction: column; + gap: 4px; + padding: 6px 0 2px 12px; +} + +.vk-bot-field { + display: flex; + align-items: center; + justify-content: space-between; + gap: 10px; +} + +.vk-bot-field-label { + font: 400 11px/1.4 ui-monospace, Menlo, monospace; + color: var(--color-neutral-400); +} + +.vk-bot-save { + align-self: flex-start; + margin-top: 4px; +} + +.vk-media-limited { + color: #ff8080; +} + +.vk-media-done-note { + color: color-mix(in srgb, var(--color-neutral-400) 55%, transparent); + font-style: italic; +} diff --git a/poc/public/media.js b/poc/public/media.js index c121beb..34b4622 100644 --- a/poc/public/media.js +++ b/poc/public/media.js @@ -35,7 +35,11 @@ zoom: Number(localStorage.getItem("vk-media-zoom") ?? 50), draft: null, prefetched: null, - pageSize: 60 + pageSize: 60, + backfill: null, + preferences: null, + /** Scrapes that finished while this page was open, until the tab is reloaded. */ + completed: new Set() }; const overlay = document.createElement("div"); @@ -52,6 +56,35 @@
+ `; document.body.append(overlay); @@ -61,6 +94,12 @@ const statusLine = overlay.querySelector("#media-status"); const scrapeList = overlay.querySelector("#media-scrapes"); const grid = overlay.querySelector("#media-grid"); + const botPanel = overlay.querySelector("#media-bot"); + const botLimited = overlay.querySelector("#bot-limited"); + const backfillSummary = overlay.querySelector("#backfill-summary"); + const backfillJobs = overlay.querySelector("#backfill-jobs"); + const tunablesHost = overlay.querySelector("#bot-tunables"); + const botErrors = overlay.querySelector("#bot-errors"); const filterDrawer = el("div", "vk-media-drawer"); filterDrawer.hidden = true; @@ -177,12 +216,27 @@ } async function loadScrapes() { + const wasActive = new Set( + state.scrapes.filter(scrape => scrape.state === "running" || scrape.state === "queued").map(s => s.id) + ); + try { state.scrapes = await window.VK.api.get("/api/media/scrapes"); state.status = await window.VK.api.get("/api/media/status"); } catch (error) { setStatus(`media status failed: ${error.message}`); } + + // A scrape that was running and is now done counts once for the strip notice. + const ids = new Set(state.scrapes.map(scrape => scrape.id)); + for (const scrape of state.scrapes) { + if (wasActive.has(scrape.id) && scrape.state === "done") state.completed.add(scrape.id); + } + state.completed = new Set([...state.completed].filter(id => ids.has(id))); + + const dismissed = prefs.dismissed.filter(id => ids.has(id)); + if (dismissed.length !== prefs.dismissed.length) prefs.setDismissed(dismissed); + renderScrapes(); } @@ -327,11 +381,23 @@ function renderScrapes() { scrapeList.replaceChildren(); - // Only live jobs: a paused scrape is not running and belongs in History, not the strip. - const active = state.scrapes.filter(scrape => ["queued", "running"].includes(scrape.state)); - for (const scrape of active) { - const row = el("div", "vk-media-scrape"); + // The strip shows live jobs plus the newest paused scrape. Stop on a paused one only + // hides it here; it stays resumable, and the next paused scrape can take its place. + const active = state.scrapes.filter(scrape => ["queued", "running"].includes(scrape.state)); + const paused = state.scrapes + .filter(scrape => scrape.state === "paused" && !prefs.dismissed.includes(scrape.id)) + .sort((a, b) => (a.updatedAt < b.updatedAt ? 1 : -1)) + .slice(0, 1); + + const limitedUntil = state.status?.limitedUntil ? new Date(state.status.limitedUntil).getTime() : 0; + const limited = limitedUntil > Date.now(); + const limitedText = limited + ? ` · API limited, retry in ${Math.max(0, Math.ceil((limitedUntil - Date.now()) / 1000))}s` + : ""; + + for (const scrape of [...active, ...paused]) { + const row = el("div", `vk-media-scrape${limited ? " vk-media-limited" : ""}`); const media = scrape.media ?? {}; if (scrape.state === "running" || scrape.state === "queued") { @@ -354,12 +420,32 @@ `${channelLabel(scrape.channelId)} · ${scrape.state} · ${pageText}${percent} · ${scrape.messages}m · ` + `img ${media.image ?? 0} gif ${media.gif ?? 0} vid ${media.video ?? 0} aud ${media.audio ?? 0}` + (scrape.downloaded ? ` · files ${scrape.downloaded}` : "") + - (scrape.previewed ? ` · previews ${scrape.previewed}` : "") + (scrape.previewed ? ` · previews ${scrape.previewed}` : "") + + limitedText ) ); + if (scrape.state === "paused") { + row.append( + button("Resume", async () => { + try { + await window.VK.api.post("/api/media/scrape/resume", { id: scrape.id }); + } catch (error) { + setStatus(`resume failed: ${error.message}`); + } + void loadScrapes(); + }) + ); + } + row.append( button("Stop", async () => { + if (scrape.state === "paused") { + // Hidden here, still resumable in the Bot tab. + prefs.setDismissed([...prefs.dismissed, scrape.id]); + renderScrapes(); + return; + } await window.VK.api.post("/api/media/scrape/stop", { id: scrape.id }); void loadScrapes(); }) @@ -367,6 +453,285 @@ scrapeList.append(row); } + + // Session notice only: it clears on reload, not because the data changed. + if (state.completed.size) { + scrapeList.append( + el( + "div", + "vk-media-scrape vk-media-done-note", + `${state.completed.size} ${state.completed.size === 1 ? "scrape" : "scrapes"} completed` + ) + ); + } + } + + const prefs = { + get dismissed() { + try { + const raw = JSON.parse(localStorage.getItem("vk-media-dismissed") ?? "[]"); + return Array.isArray(raw) ? raw.map(String) : []; + } catch { + return []; + } + }, + setDismissed(ids) { + localStorage.setItem("vk-media-dismissed", JSON.stringify([...new Set(ids.map(String))])); + } + }; + + // ---- Backfill (Bot tab) ---- + + const TUNABLE_GROUPS = { + backfill: { + title: "Backfilling", + fields: [ + ["batchSize", "Batch size (rows)", 1, 500, 1], + ["batchPauseMs", "Cooldown between batches (ms)", 0, 600000, 100], + ["maxAttempts", "Tries before giving up", 1, 100, 1], + ["previewMinGapMs", "Preview gap min (ms)", 0, 60000, 10], + ["previewMaxGapMs", "Preview gap max (ms)", 0, 60000, 10], + ["maxBytes", "Max preview size (MiB)", 0.0625, 2048, 0.5, "mib"], + ["auto", "Start automatically on daemon boot", null, null, null, "bool"] + ] + }, + scrape: { + title: "Scraping", + fields: [ + ["walkCyclePages", "Preview every N walk pages", 0, 100, 1], + ["maxFileBytes", "Max file size (MiB)", 1, 8192, 1, "mib"], + ["downloadMinGapMs", "Download gap min (ms)", 0, 600000, 100], + ["downloadMaxGapMs", "Download gap max (ms)", 0, 600000, 100], + ["maxRetries", "Page fetch retries", 0, 10, 1] + ] + }, + timers: { + title: "Timers", + fields: [ + ["pageMinMs", "Walk page gap min (ms)", 0, 600000, 100], + ["pageMaxMs", "Walk page gap max (ms)", 0, 600000, 100], + ["rateLimitPauseMs", "Rate-limit cooldown (ms)", 0, 3600000, 5000] + ] + } + }; + + function formatEta(ms) { + if (ms === null || ms === undefined || !Number.isFinite(ms)) return "\u2014"; + const seconds = Math.max(0, Math.round(ms / 1000)); + if (seconds < 90) return `${seconds}s`; + const minutes = Math.round(seconds / 60); + if (minutes < 90) return `${minutes}m`; + const hours = Math.floor(minutes / 60); + const rest = minutes % 60; + return rest ? `${hours}h ${rest}m` : `${hours}h`; + } + + function pendingKinds(member) { + const by = member.pendingByKind ?? {}; + return ["video", "image", "gif", "audio"] + .filter(kind => by[kind] > 0) + .map(kind => `${by[kind]} ${kind}`) + .join(" \u00b7 "); + } + + function renderBackfill() { + const status = state.backfill; + if (!status) return; + + const until = status.limitedUntil ? new Date(status.limitedUntil).getTime() : 0; + const limited = until > Date.now(); + botLimited.hidden = !limited; + if (limited) { + botLimited.textContent = + `API timed out, increase timers \u2014 media traffic paused for ` + + formatEta(until - Date.now()); + } + + const totals = status.totals ?? {}; + const backend = totals.byBackend ?? {}; + const bot = backend.slopbot ?? { pending: 0, cached: 0, failed: 0, total: 0 }; + const puppet = backend.puppet ?? { pending: 0, cached: 0, failed: 0, total: 0 }; + + backfillSummary.replaceChildren( + el( + "div", + "vk-bot-summary-line", + `${totals.pending ?? 0} pending \u00b7 ${totals.cached ?? 0} cached \u00b7 ` + + `${totals.failed ?? 0} failed \u00b7 ${totals.downloaded ?? 0} downloaded \u00b7 ` + + `${formatBytes(totals.bytes ?? 0)} stored` + ), + el( + "div", + "vk-bot-summary-line", + `ETA ${formatEta(totals.etaMs)} at ${status.estimatedPerMinute ?? 0} rows/min ` + + `(current settings)` + ), + el( + "div", + "vk-bot-summary-line vk-dim", + `Bot: ${bot.pending} pending, ${bot.cached} cached, ${bot.failed} failed \u00b7 ` + + `Puppet: ${puppet.pending} pending, ${puppet.cached} cached, ${puppet.failed} failed` + ) + ); + + backfillJobs.replaceChildren(); + const jobs = (status.jobs ?? []).filter(job => job.pending > 0 || job.state !== "idle"); + if (!jobs.length) { + backfillJobs.append(el("div", "vk-bot-empty", "Nothing waiting to backfill.")); + return; + } + + for (const job of jobs) { + const row = el("div", `vk-bot-job vk-bot-job-${job.state}`); + const head = el("div", "vk-bot-job-head"); + head.append(el("span", "vk-bot-job-title", channelLabel(job.chatId))); + head.append(el("span", `vk-bot-state vk-bot-state-${job.state}`, job.state)); + + const parts = [`${job.pending} pending`]; + if (job.cached) parts.push(`${job.cached} cached`); + if (job.failed) parts.push(`${job.failed} failed`); + if (job.pending > 0) parts.push(`ETA ${formatEta(job.etaMs)}`); + head.append(el("span", "vk-bot-job-meta", parts.join(" \u00b7 "))); + row.append(head); + + for (const member of job.members) { + if (member.pending === 0 && member.state !== "running") continue; + const kinds = pendingKinds(member); + row.append( + el( + "div", + "vk-bot-job-member", + `${member.scrapeId} \u00b7 ${member.state} \u00b7 ${member.mode} \u00b7 ` + + `${member.backend} \u00b7 ${member.pending} pending` + + (kinds ? ` (${kinds})` : "") + ) + ); + } + + const progress = el("div", "vk-bot-progress"); + const fill = el("div", "vk-bot-progress-fill"); + fill.style.width = `${Math.min(100, Math.round((job.cached / Math.max(1, job.total)) * 100))}%`; + progress.append(fill); + row.append(progress); + + const actions = el("div", "vk-bot-job-actions"); + if (job.state === "running") { + actions.append(button("Pause", () => void backfillAction("pause", job.chatId))); + } else if (job.pending > 0) { + actions.append(button("Start", () => void backfillAction("start", job.chatId))); + } + row.append(actions); + + backfillJobs.append(row); + } + } + + async function backfillAction(action, chatId) { + try { + state.backfill = await window.VK.api.post( + `/api/media/backfill/${action}`, + chatId ? { chatId } : {} + ); + renderBackfill(); + } catch (error) { + setStatus(`backfill ${action} failed: ${error.message}`); + } + } + + async function refreshBackfill() { + try { + state.backfill = await window.VK.api.get("/api/media/backfill"); + } catch { + state.backfill = null; + } + renderBackfill(); + } + + function tunableValue(raw, key, scale) { + const value = raw?.[key]; + if (scale === "bool") return value !== false; + const number = Number(value); + if (!Number.isFinite(number)) return null; + return scale === "mib" ? Math.round((number / 1024 ** 2) * 100) / 100 : number; + } + + function renderTunables() { + const tunables = state.preferences?.media?.tunables; + if (!tunables) return; + + tunablesHost.replaceChildren(); + + for (const [group, spec] of Object.entries(TUNABLE_GROUPS)) { + const details = el("details", "vk-bot-group"); + details.open = group === "backfill"; + details.append(el("summary", "vk-bot-group-title", spec.title)); + + const body = el("div", "vk-bot-group-body"); + for (const [key, label, min, max, step, scale] of spec.fields) { + const row = el("label", "vk-bot-field"); + row.append(el("span", "vk-bot-field-label", label)); + + const input = document.createElement("input"); + input.dataset.group = group; + input.dataset.key = key; + input.dataset.scale = scale ?? ""; + + if (scale === "bool") { + input.type = "checkbox"; + input.checked = tunables[group]?.[key] !== false; + } else { + input.type = "number"; + input.className = "vk-media-input"; + input.min = String(min); + input.max = String(max); + input.step = String(step); + input.value = String(tunableValue(tunables[group] ?? {}, key, scale) ?? ""); + } + + row.append(input); + body.append(row); + } + + const save = button("Save", () => void saveTunables(group)); + save.classList.add("vk-bot-save"); + body.append(save); + details.append(body); + tunablesHost.append(details); + } + } + + async function saveTunables(group) { + const patch = {}; + + for (const input of tunablesHost.querySelectorAll(`input[data-group="${group}"]`)) { + const scale = input.dataset.scale; + if (scale === "bool") { + patch[input.dataset.key] = input.checked; + continue; + } + + const value = Number(input.value); + if (!Number.isFinite(value)) continue; + patch[input.dataset.key] = scale === "mib" ? Math.round(value * 1024 ** 2) : Math.round(value); + } + + try { + await window.VK.api.post("/api/preferences", { media: { tunables: { [group]: patch } } }); + setStatus(`${TUNABLE_GROUPS[group].title.toLowerCase()} settings saved`); + await refreshPreferences(); + void refreshBackfill(); + } catch (error) { + setStatus(`save failed: ${error.message}`); + } + } + + async function refreshPreferences() { + try { + state.preferences = await window.VK.api.get("/api/preferences"); + } catch { + return; + } + renderTunables(); } // ---- Bot errors ---- @@ -387,7 +752,7 @@ } function renderErrors() { - scrapeList.replaceChildren(); + botErrors.replaceChildren(); const head = el("div", "vk-media-scrape"); head.append(el("span", null, `${(state.errors ?? []).length} error(s)`)); @@ -408,7 +773,7 @@ scrapeList.append(head); for (const entry of state.errors ?? []) { - scrapeList.append( + botErrors.append( el( "div", "vk-media-scrape vk-media-error", @@ -417,7 +782,7 @@ ); } - if (!(state.errors ?? []).length) scrapeList.append(el("div", "vk-media-scrape", "No errors.")); + if (!(state.errors ?? []).length) botErrors.append(el("div", "vk-media-scrape", "No errors.")); } async function refreshErrors() { @@ -972,6 +1337,8 @@ title.textContent = state.view === "bot" ? "Slopbot" : "Media archive"; toolbar.hidden = false; grid.hidden = state.view === "bot"; + botPanel.hidden = state.view !== "bot"; + scrapeList.hidden = state.view === "bot"; renderToolbar(); if (state.view === "bot") { @@ -979,6 +1346,8 @@ statusLine.textContent = bot?.connected ? `connected as ${bot.user} in ${bot.guilds.length} guild(s)` : "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)"; + if (!state.preferences) void refreshPreferences(); + void refreshBackfill(); void loadScrapes().then(renderScrapes); void refreshErrors(); } else { @@ -993,10 +1362,15 @@ overlay.querySelector("#media-close").addEventListener("click", hide); overlay.querySelector("#media-refresh").addEventListener("click", () => { + state.completed.clear(); void loadScrapes(); if (state.view === "media") void loadMedia(true); + if (state.view === "bot") void refreshBackfill(); }); + overlay.querySelector("#backfill-start").addEventListener("click", () => void backfillAction("start")); + overlay.querySelector("#backfill-pause").addEventListener("click", () => void backfillAction("pause")); + const scopeMedia = document.querySelector("#scope-media"); const scopeBot = document.querySelector("#scope-bot"); scopeMedia?.addEventListener("click", () => show("media")); @@ -1019,6 +1393,11 @@ if (state.view === "media") void loadScrapes(); }, 4000); + // Backfill progress while the Bot tab is open. + setInterval(() => { + if (state.view === "bot") void refreshBackfill(); + }, 3000); + // Infinite scroll: fetch the next page before the grid bottom is reached. grid.addEventListener("scroll", () => { if (state.loading || !state.loaded) return; diff --git a/poc/src/control-bridge.ts b/poc/src/control-bridge.ts index b3ac1f3..84fabaf 100644 --- a/poc/src/control-bridge.ts +++ b/poc/src/control-bridge.ts @@ -42,13 +42,6 @@ let puppet: Puppet | undefined; let coreConnected = false; let connecting = false; -const media = new MediaService({ - getCore: () => core, - isCoreConnected: () => coreConnected, - getMediaPrefs: () => preferences.get().media, - log: line => console.log(`[media] ${line}`) -}); - async function connectLoop(): Promise { if (connecting) return; connecting = true; @@ -185,6 +178,14 @@ const preferences = new PreferencesStore( ); const stt = new Stt(); +// After `preferences`: the media engine and backfill queue read their tunables at construction. +const media = new MediaService({ + getCore: () => core, + isCoreConnected: () => coreConnected, + getMediaPrefs: () => preferences.get().media, + log: line => console.log(`[media] ${line}`) +}); + const audio = new AudioService({ cacheDir: process.env.VESKSLOP_CACHE_DIR, transcripts, diff --git a/poc/src/media.ts b/poc/src/media.ts index 7ad1861..dc3b31e 100644 --- a/poc/src/media.ts +++ b/poc/src/media.ts @@ -8,6 +8,7 @@ import { rmSync } from "node:fs"; import { isAbsolute, join } from "node:path"; import { + BackfillQueue, betterCandidate, dirSizeBytes, isNearLimit, @@ -19,6 +20,8 @@ import { resolveDbRoot, type MediaKind, type MediaRow, + type BackfillSettings, + type BackfillStatus, type ScrapeBackend, type ScrapeMeta } from "../../media-scrape/src/index.js"; @@ -153,11 +156,14 @@ const FILE_CONTENT_TYPES: Record = { export interface MediaStatus { slopbot: SlopbotStatus; scrapes: ScrapeMeta[]; + /** ISO time while a Discord rate limit holds media traffic. */ + limitedUntil: string | null; } export class MediaService { private readonly rootDir = resolveDbRoot(); private readonly engine: ScrapeEngine; + private readonly backfill: BackfillQueue; private bot: BotClient | null = null; private botFetcher: SlopbotFetcher | null = null; @@ -167,50 +173,102 @@ export class MediaService { private readonly errors: MediaErrorEntry[] = []; constructor(private readonly deps: MediaServiceDeps) { - const number = (value: string | undefined, fallback: number) => { - const parsed = Number(value); - return Number.isFinite(parsed) && parsed >= 0 ? parsed : fallback; - }; + const tunables = this.deps.getMediaPrefs().tunables; this.engine = new ScrapeEngine({ rootDir: this.rootDir, log: line => this.deps.log?.(line), onError: (scrapeId, message) => this.recordError("scrape", `${scrapeId}: ${message}`), - walkCyclePages: number(process.env.VESKSLOP_MEDIA_WALK_CYCLE_PAGES, 5), - rateLimitPauseMs: number(process.env.VESKSLOP_MEDIA_RATELIMIT_PAUSE_MS, 60_000), - previewMaxAttempts: number(process.env.VESKSLOP_MEDIA_PREVIEW_MAX_ATTEMPTS, 3), + walkCyclePages: tunables.scrape.walkCyclePages, + rateLimitPauseMs: tunables.timers.rateLimitPauseMs, + pageMinMs: tunables.timers.pageMinMs, + pageMaxMs: tunables.timers.pageMaxMs, + maxRetries: tunables.scrape.maxRetries, + previewMaxAttempts: tunables.backfill.maxAttempts, download: { - maxFileBytes: number(process.env.VESKSLOP_MEDIA_MAX_FILE_BYTES, 50 * 1024 * 1024), - previewMaxBytes: number(process.env.VESKSLOP_MEDIA_PREVIEW_MAX_BYTES, 5 * 1024 * 1024), - previewMinGapMs: number(process.env.VESKSLOP_MEDIA_PREVIEW_MIN_MS, 200), - previewMaxGapMs: number(process.env.VESKSLOP_MEDIA_PREVIEW_MAX_MS, 600), - previewBatchSize: number(process.env.VESKSLOP_MEDIA_PREVIEW_BATCH, 25), - previewBatchPauseMs: number(process.env.VESKSLOP_MEDIA_PREVIEW_BATCH_PAUSE_MS, 60_000), - minGapMs: number(process.env.VESKSLOP_MEDIA_DOWNLOAD_MIN_MS, 1000), - maxGapMs: number(process.env.VESKSLOP_MEDIA_DOWNLOAD_MAX_MS, 3000) + maxFileBytes: tunables.scrape.maxFileBytes, + previewMaxBytes: tunables.backfill.maxBytes, + previewMinGapMs: tunables.backfill.previewMinGapMs, + previewMaxGapMs: tunables.backfill.previewMaxGapMs, + previewBatchSize: tunables.backfill.batchSize, + previewBatchPauseMs: tunables.backfill.batchPauseMs, + minGapMs: tunables.scrape.downloadMinGapMs, + maxGapMs: tunables.scrape.downloadMaxGapMs } }); + this.backfill = new BackfillQueue(this.engine, { + rootDir: this.rootDir, + getSettings: () => this.backfillSettings(), + getGaps: () => { + const live = this.deps.getMediaPrefs().tunables; + return { minGapMs: live.backfill.previewMinGapMs, maxGapMs: live.backfill.previewMaxGapMs }; + }, + log: line => this.deps.log?.(line) + }); + const stale = this.engine.reconcileStaleJobs(); if (stale.length) { this.deps.log?.(`paused ${stale.length} stale scrape(s) left running by a previous daemon`); } - // Preview passes only run when a walk ends; fill in scrapes that paused early. - void this.backfillPreviews(); + // Preview passes only run when a walk ends; the queue fills in scrapes that paused early. + this.backfill.autoStart(); } - private async backfillPreviews(): Promise { - for (const meta of this.engine.list()) { - if (meta.state === "running" || meta.state === "queued") continue; + private backfillSettings(): BackfillSettings { + const tunables = this.deps.getMediaPrefs().tunables; + return { + batchSize: tunables.backfill.batchSize, + batchPauseMs: tunables.backfill.batchPauseMs, + auto: tunables.backfill.auto + }; + } - try { - const added = await this.engine.backfillPreviews(meta.id); - if (added) this.deps.log?.(`backfilled ${added} previews for ${meta.id}`); - } catch (error) { - this.deps.log?.(`preview backfill failed for ${meta.id}: ${String(error)}`); + /** Push preference changes into the live engine and backfill queue. */ + applyTunables(): void { + const tunables = this.deps.getMediaPrefs().tunables; + this.engine.updateOptions({ + walkCyclePages: tunables.scrape.walkCyclePages, + rateLimitPauseMs: tunables.timers.rateLimitPauseMs, + pageMinMs: tunables.timers.pageMinMs, + pageMaxMs: tunables.timers.pageMaxMs, + maxRetries: tunables.scrape.maxRetries, + previewMaxAttempts: tunables.backfill.maxAttempts, + download: { + maxFileBytes: tunables.scrape.maxFileBytes, + previewMaxBytes: tunables.backfill.maxBytes, + previewMinGapMs: tunables.backfill.previewMinGapMs, + previewMaxGapMs: tunables.backfill.previewMaxGapMs, + previewBatchSize: tunables.backfill.batchSize, + previewBatchPauseMs: tunables.backfill.batchPauseMs, + minGapMs: tunables.scrape.downloadMinGapMs, + maxGapMs: tunables.scrape.downloadMaxGapMs } - } + }); + } + + /** Backfill jobs, totals, and pacing, plus the engine's live cooldown. */ + backfillStatus(): BackfillStatus { + const status = this.backfill.list(); + const limit = this.engine.rateLimitStatus(); + + return { + ...status, + limitedUntil: limit.active + ? new Date(limit.until ?? Date.now() + limit.remainingMs).toISOString() + : status.limitedUntil + }; + } + + startBackfill(chatId?: string): BackfillStatus { + this.backfill.start(chatId); + return this.backfillStatus(); + } + + pauseBackfill(chatId?: string): BackfillStatus { + this.backfill.pause(chatId); + return this.backfillStatus(); } /** Log slopbot in on first use. False when it is disabled or the login failed. */ @@ -425,9 +483,13 @@ export class MediaService { } status(): MediaStatus { + const limit = this.engine.rateLimitStatus(); return { slopbot: slopbotStatus(this.bot), - scrapes: this.engine.list() + scrapes: this.engine.list(), + limitedUntil: limit.active + ? new Date(limit.until ?? Date.now() + limit.remainingMs).toISOString() + : null }; } diff --git a/poc/src/preferences.ts b/poc/src/preferences.ts index ac2d28b..83369e2 100644 --- a/poc/src/preferences.ts +++ b/poc/src/preferences.ts @@ -76,6 +76,43 @@ export interface AudioPreferences { duck: boolean; } +/** Pacing and limit knobs for preview backfilling, scraping, and cooldowns. */ +export interface MediaTunables { + backfill: { + /** Preview rows per batch. */ + batchSize: number; + /** Cooldown between batches, ms. */ + batchPauseMs: number; + /** Transient failures allowed before a row is closed as unreachable. */ + maxAttempts: number; + /** Gap between preview fetches, ms. */ + previewMinGapMs: number; + previewMaxGapMs: number; + /** Largest preview to cache, bytes. */ + maxBytes: number; + /** Start backfill jobs automatically on daemon boot. */ + auto: boolean; + }; + scrape: { + /** Walk pages between inline preview cycles. 0 defers previews to the end. */ + walkCyclePages: number; + /** Largest file to download, bytes. */ + maxFileBytes: number; + /** Gap between file downloads, ms. */ + downloadMinGapMs: number; + downloadMaxGapMs: number; + /** Page fetch retries before a scrape fails. */ + maxRetries: number; + }; + timers: { + /** Gap between walk pages, ms. */ + pageMinMs: number; + pageMaxMs: number; + /** Cooldown after a Discord rate limit, ms. */ + rateLimitPauseMs: number; + }; +} + export interface MediaPreferences { /** Combined size limit for the media DB root: databases, downloads, previews. */ limitBytes: number; @@ -83,6 +120,7 @@ export interface MediaPreferences { warnPercent: number; /** No-delete overrun allowance in percent, 0-100. */ overrunPercent: number; + tunables: MediaTunables; } export interface WebPreferences { @@ -115,13 +153,89 @@ const DEFAULTS: WebPreferences = { media: { limitBytes: 30 * 1024 * 1024 * 1024, warnPercent: 90, - overrunPercent: 5 + overrunPercent: 5, + tunables: { + backfill: { + batchSize: 25, + batchPauseMs: 60_000, + maxAttempts: 3, + previewMinGapMs: 200, + previewMaxGapMs: 600, + maxBytes: 5 * 1024 * 1024, + auto: true + }, + scrape: { + walkCyclePages: 5, + maxFileBytes: 50 * 1024 * 1024, + downloadMinGapMs: 1000, + downloadMaxGapMs: 3000, + maxRetries: 2 + }, + timers: { + pageMinMs: 1200, + pageMaxMs: 3500, + rateLimitPauseMs: 60_000 + } + } } }; const MEDIA_LIMIT_MIN = 256 * 1024 * 1024; const MEDIA_LIMIT_MAX = 10 * 1024 * 1024 * 1024 * 1024; +function intIn(value: unknown, min: number, max: number, fallback: number): number { + const number = Number(value); + if (!Number.isFinite(number)) return fallback; + return Math.min(max, Math.max(min, Math.round(number))); +} + +function normalizeTunables(raw: any, fallback: MediaTunables): MediaTunables { + const backfill = raw?.backfill ?? {}; + const scrape = raw?.scrape ?? {}; + const timers = raw?.timers ?? {}; + + const previewMinGapMs = intIn(backfill.previewMinGapMs, 0, 60_000, fallback.backfill.previewMinGapMs); + const previewMaxGapMs = intIn( + backfill.previewMaxGapMs, + previewMinGapMs, + 60_000, + fallback.backfill.previewMaxGapMs + ); + const downloadMinGapMs = intIn(scrape.downloadMinGapMs, 0, 600_000, fallback.scrape.downloadMinGapMs); + const downloadMaxGapMs = intIn( + scrape.downloadMaxGapMs, + downloadMinGapMs, + 600_000, + fallback.scrape.downloadMaxGapMs + ); + const pageMinMs = intIn(timers.pageMinMs, 0, 600_000, fallback.timers.pageMinMs); + const pageMaxMs = intIn(timers.pageMaxMs, pageMinMs, 600_000, fallback.timers.pageMaxMs); + + return { + backfill: { + batchSize: intIn(backfill.batchSize, 1, 500, fallback.backfill.batchSize), + batchPauseMs: intIn(backfill.batchPauseMs, 0, 3_600_000, fallback.backfill.batchPauseMs), + maxAttempts: intIn(backfill.maxAttempts, 1, 100, fallback.backfill.maxAttempts), + previewMinGapMs, + previewMaxGapMs, + maxBytes: intIn(backfill.maxBytes, 65_536, 2 * 1024 ** 3, fallback.backfill.maxBytes), + auto: typeof backfill.auto === "boolean" ? backfill.auto : fallback.backfill.auto + }, + scrape: { + walkCyclePages: intIn(scrape.walkCyclePages, 0, 100, fallback.scrape.walkCyclePages), + maxFileBytes: intIn(scrape.maxFileBytes, 1024 ** 2, 8 * 1024 ** 3, fallback.scrape.maxFileBytes), + downloadMinGapMs, + downloadMaxGapMs, + maxRetries: intIn(scrape.maxRetries, 0, 10, fallback.scrape.maxRetries) + }, + timers: { + pageMinMs, + pageMaxMs, + rateLimitPauseMs: intIn(timers.rateLimitPauseMs, 0, 3_600_000, fallback.timers.rateLimitPauseMs) + } + }; +} + function normalizeMedia(raw: any, fallback: MediaPreferences): MediaPreferences { const limit = Number(raw?.limitBytes); const warn = Number(raw?.warnPercent); @@ -136,7 +250,8 @@ function normalizeMedia(raw: any, fallback: MediaPreferences): MediaPreferences overrunPercent: Number.isFinite(overrun) && overrun >= 0 && overrun <= 100 ? Math.round(overrun) - : fallback.overrunPercent + : fallback.overrunPercent, + tunables: normalizeTunables(raw?.tunables, fallback.tunables) }; } diff --git a/poc/src/webui.ts b/poc/src/webui.ts index 806e2b2..a30fc32 100644 --- a/poc/src/webui.ts +++ b/poc/src/webui.ts @@ -478,7 +478,8 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { : current.warnPercent, overrunPercent: Number.isFinite(Number(mediaPrefs?.overrunPercent)) ? Number(mediaPrefs.overrunPercent) - : current.overrunPercent + : current.overrunPercent, + tunables: mediaPrefs?.tunables ?? current.tunables }; } @@ -487,7 +488,9 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { deps.audio.setDuckEnabled(patch.audio.duck); } - res.json({ ok: true, result: deps.preferences.update(patch) }); + const updated = deps.preferences.update(patch); + if (mediaPrefs !== undefined) deps.media.applyTunables(); + res.json({ ok: true, result: updated }); }) ); @@ -773,6 +776,29 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { }) ); + app.get( + "/api/media/backfill", + wrap(async (_req, res) => { + res.json({ ok: true, result: deps.media.backfillStatus() }); + }) + ); + + app.post( + "/api/media/backfill/start", + wrap(async (req, res) => { + const chatId = req.body?.chatId ? String(req.body.chatId) : undefined; + res.json({ ok: true, result: deps.media.startBackfill(chatId) }); + }) + ); + + app.post( + "/api/media/backfill/pause", + wrap(async (req, res) => { + const chatId = req.body?.chatId ? String(req.body.chatId) : undefined; + res.json({ ok: true, result: deps.media.pauseBackfill(chatId) }); + }) + ); + app.get( "/api/media/:scrapeId/:itemId/thumb", wrap(async (req, res) => { diff --git a/poc/test/media-ui-smoke.mjs b/poc/test/media-ui-smoke.mjs index 60578f7..dd7357c 100644 --- a/poc/test/media-ui-smoke.mjs +++ b/poc/test/media-ui-smoke.mjs @@ -126,6 +126,29 @@ try { step("prompt after reset", resetPrompt); check("reset returns the fresh view", resetPrompt); + // Bot tab: backfill progress, stats, and the three settings groups. + await page.click("#scope-bot"); + await page.waitForSelector("#media-bot:not([hidden])", { timeout: 10000 }); + await page.waitForFunction( + () => document.querySelector("#backfill-summary")?.textContent?.includes("pending"), + { timeout: 20000 } + ); + + const summaryLines = await page.locator(".vk-bot-summary-line").count(); + step("backfill summary lines", summaryLines); + check("backfill summary renders", summaryLines >= 3, summaryLines); + step("backfill totals", (await page.locator("#backfill-summary").textContent())?.trim().slice(0, 160)); + step("backend split", (await page.locator(".vk-bot-summary-line.vk-dim").first().textContent())?.trim()); + + const groups = await page.locator("#bot-tunables details").count(); + step("tunable groups", groups); + check("three tunable groups", groups === 3, groups); + + const fields = await page.locator("#bot-tunables input").count(); + step("tunable fields", fields); + check("tunable fields render", fields >= 12, fields); + check("start button visible", await page.locator("#backfill-start").isVisible()); + step("console errors", results.consoleErrors.length); check("no console errors", results.consoleErrors.length === 0, results.consoleErrors.slice(0, 3));