Run the title catch-up in its own YouTube lane

This commit is contained in:
Fisto
2026-09-17 14:25:08 +02:00
parent 8befc05591
commit 9481286169
16 changed files with 495 additions and 82 deletions
+9 -6
View File
@@ -168,12 +168,15 @@ Previews run after every walk:
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
before their previews were cached. It holds preview work only.
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
+4 -2
View File
@@ -200,8 +200,10 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
that disappears when the tab is opened and a Clear button.
- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
**Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing
YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its
current settings, and a Bot/Puppet split of pending/cached/failed. A separate **Titles
(YouTube)** card tracks the one-time title catch-up: pending/resolved/failed, throughput, ETA,
and Start/Pause. It talks to YouTube only (hard-coded 100 per batch, 10 s between batches) and
runs beside the preview queue. One job card per chat lists its
member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
+5 -1
View File
@@ -165,7 +165,8 @@ box live in Chat, not Call, and the connection stats are not chips yet.
the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
tooltip.
- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
@@ -302,6 +303,9 @@ replaceable.
| GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, `etaStartMs`/`etaRunMs`/`etaTotalMs`, totals with a Bot/Puppet split, the live settings, and `limitedUntil` |
| POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job |
| POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch |
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
| POST | `/api/media/titles/start` | none | Start the title lane |
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |