Run the title catch-up in its own YouTube lane

This commit is contained in:
Fisto
2026-09-17 14:25:08 +02:00
parent 8befc05591
commit 9481286169
16 changed files with 495 additions and 82 deletions
+9 -6
View File
@@ -168,12 +168,15 @@ Previews run after every walk:
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
before their previews were cached. It holds preview work only.
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the