Run the title catch-up in its own YouTube lane
This commit is contained in:
@@ -168,12 +168,15 @@ Previews run after every walk:
|
||||
- The job stays `running` through the download and preview passes, so a client never sees `done`
|
||||
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
|
||||
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
|
||||
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
|
||||
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
|
||||
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
|
||||
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
|
||||
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
|
||||
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
|
||||
before their previews were cached. It holds preview work only.
|
||||
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
|
||||
indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
|
||||
job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
|
||||
media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
|
||||
beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
|
||||
youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
|
||||
stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
|
||||
throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
|
||||
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
|
||||
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
|
||||
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
|
||||
|
||||
Reference in New Issue
Block a user