Run the title catch-up in its own YouTube lane
This commit is contained in:
@@ -168,12 +168,15 @@ Previews run after every walk:
|
||||
- The job stays `running` through the download and preview passes, so a client never sees `done`
|
||||
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
|
||||
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
|
||||
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
|
||||
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
|
||||
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
|
||||
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
|
||||
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
|
||||
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
|
||||
before their previews were cached. It holds preview work only.
|
||||
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
|
||||
indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
|
||||
job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
|
||||
media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
|
||||
beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
|
||||
youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
|
||||
stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
|
||||
throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
|
||||
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
|
||||
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
|
||||
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
|
||||
|
||||
+4
-2
@@ -200,8 +200,10 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
|
||||
that disappears when the tab is opened and a Clear button.
|
||||
- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
|
||||
**Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
|
||||
current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing
|
||||
YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its
|
||||
current settings, and a Bot/Puppet split of pending/cached/failed. A separate **Titles
|
||||
(YouTube)** card tracks the one-time title catch-up: pending/resolved/failed, throughput, ETA,
|
||||
and Start/Pause. It talks to YouTube only (hard-coded 100 per batch, 10 s between batches) and
|
||||
runs beside the preview queue. One job card per chat lists its
|
||||
member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
|
||||
counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
|
||||
carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
|
||||
|
||||
+5
-1
@@ -165,7 +165,8 @@ box live in Chat, not Call, and the connection stats are not chips yet.
|
||||
the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local
|
||||
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
|
||||
tooltip.
|
||||
- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has
|
||||
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
|
||||
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
|
||||
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
|
||||
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
|
||||
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
|
||||
@@ -302,6 +303,9 @@ replaceable.
|
||||
| GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, `etaStartMs`/`etaRunMs`/`etaTotalMs`, totals with a Bot/Puppet split, the live settings, and `limitedUntil` |
|
||||
| POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job |
|
||||
| POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch |
|
||||
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
|
||||
| POST | `/api/media/titles/start` | none | Start the title lane |
|
||||
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
|
||||
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
||||
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
|
||||
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
||||
|
||||
Reference in New Issue
Block a user