zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Backfill YouTube titles through the queue

Browse Source
This commit is contained in:
Fisto
2026-09-17 13:19:22 +02:00
parent 557ba5ca7e
commit 774183b6fa
11 changed files with 295 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+7 -2
View File
@@ -88,7 +88,7 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `title_error`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
@@ -163,7 +163,12 @@ Previews run after every walk:
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
before their previews were cached. One job per chat groups every paused, done, or failed scrape
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
docs/media.md
+2 -1
View File
@@ -200,7 +200,8 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
that disappears when the tab is opened and a Clear button.
- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
**Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
current settings, and a Bot/Puppet split of pending/cached/failed. One job card per chat lists its
current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing
YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its
member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`