diff --git a/CURRENT.md b/CURRENT.md index 39d9aeb..9aa7920 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -40,6 +40,14 @@ mobile draft at 390px. - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: `previewed: 2`; thumbs served over `/api/media///thumb` (200). + - [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated + `TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause + (hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no + media-traffic lock because it only calls YouTube's oEmbed. New `titleBatch` on the engine, + per-scrape `titlePending/titled/titleFailed` counters, `/api/media/titles` + start/pause + routes, and a Bot-tab card with progress, throughput, ETA, and Start/Pause. The lane + auto-starts on boot; with nothing pending it exits immediately. Preview backfill goes back to + previews only. Unit tests for `titleBatch` and a lane catch-up run; smoke covers the card. - [x] C52 Wildcard title search. `*` now maps to a LIKE wildcard in both title modes: a leading `*` drops the word anchor (`*car` finds *scar* and *discard*), in-token stars match between letters (`c*r*`), and a trailing star matches the plain prefix. `%` and `_` are escaped with an diff --git a/TODO.md b/TODO.md index 434cbaa..2dd7996 100644 --- a/TODO.md +++ b/TODO.md @@ -64,13 +64,6 @@ Ordered backlog. `CURRENT.md` tracks the live session state. ## Media archive follow-ups -2. **Title work in its own lane.** Title resolution talks straight to `youtube.com/oembed` - (daemon fetch; no Discord API, no bot account), but it currently shares the media-traffic lock - and the preview batch pacing, so it serializes behind walks and previews. Move it to its own - lane: skip the media-traffic lock, give it its own pacing/concurrency settings, and handle - YouTube `429`s as a cooldown rather than a normal failed try (they must not consume the tries - budget). Show the lane's throughput separately in the Bot tab. - ## Web UI follow-ups Ordered after the voice backend unless the owner asks first. diff --git a/docs/media-scraper.md b/docs/media-scraper.md index 1f35dbd..b5e6d7e 100644 --- a/docs/media-scraper.md +++ b/docs/media-scraper.md @@ -168,12 +168,15 @@ Previews run after every walk: - The job stays `running` through the download and preview passes, so a client never sees `done` on a half-archived scrape. A paused walk still gets these passes for the pages it collected. - The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished - before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows - (`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved - through the provider oEmbed and stored as `title` + `title_key`; failures count toward - `title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted - videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while - both kinds of work remain. One job per chat groups every paused, done, or failed scrape + before their previews were cached. It holds preview work only. +- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows + indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time + job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no + media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs + beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches + youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos + stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts, + throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the diff --git a/docs/media.md b/docs/media.md index ac30f14..5510cfe 100644 --- a/docs/media.md +++ b/docs/media.md @@ -200,8 +200,10 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun that disappears when the tab is opened and a Clear button. - **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables. **Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the - current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing - YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its + current settings, and a Bot/Puppet split of pending/cached/failed. A separate **Titles + (YouTube)** card tracks the one-time title catch-up: pending/resolved/failed, throughput, ETA, + and Start/Pause. It talks to YouTube only (hard-coded 100 per batch, 10 s between batches) and + runs beside the preview queue. One job card per chat lists its member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers` diff --git a/docs/webui.md b/docs/webui.md index f282a29..1ffc372 100644 --- a/docs/webui.md +++ b/docs/webui.md @@ -165,7 +165,8 @@ box live in Chat, not Call, and the connection stats are not chips yet. the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local file, then the CDN URL; videos and audio are link cards with the poster and posted time in the tooltip. -- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has +- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)** + catch-up lane with its own Start/Pause, and the media tunables. Backfill has totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes, per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The @@ -302,6 +303,9 @@ replaceable. | GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, `etaStartMs`/`etaRunMs`/`etaTotalMs`, totals with a Bot/Puppet split, the live settings, and `limitedUntil` | | POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job | | POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch | +| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape | +| POST | `/api/media/titles/start` | none | Start the title lane | +| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch | | GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts | | GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` | | GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over | diff --git a/media-scrape/src/backfill.ts b/media-scrape/src/backfill.ts index e03333c..d9b5f4e 100644 --- a/media-scrape/src/backfill.ts +++ b/media-scrape/src/backfill.ts @@ -29,8 +29,6 @@ export interface BackfillMember { backend: BackendName; mode: ScrapeMode; pending: number; - /** YouTube-like rows still missing a title. */ - pendingTitles: number; cached: number; failed: number; total: number; @@ -154,7 +152,6 @@ export class BackfillQueue { backend: meta.backend, mode: meta.mode, pending: 0, - pendingTitles: 0, cached: 0, failed: 0, total: 0, @@ -168,7 +165,6 @@ export class BackfillQueue { try { const counts = store.previewCounts(); member.pending = counts.pending; - member.pendingTitles = counts.titlePending; member.cached = counts.cached; member.failed = counts.failed; member.total = counts.total; @@ -209,10 +205,7 @@ export class BackfillQueue { for (const [chatId, group] of groups) { // Hide chats that are fully backfilled unless their job did work this session. const tracked = this.jobs.get(chatId); - const pending = group.members.reduce( - (sum, member) => sum + member.pending + member.pendingTitles, - 0 - ); + const pending = group.members.reduce((sum, member) => sum + member.pending, 0); if (!tracked && pending === 0) continue; const perMinute = tracked?.perMinute ?? this.estimatedPerMinute(settings, gaps); @@ -418,9 +411,7 @@ export class BackfillQueue { while (!job.stop) { const group = this.plan(false).get(job.chatId); - const member = group?.members.find( - candidate => candidate.pending > 0 || candidate.pendingTitles > 0 - ); + const member = group?.members.find(candidate => candidate.pending > 0); if (!member) break; const limit = this.engine.rateLimitStatus(); diff --git a/media-scrape/src/engine.ts b/media-scrape/src/engine.ts index 6f311cd..99e05a8 100644 --- a/media-scrape/src/engine.ts +++ b/media-scrape/src/engine.ts @@ -46,14 +46,21 @@ export interface ScrapeBackend { fetchPage(channelId: string, before: string | null, limit: number): Promise; } +/** One batch of title resolution, as returned by `titleBatch`. */ +export interface TitleBatchResult { + titled: number; + /** Rows closed out because the provider never returned a title. */ + failed: number; + remaining: number; + stopped: boolean; +} + /** One batch of preview work, as returned by `previewBatch`. */ export interface PreviewBatchResult { cached: number; skipped: number; failed: number; ratelimited: number; - /** Titles resolved in this batch. */ - titled: number; /** Rows still pending after the batch. */ remaining: number; /** True when a stop request interrupted the batch. */ @@ -550,7 +557,7 @@ export class ScrapeEngine { const dir = join(this.options.rootDir, scrapeId); const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir); if (!meta) { - return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, titled: 0, remaining: 0, stopped: false }; + return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, remaining: 0, stopped: false }; } const releaseTraffic = await this.acquireTraffic(); @@ -563,6 +570,56 @@ export class ScrapeEngine { } } + /** + * Resolve YouTube titles for up to `limit` rows. No media-traffic lock: these requests go to + * YouTube's oEmbed, never Discord, so the lane can run beside a walk. The caller paces. + */ + async titleBatch(scrapeId: string, limit: number): Promise { + const dir = join(this.options.rootDir, scrapeId); + const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir); + const result: TitleBatchResult = { titled: 0, failed: 0, remaining: 0, stopped: false }; + if (!meta) return result; + + const store = new ScrapeStore(dir); + const options = this.previewOptions(meta); + try { + for (const row of store.pendingTitles(Math.max(1, limit))) { + if (this.stopRequested.has(meta.id)) { + meta.state = "paused"; + result.stopped = true; + break; + } + + const title = await resolveEmbedTitle(row.url, options); + if (title) { + store.markTitle(row.id, title); + result.titled++; + continue; + } + + const attempts = store.bumpTitleAttempt(row.id); + const maxAttempts = this.options.previewMaxAttempts ?? 3; + if (attempts >= maxAttempts) { + store.markTitleError(row.id, `no title after ${attempts} attempt(s)`); + result.failed++; + this.log( + `${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})` + ); + } else { + this.log( + `${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})` + ); + } + } + + result.remaining = store.pendingTitles(5000).length; + } finally { + store.close(); + } + + return result; + } + /** One batch of preview rows. Mutates the store and the meta counters. */ private async previewRows( store: ScrapeStore, @@ -576,47 +633,11 @@ export class ScrapeEngine { skipped: 0, failed: 0, ratelimited: 0, - titled: 0, remaining: 0, stopped: false }; - // Title work first, then previews with what is left of the batch. YouTube rows that - // never got a title share the queue with the preview pass. - const titleBudget = store.pendingTitles(1).length > 0 && limit > 1 ? Math.ceil(limit / 2) : limit; - for (const row of store.pendingTitles(titleBudget)) { - if (this.stopRequested.has(meta.id)) { - meta.state = "paused"; - result.stopped = true; - break; - } - - const title = await resolveEmbedTitle(row.url, options); - if (title) { - store.markTitle(row.id, title); - result.titled++; - } else { - const attempts = store.bumpTitleAttempt(row.id); - const maxAttempts = this.options.previewMaxAttempts ?? 3; - if (attempts >= maxAttempts) { - store.markTitleError(row.id, `no title after ${attempts} attempt(s)`); - this.log(`${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})`); - } else { - this.log(`${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})`); - } - } - - this.persist(dir, meta); - await this.sleep(previewGapMs(options)); - } - - if (result.stopped) { - result.remaining = store.pendingPreviews(5000, true).length; - return result; - } - - const previewLimit = Math.max(1, limit - result.titled); - const pending = store.pendingPreviews(previewLimit, true); + const pending = store.pendingPreviews(limit, true); for (const row of pending) { if (this.stopRequested.has(meta.id)) { @@ -752,10 +773,10 @@ export class ScrapeEngine { } result.remaining = store.pendingPreviews(5000, true).length; - if (pending.length || result.titled) { + if (pending.length) { this.log( `${meta.id}: preview pass done: ${result.cached} cached, ${result.skipped} skipped, ` + - `${result.failed} failed, ${result.titled} titled, ${result.remaining} still pending` + `${result.failed} failed, ${result.remaining} still pending` ); } diff --git a/media-scrape/src/index.ts b/media-scrape/src/index.ts index 70f3ac3..daef669 100644 --- a/media-scrape/src/index.ts +++ b/media-scrape/src/index.ts @@ -37,6 +37,7 @@ export { type BackfillStatus, type BackfillTotals } from "./backfill.js"; +export { TitleLane, type TitleLaneOptions, type TitleLaneStatus } from "./titles.js"; export { ScrapeEngine, type FetchPage, diff --git a/media-scrape/src/store.ts b/media-scrape/src/store.ts index c4ca2a0..3f12664 100644 --- a/media-scrape/src/store.ts +++ b/media-scrape/src/store.ts @@ -538,6 +538,10 @@ export class ScrapeStore { failed: number; /** YouTube-like rows still missing a title. */ titlePending: number; + /** Rows with a stored title. */ + titled: number; + /** Rows closed out without a title. */ + titleFailed: number; byKind: Record; } { const empty = () => ({ total: 0, pending: 0, cached: 0, failed: 0 }); @@ -583,15 +587,24 @@ export class ScrapeStore { const titles = this.db .prepare( - `SELECT COUNT(*) AS count FROM media - WHERE title IS NULL AND title_error IS NULL - AND (lower(COALESCE(provider, '')) LIKE '%youtube%' - OR lower(url) LIKE '%youtube.com/%' - OR lower(url) LIKE '%youtu.be/%')` + `SELECT + SUM(title IS NULL AND title_error IS NULL + AND (lower(COALESCE(provider, '')) LIKE '%youtube%' + OR lower(url) LIKE '%youtube.com/%' + OR lower(url) LIKE '%youtu.be/%')) AS pending, + SUM(title IS NOT NULL) AS titled, + SUM(title IS NULL AND title_error IS NOT NULL) AS failed + FROM media` ) - .get() as { count: number | null } | undefined; + .get() as { pending: number | null; titled: number | null; failed: number | null } | undefined; - return { ...totals, titlePending: titles?.count ?? 0, byKind }; + return { + ...totals, + titlePending: titles?.pending ?? 0, + titled: titles?.titled ?? 0, + titleFailed: titles?.failed ?? 0, + byKind + }; } /** Count one transient preview failure and return the new total. */ diff --git a/media-scrape/src/titles.ts b/media-scrape/src/titles.ts new file mode 100644 index 0000000..9cd1410 --- /dev/null +++ b/media-scrape/src/titles.ts @@ -0,0 +1,202 @@ +/* + * The YouTube title catch-up lane. + * + * Titles normally arrive with the scrape (Discord embeds carry them), so this lane only exists + * for rows indexed before titles were stored. It talks to YouTube's oEmbed directly, never to + * Discord, so it takes no media-traffic lock and can run beside a walk. Pacing is hard-coded + * (100 rows per batch, 10 s between batches): the lane is a one-time catch-up, not a tuned knob. + */ + +import { ScrapeEngine } from "./engine.js"; +import { listScrapes, ScrapeStore, scrapeDir } from "./store.js"; + +export interface TitleLaneStatus { + state: "idle" | "running" | "paused" | "done"; + pending: number; + resolved: number; + failed: number; + /** Rows handled since the lane started. */ + processed: number; + startedAt: string | null; + finishedAt: string | null; + perMinute: number | null; + etaMs: number | null; + /** The scrape being worked on right now. */ + currentScrapeId: string | null; +} + +export interface TitleLaneOptions { + rootDir: string; + log?: (line: string) => void; + now?: () => number; + sleep?: (ms: number) => Promise; + /** Hard-coded in production; tests override. */ + batchSize?: number; + batchPauseMs?: number; +} + +const BATCH_SIZE = 100; +const BATCH_PAUSE_MS = 10_000; + +interface LaneState { + state: TitleLaneStatus["state"]; + stop: boolean; + startedAt: number | null; + finishedAt: number | null; + processed: number; + perMinute: number | null; + currentScrapeId: string | null; +} + +const defaultSleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export class TitleLane { + private readonly state: LaneState = { + state: "idle", + stop: false, + startedAt: null, + finishedAt: null, + processed: 0, + perMinute: null, + currentScrapeId: null + }; + + private active: Promise = Promise.resolve(); + + constructor( + private readonly engine: ScrapeEngine, + private readonly options: TitleLaneOptions + ) {} + + private now(): number { + return this.options.now?.() ?? Date.now(); + } + + private sleep(ms: number): Promise { + return (this.options.sleep ?? defaultSleep)(ms); + } + + /** Totals across every scrape, with the lane's own run state. */ + status(): TitleLaneStatus { + let pending = 0; + let resolved = 0; + let failed = 0; + + for (const meta of listScrapes(this.options.rootDir)) { + if (meta.state === "deleted" || meta.state === "deleting" || meta.state === "pending-delete") { + continue; + } + + try { + const store = new ScrapeStore(scrapeDir(this.options.rootDir, meta.id)); + try { + const counts = store.previewCounts(); + pending += counts.titlePending; + resolved += counts.titled ?? 0; + failed += counts.titleFailed ?? 0; + } finally { + store.close(); + } + } catch { + // an unreadable index counts as nothing to do + } + } + + const runMs = this.state.startedAt ? Math.max(1, this.now() - this.state.startedAt) : 0; + const perMinute = + this.state.perMinute ?? + (runMs && this.state.processed ? this.state.processed / (runMs / 60_000) : null); + + return { + state: this.state.state, + pending, + resolved, + failed, + processed: this.state.processed, + startedAt: this.state.startedAt ? new Date(this.state.startedAt).toISOString() : null, + finishedAt: this.state.finishedAt ? new Date(this.state.finishedAt).toISOString() : null, + perMinute: perMinute === null ? null : Math.round(perMinute * 10) / 10, + etaMs: + pending > 0 && perMinute && perMinute > 0 + ? Math.round((pending / perMinute) * 60_000) + : null, + currentScrapeId: this.state.currentScrapeId + }; + } + + start(): void { + this.state.stop = false; + this.state.state = "running"; + this.state.startedAt = this.state.startedAt ?? this.now(); + this.state.finishedAt = null; + this.options.log?.("titles: lane started"); + this.schedule(); + } + + pause(): void { + if (this.state.state !== "running") return; + this.state.stop = true; + this.state.state = "paused"; + this.options.log?.("titles: pausing"); + } + + /** Resolves when the lane has stopped for now. */ + idle(): Promise { + return this.active; + } + + private schedule(): void { + this.active = this.active.then(async () => { + if (this.state.state !== "running") return; + await this.run(); + }); + } + + private async run(): Promise { + const batchSize = Math.max(1, this.options.batchSize ?? BATCH_SIZE); + const batchPauseMs = Math.max(0, this.options.batchPauseMs ?? BATCH_PAUSE_MS); + + while (!this.state.stop) { + const meta = listScrapes(this.options.rootDir).find(candidate => { + if (candidate.state === "deleted" || candidate.state === "deleting") return false; + try { + const store = new ScrapeStore(scrapeDir(this.options.rootDir, candidate.id)); + try { + return store.previewCounts().titlePending > 0; + } finally { + store.close(); + } + } catch { + return false; + } + }); + + if (!meta) break; + + this.state.currentScrapeId = meta.id; + const result = await this.engine.titleBatch(meta.id, batchSize); + this.state.processed += result.titled + result.failed; + + const elapsed = Math.max(1, this.now() - (this.state.startedAt ?? this.now())); + this.state.perMinute = this.state.processed / (elapsed / 60_000); + + this.options.log?.( + `titles: ${meta.id} batch done: +${result.titled} titled, ${result.failed} closed, ` + + `${result.remaining} left in scrape` + ); + + if (this.state.stop) break; + if (result.stopped) { + this.state.state = "paused"; + break; + } + + if (batchPauseMs > 0) await this.sleep(batchPauseMs); + } + + this.state.currentScrapeId = null; + if (this.state.state === "running") this.state.state = this.state.stop ? "paused" : "done"; + this.state.finishedAt = this.state.state === "done" ? this.now() : null; + this.options.log?.(`titles: lane ${this.state.state} (${this.state.processed} rows this run)`); + } +} diff --git a/media-scrape/test/run.ts b/media-scrape/test/run.ts index 7561881..62fed9a 100644 --- a/media-scrape/test/run.ts +++ b/media-scrape/test/run.ts @@ -21,6 +21,7 @@ import { import { ogImageFromHtml, resolveEmbedPreview } from "../src/embed-preview.js"; import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js"; import { BackfillQueue } from "../src/backfill.js"; +import { TitleLane } from "../src/titles.js"; import { extractMedia, normalizeYouTubeUrl } from "../src/extract.js"; import { PuppetBackend } from "../src/puppet-backend.js"; import { betterCandidate, sortCandidates } from "../src/dedupe.js"; @@ -1502,7 +1503,7 @@ void (async () => { store.close(); }); - await test("previewBatch resolves titles and gives up after the cap", async () => { + await test("titleBatch resolves titles and gives up after the cap", async () => { const embed = { type: "video", url: "https://www.youtube.com/watch?v=titlepass", @@ -1534,7 +1535,8 @@ void (async () => { const started = await engine.start({ channelId: "135", guildId: null, mode: "links", backend }); await engine.processQueue(); - await engine.previewBatch(started.id, 5); + const batch = await engine.titleBatch(started.id, 5); + assert.equal(batch.titled, 1); const store = new ScrapeStore(join(root, started.id)); const row = store.queryMedia({ kind: "video" })[0]; @@ -1561,7 +1563,8 @@ void (async () => { const dead = await deadEngine.start({ channelId: "136", guildId: null, mode: "links", backend: deadBackend }); await deadEngine.processQueue(); - await deadEngine.previewBatch(dead.id, 5); + const deadBatch = await deadEngine.titleBatch(dead.id, 5); + assert.equal(deadBatch.failed, 1); const deadStore = new ScrapeStore(join(root, dead.id)); const deadRow = deadStore.queryMedia({ kind: "video" })[0]; @@ -1570,6 +1573,55 @@ void (async () => { deadStore.close(); }); + await test("title lane runs a catch-up across scrapes", async () => { + const laneRoot = join(root, "lane-root"); + mkdirSync(laneRoot, { recursive: true }); + + const embed = (id: string) => ({ + type: "video", + url: `https://www.youtube.com/watch?v=${id}`, + title: null, + provider: "YouTube", + videoUrl: `https://www.youtube.com/watch?v=${id}`, + imageUrl: null, + thumbnailUrl: null + }); + const backend = new FakeBackend([ + { messages: [message("1500", { embeds: [embed("lane1"), embed("lane2")] })] } + ]); + const engine = engineFor(laneRoot, backend, { + download: { + fetchImpl: async (input: any) => + new Response(JSON.stringify({ title: `T ${String(input).length}` }), { + status: 200, + headers: { "content-type": "application/json" } + }), + minGapMs: 0, + maxGapMs: 0, + sleep: async () => {} + } + }); + await engine.start({ channelId: "137", guildId: null, mode: "links", backend }); + await engine.processQueue(); + + const lane = new TitleLane(engine, { + rootDir: laneRoot, + batchSize: 1, + batchPauseMs: 0, + sleep: async () => {}, + log: () => {} + }); + lane.start(); + await lane.idle(); + + const status = lane.status(); + assert.equal(status.state, "done"); + assert.equal(status.pending, 0); + assert.equal(status.resolved, 2); + assert.equal(status.failed, 0); + assert.ok(status.processed >= 2); + }); + await test("title search matches word prefixes and exact phrases", () => { const dir = join(root, "title-search"); mkdirSync(dir, { recursive: true }); diff --git a/poc/public/media.css b/poc/public/media.css index 38644a7..c782882 100644 --- a/poc/public/media.css +++ b/poc/public/media.css @@ -791,3 +791,14 @@ .vk-chip-strike { text-decoration: line-through; } + +/* The YouTube title catch-up lane, inside the Bot tab backfill section. */ +.vk-bot-lane { + border: 1px dashed var(--color-neutral-700); + border-radius: 5px; + padding: 6px 8px; + margin-bottom: 8px; + display: flex; + flex-direction: column; + gap: 4px; +} diff --git a/poc/public/media.js b/poc/public/media.js index 5c7dce3..5e96658 100644 --- a/poc/public/media.js +++ b/poc/public/media.js @@ -40,6 +40,7 @@ loadError: false, pageSize: 60, backfill: null, + titles: null, preferences: null, /** Scrapes that finished while this page was open, until the tab is reloaded. */ completed: new Set() @@ -75,6 +76,7 @@
+
@@ -101,6 +103,7 @@ const botLimited = overlay.querySelector("#bot-limited"); const backfillSummary = overlay.querySelector("#backfill-summary"); const backfillJobs = overlay.querySelector("#backfill-jobs"); + const titleLane = overlay.querySelector("#title-lane"); const tunablesHost = overlay.querySelector("#bot-tunables"); const botErrors = overlay.querySelector("#bot-errors"); @@ -790,7 +793,6 @@ "vk-bot-job-member", `${member.scrapeId} \u00b7 ${member.state} \u00b7 ${member.mode} \u00b7 ` + `${member.backend} \u00b7 ${member.pending} pending` + - (member.pendingTitles ? ` + ${member.pendingTitles} titles` : "") + (kinds ? ` (${kinds})` : "") ) ); @@ -826,6 +828,57 @@ } } + function renderTitleLane(status) { + if (!status) return; + titleLane.replaceChildren(); + + const head = el("div", "vk-bot-job-head"); + head.append(el("span", "vk-bot-job-title", "Titles (YouTube)")); + head.append(el("span", `vk-bot-state vk-bot-state-${status.state}`, status.state)); + + const parts = [`${status.pending} pending`, `${status.resolved} titles`]; + if (status.failed) parts.push(`${status.failed} failed`); + if (status.perMinute) parts.push(`${status.perMinute}/min`); + if (status.pending > 0) parts.push(`ETA ${formatEta(status.etaMs)}`); + if (status.currentScrapeId) parts.push(`on ${channelLabel(status.currentScrapeId)}`); + head.append(el("span", "vk-bot-job-meta", parts.join(" \u00b7 "))); + titleLane.append(head); + + const actions = el("div", "vk-bot-job-actions"); + if (status.state === "running") { + actions.append(button("Pause", () => void titleAction("pause"))); + } else if (status.pending > 0) { + actions.append(button("Start", () => void titleAction("start"))); + } + titleLane.append(actions); + + const note = el( + "div", + "vk-bot-job-member", + "One-time catch-up for rows indexed before titles were stored; talks to YouTube only, " + + "100 per batch, 10 s between batches." + ); + titleLane.append(note); + } + + async function titleAction(action) { + try { + state.titles = await window.VK.api.post(`/api/media/titles/${action}`, {}); + renderTitleLane(state.titles); + } catch (error) { + setStatus(`titles ${action} failed: ${error.message}`); + } + } + + async function refreshTitles() { + try { + state.titles = await window.VK.api.get("/api/media/titles"); + } catch { + state.titles = null; + } + renderTitleLane(state.titles); + } + async function refreshBackfill() { try { state.backfill = await window.VK.api.get("/api/media/backfill"); @@ -1765,6 +1818,7 @@ : "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)"; if (!state.preferences) void refreshPreferences(); void refreshBackfill(); + void refreshTitles(); void loadScrapes().then(renderScrapes); void refreshErrors(); } else { @@ -1782,7 +1836,10 @@ state.completed.clear(); void loadScrapes(); if (state.view === "media") void loadMedia(true); - if (state.view === "bot") void refreshBackfill(); + if (state.view === "bot") { + void refreshBackfill(); + void refreshTitles(); + } }); overlay.querySelector("#backfill-start").addEventListener("click", () => void backfillAction("start")); @@ -1812,7 +1869,10 @@ // Backfill progress while the Bot tab is open. setInterval(() => { - if (state.view === "bot") void refreshBackfill(); + if (state.view === "bot") { + void refreshBackfill(); + void refreshTitles(); + } }, 3000); // Infinite scroll: the buffer entering the viewport starts the next page. diff --git a/poc/src/media.ts b/poc/src/media.ts index ed18ee1..dc35d76 100644 --- a/poc/src/media.ts +++ b/poc/src/media.ts @@ -17,6 +17,8 @@ import { ScrapeEngine, ScrapeStore, scrapeDir, + TitleLane, + type TitleLaneStatus, snowflakeFromIso, resolveDbRoot, type MediaKind, @@ -171,6 +173,7 @@ export class MediaService { private readonly rootDir = resolveDbRoot(); private readonly engine: ScrapeEngine; private readonly backfill: BackfillQueue; + private readonly titles: TitleLane; private bot: BotClient | null = null; private botFetcher: SlopbotFetcher | null = null; @@ -214,6 +217,14 @@ export class MediaService { log: line => this.deps.log?.(line) }); + // The title catch-up lane talks to YouTube only, so it runs beside the backfill queue. + // It auto-starts because it is a one-time catch-up: with nothing pending it exits at once. + this.titles = new TitleLane(this.engine, { + rootDir: this.rootDir, + log: line => this.deps.log?.(line) + }); + this.titles.start(); + const stale = this.engine.reconcileStaleJobs(); if (stale.length) { this.deps.log?.(`paused ${stale.length} stale scrape(s) left running by a previous daemon`); @@ -268,6 +279,20 @@ export class MediaService { }; } + titleStatus(): TitleLaneStatus { + return this.titles.status(); + } + + startTitles(): TitleLaneStatus { + this.titles.start(); + return this.titles.status(); + } + + pauseTitles(): TitleLaneStatus { + this.titles.pause(); + return this.titles.status(); + } + startBackfill(chatId?: string): BackfillStatus { this.backfill.start(chatId); return this.backfillStatus(); diff --git a/poc/src/webui.ts b/poc/src/webui.ts index 6570255..19f64b3 100644 --- a/poc/src/webui.ts +++ b/poc/src/webui.ts @@ -835,6 +835,27 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle { }) ); + app.get( + "/api/media/titles", + wrap(async (_req, res) => { + res.json({ ok: true, result: deps.media.titleStatus() }); + }) + ); + + app.post( + "/api/media/titles/start", + wrap(async (_req, res) => { + res.json({ ok: true, result: deps.media.startTitles() }); + }) + ); + + app.post( + "/api/media/titles/pause", + wrap(async (_req, res) => { + res.json({ ok: true, result: deps.media.pauseTitles() }); + }) + ); + app.post( "/api/media/backfill/start", wrap(async (req, res) => { diff --git a/poc/test/media-ui-smoke.mjs b/poc/test/media-ui-smoke.mjs index 4a5f88e..27bd38b 100644 --- a/poc/test/media-ui-smoke.mjs +++ b/poc/test/media-ui-smoke.mjs @@ -202,6 +202,12 @@ try { check("media toolbar hidden on the bot tab", !(await page.locator("#media-toolbar").isVisible())); check("media filter drawer hidden on the bot tab", !(await page.locator(".vk-media-drawer").isVisible())); + await page.waitForFunction( + () => document.querySelector("#title-lane")?.textContent?.includes("Titles (YouTube)"), + { timeout: 15000 } + ); + check("title lane card renders", (await page.locator("#title-lane .vk-bot-job-head").count()) === 1); + const summaryLines = await page.locator(".vk-bot-summary-line").count(); step("backfill summary lines", summaryLines); check("backfill summary renders", summaryLines >= 3, summaryLines);