Run the title catch-up in its own YouTube lane

2026-09-17 14:25:08 +02:00
parent 8befc05591
commit 9481286169
16 changed files with 495 additions and 82 deletions
@@ -40,6 +40,14 @@ mobile draft at 390px.
  - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
    cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
    `previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
+  - [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated
+    `TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause
+    (hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no
+    media-traffic lock because it only calls YouTube's oEmbed. New `titleBatch` on the engine,
+    per-scrape `titlePending/titled/titleFailed` counters, `/api/media/titles` + start/pause
+    routes, and a Bot-tab card with progress, throughput, ETA, and Start/Pause. The lane
+    auto-starts on boot; with nothing pending it exits immediately. Preview backfill goes back to
+    previews only. Unit tests for `titleBatch` and a lane catch-up run; smoke covers the card.
  - [x] C52 Wildcard title search. `*` now maps to a LIKE wildcard in both title modes: a leading
    `*` drops the word anchor (`*car` finds *scar* and *discard*), in-token stars match between
    letters (`c*r*`), and a trailing star matches the plain prefix. `%` and `_` are escaped with an
@@ -64,13 +64,6 @@ Ordered backlog. `CURRENT.md` tracks the live session state.

 ## Media archive follow-ups

-2. **Title work in its own lane.** Title resolution talks straight to `youtube.com/oembed`
-   (daemon fetch; no Discord API, no bot account), but it currently shares the media-traffic lock
-   and the preview batch pacing, so it serializes behind walks and previews. Move it to its own
-   lane: skip the media-traffic lock, give it its own pacing/concurrency settings, and handle
-   YouTube `429`s as a cooldown rather than a normal failed try (they must not consume the tries
-   budget). Show the lane's throughput separately in the Bot tab.
-
 ## Web UI follow-ups

 Ordered after the voice backend unless the owner asks first.
@@ -168,12 +168,15 @@ Previews run after every walk:
 - The job stays `running` through the download and preview passes, so a client never sees `done`
  on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
 - The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
-  before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
-  (`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
-  through the provider oEmbed and stored as `title` + `title_key`; failures count toward
-  `title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
-  videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
-  both kinds of work remain. One job per chat groups every paused, done, or failed scrape
+  before their previews were cached. It holds preview work only.
+- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
+  indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
+  job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
+  media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
+  beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
+  youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
+  stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
+  throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
  of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
  batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
  Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
@@ -200,8 +200,10 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
  that disappears when the tab is opened and a Clear button.
 - **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
  **Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
-  current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing
-  YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its
+  current settings, and a Bot/Puppet split of pending/cached/failed. A separate **Titles
+  (YouTube)** card tracks the one-time title catch-up: pending/resolved/failed, throughput, ETA,
+  and Start/Pause. It talks to YouTube only (hard-coded 100 per batch, 10 s between batches) and
+  runs beside the preview queue. One job card per chat lists its
  member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
  counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
  carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
@@ -165,7 +165,8 @@ box live in Chat, not Call, and the connection stats are not chips yet.
  the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local
  file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
  tooltip.
- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has
+- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
+  catch-up lane with its own Start/Pause, and the media tunables. Backfill has
  totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
  per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
  **Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
@@ -302,6 +303,9 @@ replaceable.
 | GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, `etaStartMs`/`etaRunMs`/`etaTotalMs`, totals with a Bot/Puppet split, the live settings, and `limitedUntil` |
 | POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job |
 | POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch |
+| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
+| POST | `/api/media/titles/start` | none | Start the title lane |
+| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
 | GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
 | GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
 | GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
@@ -29,8 +29,6 @@ export interface BackfillMember {
    backend: BackendName;
    mode: ScrapeMode;
    pending: number;
-    /** YouTube-like rows still missing a title. */
-    pendingTitles: number;
    cached: number;
    failed: number;
    total: number;
@@ -154,7 +152,6 @@ export class BackfillQueue {
                backend: meta.backend,
                mode: meta.mode,
                pending: 0,
-                pendingTitles: 0,
                cached: 0,
                failed: 0,
                total: 0,
@@ -168,7 +165,6 @@ export class BackfillQueue {
                try {
                    const counts = store.previewCounts();
                    member.pending = counts.pending;
-                    member.pendingTitles = counts.titlePending;
                    member.cached = counts.cached;
                    member.failed = counts.failed;
                    member.total = counts.total;
@@ -209,10 +205,7 @@ export class BackfillQueue {
        for (const [chatId, group] of groups) {
            // Hide chats that are fully backfilled unless their job did work this session.
            const tracked = this.jobs.get(chatId);
-            const pending = group.members.reduce(
-                (sum, member) => sum + member.pending + member.pendingTitles,
-                0
-            );
+            const pending = group.members.reduce((sum, member) => sum + member.pending, 0);
            if (!tracked && pending === 0) continue;

            const perMinute = tracked?.perMinute ?? this.estimatedPerMinute(settings, gaps);
@@ -418,9 +411,7 @@ export class BackfillQueue {

        while (!job.stop) {
            const group = this.plan(false).get(job.chatId);
-            const member = group?.members.find(
-                candidate => candidate.pending > 0 || candidate.pendingTitles > 0
-            );
+            const member = group?.members.find(candidate => candidate.pending > 0);
            if (!member) break;

            const limit = this.engine.rateLimitStatus();
@@ -46,14 +46,21 @@ export interface ScrapeBackend {
    fetchPage(channelId: string, before: string | null, limit: number): Promise<FetchPage>;
 }

+/** One batch of title resolution, as returned by `titleBatch`. */
+export interface TitleBatchResult {
+    titled: number;
+    /** Rows closed out because the provider never returned a title. */
+    failed: number;
+    remaining: number;
+    stopped: boolean;
+}
+
 /** One batch of preview work, as returned by `previewBatch`. */
 export interface PreviewBatchResult {
    cached: number;
    skipped: number;
    failed: number;
    ratelimited: number;
-    /** Titles resolved in this batch. */
-    titled: number;
    /** Rows still pending after the batch. */
    remaining: number;
    /** True when a stop request interrupted the batch. */
@@ -550,7 +557,7 @@ export class ScrapeEngine {
        const dir = join(this.options.rootDir, scrapeId);
        const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
        if (!meta) {
-            return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, titled: 0, remaining: 0, stopped: false };
+            return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, remaining: 0, stopped: false };
        }

        const releaseTraffic = await this.acquireTraffic();
@@ -563,6 +570,56 @@ export class ScrapeEngine {
        }
    }

+    /**
+     * Resolve YouTube titles for up to `limit` rows. No media-traffic lock: these requests go to
+     * YouTube's oEmbed, never Discord, so the lane can run beside a walk. The caller paces.
+     */
+    async titleBatch(scrapeId: string, limit: number): Promise<TitleBatchResult> {
+        const dir = join(this.options.rootDir, scrapeId);
+        const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
+        const result: TitleBatchResult = { titled: 0, failed: 0, remaining: 0, stopped: false };
+        if (!meta) return result;
+
+        const store = new ScrapeStore(dir);
+        const options = this.previewOptions(meta);
+        try {
+            for (const row of store.pendingTitles(Math.max(1, limit))) {
+                if (this.stopRequested.has(meta.id)) {
+                    meta.state = "paused";
+                    result.stopped = true;
+                    break;
+                }
+
+                const title = await resolveEmbedTitle(row.url, options);
+                if (title) {
+                    store.markTitle(row.id, title);
+                    result.titled++;
+                    continue;
+                }
+
+                const attempts = store.bumpTitleAttempt(row.id);
+                const maxAttempts = this.options.previewMaxAttempts ?? 3;
+                if (attempts >= maxAttempts) {
+                    store.markTitleError(row.id, `no title after ${attempts} attempt(s)`);
+                    result.failed++;
+                    this.log(
+                        `${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})`
+                    );
+                } else {
+                    this.log(
+                        `${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})`
+                    );
+                }
+            }
+
+            result.remaining = store.pendingTitles(5000).length;
+        } finally {
+            store.close();
+        }
+
+        return result;
+    }
+
    /** One batch of preview rows. Mutates the store and the meta counters. */
    private async previewRows(
        store: ScrapeStore,
@@ -576,47 +633,11 @@ export class ScrapeEngine {
            skipped: 0,
            failed: 0,
            ratelimited: 0,
-            titled: 0,
            remaining: 0,
            stopped: false
        };

-        // Title work first, then previews with what is left of the batch. YouTube rows that
-        // never got a title share the queue with the preview pass.
-        const titleBudget = store.pendingTitles(1).length > 0 && limit > 1 ? Math.ceil(limit / 2) : limit;
-        for (const row of store.pendingTitles(titleBudget)) {
-            if (this.stopRequested.has(meta.id)) {
-                meta.state = "paused";
-                result.stopped = true;
-                break;
-            }
-
-            const title = await resolveEmbedTitle(row.url, options);
-            if (title) {
-                store.markTitle(row.id, title);
-                result.titled++;
-            } else {
-                const attempts = store.bumpTitleAttempt(row.id);
-                const maxAttempts = this.options.previewMaxAttempts ?? 3;
-                if (attempts >= maxAttempts) {
-                    store.markTitleError(row.id, `no title after ${attempts} attempt(s)`);
-                    this.log(`${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})`);
-                } else {
-                    this.log(`${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})`);
-                }
-            }
-
-            this.persist(dir, meta);
-            await this.sleep(previewGapMs(options));
-        }
-
-        if (result.stopped) {
-            result.remaining = store.pendingPreviews(5000, true).length;
-            return result;
-        }
-
-        const previewLimit = Math.max(1, limit - result.titled);
-        const pending = store.pendingPreviews(previewLimit, true);
+        const pending = store.pendingPreviews(limit, true);

        for (const row of pending) {
            if (this.stopRequested.has(meta.id)) {
@@ -752,10 +773,10 @@ export class ScrapeEngine {
        }

        result.remaining = store.pendingPreviews(5000, true).length;
-        if (pending.length || result.titled) {
+        if (pending.length) {
            this.log(
                `${meta.id}: preview pass done: ${result.cached} cached, ${result.skipped} skipped, ` +
-                    `${result.failed} failed, ${result.titled} titled, ${result.remaining} still pending`
+                    `${result.failed} failed, ${result.remaining} still pending`
            );
        }

@@ -37,6 +37,7 @@ export {
    type BackfillStatus,
    type BackfillTotals
 } from "./backfill.js";
+export { TitleLane, type TitleLaneOptions, type TitleLaneStatus } from "./titles.js";
 export {
    ScrapeEngine,
    type FetchPage,
@@ -538,6 +538,10 @@ export class ScrapeStore {
        failed: number;
        /** YouTube-like rows still missing a title. */
        titlePending: number;
+        /** Rows with a stored title. */
+        titled: number;
+        /** Rows closed out without a title. */
+        titleFailed: number;
        byKind: Record<MediaKind, { total: number; pending: number; cached: number; failed: number }>;
    } {
        const empty = () => ({ total: 0, pending: 0, cached: 0, failed: 0 });
@@ -583,15 +587,24 @@ export class ScrapeStore {

        const titles = this.db
            .prepare(
-                `SELECT COUNT(*) AS count FROM media
-                 WHERE title IS NULL AND title_error IS NULL
+                `SELECT
+                   SUM(title IS NULL AND title_error IS NULL
                       AND (lower(COALESCE(provider, '')) LIKE '%youtube%'
                            OR lower(url) LIKE '%youtube.com/%'
-                        OR lower(url) LIKE '%youtu.be/%')`
+                            OR lower(url) LIKE '%youtu.be/%')) AS pending,
+                   SUM(title IS NOT NULL) AS titled,
+                   SUM(title IS NULL AND title_error IS NOT NULL) AS failed
+                 FROM media`
            )
-            .get() as { count: number | null } | undefined;
+            .get() as { pending: number | null; titled: number | null; failed: number | null } | undefined;

-        return { ...totals, titlePending: titles?.count ?? 0, byKind };
+        return {
+            ...totals,
+            titlePending: titles?.pending ?? 0,
+            titled: titles?.titled ?? 0,
+            titleFailed: titles?.failed ?? 0,
+            byKind
+        };
    }

    /** Count one transient preview failure and return the new total. */
@@ -0,0 +1,202 @@
+/*
+ * The YouTube title catch-up lane.
+ *
+ * Titles normally arrive with the scrape (Discord embeds carry them), so this lane only exists
+ * for rows indexed before titles were stored. It talks to YouTube's oEmbed directly, never to
+ * Discord, so it takes no media-traffic lock and can run beside a walk. Pacing is hard-coded
+ * (100 rows per batch, 10 s between batches): the lane is a one-time catch-up, not a tuned knob.
+ */
+
+import { ScrapeEngine } from "./engine.js";
+import { listScrapes, ScrapeStore, scrapeDir } from "./store.js";
+
+export interface TitleLaneStatus {
+    state: "idle" | "running" | "paused" | "done";
+    pending: number;
+    resolved: number;
+    failed: number;
+    /** Rows handled since the lane started. */
+    processed: number;
+    startedAt: string | null;
+    finishedAt: string | null;
+    perMinute: number | null;
+    etaMs: number | null;
+    /** The scrape being worked on right now. */
+    currentScrapeId: string | null;
+}
+
+export interface TitleLaneOptions {
+    rootDir: string;
+    log?: (line: string) => void;
+    now?: () => number;
+    sleep?: (ms: number) => Promise<void>;
+    /** Hard-coded in production; tests override. */
+    batchSize?: number;
+    batchPauseMs?: number;
+}
+
+const BATCH_SIZE = 100;
+const BATCH_PAUSE_MS = 10_000;
+
+interface LaneState {
+    state: TitleLaneStatus["state"];
+    stop: boolean;
+    startedAt: number | null;
+    finishedAt: number | null;
+    processed: number;
+    perMinute: number | null;
+    currentScrapeId: string | null;
+}
+
+const defaultSleep = (ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms));
+
+export class TitleLane {
+    private readonly state: LaneState = {
+        state: "idle",
+        stop: false,
+        startedAt: null,
+        finishedAt: null,
+        processed: 0,
+        perMinute: null,
+        currentScrapeId: null
+    };
+
+    private active: Promise<void> = Promise.resolve();
+
+    constructor(
+        private readonly engine: ScrapeEngine,
+        private readonly options: TitleLaneOptions
+    ) {}
+
+    private now(): number {
+        return this.options.now?.() ?? Date.now();
+    }
+
+    private sleep(ms: number): Promise<void> {
+        return (this.options.sleep ?? defaultSleep)(ms);
+    }
+
+    /** Totals across every scrape, with the lane's own run state. */
+    status(): TitleLaneStatus {
+        let pending = 0;
+        let resolved = 0;
+        let failed = 0;
+
+        for (const meta of listScrapes(this.options.rootDir)) {
+            if (meta.state === "deleted" || meta.state === "deleting" || meta.state === "pending-delete") {
+                continue;
+            }
+
+            try {
+                const store = new ScrapeStore(scrapeDir(this.options.rootDir, meta.id));
+                try {
+                    const counts = store.previewCounts();
+                    pending += counts.titlePending;
+                    resolved += counts.titled ?? 0;
+                    failed += counts.titleFailed ?? 0;
+                } finally {
+                    store.close();
+                }
+            } catch {
+                // an unreadable index counts as nothing to do
+            }
+        }
+
+        const runMs = this.state.startedAt ? Math.max(1, this.now() - this.state.startedAt) : 0;
+        const perMinute =
+            this.state.perMinute ??
+            (runMs && this.state.processed ? this.state.processed / (runMs / 60_000) : null);
+
+        return {
+            state: this.state.state,
+            pending,
+            resolved,
+            failed,
+            processed: this.state.processed,
+            startedAt: this.state.startedAt ? new Date(this.state.startedAt).toISOString() : null,
+            finishedAt: this.state.finishedAt ? new Date(this.state.finishedAt).toISOString() : null,
+            perMinute: perMinute === null ? null : Math.round(perMinute * 10) / 10,
+            etaMs:
+                pending > 0 && perMinute && perMinute > 0
+                    ? Math.round((pending / perMinute) * 60_000)
+                    : null,
+            currentScrapeId: this.state.currentScrapeId
+        };
+    }
+
+    start(): void {
+        this.state.stop = false;
+        this.state.state = "running";
+        this.state.startedAt = this.state.startedAt ?? this.now();
+        this.state.finishedAt = null;
+        this.options.log?.("titles: lane started");
+        this.schedule();
+    }
+
+    pause(): void {
+        if (this.state.state !== "running") return;
+        this.state.stop = true;
+        this.state.state = "paused";
+        this.options.log?.("titles: pausing");
+    }
+
+    /** Resolves when the lane has stopped for now. */
+    idle(): Promise<void> {
+        return this.active;
+    }
+
+    private schedule(): void {
+        this.active = this.active.then(async () => {
+            if (this.state.state !== "running") return;
+            await this.run();
+        });
+    }
+
+    private async run(): Promise<void> {
+        const batchSize = Math.max(1, this.options.batchSize ?? BATCH_SIZE);
+        const batchPauseMs = Math.max(0, this.options.batchPauseMs ?? BATCH_PAUSE_MS);
+
+        while (!this.state.stop) {
+            const meta = listScrapes(this.options.rootDir).find(candidate => {
+                if (candidate.state === "deleted" || candidate.state === "deleting") return false;
+                try {
+                    const store = new ScrapeStore(scrapeDir(this.options.rootDir, candidate.id));
+                    try {
+                        return store.previewCounts().titlePending > 0;
+                    } finally {
+                        store.close();
+                    }
+                } catch {
+                    return false;
+                }
+            });
+
+            if (!meta) break;
+
+            this.state.currentScrapeId = meta.id;
+            const result = await this.engine.titleBatch(meta.id, batchSize);
+            this.state.processed += result.titled + result.failed;
+
+            const elapsed = Math.max(1, this.now() - (this.state.startedAt ?? this.now()));
+            this.state.perMinute = this.state.processed / (elapsed / 60_000);
+
+            this.options.log?.(
+                `titles: ${meta.id} batch done: +${result.titled} titled, ${result.failed} closed, ` +
+                    `${result.remaining} left in scrape`
+            );
+
+            if (this.state.stop) break;
+            if (result.stopped) {
+                this.state.state = "paused";
+                break;
+            }
+
+            if (batchPauseMs > 0) await this.sleep(batchPauseMs);
+        }
+
+        this.state.currentScrapeId = null;
+        if (this.state.state === "running") this.state.state = this.state.stop ? "paused" : "done";
+        this.state.finishedAt = this.state.state === "done" ? this.now() : null;
+        this.options.log?.(`titles: lane ${this.state.state} (${this.state.processed} rows this run)`);
+    }
+}
@@ -21,6 +21,7 @@ import {
 import { ogImageFromHtml, resolveEmbedPreview } from "../src/embed-preview.js";
 import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js";
 import { BackfillQueue } from "../src/backfill.js";
+import { TitleLane } from "../src/titles.js";
 import { extractMedia, normalizeYouTubeUrl } from "../src/extract.js";
 import { PuppetBackend } from "../src/puppet-backend.js";
 import { betterCandidate, sortCandidates } from "../src/dedupe.js";
@@ -1502,7 +1503,7 @@ void (async () => {
        store.close();
    });

-    await test("previewBatch resolves titles and gives up after the cap", async () => {
+    await test("titleBatch resolves titles and gives up after the cap", async () => {
        const embed = {
            type: "video",
            url: "https://www.youtube.com/watch?v=titlepass",
@@ -1534,7 +1535,8 @@ void (async () => {

        const started = await engine.start({ channelId: "135", guildId: null, mode: "links", backend });
        await engine.processQueue();
-        await engine.previewBatch(started.id, 5);
+        const batch = await engine.titleBatch(started.id, 5);
+        assert.equal(batch.titled, 1);

        const store = new ScrapeStore(join(root, started.id));
        const row = store.queryMedia({ kind: "video" })[0];
@@ -1561,7 +1563,8 @@ void (async () => {

        const dead = await deadEngine.start({ channelId: "136", guildId: null, mode: "links", backend: deadBackend });
        await deadEngine.processQueue();
-        await deadEngine.previewBatch(dead.id, 5);
+        const deadBatch = await deadEngine.titleBatch(dead.id, 5);
+        assert.equal(deadBatch.failed, 1);

        const deadStore = new ScrapeStore(join(root, dead.id));
        const deadRow = deadStore.queryMedia({ kind: "video" })[0];
@@ -1570,6 +1573,55 @@ void (async () => {
        deadStore.close();
    });

+    await test("title lane runs a catch-up across scrapes", async () => {
+        const laneRoot = join(root, "lane-root");
+        mkdirSync(laneRoot, { recursive: true });
+
+        const embed = (id: string) => ({
+            type: "video",
+            url: `https://www.youtube.com/watch?v=${id}`,
+            title: null,
+            provider: "YouTube",
+            videoUrl: `https://www.youtube.com/watch?v=${id}`,
+            imageUrl: null,
+            thumbnailUrl: null
+        });
+        const backend = new FakeBackend([
+            { messages: [message("1500", { embeds: [embed("lane1"), embed("lane2")] })] }
+        ]);
+        const engine = engineFor(laneRoot, backend, {
+            download: {
+                fetchImpl: async (input: any) =>
+                    new Response(JSON.stringify({ title: `T ${String(input).length}` }), {
+                        status: 200,
+                        headers: { "content-type": "application/json" }
+                    }),
+                minGapMs: 0,
+                maxGapMs: 0,
+                sleep: async () => {}
+            }
+        });
+        await engine.start({ channelId: "137", guildId: null, mode: "links", backend });
+        await engine.processQueue();
+
+        const lane = new TitleLane(engine, {
+            rootDir: laneRoot,
+            batchSize: 1,
+            batchPauseMs: 0,
+            sleep: async () => {},
+            log: () => {}
+        });
+        lane.start();
+        await lane.idle();
+
+        const status = lane.status();
+        assert.equal(status.state, "done");
+        assert.equal(status.pending, 0);
+        assert.equal(status.resolved, 2);
+        assert.equal(status.failed, 0);
+        assert.ok(status.processed >= 2);
+    });
+
    await test("title search matches word prefixes and exact phrases", () => {
        const dir = join(root, "title-search");
        mkdirSync(dir, { recursive: true });
@@ -791,3 +791,14 @@
 .vk-chip-strike {
    text-decoration: line-through;
 }
+
+/* The YouTube title catch-up lane, inside the Bot tab backfill section. */
+.vk-bot-lane {
+    border: 1px dashed var(--color-neutral-700);
+    border-radius: 5px;
+    padding: 6px 8px;
+    margin-bottom: 8px;
+    display: flex;
+    flex-direction: column;
+    gap: 4px;
+}
@@ -40,6 +40,7 @@
        loadError: false,
        pageSize: 60,
        backfill: null,
+        titles: null,
        preferences: null,
        /** Scrapes that finished while this page was open, until the tab is reloaded. */
        completed: new Set()
@@ -75,6 +76,7 @@
                        </span>
                    </div>
                    <div id="backfill-summary" class="vk-bot-summary"></div>
+                    <div id="title-lane" class="vk-bot-lane"></div>
                    <div id="backfill-jobs" class="vk-bot-jobs"></div>
                </div>

@@ -101,6 +103,7 @@
    const botLimited = overlay.querySelector("#bot-limited");
    const backfillSummary = overlay.querySelector("#backfill-summary");
    const backfillJobs = overlay.querySelector("#backfill-jobs");
+    const titleLane = overlay.querySelector("#title-lane");
    const tunablesHost = overlay.querySelector("#bot-tunables");
    const botErrors = overlay.querySelector("#bot-errors");

@@ -790,7 +793,6 @@
                        "vk-bot-job-member",
                        `${member.scrapeId} \u00b7 ${member.state} \u00b7 ${member.mode} \u00b7 ` +
                            `${member.backend} \u00b7 ${member.pending} pending` +
-                            (member.pendingTitles ? ` + ${member.pendingTitles} titles` : "") +
                            (kinds ? ` (${kinds})` : "")
                    )
                );
@@ -826,6 +828,57 @@
        }
    }

+    function renderTitleLane(status) {
+        if (!status) return;
+        titleLane.replaceChildren();
+
+        const head = el("div", "vk-bot-job-head");
+        head.append(el("span", "vk-bot-job-title", "Titles (YouTube)"));
+        head.append(el("span", `vk-bot-state vk-bot-state-${status.state}`, status.state));
+
+        const parts = [`${status.pending} pending`, `${status.resolved} titles`];
+        if (status.failed) parts.push(`${status.failed} failed`);
+        if (status.perMinute) parts.push(`${status.perMinute}/min`);
+        if (status.pending > 0) parts.push(`ETA ${formatEta(status.etaMs)}`);
+        if (status.currentScrapeId) parts.push(`on ${channelLabel(status.currentScrapeId)}`);
+        head.append(el("span", "vk-bot-job-meta", parts.join(" \u00b7 ")));
+        titleLane.append(head);
+
+        const actions = el("div", "vk-bot-job-actions");
+        if (status.state === "running") {
+            actions.append(button("Pause", () => void titleAction("pause")));
+        } else if (status.pending > 0) {
+            actions.append(button("Start", () => void titleAction("start")));
+        }
+        titleLane.append(actions);
+
+        const note = el(
+            "div",
+            "vk-bot-job-member",
+            "One-time catch-up for rows indexed before titles were stored; talks to YouTube only, " +
+                "100 per batch, 10 s between batches."
+        );
+        titleLane.append(note);
+    }
+
+    async function titleAction(action) {
+        try {
+            state.titles = await window.VK.api.post(`/api/media/titles/${action}`, {});
+            renderTitleLane(state.titles);
+        } catch (error) {
+            setStatus(`titles ${action} failed: ${error.message}`);
+        }
+    }
+
+    async function refreshTitles() {
+        try {
+            state.titles = await window.VK.api.get("/api/media/titles");
+        } catch {
+            state.titles = null;
+        }
+        renderTitleLane(state.titles);
+    }
+
    async function refreshBackfill() {
        try {
            state.backfill = await window.VK.api.get("/api/media/backfill");
@@ -1765,6 +1818,7 @@
                : "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)";
            if (!state.preferences) void refreshPreferences();
            void refreshBackfill();
+            void refreshTitles();
            void loadScrapes().then(renderScrapes);
            void refreshErrors();
        } else {
@@ -1782,7 +1836,10 @@
        state.completed.clear();
        void loadScrapes();
        if (state.view === "media") void loadMedia(true);
-        if (state.view === "bot") void refreshBackfill();
+        if (state.view === "bot") {
+            void refreshBackfill();
+            void refreshTitles();
+        }
    });

    overlay.querySelector("#backfill-start").addEventListener("click", () => void backfillAction("start"));
@@ -1812,7 +1869,10 @@

    // Backfill progress while the Bot tab is open.
    setInterval(() => {
-        if (state.view === "bot") void refreshBackfill();
+        if (state.view === "bot") {
+            void refreshBackfill();
+            void refreshTitles();
+        }
    }, 3000);

    // Infinite scroll: the buffer entering the viewport starts the next page.
@@ -17,6 +17,8 @@ import {
    ScrapeEngine,
    ScrapeStore,
    scrapeDir,
+    TitleLane,
+    type TitleLaneStatus,
    snowflakeFromIso,
    resolveDbRoot,
    type MediaKind,
@@ -171,6 +173,7 @@ export class MediaService {
    private readonly rootDir = resolveDbRoot();
    private readonly engine: ScrapeEngine;
    private readonly backfill: BackfillQueue;
+    private readonly titles: TitleLane;

    private bot: BotClient | null = null;
    private botFetcher: SlopbotFetcher | null = null;
@@ -214,6 +217,14 @@ export class MediaService {
            log: line => this.deps.log?.(line)
        });

+        // The title catch-up lane talks to YouTube only, so it runs beside the backfill queue.
+        // It auto-starts because it is a one-time catch-up: with nothing pending it exits at once.
+        this.titles = new TitleLane(this.engine, {
+            rootDir: this.rootDir,
+            log: line => this.deps.log?.(line)
+        });
+        this.titles.start();
+
        const stale = this.engine.reconcileStaleJobs();
        if (stale.length) {
            this.deps.log?.(`paused ${stale.length} stale scrape(s) left running by a previous daemon`);
@@ -268,6 +279,20 @@ export class MediaService {
        };
    }

+    titleStatus(): TitleLaneStatus {
+        return this.titles.status();
+    }
+
+    startTitles(): TitleLaneStatus {
+        this.titles.start();
+        return this.titles.status();
+    }
+
+    pauseTitles(): TitleLaneStatus {
+        this.titles.pause();
+        return this.titles.status();
+    }
+
    startBackfill(chatId?: string): BackfillStatus {
        this.backfill.start(chatId);
        return this.backfillStatus();
@@ -835,6 +835,27 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
        })
    );

+    app.get(
+        "/api/media/titles",
+        wrap(async (_req, res) => {
+            res.json({ ok: true, result: deps.media.titleStatus() });
+        })
+    );
+
+    app.post(
+        "/api/media/titles/start",
+        wrap(async (_req, res) => {
+            res.json({ ok: true, result: deps.media.startTitles() });
+        })
+    );
+
+    app.post(
+        "/api/media/titles/pause",
+        wrap(async (_req, res) => {
+            res.json({ ok: true, result: deps.media.pauseTitles() });
+        })
+    );
+
    app.post(
        "/api/media/backfill/start",
        wrap(async (req, res) => {
@@ -202,6 +202,12 @@ try {
    check("media toolbar hidden on the bot tab", !(await page.locator("#media-toolbar").isVisible()));
    check("media filter drawer hidden on the bot tab", !(await page.locator(".vk-media-drawer").isVisible()));

+    await page.waitForFunction(
+        () => document.querySelector("#title-lane")?.textContent?.includes("Titles (YouTube)"),
+        { timeout: 15000 }
+    );
+    check("title lane card renders", (await page.locator("#title-lane .vk-bot-job-head").count()) === 1);
+
    const summaryLines = await page.locator(".vk-bot-summary-line").count();
    step("backfill summary lines", summaryLines);
    check("backfill summary renders", summaryLines >= 3, summaryLines);