zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Run the title catch-up in its own YouTube lane

Browse Source
This commit is contained in:
Fisto
2026-09-17 14:25:08 +02:00
parent 8befc05591
commit 9481286169
16 changed files with 495 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CURRENT.md
+8
View File
@@ -40,6 +40,14 @@ mobile draft at 390px.
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
- [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated
`TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause
(hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no
media-traffic lock because it only calls YouTube's oEmbed. New `titleBatch` on the engine,
per-scrape `titlePending/titled/titleFailed` counters, `/api/media/titles` + start/pause
routes, and a Bot-tab card with progress, throughput, ETA, and Start/Pause. The lane
auto-starts on boot; with nothing pending it exits immediately. Preview backfill goes back to
previews only. Unit tests for `titleBatch` and a lane catch-up run; smoke covers the card.
- [x] C52 Wildcard title search. `*` now maps to a LIKE wildcard in both title modes: a leading
`*` drops the word anchor (`*car` finds *scar* and *discard*), in-token stars match between
letters (`c*r*`), and a trailing star matches the plain prefix. `%` and `_` are escaped with an
TODO.md
-7
View File
@@ -64,13 +64,6 @@ Ordered backlog. `CURRENT.md` tracks the live session state.
## Media archive follow-ups
2. **Title work in its own lane.** Title resolution talks straight to `youtube.com/oembed`
(daemon fetch; no Discord API, no bot account), but it currently shares the media-traffic lock
and the preview batch pacing, so it serializes behind walks and previews. Move it to its own
lane: skip the media-traffic lock, give it its own pacing/concurrency settings, and handle
YouTube `429`s as a cooldown rather than a normal failed try (they must not consume the tries
budget). Show the lane's throughput separately in the Bot tab.
## Web UI follow-ups
Ordered after the voice backend unless the owner asks first.
docs/media-scraper.md
+9 -6
View File
@@ -168,12 +168,15 @@ Previews run after every walk:
- The job stays `running` through the download and preview passes, so a client never sees `done`
on a half-archived scrape. A paused walk still gets these passes for the pages it collected.
- The **backfill queue** (`media-scrape/src/backfill.ts`) fills in scrapes that paused or finished
before their previews were cached, and resolves missing titles. **Titles**: YouTube-like rows
(`provider` or URL matching youtube/youtu.be) whose Discord embed carried no title are resolved
through the provider oEmbed and stored as `title` + `title_key`; failures count toward
`title_attempts` and close out as `title_error` after the tries-per-item setting, so deleted
videos stop rotating. Titles and previews share a batch: up to half of it goes to titles while
both kinds of work remain. One job per chat groups every paused, done, or failed scrape
before their previews were cached. It holds preview work only.
- The **title catch-up lane** (`media-scrape/src/titles.ts`) resolves YouTube titles for rows
indexed before titles were stored. Titles normally arrive with the scrape, so this is a one-time
job with hard-coded pacing — 100 rows per batch, 10 s between batches — and it takes **no
media-traffic lock**, because every request goes to `youtube.com/oembed`, never Discord; it runs
beside walks and the preview queue. YouTube-like rows are those whose `provider` or URL matches
youtube/youtu.be. Failures count `title_attempts` and close out as `title_error` (deleted videos
stop rotating). The Bot tab shows the lane as its own card with pending/resolved/failed counts,
throughput, an ETA, and Start/Pause. One job per chat groups every paused, done, or failed scrape
of that chat, so a paused-and-resumed scrape stays in one job. Jobs run one at a time, in
batches, and report pending/cached/failed counts, per-kind pending, throughput, and an ETA. The
Bot tab drives them: **Start backfill**, per-job Start/Pause, and **Pause all**. Jobs with the
docs/media.md
+4 -2
View File
@@ -200,8 +200,10 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
that disappears when the tab is opened and a Clear button.
- **Bot** tab: slopbot connection state, the backfill queue, per-backend stats, and the tunables.
**Backfill** shows totals (pending, cached, failed, downloaded, stored bytes), an ETA at the
current settings, and a Bot/Puppet split of pending/cached/failed. Pending includes missing
YouTube titles; each member line shows its share as `+ N titles`. One job card per chat lists its
current settings, and a Bot/Puppet split of pending/cached/failed. A separate **Titles
(YouTube)** card tracks the one-time title catch-up: pending/resolved/failed, throughput, ETA,
and Start/Pause. It talks to YouTube only (hard-coded 100 per batch, 10 s between batches) and
runs beside the preview queue. One job card per chat lists its
member scrapes (paused and resumed scrapes of the same chat are combined), per-kind pending
counts, a cached/total progress bar, a measured ETA, and **Start**/**Pause**; the section header
carries **Start backfill** and **Pause all**. A red banner reads `API timed out, increase timers`
docs/webui.md
+5 -1
View File
@@ -165,7 +165,8 @@ box live in Chat, not Call, and the connection stats are not chips yet.
the tab is reloaded. Images and GIFs render from the cached preview when one exists, then the local
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
tooltip.
- Bot shows the slopbot connection state, the backfill queue, and the media tunables. Backfill has
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
@@ -302,6 +303,9 @@ replaceable.
| GET | `/api/media/backfill` | none | Backfill jobs per chat with pending/cached/failed, per-kind pending, progress, throughput, `etaStartMs`/`etaRunMs`/`etaTotalMs`, totals with a Bot/Puppet split, the live settings, and `limitedUntil` |
| POST | `/api/media/backfill/start` | `chatId?` | Start every job with pending rows, or one chat's job |
| POST | `/api/media/backfill/pause` | `chatId?` | Pause all jobs, or one chat's job, after the current batch |
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
| POST | `/api/media/titles/start` | none | Start the title lane |
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
media-scrape/src/backfill.ts
+2 -11
View File
@@ -29,8 +29,6 @@ export interface BackfillMember {
backend: BackendName;
mode: ScrapeMode;
pending: number;
/** YouTube-like rows still missing a title. */
pendingTitles: number;
cached: number;
failed: number;
total: number;
@@ -154,7 +152,6 @@ export class BackfillQueue {
backend: meta.backend,
mode: meta.mode,
pending: 0,
pendingTitles: 0,
cached: 0,
failed: 0,
total: 0,
@@ -168,7 +165,6 @@ export class BackfillQueue {
try {
const counts = store.previewCounts();
member.pending = counts.pending;
member.pendingTitles = counts.titlePending;
member.cached = counts.cached;
member.failed = counts.failed;
member.total = counts.total;
@@ -209,10 +205,7 @@ export class BackfillQueue {
for (const [chatId, group] of groups) {
// Hide chats that are fully backfilled unless their job did work this session.
const tracked = this.jobs.get(chatId);
const pending = group.members.reduce(
(sum, member) => sum + member.pending + member.pendingTitles,
0
);
const pending = group.members.reduce((sum, member) => sum + member.pending, 0);
if (!tracked && pending === 0) continue;
const perMinute = tracked?.perMinute ?? this.estimatedPerMinute(settings, gaps);
@@ -418,9 +411,7 @@ export class BackfillQueue {
while (!job.stop) {
const group = this.plan(false).get(job.chatId);
const member = group?.members.find(
candidate => candidate.pending > 0 || candidate.pendingTitles > 0
);
const member = group?.members.find(candidate => candidate.pending > 0);
if (!member) break;
const limit = this.engine.rateLimitStatus();
media-scrape/src/engine.ts
+63 -42
View File
@@ -46,14 +46,21 @@ export interface ScrapeBackend {
fetchPage(channelId: string, before: string | null, limit: number): Promise<FetchPage>;
}
/** One batch of title resolution, as returned by `titleBatch`. */
export interface TitleBatchResult {
titled: number;
/** Rows closed out because the provider never returned a title. */
failed: number;
remaining: number;
stopped: boolean;
}
/** One batch of preview work, as returned by `previewBatch`. */
export interface PreviewBatchResult {
cached: number;
skipped: number;
failed: number;
ratelimited: number;
/** Titles resolved in this batch. */
titled: number;
/** Rows still pending after the batch. */
remaining: number;
/** True when a stop request interrupted the batch. */
@@ -550,7 +557,7 @@ export class ScrapeEngine {
const dir = join(this.options.rootDir, scrapeId);
const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
if (!meta) {
return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, titled: 0, remaining: 0, stopped: false };
return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, remaining: 0, stopped: false };
}
const releaseTraffic = await this.acquireTraffic();
@@ -563,6 +570,56 @@ export class ScrapeEngine {
}
}
/**
* Resolve YouTube titles for up to `limit` rows. No media-traffic lock: these requests go to
* YouTube's oEmbed, never Discord, so the lane can run beside a walk. The caller paces.
*/
async titleBatch(scrapeId: string, limit: number): Promise<TitleBatchResult> {
const dir = join(this.options.rootDir, scrapeId);
const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
const result: TitleBatchResult = { titled: 0, failed: 0, remaining: 0, stopped: false };
if (!meta) return result;
const store = new ScrapeStore(dir);
const options = this.previewOptions(meta);
try {
for (const row of store.pendingTitles(Math.max(1, limit))) {
if (this.stopRequested.has(meta.id)) {
meta.state = "paused";
result.stopped = true;
break;
}
const title = await resolveEmbedTitle(row.url, options);
if (title) {
store.markTitle(row.id, title);
result.titled++;
continue;
}
const attempts = store.bumpTitleAttempt(row.id);
const maxAttempts = this.options.previewMaxAttempts ?? 3;
if (attempts >= maxAttempts) {
store.markTitleError(row.id, `no title after ${attempts} attempt(s)`);
result.failed++;
this.log(
`${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})`
);
} else {
this.log(
`${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})`
);
}
}
result.remaining = store.pendingTitles(5000).length;
} finally {
store.close();
}
return result;
}
/** One batch of preview rows. Mutates the store and the meta counters. */
private async previewRows(
store: ScrapeStore,
@@ -576,47 +633,11 @@ export class ScrapeEngine {
skipped: 0,
failed: 0,
ratelimited: 0,
titled: 0,
remaining: 0,
stopped: false
};
// Title work first, then previews with what is left of the batch. YouTube rows that
// never got a title share the queue with the preview pass.
const titleBudget = store.pendingTitles(1).length > 0 && limit > 1 ? Math.ceil(limit / 2) : limit;
for (const row of store.pendingTitles(titleBudget)) {
if (this.stopRequested.has(meta.id)) {
meta.state = "paused";
result.stopped = true;
break;
}
const title = await resolveEmbedTitle(row.url, options);
if (title) {
store.markTitle(row.id, title);
result.titled++;
} else {
const attempts = store.bumpTitleAttempt(row.id);
const maxAttempts = this.options.previewMaxAttempts ?? 3;
if (attempts >= maxAttempts) {
store.markTitleError(row.id, `no title after ${attempts} attempt(s)`);
this.log(`${meta.id}: title #${row.id} gave up after ${attempts} attempts (${row.url.slice(0, 60)})`);
} else {
this.log(`${meta.id}: title #${row.id} unresolved (${attempts}/${maxAttempts}) (${row.url.slice(0, 60)})`);
}
}
this.persist(dir, meta);
await this.sleep(previewGapMs(options));
}
if (result.stopped) {
result.remaining = store.pendingPreviews(5000, true).length;
return result;
}
const previewLimit = Math.max(1, limit - result.titled);
const pending = store.pendingPreviews(previewLimit, true);
const pending = store.pendingPreviews(limit, true);
for (const row of pending) {
if (this.stopRequested.has(meta.id)) {
@@ -752,10 +773,10 @@ export class ScrapeEngine {
}
result.remaining = store.pendingPreviews(5000, true).length;
if (pending.length || result.titled) {
if (pending.length) {
this.log(
`${meta.id}: preview pass done: ${result.cached} cached, ${result.skipped} skipped, ` +
`${result.failed} failed, ${result.titled} titled, ${result.remaining} still pending`
`${result.failed} failed, ${result.remaining} still pending`
);
}
media-scrape/src/index.ts
+1
View File
@@ -37,6 +37,7 @@ export {
type BackfillStatus,
type BackfillTotals
} from "./backfill.js";
export { TitleLane, type TitleLaneOptions, type TitleLaneStatus } from "./titles.js";
export {
ScrapeEngine,
type FetchPage,
media-scrape/src/store.ts
+20 -7
View File
@@ -538,6 +538,10 @@ export class ScrapeStore {
failed: number;
/** YouTube-like rows still missing a title. */
titlePending: number;
/** Rows with a stored title. */
titled: number;
/** Rows closed out without a title. */
titleFailed: number;
byKind: Record<MediaKind, { total: number; pending: number; cached: number; failed: number }>;
} {
const empty = () => ({ total: 0, pending: 0, cached: 0, failed: 0 });
@@ -583,15 +587,24 @@ export class ScrapeStore {
const titles = this.db
.prepare(
`SELECT COUNT(*) AS count FROM media
WHERE title IS NULL AND title_error IS NULL
AND (lower(COALESCE(provider, '')) LIKE '%youtube%'
OR lower(url) LIKE '%youtube.com/%'
OR lower(url) LIKE '%youtu.be/%')`
`SELECT
SUM(title IS NULL AND title_error IS NULL
AND (lower(COALESCE(provider, '')) LIKE '%youtube%'
OR lower(url) LIKE '%youtube.com/%'
OR lower(url) LIKE '%youtu.be/%')) AS pending,
SUM(title IS NOT NULL) AS titled,
SUM(title IS NULL AND title_error IS NOT NULL) AS failed
FROM media`
)
.get() as { count: number | null } | undefined;
.get() as { pending: number | null; titled: number | null; failed: number | null } | undefined;
return { ...totals, titlePending: titles?.count ?? 0, byKind };
return {
...totals,
titlePending: titles?.pending ?? 0,
titled: titles?.titled ?? 0,
titleFailed: titles?.failed ?? 0,
byKind
};
}
/** Count one transient preview failure and return the new total. */
media-scrape/src/titles.ts
+202
View File
@@ -0,0 +1,202 @@
/*
* The YouTube title catch-up lane.
*
* Titles normally arrive with the scrape (Discord embeds carry them), so this lane only exists
* for rows indexed before titles were stored. It talks to YouTube's oEmbed directly, never to
* Discord, so it takes no media-traffic lock and can run beside a walk. Pacing is hard-coded
* (100 rows per batch, 10 s between batches): the lane is a one-time catch-up, not a tuned knob.
*/
import { ScrapeEngine } from "./engine.js";
import { listScrapes, ScrapeStore, scrapeDir } from "./store.js";
export interface TitleLaneStatus {
state: "idle" | "running" | "paused" | "done";
pending: number;
resolved: number;
failed: number;
/** Rows handled since the lane started. */
processed: number;
startedAt: string | null;
finishedAt: string | null;
perMinute: number | null;
etaMs: number | null;
/** The scrape being worked on right now. */
currentScrapeId: string | null;
}
export interface TitleLaneOptions {
rootDir: string;
log?: (line: string) => void;
now?: () => number;
sleep?: (ms: number) => Promise<void>;
/** Hard-coded in production; tests override. */
batchSize?: number;
batchPauseMs?: number;
}
const BATCH_SIZE = 100;
const BATCH_PAUSE_MS = 10_000;
interface LaneState {
state: TitleLaneStatus["state"];
stop: boolean;
startedAt: number | null;
finishedAt: number | null;
processed: number;
perMinute: number | null;
currentScrapeId: string | null;
}
const defaultSleep = (ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms));
export class TitleLane {
private readonly state: LaneState = {
state: "idle",
stop: false,
startedAt: null,
finishedAt: null,
processed: 0,
perMinute: null,
currentScrapeId: null
};
private active: Promise<void> = Promise.resolve();
constructor(
private readonly engine: ScrapeEngine,
private readonly options: TitleLaneOptions
) {}
private now(): number {
return this.options.now?.() ?? Date.now();
}
private sleep(ms: number): Promise<void> {
return (this.options.sleep ?? defaultSleep)(ms);
}
/** Totals across every scrape, with the lane's own run state. */
status(): TitleLaneStatus {
let pending = 0;
let resolved = 0;
let failed = 0;
for (const meta of listScrapes(this.options.rootDir)) {
if (meta.state === "deleted" || meta.state === "deleting" || meta.state === "pending-delete") {
continue;
}
try {
const store = new ScrapeStore(scrapeDir(this.options.rootDir, meta.id));
try {
const counts = store.previewCounts();
pending += counts.titlePending;
resolved += counts.titled ?? 0;
failed += counts.titleFailed ?? 0;
} finally {
store.close();
}
} catch {
// an unreadable index counts as nothing to do
}
}
const runMs = this.state.startedAt ? Math.max(1, this.now() - this.state.startedAt) : 0;
const perMinute =
this.state.perMinute ??
(runMs && this.state.processed ? this.state.processed / (runMs / 60_000) : null);
return {
state: this.state.state,
pending,
resolved,
failed,
processed: this.state.processed,
startedAt: this.state.startedAt ? new Date(this.state.startedAt).toISOString() : null,
finishedAt: this.state.finishedAt ? new Date(this.state.finishedAt).toISOString() : null,
perMinute: perMinute === null ? null : Math.round(perMinute * 10) / 10,
etaMs:
pending > 0 && perMinute && perMinute > 0
? Math.round((pending / perMinute) * 60_000)
: null,
currentScrapeId: this.state.currentScrapeId
};
}
start(): void {
this.state.stop = false;
this.state.state = "running";
this.state.startedAt = this.state.startedAt ?? this.now();
this.state.finishedAt = null;
this.options.log?.("titles: lane started");
this.schedule();
}
pause(): void {
if (this.state.state !== "running") return;
this.state.stop = true;
this.state.state = "paused";
this.options.log?.("titles: pausing");
}
/** Resolves when the lane has stopped for now. */
idle(): Promise<void> {
return this.active;
}
private schedule(): void {
this.active = this.active.then(async () => {
if (this.state.state !== "running") return;
await this.run();
});
}
private async run(): Promise<void> {
const batchSize = Math.max(1, this.options.batchSize ?? BATCH_SIZE);
const batchPauseMs = Math.max(0, this.options.batchPauseMs ?? BATCH_PAUSE_MS);
while (!this.state.stop) {
const meta = listScrapes(this.options.rootDir).find(candidate => {
if (candidate.state === "deleted" || candidate.state === "deleting") return false;
try {
const store = new ScrapeStore(scrapeDir(this.options.rootDir, candidate.id));
try {
return store.previewCounts().titlePending > 0;
} finally {
store.close();
}
} catch {
return false;
}
});
if (!meta) break;
this.state.currentScrapeId = meta.id;
const result = await this.engine.titleBatch(meta.id, batchSize);
this.state.processed += result.titled + result.failed;
const elapsed = Math.max(1, this.now() - (this.state.startedAt ?? this.now()));
this.state.perMinute = this.state.processed / (elapsed / 60_000);
this.options.log?.(
`titles: ${meta.id} batch done: +${result.titled} titled, ${result.failed} closed, ` +
`${result.remaining} left in scrape`
);
if (this.state.stop) break;
if (result.stopped) {
this.state.state = "paused";
break;
}
if (batchPauseMs > 0) await this.sleep(batchPauseMs);
}
this.state.currentScrapeId = null;
if (this.state.state === "running") this.state.state = this.state.stop ? "paused" : "done";
this.state.finishedAt = this.state.state === "done" ? this.now() : null;
this.options.log?.(`titles: lane ${this.state.state} (${this.state.processed} rows this run)`);
}
}
media-scrape/test/run.ts
+55 -3
View File
@@ -21,6 +21,7 @@ import {
import { ogImageFromHtml, resolveEmbedPreview } from "../src/embed-preview.js";
import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js";
import { BackfillQueue } from "../src/backfill.js";
import { TitleLane } from "../src/titles.js";
import { extractMedia, normalizeYouTubeUrl } from "../src/extract.js";
import { PuppetBackend } from "../src/puppet-backend.js";
import { betterCandidate, sortCandidates } from "../src/dedupe.js";
@@ -1502,7 +1503,7 @@ void (async () => {
store.close();
});
await test("previewBatch resolves titles and gives up after the cap", async () => {
await test("titleBatch resolves titles and gives up after the cap", async () => {
const embed = {
type: "video",
url: "https://www.youtube.com/watch?v=titlepass",
@@ -1534,7 +1535,8 @@ void (async () => {
const started = await engine.start({ channelId: "135", guildId: null, mode: "links", backend });
await engine.processQueue();
await engine.previewBatch(started.id, 5);
const batch = await engine.titleBatch(started.id, 5);
assert.equal(batch.titled, 1);
const store = new ScrapeStore(join(root, started.id));
const row = store.queryMedia({ kind: "video" })[0];
@@ -1561,7 +1563,8 @@ void (async () => {
const dead = await deadEngine.start({ channelId: "136", guildId: null, mode: "links", backend: deadBackend });
await deadEngine.processQueue();
await deadEngine.previewBatch(dead.id, 5);
const deadBatch = await deadEngine.titleBatch(dead.id, 5);
assert.equal(deadBatch.failed, 1);
const deadStore = new ScrapeStore(join(root, dead.id));
const deadRow = deadStore.queryMedia({ kind: "video" })[0];
@@ -1570,6 +1573,55 @@ void (async () => {
deadStore.close();
});
await test("title lane runs a catch-up across scrapes", async () => {
const laneRoot = join(root, "lane-root");
mkdirSync(laneRoot, { recursive: true });
const embed = (id: string) => ({
type: "video",
url: `https://www.youtube.com/watch?v=${id}`,
title: null,
provider: "YouTube",
videoUrl: `https://www.youtube.com/watch?v=${id}`,
imageUrl: null,
thumbnailUrl: null
});
const backend = new FakeBackend([
{ messages: [message("1500", { embeds: [embed("lane1"), embed("lane2")] })] }
]);
const engine = engineFor(laneRoot, backend, {
download: {
fetchImpl: async (input: any) =>
new Response(JSON.stringify({ title: `T ${String(input).length}` }), {
status: 200,
headers: { "content-type": "application/json" }
}),
minGapMs: 0,
maxGapMs: 0,
sleep: async () => {}
}
});
await engine.start({ channelId: "137", guildId: null, mode: "links", backend });
await engine.processQueue();
const lane = new TitleLane(engine, {
rootDir: laneRoot,
batchSize: 1,
batchPauseMs: 0,
sleep: async () => {},
log: () => {}
});
lane.start();
await lane.idle();
const status = lane.status();
assert.equal(status.state, "done");
assert.equal(status.pending, 0);
assert.equal(status.resolved, 2);
assert.equal(status.failed, 0);
assert.ok(status.processed >= 2);
});
await test("title search matches word prefixes and exact phrases", () => {
const dir = join(root, "title-search");
mkdirSync(dir, { recursive: true });
poc/public/media.css
+11
View File
@@ -791,3 +791,14 @@
.vk-chip-strike {
text-decoration: line-through;
}
/* The YouTube title catch-up lane, inside the Bot tab backfill section. */
.vk-bot-lane {
border: 1px dashed var(--color-neutral-700);
border-radius: 5px;
padding: 6px 8px;
margin-bottom: 8px;
display: flex;
flex-direction: column;
gap: 4px;
}
poc/public/media.js
+63 -3
View File
@@ -40,6 +40,7 @@
loadError: false,
pageSize: 60,
backfill: null,
titles: null,
preferences: null,
/** Scrapes that finished while this page was open, until the tab is reloaded. */
completed: new Set()
@@ -75,6 +76,7 @@
</span>
</div>
<div id="backfill-summary" class="vk-bot-summary"></div>
<div id="title-lane" class="vk-bot-lane"></div>
<div id="backfill-jobs" class="vk-bot-jobs"></div>
</div>
@@ -101,6 +103,7 @@
const botLimited = overlay.querySelector("#bot-limited");
const backfillSummary = overlay.querySelector("#backfill-summary");
const backfillJobs = overlay.querySelector("#backfill-jobs");
const titleLane = overlay.querySelector("#title-lane");
const tunablesHost = overlay.querySelector("#bot-tunables");
const botErrors = overlay.querySelector("#bot-errors");
@@ -790,7 +793,6 @@
"vk-bot-job-member",
`${member.scrapeId} \u00b7 ${member.state} \u00b7 ${member.mode} \u00b7 ` +
`${member.backend} \u00b7 ${member.pending} pending` +
(member.pendingTitles ? ` + ${member.pendingTitles} titles` : "") +
(kinds ? ` (${kinds})` : "")
)
);
@@ -826,6 +828,57 @@
}
}
function renderTitleLane(status) {
if (!status) return;
titleLane.replaceChildren();
const head = el("div", "vk-bot-job-head");
head.append(el("span", "vk-bot-job-title", "Titles (YouTube)"));
head.append(el("span", `vk-bot-state vk-bot-state-${status.state}`, status.state));
const parts = [`${status.pending} pending`, `${status.resolved} titles`];
if (status.failed) parts.push(`${status.failed} failed`);
if (status.perMinute) parts.push(`${status.perMinute}/min`);
if (status.pending > 0) parts.push(`ETA ${formatEta(status.etaMs)}`);
if (status.currentScrapeId) parts.push(`on ${channelLabel(status.currentScrapeId)}`);
head.append(el("span", "vk-bot-job-meta", parts.join(" \u00b7 ")));
titleLane.append(head);
const actions = el("div", "vk-bot-job-actions");
if (status.state === "running") {
actions.append(button("Pause", () => void titleAction("pause")));
} else if (status.pending > 0) {
actions.append(button("Start", () => void titleAction("start")));
}
titleLane.append(actions);
const note = el(
"div",
"vk-bot-job-member",
"One-time catch-up for rows indexed before titles were stored; talks to YouTube only, " +
"100 per batch, 10 s between batches."
);
titleLane.append(note);
}
async function titleAction(action) {
try {
state.titles = await window.VK.api.post(`/api/media/titles/${action}`, {});
renderTitleLane(state.titles);
} catch (error) {
setStatus(`titles ${action} failed: ${error.message}`);
}
}
async function refreshTitles() {
try {
state.titles = await window.VK.api.get("/api/media/titles");
} catch {
state.titles = null;
}
renderTitleLane(state.titles);
}
async function refreshBackfill() {
try {
state.backfill = await window.VK.api.get("/api/media/backfill");
@@ -1765,6 +1818,7 @@
: "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)";
if (!state.preferences) void refreshPreferences();
void refreshBackfill();
void refreshTitles();
void loadScrapes().then(renderScrapes);
void refreshErrors();
} else {
@@ -1782,7 +1836,10 @@
state.completed.clear();
void loadScrapes();
if (state.view === "media") void loadMedia(true);
if (state.view === "bot") void refreshBackfill();
if (state.view === "bot") {
void refreshBackfill();
void refreshTitles();
}
});
overlay.querySelector("#backfill-start").addEventListener("click", () => void backfillAction("start"));
@@ -1812,7 +1869,10 @@
// Backfill progress while the Bot tab is open.
setInterval(() => {
if (state.view === "bot") void refreshBackfill();
if (state.view === "bot") {
void refreshBackfill();
void refreshTitles();
}
}, 3000);
// Infinite scroll: the buffer entering the viewport starts the next page.
poc/src/media.ts
+25
View File
@@ -17,6 +17,8 @@ import {
ScrapeEngine,
ScrapeStore,
scrapeDir,
TitleLane,
type TitleLaneStatus,
snowflakeFromIso,
resolveDbRoot,
type MediaKind,
@@ -171,6 +173,7 @@ export class MediaService {
private readonly rootDir = resolveDbRoot();
private readonly engine: ScrapeEngine;
private readonly backfill: BackfillQueue;
private readonly titles: TitleLane;
private bot: BotClient | null = null;
private botFetcher: SlopbotFetcher | null = null;
@@ -214,6 +217,14 @@ export class MediaService {
log: line => this.deps.log?.(line)
});
// The title catch-up lane talks to YouTube only, so it runs beside the backfill queue.
// It auto-starts because it is a one-time catch-up: with nothing pending it exits at once.
this.titles = new TitleLane(this.engine, {
rootDir: this.rootDir,
log: line => this.deps.log?.(line)
});
this.titles.start();
const stale = this.engine.reconcileStaleJobs();
if (stale.length) {
this.deps.log?.(`paused ${stale.length} stale scrape(s) left running by a previous daemon`);
@@ -268,6 +279,20 @@ export class MediaService {
};
}
titleStatus(): TitleLaneStatus {
return this.titles.status();
}
startTitles(): TitleLaneStatus {
this.titles.start();
return this.titles.status();
}
pauseTitles(): TitleLaneStatus {
this.titles.pause();
return this.titles.status();
}
startBackfill(chatId?: string): BackfillStatus {
this.backfill.start(chatId);
return this.backfillStatus();
poc/src/webui.ts
+21
View File
@@ -835,6 +835,27 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
})
);
app.get(
"/api/media/titles",
wrap(async (_req, res) => {
res.json({ ok: true, result: deps.media.titleStatus() });
})
);
app.post(
"/api/media/titles/start",
wrap(async (_req, res) => {
res.json({ ok: true, result: deps.media.startTitles() });
})
);
app.post(
"/api/media/titles/pause",
wrap(async (_req, res) => {
res.json({ ok: true, result: deps.media.pauseTitles() });
})
);
app.post(
"/api/media/backfill/start",
wrap(async (req, res) => {
poc/test/media-ui-smoke.mjs
+6
View File
@@ -202,6 +202,12 @@ try {
check("media toolbar hidden on the bot tab", !(await page.locator("#media-toolbar").isVisible()));
check("media filter drawer hidden on the bot tab", !(await page.locator(".vk-media-drawer").isVisible()));
await page.waitForFunction(
() => document.querySelector("#title-lane")?.textContent?.includes("Titles (YouTube)"),
{ timeout: 15000 }
);
check("title lane card renders", (await page.locator("#title-lane .vk-bot-job-head").count()) === 1);
const summaryLines = await page.locator(".vk-bot-summary-line").count();
step("backfill summary lines", summaryLines);
check("backfill summary renders", summaryLines >= 3, summaryLines);