Add message links, suspect retries, and Discord preview recovery
This commit is contained in:
Fisto
+13
@@ -40,6 +40,19 @@ mobile draft at 390px.
|
||||
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
||||
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
||||
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
||||
- [x] C54 Message links, suspect retries, and preview recovery. Items now carry `messageUrl`
|
||||
(from `media_sources` + scrape guild), the viewer shows/copies it instead of the expiring CDN
|
||||
link and opens a fresh stream for View full, YouTube items offer an **Open on YouTube**
|
||||
fallback, and the metadata gained Channel/Scrape/Title (title includes the channel kept from
|
||||
oEmbed, `title_author`). Preview and download HTTP failures on Discord hosts first refresh the
|
||||
URL from the source message and retry; if that fails the row parks as a **suspect** with one
|
||||
delayed retry (50 s, runs after the batch; a batch two-thirds suspect doubles the cooldown,
|
||||
60 s floor). Still-failing rows are **unverified** and skipped by normal passes; the Bot tab
|
||||
shows suspect/unverified counts and a segmented progress bar (cached blue, pending gray,
|
||||
failed red). **Recover Discord previews** is a paused one-shot task that re-fetches every
|
||||
Discord-CDN row closed with an HTTP error across all scrapes (mixed bot/puppet via the
|
||||
existing refresh path) and lands in the task history when done. All 1,208 Discord 404 rows
|
||||
sampled 15/15 recoverable.
|
||||
- [x] C53 Title catch-up as its own lane. Titles no longer share the preview batch: a dedicated
|
||||
`TitleLane` walks every scrape with a pending title, 100 rows per batch with a 10 s pause
|
||||
(hard-coded; it is a one-time catch-up and future titles arrive with the scrape), and takes no
|
||||
|
||||
@@ -210,6 +210,25 @@ a cached still show a labeled placeholder, and playback streams only when the us
|
||||
GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
|
||||
frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.
|
||||
|
||||
## Failure policy and recovery
|
||||
|
||||
The archive never treats a transient HTTP answer as a dead link on the first try.
|
||||
|
||||
- A preview or download that fails with an HTTP status on Discord's CDN is retried once with a
|
||||
**refreshed URL** from the source message (the signed link expires in about a day). Only if the
|
||||
refreshed fetch fails is the row parked.
|
||||
- Parked rows become **suspects**: they get one delayed retry (about 50 s). The retry runs after
|
||||
the batch, not immediately. If a batch is two-thirds suspect failures, the cooldown doubles
|
||||
(floor 60 s) and the retries run before the next batch.
|
||||
- A suspect whose delayed retry fails again is left **unverified**: it is skipped by later passes,
|
||||
so a finished backfill never churns over genuinely unavailable media. The Bot tab shows suspects
|
||||
and unverified counts per member, and the progress bar fills blue for cached, gray for pending,
|
||||
and red for failed rows.
|
||||
- **Recover Discord previews** is a one-shot task in the Bot tab: it walks every scrape's
|
||||
Discord-CDN rows closed with HTTP errors, refreshes each URL through the message (slopbot REST or
|
||||
the puppet, whichever the scrape used), and fetches the preview again. It starts paused; when it
|
||||
finishes it moves to the task history. The code stays for reuse.
|
||||
|
||||
## Bot tab settings
|
||||
|
||||
The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.
|
||||
|
||||
@@ -101,6 +101,9 @@ media-scrape/db/
|
||||
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
|
||||
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
|
||||
labeled placeholder, and a failed image falls back to that placeholder.
|
||||
- Every item carries the permanent Discord message link (`messageUrl`) derived from
|
||||
`media_sources`; the viewer shows it (with a copy button) instead of the expiring CDN URL, and
|
||||
"View full" streams through the daemon so the source is refreshed automatically.
|
||||
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
|
||||
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
|
||||
filled in with `ffprobe` after the file lands.
|
||||
|
||||
+7
-1
@@ -166,7 +166,10 @@ box live in Chat, not Call, and the connection stats are not chips yet.
|
||||
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
|
||||
tooltip.
|
||||
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
|
||||
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
|
||||
catch-up lane with its own Start/Pause, the one-shot **Recover Discord previews** task, the task
|
||||
history, the failure-policy hint, and the media tunables. Media items carry their permanent
|
||||
`messageUrl`; the viewer shows it with a **Copy link** button in the link popup and offers
|
||||
**Open on YouTube** as a fallback for embeds that refuse to play. Backfill has
|
||||
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
|
||||
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
|
||||
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
|
||||
@@ -306,6 +309,9 @@ replaceable.
|
||||
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
|
||||
| POST | `/api/media/titles/start` | none | Start the title lane |
|
||||
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
|
||||
| GET | `/api/media/recover` | none | Recovery task state, remaining Discord-CDN preview failures, counts, and the task history |
|
||||
| POST | `/api/media/recover/start` | none | Start the Discord preview recovery task |
|
||||
| POST | `/api/media/recover/pause` | none | Pause the recovery task after the current row |
|
||||
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
||||
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
|
||||
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
||||
|
||||
@@ -31,6 +31,10 @@ export interface BackfillMember {
|
||||
pending: number;
|
||||
cached: number;
|
||||
failed: number;
|
||||
/** Preview failures kept for the delayed suspect retry. */
|
||||
suspect: number;
|
||||
/** Suspect rows whose retry failed too. */
|
||||
unverified: number;
|
||||
total: number;
|
||||
downloaded: number;
|
||||
bytes: number;
|
||||
@@ -154,6 +158,8 @@ export class BackfillQueue {
|
||||
pending: 0,
|
||||
cached: 0,
|
||||
failed: 0,
|
||||
suspect: 0,
|
||||
unverified: 0,
|
||||
total: 0,
|
||||
downloaded: meta.downloaded ?? 0,
|
||||
bytes: 0,
|
||||
@@ -167,6 +173,8 @@ export class BackfillQueue {
|
||||
member.pending = counts.pending;
|
||||
member.cached = counts.cached;
|
||||
member.failed = counts.failed;
|
||||
member.suspect = counts.suspect;
|
||||
member.unverified = counts.unverified;
|
||||
member.total = counts.total;
|
||||
for (const kind of Object.keys(member.pendingByKind) as MediaKind[]) {
|
||||
member.pendingByKind[kind] = counts.byKind[kind].pending;
|
||||
@@ -408,6 +416,9 @@ export class BackfillQueue {
|
||||
|
||||
private async runJob(job: Job): Promise<void> {
|
||||
let stalled = 0;
|
||||
// The cooldown doubles when a batch is mostly suspect failures; it stays raised for the
|
||||
// rest of this job so throttling is not immediately forgotten.
|
||||
let pauseMs = this.options.getSettings().batchPauseMs;
|
||||
|
||||
while (!job.stop) {
|
||||
const group = this.plan(false).get(job.chatId);
|
||||
@@ -457,8 +468,30 @@ export class BackfillQueue {
|
||||
stalled = 0;
|
||||
}
|
||||
|
||||
if (settings.batchPauseMs > 0 && progressed > 0) {
|
||||
await this.sleep(settings.batchPauseMs);
|
||||
// Suspect failures get one delayed retry. When they reach two thirds of a batch the
|
||||
// cooldown doubles (floor 60 s) and the retry runs before the next batch.
|
||||
const suspectThreshold = Math.ceil((settings.batchSize * 2) / 3);
|
||||
if (result.suspect > 0 && result.suspect >= suspectThreshold) {
|
||||
pauseMs = Math.min(300_000, Math.max(60_000, pauseMs * 2));
|
||||
this.options.log?.(
|
||||
`backfill: ${job.chatId} ${result.suspect} suspect failures, cooldown now ${pauseMs} ms`
|
||||
);
|
||||
}
|
||||
|
||||
if (pauseMs > 0 && (progressed > 0 || result.suspect > 0)) {
|
||||
await this.sleep(pauseMs);
|
||||
if (job.stop) break;
|
||||
}
|
||||
|
||||
if (result.suspect > 0 || result.unverified > 0) {
|
||||
const retried = await this.engine.previewSuspects(
|
||||
member.scrapeId,
|
||||
Math.max(2, Math.ceil(settings.batchSize / 2))
|
||||
);
|
||||
this.options.log?.(
|
||||
`backfill: ${job.chatId} suspect retry: ${retried.cached} recovered, ` +
|
||||
`${retried.unverified} left unverified`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
/*
|
||||
* Permanent Discord links. CDN URLs expire; the message link never does.
|
||||
*/
|
||||
|
||||
/** `discord.com/channels/<guildId|@me>/<channelId>/<messageId>`, or null without a message. */
|
||||
export function messageUrlFor(
|
||||
guildId: string | null,
|
||||
channelId: string,
|
||||
messageId: string | null
|
||||
): string | null {
|
||||
if (!messageId) return null;
|
||||
const scope = guildId ?? "@me";
|
||||
return `https://discord.com/channels/${scope}/${channelId}/${messageId}`;
|
||||
}
|
||||
@@ -66,7 +66,7 @@ export function ogImageFromHtml(html: string): string | null {
|
||||
export async function resolveEmbedTitle(
|
||||
url: string,
|
||||
options: DownloadOptions = {}
|
||||
): Promise<string | null> {
|
||||
): Promise<{ title: string; author: string | null } | null> {
|
||||
let parsed: URL;
|
||||
try {
|
||||
parsed = new URL(url);
|
||||
@@ -80,7 +80,12 @@ export async function resolveEmbedTitle(
|
||||
}
|
||||
|
||||
const data = await getJson(`https://www.youtube.com/oembed?url=${encodeURIComponent(url)}&format=json`, options);
|
||||
return typeof data?.title === "string" && data.title.trim() ? data.title.trim() : null;
|
||||
if (typeof data?.title !== "string" || !data.title.trim()) return null;
|
||||
|
||||
return {
|
||||
title: data.title.trim(),
|
||||
author: typeof data.author_name === "string" && data.author_name.trim() ? data.author_name.trim() : null
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+98
-12
@@ -61,6 +61,10 @@ export interface PreviewBatchResult {
|
||||
skipped: number;
|
||||
failed: number;
|
||||
ratelimited: number;
|
||||
/** Rows parked for one delayed retry (possible throttling, not confirmed gone). */
|
||||
suspect: number;
|
||||
/** Suspect rows whose delayed retry failed too. */
|
||||
unverified: number;
|
||||
/** Rows still pending after the batch. */
|
||||
remaining: number;
|
||||
/** True when a stop request interrupted the batch. */
|
||||
@@ -89,6 +93,13 @@ export interface ScrapeEngineOptions {
|
||||
rateLimitPauseMs?: number;
|
||||
/** Transient preview failures allowed before a row is marked permanently unreachable. Default 3. */
|
||||
previewMaxAttempts?: number;
|
||||
/** Fresh URL for a row whose stored URL expired (message re-fetch). Best effort. */
|
||||
refreshUrl?: (
|
||||
scrapeId: string,
|
||||
media: { id: number; url: string; playUrl: string | null }
|
||||
) => Promise<string | null>;
|
||||
/** Delay before the single suspect retry. Default 50 s. */
|
||||
suspectRetryMs?: number;
|
||||
/** Read dimensions and duration from a downloaded file. Defaults to ffprobe. */
|
||||
probe?: (
|
||||
path: string
|
||||
@@ -113,6 +124,10 @@ function oldestId(messages: CoreMessage[]): string | null {
|
||||
return oldest;
|
||||
}
|
||||
|
||||
function isDiscordHost(url: string): boolean {
|
||||
return /^https?:\/\/[^/]*discordapp\.(com|net)\//i.test(url);
|
||||
}
|
||||
|
||||
function isRateLimited(error: unknown): boolean {
|
||||
const anyError = error as { status?: number; message?: string } | undefined;
|
||||
return anyError?.status === 429 || /\b429\b|rate.?limit/i.test(String(anyError?.message ?? error));
|
||||
@@ -557,7 +572,7 @@ export class ScrapeEngine {
|
||||
const dir = join(this.options.rootDir, scrapeId);
|
||||
const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
|
||||
if (!meta) {
|
||||
return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, remaining: 0, stopped: false };
|
||||
return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, suspect: 0, unverified: 0, remaining: 0, stopped: false };
|
||||
}
|
||||
|
||||
const releaseTraffic = await this.acquireTraffic();
|
||||
@@ -590,9 +605,9 @@ export class ScrapeEngine {
|
||||
break;
|
||||
}
|
||||
|
||||
const title = await resolveEmbedTitle(row.url, options);
|
||||
if (title) {
|
||||
store.markTitle(row.id, title);
|
||||
const resolved = await resolveEmbedTitle(row.url, options);
|
||||
if (resolved) {
|
||||
store.markTitle(row.id, resolved.title, resolved.author);
|
||||
result.titled++;
|
||||
continue;
|
||||
}
|
||||
@@ -620,24 +635,47 @@ export class ScrapeEngine {
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Retry the due suspect rows (one delayed attempt each) for one scrape. */
|
||||
async previewSuspects(scrapeId: string, limit: number): Promise<PreviewBatchResult> {
|
||||
const dir = join(this.options.rootDir, scrapeId);
|
||||
const meta = this.active.get(scrapeId) ?? readScrapeMeta(dir);
|
||||
if (!meta) {
|
||||
return { cached: 0, skipped: 0, failed: 0, ratelimited: 0, suspect: 0, unverified: 0, remaining: 0, stopped: false };
|
||||
}
|
||||
|
||||
const releaseTraffic = await this.acquireTraffic();
|
||||
const store = new ScrapeStore(dir);
|
||||
try {
|
||||
return await this.previewRows(store, dir, meta, Math.max(1, limit), this.previewOptions(meta), true);
|
||||
} finally {
|
||||
store.close();
|
||||
releaseTraffic();
|
||||
}
|
||||
}
|
||||
|
||||
/** One batch of preview rows. Mutates the store and the meta counters. */
|
||||
private async previewRows(
|
||||
store: ScrapeStore,
|
||||
dir: string,
|
||||
meta: ScrapeMeta,
|
||||
limit: number,
|
||||
options: DownloadOptions & { embedPreviews: boolean; destinationDir: string | null }
|
||||
options: DownloadOptions & { embedPreviews: boolean; destinationDir: string | null },
|
||||
retryingSuspects = false
|
||||
): Promise<PreviewBatchResult> {
|
||||
const result: PreviewBatchResult = {
|
||||
cached: 0,
|
||||
skipped: 0,
|
||||
failed: 0,
|
||||
ratelimited: 0,
|
||||
suspect: 0,
|
||||
unverified: 0,
|
||||
remaining: 0,
|
||||
stopped: false
|
||||
};
|
||||
|
||||
const pending = store.pendingPreviews(limit, true);
|
||||
const pending = retryingSuspects
|
||||
? store.dueSuspects(limit, this.now())
|
||||
: store.pendingPreviews(limit, true);
|
||||
|
||||
for (const row of pending) {
|
||||
if (this.stopRequested.has(meta.id)) {
|
||||
@@ -704,6 +742,7 @@ export class ScrapeEngine {
|
||||
}
|
||||
|
||||
store.markPreview(row.id, { hash, bytes, ext, animHash });
|
||||
store.clearSuspect(row.id);
|
||||
meta.previewed = (meta.previewed ?? 0) + 1;
|
||||
result.cached++;
|
||||
|
||||
@@ -723,6 +762,7 @@ export class ScrapeEngine {
|
||||
const frame = await remoteFrame(previewRow.preview_url ?? previewRow.url, join(dir, "thumbs"));
|
||||
if (frame) {
|
||||
store.markPreview(row.id, { hash: frame.hash, bytes: frame.bytes, ext: "webp" });
|
||||
store.clearSuspect(row.id);
|
||||
meta.previewed = (meta.previewed ?? 0) + 1;
|
||||
result.cached++;
|
||||
} else {
|
||||
@@ -760,12 +800,58 @@ export class ScrapeEngine {
|
||||
);
|
||||
}
|
||||
} else {
|
||||
store.markPreviewError(row.id, outcome.error ?? outcome.state);
|
||||
result.skipped++;
|
||||
this.log(
|
||||
`${meta.id}: preview #${row.id} ${row.kind}/${row.provider ?? "?"} skipped: ` +
|
||||
`${outcome.error ?? outcome.state} (${row.url.slice(0, 60)})`
|
||||
);
|
||||
const error = outcome.error ?? outcome.state;
|
||||
const httpError = /^http \d/.test(error);
|
||||
|
||||
if (httpError && !retryingSuspects && this.options.refreshUrl && isDiscordHost(row.url)) {
|
||||
// The signed CDN URL may have expired. Refresh it from the source message,
|
||||
// persist the fresh one, and try once more before calling it gone.
|
||||
const refreshed = await this.options.refreshUrl(meta.id, {
|
||||
id: row.id,
|
||||
url: row.url,
|
||||
playUrl: row.play_url
|
||||
});
|
||||
if (refreshed && refreshed !== previewRow.url) {
|
||||
const retry = await downloadPreview({ ...previewRow, url: refreshed }, join(dir, "thumbs"), options);
|
||||
if (retry.state === "downloaded" && retry.hash) {
|
||||
store.markPreview(row.id, {
|
||||
hash: retry.hash,
|
||||
bytes: retry.bytes ?? 0,
|
||||
ext: retry.ext ?? "webp"
|
||||
});
|
||||
store.clearSuspect(row.id);
|
||||
meta.previewed = (meta.previewed ?? 0) + 1;
|
||||
result.cached++;
|
||||
this.persist(dir, meta);
|
||||
await this.sleep(previewGapMs(options));
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (httpError) {
|
||||
if (retryingSuspects || row.preview_suspect) {
|
||||
store.confirmSuspect(row.id, `still ${error} after retry`);
|
||||
result.unverified++;
|
||||
this.log(
|
||||
`${meta.id}: preview #${row.id} still ${error} after retry, left unverified (${row.url.slice(0, 60)})`
|
||||
);
|
||||
} else {
|
||||
const retryAt = this.now() + (this.options.suspectRetryMs ?? 50_000);
|
||||
store.markSuspect(row.id, error, retryAt);
|
||||
result.suspect++;
|
||||
this.log(
|
||||
`${meta.id}: preview #${row.id} suspect (${error}), retrying after a pause (${row.url.slice(0, 60)})`
|
||||
);
|
||||
}
|
||||
} else {
|
||||
store.markPreviewError(row.id, error);
|
||||
result.skipped++;
|
||||
this.log(
|
||||
`${meta.id}: preview #${row.id} ${row.kind}/${row.provider ?? "?"} skipped: ` +
|
||||
`${error} (${row.url.slice(0, 60)})`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
this.persist(dir, meta);
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
|
||||
export { extractMedia, type ExtractedMedia } from "./extract.js";
|
||||
export { betterCandidate, candidateScore, sortCandidates, type DedupeCandidate } from "./dedupe.js";
|
||||
export { messageUrlFor } from "./discord-links.js";
|
||||
export { ogImageFromHtml, resolveEmbedPreview, resolveEmbedTitle } from "./embed-preview.js";
|
||||
export { isNearLimit, planRetention, type RetentionPlan, type RetentionScrape } from "./retention.js";
|
||||
export { isoFromSnowflake, snowflakeFromIso } from "./snowflake.js";
|
||||
|
||||
+119
-6
@@ -169,10 +169,18 @@ export interface MediaRow {
|
||||
download_error: string | null;
|
||||
preview_error: string | null;
|
||||
preview_attempts: number;
|
||||
/** First message that carried this media; null on rows written before linking. */
|
||||
message_id: string | null;
|
||||
/** Embed title from Discord (YouTube and friends). */
|
||||
title: string | null;
|
||||
title_attempts: number;
|
||||
title_error: string | null;
|
||||
/** YouTube channel name from oEmbed, kept even if the video later disappears. */
|
||||
title_author: string | null;
|
||||
/** Last suspect (possibly throttled) preview failure; retried once, then left alone. */
|
||||
preview_suspect: string | null;
|
||||
/** Epoch ms when the suspect row may be retried. Null once confirmed unavailable. */
|
||||
preview_retry_at: number | null;
|
||||
/** Lowercased, punctuation-collapsed title for token-prefix search. */
|
||||
title_key: string | null;
|
||||
created_at: string;
|
||||
@@ -242,6 +250,9 @@ CREATE TABLE IF NOT EXISTS media (
|
||||
title_key TEXT,
|
||||
title_attempts INTEGER NOT NULL DEFAULT 0,
|
||||
title_error TEXT,
|
||||
title_author TEXT,
|
||||
preview_suspect TEXT,
|
||||
preview_retry_at INTEGER,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
@@ -295,6 +306,11 @@ export class ScrapeStore {
|
||||
this.db.exec("ALTER TABLE media ADD COLUMN title_attempts INTEGER NOT NULL DEFAULT 0");
|
||||
this.db.exec("ALTER TABLE media ADD COLUMN title_error TEXT");
|
||||
}
|
||||
if (!names.has("title_author")) this.db.exec("ALTER TABLE media ADD COLUMN title_author TEXT");
|
||||
if (!names.has("preview_suspect")) {
|
||||
this.db.exec("ALTER TABLE media ADD COLUMN preview_suspect TEXT");
|
||||
this.db.exec("ALTER TABLE media ADD COLUMN preview_retry_at INTEGER");
|
||||
}
|
||||
|
||||
this.migrateYouTubeShorts();
|
||||
}
|
||||
@@ -499,6 +515,88 @@ export class ScrapeStore {
|
||||
* Rows that still need a cached preview. Attachment/sticker images and GIFs always;
|
||||
* embed thumbnails only when `includeEmbeds` is set (download mode).
|
||||
*/
|
||||
/**
|
||||
* Rows whose preview failed in a way that may be throttling rather than a dead link. Each is
|
||||
* retried once after a delay; confirmed rows stay here and are skipped by normal passes, so a
|
||||
* finished backfill never churns over genuinely unavailable media.
|
||||
*/
|
||||
dueSuspects(limit = 500, nowMs = Date.now()): MediaRow[] {
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT * FROM media
|
||||
WHERE preview_hash IS NULL AND preview_error IS NULL
|
||||
AND preview_suspect IS NOT NULL
|
||||
AND preview_retry_at IS NOT NULL
|
||||
AND preview_retry_at <= ?
|
||||
ORDER BY preview_retry_at LIMIT ?`
|
||||
)
|
||||
.all(nowMs, limit) as unknown as MediaRow[];
|
||||
}
|
||||
|
||||
/** Suspect rows that still have a retry coming. */
|
||||
suspectCount(): number {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
"SELECT COUNT(*) AS count FROM media WHERE preview_hash IS NULL AND preview_suspect IS NOT NULL AND preview_retry_at IS NOT NULL"
|
||||
)
|
||||
.get() as { count: number | null } | undefined;
|
||||
return row?.count ?? 0;
|
||||
}
|
||||
|
||||
/** Suspect rows whose single retry failed too; skipped by normal passes. */
|
||||
unverifiedCount(): number {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
"SELECT COUNT(*) AS count FROM media WHERE preview_hash IS NULL AND preview_suspect IS NOT NULL AND preview_retry_at IS NULL"
|
||||
)
|
||||
.get() as { count: number | null } | undefined;
|
||||
return row?.count ?? 0;
|
||||
}
|
||||
|
||||
/** Schedule the single delayed retry for a suspect row. */
|
||||
markSuspect(id: number, error: string, retryAtMs: number): void {
|
||||
this.db
|
||||
.prepare("UPDATE media SET preview_suspect = ?, preview_retry_at = ? WHERE id = ?")
|
||||
.run(error.slice(0, 200), retryAtMs, id);
|
||||
}
|
||||
|
||||
/** The delayed retry failed too: keep the note but stop retrying. */
|
||||
confirmSuspect(id: number, error: string): void {
|
||||
this.db
|
||||
.prepare("UPDATE media SET preview_suspect = ?, preview_retry_at = NULL WHERE id = ?")
|
||||
.run(error.slice(0, 200), id);
|
||||
}
|
||||
|
||||
/** The retry succeeded or the preview arrived another way. */
|
||||
clearSuspect(id: number): void {
|
||||
this.db.prepare("UPDATE media SET preview_suspect = NULL, preview_retry_at = NULL WHERE id = ?").run(id);
|
||||
}
|
||||
|
||||
/** Discord-CDN rows closed out with an HTTP error; the recovery task retries these. */
|
||||
discordPreviewFailures(limit = 5000): MediaRow[] {
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT * FROM media
|
||||
WHERE preview_hash IS NULL AND preview_error IS NOT NULL
|
||||
AND lower(url) LIKE '%discordapp%'
|
||||
ORDER BY id LIMIT ?`
|
||||
)
|
||||
.all(limit) as unknown as MediaRow[];
|
||||
}
|
||||
|
||||
clearPreviewError(id: number): void {
|
||||
this.db.prepare("UPDATE media SET preview_error = NULL WHERE id = ?").run(id);
|
||||
}
|
||||
|
||||
discordPreviewFailureCount(): number {
|
||||
const row = this.db
|
||||
.prepare(
|
||||
"SELECT COUNT(*) AS count FROM media WHERE preview_hash IS NULL AND preview_error IS NOT NULL AND lower(url) LIKE '%discordapp%'"
|
||||
)
|
||||
.get() as { count: number | null } | undefined;
|
||||
return row?.count ?? 0;
|
||||
}
|
||||
|
||||
pendingPreviews(limit = 5000, includeEmbeds = false): MediaRow[] {
|
||||
const local =
|
||||
"((source IS NULL OR source != 'embed') AND kind IN ('image', 'gif', 'video', 'audio'))";
|
||||
@@ -508,7 +606,10 @@ export class ScrapeStore {
|
||||
|
||||
return this.db
|
||||
.prepare(
|
||||
`SELECT * FROM media WHERE ${condition} AND preview_hash IS NULL AND preview_error IS NULL ORDER BY id LIMIT ?`
|
||||
`SELECT * FROM media
|
||||
WHERE ${condition} AND preview_hash IS NULL AND preview_error IS NULL
|
||||
AND preview_suspect IS NULL
|
||||
ORDER BY id LIMIT ?`
|
||||
)
|
||||
.all(limit) as unknown as MediaRow[];
|
||||
}
|
||||
@@ -542,6 +643,10 @@ export class ScrapeStore {
|
||||
titled: number;
|
||||
/** Rows closed out without a title. */
|
||||
titleFailed: number;
|
||||
/** Preview failures kept for one delayed retry. */
|
||||
suspect: number;
|
||||
/** Suspect rows whose retry failed too. */
|
||||
unverified: number;
|
||||
byKind: Record<MediaKind, { total: number; pending: number; cached: number; failed: number }>;
|
||||
} {
|
||||
const empty = () => ({ total: 0, pending: 0, cached: 0, failed: 0 });
|
||||
@@ -603,6 +708,8 @@ export class ScrapeStore {
|
||||
titlePending: titles?.pending ?? 0,
|
||||
titled: titles?.titled ?? 0,
|
||||
titleFailed: titles?.failed ?? 0,
|
||||
suspect: this.suspectCount(),
|
||||
unverified: this.unverifiedCount(),
|
||||
byKind
|
||||
};
|
||||
}
|
||||
@@ -625,11 +732,13 @@ export class ScrapeStore {
|
||||
.all(limit) as unknown as MediaRow[];
|
||||
}
|
||||
|
||||
/** Store a resolved title and its search key. */
|
||||
markTitle(id: number, title: string): void {
|
||||
/** Store a resolved title, its search key, and the provider's channel name. */
|
||||
markTitle(id: number, title: string, author?: string | null): void {
|
||||
this.db
|
||||
.prepare("UPDATE media SET title = ?, title_key = ?, title_error = NULL WHERE id = ?")
|
||||
.run(title.slice(0, 300), normalizeTitleKey(title), id);
|
||||
.prepare(
|
||||
"UPDATE media SET title = ?, title_key = ?, title_error = NULL, title_author = COALESCE(?, title_author) WHERE id = ?"
|
||||
)
|
||||
.run(title.slice(0, 300), normalizeTitleKey(title), author ?? null, id);
|
||||
}
|
||||
|
||||
/** Count one transient title failure and return the new total. */
|
||||
@@ -873,7 +982,11 @@ export class ScrapeStore {
|
||||
const { where, values } = this.mediaWhere(options);
|
||||
|
||||
const rows = this.db
|
||||
.prepare(`SELECT * FROM media ${where} ORDER BY id DESC LIMIT ? OFFSET ?`)
|
||||
.prepare(
|
||||
`SELECT media.*,
|
||||
(SELECT message_id FROM media_sources WHERE media_id = media.id LIMIT 1) AS message_id
|
||||
FROM media ${where} ORDER BY id DESC LIMIT ? OFFSET ?`
|
||||
)
|
||||
.all(...values, limit, offset);
|
||||
|
||||
return rows as unknown as MediaRow[];
|
||||
|
||||
@@ -24,6 +24,7 @@ import { BackfillQueue } from "../src/backfill.js";
|
||||
import { TitleLane } from "../src/titles.js";
|
||||
import { extractMedia, normalizeYouTubeUrl } from "../src/extract.js";
|
||||
import { PuppetBackend } from "../src/puppet-backend.js";
|
||||
import { messageUrlFor } from "../src/discord-links.js";
|
||||
import { betterCandidate, sortCandidates } from "../src/dedupe.js";
|
||||
import { aspectCondition, PROVIDER_FILTERS, resolutionCondition } from "../src/filter-presets.js";
|
||||
import { isNearLimit, planRetention } from "../src/retention.js";
|
||||
@@ -1436,6 +1437,116 @@ void (async () => {
|
||||
reopened.close();
|
||||
});
|
||||
|
||||
await test("message links build for guilds and DMs", () => {
|
||||
assert.equal(messageUrlFor("123", "456", "789"), "https://discord.com/channels/123/456/789");
|
||||
assert.equal(messageUrlFor(null, "456", "789"), "https://discord.com/channels/@me/456/789");
|
||||
assert.equal(messageUrlFor("123", "456", null), null);
|
||||
});
|
||||
|
||||
await test("suspect rows park, due, and confirm", () => {
|
||||
const dir = join(root, "suspects");
|
||||
mkdirSync(dir, { recursive: true });
|
||||
const store = new ScrapeStore(dir);
|
||||
store.insertPage([
|
||||
message("1700", {
|
||||
attachments: [attachment({ filename: "gone.mp4", contentType: "video/mp4" })]
|
||||
})
|
||||
]);
|
||||
|
||||
const row = store.queryMedia({ kind: "video" })[0];
|
||||
const now = Date.now();
|
||||
store.markSuspect(row.id, "http 404", now + 10_000);
|
||||
|
||||
assert.equal(store.previewCounts().suspect, 1);
|
||||
assert.equal(store.previewCounts().unverified, 0);
|
||||
assert.equal(store.pendingPreviews(10, true).length, 0);
|
||||
assert.equal(store.dueSuspects(10, now).length, 0);
|
||||
assert.equal(store.dueSuspects(10, now + 20_000).length, 1);
|
||||
|
||||
store.confirmSuspect(row.id, "still http 404 after retry");
|
||||
assert.equal(store.previewCounts().suspect, 0);
|
||||
assert.equal(store.previewCounts().unverified, 1);
|
||||
assert.equal(store.dueSuspects(10, now + 20_000).length, 0);
|
||||
|
||||
store.markSuspect(row.id, "http 503", now);
|
||||
assert.equal(store.previewCounts().suspect, 1);
|
||||
assert.equal(store.previewCounts().unverified, 0);
|
||||
assert.equal(store.dueSuspects(10, now).length, 1);
|
||||
store.close();
|
||||
});
|
||||
|
||||
await test("discord 404s become suspects and the delayed retry resolves them", async () => {
|
||||
let failing = true;
|
||||
const backend = new FakeBackend([
|
||||
{
|
||||
messages: [
|
||||
message("1701", {
|
||||
attachments: [attachment({ filename: "clip.mp4", contentType: "video/mp4" })]
|
||||
})
|
||||
]
|
||||
}
|
||||
]);
|
||||
const engine = engineFor(root, backend, {
|
||||
suspectRetryMs: 0,
|
||||
download: {
|
||||
fetchImpl: async () =>
|
||||
failing
|
||||
? new Response("gone", { status: 404 })
|
||||
: new Response("png", { status: 200, headers: { "content-type": "image/png" } }),
|
||||
minGapMs: 0,
|
||||
maxGapMs: 0,
|
||||
sleep: async () => {}
|
||||
}
|
||||
});
|
||||
|
||||
const started = await engine.start({ channelId: "138", guildId: null, mode: "links", backend });
|
||||
await engine.processQueue();
|
||||
|
||||
const first = await engine.previewBatch(started.id, 5);
|
||||
assert.equal(first.suspect, 1);
|
||||
assert.equal(first.skipped, 0);
|
||||
|
||||
// Normal passes leave suspects alone.
|
||||
assert.equal((await engine.previewBatch(started.id, 5)).suspect, 0);
|
||||
|
||||
failing = false;
|
||||
const retry = await engine.previewSuspects(started.id, 5);
|
||||
assert.equal(retry.cached, 1);
|
||||
|
||||
const store = new ScrapeStore(join(root, started.id));
|
||||
assert.equal(store.previewCounts().cached, 1);
|
||||
assert.equal(store.previewCounts().suspect, 0);
|
||||
store.close();
|
||||
|
||||
// A retry that fails again is confirmed unverified.
|
||||
const dead = new FakeBackend([
|
||||
{
|
||||
messages: [
|
||||
message("1702", {
|
||||
attachments: [attachment({ filename: "dead.mp4", contentType: "video/mp4" })]
|
||||
})
|
||||
]
|
||||
}
|
||||
]);
|
||||
const deadEngine = engineFor(root, dead, {
|
||||
suspectRetryMs: 0,
|
||||
download: {
|
||||
fetchImpl: async () => new Response("gone", { status: 404 }),
|
||||
minGapMs: 0,
|
||||
maxGapMs: 0,
|
||||
sleep: async () => {}
|
||||
}
|
||||
});
|
||||
const deadScrape = await deadEngine.start({ channelId: "139", guildId: null, mode: "links", backend: dead });
|
||||
await deadEngine.processQueue();
|
||||
assert.equal((await deadEngine.previewBatch(deadScrape.id, 5)).suspect, 1);
|
||||
assert.equal((await deadEngine.previewSuspects(deadScrape.id, 5)).unverified, 1);
|
||||
const deadStore = new ScrapeStore(join(root, deadScrape.id));
|
||||
assert.equal(deadStore.previewCounts().unverified, 1);
|
||||
assert.equal(deadStore.pendingPreviews(10, true).length, 0);
|
||||
deadStore.close();
|
||||
});
|
||||
|
||||
await test("pendingTitles selects YouTube rows without a title", () => {
|
||||
const dir = join(root, "pending-titles");
|
||||
mkdirSync(dir, { recursive: true });
|
||||
|
||||
@@ -802,3 +802,40 @@
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.vk-bot-progress {
|
||||
display: flex;
|
||||
}
|
||||
|
||||
.vk-bot-progress-cached {
|
||||
background: var(--color-accent-400, #4f8cff);
|
||||
}
|
||||
|
||||
.vk-bot-progress-pending {
|
||||
background: var(--color-neutral-700);
|
||||
}
|
||||
|
||||
.vk-bot-progress-failed {
|
||||
background: #c04040;
|
||||
}
|
||||
|
||||
.vk-bot-history {
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.vk-bot-history-title {
|
||||
font: 600 11px/1.6 ui-monospace, Menlo, monospace;
|
||||
color: var(--color-neutral-300);
|
||||
margin-bottom: 2px;
|
||||
}
|
||||
|
||||
.vk-bot-history-entry {
|
||||
font: 400 10px/1.5 ui-monospace, Menlo, monospace;
|
||||
color: var(--color-neutral-500);
|
||||
}
|
||||
|
||||
.vk-bot-hint {
|
||||
margin-top: 8px;
|
||||
font: 400 10px/1.5 ui-monospace, Menlo, monospace;
|
||||
color: var(--color-neutral-500);
|
||||
}
|
||||
|
||||
+162
-12
@@ -41,6 +41,7 @@
|
||||
pageSize: 60,
|
||||
backfill: null,
|
||||
titles: null,
|
||||
recover: null,
|
||||
preferences: null,
|
||||
/** Scrapes that finished while this page was open, until the tab is reloaded. */
|
||||
completed: new Set()
|
||||
@@ -77,7 +78,16 @@
|
||||
</div>
|
||||
<div id="backfill-summary" class="vk-bot-summary"></div>
|
||||
<div id="title-lane" class="vk-bot-lane"></div>
|
||||
<div id="recover-lane" class="vk-bot-lane"></div>
|
||||
<div id="backfill-jobs" class="vk-bot-jobs"></div>
|
||||
<div id="task-history" class="vk-bot-history"></div>
|
||||
<div class="vk-bot-hint">
|
||||
Failure policy: a failed preview gets one delayed retry (about 50 s). If a
|
||||
batch is two-thirds suspect failures, the cooldown doubles (60 s floor) and
|
||||
the retries run before the next batch. Rows that still fail are left
|
||||
unverified and skipped by later passes; the recovery task above retries
|
||||
Discord-CDN ones on demand.
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="vk-bot-section">
|
||||
@@ -104,6 +114,8 @@
|
||||
const backfillSummary = overlay.querySelector("#backfill-summary");
|
||||
const backfillJobs = overlay.querySelector("#backfill-jobs");
|
||||
const titleLane = overlay.querySelector("#title-lane");
|
||||
const recoverLane = overlay.querySelector("#recover-lane");
|
||||
const taskHistory = overlay.querySelector("#task-history");
|
||||
const tunablesHost = overlay.querySelector("#bot-tunables");
|
||||
const botErrors = overlay.querySelector("#bot-errors");
|
||||
|
||||
@@ -793,15 +805,24 @@
|
||||
"vk-bot-job-member",
|
||||
`${member.scrapeId} \u00b7 ${member.state} \u00b7 ${member.mode} \u00b7 ` +
|
||||
`${member.backend} \u00b7 ${member.pending} pending` +
|
||||
(member.suspect ? ` + ${member.suspect} suspect` : "") +
|
||||
(member.unverified ? ` + ${member.unverified} unverified` : "") +
|
||||
(kinds ? ` (${kinds})` : "")
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
const total = Math.max(1, job.total);
|
||||
const progress = el("div", "vk-bot-progress");
|
||||
const fill = el("div", "vk-bot-progress-fill");
|
||||
fill.style.width = `${Math.min(100, Math.round((job.cached / Math.max(1, job.total)) * 100))}%`;
|
||||
progress.append(fill);
|
||||
for (const [className, value] of [
|
||||
["vk-bot-progress-cached", job.cached],
|
||||
["vk-bot-progress-pending", job.pending + (job.suspect ?? 0)],
|
||||
["vk-bot-progress-failed", job.failed]
|
||||
]) {
|
||||
const segment = el("div", className);
|
||||
segment.style.width = `${Math.min(100, (value / total) * 100)}%`;
|
||||
progress.append(segment);
|
||||
}
|
||||
row.append(progress);
|
||||
|
||||
const actions = el("div", "vk-bot-job-actions");
|
||||
@@ -879,6 +900,76 @@
|
||||
renderTitleLane(state.titles);
|
||||
}
|
||||
|
||||
function renderRecover(status) {
|
||||
if (!status) return;
|
||||
recoverLane.replaceChildren();
|
||||
|
||||
const head = el("div", "vk-bot-job-head");
|
||||
head.append(el("span", "vk-bot-job-title", "Recover Discord previews"));
|
||||
head.append(el("span", `vk-bot-state vk-bot-state-${status.state}`, status.state));
|
||||
|
||||
const parts = [`${status.total} to retry`];
|
||||
if (status.processed) parts.push(`${status.processed} processed`);
|
||||
if (status.recovered) parts.push(`${status.recovered} recovered`);
|
||||
if (status.gone) parts.push(`${status.gone} unavailable`);
|
||||
if (status.currentScrapeId) parts.push(`on ${channelLabel(status.currentScrapeId)}`);
|
||||
head.append(el("span", "vk-bot-job-meta", parts.join(" \u00b7 ")));
|
||||
recoverLane.append(head);
|
||||
|
||||
const actions = el("div", "vk-bot-job-actions");
|
||||
if (status.state === "running") {
|
||||
actions.append(button("Pause", () => void recoverAction("pause")));
|
||||
} else if (status.total > 0) {
|
||||
actions.append(button("Start", () => void recoverAction("start")));
|
||||
}
|
||||
recoverLane.append(actions);
|
||||
recoverLane.append(
|
||||
el(
|
||||
"div",
|
||||
"vk-bot-job-member",
|
||||
"Re-fetches previews for Discord CDN rows closed as gone, refreshing each signed " +
|
||||
"URL through the source message (bot or puppet). Runs once; the result lands in " +
|
||||
"the history below."
|
||||
)
|
||||
);
|
||||
|
||||
renderTaskHistory(status.history ?? []);
|
||||
}
|
||||
|
||||
function renderTaskHistory(history) {
|
||||
taskHistory.replaceChildren();
|
||||
if (!history.length) return;
|
||||
|
||||
taskHistory.append(el("div", "vk-bot-history-title", "Task history"));
|
||||
for (const entry of history) {
|
||||
taskHistory.append(
|
||||
el(
|
||||
"div",
|
||||
"vk-bot-history-entry",
|
||||
`${new Date(entry.at).toLocaleString()} \u00b7 ${entry.name} \u00b7 ${entry.detail}`
|
||||
)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function recoverAction(action) {
|
||||
try {
|
||||
state.recover = await window.VK.api.post(`/api/media/recover/${action}`, {});
|
||||
renderRecover(state.recover);
|
||||
} catch (error) {
|
||||
setStatus(`recover ${action} failed: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function refreshRecover() {
|
||||
try {
|
||||
state.recover = await window.VK.api.get("/api/media/recover");
|
||||
} catch {
|
||||
state.recover = null;
|
||||
}
|
||||
renderRecover(state.recover);
|
||||
}
|
||||
|
||||
async function refreshBackfill() {
|
||||
try {
|
||||
state.backfill = await window.VK.api.get("/api/media/backfill");
|
||||
@@ -1043,7 +1134,10 @@
|
||||
["date", "Date"],
|
||||
["source", "Source"],
|
||||
["resolution", "Resolution"],
|
||||
["length", "Length"]
|
||||
["length", "Length"],
|
||||
["channel", "Channel"],
|
||||
["scrape", "Scrape"],
|
||||
["title", "Title"]
|
||||
];
|
||||
|
||||
/** "mp4 video", "webp image", "YouTube video", ... */
|
||||
@@ -1062,7 +1156,14 @@
|
||||
date: item.postedAt ? new Date(item.postedAt).toLocaleString() : "—",
|
||||
source: item.provider ?? "discord",
|
||||
resolution: item.width && item.height ? `${item.width}x${item.height}` : "—",
|
||||
length: item.durationSecs ? `${item.durationSecs.toFixed(1)}s` : "—"
|
||||
length: item.durationSecs ? `${item.durationSecs.toFixed(1)}s` : "—",
|
||||
channel: channelLabel(item.channelId),
|
||||
scrape: item.scrapeId,
|
||||
title: item.title
|
||||
? item.titleAuthor
|
||||
? `${item.title} — ${item.titleAuthor}`
|
||||
: item.title
|
||||
: ""
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1819,6 +1920,7 @@
|
||||
if (!state.preferences) void refreshPreferences();
|
||||
void refreshBackfill();
|
||||
void refreshTitles();
|
||||
void refreshRecover();
|
||||
void loadScrapes().then(renderScrapes);
|
||||
void refreshErrors();
|
||||
} else {
|
||||
@@ -1839,6 +1941,7 @@
|
||||
if (state.view === "bot") {
|
||||
void refreshBackfill();
|
||||
void refreshTitles();
|
||||
void refreshRecover();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -1872,6 +1975,7 @@
|
||||
if (state.view === "bot") {
|
||||
void refreshBackfill();
|
||||
void refreshTitles();
|
||||
void refreshRecover();
|
||||
}
|
||||
}, 3000);
|
||||
|
||||
@@ -2290,6 +2394,22 @@
|
||||
|
||||
// ---- Media viewer ----
|
||||
|
||||
/** The canonical watch URL for YouTube items, for the fallback button. */
|
||||
function youtubeWatchUrl(item) {
|
||||
try {
|
||||
const url = new URL(item.url);
|
||||
const host = url.hostname.replace(/^www\./, "");
|
||||
if (host === "youtube.com" || host === "m.youtube.com") {
|
||||
const id = url.searchParams.get("v") ?? /^\/shorts\/([A-Za-z0-9_-]+)/.exec(url.pathname)?.[1];
|
||||
return id ? `https://www.youtube.com/watch?v=${id}` : null;
|
||||
}
|
||||
if (host === "youtu.be") return `https://www.youtube.com/watch${url.pathname}`;
|
||||
} catch {
|
||||
// not a URL
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function embedUrlFor(item) {
|
||||
if (item.source !== "embed") return null;
|
||||
|
||||
@@ -2396,26 +2516,38 @@
|
||||
meta.replaceChildren();
|
||||
const values = metaValues(item);
|
||||
for (const [key, label] of META_FIELDS) {
|
||||
if (!values[key]) continue;
|
||||
const line = el("div", "vk-media-meta-line");
|
||||
line.append(el("span", "vk-media-meta-key", `${label}: `));
|
||||
line.append(el("span", null, values[key]));
|
||||
meta.append(line);
|
||||
}
|
||||
|
||||
const link = el("button", "vk-media-link", item.url);
|
||||
// The permanent message link, not the expiring CDN URL.
|
||||
const linkUrl = item.messageUrl ?? item.url;
|
||||
const link = el("button", "vk-media-link", linkUrl);
|
||||
link.type = "button";
|
||||
link.title = "Open the original link";
|
||||
link.addEventListener("click", () => openLinkWarning(item.url));
|
||||
link.title = item.messageUrl ? "Open the message" : "Open the original link";
|
||||
link.addEventListener("click", () => openLinkWarning(linkUrl));
|
||||
meta.append(link);
|
||||
|
||||
// "View full" opens the embedded media (player, local file, or a refreshed stream).
|
||||
// "View full" plays in a new tab: the stream endpoint for Discord media (always a fresh
|
||||
// source), the provider player for embeddable items.
|
||||
const streamPath = `/api/media/${item.scrapeId}/${item.itemId}/stream`;
|
||||
const fullUrl =
|
||||
embedUrlFor(item) ??
|
||||
item.localUrl ??
|
||||
(item.kind === "video" ? streamPath : item.playUrl ?? item.thumbUrl ?? streamPath);
|
||||
item.messageUrl && item.kind === "video"
|
||||
? streamPath
|
||||
: embedUrlFor(item) ??
|
||||
item.localUrl ??
|
||||
(item.kind === "video" ? streamPath : item.playUrl ?? item.thumbUrl ?? streamPath);
|
||||
meta.append(button("View full", () => window.open(fullUrl, "_blank", "noreferrer")));
|
||||
|
||||
// Embeds break for age-gated or embed-disabled videos: keep a direct provider link.
|
||||
const watchUrl = youtubeWatchUrl(item);
|
||||
if (watchUrl) {
|
||||
meta.append(button("Open on YouTube", () => window.open(watchUrl, "_blank", "noreferrer")));
|
||||
}
|
||||
|
||||
viewerModal.hidden = false;
|
||||
}
|
||||
|
||||
@@ -2437,6 +2569,7 @@
|
||||
<div id="link-target" class="vk-media-status"></div>
|
||||
<div class="vk-media-status">This opens a new website in a new tab.</div>
|
||||
<div class="vk-media-toolbar">
|
||||
<button id="link-copy" class="btn btn-secondary vk-small-btn" type="button">Copy link</button>
|
||||
<button id="link-cancel" class="btn btn-secondary vk-small-btn" type="button">Cancel</button>
|
||||
<button id="link-continue" class="btn btn-primary vk-small-btn" type="button">Continue</button>
|
||||
</div>
|
||||
@@ -2448,12 +2581,29 @@
|
||||
|
||||
function openLinkWarning(url) {
|
||||
pendingLink = url;
|
||||
linkModal.querySelector("#link-copy").textContent = "Copy link";
|
||||
linkModal.querySelector("#link-target").textContent = url;
|
||||
linkModal.hidden = false;
|
||||
}
|
||||
|
||||
linkModal.querySelector("#link-copy").addEventListener("click", async () => {
|
||||
if (!pendingLink) return;
|
||||
try {
|
||||
await navigator.clipboard.writeText(pendingLink);
|
||||
} catch {
|
||||
const area = document.createElement("textarea");
|
||||
area.value = pendingLink;
|
||||
document.body.append(area);
|
||||
area.select();
|
||||
document.execCommand("copy");
|
||||
area.remove();
|
||||
}
|
||||
linkModal.querySelector("#link-copy").textContent = "Copied";
|
||||
});
|
||||
|
||||
linkModal.querySelector("#link-cancel").addEventListener("click", () => {
|
||||
linkModal.hidden = true;
|
||||
linkModal.querySelector("#link-copy").textContent = "Copy link";
|
||||
});
|
||||
linkModal.querySelector("#link-continue").addEventListener("click", () => {
|
||||
if (pendingLink) window.open(pendingLink, "_blank", "noreferrer");
|
||||
|
||||
@@ -10,6 +10,7 @@ import { isAbsolute, join } from "node:path";
|
||||
import {
|
||||
BackfillQueue,
|
||||
betterCandidate,
|
||||
messageUrlFor,
|
||||
sortCandidates,
|
||||
dirSizeBytes,
|
||||
isNearLimit,
|
||||
@@ -115,6 +116,10 @@ export interface MediaItemResult {
|
||||
scrapeId: string;
|
||||
itemId: number;
|
||||
channelId: string;
|
||||
guildId: string | null;
|
||||
/** The message that carried the media; the permanent link target. */
|
||||
messageId: string | null;
|
||||
messageUrl: string | null;
|
||||
kind: MediaKind;
|
||||
source: string | null;
|
||||
authorId: string | null;
|
||||
@@ -139,6 +144,8 @@ export interface MediaItemResult {
|
||||
downloaded: boolean;
|
||||
/** Embed title (YouTube and friends), when Discord provided one. */
|
||||
title: string | null;
|
||||
/** YouTube channel name kept from oEmbed, even if the video later disappears. */
|
||||
titleAuthor: string | null;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
@@ -182,6 +189,19 @@ export class MediaService {
|
||||
/** The newest bot and scrape errors, capped. Surfaced in the Bot tab. */
|
||||
private readonly errors: MediaErrorEntry[] = [];
|
||||
|
||||
/** One-shot maintenance: re-fetch previews for Discord rows closed out with HTTP errors. */
|
||||
private readonly recover = {
|
||||
state: "idle" as "idle" | "running" | "paused" | "done",
|
||||
processed: 0,
|
||||
recovered: 0,
|
||||
gone: 0,
|
||||
startedAt: null as number | null,
|
||||
finishedAt: null as number | null,
|
||||
currentScrapeId: null as string | null
|
||||
};
|
||||
/** Finished maintenance tasks, newest first. */
|
||||
private readonly taskHistory: Array<{ name: string; at: string; detail: string }> = [];
|
||||
|
||||
constructor(private readonly deps: MediaServiceDeps) {
|
||||
const tunables = this.deps.getMediaPrefs().tunables;
|
||||
|
||||
@@ -195,6 +215,14 @@ export class MediaService {
|
||||
pageMaxMs: tunables.timers.pageMaxMs,
|
||||
maxRetries: tunables.scrape.maxRetries,
|
||||
previewMaxAttempts: tunables.backfill.maxAttempts,
|
||||
suspectRetryMs: 50_000,
|
||||
refreshUrl: async (scrapeId, media) => {
|
||||
const meta = this.engine.get(scrapeId);
|
||||
if (!meta) return null;
|
||||
const item = this.getMediaItem(scrapeId, media.id);
|
||||
if (!item) return null;
|
||||
return this.refreshUrls(meta, item);
|
||||
},
|
||||
download: {
|
||||
maxFileBytes: tunables.scrape.maxFileBytes,
|
||||
previewMaxBytes: tunables.backfill.maxBytes,
|
||||
@@ -293,6 +321,145 @@ export class MediaService {
|
||||
return this.titles.status();
|
||||
}
|
||||
|
||||
/** The recovery task with its live remaining count and the task history. */
|
||||
recoverStatus(): {
|
||||
state: string;
|
||||
total: number;
|
||||
processed: number;
|
||||
recovered: number;
|
||||
gone: number;
|
||||
startedAt: string | null;
|
||||
finishedAt: string | null;
|
||||
currentScrapeId: string | null;
|
||||
history: Array<{ name: string; at: string; detail: string }>;
|
||||
} {
|
||||
let total = 0;
|
||||
for (const meta of this.engine.list()) {
|
||||
if (meta.state === "deleted" || meta.state === "deleting") continue;
|
||||
try {
|
||||
const store = new ScrapeStore(scrapeDir(this.rootDir, meta.id));
|
||||
try {
|
||||
total += store.discordPreviewFailureCount();
|
||||
} finally {
|
||||
store.close();
|
||||
}
|
||||
} catch {
|
||||
// an unreadable index has nothing to recover
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
state: this.recover.state,
|
||||
total,
|
||||
processed: this.recover.processed,
|
||||
recovered: this.recover.recovered,
|
||||
gone: this.recover.gone,
|
||||
startedAt: this.recover.startedAt ? new Date(this.recover.startedAt).toISOString() : null,
|
||||
finishedAt: this.recover.finishedAt ? new Date(this.recover.finishedAt).toISOString() : null,
|
||||
currentScrapeId: this.recover.currentScrapeId,
|
||||
history: [...this.taskHistory]
|
||||
};
|
||||
}
|
||||
|
||||
startRecover(): void {
|
||||
if (this.recover.state === "running") return;
|
||||
this.recover.state = "running";
|
||||
this.recover.startedAt = this.recover.startedAt ?? Date.now();
|
||||
this.recover.processed = 0;
|
||||
this.recover.recovered = 0;
|
||||
this.recover.gone = 0;
|
||||
this.deps.log?.("recover: retrying Discord-CDN previews marked gone");
|
||||
void this.runRecover();
|
||||
}
|
||||
|
||||
pauseRecover(): void {
|
||||
if (this.recover.state !== "running") return;
|
||||
this.recover.state = "paused";
|
||||
this.deps.log?.("recover: pausing");
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk every scrape's Discord-CDN preview failures, refresh the signed URL from the source
|
||||
* message (slopbot REST for guild scrapes, the puppet for DMs and group chats), then let the
|
||||
* engine fetch the preview. Rows whose URL cannot be refreshed are counted as gone.
|
||||
*/
|
||||
private async runRecover(): Promise<void> {
|
||||
const scrapeIds = this.engine
|
||||
.list()
|
||||
.filter(meta => meta.state !== "deleted" && meta.state !== "deleting")
|
||||
.map(meta => meta.id);
|
||||
|
||||
while (this.recover.state === "running") {
|
||||
let acted = false;
|
||||
|
||||
for (const scrapeId of scrapeIds) {
|
||||
if (this.recover.state !== "running") break;
|
||||
|
||||
const meta = this.engine.get(scrapeId);
|
||||
if (!meta) continue;
|
||||
|
||||
let store: ScrapeStore | null = null;
|
||||
try {
|
||||
store = new ScrapeStore(scrapeDir(this.rootDir, scrapeId));
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
const rows = store.discordPreviewFailures(10);
|
||||
if (!rows.length) continue;
|
||||
|
||||
acted = true;
|
||||
this.recover.currentScrapeId = scrapeId;
|
||||
const cleared: MediaRow[] = [];
|
||||
|
||||
for (const row of rows) {
|
||||
if (this.recover.state !== "running") break;
|
||||
const fresh = await this.refreshUrls(meta, {
|
||||
...row,
|
||||
scrapeId,
|
||||
channelId: meta.channelId
|
||||
} as MediaRow & { scrapeId: string; channelId: string });
|
||||
if (fresh) {
|
||||
store.clearPreviewError(row.id);
|
||||
cleared.push(row);
|
||||
} else {
|
||||
this.recover.gone++;
|
||||
}
|
||||
this.recover.processed++;
|
||||
}
|
||||
|
||||
if (cleared.length && this.recover.state === "running") {
|
||||
await this.engine.previewBatch(scrapeId, cleared.length);
|
||||
for (const row of cleared) {
|
||||
if (store.getMedia(row.id)?.preview_hash) this.recover.recovered++;
|
||||
else this.recover.gone++;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
store?.close();
|
||||
}
|
||||
|
||||
if (acted) break;
|
||||
}
|
||||
|
||||
if (!acted) break;
|
||||
}
|
||||
|
||||
if (this.recover.state === "running") this.recover.state = "done";
|
||||
this.recover.finishedAt = Date.now();
|
||||
this.recover.currentScrapeId = null;
|
||||
this.taskHistory.unshift({
|
||||
name: "Recover Discord previews",
|
||||
at: new Date().toISOString(),
|
||||
detail: `${this.recover.recovered} recovered, ${this.recover.gone} unavailable`
|
||||
});
|
||||
if (this.taskHistory.length > 10) this.taskHistory.length = 10;
|
||||
this.deps.log?.(
|
||||
`recover: done, ${this.recover.recovered} recovered, ${this.recover.gone} unavailable`
|
||||
);
|
||||
}
|
||||
|
||||
startBackfill(chatId?: string): BackfillStatus {
|
||||
this.backfill.start(chatId);
|
||||
return this.backfillStatus();
|
||||
@@ -1006,6 +1173,9 @@ export class MediaService {
|
||||
scrapeId: meta.id,
|
||||
itemId: row.id,
|
||||
channelId: meta.channelId,
|
||||
guildId: meta.guildId,
|
||||
messageId: row.message_id,
|
||||
messageUrl: messageUrlFor(meta.guildId, meta.channelId, row.message_id),
|
||||
kind: row.kind,
|
||||
source: row.source,
|
||||
authorId: row.author_id,
|
||||
@@ -1029,6 +1199,7 @@ export class MediaService {
|
||||
provider: row.provider,
|
||||
downloaded: row.downloaded === 1,
|
||||
title: row.title,
|
||||
titleAuthor: row.title_author,
|
||||
createdAt: row.created_at
|
||||
}));
|
||||
}
|
||||
|
||||
@@ -842,6 +842,29 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
||||
})
|
||||
);
|
||||
|
||||
app.get(
|
||||
"/api/media/recover",
|
||||
wrap(async (_req, res) => {
|
||||
res.json({ ok: true, result: deps.media.recoverStatus() });
|
||||
})
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/api/media/recover/start",
|
||||
wrap(async (_req, res) => {
|
||||
deps.media.startRecover();
|
||||
res.json({ ok: true, result: deps.media.recoverStatus() });
|
||||
})
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/api/media/recover/pause",
|
||||
wrap(async (_req, res) => {
|
||||
deps.media.pauseRecover();
|
||||
res.json({ ok: true, result: deps.media.recoverStatus() });
|
||||
})
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/api/media/titles/start",
|
||||
wrap(async (_req, res) => {
|
||||
|
||||
@@ -206,6 +206,17 @@ try {
|
||||
() => document.querySelector("#title-lane")?.textContent?.includes("Titles (YouTube)"),
|
||||
{ timeout: 15000 }
|
||||
);
|
||||
check(
|
||||
"recovery card renders",
|
||||
(await page.locator("#recover-lane").textContent())?.includes("Recover Discord previews") === true
|
||||
);
|
||||
const withMessage = await page.evaluate(async () => {
|
||||
const response = await fetch("/api/media?kind=video&limit=5");
|
||||
const body = await response.json();
|
||||
return (body.result ?? []).filter(item => item.messageUrl).length;
|
||||
});
|
||||
step("items with a message link", withMessage);
|
||||
check("message links present", withMessage > 0, withMessage);
|
||||
check("title lane card renders", (await page.locator("#title-lane .vk-bot-job-head").count()) === 1);
|
||||
|
||||
const summaryLines = await page.locator(".vk-bot-summary-line").count();
|
||||
|
||||
Reference in New Issue
Block a user