Add custom download destinations, confirmations, and the bot error feed
This commit is contained in:
@@ -40,6 +40,13 @@ mobile draft at 390px.
|
|||||||
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
||||||
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
||||||
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
||||||
|
- [x] C15 Scrape UX: custom download destination (absolute path per scrape; files there,
|
||||||
|
previews local, embed thumbnails copied to both, custom files not served), confirmation popup
|
||||||
|
before Links/Download, New scrape/History right-aligned with New scrape furthest right, modal
|
||||||
|
X and backdrop close, active-only progress strip, and a Bot error feed with a red unseen badge
|
||||||
|
that clears on open plus a Clear button. Live run `...-20260916095029`: 6 files and 7 embed
|
||||||
|
thumbnails in `/tmp/opencode/media-dest/<id>/`, 11 local previews, 0 local files, `localUrl`
|
||||||
|
null; error record and clear verified over the API.
|
||||||
- [x] C14 Links-mode embed thumbnails, item metadata, filters, and the scrape popup. Metadata:
|
- [x] C14 Links-mode embed thumbnails, item metadata, filters, and the scrape popup. Metadata:
|
||||||
poster (`author_id`/`author_name`), `posted_at`, `width`/`height`, `aspect_ratio`, with an
|
poster (`author_id`/`author_name`), `posted_at`, `width`/`height`, `aspect_ratio`, with an
|
||||||
`ffprobe` fallback after downloads. Filters: poster, aspect preset (vertical/square/4:3/16:9/
|
`ffprobe` fallback after downloads. Filters: poster, aspect preset (vertical/square/4:3/16:9/
|
||||||
|
|||||||
+1
-1
@@ -35,7 +35,7 @@ preset keys from `filter-presets.ts`; `after`/`before` are ISO times on `posted_
|
|||||||
|
|
||||||
| Tool | Access | Purpose |
|
| Tool | Access | Purpose |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `scrape_media` | trusted | Start a scrape (`channelId`, `mode`) |
|
| `scrape_media` | trusted | Start a scrape (`channelId`, `guildId?`, `mode?`, `backend?`, `downloadDir?`) |
|
||||||
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
|
| `list_media_scrapes` | trusted | Every scrape with state, size, and counts |
|
||||||
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
|
| `get_media_scrape` | trusted | One scrape, live counters and cursor |
|
||||||
| `stop_media_scrape` | trusted | Stop the active scrape |
|
| `stop_media_scrape` | trusted | Stop the active scrape |
|
||||||
|
|||||||
@@ -103,6 +103,9 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
|
|||||||
`file_bytes` are written to the row, so the run resumes and dedupes by content.
|
`file_bytes` are written to the row, so the run resumes and dedupes by content.
|
||||||
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
|
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
|
||||||
(`source != 'embed'`).
|
(`source != 'embed'`).
|
||||||
|
- With a custom destination, files go to `<destination>/<scrape-id>/media/` and the row stores the
|
||||||
|
absolute path; without one they stay in `<scrape>/media/`. Embed previews are copied to
|
||||||
|
`<destination>/<scrape-id>/thumbnails/` while the local copy stays for the Media tab.
|
||||||
- Progress counters: `downloaded`, `downloadErrors`.
|
- Progress counters: `downloaded`, `downloadErrors`.
|
||||||
|
|
||||||
Previews run after every walk:
|
Previews run after every walk:
|
||||||
|
|||||||
+11
-4
@@ -80,6 +80,10 @@ media-scrape/db/
|
|||||||
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
|
- `download` fetches attachment bytes into `media/`, one file at a time, hash-addressed. Embed rows
|
||||||
are never download targets. The Media tab prefers the local file when one exists, so the archive
|
are never download targets. The Media tab prefers the local file when one exists, so the archive
|
||||||
survives CDN URL expiry.
|
survives CDN URL expiry.
|
||||||
|
- A custom download destination sends the bytes to `<dir>/<scrape-id>/media/` instead. Previews
|
||||||
|
stay local, and embed thumbnails are copied to `<dir>/<scrape-id>/thumbnails/` so the download
|
||||||
|
has a poster. Files in a custom destination are not served or linked by the Media tab; the user
|
||||||
|
owns that path.
|
||||||
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
|
- Third-party embeds are **never downloaded as media**: YouTube, Instagram, X, TikTok, and every
|
||||||
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
|
other embed stays a link. Its thumbnail is cached (below); the video itself is never fetched.
|
||||||
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk, in both modes: Discord
|
- Previews are cached small (WebP, 320px) in `thumbs/` after every walk, in both modes: Discord
|
||||||
@@ -134,10 +138,13 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
|
|||||||
21:9, Other), resolution (SD, HD, FHD, QHD, 4K+, Unknown), and a posted-date range (from/to).
|
21:9, Other), resolution (SD, HD, FHD, QHD, 4K+, Unknown), and a posted-date range (from/to).
|
||||||
Grid with lazy loading; previews come from the cache, then the local file, then the CDN; each
|
Grid with lazy loading; previews come from the cache, then the local file, then the CDN; each
|
||||||
tile's tooltip shows the poster and posted time. A scrape strip shows live progress with Stop.
|
tile's tooltip shows the poster and posted time. A scrape strip shows live progress with Stop.
|
||||||
**New scrape…** opens a popup with Server and Group chat trees (servers expand to channels; each
|
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to
|
||||||
row has Links and Download), and **History** lists each scrape with its start time, duration,
|
channels; each row has Links and Download). Every scrape button asks for confirmation, and
|
||||||
pages, messages, media counts, bytes, files, and previews. The view fills the window below the
|
Download can target a custom absolute destination. **History** lists each scrape with its start
|
||||||
header and the call strip.
|
time, duration, pages, messages, media counts, bytes, files, and previews. The popups close on
|
||||||
|
backdrop click, X, or Close. Only active scrapes show in the strip; the view fills the window
|
||||||
|
below the header and the call strip. The Bot tab lists bot and scrape errors with a red badge
|
||||||
|
that disappears when the tab is opened and a Clear button.
|
||||||
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
|
- **Bot** tab: slopbot connection state, the scrape queue, and the storage limit with a usage bar.
|
||||||
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
|
The bar is segmented and colour coded by media kind (images, GIFs, video, audio, other) plus the
|
||||||
database overhead.
|
database overhead.
|
||||||
|
|||||||
+3
-1
@@ -263,7 +263,9 @@ replaceable.
|
|||||||
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
|
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
|
||||||
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
||||||
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
||||||
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`. Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
|
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `backend?`, `confirm?`, `downloadDir?` | Start a media scrape. `backend` is `auto` (default), `slopbot`, or `puppet`; `downloadDir` is an absolute path for downloaded files (previews stay local). Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
|
||||||
|
| GET | `/api/media/errors` | none | Bot and scrape errors, newest first |
|
||||||
|
| POST | `/api/media/errors/clear` | none | Clear the error list |
|
||||||
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
|
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
|
||||||
| POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent |
|
| POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent |
|
||||||
| GET | `/api/logs` | `limit` | The persistent action log |
|
| GET | `/api/logs` | `limit` | The persistent action log |
|
||||||
|
|||||||
@@ -44,8 +44,10 @@ export interface DownloadOptions {
|
|||||||
maxRetries?: number;
|
maxRetries?: number;
|
||||||
/** Preview size cap. Default 5 MiB. */
|
/** Preview size cap. Default 5 MiB. */
|
||||||
previewMaxBytes?: number;
|
previewMaxBytes?: number;
|
||||||
/** Cache third-party embed thumbnails. Download mode only. */
|
/** Cache third-party embed thumbnails. */
|
||||||
embedPreviews?: boolean;
|
embedPreviews?: boolean;
|
||||||
|
/** Absolute destination for downloaded files. Null or unset keeps them under the scrape dir. */
|
||||||
|
destinationDir?: string | null;
|
||||||
sleep?: (ms: number) => Promise<void>;
|
sleep?: (ms: number) => Promise<void>;
|
||||||
random?: () => number;
|
random?: () => number;
|
||||||
log?: (line: string) => void;
|
log?: (line: string) => void;
|
||||||
@@ -134,7 +136,7 @@ export async function downloadOne(
|
|||||||
const file = `${hash}.${extension}`;
|
const file = `${hash}.${extension}`;
|
||||||
writeFileSync(join(dir, file), buffer);
|
writeFileSync(join(dir, file), buffer);
|
||||||
|
|
||||||
return { id: media.id, state: "downloaded", path: join("media", file), hash, bytes: buffer.byteLength };
|
return { id: media.id, state: "downloaded", path: file, hash, bytes: buffer.byteLength };
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
lastError = error instanceof Error ? error.message : String(error);
|
lastError = error instanceof Error ? error.message : String(error);
|
||||||
|
|||||||
@@ -3,7 +3,8 @@
|
|||||||
* Rules: docs/media-scraper.md.
|
* Rules: docs/media-scraper.md.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { join } from "node:path";
|
import { copyFileSync, mkdirSync } from "node:fs";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
import {
|
import {
|
||||||
downloadGapMs,
|
downloadGapMs,
|
||||||
downloadOne,
|
downloadOne,
|
||||||
@@ -58,6 +59,8 @@ export interface ScrapeEngineOptions {
|
|||||||
previews?: boolean;
|
previews?: boolean;
|
||||||
/** Read dimensions from a downloaded file. Defaults to ffprobe. */
|
/** Read dimensions from a downloaded file. Defaults to ffprobe. */
|
||||||
probe?: (path: string) => Promise<{ width: number; height: number } | null>;
|
probe?: (path: string) => Promise<{ width: number; height: number } | null>;
|
||||||
|
/** Called when a scrape fails. */
|
||||||
|
onError?: (scrapeId: string, message: string) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface QueuedJob {
|
interface QueuedJob {
|
||||||
@@ -91,6 +94,7 @@ export class ScrapeEngine {
|
|||||||
guildId: string | null;
|
guildId: string | null;
|
||||||
mode: ScrapeMode;
|
mode: ScrapeMode;
|
||||||
backend: ScrapeBackend;
|
backend: ScrapeBackend;
|
||||||
|
downloadDir?: string | null;
|
||||||
}): Promise<ScrapeMeta> {
|
}): Promise<ScrapeMeta> {
|
||||||
const now = this.now();
|
const now = this.now();
|
||||||
const id = uniqueScrapeId(this.options.rootDir, input.channelId, new Date(now));
|
const id = uniqueScrapeId(this.options.rootDir, input.channelId, new Date(now));
|
||||||
@@ -111,7 +115,8 @@ export class ScrapeEngine {
|
|||||||
messages: 0,
|
messages: 0,
|
||||||
media: emptyMediaCounts(),
|
media: emptyMediaCounts(),
|
||||||
errors: 0,
|
errors: 0,
|
||||||
lastError: null
|
lastError: null,
|
||||||
|
downloadDir: input.downloadDir ?? null
|
||||||
};
|
};
|
||||||
|
|
||||||
writeScrapeMeta(dir, meta);
|
writeScrapeMeta(dir, meta);
|
||||||
@@ -251,6 +256,7 @@ export class ScrapeEngine {
|
|||||||
meta.state = "failed";
|
meta.state = "failed";
|
||||||
meta.lastError = error instanceof Error ? error.message : String(error);
|
meta.lastError = error instanceof Error ? error.message : String(error);
|
||||||
this.log(`${meta.id}: failed: ${meta.lastError}`);
|
this.log(`${meta.id}: failed: ${meta.lastError}`);
|
||||||
|
this.options.onError?.(meta.id, meta.lastError);
|
||||||
} finally {
|
} finally {
|
||||||
if ((meta.state === "done" || meta.state === "failed") && !meta.finishedAt) {
|
if ((meta.state === "done" || meta.state === "failed") && !meta.finishedAt) {
|
||||||
meta.finishedAt = new Date(this.now()).toISOString();
|
meta.finishedAt = new Date(this.now()).toISOString();
|
||||||
@@ -263,7 +269,13 @@ export class ScrapeEngine {
|
|||||||
|
|
||||||
/** Fetch attachment bytes for a finished walk. Serial, hash-addressed, resumable. */
|
/** Fetch attachment bytes for a finished walk. Serial, hash-addressed, resumable. */
|
||||||
private async runDownloads(store: ScrapeStore, dir: string, meta: ScrapeMeta): Promise<void> {
|
private async runDownloads(store: ScrapeStore, dir: string, meta: ScrapeMeta): Promise<void> {
|
||||||
const options = this.options.download ?? {};
|
const customDir = meta.downloadDir ?? null;
|
||||||
|
const options: DownloadOptions = {
|
||||||
|
...(this.options.download ?? {}),
|
||||||
|
destinationDir: customDir
|
||||||
|
};
|
||||||
|
const targetDir = customDir ? join(customDir, meta.id, "media") : join(dir, "media");
|
||||||
|
mkdirSync(targetDir, { recursive: true });
|
||||||
const pending = store.pendingDownloads();
|
const pending = store.pendingDownloads();
|
||||||
|
|
||||||
for (const row of pending) {
|
for (const row of pending) {
|
||||||
@@ -272,21 +284,24 @@ export class ScrapeEngine {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const outcome = await downloadOne(row, join(dir, "media"), {
|
const outcome = await downloadOne(row, targetDir, {
|
||||||
...options,
|
...options,
|
||||||
log: line => this.log(`${meta.id}: ${line}`)
|
log: line => this.log(`${meta.id}: ${line}`)
|
||||||
});
|
});
|
||||||
|
|
||||||
if (outcome.state === "downloaded" && outcome.path && outcome.hash) {
|
if (outcome.state === "downloaded" && outcome.path && outcome.hash) {
|
||||||
|
const filePath = customDir ? join(targetDir, outcome.path) : join("media", outcome.path);
|
||||||
store.markDownloaded(row.id, {
|
store.markDownloaded(row.id, {
|
||||||
path: outcome.path,
|
path: filePath,
|
||||||
hash: outcome.hash,
|
hash: outcome.hash,
|
||||||
bytes: outcome.bytes ?? 0
|
bytes: outcome.bytes ?? 0
|
||||||
});
|
});
|
||||||
meta.downloaded = (meta.downloaded ?? 0) + 1;
|
meta.downloaded = (meta.downloaded ?? 0) + 1;
|
||||||
|
|
||||||
if (row.width === null || row.height === null) {
|
if (row.width === null || row.height === null) {
|
||||||
const dimensions = await (this.options.probe ?? probeDimensions)(join(dir, outcome.path));
|
const dimensions = await (this.options.probe ?? probeDimensions)(
|
||||||
|
join(targetDir, outcome.path)
|
||||||
|
);
|
||||||
if (dimensions) store.updateDimensions(row.id, dimensions.width, dimensions.height);
|
if (dimensions) store.updateDimensions(row.id, dimensions.width, dimensions.height);
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
@@ -301,7 +316,11 @@ export class ScrapeEngine {
|
|||||||
|
|
||||||
/** Cache small previews. Embed thumbnails are cached in both modes. */
|
/** Cache small previews. Embed thumbnails are cached in both modes. */
|
||||||
private async runPreviews(store: ScrapeStore, dir: string, meta: ScrapeMeta): Promise<void> {
|
private async runPreviews(store: ScrapeStore, dir: string, meta: ScrapeMeta): Promise<void> {
|
||||||
const options = { ...(this.options.download ?? {}), embedPreviews: true };
|
const options = {
|
||||||
|
...(this.options.download ?? {}),
|
||||||
|
embedPreviews: true,
|
||||||
|
destinationDir: meta.downloadDir ?? null
|
||||||
|
};
|
||||||
const pending = store.pendingPreviews(5000, true);
|
const pending = store.pendingPreviews(5000, true);
|
||||||
|
|
||||||
for (const row of pending) {
|
for (const row of pending) {
|
||||||
@@ -315,6 +334,14 @@ export class ScrapeEngine {
|
|||||||
if (outcome.state === "downloaded" && outcome.hash) {
|
if (outcome.state === "downloaded" && outcome.hash) {
|
||||||
store.markPreview(row.id, { hash: outcome.hash, bytes: outcome.bytes ?? 0 });
|
store.markPreview(row.id, { hash: outcome.hash, bytes: outcome.bytes ?? 0 });
|
||||||
meta.previewed = (meta.previewed ?? 0) + 1;
|
meta.previewed = (meta.previewed ?? 0) + 1;
|
||||||
|
|
||||||
|
// Embed thumbnails also land in the custom destination, so the download has a
|
||||||
|
// viewable poster for embeds while the media tab keeps its local copy.
|
||||||
|
if (options.destinationDir && row.source === "embed") {
|
||||||
|
const target = join(options.destinationDir, meta.id, "thumbnails", `${outcome.hash}.webp`);
|
||||||
|
mkdirSync(dirname(target), { recursive: true });
|
||||||
|
copyFileSync(join(dir, "thumbs", `${outcome.hash}.webp`), target);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
store.markPreviewError(row.id, outcome.error ?? outcome.state);
|
store.markPreviewError(row.id, outcome.error ?? outcome.state);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -94,6 +94,8 @@ export interface ScrapeMeta {
|
|||||||
previewed?: number;
|
previewed?: number;
|
||||||
/** When the scrape reached done or failed. */
|
/** When the scrape reached done or failed. */
|
||||||
finishedAt?: string | null;
|
finishedAt?: string | null;
|
||||||
|
/** Absolute destination for downloaded files. Null means the local `media/` folder. */
|
||||||
|
downloadDir?: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function emptyMediaCounts(): Record<MediaKind, number> {
|
export function emptyMediaCounts(): Record<MediaKind, number> {
|
||||||
|
|||||||
@@ -637,6 +637,79 @@ void (async () => {
|
|||||||
assert.equal(existsSync(join(root, started.id, "thumbs", `${row.preview_hash}.webp`)), true);
|
assert.equal(existsSync(join(root, started.id, "thumbs", `${row.preview_hash}.webp`)), true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
await test("custom download destination keeps files off-archive and copies embed previews", async () => {
|
||||||
|
const dest = join(root, "custom-dest");
|
||||||
|
mkdirSync(dest, { recursive: true });
|
||||||
|
|
||||||
|
const backend = new FakeBackend([
|
||||||
|
{
|
||||||
|
messages: [
|
||||||
|
message("702", {
|
||||||
|
attachments: [attachment({ filename: "offsite.png", contentType: "image/png" })],
|
||||||
|
embeds: [
|
||||||
|
{
|
||||||
|
type: "video",
|
||||||
|
url: "https://www.youtube.com/watch?v=x",
|
||||||
|
title: null,
|
||||||
|
provider: "YouTube",
|
||||||
|
videoUrl: "https://www.youtube.com/embed/x",
|
||||||
|
imageUrl: null,
|
||||||
|
thumbnailUrl: "https://i.ytimg.com/vi/x/hqdefault.jpg"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
})
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]);
|
||||||
|
|
||||||
|
const engine = new ScrapeEngine({
|
||||||
|
rootDir: root,
|
||||||
|
pageSize: 2,
|
||||||
|
pageMinMs: 0,
|
||||||
|
pageMaxMs: 0,
|
||||||
|
sleep: async () => {},
|
||||||
|
random: () => 0,
|
||||||
|
log: () => {},
|
||||||
|
previews: true,
|
||||||
|
download: {
|
||||||
|
fetchImpl: async () =>
|
||||||
|
new Response("bytes", { status: 200, headers: { "content-type": "image/png" } }),
|
||||||
|
minGapMs: 0,
|
||||||
|
maxGapMs: 0,
|
||||||
|
sleep: async () => {}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const started = await engine.start({
|
||||||
|
channelId: "130",
|
||||||
|
guildId: null,
|
||||||
|
mode: "download",
|
||||||
|
backend,
|
||||||
|
downloadDir: dest
|
||||||
|
});
|
||||||
|
await engine.processQueue();
|
||||||
|
|
||||||
|
const meta = engine.get(started.id);
|
||||||
|
assert.equal(meta?.downloadDir, dest);
|
||||||
|
assert.equal(meta?.downloaded, 1);
|
||||||
|
|
||||||
|
const store = new ScrapeStore(join(root, started.id));
|
||||||
|
const file = store.queryMedia({ kind: "image" })[0];
|
||||||
|
const filePath = file.file_path;
|
||||||
|
assert.ok(filePath && filePath.startsWith(dest));
|
||||||
|
assert.equal(existsSync(filePath), true);
|
||||||
|
assert.equal(existsSync(join(root, started.id, "media", filePath.split("/").pop() ?? "")), false);
|
||||||
|
|
||||||
|
const embed = store.queryMedia({ kind: "video" })[0];
|
||||||
|
assert.ok(embed.preview_hash);
|
||||||
|
assert.equal(
|
||||||
|
existsSync(join(dest, started.id, "thumbnails", `${embed.preview_hash}.webp`)),
|
||||||
|
true
|
||||||
|
);
|
||||||
|
assert.equal(existsSync(join(root, started.id, "thumbs", `${embed.preview_hash}.webp`)), true);
|
||||||
|
store.close();
|
||||||
|
});
|
||||||
|
|
||||||
await test("retention marks the oldest completed scrapes up to the limit", () => {
|
await test("retention marks the oldest completed scrapes up to the limit", () => {
|
||||||
const scrapes = [
|
const scrapes = [
|
||||||
{ id: "old", state: "done", createdAt: "2026-09-01T00:00:00.000Z", bytes: 600 },
|
{ id: "old", state: "done", createdAt: "2026-09-01T00:00:00.000Z", bytes: 600 },
|
||||||
|
|||||||
@@ -286,3 +286,37 @@
|
|||||||
background: var(--color-accent-400, #4f8cff);
|
background: var(--color-accent-400, #4f8cff);
|
||||||
animation: vk-media-pulse 1.2s ease-in-out infinite;
|
animation: vk-media-pulse 1.2s ease-in-out infinite;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.vk-media-spacer {
|
||||||
|
flex: 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
.vk-media-check {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 6px;
|
||||||
|
font: 400 11px/1.4 ui-monospace, Menlo, monospace;
|
||||||
|
color: var(--color-neutral-300);
|
||||||
|
}
|
||||||
|
|
||||||
|
.vk-media-dest {
|
||||||
|
width: 100%;
|
||||||
|
}
|
||||||
|
|
||||||
|
.vk-media-error {
|
||||||
|
color: #ff8080;
|
||||||
|
}
|
||||||
|
|
||||||
|
.vk-media-badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
min-width: 14px;
|
||||||
|
height: 14px;
|
||||||
|
margin-left: 4px;
|
||||||
|
padding: 0 4px;
|
||||||
|
border-radius: 7px;
|
||||||
|
background: #e5484d;
|
||||||
|
color: #fff;
|
||||||
|
font: 600 9px/1 ui-monospace, Menlo, monospace;
|
||||||
|
}
|
||||||
|
|||||||
+189
-21
@@ -22,6 +22,7 @@
|
|||||||
dms: [],
|
dms: [],
|
||||||
loadedChannels: false,
|
loadedChannels: false,
|
||||||
authors: null,
|
authors: null,
|
||||||
|
errors: [],
|
||||||
filters: {
|
filters: {
|
||||||
source: "",
|
source: "",
|
||||||
channelId: "",
|
channelId: "",
|
||||||
@@ -92,6 +93,13 @@
|
|||||||
document.querySelector("#scope-bot")?.classList.toggle("active", view === "bot");
|
document.querySelector("#scope-bot")?.classList.toggle("active", view === "bot");
|
||||||
document.querySelector("#scope-guild")?.classList.remove("active");
|
document.querySelector("#scope-guild")?.classList.remove("active");
|
||||||
document.querySelector("#scope-dm")?.classList.remove("active");
|
document.querySelector("#scope-dm")?.classList.remove("active");
|
||||||
|
|
||||||
|
if (view === "bot") {
|
||||||
|
localStorage.setItem("vk-media-errors-seen", String(errorCount));
|
||||||
|
updateErrorBadge();
|
||||||
|
void refreshErrors();
|
||||||
|
}
|
||||||
|
|
||||||
render();
|
render();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -191,7 +199,7 @@
|
|||||||
renderGrid();
|
renderGrid();
|
||||||
}
|
}
|
||||||
|
|
||||||
async function startScrape(channelId, guildId, mode) {
|
async function startScrape(channelId, guildId, mode, downloadDir) {
|
||||||
if (!channelId) {
|
if (!channelId) {
|
||||||
setStatus("no channel selected");
|
setStatus("no channel selected");
|
||||||
return;
|
return;
|
||||||
@@ -202,7 +210,8 @@
|
|||||||
channelId,
|
channelId,
|
||||||
guildId: guildId ?? null,
|
guildId: guildId ?? null,
|
||||||
mode,
|
mode,
|
||||||
backend: guildId ? "auto" : "puppet"
|
backend: guildId ? "auto" : "puppet",
|
||||||
|
downloadDir: downloadDir ?? null
|
||||||
});
|
});
|
||||||
setStatus(`started ${scrape.id} (${mode})`);
|
setStatus(`started ${scrape.id} (${mode})`);
|
||||||
scrapeModal.hidden = true;
|
scrapeModal.hidden = true;
|
||||||
@@ -219,8 +228,8 @@
|
|||||||
function renderScrapes() {
|
function renderScrapes() {
|
||||||
scrapeList.replaceChildren();
|
scrapeList.replaceChildren();
|
||||||
const active = state.scrapes.filter(scrape => ["queued", "running", "paused"].includes(scrape.state));
|
const active = state.scrapes.filter(scrape => ["queued", "running", "paused"].includes(scrape.state));
|
||||||
const rows = active.length ? active : state.scrapes.slice(0, 5);
|
|
||||||
for (const scrape of rows) {
|
for (const scrape of active) {
|
||||||
const row = el("div", "vk-media-scrape");
|
const row = el("div", "vk-media-scrape");
|
||||||
const media = scrape.media ?? {};
|
const media = scrape.media ?? {};
|
||||||
|
|
||||||
@@ -239,21 +248,79 @@
|
|||||||
)
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
if (active.includes(scrape)) {
|
row.append(
|
||||||
row.append(
|
button("Stop", async () => {
|
||||||
button("Stop", async () => {
|
await window.VK.api.post("/api/media/scrape/stop", { id: scrape.id });
|
||||||
await window.VK.api.post("/api/media/scrape/stop", { id: scrape.id });
|
void loadScrapes();
|
||||||
void loadScrapes();
|
})
|
||||||
})
|
);
|
||||||
);
|
|
||||||
} else {
|
|
||||||
row.append(el("span", "vk-media-status", durationLabel(scrape)));
|
|
||||||
}
|
|
||||||
|
|
||||||
scrapeList.append(row);
|
scrapeList.append(row);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- Bot errors ----
|
||||||
|
|
||||||
|
let errorCount = 0;
|
||||||
|
|
||||||
|
function seenErrorCount() {
|
||||||
|
return Number(localStorage.getItem("vk-media-errors-seen") ?? 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateErrorBadge() {
|
||||||
|
const badge = document.querySelector("#scope-bot-badge");
|
||||||
|
if (!badge) return;
|
||||||
|
|
||||||
|
const unseen = Math.max(0, errorCount - seenErrorCount());
|
||||||
|
badge.textContent = String(unseen);
|
||||||
|
badge.hidden = unseen === 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderErrors() {
|
||||||
|
scrapeList.replaceChildren();
|
||||||
|
|
||||||
|
const head = el("div", "vk-media-scrape");
|
||||||
|
head.append(el("span", null, `${(state.errors ?? []).length} error(s)`));
|
||||||
|
head.append(
|
||||||
|
button("Clear errors", async () => {
|
||||||
|
try {
|
||||||
|
await window.VK.api.post("/api/media/errors/clear", {});
|
||||||
|
} catch {
|
||||||
|
// the list stays until the next refresh
|
||||||
|
}
|
||||||
|
state.errors = [];
|
||||||
|
errorCount = 0;
|
||||||
|
localStorage.setItem("vk-media-errors-seen", "0");
|
||||||
|
updateErrorBadge();
|
||||||
|
renderErrors();
|
||||||
|
})
|
||||||
|
);
|
||||||
|
scrapeList.append(head);
|
||||||
|
|
||||||
|
for (const entry of state.errors ?? []) {
|
||||||
|
scrapeList.append(
|
||||||
|
el(
|
||||||
|
"div",
|
||||||
|
"vk-media-scrape vk-media-error",
|
||||||
|
`${new Date(entry.at).toLocaleTimeString()} · ${entry.source} · ${entry.message}`
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!(state.errors ?? []).length) scrapeList.append(el("div", "vk-media-scrape", "No errors."));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function refreshErrors() {
|
||||||
|
try {
|
||||||
|
state.errors = await window.VK.api.get("/api/media/errors");
|
||||||
|
} catch {
|
||||||
|
state.errors = [];
|
||||||
|
}
|
||||||
|
errorCount = (state.errors ?? []).length;
|
||||||
|
updateErrorBadge();
|
||||||
|
if (state.view === "bot") renderErrors();
|
||||||
|
}
|
||||||
|
|
||||||
function renderGrid() {
|
function renderGrid() {
|
||||||
grid.replaceChildren();
|
grid.replaceChildren();
|
||||||
for (const item of state.items) {
|
for (const item of state.items) {
|
||||||
@@ -319,8 +386,7 @@
|
|||||||
toolbar.append(chip);
|
toolbar.append(chip);
|
||||||
}
|
}
|
||||||
|
|
||||||
toolbar.append(button("New scrape…", openScrapeModal));
|
|
||||||
toolbar.append(button("History", openHistoryModal));
|
|
||||||
|
|
||||||
const sourceSelect = document.createElement("select");
|
const sourceSelect = document.createElement("select");
|
||||||
sourceSelect.className = "vk-chip vk-chip-off";
|
sourceSelect.className = "vk-chip vk-chip-off";
|
||||||
@@ -428,6 +494,10 @@
|
|||||||
void loadMedia(true);
|
void loadMedia(true);
|
||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
|
||||||
|
toolbar.append(el("span", "vk-media-spacer"));
|
||||||
|
toolbar.append(button("History", openHistoryModal));
|
||||||
|
toolbar.append(button("New scrape…", openScrapeModal));
|
||||||
} else {
|
} else {
|
||||||
toolbar.append(el("span", null, "Slopbot is the dedicated bot core. Setup: docs/slopbot.md."));
|
toolbar.append(el("span", null, "Slopbot is the dedicated bot core. Setup: docs/slopbot.md."));
|
||||||
}
|
}
|
||||||
@@ -444,12 +514,14 @@
|
|||||||
statusLine.textContent = bot?.connected
|
statusLine.textContent = bot?.connected
|
||||||
? `connected as ${bot.user} in ${bot.guilds.length} guild(s)`
|
? `connected as ${bot.user} in ${bot.guilds.length} guild(s)`
|
||||||
: "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)";
|
: "slopbot is disabled or not logged in (VESKSLOP_SLOPBOT_ENABLED=1 and a token)";
|
||||||
|
void loadScrapes().then(renderScrapes);
|
||||||
|
void refreshErrors();
|
||||||
} else {
|
} else {
|
||||||
void loadChannels().then(renderToolbar);
|
void loadChannels().then(renderToolbar);
|
||||||
void loadAuthors().then(renderToolbar);
|
void loadAuthors().then(renderToolbar);
|
||||||
void loadMedia(true);
|
void loadMedia(true);
|
||||||
|
void loadScrapes();
|
||||||
}
|
}
|
||||||
void loadScrapes();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
overlay.querySelector("#media-close").addEventListener("click", hide);
|
overlay.querySelector("#media-close").addEventListener("click", hide);
|
||||||
@@ -465,6 +537,16 @@
|
|||||||
document.querySelector("#scope-guild")?.addEventListener("click", hide);
|
document.querySelector("#scope-guild")?.addEventListener("click", hide);
|
||||||
document.querySelector("#scope-dm")?.addEventListener("click", hide);
|
document.querySelector("#scope-dm")?.addEventListener("click", hide);
|
||||||
|
|
||||||
|
const errorBadge = el("span", "vk-media-badge");
|
||||||
|
errorBadge.id = "scope-bot-badge";
|
||||||
|
errorBadge.hidden = true;
|
||||||
|
scopeBot?.append(errorBadge);
|
||||||
|
|
||||||
|
void refreshErrors();
|
||||||
|
setInterval(() => {
|
||||||
|
if (state.view !== "bot") void refreshErrors();
|
||||||
|
}, 15000);
|
||||||
|
|
||||||
// ---- Scrape popup: server or group-chat trees ----
|
// ---- Scrape popup: server or group-chat trees ----
|
||||||
|
|
||||||
const scrapeModal = document.createElement("div");
|
const scrapeModal = document.createElement("div");
|
||||||
@@ -472,7 +554,11 @@
|
|||||||
scrapeModal.hidden = true;
|
scrapeModal.hidden = true;
|
||||||
scrapeModal.innerHTML = `
|
scrapeModal.innerHTML = `
|
||||||
<div class="vk-media-modal-box vk-media-wide">
|
<div class="vk-media-modal-box vk-media-wide">
|
||||||
<strong>New scrape</strong>
|
<div class="vk-media-head">
|
||||||
|
<strong>New scrape</strong>
|
||||||
|
<span class="vk-media-spacer"></span>
|
||||||
|
<button id="scrape-x" class="btn btn-secondary vk-small-btn" type="button">X</button>
|
||||||
|
</div>
|
||||||
<div class="vk-media-toolbar">
|
<div class="vk-media-toolbar">
|
||||||
<button id="scrape-tab-server" class="vk-chip vk-chip-on" type="button">Server</button>
|
<button id="scrape-tab-server" class="vk-chip vk-chip-on" type="button">Server</button>
|
||||||
<button id="scrape-tab-group" class="vk-chip vk-chip-off" type="button">Group chat</button>
|
<button id="scrape-tab-group" class="vk-chip vk-chip-off" type="button">Group chat</button>
|
||||||
@@ -491,8 +577,8 @@
|
|||||||
function channelRow(channelId, guildId, label) {
|
function channelRow(channelId, guildId, label) {
|
||||||
const row = el("div", "vk-media-channel");
|
const row = el("div", "vk-media-channel");
|
||||||
row.append(el("span", "vk-media-channel-name", label));
|
row.append(el("span", "vk-media-channel-name", label));
|
||||||
row.append(button("Links", () => void startScrape(channelId, guildId, "links")));
|
row.append(button("Links", () => askScrape(channelId, guildId, "links", label)));
|
||||||
row.append(button("Download", () => void startScrape(channelId, guildId, "download")));
|
row.append(button("Download", () => askScrape(channelId, guildId, "download", label)));
|
||||||
return row;
|
return row;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -558,6 +644,78 @@
|
|||||||
scrapeModal.querySelector("#scrape-cancel").addEventListener("click", () => {
|
scrapeModal.querySelector("#scrape-cancel").addEventListener("click", () => {
|
||||||
scrapeModal.hidden = true;
|
scrapeModal.hidden = true;
|
||||||
});
|
});
|
||||||
|
scrapeModal.querySelector("#scrape-x").addEventListener("click", () => {
|
||||||
|
scrapeModal.hidden = true;
|
||||||
|
});
|
||||||
|
scrapeModal.addEventListener("click", event => {
|
||||||
|
if (event.target === scrapeModal) scrapeModal.hidden = true;
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---- Confirmation popup for Links / Download ----
|
||||||
|
|
||||||
|
const confirmModal = document.createElement("div");
|
||||||
|
confirmModal.className = "vk-media-modal";
|
||||||
|
confirmModal.hidden = true;
|
||||||
|
confirmModal.innerHTML = `
|
||||||
|
<div class="vk-media-modal-box">
|
||||||
|
<strong id="confirm-title">Start scrape?</strong>
|
||||||
|
<div id="confirm-note" class="vk-media-status"></div>
|
||||||
|
<label id="confirm-dest-row" class="vk-media-check" hidden>
|
||||||
|
<input id="confirm-custom-dest" type="checkbox" /> Custom download destination
|
||||||
|
</label>
|
||||||
|
<input
|
||||||
|
id="confirm-dest"
|
||||||
|
class="vk-media-input vk-media-dest"
|
||||||
|
type="text"
|
||||||
|
placeholder="/path/to/download/"
|
||||||
|
hidden
|
||||||
|
/>
|
||||||
|
<div class="vk-media-toolbar">
|
||||||
|
<button id="confirm-cancel" class="btn btn-secondary vk-small-btn" type="button">Cancel</button>
|
||||||
|
<button id="confirm-start" class="btn btn-primary vk-small-btn" type="button">Start</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
document.body.append(confirmModal);
|
||||||
|
|
||||||
|
let confirmAction = null;
|
||||||
|
|
||||||
|
function askScrape(channelId, guildId, mode, label) {
|
||||||
|
const isDownload = mode === "download";
|
||||||
|
const destRow = confirmModal.querySelector("#confirm-dest-row");
|
||||||
|
const destInput = confirmModal.querySelector("#confirm-dest");
|
||||||
|
const checkbox = confirmModal.querySelector("#confirm-custom-dest");
|
||||||
|
|
||||||
|
confirmModal.querySelector("#confirm-title").textContent =
|
||||||
|
isDownload ? "Download scrape?" : "Links scrape?";
|
||||||
|
confirmModal.querySelector("#confirm-note").textContent = isDownload
|
||||||
|
? `${label} — downloads files${guildId ? "" : " with the puppet"}.`
|
||||||
|
: `${label} — indexes links and caches previews${guildId ? "" : " with the puppet"}.`;
|
||||||
|
|
||||||
|
destRow.hidden = !isDownload;
|
||||||
|
destInput.hidden = true;
|
||||||
|
destInput.value = "";
|
||||||
|
checkbox.checked = false;
|
||||||
|
checkbox.onchange = () => {
|
||||||
|
destInput.hidden = !checkbox.checked;
|
||||||
|
};
|
||||||
|
|
||||||
|
confirmAction = () =>
|
||||||
|
startScrape(channelId, guildId, mode, isDownload && checkbox.checked ? destInput.value.trim() : null);
|
||||||
|
confirmModal.hidden = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
confirmModal.querySelector("#confirm-cancel").addEventListener("click", () => {
|
||||||
|
confirmModal.hidden = true;
|
||||||
|
});
|
||||||
|
confirmModal.querySelector("#confirm-start").addEventListener("click", () => {
|
||||||
|
const action = confirmAction;
|
||||||
|
confirmModal.hidden = true;
|
||||||
|
void action?.();
|
||||||
|
});
|
||||||
|
confirmModal.addEventListener("click", event => {
|
||||||
|
if (event.target === confirmModal) confirmModal.hidden = true;
|
||||||
|
});
|
||||||
|
|
||||||
// ---- History popup ----
|
// ---- History popup ----
|
||||||
|
|
||||||
@@ -566,7 +724,11 @@
|
|||||||
historyModal.hidden = true;
|
historyModal.hidden = true;
|
||||||
historyModal.innerHTML = `
|
historyModal.innerHTML = `
|
||||||
<div class="vk-media-modal-box vk-media-wide">
|
<div class="vk-media-modal-box vk-media-wide">
|
||||||
<strong>Scrape history</strong>
|
<div class="vk-media-head">
|
||||||
|
<strong>Scrape history</strong>
|
||||||
|
<span class="vk-media-spacer"></span>
|
||||||
|
<button id="history-x" class="btn btn-secondary vk-small-btn" type="button">X</button>
|
||||||
|
</div>
|
||||||
<div id="history-list" class="vk-media-modal-list"></div>
|
<div id="history-list" class="vk-media-modal-list"></div>
|
||||||
<div class="vk-media-toolbar">
|
<div class="vk-media-toolbar">
|
||||||
<button id="history-close" class="btn btn-secondary vk-small-btn" type="button">Close</button>
|
<button id="history-close" class="btn btn-secondary vk-small-btn" type="button">Close</button>
|
||||||
@@ -630,6 +792,12 @@
|
|||||||
historyModal.querySelector("#history-close").addEventListener("click", () => {
|
historyModal.querySelector("#history-close").addEventListener("click", () => {
|
||||||
historyModal.hidden = true;
|
historyModal.hidden = true;
|
||||||
});
|
});
|
||||||
|
historyModal.querySelector("#history-x").addEventListener("click", () => {
|
||||||
|
historyModal.hidden = true;
|
||||||
|
});
|
||||||
|
historyModal.addEventListener("click", event => {
|
||||||
|
if (event.target === historyModal) historyModal.hidden = true;
|
||||||
|
});
|
||||||
|
|
||||||
// ---- Settings: media storage ----
|
// ---- Settings: media storage ----
|
||||||
|
|
||||||
|
|||||||
@@ -774,11 +774,24 @@ function buildServer(): McpServer {
|
|||||||
.enum(["auto", "slopbot", "puppet"])
|
.enum(["auto", "slopbot", "puppet"])
|
||||||
.optional()
|
.optional()
|
||||||
.describe("auto (default) picks slopbot for its guilds, the client otherwise"),
|
.describe("auto (default) picks slopbot for its guilds, the client otherwise"),
|
||||||
confirm: z.boolean().optional().describe("Accept the near-limit warning")
|
confirm: z.boolean().optional().describe("Accept the near-limit warning"),
|
||||||
|
downloadDir: z
|
||||||
|
.string()
|
||||||
|
.optional()
|
||||||
|
.describe("Absolute destination for downloads; previews stay local")
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
async ({ channelId, guildId, mode, backend, confirm }) =>
|
async ({ channelId, guildId, mode, backend, confirm, downloadDir }) =>
|
||||||
jsonText(await media.startScrape({ channelId, guildId: guildId ?? null, mode, backend, confirm }))
|
jsonText(
|
||||||
|
await media.startScrape({
|
||||||
|
channelId,
|
||||||
|
guildId: guildId ?? null,
|
||||||
|
mode,
|
||||||
|
backend,
|
||||||
|
confirm,
|
||||||
|
downloadDir: downloadDir ?? null
|
||||||
|
})
|
||||||
|
)
|
||||||
);
|
);
|
||||||
|
|
||||||
server.registerTool(
|
server.registerTool(
|
||||||
|
|||||||
+71
-22
@@ -6,7 +6,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { rmSync } from "node:fs";
|
import { rmSync } from "node:fs";
|
||||||
import { join } from "node:path";
|
import { isAbsolute, join } from "node:path";
|
||||||
import {
|
import {
|
||||||
dirSizeBytes,
|
dirSizeBytes,
|
||||||
isNearLimit,
|
isNearLimit,
|
||||||
@@ -49,6 +49,14 @@ export interface StartScrapeInput {
|
|||||||
backend?: "auto" | "slopbot" | "puppet";
|
backend?: "auto" | "slopbot" | "puppet";
|
||||||
/** Set by a re-run of the command after the near-limit warning. */
|
/** Set by a re-run of the command after the near-limit warning. */
|
||||||
confirm?: boolean;
|
confirm?: boolean;
|
||||||
|
/** Absolute destination for downloaded files. Null keeps them under the scrape dir. */
|
||||||
|
downloadDir?: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface MediaErrorEntry {
|
||||||
|
at: string;
|
||||||
|
source: string;
|
||||||
|
message: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface ScrapeBytes {
|
export interface ScrapeBytes {
|
||||||
@@ -138,6 +146,9 @@ export class MediaService {
|
|||||||
private botFetcher: SlopbotFetcher | null = null;
|
private botFetcher: SlopbotFetcher | null = null;
|
||||||
private botStarting: Promise<void> | null = null;
|
private botStarting: Promise<void> | null = null;
|
||||||
|
|
||||||
|
/** The newest bot and scrape errors, capped. Surfaced in the Bot tab. */
|
||||||
|
private readonly errors: MediaErrorEntry[] = [];
|
||||||
|
|
||||||
constructor(private readonly deps: MediaServiceDeps) {
|
constructor(private readonly deps: MediaServiceDeps) {
|
||||||
const number = (value: string | undefined, fallback: number) => {
|
const number = (value: string | undefined, fallback: number) => {
|
||||||
const parsed = Number(value);
|
const parsed = Number(value);
|
||||||
@@ -147,6 +158,7 @@ export class MediaService {
|
|||||||
this.engine = new ScrapeEngine({
|
this.engine = new ScrapeEngine({
|
||||||
rootDir: this.rootDir,
|
rootDir: this.rootDir,
|
||||||
log: line => this.deps.log?.(line),
|
log: line => this.deps.log?.(line),
|
||||||
|
onError: (scrapeId, message) => this.recordError("scrape", `${scrapeId}: ${message}`),
|
||||||
download: {
|
download: {
|
||||||
maxFileBytes: number(process.env.VESKSLOP_MEDIA_MAX_FILE_BYTES, 50 * 1024 * 1024),
|
maxFileBytes: number(process.env.VESKSLOP_MEDIA_MAX_FILE_BYTES, 50 * 1024 * 1024),
|
||||||
previewMaxBytes: number(process.env.VESKSLOP_MEDIA_PREVIEW_MAX_BYTES, 5 * 1024 * 1024),
|
previewMaxBytes: number(process.env.VESKSLOP_MEDIA_PREVIEW_MAX_BYTES, 5 * 1024 * 1024),
|
||||||
@@ -171,7 +183,9 @@ export class MediaService {
|
|||||||
this.deps.log?.("slopbot connected");
|
this.deps.log?.("slopbot connected");
|
||||||
})
|
})
|
||||||
.catch(error => {
|
.catch(error => {
|
||||||
this.deps.log?.(`slopbot login failed: ${String(error)}`);
|
const message = `slopbot login failed: ${String(error)}`;
|
||||||
|
this.deps.log?.(message);
|
||||||
|
this.recordError("slopbot", message);
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -264,6 +278,10 @@ export class MediaService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async startScrape(input: StartScrapeInput): Promise<ScrapeMeta> {
|
async startScrape(input: StartScrapeInput): Promise<ScrapeMeta> {
|
||||||
|
if (input.downloadDir && !isAbsolute(input.downloadDir)) {
|
||||||
|
throw new Error("downloadDir must be an absolute path");
|
||||||
|
}
|
||||||
|
|
||||||
this.applyRetention();
|
this.applyRetention();
|
||||||
|
|
||||||
const status = this.storageStatus();
|
const status = this.storageStatus();
|
||||||
@@ -275,30 +293,38 @@ export class MediaService {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
const mode = input.mode ?? "links";
|
try {
|
||||||
let backend: ScrapeBackend;
|
const mode = input.mode ?? "links";
|
||||||
|
let backend: ScrapeBackend;
|
||||||
|
|
||||||
if (input.backend === "puppet") {
|
if (input.backend === "puppet") {
|
||||||
backend = this.puppetBackend();
|
backend = this.puppetBackend();
|
||||||
} else {
|
} else {
|
||||||
const available = await this.ensureSlopbot();
|
const available = await this.ensureSlopbot();
|
||||||
const inGuild = !!input.guildId && (this.bot?.guilds.cache.has(input.guildId) ?? false);
|
const inGuild = !!input.guildId && (this.bot?.guilds.cache.has(input.guildId) ?? false);
|
||||||
|
|
||||||
if (input.backend === "slopbot" && (!available || !inGuild)) {
|
if (input.backend === "slopbot" && (!available || !inGuild)) {
|
||||||
throw new Error("slopbot is not connected to that guild");
|
throw new Error("slopbot is not connected to that guild");
|
||||||
|
}
|
||||||
|
|
||||||
|
const fetcher = this.botFetcher;
|
||||||
|
backend =
|
||||||
|
available && inGuild && fetcher
|
||||||
|
? new SlopbotBackend(fetcher)
|
||||||
|
: this.puppetBackend();
|
||||||
}
|
}
|
||||||
|
|
||||||
const fetcher = this.botFetcher;
|
return this.engine.start({
|
||||||
backend =
|
channelId: input.channelId,
|
||||||
available && inGuild && fetcher ? new SlopbotBackend(fetcher) : this.puppetBackend();
|
guildId: input.guildId ?? null,
|
||||||
|
mode,
|
||||||
|
backend,
|
||||||
|
downloadDir: input.downloadDir ?? null
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
this.recordError("scrape", error instanceof Error ? error.message : String(error));
|
||||||
|
throw error;
|
||||||
}
|
}
|
||||||
|
|
||||||
return this.engine.start({
|
|
||||||
channelId: input.channelId,
|
|
||||||
guildId: input.guildId ?? null,
|
|
||||||
mode,
|
|
||||||
backend
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
|
|
||||||
stopScrape(id: string): void {
|
stopScrape(id: string): void {
|
||||||
@@ -322,6 +348,10 @@ export class MediaService {
|
|||||||
|
|
||||||
/** Absolute path and content type for a downloaded file, or null. */
|
/** Absolute path and content type for a downloaded file, or null. */
|
||||||
getMediaFile(scrapeId: string, itemId: number): { path: string; contentType: string } | null {
|
getMediaFile(scrapeId: string, itemId: number): { path: string; contentType: string } | null {
|
||||||
|
// A custom destination is the user's filesystem, outside the archive: not served.
|
||||||
|
const meta = this.engine.get(scrapeId);
|
||||||
|
if (meta?.downloadDir) return null;
|
||||||
|
|
||||||
const item = this.getMediaItem(scrapeId, itemId);
|
const item = this.getMediaItem(scrapeId, itemId);
|
||||||
if (!item?.file_path) return null;
|
if (!item?.file_path) return null;
|
||||||
|
|
||||||
@@ -343,6 +373,22 @@ export class MediaService {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private recordError(source: string, message: string): void {
|
||||||
|
this.errors.unshift({ at: new Date().toISOString(), source, message });
|
||||||
|
if (this.errors.length > 50) this.errors.length = 50;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Bot and scrape errors, newest first. */
|
||||||
|
listErrors(): MediaErrorEntry[] {
|
||||||
|
return [...this.errors];
|
||||||
|
}
|
||||||
|
|
||||||
|
clearErrors(): number {
|
||||||
|
const count = this.errors.length;
|
||||||
|
this.errors.length = 0;
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
/** Distinct posters across every scrape, biggest first. */
|
/** Distinct posters across every scrape, biggest first. */
|
||||||
listAuthors(): Array<{ id: string | null; name: string | null; count: number }> {
|
listAuthors(): Array<{ id: string | null; name: string | null; count: number }> {
|
||||||
const totals = new Map<string, { id: string | null; name: string | null; count: number }>();
|
const totals = new Map<string, { id: string | null; name: string | null; count: number }>();
|
||||||
@@ -434,7 +480,10 @@ export class MediaService {
|
|||||||
height: row.height,
|
height: row.height,
|
||||||
aspectRatio: row.aspect_ratio,
|
aspectRatio: row.aspect_ratio,
|
||||||
url: row.url,
|
url: row.url,
|
||||||
localUrl: row.file_path ? `/api/media/${meta.id}/${row.id}/file` : null,
|
localUrl:
|
||||||
|
row.file_path && !meta.downloadDir
|
||||||
|
? `/api/media/${meta.id}/${row.id}/file`
|
||||||
|
: null,
|
||||||
thumbUrl: row.preview_hash ? `/api/media/${meta.id}/${row.id}/thumb` : null,
|
thumbUrl: row.preview_hash ? `/api/media/${meta.id}/${row.id}/thumb` : null,
|
||||||
provider: row.provider,
|
provider: row.provider,
|
||||||
downloaded: row.downloaded === 1,
|
downloaded: row.downloaded === 1,
|
||||||
|
|||||||
+23
-1
@@ -832,6 +832,20 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
|||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
|
||||||
|
app.get(
|
||||||
|
"/api/media/errors",
|
||||||
|
wrap(async (_req, res) => {
|
||||||
|
res.json({ ok: true, result: deps.media.listErrors() });
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
app.post(
|
||||||
|
"/api/media/errors/clear",
|
||||||
|
wrap(async (_req, res) => {
|
||||||
|
res.json({ ok: true, result: { cleared: deps.media.clearErrors() } });
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
app.post(
|
app.post(
|
||||||
"/api/media/scrape",
|
"/api/media/scrape",
|
||||||
wrap(async (req, res) => {
|
wrap(async (req, res) => {
|
||||||
@@ -844,8 +858,16 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
|
|||||||
const requested = String(req.body?.backend ?? "auto");
|
const requested = String(req.body?.backend ?? "auto");
|
||||||
const backend =
|
const backend =
|
||||||
requested === "slopbot" || requested === "puppet" ? requested : "auto";
|
requested === "slopbot" || requested === "puppet" ? requested : "auto";
|
||||||
|
const downloadDir = req.body?.downloadDir ? String(req.body.downloadDir) : null;
|
||||||
|
|
||||||
const scrape = await deps.media.startScrape({ channelId, guildId, mode, confirm, backend });
|
const scrape = await deps.media.startScrape({
|
||||||
|
channelId,
|
||||||
|
guildId,
|
||||||
|
mode,
|
||||||
|
confirm,
|
||||||
|
backend,
|
||||||
|
downloadDir
|
||||||
|
});
|
||||||
deps.getPuppet()?.appendAction({
|
deps.getPuppet()?.appendAction({
|
||||||
type: "media",
|
type: "media",
|
||||||
action: "scrape",
|
action: "scrape",
|
||||||
|
|||||||
Reference in New Issue
Block a user