Add media storage limits, retention, and the near-limit flow

This commit is contained in:
Fisto
2026-09-16 08:49:42 +02:00
parent 6b961ddcef
commit 8f3f153c55
15 changed files with 394 additions and 12 deletions
+2 -1
View File
@@ -30,7 +30,8 @@ mobile draft at 390px.
- [x] C4 slopbot backend, tests - [x] C4 slopbot backend, tests
- [x] C5 daemon wiring: `scrape media` command, MediaService, `/api/media/*` - [x] C5 daemon wiring: `scrape media` command, MediaService, `/api/media/*`
- [x] C6 Media and Bot tabs in the Web UI - [x] C6 Media and Bot tabs in the Web UI
- [ ] C7 storage limit, warning flow, clean modal, settings - [x] C7a storage limits, retention, warning/confirm flow, storage and clean API
- [ ] C7b Settings group: limit field with info hover, usage bar, clean modal
- [ ] C8 MCP service hooks - [ ] C8 MCP service hooks
- [ ] C9 live verification, tuning, docs - [ ] C9 live verification, tuning, docs
- Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each - Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
+3
View File
@@ -58,6 +58,9 @@ Ordered backlog. `CURRENT.md` tracks the live session state.
10. Mobile web UI: first draft at 390px with the four tabs (Users, Chat, Call, Activity) is 10. Mobile web UI: first draft at 390px with the four tabs (Users, Chat, Call, Activity) is
implemented and rendered headless. Next: move the transcript into the Call tab, put the TTS implemented and rendered headless. Next: move the transcript into the Call tab, put the TTS
box in the Call tab as in 1c, and verify with real data on a phone-width viewport. box in the Call tab as in 1c, and verify with real data on a phone-width viewport.
11. TTS VOX preference: a `tts.vox` toggle that exports `VESKSLOP_TTS_VOX` (like `words`), an
optional `VOX_CLEAN`, and a help line in the TTS panel. Requested in `TTS-DESIGN.md`; the
wrapper side is ready and verified. Inline `/rate` and `/pitch` need no UI.
## Web UI follow-ups ## Web UI follow-ups
+8 -3
View File
@@ -1,8 +1,8 @@
# Media archive # Media archive
Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command, Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
and the `/api/media/*` routes are implemented (C3-C5). The Media and Bot tabs are C6. Checkpoints the `/api/media/*` routes, and storage limits with retention are implemented (C3-C7a). The Media
live in [CURRENT.md](../CURRENT.md). and Bot tabs are C6; the Settings group is C7b. Checkpoints live in [CURRENT.md](../CURRENT.md).
The media archive saves the media history of a channel. A trusted user runs `scrape media` in the The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
channel. The daemon walks the history through a scrape backend and indexes every image, GIF, channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
@@ -101,12 +101,17 @@ The storage limit covers everything under the DB root: databases, downloads, and
Behavior: Behavior:
1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the 1. At or above the warn threshold, a new `scrape media` command is refused and the reply shows the
usage. Re-running the same command accepts. The confirmation lives for a short window. usage. Re-running the same command within 2 minutes accepts. The API mirror is `confirm: true`
on `POST /api/media/scrape` (otherwise `409 MEDIA_LIMIT`).
2. At or above 100% the oldest completed scrapes are marked `pending-delete`. 2. At or above 100% the oldest completed scrapes are marked `pending-delete`.
3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits. 3. Above limit + 5% overrun, marked scrapes are deleted oldest first until the usage fits.
4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field) 4. No data is deleted inside the 5% overrun band. The hover info (small `i` beside the limit field)
explains this. explains this.
The planning is a pure function (`media-scrape/src/retention.ts`) with tests. Only `done`,
`failed`, and `paused` scrapes are ever marked; a running scrape is untouched. Deletion removes the
scrape directory and forgets it in the engine.
**Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states **Clean media** in Settings lists exactly what will be deleted, with sizes and counts, and states
that deletion is irreversible. The delete button is disabled for a 5 second countdown. that deletion is irreversible. The delete button is disabled for a 5 second countdown.
+3 -1
View File
@@ -249,8 +249,10 @@ replaceable.
| GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first | | GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first |
| GET | `/api/media/scrapes` | none | Every scrape with state and counters | | GET | `/api/media/scrapes` | none | Every scrape with state and counters |
| GET | `/api/media/status` | none | Slopbot connection state and the scrape list | | GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?` | Start a media scrape | | GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?`, `confirm?` | Start a media scrape. Near the limit: `409` with code `MEDIA_LIMIT` unless `confirm: true` |
| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape | | POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
| POST | `/api/media/clean` | `ids?` | Delete the given scrapes, or every scrape when `ids` is absent |
| GET | `/api/logs` | `limit` | The persistent action log | | GET | `/api/logs` | `limit` | The persistent action log |
| GET | `/api/events` | none | Server-sent events | | GET | `/api/events` | none | Server-sent events |
+20 -1
View File
@@ -18,7 +18,8 @@ import {
type BackendName, type BackendName,
type CoreMessage, type CoreMessage,
type ScrapeMeta, type ScrapeMeta,
type ScrapeMode type ScrapeMode,
type ScrapeState
} from "./types.js"; } from "./types.js";
export interface FetchPage { export interface FetchPage {
@@ -146,6 +147,24 @@ export class ScrapeEngine {
this.stopRequested.add(id); this.stopRequested.add(id);
} }
/** Persist a state change made outside a job, such as retention marking. */
setState(id: string, state: ScrapeState): ScrapeMeta | null {
const meta = this.active.get(id) ?? readScrapeMeta(join(this.options.rootDir, id));
if (!meta) return null;
meta.state = state;
meta.updatedAt = new Date(this.now()).toISOString();
writeScrapeMeta(join(this.options.rootDir, id), meta);
this.active.set(id, meta);
return { ...meta };
}
/** Drop a deleted scrape from the in-memory list. */
forget(id: string): void {
this.active.delete(id);
this.stopRequested.delete(id);
}
private async runJob(job: QueuedJob): Promise<void> { private async runJob(job: QueuedJob): Promise<void> {
const { meta, dir, backend } = job; const { meta, dir, backend } = job;
const store = new ScrapeStore(dir); const store = new ScrapeStore(dir);
+1
View File
@@ -4,6 +4,7 @@
*/ */
export { extractMedia, type ExtractedMedia } from "./extract.js"; export { extractMedia, type ExtractedMedia } from "./extract.js";
export { isNearLimit, planRetention, type RetentionPlan, type RetentionScrape } from "./retention.js";
export { export {
ScrapeEngine, ScrapeEngine,
type FetchPage, type FetchPage,
+65
View File
@@ -0,0 +1,65 @@
/*
* Pure retention planning for the media archive. Rules: docs/media.md.
*/
export interface RetentionScrape {
id: string;
state: string;
createdAt: string;
bytes: number;
}
export interface RetentionPlan {
/** Completed scrapes to mark pending-delete so the usage can fit the limit. */
mark: string[];
/** Marked scrapes to delete now: the usage is past the overrun allowance. */
delete: string[];
totalBytes: number;
}
/** States that may be marked for deletion. A running scrape is never touched. */
const MARKABLE = new Set(["done", "failed", "paused"]);
export function isNearLimit(usedBytes: number, limitBytes: number, warnPercent: number): boolean {
return usedBytes >= limitBytes * (warnPercent / 100);
}
export function planRetention(
scrapes: RetentionScrape[],
limitBytes: number,
overrunPercent: number
): RetentionPlan {
const totalBytes = scrapes.reduce((sum, scrape) => sum + scrape.bytes, 0);
const oldest = [...scrapes].sort((a, b) => (a.createdAt < b.createdAt ? -1 : 1));
const overrunLimit = limitBytes * (1 + overrunPercent / 100);
// Past the allowance: delete marked scrapes, oldest first, until the usage fits.
if (totalBytes > overrunLimit) {
const remove: string[] = [];
let planned = totalBytes;
for (const scrape of oldest) {
if (planned <= limitBytes) break;
if (scrape.state !== "pending-delete") continue;
remove.push(scrape.id);
planned -= scrape.bytes;
}
return { mark: [], delete: remove, totalBytes };
}
if (totalBytes <= limitBytes) return { mark: [], delete: [], totalBytes };
// Between the limit and the allowance: mark oldest completed scrapes, no deletion.
const mark: string[] = [];
let planned = totalBytes;
for (const scrape of oldest) {
if (planned <= limitBytes) break;
if (!MARKABLE.has(scrape.state)) continue;
mark.push(scrape.id);
planned -= scrape.bytes;
}
return { mark, delete: [], totalBytes };
}
+13
View File
@@ -216,6 +216,19 @@ export class ScrapeStore {
return added; return added;
} }
/** Sum of the recorded source sizes per kind. Used by the storage bar. */
sumBytesByKind(): Record<MediaKind, number> {
const rows = this.db
.prepare("SELECT kind, COALESCE(SUM(size), 0) AS bytes FROM media GROUP BY kind")
.all() as unknown as Array<{ kind: string; bytes: number }>;
const result = emptyMediaCounts();
for (const row of rows) {
if (row.kind in result) result[row.kind as MediaKind] = row.bytes;
}
return result;
}
counts(): Record<MediaKind, number> { counts(): Record<MediaKind, number> {
const rows = this.db const rows = this.db
.prepare("SELECT kind, COUNT(*) AS n FROM media GROUP BY kind") .prepare("SELECT kind, COUNT(*) AS n FROM media GROUP BY kind")
+41 -1
View File
@@ -10,6 +10,7 @@ import { join } from "node:path";
import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js"; import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js";
import { extractMedia } from "../src/extract.js"; import { extractMedia } from "../src/extract.js";
import { PuppetBackend } from "../src/puppet-backend.js"; import { PuppetBackend } from "../src/puppet-backend.js";
import { isNearLimit, planRetention } from "../src/retention.js";
import { dirSizeBytes, listScrapes, ScrapeStore } from "../src/store.js"; import { dirSizeBytes, listScrapes, ScrapeStore } from "../src/store.js";
import type { CoreAttachment, CoreMessage } from "../src/types.js"; import type { CoreAttachment, CoreMessage } from "../src/types.js";
@@ -173,7 +174,9 @@ void (async () => {
const backend = new FakeBackend([ const backend = new FakeBackend([
{ {
messages: [ messages: [
message("100", { attachments: [attachment({ filename: "one.png", contentType: "image/png" })] }), message("100", {
attachments: [attachment({ filename: "one.png", contentType: "image/png", size: 10 })]
}),
message("101") message("101")
] ]
}, },
@@ -202,6 +205,7 @@ void (async () => {
assert.equal(stored.counts().image, 1); assert.equal(stored.counts().image, 1);
assert.equal(stored.queryMedia({ kind: "image" }).length, 1); assert.equal(stored.queryMedia({ kind: "image" }).length, 1);
assert.equal(stored.queryMedia({ kind: "video" }).length, 0); assert.equal(stored.queryMedia({ kind: "video" }).length, 0);
assert.equal(stored.sumBytesByKind().image, 10);
stored.close(); stored.close();
assert.equal(listScrapes(root).length, 1); assert.equal(listScrapes(root).length, 1);
@@ -272,6 +276,42 @@ void (async () => {
assert.deepEqual(calls[0], { channelId: "123", limit: 5, before: "60" }); assert.deepEqual(calls[0], { channelId: "123", limit: 5, before: "60" });
}); });
await test("retention marks the oldest completed scrapes up to the limit", () => {
const scrapes = [
{ id: "old", state: "done", createdAt: "2026-09-01T00:00:00.000Z", bytes: 600 },
{ id: "new", state: "done", createdAt: "2026-09-02T00:00:00.000Z", bytes: 600 },
{ id: "run", state: "running", createdAt: "2026-09-03T00:00:00.000Z", bytes: 600 }
];
const plan = planRetention(scrapes, 1000, 100);
assert.deepEqual(plan.mark, ["old", "new"]);
assert.deepEqual(plan.delete, []);
});
await test("retention deletes marked scrapes past the overrun allowance", () => {
const scrapes = [
{ id: "old", state: "pending-delete", createdAt: "2026-09-01T00:00:00.000Z", bytes: 600 },
{ id: "new", state: "pending-delete", createdAt: "2026-09-02T00:00:00.000Z", bytes: 600 },
{ id: "run", state: "running", createdAt: "2026-09-03T00:00:00.000Z", bytes: 600 }
];
const plan = planRetention(scrapes, 1000, 5);
assert.deepEqual(plan.delete, ["old", "new"]);
assert.deepEqual(plan.mark, []);
});
await test("retention leaves usage under the limit alone", () => {
const plan = planRetention(
[{ id: "one", state: "done", createdAt: "2026-09-01T00:00:00.000Z", bytes: 500 }],
1000,
5
);
assert.deepEqual(plan, { mark: [], delete: [], totalBytes: 500 });
assert.equal(isNearLimit(900, 1000, 90), true);
assert.equal(isNearLimit(899, 1000, 90), false);
});
rmSync(root, { recursive: true, force: true }); rmSync(root, { recursive: true, force: true });
if (failures > 0) { if (failures > 0) {
+1
View File
@@ -8,5 +8,6 @@ setup/out-*.reg
*.wav *.wav
# Generated per-checkout configuration. # Generated per-checkout configuration.
setup/microslopsam.env setup/microslopsam.env
__pycache__/
# Engine DLLs and voice data are proprietary; never commit them. # Engine DLLs and voice data are proprietary; never commit them.
third_party/ third_party/
+1
View File
@@ -43,6 +43,7 @@ let connecting = false;
const media = new MediaService({ const media = new MediaService({
getCore: () => core, getCore: () => core,
isCoreConnected: () => coreConnected, isCoreConnected: () => coreConnected,
getMediaPrefs: () => preferences.get().media,
log: line => console.log(`[media] ${line}`) log: line => console.log(`[media] ${line}`)
}); });
+129
View File
@@ -5,7 +5,11 @@
* (later) MCP tools call. * (later) MCP tools call.
*/ */
import { rmSync } from "node:fs";
import { import {
dirSizeBytes,
isNearLimit,
planRetention,
ScrapeEngine, ScrapeEngine,
ScrapeStore, ScrapeStore,
scrapeDir, scrapeDir,
@@ -25,12 +29,14 @@ import {
type SlopbotStatus type SlopbotStatus
} from "../../media-scrape/slopbot/index.js"; } from "../../media-scrape/slopbot/index.js";
import type { CoreClient } from "./core-client.js"; import type { CoreClient } from "./core-client.js";
import type { MediaPreferences } from "./preferences.js";
type BotClient = Awaited<ReturnType<typeof startSlopbot>>; type BotClient = Awaited<ReturnType<typeof startSlopbot>>;
export interface MediaServiceDeps { export interface MediaServiceDeps {
getCore: () => CoreClient | undefined; getCore: () => CoreClient | undefined;
isCoreConnected: () => boolean; isCoreConnected: () => boolean;
getMediaPrefs: () => MediaPreferences;
log?: (line: string) => void; log?: (line: string) => void;
} }
@@ -39,6 +45,44 @@ export interface StartScrapeInput {
guildId?: string | null; guildId?: string | null;
mode?: "links" | "download"; mode?: "links" | "download";
backend?: "auto" | "slopbot" | "puppet"; backend?: "auto" | "slopbot" | "puppet";
/** Set by a re-run of the command after the near-limit warning. */
confirm?: boolean;
}
export interface ScrapeBytes {
id: string;
channelId: string;
state: string;
createdAt: string;
bytes: number;
}
export interface MediaStorageStatus {
usedBytes: number;
limitBytes: number;
warnPercent: number;
overrunPercent: number;
nearLimit: boolean;
byKind: Record<MediaKind, number>;
scrapes: ScrapeBytes[];
}
/** Thrown when a scrape starts near the storage limit without confirmation. */
export class MediaLimitError extends Error {
readonly code = "MEDIA_LIMIT";
constructor(
message: string,
readonly usedBytes: number,
readonly limitBytes: number
) {
super(message);
}
}
function formatBytes(bytes: number): string {
const gib = bytes / 1024 ** 3;
return gib >= 1 ? `${gib.toFixed(2)} GiB` : `${Math.round(bytes / 1024 ** 2)} MiB`;
} }
export interface MediaItemResult { export interface MediaItemResult {
@@ -104,7 +148,92 @@ export class MediaService {
}); });
} }
/** Combined size of every scrape directory, with the limit settings applied. */
storageStatus(): MediaStorageStatus {
const prefs = this.deps.getMediaPrefs();
const byKind: Record<MediaKind, number> = { image: 0, gif: 0, video: 0, audio: 0 };
const scrapes = this.engine.list().map(meta => {
const dir = scrapeDir(this.rootDir, meta.id);
const bytes = dirSizeBytes(dir);
try {
const store = new ScrapeStore(dir);
try {
const sums = store.sumBytesByKind();
for (const kind of Object.keys(byKind) as MediaKind[]) byKind[kind] += sums[kind];
} finally {
store.close();
}
} catch {
// an unreadable index counts as zero bytes
}
return { id: meta.id, channelId: meta.channelId, state: meta.state, createdAt: meta.createdAt, bytes };
});
const usedBytes = scrapes.reduce((sum, scrape) => sum + scrape.bytes, 0);
return {
usedBytes,
limitBytes: prefs.limitBytes,
warnPercent: prefs.warnPercent,
overrunPercent: prefs.overrunPercent,
nearLimit: isNearLimit(usedBytes, prefs.limitBytes, prefs.warnPercent),
byKind,
scrapes
};
}
/** Mark the oldest completed scrapes when the usage is at or over the limit. */
refreshRetention(): MediaStorageStatus {
const status = this.storageStatus();
const plan = planRetention(status.scrapes, status.limitBytes, status.overrunPercent);
for (const id of plan.mark) this.engine.setState(id, "pending-delete");
return this.storageStatus();
}
private applyRetention(): void {
const status = this.storageStatus();
const plan = planRetention(status.scrapes, status.limitBytes, status.overrunPercent);
for (const id of plan.mark) this.engine.setState(id, "pending-delete");
for (const id of plan.delete) this.deleteScrape(id);
}
deleteScrape(id: string): boolean {
try {
rmSync(scrapeDir(this.rootDir, id), { recursive: true, force: true });
this.engine.forget(id);
this.deps.log?.(`deleted scrape ${id}`);
return true;
} catch (error) {
this.deps.log?.(`delete failed for ${id}: ${String(error)}`);
return false;
}
}
/** Delete the given scrapes, or every scrape when no IDs are given. */
cleanMedia(ids?: string[]): number {
const targets = ids?.length ? ids : this.engine.list().map(meta => meta.id);
let deleted = 0;
for (const id of targets) {
if (this.deleteScrape(id)) deleted++;
}
return deleted;
}
async startScrape(input: StartScrapeInput): Promise<ScrapeMeta> { async startScrape(input: StartScrapeInput): Promise<ScrapeMeta> {
this.applyRetention();
const status = this.storageStatus();
if (status.nearLimit && !input.confirm) {
throw new MediaLimitError(
`media storage is at ${formatBytes(status.usedBytes)} of ${formatBytes(status.limitBytes)}`,
status.usedBytes,
status.limitBytes
);
}
const mode = input.mode ?? "links"; const mode = input.mode ?? "links";
let backend: ScrapeBackend; let backend: ScrapeBackend;
+42 -1
View File
@@ -76,12 +76,22 @@ export interface AudioPreferences {
duck: boolean; duck: boolean;
} }
export interface MediaPreferences {
/** Combined size limit for the media DB root: databases, downloads, previews. */
limitBytes: number;
/** Warn threshold in percent, 1-100. */
warnPercent: number;
/** No-delete overrun allowance in percent, 0-100. */
overrunPercent: number;
}
export interface WebPreferences { export interface WebPreferences {
/** null means every guild is visible. An empty array hides all. */ /** null means every guild is visible. An empty array hides all. */
visibleGuilds: string[] | null; visibleGuilds: string[] | null;
tts: TtsPreferences; tts: TtsPreferences;
audio: AudioPreferences; audio: AudioPreferences;
favourites: FavouritesPreferences; favourites: FavouritesPreferences;
media: MediaPreferences;
} }
const DEFAULTS: WebPreferences = { const DEFAULTS: WebPreferences = {
@@ -101,9 +111,35 @@ const DEFAULTS: WebPreferences = {
wordGapMs: null wordGapMs: null
}, },
audio: { duck: true }, audio: { duck: true },
favourites: { users: [], channels: [] } favourites: { users: [], channels: [] },
media: {
limitBytes: 30 * 1024 * 1024 * 1024,
warnPercent: 90,
overrunPercent: 5
}
}; };
const MEDIA_LIMIT_MIN = 256 * 1024 * 1024;
const MEDIA_LIMIT_MAX = 10 * 1024 * 1024 * 1024 * 1024;
function normalizeMedia(raw: any, fallback: MediaPreferences): MediaPreferences {
const limit = Number(raw?.limitBytes);
const warn = Number(raw?.warnPercent);
const overrun = Number(raw?.overrunPercent);
return {
limitBytes:
Number.isFinite(limit) && limit >= MEDIA_LIMIT_MIN && limit <= MEDIA_LIMIT_MAX
? Math.round(limit)
: fallback.limitBytes,
warnPercent: Number.isFinite(warn) && warn >= 1 && warn <= 100 ? Math.round(warn) : fallback.warnPercent,
overrunPercent:
Number.isFinite(overrun) && overrun >= 0 && overrun <= 100
? Math.round(overrun)
: fallback.overrunPercent
};
}
export interface PreferencesOptions { export interface PreferencesOptions {
/** The old state file that may still hold preferences. */ /** The old state file that may still hold preferences. */
legacyFile?: string; legacyFile?: string;
@@ -199,6 +235,10 @@ export class PreferencesStore {
}; };
} }
if (patch.media !== undefined) {
this.preferences.media = normalizeMedia(patch.media, this.preferences.media);
}
if (patch.favourites !== undefined) { if (patch.favourites !== undefined) {
this.preferences.favourites = { this.preferences.favourites = {
users: Array.isArray(patch.favourites?.users) users: Array.isArray(patch.favourites?.users)
@@ -255,6 +295,7 @@ export class PreferencesStore {
audio: { audio: {
duck: raw?.audio?.duck !== false duck: raw?.audio?.duck !== false
}, },
media: normalizeMedia(raw?.media, DEFAULTS.media),
favourites: { favourites: {
users: Array.isArray(raw?.favourites?.users) users: Array.isArray(raw?.favourites?.users)
? [...new Set<string>(raw.favourites.users.map((id: unknown) => String(id)))].slice(0, 100) ? [...new Set<string>(raw.favourites.users.map((id: unknown) => String(id)))].slice(0, 100)
+19 -1
View File
@@ -123,6 +123,7 @@ export interface PuppetMedia {
channelId: string; channelId: string;
guildId: string | null; guildId: string | null;
mode: "links" | "download"; mode: "links" | "download";
confirm?: boolean;
}): Promise<{ id: string }>; }): Promise<{ id: string }>;
stopScrape(id: string): void; stopScrape(id: string): void;
listScrapes(): PuppetScrape[]; listScrapes(): PuppetScrape[];
@@ -174,6 +175,8 @@ export class Puppet {
private readonly lastFetchAt = new Map<string, number>(); private readonly lastFetchAt = new Map<string, number>();
private readonly pending = new Map<string, { entryId: string; expiresAt: number }>(); private readonly pending = new Map<string, { entryId: string; expiresAt: number }>();
private readonly polls = new Map<string, PollState>(); private readonly polls = new Map<string, PollState>();
/** Near-limit scrape warnings waiting for the same user to re-run the command. */
private readonly scrapeConfirm = new Map<string, number>();
private voiceChannelId?: string | null; private voiceChannelId?: string | null;
private ownerVoiceChannelId?: string | null; private ownerVoiceChannelId?: string | null;
@@ -753,11 +756,26 @@ export class Puppet {
return this.respond(authorId, channelId, "scrape", "Download mode needs the admin tier."); return this.respond(authorId, channelId, "scrape", "Download mode needs the admin tier.");
} }
const key = `${channelId}:${authorId}`;
const confirmUntil = this.scrapeConfirm.get(key);
const confirm = confirmUntil !== undefined && Date.now() < confirmUntil;
if (confirm) this.scrapeConfirm.delete(key);
try { try {
const scrape = await this.media.startScrape({ channelId, guildId: guildId ?? null, mode }); const scrape = await this.media.startScrape({ channelId, guildId: guildId ?? null, mode, confirm });
this.actionLog.append({ type: "media", action: "scrape", channelId, mode, scrapeId: scrape.id }); this.actionLog.append({ type: "media", action: "scrape", channelId, mode, scrapeId: scrape.id });
return this.respond(authorId, channelId, "scrape", `Media scrape started (${mode}): ${scrape.id}.`); return this.respond(authorId, channelId, "scrape", `Media scrape started (${mode}): ${scrape.id}.`);
} catch (error) { } catch (error) {
if ((error as { code?: string }).code === "MEDIA_LIMIT" && !confirm) {
this.scrapeConfirm.set(key, Date.now() + 120_000);
return this.respond(
authorId,
channelId,
"scrape",
`${(error as Error).message}. Run the command again within 2 minutes to accept.`
);
}
this.actionLog.append({ type: "error", scope: "media", channelId, message: String(error) }); this.actionLog.append({ type: "error", scope: "media", channelId, message: String(error) });
return this.respond( return this.respond(
authorId, authorId,
+46 -3
View File
@@ -20,7 +20,7 @@ import type { AuthService, AuthStatus } from "./auth-service.js";
import type { AudioService } from "./audio-service.js"; import type { AudioService } from "./audio-service.js";
import { CoreError, type CoreClient } from "./core-client.js"; import { CoreError, type CoreClient } from "./core-client.js";
import type { CoreProcess } from "./core-process.js"; import type { CoreProcess } from "./core-process.js";
import type { MediaService } from "./media.js"; import { MediaLimitError, type MediaService } from "./media.js";
import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js"; import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js";
import type { PreferencesStore, WebPreferences } from "./preferences.js"; import type { PreferencesStore, WebPreferences } from "./preferences.js";
import type { Puppet } from "./puppet.js"; import type { Puppet } from "./puppet.js";
@@ -206,6 +206,16 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
return; return;
} }
if (error instanceof MediaLimitError) {
res.status(409).json({
ok: false,
code: error.code,
error: error.message,
result: { usedBytes: error.usedBytes, limitBytes: error.limitBytes }
});
return;
}
const status = error instanceof HttpError ? error.status : 500; const status = error instanceof HttpError ? error.status : 500;
res.status(status).json({ ok: false, error: String(error?.message ?? error) }); res.status(status).json({ ok: false, error: String(error?.message ?? error) });
}); });
@@ -403,7 +413,7 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
app.post( app.post(
"/api/preferences", "/api/preferences",
wrap(async (req, res) => { wrap(async (req, res) => {
const { visibleGuilds, tts, audio, favourites } = req.body ?? {}; const { visibleGuilds, tts, audio, favourites, media: mediaPrefs } = req.body ?? {};
if (visibleGuilds !== undefined && visibleGuilds !== null && !Array.isArray(visibleGuilds)) { if (visibleGuilds !== undefined && visibleGuilds !== null && !Array.isArray(visibleGuilds)) {
throw new HttpError(400, "visibleGuilds must be an array of guild IDs or null"); throw new HttpError(400, "visibleGuilds must be an array of guild IDs or null");
@@ -438,6 +448,21 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
}; };
} }
if (mediaPrefs !== undefined) {
const current = deps.preferences.get().media;
patch.media = {
limitBytes: Number.isFinite(Number(mediaPrefs?.limitBytes))
? Number(mediaPrefs.limitBytes)
: current.limitBytes,
warnPercent: Number.isFinite(Number(mediaPrefs?.warnPercent))
? Number(mediaPrefs.warnPercent)
: current.warnPercent,
overrunPercent: Number.isFinite(Number(mediaPrefs?.overrunPercent))
? Number(mediaPrefs.overrunPercent)
: current.overrunPercent
};
}
if (audio !== undefined) { if (audio !== undefined) {
patch.audio = { duck: audio?.duck !== false }; patch.audio = { duck: audio?.duck !== false };
deps.audio.setDuckEnabled(patch.audio.duck); deps.audio.setDuckEnabled(patch.audio.duck);
@@ -729,6 +754,23 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
}) })
); );
app.get(
"/api/media/storage",
wrap(async (_req, res) => {
res.json({ ok: true, result: deps.media.refreshRetention() });
})
);
app.post(
"/api/media/clean",
wrap(async (req, res) => {
const ids = Array.isArray(req.body?.ids) ? req.body.ids.map(String) : undefined;
const deleted = deps.media.cleanMedia(ids);
deps.getPuppet()?.appendAction({ type: "media", action: "clean", deleted });
res.json({ ok: true, result: { deleted } });
})
);
app.get( app.get(
"/api/media", "/api/media",
wrap(async (req, res) => { wrap(async (req, res) => {
@@ -750,8 +792,9 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
const mode = req.body?.mode === "download" ? "download" : "links"; const mode = req.body?.mode === "download" ? "download" : "links";
const guildId = req.body?.guildId ? String(req.body.guildId) : null; const guildId = req.body?.guildId ? String(req.body.guildId) : null;
const confirm = req.body?.confirm === true;
const scrape = await deps.media.startScrape({ channelId, guildId, mode }); const scrape = await deps.media.startScrape({ channelId, guildId, mode, confirm });
deps.getPuppet()?.appendAction({ deps.getPuppet()?.appendAction({
type: "media", type: "media",
action: "scrape", action: "scrape",