Wire the media archive into the daemon: command, service, and API

CURRENT.md

@@ -19,19 +19,20 @@ mobile draft at 390px.
   bridge for group DMs; one self-contained SQLite + media dir per scrape under `media-scrape/db/`
   (gitignored); links indexed by default, downloads optional, third-party embeds never downloaded;
   30 GiB default limit with a 90% warning and 5% no-delete overrun, usage bar and clean button in
-  Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C7. The slopbot scaffold
+  Settings → Bot; Media and Bot top-level tabs; MCP hooks designed for C8. The slopbot scaffold
-  (config loader, invite helper) and its discord.js dependency were added 2026-09-16 and stay
+  (config loader, invite helper) and its discord.js dependency were added 2026-09-16. C5 smoke
-  unused until C4.
+  test: `/api/media/status` and `/api/media` answer on a daemon with no core.
 
   Checkpoints, push after each:
   - [x] C1 design docs, AGENTS/INDEX pointers, `.gitignore`
   - [x] C2 core message shape: embeds, forwards, attachment metadata
   - [x] C3 media-scrape store and puppet walker, tests
   - [x] C4 slopbot backend, tests
-  - [ ] C5 Web UI Media and Bot tabs, API routes
+  - [x] C5 daemon wiring: `scrape media` command, MediaService, `/api/media/*`
-  - [ ] C6 storage limit, warning flow, clean modal, settings
+  - [ ] C6 Media and Bot tabs in the Web UI
-  - [ ] C7 MCP service hooks
+  - [ ] C7 storage limit, warning flow, clean modal, settings
-  - [ ] C8 live verification, tuning, docs
+  - [ ] C8 MCP service hooks
+  - [ ] C9 live verification, tuning, docs
 - Flextalk voice profile, end to end. The profile window is a movable panel that recenters on each
   open; it carries the character sliders with engine defaults and +/-2 snapping, five named
   presets, radio-kit effects (high-pass 40-400 Hz, echo 0-100, word mode with a gap in ms), and

docs/communication.md

@@ -181,6 +181,7 @@ event. The log rotates at the size limit.
 | `command` | Command text, requester, result |
 | `reply` | Command, mode, target, result, error |
 | `voice` | Voice action, channel, mute |
+| `media` | Scrape start or stop: scrape ID, channel, mode |
 | `audio` | Audio action (speak, stop, devices, capture), provider, size |
 | `message-edit` | Message ID, channel, author, content length, whether the cache had it |
 | `message-delete` | Message ID, channel, whether the cache had it |

docs/media-mcp.md

@@ -4,7 +4,7 @@ The media archive is designed for agent access. The daemon exposes one service,
 are thin wrappers around it. The same service backs the Web UI routes, so the two clients cannot
 drift.
 
-Status: hooks designed, implementation at checkpoint C7 in [CURRENT.md](../CURRENT.md).
+Status: hooks designed, implementation at checkpoint C8 in [CURRENT.md](../CURRENT.md).
 
 ## Service interface
 
@@ -54,7 +54,7 @@ the action log apply.
 - Media bytes are never returned inline. Tools return metadata, the original URL, and the local
   path; an agent that wants the bytes reads the file through the daemon host.
 
-## Hooks to leave in place before C7
+## Hooks to leave in place before C8
 
 1. `MediaService` is constructed once in the daemon and passed to both the Web UI routes and the
    MCP registration. No globals.

docs/media-scraper.md

@@ -111,4 +111,5 @@ and the MCP `get_media_scrape` tool read the same fields.
 
 ## Checkpoints
 
-Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4.
+Implementation goals live in [CURRENT.md](../CURRENT.md). The engine is C3; slopbot is C4; daemon
+wiring (command, service, routes) is C5.

docs/media.md

@@ -1,7 +1,8 @@
 # Media archive
 
-Status: design approved 2026-09-16. Implementation in progress. Checkpoints live in
+Status: design approved 2026-09-16. The engine, the slopbot backend, the `scrape media` command,
-[CURRENT.md](../CURRENT.md).
+and the `/api/media/*` routes are implemented (C3-C5). The Media and Bot tabs are C6. Checkpoints
+live in [CURRENT.md](../CURRENT.md).
 
 The media archive saves the media history of a channel. A trusted user runs `scrape media` in the
 channel. The daemon walks the history through a scrape backend and indexes every image, GIF,
@@ -125,7 +126,7 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
 ## MCP
 
 The agent can query the archive and start scrapes with MCP tools. The hooks are designed in
-[media-mcp.md](media-mcp.md) and implemented at checkpoint C7.
+[media-mcp.md](media-mcp.md) and implemented at checkpoint C8.
 
 ## Security
 

docs/slopbot.md

@@ -75,6 +75,6 @@ Slopbot runs with the faster bot budget; the puppet backend uses the slower user
 
 ## Status
 
-Backend implemented 2026-09-16 (C4): mapping, paging, and client lifecycle, tested with a fake
+Backend implemented 2026-09-16 (C4), wired into the daemon at C5: `MediaService` logs the client in
-fetcher. Not wired into the daemon yet; that is C5. A live login with a real token is owed at C8.
+on the first scrape and picks it for guild channels it is in. A live login with a real token is owed
-Checkpoints: [CURRENT.md](../CURRENT.md).
+at C9. Checkpoints: [CURRENT.md](../CURRENT.md).

docs/webui.md

@@ -235,6 +235,11 @@ replaceable.
 | POST | `/api/audio/stop` | none | Stop speech and clear the queue |
 | POST | `/api/audio/capture` | `start` | Start or stop capture |
 | GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
+| GET | `/api/media` | `kind?`, `limit?`, `offset?` | Media items across every scrape, newest first |
+| GET | `/api/media/scrapes` | none | Every scrape with state and counters |
+| GET | `/api/media/status` | none | Slopbot connection state and the scrape list |
+| POST | `/api/media/scrape` | `channelId`, `guildId?`, `mode?` | Start a media scrape |
+| POST | `/api/media/scrape/stop` | `id` | Stop a media scrape |
 | GET | `/api/logs` | `limit` | The persistent action log |
 | GET | `/api/events` | none | Server-sent events |
 

media-scrape/src/puppet-backend.ts

@@ -0,0 +1,25 @@
+/*
+ * The puppet scrape backend: history pages over the validated core bridge.
+ */
+
+import type { FetchPage, ScrapeBackend } from "./engine.js";
+import type { CoreMessage } from "./types.js";
+
+/** The core `message.fetch` call the daemon injects. */
+export type CoreFetch = (
+    channelId: string,
+    limit: number,
+    options?: { before?: string; after?: string }
+) => Promise<{ messages?: unknown[] }>;
+
+export class PuppetBackend implements ScrapeBackend {
+    readonly name = "puppet" as const;
+
+    constructor(private readonly fetchMessages: CoreFetch) {}
+
+    async fetchPage(channelId: string, before: string | null, limit: number): Promise<FetchPage> {
+        const result = await this.fetchMessages(channelId, limit, before ? { before } : {});
+        const messages = (result?.messages ?? []) as CoreMessage[];
+        return { messages };
+    }
+}

media-scrape/test/run.ts

@@ -9,6 +9,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { ScrapeEngine, type FetchPage, type ScrapeBackend } from "../src/engine.js";
 import { extractMedia } from "../src/extract.js";
+import { PuppetBackend } from "../src/puppet-backend.js";
 import { dirSizeBytes, listScrapes, ScrapeStore } from "../src/store.js";
 import type { CoreAttachment, CoreMessage } from "../src/types.js";
 
@@ -258,6 +259,19 @@ void (async () => {
         assert.equal(meta?.pages, 1);
     });
 
+    await test("puppet backend passes before and limit to the core", async () => {
+        const calls: Array<Record<string, unknown>> = [];
+        const backend = new PuppetBackend(async (channelId, limit, options) => {
+            calls.push({ channelId, limit, ...options });
+            return { messages: [message("50")] };
+        });
+
+        const page = await backend.fetchPage("123", "60", 5);
+        assert.equal(page.messages.length, 1);
+        assert.equal(page.messages[0].id, "50");
+        assert.deepEqual(calls[0], { channelId: "123", limit: 5, before: "60" });
+    });
+
     rmSync(root, { recursive: true, force: true });
 
     if (failures > 0) {

poc/src/control-bridge.ts

@@ -25,6 +25,7 @@ import { DEFAULT_VAD } from "./vad.js";
 import { CoreClient, coreSocketPath } from "./core-client.js";
 import { CoreProcess, staleBundleWarning } from "./core-process.js";
 import { dedupe, getQueryCount, paced, pacedSend, pacedVoice, sleep } from "./pacing.js";
+import { MediaService } from "./media.js";
 import { PreferencesStore } from "./preferences.js";
 import { Puppet } from "./puppet.js";
 import { startWebUi } from "./webui.js";
@@ -39,6 +40,12 @@ let puppet: Puppet | undefined;
 let coreConnected = false;
 let connecting = false;
 
+const media = new MediaService({
+    getCore: () => core,
+    isCoreConnected: () => coreConnected,
+    log: line => console.log(`[media] ${line}`)
+});
+
 async function connectLoop(): Promise<void> {
     if (connecting) return;
     connecting = true;
@@ -49,7 +56,7 @@ async function connectLoop(): Promise<void> {
                 const client = await CoreClient.connect();
 
                 core = client;
-                puppet = new Puppet(client, audio);
+                puppet = new Puppet(client, audio, media);
 
                 client.onEvent(event => {
                     // Speaking hints add to the reconciled set so the poller can
@@ -277,6 +284,7 @@ const webui = startWebUi({
     audio,
     transcripts,
     preferences,
+    media,
     connectAudioDevices: () => autoConnectAudio(requireCore())
 });
 

poc/src/media.ts

@@ -0,0 +1,188 @@
+/*
+ * Media archive service: the daemon side of media-scrape/. See docs/media.md.
+ *
+ * Owns the scrape engine, the slopbot client, and the read path the Web UI and
+ * (later) MCP tools call.
+ */
+
+import {
+    ScrapeEngine,
+    ScrapeStore,
+    scrapeDir,
+    resolveDbRoot,
+    type MediaKind,
+    type ScrapeBackend,
+    type ScrapeMeta
+} from "../../media-scrape/src/index.js";
+import { PuppetBackend } from "../../media-scrape/src/puppet-backend.js";
+import {
+    loadSlopbotConfig,
+    slopbotFetcher,
+    slopbotStatus,
+    SlopbotBackend,
+    startSlopbot,
+    type SlopbotFetcher,
+    type SlopbotStatus
+} from "../../media-scrape/slopbot/index.js";
+import type { CoreClient } from "./core-client.js";
+
+type BotClient = Awaited<ReturnType<typeof startSlopbot>>;
+
+export interface MediaServiceDeps {
+    getCore: () => CoreClient | undefined;
+    isCoreConnected: () => boolean;
+    log?: (line: string) => void;
+}
+
+export interface StartScrapeInput {
+    channelId: string;
+    guildId?: string | null;
+    mode?: "links" | "download";
+    backend?: "auto" | "slopbot" | "puppet";
+}
+
+export interface MediaItemResult {
+    scrapeId: string;
+    channelId: string;
+    kind: MediaKind;
+    url: string;
+    provider: string | null;
+    downloaded: boolean;
+    createdAt: string;
+}
+
+export interface MediaStatus {
+    slopbot: SlopbotStatus;
+    scrapes: ScrapeMeta[];
+}
+
+export class MediaService {
+    private readonly rootDir = resolveDbRoot();
+    private readonly engine: ScrapeEngine;
+
+    private bot: BotClient | null = null;
+    private botFetcher: SlopbotFetcher | null = null;
+    private botStarting: Promise<void> | null = null;
+
+    constructor(private readonly deps: MediaServiceDeps) {
+        this.engine = new ScrapeEngine({
+            rootDir: this.rootDir,
+            log: line => this.deps.log?.(line)
+        });
+    }
+
+    /** Log slopbot in on first use. False when it is disabled or the login failed. */
+    private async ensureSlopbot(): Promise<boolean> {
+        if (this.bot) return true;
+
+        const config = loadSlopbotConfig();
+        if (!config.enabled) return false;
+
+        if (!this.botStarting) {
+            this.botStarting = startSlopbot(config)
+                .then(client => {
+                    this.bot = client;
+                    this.botFetcher = slopbotFetcher(client);
+                    this.deps.log?.("slopbot connected");
+                })
+                .catch(error => {
+                    this.deps.log?.(`slopbot login failed: ${String(error)}`);
+                });
+        }
+
+        await this.botStarting;
+        return this.bot !== null;
+    }
+
+    private puppetBackend(): ScrapeBackend {
+        return new PuppetBackend(async (channelId, limit, options) => {
+            const core = this.deps.getCore();
+            if (!core || !this.deps.isCoreConnected()) {
+                throw new Error("the core bridge is not connected");
+            }
+            return (await core.fetchMessages(channelId, limit, options)) as { messages?: unknown[] };
+        });
+    }
+
+    async startScrape(input: StartScrapeInput): Promise<ScrapeMeta> {
+        const mode = input.mode ?? "links";
+        let backend: ScrapeBackend;
+
+        if (input.backend === "puppet") {
+            backend = this.puppetBackend();
+        } else {
+            const available = await this.ensureSlopbot();
+            const inGuild = !!input.guildId && (this.bot?.guilds.cache.has(input.guildId) ?? false);
+
+            if (input.backend === "slopbot" && (!available || !inGuild)) {
+                throw new Error("slopbot is not connected to that guild");
+            }
+
+            const fetcher = this.botFetcher;
+            backend =
+                available && inGuild && fetcher ? new SlopbotBackend(fetcher) : this.puppetBackend();
+        }
+
+        return this.engine.start({
+            channelId: input.channelId,
+            guildId: input.guildId ?? null,
+            mode,
+            backend
+        });
+    }
+
+    stopScrape(id: string): void {
+        this.engine.stop(id);
+    }
+
+    listScrapes(): ScrapeMeta[] {
+        return this.engine.list();
+    }
+
+    getScrape(id: string): ScrapeMeta | null {
+        return this.engine.get(id);
+    }
+
+    status(): MediaStatus {
+        return {
+            slopbot: slopbotStatus(this.bot),
+            scrapes: this.engine.list()
+        };
+    }
+
+    /** Read media rows across every scrape, newest scrape first. */
+    queryMedia(options: { kind?: MediaKind; limit?: number; offset?: number } = {}): MediaItemResult[] {
+        const limit = Math.max(1, Math.min(options.limit ?? 60, 500));
+        const offset = Math.max(0, options.offset ?? 0);
+        const results: MediaItemResult[] = [];
+        let skipped = 0;
+
+        for (const meta of this.engine.list()) {
+            if (meta.state === "deleted") continue;
+
+            const store = new ScrapeStore(scrapeDir(this.rootDir, meta.id));
+            try {
+                const rows = store.queryMedia({ kind: options.kind, limit: limit + offset });
+                for (const row of rows) {
+                    if (skipped++ < offset) continue;
+                    if (results.length >= limit) break;
+                    results.push({
+                        scrapeId: meta.id,
+                        channelId: meta.channelId,
+                        kind: row.kind,
+                        url: row.url,
+                        provider: row.provider,
+                        downloaded: row.downloaded === 1,
+                        createdAt: row.created_at
+                    });
+                }
+            } finally {
+                store.close();
+            }
+
+            if (results.length >= limit) break;
+        }
+
+        return results;
+    }
+}

poc/src/puppet.ts

@@ -110,6 +110,24 @@ export interface PuppetAudio {
     stop(): void;
 }
 
+/** The scrape state a chat command reads. */
+export interface PuppetScrape {
+    id: string;
+    channelId: string;
+    state: string;
+}
+
+/** The media actions a chat command can trigger. */
+export interface PuppetMedia {
+    startScrape(input: {
+        channelId: string;
+        guildId: string | null;
+        mode: "links" | "download";
+    }): Promise<{ id: string }>;
+    stopScrape(id: string): void;
+    listScrapes(): PuppetScrape[];
+}
+
 interface PollState {
     channelId: string;
     authorId: string;
@@ -130,6 +148,15 @@ const WHITELIST_HELP = [
     "  whitelist remove @user      Remove a user"
 ].join("\n");
 
+const SCRAPE_HELP = [
+    "scrape media — archive this channel's media.",
+    "Usage:",
+    "  scrape media            Index links (default)",
+    "  scrape media download   Also download attachment bytes (admin)",
+    "  scrape media status     Show this channel's latest scrape",
+    "  scrape media stop       Stop it (admin)"
+].join("\n");
+
 function clamp(value: number, min: number, max: number): number {
     return Math.min(Math.max(value, min), max);
 }
@@ -164,7 +191,8 @@ export class Puppet {
 
     constructor(
         private readonly core: PuppetTransport,
-        private readonly audio?: PuppetAudio
+        private readonly audio?: PuppetAudio,
+        private readonly media?: PuppetMedia
     ) {
         try {
             const state = JSON.parse(readFileSync(STATE_FILE, "utf8")) as PersistedState;
@@ -531,6 +559,8 @@ export class Puppet {
             await this.handleWhitelistCommand(command, author.id, tier, channelId);
         } else if (command?.name === "speak") {
             await this.handleSpeakCommand(command, author.id, channelId);
+        } else if (command?.name === "scrape") {
+            await this.handleScrapeCommand(command, author.id, channelId, guildId, tier);
         } else if (POLL_ENABLED && command?.name === "poll") {
             await this.handlePollCommand(author.id, channelId, text);
         }
@@ -671,6 +701,73 @@ export class Puppet {
         }
     }
 
+    private async handleScrapeCommand(
+        command: ParsedCommand,
+        authorId: string,
+        channelId: string,
+        guildId: string | undefined,
+        tier: TrustTier
+    ) {
+        const [kind, sub] = command.args;
+        const knownSub = sub === undefined || sub === "download" || sub === "status" || sub === "stop";
+
+        if (kind !== "media" || sub === "help" || !knownSub) {
+            return this.respond(authorId, channelId, "scrape", SCRAPE_HELP);
+        }
+
+        if (!this.media) {
+            return this.respond(authorId, channelId, "scrape", "The media archive is not available.");
+        }
+
+        if (sub === "status") {
+            const scrape = this.media.listScrapes().find(entry => entry.channelId === channelId);
+            return this.respond(
+                authorId,
+                channelId,
+                "scrape",
+                scrape
+                    ? `Media scrape ${scrape.id}: ${scrape.state}.`
+                    : "No media scrape for this channel yet."
+            );
+        }
+
+        if (sub === "stop") {
+            if (!hasTier(tier, "admin")) {
+                return this.respond(authorId, channelId, "scrape", "You do not have permission for this command.");
+            }
+
+            const scrape = this.media
+                .listScrapes()
+                .find(entry => entry.channelId === channelId && ["queued", "running", "paused"].includes(entry.state));
+            if (!scrape) {
+                return this.respond(authorId, channelId, "scrape", "No active media scrape for this channel.");
+            }
+
+            this.media.stopScrape(scrape.id);
+            this.actionLog.append({ type: "media", action: "stop", channelId, scrapeId: scrape.id });
+            return this.respond(authorId, channelId, "scrape", `Stopping media scrape ${scrape.id}.`);
+        }
+
+        const mode = sub === "download" ? "download" : "links";
+        if (mode === "download" && !hasTier(tier, "admin")) {
+            return this.respond(authorId, channelId, "scrape", "Download mode needs the admin tier.");
+        }
+
+        try {
+            const scrape = await this.media.startScrape({ channelId, guildId: guildId ?? null, mode });
+            this.actionLog.append({ type: "media", action: "scrape", channelId, mode, scrapeId: scrape.id });
+            return this.respond(authorId, channelId, "scrape", `Media scrape started (${mode}): ${scrape.id}.`);
+        } catch (error) {
+            this.actionLog.append({ type: "error", scope: "media", channelId, message: String(error) });
+            return this.respond(
+                authorId,
+                channelId,
+                "scrape",
+                `Media scrape failed: ${error instanceof Error ? error.message : String(error)}`
+            );
+        }
+    }
+
     /* Polls */
 
     private async handlePollCommand(authorId: string, channelId: string, text: string) {

poc/src/webui.ts

@@ -20,6 +20,7 @@ import type { AuthService, AuthStatus } from "./auth-service.js";
 import type { AudioService } from "./audio-service.js";
 import { CoreError, type CoreClient } from "./core-client.js";
 import type { CoreProcess } from "./core-process.js";
+import type { MediaService } from "./media.js";
 import { dedupe, getQueryCount, paced, pacedSend, pacedVoice } from "./pacing.js";
 import type { PreferencesStore, WebPreferences } from "./preferences.js";
 import type { Puppet } from "./puppet.js";
@@ -72,6 +73,7 @@ export interface WebUiDeps {
     audio: AudioService;
     transcripts: TranscriptStore;
     preferences: PreferencesStore;
+    media: MediaService;
     connectAudioDevices: () => Promise<{ connected: boolean; message: string }>;
 }
 
@@ -713,6 +715,66 @@ export function startWebUi(deps: WebUiDeps): WebUiHandle {
         })
     );
 
+    app.get(
+        "/api/media/scrapes",
+        wrap(async (_req, res) => {
+            res.json({ ok: true, result: deps.media.listScrapes() });
+        })
+    );
+
+    app.get(
+        "/api/media/status",
+        wrap(async (_req, res) => {
+            res.json({ ok: true, result: deps.media.status() });
+        })
+    );
+
+    app.get(
+        "/api/media",
+        wrap(async (req, res) => {
+            const kind = ["image", "gif", "video", "audio"].includes(String(req.query.kind ?? ""))
+                ? (String(req.query.kind) as "image" | "gif" | "video" | "audio")
+                : undefined;
+            const limit = clampLimit(req.query.limit, 60, 500);
+            const offset = Math.max(0, Number(req.query.offset ?? 0) || 0);
+
+            res.json({ ok: true, result: deps.media.queryMedia({ kind, limit, offset }) });
+        })
+    );
+
+    app.post(
+        "/api/media/scrape",
+        wrap(async (req, res) => {
+            const channelId = String(req.body?.channelId ?? "");
+            if (!channelId) throw new HttpError(400, "channelId is required");
+
+            const mode = req.body?.mode === "download" ? "download" : "links";
+            const guildId = req.body?.guildId ? String(req.body.guildId) : null;
+
+            const scrape = await deps.media.startScrape({ channelId, guildId, mode });
+            deps.getPuppet()?.appendAction({
+                type: "media",
+                action: "scrape",
+                channelId,
+                mode,
+                scrapeId: scrape.id
+            });
+            res.json({ ok: true, result: scrape });
+        })
+    );
+
+    app.post(
+        "/api/media/scrape/stop",
+        wrap(async (req, res) => {
+            const id = String(req.body?.id ?? "");
+            if (!id) throw new HttpError(400, "id is required");
+
+            deps.media.stopScrape(id);
+            deps.getPuppet()?.appendAction({ type: "media", action: "stop", scrapeId: id });
+            res.json({ ok: true, result: { stopped: id } });
+        })
+    );
+
     app.get("/api/logs", (req, res) => {
         const puppet = requirePuppet();
         const limit = clampLimit(req.query.limit, 100, 500);