From 308e771f2a03a5b30d44e057e9aeeec79048bcb4 Mon Sep 17 00:00:00 2001 From: Fisto Date: Wed, 16 Sep 2026 09:54:44 +0200 Subject: [PATCH] Mark the byte downloader as the next checkpoint --- CURRENT.md | 2 ++ docs/media-scraper.md | 7 ++++++- docs/media.md | 4 +++- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/CURRENT.md b/CURRENT.md index 56e5068..c214807 100644 --- a/CURRENT.md +++ b/CURRENT.md @@ -33,6 +33,8 @@ mobile draft at 390px. - [x] C7a storage limits, retention, warning/confirm flow, storage and clean API - [x] C7b Settings group: limit field with info hover, usage bar, clean modal - [x] C8 MCP service hooks (`scrape_media`, scrape reads, storage, clean; reads in `run_commands`) + - [ ] C10 downloader: fetch attachment bytes in `mode: "download"`, hash-dedupe, size cap, + serial queue with 1-3 s gaps, retries. The command accepts the mode today but only indexes. - [~] C9 live verification. Slopbot path verified live on 2026-09-16: login Slopbot#5775, guild `177931418704281600`, test channel `1549681689222053988`. Scrape `1549681689222053988- 20260916075150`: state done, 1 page, 17 messages, 4 images + 13 videos, 0 errors, ~20 MB of diff --git a/docs/media-scraper.md b/docs/media-scraper.md index feba139..7943cb2 100644 --- a/docs/media-scraper.md +++ b/docs/media-scraper.md @@ -90,7 +90,12 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape. ## Downloads and previews -- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap +**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`, +and the command accepts `mode: "download"`, but nothing fetches bytes yet. + +The plan: + +- A serial download queue, one file at a time, 1-3 s apart, with a size cap (`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors. - `4xx` marks the URL expired. The item is re-fetched with its message later. - Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash diff --git a/docs/media.md b/docs/media.md index 9b8c937..9139c60 100644 --- a/docs/media.md +++ b/docs/media.md @@ -72,7 +72,9 @@ media-scrape/db/ - `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is downloaded. -- `download` fetches attachment bytes into `media/`. Deduplicated by hash. +- `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet** + (checkpoint C10): the command accepts the mode and the schema has the columns, but the download + queue is still owed. - Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other embed stays a link with metadata. - Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN