Mark the byte downloader as the next checkpoint

This commit is contained in:
Fisto
2026-09-16 09:54:44 +02:00
parent d483a2fa47
commit 308e771f2a
3 changed files with 11 additions and 2 deletions
+6 -1
View File
@@ -90,7 +90,12 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.
## Downloads and previews
- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap
**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
The plan:
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
+3 -1
View File
@@ -72,7 +72,9 @@ media-scrape/db/
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
downloaded.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet**
(checkpoint C10): the command accepts the mode and the schema has the columns, but the download
queue is still owed.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN