Mark the byte downloader as the next checkpoint
This commit is contained in:
@@ -90,7 +90,12 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.
|
||||
|
||||
## Downloads and previews
|
||||
|
||||
- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap
|
||||
**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
|
||||
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
|
||||
|
||||
The plan:
|
||||
|
||||
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
|
||||
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
|
||||
- `4xx` marks the URL expired. The item is re-fetched with its message later.
|
||||
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
|
||||
|
||||
+3
-1
@@ -72,7 +72,9 @@ media-scrape/db/
|
||||
|
||||
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
|
||||
downloaded.
|
||||
- `download` fetches attachment bytes into `media/`. Deduplicated by hash.
|
||||
- `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet**
|
||||
(checkpoint C10): the command accepts the mode and the schema has the columns, but the download
|
||||
queue is still owed.
|
||||
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
|
||||
embed stays a link with metadata.
|
||||
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN
|
||||
|
||||
Reference in New Issue
Block a user