zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the media downloader with hash storage and local file serving

Browse Source
This commit is contained in:
Fisto
2026-09-16 10:35:55 +02:00
parent 3ea72f587d
commit f7664256d1
15 changed files with 431 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+11 -8
View File
@@ -90,16 +90,19 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.
## Downloads and previews
**Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
Downloads run after a finished walk when the scrape used `mode: "download"`:
The plan:
- A serial queue, one file at a time, 1-3 s apart, with a size cap
(`VESKSLOP_MEDIA_MAX_FILE_BYTES`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` from the skip set (400, 401, 403, 404, 410) marks the URL expired; the row is not retried.
- The file name is the SHA-256 of the bytes under `media/`. `file_path`, `file_hash`, and
`file_bytes` are written to the row, so the run resumes and dedupes by content.
- Only attachment and sticker rows are targets. Embed rows are excluded in SQL
(`source != 'embed'`).
- Progress counters: `downloaded`, `downloadErrors`.
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
in `thumbs/`. A preview failure never fails the scrape.
Previews are **planned (C11)**: the Discord CDN resizer (`?width=320&height=320&format=webp`) into
`thumbs/`, stored by hash, failure never fails the scrape.
## Job states