zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Mark the byte downloader as the next checkpoint

Browse Source
This commit is contained in:
Fisto
2026-09-16 09:54:44 +02:00
parent d483a2fa47
commit 308e771f2a
3 changed files with 11 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CURRENT.md
+2
View File
@@ -33,6 +33,8 @@ mobile draft at 390px.
- [x] C7a storage limits, retention, warning/confirm flow, storage and clean API - [x] C7a storage limits, retention, warning/confirm flow, storage and clean API
- [x] C7b Settings group: limit field with info hover, usage bar, clean modal - [x] C7b Settings group: limit field with info hover, usage bar, clean modal
- [x] C8 MCP service hooks (`scrape_media`, scrape reads, storage, clean; reads in `run_commands`) - [x] C8 MCP service hooks (`scrape_media`, scrape reads, storage, clean; reads in `run_commands`)
- [ ] C10 downloader: fetch attachment bytes in `mode: "download"`, hash-dedupe, size cap,
serial queue with 1-3 s gaps, retries. The command accepts the mode today but only indexes.
- [~] C9 live verification. Slopbot path verified live on 2026-09-16: login Slopbot#5775, guild - [~] C9 live verification. Slopbot path verified live on 2026-09-16: login Slopbot#5775, guild
`177931418704281600`, test channel `1549681689222053988`. Scrape `1549681689222053988- `177931418704281600`, test channel `1549681689222053988`. Scrape `1549681689222053988-
20260916075150`: state done, 1 page, 17 messages, 4 images + 13 videos, 0 errors, ~20 MB of 20260916075150`: state done, 1 page, 17 messages, 4 images + 13 videos, 0 errors, ~20 MB of
docs/media-scraper.md
+6 -1
View File
@@ -90,7 +90,12 @@ Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.
## Downloads and previews ## Downloads and previews
- `downloads` is a serial queue, one file at a time, 1-3 s apart, with a size cap **Planned (C10).** The schema has `file_bytes`, `preview_bytes`, `file_hash`, and `preview_hash`,
and the command accepts `mode: "download"`, but nothing fetches bytes yet.
The plan:
- A serial download queue, one file at a time, 1-3 s apart, with a size cap
(`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors. (`media.maxFileBytes`, default 50 MiB) and 2 retries on network or 5xx errors.
- `4xx` marks the URL expired. The item is re-fetched with its message later. - `4xx` marks the URL expired. The item is re-fetched with its message later.
- Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash - Previews use the Discord CDN resizer (`?width=320&height=320&format=webp`) and are stored by hash
docs/media.md
+3 -1
View File
@@ -72,7 +72,9 @@ media-scrape/db/
- `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is - `links` is the default. The archive stores the Discord CDN URL and the metadata. Nothing large is
downloaded. downloaded.
- `download` fetches attachment bytes into `media/`. Deduplicated by hash. - `download` fetches attachment bytes into `media/`. Deduplicated by hash. **Not implemented yet**
(checkpoint C10): the command accepts the mode and the schema has the columns, but the download
queue is still owed.
- Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other - Third-party embeds are **never downloaded**: YouTube, Instagram, X, TikTok, and every other
embed stays a link with metadata. embed stays a link with metadata.
- Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN - Discord-hosted image and sticker previews are cached small (`thumbs/`, WebP, via the CDN