zib/Veskslop-t
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add links-mode embed thumbnails, item metadata, and media filters

Browse Source
This commit is contained in:
Fisto
2026-09-16 11:44:09 +02:00
parent 1fe204b91e
commit 973b9cff92
23 changed files with 863 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/media-scraper.md
+10 -6
View File
@@ -81,12 +81,16 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `duration_ms`, `downloaded`, `created_at` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
Indexes: `(kind, timestamp)` via `media_sources`, `channel` metadata in `scrape.json`, and
`file_hash` for dedupe. The database is an index; the media folder holds bytes.
Indexes: `kind`, `posted_at`, and `media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
## Downloads and previews
@@ -105,9 +109,9 @@ Previews run after every walk:
- In both modes, Discord-hosted image and GIF URLs (`cdn.discordapp.com`, `media.discordapp.net`)
are transformed to the CDN resizer (`media.discordapp.net?...width=320&height=320&format=webp`).
- In `download` mode only, embed rows also get their stored `preview_url` (YouTube, X, TikTok
thumbnails, and so on) cached. The embed's full media URL is never fetched; embed rows are
excluded from `pendingDownloads`.
- In both modes, embed rows also get their stored `preview_url` (YouTube, X, TikTok thumbnails,
and so on) cached. The embed's full media URL is never fetched; embed rows are excluded from
`pendingDownloads`.
- One attempt per row, no retries; a failure marks `preview_error` and never fails the scrape.
- Stored by hash in `thumbs/<sha256>.webp`; `preview_hash` and `preview_bytes` land on the row, and
the counter is `previewed`. Previews count toward the storage limit.