Deduplicate media across scrapes and polish the viewer controls

This commit is contained in:
Fisto
2026-09-16 13:05:31 +02:00
parent 1b5fcfed2f
commit d3f9d20190
12 changed files with 198 additions and 85 deletions
+2
View File
@@ -111,6 +111,8 @@ Downloads run after a finished walk when the scrape used `mode: "download"`:
`<destination>/<scrape-id>/thumbnails/` while the local copy stays for the Media tab.
- Downloaded videos get a poster frame (`ffmpeg -ss 0.5`, 480px WebP) cached in `thumbs/`, because
Discord serves no thumbnails for uploaded video files.
- The ffprobe pass fills missing dimensions and `duration_secs` after a download. Attachment
durations come from Discord; the 0 that Discord sends for non-audio files is treated as unknown.
- Progress counters: `downloaded`, `downloadErrors`.
Previews run after every walk:
+11 -3
View File
@@ -67,6 +67,10 @@ media-scrape/db/
- The database is an index. It holds the scrape state, the messages seen, the media items, and the
link between a media item and its source messages. Source channel, author, and timestamp are
indexed so filters can be added later without a re-scrape.
- Scrapes stay self-contained, but the API and UI **deduplicate by URL across scrapes**: one card
per URL, the newest scrape wins; stats count unique URLs too. Re-scraping a channel does not
duplicate anything in the grid or the counters. (Files are still stored per scrape, so the
on-disk size doubles with overlapping scrapes; a shared content store is a possible later step.)
- Storage uses `node:sqlite` (experimental in Node 22, prints a warning). If its API changes,
`better-sqlite3` is the fallback; it needs a native build, so that switch is an owner install.
- The file name in `media/` is the SHA-256 of the bytes, never the original name. The original
@@ -147,9 +151,13 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
tiles carry a translucent play icon top-right. Discord serves no thumbnails for uploaded videos,
so a video without a cached preview renders a first-frame `<video>` in the grid, and downloaded
videos get an ffmpeg poster cached in `thumbs/`. Clicking a tile opens a viewer: wide media
shows the metadata banner below, vertical media shows it to the right; the banner has the full
link (with an external-site warning) and **View full**, and the play action embeds a YouTube,
Vimeo, Twitch, or Streamable player when possible. The status row reads
shows the metadata banner below, vertical media shows it to the right, and videos carry a large
play icon top-right. The play action embeds a YouTube, Vimeo, Twitch, or Streamable player, or
plays a direct mp4/webm/m4v; when neither is possible it opens the external-site warning, since
the page is the only way to watch. **View full** opens the embedded media (player, local file,
or cached preview) in a new tab; the full link is a separate button that opens the original
page with the warning. Length comes from Discord's attachment duration and, for downloads, an
ffprobe probe. The status row reads
`(Images/Videos) | Downloaded A/B | Cached C/B (excl Downl) | Loaded L/B`, where B is the total
for the active filters. A scrape strip shows live progress with Stop.
**New scrape…** (rightmost) opens a popup with Server and Group chat trees (servers expand to