Store embed titles and search them

This commit is contained in:
Fisto
2026-09-17 13:10:16 +02:00
parent c6caa168be
commit 557ba5ca7e
13 changed files with 200 additions and 10 deletions
+6 -2
View File
@@ -88,7 +88,7 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
| Table | Columns |
| --- | --- |
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
| `media_sources` | `media_id`, `message_id` |
| `downloads` | `media_id`, `state`, `attempts`, `error` |
@@ -96,7 +96,11 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
`media_sources(message_id)`. The database is an index; the media
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Embed rows also store
Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed
form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps
over car*), one phrase with the exact-match toggle, and rows without a title drop out while a
title query is set. Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
+3 -1
View File
@@ -149,7 +149,9 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
include** and **Must include** groups with nested collapsibles for sources (the common providers
include** and **Must include** groups with nested collapsibles, a title search for embed titles
(multi-word prefix matching, with an **Exact match (quotes)** toggle) for sources (the common
providers
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
rows with no known value. The bar and drawer show only on the Media tab.
+5 -2
View File
@@ -137,7 +137,10 @@ box live in Chat, not Call, and the connection stats are not chips yet.
2ch.hk, Imgur, FunnyJunk, Internet Archive — plus embeds, attachments, stickers) and file types.
**Must include** requires them (same provider list and kinds, file-type chips, and size/length
bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as
`Giphy/Tenor`. A sort select orders the grid by newest, oldest, user, size, or length. Size and
`Giphy/Tenor`. A title field searches embed titles (YouTube): default is multi-word prefix
matching (each token must start a word of the title, order-free), and **Exact match (quotes)**
treats the input as one phrase. A sort select orders the grid by newest, oldest, user, size, or
length. Size and
length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot
tab. Filters do not
apply until **Apply filters** is pressed; **Reset** clears every filter and the zoom, unloads the
@@ -289,7 +292,7 @@ replaceable.
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
| POST | `/api/audio/capture` | `start` | Start or stop capture |
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?`, `includeProviders?`, `includeSources?` (comma lists), `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?`, `includeProviders?`, `includeSources?` (comma lists), `title?` (200 chars max), `titleExact?`, `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
| GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) |
| GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` |