Store embed titles and search them
This commit is contained in:
@@ -40,6 +40,14 @@ mobile draft at 390px.
|
|||||||
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
|
||||||
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
|
||||||
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
|
||||||
|
- [x] C49 Searchable video titles. Embed rows now store Discord's `title` plus a normalized
|
||||||
|
`title_key` (lowercase, punctuation collapsed). The Advanced drawer has a title field with an
|
||||||
|
**Exact match (quotes)** toggle: default is order-free multi-word prefix search ("Ma Ca Ju"
|
||||||
|
finds *Man who jumps over car*), exact mode is one phrase; rows without a title drop out while
|
||||||
|
the query is set. Store, `/api/media`, `/api/media/stats`, and MCP carry `title`/`titleExact`;
|
||||||
|
the API result and card tooltip expose the title. Unit tests cover prefixes, mid-word misses,
|
||||||
|
exact phrases, and punctuation folding; smoke checks the field and toggle. Note: titles only
|
||||||
|
exist for rows scraped from now on — pre-existing rows need the backfill queued in TODO.md.
|
||||||
- [x] C48 Expanded providers and the Must include mirror. The list grew to 19 entries from the
|
- [x] C48 Expanded providers and the Must include mirror. The list grew to 19 entries from the
|
||||||
user's picks: reddit, twitter, tiktok, instagram, streamable, facebook, gyazo, a merged
|
user's picks: reddit, twitter, tiktok, instagram, streamable, facebook, gyazo, a merged
|
||||||
`giphy,tenor` (one checkbox, two substrings), 2ch.hk (stand-in for 4chan; the CDN hosts stay
|
`giphy,tenor` (one checkbox, two substrings), 2ch.hk (stand-in for 4chan; the CDN hosts stay
|
||||||
|
|||||||
@@ -62,6 +62,15 @@ Ordered backlog. `CURRENT.md` tracks the live session state.
|
|||||||
optional `VOX_CLEAN`, and a help line in the TTS panel. Requested in `TTS-DESIGN.md`; the
|
optional `VOX_CLEAN`, and a help line in the TTS panel. Requested in `TTS-DESIGN.md`; the
|
||||||
wrapper side is ready and verified. Inline `/rate` and `/pitch` need no UI.
|
wrapper side is ready and verified. Inline `/rate` and `/pitch` need no UI.
|
||||||
|
|
||||||
|
## Media archive follow-ups
|
||||||
|
|
||||||
|
1. **Backfill titles for existing rows.** New scrapes store embed titles, but the ~20k rows
|
||||||
|
already indexed have `title IS NULL`. Add a paced pass that resolves titles through the provider
|
||||||
|
oEmbed endpoints (YouTube first) for rows where `title IS NULL AND lower(provider) LIKE
|
||||||
|
'%youtube%'`, with the same batch/cooldown settings as previews, an attempt cap so dead videos
|
||||||
|
close out, progress in the Bot tab, and tests. Consider folding it into the backfill queue as a
|
||||||
|
second work type.
|
||||||
|
|
||||||
## Web UI follow-ups
|
## Web UI follow-ups
|
||||||
|
|
||||||
Ordered after the voice backend unless the owner asks first.
|
Ordered after the voice backend unless the owner asks first.
|
||||||
|
|||||||
@@ -88,7 +88,7 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
|
|||||||
| Table | Columns |
|
| Table | Columns |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
|
| `messages` | `id`, `author_id`, `author_name`, `timestamp`, `content` |
|
||||||
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
|
| `media` | `id`, `kind`, `source`, `author_id`, `author_name`, `posted_at`, `url`, `provider`, `title`, `title_key`, `file_hash`, `preview_hash`, `content_type`, `size`, `width`, `height`, `aspect_ratio`, `duration_secs`, `downloaded`, `file_path`, `created_at` |
|
||||||
| `media_sources` | `media_id`, `message_id` |
|
| `media_sources` | `media_id`, `message_id` |
|
||||||
| `downloads` | `media_id`, `state`, `attempts`, `error` |
|
| `downloads` | `media_id`, `state`, `attempts`, `error` |
|
||||||
|
|
||||||
@@ -96,7 +96,11 @@ One SQLite file per scrape (`index.db`, `node:sqlite`).
|
|||||||
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
|
slopbot backend could ask for it (one request; null otherwise). Indexes: `kind`, `posted_at`, and
|
||||||
`media_sources(message_id)`. The database is an index; the media
|
`media_sources(message_id)`. The database is an index; the media
|
||||||
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
|
folder holds bytes. Every item carries its poster (`author_id`, `author_name`), the post time
|
||||||
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Filter presets live in
|
(`posted_at`), and its dimensions (`width`, `height`, `aspect_ratio`). Embed rows also store
|
||||||
|
Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed
|
||||||
|
form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps
|
||||||
|
over car*), one phrase with the exact-match toggle, and rows without a title drop out while a
|
||||||
|
title query is set. Filter presets live in
|
||||||
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
|
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
|
||||||
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
|
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
|
||||||
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
|
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
|
||||||
|
|||||||
+3
-1
@@ -149,7 +149,9 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
|
|||||||
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
|
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
|
||||||
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
|
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
|
||||||
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
|
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
|
||||||
include** and **Must include** groups with nested collapsibles for sources (the common providers
|
include** and **Must include** groups with nested collapsibles, a title search for embed titles
|
||||||
|
(multi-word prefix matching, with an **Exact match (quotes)** toggle) for sources (the common
|
||||||
|
providers
|
||||||
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
|
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
|
||||||
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
|
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
|
||||||
rows with no known value. The bar and drawer show only on the Media tab.
|
rows with no known value. The bar and drawer show only on the Media tab.
|
||||||
|
|||||||
+5
-2
@@ -137,7 +137,10 @@ box live in Chat, not Call, and the connection stats are not chips yet.
|
|||||||
2ch.hk, Imgur, FunnyJunk, Internet Archive — plus embeds, attachments, stickers) and file types.
|
2ch.hk, Imgur, FunnyJunk, Internet Archive — plus embeds, attachments, stickers) and file types.
|
||||||
**Must include** requires them (same provider list and kinds, file-type chips, and size/length
|
**Must include** requires them (same provider list and kinds, file-type chips, and size/length
|
||||||
bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as
|
bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as
|
||||||
`Giphy/Tenor`. A sort select orders the grid by newest, oldest, user, size, or length. Size and
|
`Giphy/Tenor`. A title field searches embed titles (YouTube): default is multi-word prefix
|
||||||
|
matching (each token must start a word of the title, order-free), and **Exact match (quotes)**
|
||||||
|
treats the input as one phrase. A sort select orders the grid by newest, oldest, user, size, or
|
||||||
|
length. Size and
|
||||||
length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot
|
length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot
|
||||||
tab. Filters do not
|
tab. Filters do not
|
||||||
apply until **Apply filters** is pressed; **Reset** clears every filter and the zoom, unloads the
|
apply until **Apply filters** is pressed; **Reset** clears every filter and the zoom, unloads the
|
||||||
@@ -289,7 +292,7 @@ replaceable.
|
|||||||
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
|
||||||
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
| POST | `/api/audio/capture` | `start` | Start or stop capture |
|
||||||
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
|
||||||
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?`, `includeProviders?`, `includeSources?` (comma lists), `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
|
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?`, `includeProviders?`, `includeSources?` (comma lists), `title?` (200 chars max), `titleExact?`, `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
|
||||||
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
|
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
|
||||||
| GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) |
|
| GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) |
|
||||||
| GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` |
|
| GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` |
|
||||||
|
|||||||
@@ -32,6 +32,8 @@ export interface ExtractedMedia {
|
|||||||
/** False for every embed. Downloads are a later step, never for third-party embeds. */
|
/** False for every embed. Downloads are a later step, never for third-party embeds. */
|
||||||
downloaded: boolean;
|
downloaded: boolean;
|
||||||
previewUrl: string | null;
|
previewUrl: string | null;
|
||||||
|
/** Embed title, when Discord provides one (YouTube and friends). Null elsewhere. */
|
||||||
|
title: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
function extension(filename: string | null | undefined): string {
|
function extension(filename: string | null | undefined): string {
|
||||||
@@ -81,7 +83,8 @@ function extractEmbed(embed: CoreEmbed): ExtractedMedia | null {
|
|||||||
size: null,
|
size: null,
|
||||||
width: embed.width ?? null,
|
width: embed.width ?? null,
|
||||||
height: embed.height ?? null,
|
height: embed.height ?? null,
|
||||||
durationSecs: null
|
durationSecs: null,
|
||||||
|
title: embed.title ?? null
|
||||||
};
|
};
|
||||||
|
|
||||||
if (embed.videoUrl) {
|
if (embed.videoUrl) {
|
||||||
@@ -135,7 +138,8 @@ function extractSticker(sticker: CoreSticker): ExtractedMedia | null {
|
|||||||
height: null,
|
height: null,
|
||||||
durationSecs: null,
|
durationSecs: null,
|
||||||
downloaded: false,
|
downloaded: false,
|
||||||
previewUrl: null
|
previewUrl: null,
|
||||||
|
title: null
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -162,7 +166,8 @@ function extractFrom(message: MediaMessage): ExtractedMedia[] {
|
|||||||
? attachment.durationSecs
|
? attachment.durationSecs
|
||||||
: null,
|
: null,
|
||||||
downloaded: false,
|
downloaded: false,
|
||||||
previewUrl: null
|
previewUrl: null,
|
||||||
|
title: null
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -136,6 +136,9 @@ export interface MediaFilterOptions {
|
|||||||
limit?: number;
|
limit?: number;
|
||||||
offset?: number;
|
offset?: number;
|
||||||
sort?: MediaSort;
|
sort?: MediaSort;
|
||||||
|
/** Title search: multi-word prefix tokens by default, a phrase when `titleExact`. */
|
||||||
|
titleQuery?: string;
|
||||||
|
titleExact?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface MediaRow {
|
export interface MediaRow {
|
||||||
@@ -166,9 +169,22 @@ export interface MediaRow {
|
|||||||
download_error: string | null;
|
download_error: string | null;
|
||||||
preview_error: string | null;
|
preview_error: string | null;
|
||||||
preview_attempts: number;
|
preview_attempts: number;
|
||||||
|
/** Embed title from Discord (YouTube and friends). */
|
||||||
|
title: string | null;
|
||||||
|
/** Lowercased, punctuation-collapsed title for token-prefix search. */
|
||||||
|
title_key: string | null;
|
||||||
created_at: string;
|
created_at: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Normalize a title for prefix-token search: lowercase, punctuation to spaces, collapsed. */
|
||||||
|
export function normalizeTitleKey(title: string): string {
|
||||||
|
return title
|
||||||
|
.toLowerCase()
|
||||||
|
.replace(/[^a-z0-9]+/g, " ")
|
||||||
|
.trim()
|
||||||
|
.replace(/\s+/g, " ");
|
||||||
|
}
|
||||||
|
|
||||||
const SCHEMA = `
|
const SCHEMA = `
|
||||||
CREATE TABLE IF NOT EXISTS messages (
|
CREATE TABLE IF NOT EXISTS messages (
|
||||||
id TEXT PRIMARY KEY,
|
id TEXT PRIMARY KEY,
|
||||||
@@ -206,6 +222,8 @@ CREATE TABLE IF NOT EXISTS media (
|
|||||||
download_error TEXT,
|
download_error TEXT,
|
||||||
preview_error TEXT,
|
preview_error TEXT,
|
||||||
preview_attempts INTEGER NOT NULL DEFAULT 0,
|
preview_attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
title TEXT,
|
||||||
|
title_key TEXT,
|
||||||
created_at TEXT NOT NULL
|
created_at TEXT NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
@@ -251,6 +269,10 @@ export class ScrapeStore {
|
|||||||
if (!names.has("preview_attempts")) {
|
if (!names.has("preview_attempts")) {
|
||||||
this.db.exec("ALTER TABLE media ADD COLUMN preview_attempts INTEGER NOT NULL DEFAULT 0");
|
this.db.exec("ALTER TABLE media ADD COLUMN preview_attempts INTEGER NOT NULL DEFAULT 0");
|
||||||
}
|
}
|
||||||
|
if (!names.has("title")) {
|
||||||
|
this.db.exec("ALTER TABLE media ADD COLUMN title TEXT");
|
||||||
|
this.db.exec("ALTER TABLE media ADD COLUMN title_key TEXT");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
close(): void {
|
close(): void {
|
||||||
@@ -264,7 +286,7 @@ export class ScrapeStore {
|
|||||||
"INSERT OR REPLACE INTO messages (id, author_id, author_name, timestamp, content) VALUES (?, ?, ?, ?, ?)"
|
"INSERT OR REPLACE INTO messages (id, author_id, author_name, timestamp, content) VALUES (?, ?, ?, ?, ?)"
|
||||||
);
|
);
|
||||||
const insertMedia = this.db.prepare(
|
const insertMedia = this.db.prepare(
|
||||||
"INSERT OR IGNORE INTO media (kind, source, author_id, author_name, posted_at, url, play_url, provider, content_type, size, width, height, aspect_ratio, duration_secs, downloaded, preview_url, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
"INSERT OR IGNORE INTO media (kind, source, author_id, author_name, posted_at, url, play_url, provider, content_type, size, width, height, aspect_ratio, duration_secs, downloaded, preview_url, title, title_key, created_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"
|
||||||
);
|
);
|
||||||
const findMedia = this.db.prepare("SELECT id FROM media WHERE url = ?");
|
const findMedia = this.db.prepare("SELECT id FROM media WHERE url = ?");
|
||||||
const linkSource = this.db.prepare(
|
const linkSource = this.db.prepare(
|
||||||
@@ -301,6 +323,8 @@ export class ScrapeStore {
|
|||||||
item.durationSecs,
|
item.durationSecs,
|
||||||
item.downloaded ? 1 : 0,
|
item.downloaded ? 1 : 0,
|
||||||
item.previewUrl,
|
item.previewUrl,
|
||||||
|
item.title,
|
||||||
|
item.title ? normalizeTitleKey(item.title) : null,
|
||||||
createdAt
|
createdAt
|
||||||
);
|
);
|
||||||
if (result.changes > 0) added[item.kind]++;
|
if (result.changes > 0) added[item.kind]++;
|
||||||
@@ -640,6 +664,18 @@ export class ScrapeStore {
|
|||||||
conditions.push("duration_secs > ?");
|
conditions.push("duration_secs > ?");
|
||||||
values.push(options.lengthGreaterThan);
|
values.push(options.lengthGreaterThan);
|
||||||
}
|
}
|
||||||
|
if (options.titleQuery?.trim()) {
|
||||||
|
const query = normalizeTitleKey(options.titleQuery);
|
||||||
|
if (options.titleExact) {
|
||||||
|
conditions.push("title_key LIKE ?");
|
||||||
|
values.push(`%${query}%`);
|
||||||
|
} else {
|
||||||
|
for (const token of query.split(" ").filter(Boolean)) {
|
||||||
|
conditions.push("(' ' || COALESCE(title_key, '')) LIKE ?");
|
||||||
|
values.push(`% ${token}%`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
if (options.after) {
|
if (options.after) {
|
||||||
conditions.push("posted_at >= ?");
|
conditions.push("posted_at >= ?");
|
||||||
values.push(options.after);
|
values.push(options.after);
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ import { isoFromSnowflake, snowflakeFromIso } from "../src/snowflake.js";
|
|||||||
import {
|
import {
|
||||||
dirSizeBytes,
|
dirSizeBytes,
|
||||||
listScrapes,
|
listScrapes,
|
||||||
|
normalizeTitleKey,
|
||||||
readScrapeMeta,
|
readScrapeMeta,
|
||||||
ScrapeStore,
|
ScrapeStore,
|
||||||
writeScrapeMeta
|
writeScrapeMeta
|
||||||
@@ -1347,6 +1348,61 @@ void (async () => {
|
|||||||
store.close();
|
store.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
await test("title search matches word prefixes and exact phrases", () => {
|
||||||
|
const dir = join(root, "title-search");
|
||||||
|
mkdirSync(dir, { recursive: true });
|
||||||
|
const store = new ScrapeStore(dir);
|
||||||
|
const embed = (url: string, title: string) => ({
|
||||||
|
type: "video",
|
||||||
|
url,
|
||||||
|
title,
|
||||||
|
provider: "YouTube",
|
||||||
|
videoUrl: url,
|
||||||
|
imageUrl: null,
|
||||||
|
thumbnailUrl: null
|
||||||
|
});
|
||||||
|
|
||||||
|
store.insertPage([
|
||||||
|
message("1000", {
|
||||||
|
attachments: [attachment({ filename: "no-title.png", contentType: "image/png" })],
|
||||||
|
embeds: [
|
||||||
|
embed("https://www.youtube.com/watch?v=a", "Man who jumps over car"),
|
||||||
|
embed("https://www.youtube.com/watch?v=b", "Human cannonball"),
|
||||||
|
embed("https://www.youtube.com/watch?v=c", "Woman's Day: 3 Things!")
|
||||||
|
]
|
||||||
|
})
|
||||||
|
]);
|
||||||
|
|
||||||
|
const ids = (options: Record<string, unknown>) =>
|
||||||
|
store
|
||||||
|
.queryMedia({ limit: 50, ...options })
|
||||||
|
.map(row => row.url.split("v=")[1])
|
||||||
|
.sort();
|
||||||
|
|
||||||
|
// Prefix tokens in any order, but only at word starts.
|
||||||
|
assert.deepEqual(ids({ titleQuery: "Ma Ca Ju" }), ["a"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "man" }), ["a"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "man car" }), ["a"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "cannon" }), ["b"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "hum" }), ["b"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "an" }), []);
|
||||||
|
|
||||||
|
// Exact mode is one phrase.
|
||||||
|
assert.deepEqual(ids({ titleQuery: "man who jumps", titleExact: true }), ["a"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "man car", titleExact: true }), []);
|
||||||
|
|
||||||
|
// Punctuation folds away.
|
||||||
|
assert.deepEqual(ids({ titleQuery: "woman's day" }), ["c"]);
|
||||||
|
assert.deepEqual(ids({ titleQuery: "3 things" }), ["c"]);
|
||||||
|
|
||||||
|
store.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
await test("normalizeTitleKey collapses punctuation and case", () => {
|
||||||
|
assert.equal(normalizeTitleKey("Woman's Day: 3 Things!"), "woman s day 3 things");
|
||||||
|
assert.equal(normalizeTitleKey(" A---B "), "a b");
|
||||||
|
});
|
||||||
|
|
||||||
await test("provider filters are the curated archive list", () => {
|
await test("provider filters are the curated archive list", () => {
|
||||||
assert.ok(PROVIDER_FILTERS.length >= 12, `provider list has ${PROVIDER_FILTERS.length} entries`);
|
assert.ok(PROVIDER_FILTERS.length >= 12, `provider list has ${PROVIDER_FILTERS.length} entries`);
|
||||||
assert.equal(new Set(PROVIDER_FILTERS).size, PROVIDER_FILTERS.length);
|
assert.equal(new Set(PROVIDER_FILTERS).size, PROVIDER_FILTERS.length);
|
||||||
|
|||||||
+37
-1
@@ -291,6 +291,11 @@
|
|||||||
if (value === "" || !Number.isFinite(number) || number < 0) return "";
|
if (value === "" || !Number.isFinite(number) || number < 0) return "";
|
||||||
return String(Math.round(number * (unit ? sizeUnit(unit) : 1)));
|
return String(Math.round(number * (unit ? sizeUnit(unit) : 1)));
|
||||||
};
|
};
|
||||||
|
const title = state.filters.title.trim()
|
||||||
|
? `title=${encodeURIComponent(state.filters.title.trim())}&${
|
||||||
|
state.filters.titleExact ? "titleExact=1&" : ""
|
||||||
|
}`
|
||||||
|
: "";
|
||||||
const sizeLt = state.filters.sizeLt !== "" ? `sizeLt=${amount(state.filters.sizeLt, state.filters.sizeLtUnit)}&` : "";
|
const sizeLt = state.filters.sizeLt !== "" ? `sizeLt=${amount(state.filters.sizeLt, state.filters.sizeLtUnit)}&` : "";
|
||||||
const sizeGt = state.filters.sizeGt !== "" ? `sizeGt=${amount(state.filters.sizeGt, state.filters.sizeGtUnit)}&` : "";
|
const sizeGt = state.filters.sizeGt !== "" ? `sizeGt=${amount(state.filters.sizeGt, state.filters.sizeGtUnit)}&` : "";
|
||||||
const lenLt = state.filters.lenLt !== "" ? `lenLt=${amount(state.filters.lenLt)}&` : "";
|
const lenLt = state.filters.lenLt !== "" ? `lenLt=${amount(state.filters.lenLt)}&` : "";
|
||||||
@@ -301,7 +306,7 @@
|
|||||||
const before = state.filters.before ? `before=${state.filters.before}T23:59:59.999Z&` : "";
|
const before = state.filters.before ? `before=${state.filters.before}T23:59:59.999Z&` : "";
|
||||||
return (
|
return (
|
||||||
`${kind}${source}${channel}${authors}${origin}${cached}${exts}${excludeHosts}` +
|
`${kind}${source}${channel}${authors}${origin}${cached}${exts}${excludeHosts}` +
|
||||||
`${sort}${excludeProviders}${excludeSources}${excludeExts}` +
|
`${sort}${title}${excludeProviders}${excludeSources}${excludeExts}` +
|
||||||
`${includeProviders}${includeSources}${sizeLt}${sizeGt}${lenLt}${lenGt}` +
|
`${includeProviders}${includeSources}${sizeLt}${sizeGt}${lenLt}${lenGt}` +
|
||||||
`${aspect}${resolution}${after}${before}`
|
`${aspect}${resolution}${after}${before}`
|
||||||
);
|
);
|
||||||
@@ -1044,6 +1049,7 @@
|
|||||||
|
|
||||||
function renderCard(item) {
|
function renderCard(item) {
|
||||||
const figure = el("figure", "vk-media-figure");
|
const figure = el("figure", "vk-media-figure");
|
||||||
|
if (item.title) figure.title = item.title;
|
||||||
const media = el("div", "vk-media-media");
|
const media = el("div", "vk-media-media");
|
||||||
const imageSource = item.localUrl || item.thumbUrl;
|
const imageSource = item.localUrl || item.thumbUrl;
|
||||||
const fallbackSource = item.source !== "embed" ? item.url : null;
|
const fallbackSource = item.source !== "embed" ? item.url : null;
|
||||||
@@ -1174,6 +1180,8 @@
|
|||||||
exts: [],
|
exts: [],
|
||||||
excludeHosts: ["4chan.org", "i.4cdn.org"],
|
excludeHosts: ["4chan.org", "i.4cdn.org"],
|
||||||
sort: "newest",
|
sort: "newest",
|
||||||
|
title: "",
|
||||||
|
titleExact: false,
|
||||||
excludeProviders: [],
|
excludeProviders: [],
|
||||||
excludeSources: [],
|
excludeSources: [],
|
||||||
includeProviders: [],
|
includeProviders: [],
|
||||||
@@ -1204,6 +1212,8 @@
|
|||||||
excludeSources: [...state.filters.excludeSources],
|
excludeSources: [...state.filters.excludeSources],
|
||||||
includeProviders: [...state.filters.includeProviders],
|
includeProviders: [...state.filters.includeProviders],
|
||||||
includeSources: [...state.filters.includeSources],
|
includeSources: [...state.filters.includeSources],
|
||||||
|
title: state.filters.title,
|
||||||
|
titleExact: state.filters.titleExact,
|
||||||
excludeExts: [...state.filters.excludeExts]
|
excludeExts: [...state.filters.excludeExts]
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
@@ -1224,6 +1234,8 @@
|
|||||||
excludeSources: [...state.draft.excludeSources],
|
excludeSources: [...state.draft.excludeSources],
|
||||||
includeProviders: [...state.draft.includeProviders],
|
includeProviders: [...state.draft.includeProviders],
|
||||||
includeSources: [...state.draft.includeSources],
|
includeSources: [...state.draft.includeSources],
|
||||||
|
title: state.draft.title,
|
||||||
|
titleExact: state.draft.titleExact,
|
||||||
excludeExts: [...state.draft.excludeExts],
|
excludeExts: [...state.draft.excludeExts],
|
||||||
sizeLt: state.draft.sizeLt,
|
sizeLt: state.draft.sizeLt,
|
||||||
sizeGt: state.draft.sizeGt,
|
sizeGt: state.draft.sizeGt,
|
||||||
@@ -1382,6 +1394,30 @@
|
|||||||
draft.sort = value;
|
draft.sort = value;
|
||||||
}
|
}
|
||||||
),
|
),
|
||||||
|
(() => {
|
||||||
|
const row = el("div", "vk-media-toolbar");
|
||||||
|
const input = document.createElement("input");
|
||||||
|
input.type = "text";
|
||||||
|
input.className = "vk-media-input";
|
||||||
|
input.placeholder = "Video title (Ma Ca Ju finds Man ... car jumps)";
|
||||||
|
input.value = draft.title;
|
||||||
|
input.addEventListener("input", () => {
|
||||||
|
draft.title = input.value;
|
||||||
|
});
|
||||||
|
row.append(input);
|
||||||
|
|
||||||
|
const exact = el("label", "vk-media-check");
|
||||||
|
const box = document.createElement("input");
|
||||||
|
box.type = "checkbox";
|
||||||
|
box.checked = draft.titleExact;
|
||||||
|
box.title = "Match the whole phrase instead of per-word prefixes";
|
||||||
|
box.addEventListener("change", () => {
|
||||||
|
draft.titleExact = box.checked;
|
||||||
|
});
|
||||||
|
exact.append(box, document.createTextNode(" Exact match (quotes)"));
|
||||||
|
row.append(exact);
|
||||||
|
return row;
|
||||||
|
})(),
|
||||||
filterSelect(
|
filterSelect(
|
||||||
[
|
[
|
||||||
["", "Any ratio"],
|
["", "Any ratio"],
|
||||||
|
|||||||
@@ -904,6 +904,11 @@ function buildServer(): McpServer {
|
|||||||
.array(z.string())
|
.array(z.string())
|
||||||
.optional()
|
.optional()
|
||||||
.describe("Keep only these row sources (attachment, sticker, embed)"),
|
.describe("Keep only these row sources (attachment, sticker, embed)"),
|
||||||
|
titleQuery: z
|
||||||
|
.string()
|
||||||
|
.optional()
|
||||||
|
.describe("Search embed titles (YouTube). Multi-word prefix match by default"),
|
||||||
|
titleExact: z.boolean().optional().describe("Treat titleQuery as one exact phrase"),
|
||||||
sizeLessThan: z.number().optional().describe("Only items smaller than this many bytes"),
|
sizeLessThan: z.number().optional().describe("Only items smaller than this many bytes"),
|
||||||
sizeGreaterThan: z.number().optional().describe("Only items larger than this many bytes"),
|
sizeGreaterThan: z.number().optional().describe("Only items larger than this many bytes"),
|
||||||
lengthLessThan: z.number().optional().describe("Only items shorter than this many seconds"),
|
lengthLessThan: z.number().optional().describe("Only items shorter than this many seconds"),
|
||||||
@@ -941,6 +946,8 @@ function buildServer(): McpServer {
|
|||||||
excludeSources,
|
excludeSources,
|
||||||
includeProviders,
|
includeProviders,
|
||||||
includeSources,
|
includeSources,
|
||||||
|
titleQuery,
|
||||||
|
titleExact,
|
||||||
sizeLessThan,
|
sizeLessThan,
|
||||||
sizeGreaterThan,
|
sizeGreaterThan,
|
||||||
lengthLessThan,
|
lengthLessThan,
|
||||||
@@ -969,6 +976,8 @@ function buildServer(): McpServer {
|
|||||||
excludeSources,
|
excludeSources,
|
||||||
includeProviders,
|
includeProviders,
|
||||||
includeSources,
|
includeSources,
|
||||||
|
titleQuery,
|
||||||
|
titleExact,
|
||||||
sizeLessThan,
|
sizeLessThan,
|
||||||
sizeGreaterThan,
|
sizeGreaterThan,
|
||||||
lengthLessThan,
|
lengthLessThan,
|
||||||
|
|||||||
@@ -135,6 +135,8 @@ export interface MediaItemResult {
|
|||||||
thumbUrl: string | null;
|
thumbUrl: string | null;
|
||||||
provider: string | null;
|
provider: string | null;
|
||||||
downloaded: boolean;
|
downloaded: boolean;
|
||||||
|
/** Embed title (YouTube and friends), when Discord provided one. */
|
||||||
|
title: string | null;
|
||||||
createdAt: string;
|
createdAt: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -719,6 +721,8 @@ export class MediaService {
|
|||||||
excludeSources?: string[];
|
excludeSources?: string[];
|
||||||
includeProviders?: string[];
|
includeProviders?: string[];
|
||||||
includeSources?: string[];
|
includeSources?: string[];
|
||||||
|
titleQuery?: string;
|
||||||
|
titleExact?: boolean;
|
||||||
sizeLessThan?: number;
|
sizeLessThan?: number;
|
||||||
sizeGreaterThan?: number;
|
sizeGreaterThan?: number;
|
||||||
lengthLessThan?: number;
|
lengthLessThan?: number;
|
||||||
@@ -850,6 +854,8 @@ export class MediaService {
|
|||||||
excludeSources?: string[];
|
excludeSources?: string[];
|
||||||
includeProviders?: string[];
|
includeProviders?: string[];
|
||||||
includeSources?: string[];
|
includeSources?: string[];
|
||||||
|
titleQuery?: string;
|
||||||
|
titleExact?: boolean;
|
||||||
sizeLessThan?: number;
|
sizeLessThan?: number;
|
||||||
sizeGreaterThan?: number;
|
sizeGreaterThan?: number;
|
||||||
lengthLessThan?: number;
|
lengthLessThan?: number;
|
||||||
@@ -887,6 +893,8 @@ export class MediaService {
|
|||||||
excludeSources: options.excludeSources,
|
excludeSources: options.excludeSources,
|
||||||
includeProviders: options.includeProviders,
|
includeProviders: options.includeProviders,
|
||||||
includeSources: options.includeSources,
|
includeSources: options.includeSources,
|
||||||
|
titleQuery: options.titleQuery,
|
||||||
|
titleExact: options.titleExact,
|
||||||
sizeLessThan: options.sizeLessThan,
|
sizeLessThan: options.sizeLessThan,
|
||||||
sizeGreaterThan: options.sizeGreaterThan,
|
sizeGreaterThan: options.sizeGreaterThan,
|
||||||
lengthLessThan: options.lengthLessThan,
|
lengthLessThan: options.lengthLessThan,
|
||||||
@@ -949,6 +957,8 @@ export class MediaService {
|
|||||||
excludeSources?: string[];
|
excludeSources?: string[];
|
||||||
includeProviders?: string[];
|
includeProviders?: string[];
|
||||||
includeSources?: string[];
|
includeSources?: string[];
|
||||||
|
titleQuery?: string;
|
||||||
|
titleExact?: boolean;
|
||||||
sizeLessThan?: number;
|
sizeLessThan?: number;
|
||||||
sizeGreaterThan?: number;
|
sizeGreaterThan?: number;
|
||||||
lengthLessThan?: number;
|
lengthLessThan?: number;
|
||||||
@@ -993,6 +1003,7 @@ export class MediaService {
|
|||||||
thumbUrl: row.preview_hash ? `/api/media/${meta.id}/${row.id}/thumb` : null,
|
thumbUrl: row.preview_hash ? `/api/media/${meta.id}/${row.id}/thumb` : null,
|
||||||
provider: row.provider,
|
provider: row.provider,
|
||||||
downloaded: row.downloaded === 1,
|
downloaded: row.downloaded === 1,
|
||||||
|
title: row.title,
|
||||||
createdAt: row.created_at
|
createdAt: row.created_at
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -213,6 +213,8 @@ function parseMediaFilters(query: Record<string, unknown>) {
|
|||||||
excludeSources: list(query.excludeSources),
|
excludeSources: list(query.excludeSources),
|
||||||
includeProviders: list(query.includeProviders),
|
includeProviders: list(query.includeProviders),
|
||||||
includeSources: list(query.includeSources),
|
includeSources: list(query.includeSources),
|
||||||
|
titleQuery: query.title ? String(query.title).slice(0, 200) : undefined,
|
||||||
|
titleExact: query.titleExact === "1" || query.titleExact === "true",
|
||||||
sizeLessThan: amount(query.sizeLt),
|
sizeLessThan: amount(query.sizeLt),
|
||||||
sizeGreaterThan: amount(query.sizeGt),
|
sizeGreaterThan: amount(query.sizeGt),
|
||||||
lengthLessThan: amount(query.lenLt),
|
lengthLessThan: amount(query.lenLt),
|
||||||
|
|||||||
@@ -125,6 +125,15 @@ try {
|
|||||||
|
|
||||||
await page.getByRole("button", { name: "Advanced search" }).click();
|
await page.getByRole("button", { name: "Advanced search" }).click();
|
||||||
await page.waitForSelector(".vk-media-drawer:not([hidden])", { timeout: 5000 });
|
await page.waitForSelector(".vk-media-drawer:not([hidden])", { timeout: 5000 });
|
||||||
|
check(
|
||||||
|
"title search field present",
|
||||||
|
(await page.locator('.vk-media-drawer input[type="text"]').count()) >= 1
|
||||||
|
);
|
||||||
|
check(
|
||||||
|
"exact-match toggle present",
|
||||||
|
(await page.locator(".vk-media-drawer .vk-media-check", { hasText: "Exact match" }).count()) === 1
|
||||||
|
);
|
||||||
|
|
||||||
const sourceBoxes = await page
|
const sourceBoxes = await page
|
||||||
.locator(".vk-media-exclude-group")
|
.locator(".vk-media-exclude-group")
|
||||||
.first()
|
.first()
|
||||||
|
|||||||
Reference in New Issue
Block a user