Support wildcards in the title search

This commit is contained in:
Fisto
2026-09-17 13:43:27 +02:00
parent 0fb93c4830
commit 8befc05591
8 changed files with 91 additions and 15 deletions
+5
View File
@@ -40,6 +40,11 @@ mobile draft at 390px.
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB - [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`: cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200). `previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
- [x] C52 Wildcard title search. `*` now maps to a LIKE wildcard in both title modes: a leading
`*` drops the word anchor (`*car` finds *scar* and *discard*), in-token stars match between
letters (`c*r*`), and a trailing star matches the plain prefix. `%` and `_` are escaped with an
`ESCAPE` clause so they only match themselves. Unit tests cover prefix vs wildcard vs in-word
matches; the drawer hint documents the syntax.
- [x] C51 YouTube Shorts normalization. Shorts links broke the embedded player and View full - [x] C51 YouTube Shorts normalization. Shorts links broke the embedded player and View full
because the viewer needs `watch?v=`. Extraction now rewrites `youtube.com/shorts/<id>` (and because the viewer needs `watch?v=`. Extraction now rewrites `youtube.com/shorts/<id>` (and
`m.`) to the watch form, the viewer also accepts `/shorts/<id>` defensively, and a one-time `m.`) to the watch form, the viewer also accepts `/shorts/<id>` defensively, and a one-time
-5
View File
@@ -64,11 +64,6 @@ Ordered backlog. `CURRENT.md` tracks the live session state.
## Media archive follow-ups ## Media archive follow-ups
1. **Wildcard title search.** Support `*` in the title field: a leading `*` means substring anywhere
(`*car` finds *scar*, *discard*), `c*r` matches inside a word, and a plain token keeps today's
word-prefix rule. Map `*` to SQL `%`, escape literal `%`/`_` in the input, drop the word-boundary
anchor for tokens that start with `*`, and document it in the Advanced drawer hint.
2. **Title work in its own lane.** Title resolution talks straight to `youtube.com/oembed` 2. **Title work in its own lane.** Title resolution talks straight to `youtube.com/oembed`
(daemon fetch; no Discord API, no bot account), but it currently shares the media-traffic lock (daemon fetch; no Discord API, no bot account), but it currently shares the media-traffic lock
and the preview batch pacing, so it serializes behind walks and previews. Move it to its own and the preview batch pacing, so it serializes behind walks and previews. Move it to its own
+3 -1
View File
@@ -103,7 +103,9 @@ indexed watch URL folds into it (preview/file fields merge, message links move o
Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed Discord's `title` (YouTube and friends) plus `title_key`, the lowercased, punctuation-collapsed
form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps form used by the title search: multi-word prefix tokens by default ("Ma Ca Ju" finds *Man who jumps
over car*), one phrase with the exact-match toggle, and rows without a title drop out while a over car*), one phrase with the exact-match toggle, and rows without a title drop out while a
title query is set. Filter presets live in title query is set. `*` is a wildcard anywhere in a token: `*car` matches inside words (*scar*,
*discard*), `c*r*` matches between letters, `car*` behaves like the plain prefix. Literal `%` and
`_` are escaped, so they only match themselves. Filter presets live in
`media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and `media-scrape/src/filter-presets.ts`: aspect buckets (vertical, square, 4:3, 16:9, 21:9, other) and
resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is resolution buckets by the shortest side (SD, HD, FHD, QHD, 4K+, unknown). A missing resolution is
filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available. filled with `ffprobe` after a download when `VESKSLOP_FFPROBE` is available.
+1 -1
View File
@@ -150,7 +150,7 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
include** and **Must include** groups with nested collapsibles, a title search for embed titles include** and **Must include** groups with nested collapsibles, a title search for embed titles
(multi-word prefix matching, with an **Exact match (quotes)** toggle) for sources (the common (multi-word prefix matching with `*` wildcards, and an **Exact match (quotes)** toggle) for sources (the common
providers providers
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units), in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
+2 -1
View File
@@ -139,7 +139,8 @@ box live in Chat, not Call, and the connection stats are not chips yet.
bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as
`Giphy/Tenor`. A title field searches embed titles (YouTube): default is multi-word prefix `Giphy/Tenor`. A title field searches embed titles (YouTube): default is multi-word prefix
matching (each token must start a word of the title, order-free), and **Exact match (quotes)** matching (each token must start a word of the title, order-free), and **Exact match (quotes)**
treats the input as one phrase. A sort select orders the grid by newest, oldest, user, size, or treats the input as one phrase. `*` acts as a wildcard in both modes (`*car` matches *scar*,
`c*r*` between letters); literal `%` and `_` are escaped. A sort select orders the grid by newest, oldest, user, size, or
length. Size and length. Size and
length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot
tab. Filters do not tab. Filters do not
+27 -5
View File
@@ -187,6 +187,20 @@ export function normalizeTitleKey(title: string): string {
.replace(/\s+/g, " "); .replace(/\s+/g, " ");
} }
/** Lowercase a title query and collapse punctuation, but keep `*` as a wildcard marker. */
export function normalizeTitleQuery(query: string): string {
return query
.toLowerCase()
.replace(/[^a-z0-9*]+/g, " ")
.trim()
.replace(/\s+/g, " ");
}
/** Escape LIKE metacharacters, then turn `*` into `%`. */
function titleLikePattern(value: string): string {
return value.replace(/[\\%_]/g, match => `\\${match}`).replace(/\*/g, "%");
}
const SCHEMA = ` const SCHEMA = `
CREATE TABLE IF NOT EXISTS messages ( CREATE TABLE IF NOT EXISTS messages (
id TEXT PRIMARY KEY, id TEXT PRIMARY KEY,
@@ -803,14 +817,22 @@ export class ScrapeStore {
values.push(options.lengthGreaterThan); values.push(options.lengthGreaterThan);
} }
if (options.titleQuery?.trim()) { if (options.titleQuery?.trim()) {
const query = normalizeTitleKey(options.titleQuery); const query = normalizeTitleQuery(options.titleQuery);
if (options.titleExact) { if (options.titleExact) {
conditions.push("title_key LIKE ?"); // One phrase, substring of the title; `*` still acts as a wildcard.
values.push(`%${query}%`); conditions.push("title_key LIKE ? ESCAPE '\\'");
values.push(`%${titleLikePattern(query)}%`);
} else { } else {
for (const token of query.split(" ").filter(Boolean)) { for (const token of query.split(" ").filter(Boolean)) {
conditions.push("(' ' || COALESCE(title_key, '')) LIKE ?"); const pattern = titleLikePattern(token);
values.push(`% ${token}%`); if (token.startsWith("*")) {
// Leading `*`: anywhere in a word, not just the start.
conditions.push("COALESCE(title_key, '') LIKE ? ESCAPE '\\'");
values.push(`%${pattern}%`);
} else {
conditions.push("(' ' || COALESCE(title_key, '')) LIKE ? ESCAPE '\\'");
values.push(`% ${pattern}%`);
}
} }
} }
} }
+51
View File
@@ -31,6 +31,7 @@ import {
dirSizeBytes, dirSizeBytes,
listScrapes, listScrapes,
normalizeTitleKey, normalizeTitleKey,
normalizeTitleQuery,
readScrapeMeta, readScrapeMeta,
ScrapeStore, ScrapeStore,
writeScrapeMeta writeScrapeMeta
@@ -1619,6 +1620,56 @@ void (async () => {
store.close(); store.close();
}); });
await test("title wildcards match anywhere and stay literal for % and _", () => {
const dir = join(root, "title-wildcards");
mkdirSync(dir, { recursive: true });
const store = new ScrapeStore(dir);
const embed = (id: string, title: string) => ({
type: "video",
url: `https://www.youtube.com/watch?v=${id}`,
title,
provider: "YouTube",
videoUrl: `https://www.youtube.com/watch?v=${id}`,
imageUrl: null,
thumbnailUrl: null
});
store.insertPage([
message("1300", {
embeds: [
embed("w1", "Scar Tissue"),
embed("w2", "Discard the rest"),
embed("w3", "Car wash"),
embed("w4", "Cartoon physics"),
embed("w5", "Carly and the r")
]
})
]);
const ids = (options: Record<string, unknown>) =>
store
.queryMedia({ limit: 50, ...options })
.map(row => row.url.split("v=")[1])
.sort();
// Word-prefix default: car starts a word.
assert.deepEqual(ids({ titleQuery: "car" }), ["w3", "w4", "w5"]);
// Leading *: anywhere, so scar and discard join in.
assert.deepEqual(ids({ titleQuery: "*car" }), ["w1", "w2", "w3", "w4", "w5"]);
// Trailing * behaves like the prefix form.
assert.deepEqual(ids({ titleQuery: "car*" }), ["w3", "w4", "w5"]);
// In-word wildcard after a word start.
assert.deepEqual(ids({ titleQuery: "c*r" }), ["w3", "w4", "w5"]);
assert.deepEqual(ids({ titleQuery: "c*r*" }), ["w3", "w4", "w5"]);
assert.deepEqual(ids({ titleQuery: "*c*r*" }), ["w1", "w2", "w3", "w4", "w5"]);
store.close();
});
await test("normalizeTitleQuery keeps stars and drops punctuation", () => {
assert.equal(normalizeTitleQuery(" *CaR, C*r* "), "*car c*r*");
assert.equal(normalizeTitleQuery("Boat"), "boat");
});
await test("normalizeTitleKey collapses punctuation and case", () => { await test("normalizeTitleKey collapses punctuation and case", () => {
assert.equal(normalizeTitleKey("Woman's Day: 3 Things!"), "woman s day 3 things"); assert.equal(normalizeTitleKey("Woman's Day: 3 Things!"), "woman s day 3 things");
assert.equal(normalizeTitleKey(" A---B "), "a b"); assert.equal(normalizeTitleKey(" A---B "), "a b");
+2 -2
View File
@@ -1400,7 +1400,7 @@
const input = document.createElement("input"); const input = document.createElement("input");
input.type = "text"; input.type = "text";
input.className = "vk-media-input"; input.className = "vk-media-input";
input.placeholder = "Video title (Ma Ca Ju finds Man ... car jumps)"; input.placeholder = "Video title (Ma Ca Ju; *car, c*r* wildcards)";
input.value = draft.title; input.value = draft.title;
input.addEventListener("input", () => { input.addEventListener("input", () => {
draft.title = input.value; draft.title = input.value;
@@ -1411,7 +1411,7 @@
const box = document.createElement("input"); const box = document.createElement("input");
box.type = "checkbox"; box.type = "checkbox";
box.checked = draft.titleExact; box.checked = draft.titleExact;
box.title = "Match the whole phrase instead of per-word prefixes"; box.title = "Match the whole phrase instead of per-word prefixes; * still works as a wildcard";
box.addEventListener("change", () => { box.addEventListener("change", () => {
draft.titleExact = box.checked; draft.titleExact = box.checked;
}); });