Add the picked providers and a Must include group

This commit is contained in:
Fisto
2026-09-17 11:28:27 +02:00
parent 83dd9bc0fa
commit c6caa168be
11 changed files with 229 additions and 119 deletions
+10
View File
@@ -40,6 +40,16 @@ mobile draft at 390px.
- [x] C11 preview caching: Discord-hosted image/GIF previews (WebP 320px, hash-addressed, 5 MiB
cap, one attempt) run after every walk in both modes. Live run `...-20260916090722`:
`previewed: 2`; thumbs served over `/api/media/<scrape>/<item>/thumb` (200).
- [x] C48 Expanded providers and the Must include mirror. The list grew to 19 entries from the
user's picks: reddit, twitter, tiktok, instagram, streamable, facebook, gyazo, a merged
`giphy,tenor` (one checkbox, two substrings), 2ch.hk (stand-in for 4chan; the CDN hosts stay
behind the default URL exclusion), funnyjunk, and internet archive. The drawer gained a
**Must include** block mirroring **Do not include**: provider/source-kind checkboxes
(`includeProviders`, `includeSources`), file-type chips, and the size/length bounds (moved here
since they are inclusions, not exclusions). Store, `/api/media`, `/api/media/stats`, and MCP
all carry the two include params. Live-verified: `includeProviders=youtube` returns only
YouTube rows, `giphy,tenor` includes/excludes the same rows complementarily; smoke checks both
groups against the API list (22 = 19 + 3 kinds).
- [x] C47 Provider list from the dataset. Aggregated the `provider` column over all 11 scraper
databases (distinct URLs): discord 13631 (folding cdn.discordapp.com and media.discordapp.net),
youtube 4840, tenor 352, twitch 196, twitter 111, steam 83, medal 59, sketchfab 31, gfycat 30,
+3 -3
View File
@@ -149,9 +149,9 @@ that deletion is irreversible. The delete button is disabled for a 5 second coun
and the multi-select People popover. The **Advanced search** drawer has aspect ratio, resolution,
source channel, posted-date range, file-type chips, cached-only, a default-on hide for 4chan
links (they 404 almost always), a sort select (newest, oldest, user, size, length), and a **Do not
include** group with nested collapsibles for sources (the ten most common providers plus embeds,
attachments, and stickers), file types, size (with B/KB/MB/GB units), and length (seconds), with
**Apply filters** / **Reset** in the bar. Size and length ranges drop
include** and **Must include** groups with nested collapsibles for sources (the common providers
in the archive plus embeds, attachments, and stickers), file types, size (with B/KB/MB/GB units),
and length (seconds), with **Apply filters** / **Reset** in the bar. Size and length ranges drop
rows with no known value. The bar and drawer show only on the Media tab.
Nothing renders until **Load media** is pressed. A **Metadata** dropdown toggles which fields
show in the small banner under each tile (User, Type, Date, Source, Resolution, Length). Video
+10 -7
View File
@@ -131,12 +131,15 @@ box live in Chat, not Call, and the connection stats are not chips yet.
popover. The **Advanced search** drawer holds aspect ratio, resolution, source channel,
posted-date range, file-type chips (video mp4/webm/mov/mkv, image png/jpg/gif/webp/avif, audio),
**Show only cached previews**, a default-on **Hide 4chan links (usually dead)**, and a
**Do not include** group with nested collapsibles for sources (the ten most common providers in
the archive — Discord, YouTube, Tenor, Twitch, Twitter, Steam, Medal, Sketchfab, Gfycat, Imgur —
plus embeds, attachments, stickers), file types, size (less/greater than, with B/KB/MB/GB units),
and length (less/greater than, seconds). A sort select orders the grid by newest,
oldest, user, size, or length. Size and length ranges drop rows with no known value. The filter
bar and the drawer are hidden on the Bot tab. Filters do not
**Do not include** and **Must include** groups, each with nested collapsibles. **Do not include**
hides sources (the common providers in the archive — Discord, YouTube, Giphy/Tenor, Twitch,
Twitter, Reddit, Streamable, Instagram, Facebook, Steam, Gyazo, TikTok, Medal, Sketchfab, Gfycat,
2ch.hk, Imgur, FunnyJunk, Internet Archive — plus embeds, attachments, stickers) and file types.
**Must include** requires them (same provider list and kinds, file-type chips, and size/length
bounds with B/KB/MB/GB units or seconds). A provider entry may carry several substrings, shown as
`Giphy/Tenor`. A sort select orders the grid by newest, oldest, user, size, or length. Size and
length bounds drop rows with no known value. The filter bar and the drawer are hidden on the Bot
tab. Filters do not
apply until **Apply filters** is pressed; **Reset** clears every filter and the zoom, unloads the
grid, and brings back the initial Load media prompt.
- A **Card size** slider scales tiles from two-thirds of the old size up to one image per row; the
@@ -286,7 +289,7 @@ replaceable.
| POST | `/api/audio/stop` | none | Stop speech and clear the queue |
| POST | `/api/audio/capture` | `start` | Start or stop capture |
| GET | `/api/transcript` | `limit?`, `sessionId?` | Sessions and captured segments |
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?` (comma lists), `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
| GET | `/api/media` | `kind?`, `source?`, `channelId?`, `authorId?` (comma list), `origin?` (`guild`/`dm`), `cachedOnly?`, `ext?` (comma list), `excludeHosts?` (comma list), `excludeExt?`, `excludeProviders?`, `excludeSources?`, `includeProviders?`, `includeSources?` (comma lists), `sizeLt?`/`sizeGt?` (bytes), `lenLt?`/`lenGt?` (seconds), `sort?` (`newest` default, `oldest`, `user`, `size`, `length`), `aspect?`, `resolution?`, `after?`, `before?`, `limit?`, `offset?` | Media items across every scrape, newest first. `source` is `attachment`, `sticker`, or `embed`; `aspect` and `resolution` are the preset keys; `after`/`before` are ISO times on `posted_at`. Items carry `authorName`, `postedAt`, `width`, `height`, `aspectRatio`, `contentType`, `durationSecs`, `previewError`, `animUrl`; `localUrl`/`thumbUrl` when a file or preview exists |
| GET | `/api/media/:scrapeId/:itemId/file` | none | The downloaded file, content-typed from its extension |
| GET | `/api/media/:scrapeId/:itemId/thumb` | none | The cached preview (type from `preview_ext`) |
| GET | `/api/media/:scrapeId/:itemId/stream` | none | Playback/full-view source: the local file, or the remote URL refreshed from the source message. Supports `Range`. Non-media upstream responses (provider pages) return `415` |
+17 -6
View File
@@ -50,21 +50,32 @@ export function resolutionCondition(preset: string): string | null {
}
/**
* The providers that dominate the archive, for the "Do not include" shortcuts.
* The providers that dominate the archive, for the "Do not include" and "Must include" shortcuts.
* Aggregated 2026-09-17 over media-scrape/db (`provider` column, distinct URLs; discord folds in
* cdn.discordapp.com and media.discordapp.net). Matching is a case-insensitive substring on
* `provider`, so "twitter" also covers fxtwitter.com. The 4chan hosts stay out: the default URL
* host exclusion already hides them.
* cdn.discordapp.com, media.discordapp.net, and images-ext-*.discordapp.net). Matching is a
* case-insensitive substring on `provider`, so "twitter" also covers fxtwitter.com, vxtwitter.com,
* and twitfix; "reddit" covers www.reddit.com and vxreddit. Entries with several substrings are
* joined with a comma and the UI shows "Giphy/Tenor": checking it applies both. 4chan's CDN hosts
* are hidden by the default URL host exclusion, so 2ch.hk stands in for the provider box.
*/
export const PROVIDER_FILTERS = [
"discord",
"youtube",
"tenor",
"giphy,tenor",
"twitch",
"twitter",
"reddit",
"streamable",
"instagram",
"facebook",
"steam",
"gyazo",
"tiktok",
"medal",
"sketchfab",
"gfycat",
"imgur"
"2ch.hk",
"imgur",
"funnyjunk",
"internet archive"
] as const;
+15
View File
@@ -119,6 +119,10 @@ export interface MediaFilterOptions {
excludeProviders?: string[];
/** Row sources to hide (attachment, sticker, embed). */
excludeSources?: string[];
/** Keep only providers matching one of these substrings; empty keeps all. */
includeProviders?: string[];
/** Keep only these row sources; empty keeps all. */
includeSources?: string[];
/** Bytes; rows with no known size drop out while this is set. */
sizeLessThan?: number;
sizeGreaterThan?: number;
@@ -609,6 +613,17 @@ export class ScrapeStore {
conditions.push(`COALESCE(source, '') NOT IN (${options.excludeSources.map(() => "?").join(", ")})`);
values.push(...options.excludeSources);
}
if (options.includeProviders?.length) {
const clauses = options.includeProviders.map(() => "lower(COALESCE(provider, '')) LIKE ?");
conditions.push(`(${clauses.join(" OR ")})`);
for (const provider of options.includeProviders) {
values.push(`%${provider.toLowerCase()}%`);
}
}
if (options.includeSources?.length) {
conditions.push(`COALESCE(source, '') IN (${options.includeSources.map(() => "?").join(", ")})`);
values.push(...options.includeSources);
}
if (options.sizeLessThan !== undefined) {
conditions.push("COALESCE(file_bytes, size) < ?");
values.push(options.sizeLessThan);
+26 -8
View File
@@ -1348,20 +1348,34 @@ void (async () => {
});
await test("provider filters are the curated archive list", () => {
assert.ok(
PROVIDER_FILTERS.length >= 8 && PROVIDER_FILTERS.length <= 12,
`provider list has ${PROVIDER_FILTERS.length} entries`
);
assert.ok(PROVIDER_FILTERS.length >= 12, `provider list has ${PROVIDER_FILTERS.length} entries`);
assert.equal(new Set(PROVIDER_FILTERS).size, PROVIDER_FILTERS.length);
for (const value of PROVIDER_FILTERS) {
assert.equal(value, value.toLowerCase());
assert.ok(!value.includes(" ") && !value.includes("."), `host-like entry: ${value}`);
for (const entry of PROVIDER_FILTERS) {
const values = entry.split(",");
assert.ok(values.length >= 1 && values.every(value => value.length > 0), entry);
for (const value of values) assert.equal(value, value.toLowerCase(), entry);
}
for (const required of ["discord", "youtube", "twitch", "tenor"]) {
for (const required of [
"discord",
"youtube",
"twitch",
"twitter",
"reddit",
"instagram",
"tiktok",
"streamable",
"facebook",
"gyazo",
"funnyjunk",
"internet archive",
"2ch.hk"
]) {
assert.ok((PROVIDER_FILTERS as readonly string[]).includes(required), required);
}
assert.ok((PROVIDER_FILTERS as readonly string[]).includes("giphy,tenor"), "giphy/tenor merged");
});
await test("store exclusion, size, and length filters", () => {
@@ -1417,6 +1431,10 @@ void (async () => {
assert.deepEqual(ids({ lengthGreaterThan: 10 }), [5]);
// Unknown lengths drop out while a length range is set.
assert.deepEqual(ids({ lengthLessThan: 10 }), []);
assert.deepEqual(ids({ includeProviders: ["youtube"] }), [3]);
assert.deepEqual(ids({ includeProviders: ["youtube", "twitch"] }), [3, 4]);
assert.deepEqual(ids({ includeSources: ["embed"] }), [3, 4]);
assert.deepEqual(ids({ includeSources: ["attachment"] }), [1, 2, 5]);
store.close();
});
+105 -85
View File
@@ -280,6 +280,12 @@
const excludeExts = state.filters.excludeExts.length
? `excludeExt=${state.filters.excludeExts.join(",")}&`
: "";
const includeProviders = state.filters.includeProviders.length
? `includeProviders=${state.filters.includeProviders.join(",")}&`
: "";
const includeSources = state.filters.includeSources.length
? `includeSources=${state.filters.includeSources.join(",")}&`
: "";
const amount = (value, unit) => {
const number = Number(value);
if (value === "" || !Number.isFinite(number) || number < 0) return "";
@@ -295,7 +301,8 @@
const before = state.filters.before ? `before=${state.filters.before}T23:59:59.999Z&` : "";
return (
`${kind}${source}${channel}${authors}${origin}${cached}${exts}${excludeHosts}` +
`${sort}${excludeProviders}${excludeSources}${excludeExts}${sizeLt}${sizeGt}${lenLt}${lenGt}` +
`${sort}${excludeProviders}${excludeSources}${excludeExts}` +
`${includeProviders}${includeSources}${sizeLt}${sizeGt}${lenLt}${lenGt}` +
`${aspect}${resolution}${after}${before}`
);
}
@@ -1169,6 +1176,8 @@
sort: "newest",
excludeProviders: [],
excludeSources: [],
includeProviders: [],
includeSources: [],
excludeExts: [],
sizeLt: "",
sizeGt: "",
@@ -1193,6 +1202,8 @@
excludeHosts: [...state.filters.excludeHosts],
excludeProviders: [...state.filters.excludeProviders],
excludeSources: [...state.filters.excludeSources],
includeProviders: [...state.filters.includeProviders],
includeSources: [...state.filters.includeSources],
excludeExts: [...state.filters.excludeExts]
};
}
@@ -1211,6 +1222,8 @@
sort: state.draft.sort,
excludeProviders: [...state.draft.excludeProviders],
excludeSources: [...state.draft.excludeSources],
includeProviders: [...state.draft.includeProviders],
includeSources: [...state.draft.includeSources],
excludeExts: [...state.draft.excludeExts],
sizeLt: state.draft.sizeLt,
sizeGt: state.draft.sizeGt,
@@ -1434,34 +1447,6 @@
})
);
const typeRow = (label, extensions) => {
const row = el("div", "vk-media-toolbar");
row.append(el("span", "vk-media-status", label));
for (const extension of extensions) {
const chip = el(
"button",
`vk-chip ${draft.exts.includes(extension) ? "vk-chip-on" : "vk-chip-off"}`,
extension
);
chip.type = "button";
chip.addEventListener("click", () => {
draft.exts = draft.exts.includes(extension)
? draft.exts.filter(value => value !== extension)
: [...draft.exts, extension];
renderFilterDrawer();
});
row.append(chip);
}
return row;
};
filters.append(
el("div", "vk-media-status", "File types"),
typeRow("Video", ["mp4", "webm", "mov", "mkv"]),
typeRow("Image", ["png", "jpg", "gif", "webp", "avif"]),
typeRow("Audio", ["mp3", "ogg", "wav", "m4a", "flac"])
);
const cachedRow = el("label", "vk-media-check");
const cachedBox = document.createElement("input");
cachedBox.type = "checkbox";
@@ -1482,15 +1467,41 @@
deadRow.append(deadBox, document.createTextNode(" Hide 4chan links (usually dead)"));
filters.append(deadRow);
// "Do not include": nested collapsibles, so exclusions stay out of the way.
const exclude = document.createElement("details");
exclude.className = "vk-media-exclude";
exclude.append(el("summary", "vk-media-exclude-title", "Do not include"));
// "Do not include" and "Must include": mirrored collapsible groups.
const EXTS = [
"mp4",
"webm",
"mov",
"mkv",
"png",
"jpg",
"gif",
"webp",
"avif",
"mp3",
"ogg",
"wav",
"m4a",
"flac"
];
const SOURCE_KINDS = [
["embed", "Embeds"],
["attachment", "Attachments"],
["sticker", "Stickers"]
];
const group = title => {
const groupBlock = title => {
const detailsNode = document.createElement("details");
detailsNode.className = "vk-media-exclude";
detailsNode.append(el("summary", "vk-media-exclude-title", title));
return detailsNode;
};
const group = (parent, title) => {
const detailsNode = document.createElement("details");
detailsNode.className = "vk-media-exclude-group";
detailsNode.append(el("summary", "vk-media-status", title));
parent.append(detailsNode);
return detailsNode;
};
@@ -1510,63 +1521,65 @@
const toggle = (list, value, on) =>
on ? [...new Set([...list, value])] : list.filter(entry => entry !== value);
const sourceGroup = group("Sources");
for (const value of state.status?.providerFilters ?? []) {
const label = value.charAt(0).toUpperCase() + value.slice(1);
sourceGroup.append(
checkRow(label, draft.excludeProviders.includes(value), on => {
draft.excludeProviders = toggle(draft.excludeProviders, value, on);
})
);
}
for (const [value, label] of [
["embed", "Embeds"],
["attachment", "Attachments"],
["sticker", "Stickers"]
]) {
sourceGroup.append(
checkRow(label, draft.excludeSources.includes(value), on => {
draft.excludeSources = toggle(draft.excludeSources, value, on);
})
);
}
exclude.append(sourceGroup);
const toggleMany = (list, values, on) =>
on ? [...new Set([...list, ...values])] : list.filter(entry => !values.includes(entry));
const extGroup = group("File types");
const extRow = el("div", "vk-media-toolbar");
for (const extension of [
"mp4",
"webm",
"mov",
"mkv",
"png",
"jpg",
"gif",
"webp",
"avif",
"mp3",
"ogg",
"wav",
"m4a",
"flac"
]) {
/** Provider boxes; a list entry may carry several substrings ("giphy,tenor"). */
const providerChecks = (parent, key) => {
const node = group(parent, "Sources");
for (const entry of state.status?.providerFilters ?? []) {
const values = entry.split(",");
const label = values.map(value => value.charAt(0).toUpperCase() + value.slice(1)).join("/");
node.append(
checkRow(
label,
values.every(value => draft[key].includes(value)),
on => {
draft[key] = toggleMany(draft[key], values, on);
}
)
);
}
for (const [value, label] of SOURCE_KINDS) {
node.append(
checkRow(label, draft[key === "excludeProviders" ? "excludeSources" : "includeSources"].includes(value), on => {
const listKey = key === "excludeProviders" ? "excludeSources" : "includeSources";
draft[listKey] = toggle(draft[listKey], value, on);
})
);
}
return node;
};
const extChips = (parent, key, mode) => {
const node = group(parent, "File types");
const row = el("div", "vk-media-toolbar");
for (const extension of EXTS) {
const active = draft[key].includes(extension);
const chip = el(
"button",
`vk-chip ${draft.excludeExts.includes(extension) ? "vk-chip-off vk-chip-strike" : "vk-chip-on"}`,
mode === "exclude"
? `vk-chip ${active ? "vk-chip-off vk-chip-strike" : "vk-chip-on"}`
: `vk-chip ${active ? "vk-chip-on" : "vk-chip-off"}`,
extension
);
chip.type = "button";
chip.title = draft.excludeExts.includes(extension)
chip.title = active
? mode === "exclude"
? `Excluding ${extension}`
: `${extension} included`;
: `Only ${extension}`
: mode === "exclude"
? `${extension} included`
: `${extension} not required`;
chip.addEventListener("click", () => {
draft.excludeExts = toggle(draft.excludeExts, extension, !draft.excludeExts.includes(extension));
draft[key] = toggle(draft[key], extension, !active);
renderFilterDrawer();
});
extRow.append(chip);
row.append(chip);
}
extGroup.append(extRow);
exclude.append(extGroup);
node.append(row);
return node;
};
const amountRow = (label, key, unitKey) => {
const row = el("div", "vk-media-toolbar");
@@ -1604,18 +1617,25 @@
return row;
};
const sizeGroup = group("Size");
const exclude = groupBlock("Do not include");
providerChecks(exclude, "excludeProviders");
extChips(exclude, "excludeExts", "exclude");
filters.append(exclude);
const include = groupBlock("Must include");
providerChecks(include, "includeProviders");
extChips(include, "exts", "include");
const sizeGroup = group(include, "Size");
sizeGroup.append(
amountRow("Less than", "sizeLt", "sizeLtUnit"),
amountRow("Greater than", "sizeGt", "sizeGtUnit")
);
exclude.append(sizeGroup);
const lengthGroup = group("Length");
const lengthGroup = group(include, "Length");
lengthGroup.append(amountRow("Less than", "lenLt"), amountRow("Greater than", "lenGt"));
exclude.append(lengthGroup);
filters.append(exclude);
filters.append(include);
filterDrawer.append(filters);
// Zoom applies live; it is not part of the filter draft.
+12
View File
@@ -896,6 +896,14 @@ function buildServer(): McpServer {
.array(z.string())
.optional()
.describe("Hide these row sources (attachment, sticker, embed)"),
includeProviders: z
.array(z.string())
.optional()
.describe("Keep only providers matching these substrings"),
includeSources: z
.array(z.string())
.optional()
.describe("Keep only these row sources (attachment, sticker, embed)"),
sizeLessThan: z.number().optional().describe("Only items smaller than this many bytes"),
sizeGreaterThan: z.number().optional().describe("Only items larger than this many bytes"),
lengthLessThan: z.number().optional().describe("Only items shorter than this many seconds"),
@@ -931,6 +939,8 @@ function buildServer(): McpServer {
excludeExts,
excludeProviders,
excludeSources,
includeProviders,
includeSources,
sizeLessThan,
sizeGreaterThan,
lengthLessThan,
@@ -957,6 +967,8 @@ function buildServer(): McpServer {
excludeExts,
excludeProviders,
excludeSources,
includeProviders,
includeSources,
sizeLessThan,
sizeGreaterThan,
lengthLessThan,
+8
View File
@@ -717,6 +717,8 @@ export class MediaService {
excludeExts?: string[];
excludeProviders?: string[];
excludeSources?: string[];
includeProviders?: string[];
includeSources?: string[];
sizeLessThan?: number;
sizeGreaterThan?: number;
lengthLessThan?: number;
@@ -846,6 +848,8 @@ export class MediaService {
excludeExts?: string[];
excludeProviders?: string[];
excludeSources?: string[];
includeProviders?: string[];
includeSources?: string[];
sizeLessThan?: number;
sizeGreaterThan?: number;
lengthLessThan?: number;
@@ -881,6 +885,8 @@ export class MediaService {
excludeExts: options.excludeExts,
excludeProviders: options.excludeProviders,
excludeSources: options.excludeSources,
includeProviders: options.includeProviders,
includeSources: options.includeSources,
sizeLessThan: options.sizeLessThan,
sizeGreaterThan: options.sizeGreaterThan,
lengthLessThan: options.lengthLessThan,
@@ -941,6 +947,8 @@ export class MediaService {
excludeExts?: string[];
excludeProviders?: string[];
excludeSources?: string[];
includeProviders?: string[];
includeSources?: string[];
sizeLessThan?: number;
sizeGreaterThan?: number;
lengthLessThan?: number;
+2
View File
@@ -211,6 +211,8 @@ function parseMediaFilters(query: Record<string, unknown>) {
excludeExts: list(query.excludeExt),
excludeProviders: list(query.excludeProviders),
excludeSources: list(query.excludeSources),
includeProviders: list(query.includeProviders),
includeSources: list(query.includeSources),
sizeLessThan: amount(query.sizeLt),
sizeGreaterThan: amount(query.sizeGt),
lengthLessThan: amount(query.lenLt),
+11
View File
@@ -136,6 +136,17 @@ try {
sourceBoxes === providerFilters.length + 3,
{ sourceBoxes, providers: providerFilters.length }
);
const includeSources = await page
.locator(".vk-media-exclude-group")
.nth(2)
.locator(".vk-media-check")
.count();
step("must-include source checkboxes", includeSources);
check(
"must-include mirrors the provider list",
includeSources === providerFilters.length + 3,
{ includeSources, providers: providerFilters.length }
);
await page.locator(".vk-media-check", { hasText: "Show only cached previews" }).locator("input").check();
await page.getByRole("button", { name: "Apply filters" }).click();
await page.waitForTimeout(2500);