Add message links, suspect retries, and Discord preview recovery

This commit is contained in:
Fisto
2026-09-18 11:47:08 +02:00
parent 9481286169
commit 8752b36209
16 changed files with 831 additions and 35 deletions
+19
View File
@@ -210,6 +210,25 @@ a cached still show a labeled placeholder, and playback streams only when the us
GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.
## Failure policy and recovery
The archive never treats a transient HTTP answer as a dead link on the first try.
- A preview or download that fails with an HTTP status on Discord's CDN is retried once with a
**refreshed URL** from the source message (the signed link expires in about a day). Only if the
refreshed fetch fails is the row parked.
- Parked rows become **suspects**: they get one delayed retry (about 50 s). The retry runs after
the batch, not immediately. If a batch is two-thirds suspect failures, the cooldown doubles
(floor 60 s) and the retries run before the next batch.
- A suspect whose delayed retry fails again is left **unverified**: it is skipped by later passes,
so a finished backfill never churns over genuinely unavailable media. The Bot tab shows suspects
and unverified counts per member, and the progress bar fills blue for cached, gray for pending,
and red for failed rows.
- **Recover Discord previews** is a one-shot task in the Bot tab: it walks every scrape's
Discord-CDN rows closed with HTTP errors, refreshes each URL through the message (slopbot REST or
the puppet, whichever the scrape used), and fetches the preview again. It starts paused; when it
finishes it moves to the task history. The code stays for reuse.
## Bot tab settings
The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.
+3
View File
@@ -101,6 +101,9 @@ media-scrape/db/
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
labeled placeholder, and a failed image falls back to that placeholder.
- Every item carries the permanent Discord message link (`messageUrl`) derived from
`media_sources`; the viewer shows it (with a copy button) instead of the expiring CDN URL, and
"View full" streams through the daemon so the source is refreshed automatically.
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
filled in with `ffprobe` after the file lands.
+7 -1
View File
@@ -166,7 +166,10 @@ box live in Chat, not Call, and the connection stats are not chips yet.
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
tooltip.
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
catch-up lane with its own Start/Pause, the one-shot **Recover Discord previews** task, the task
history, the failure-policy hint, and the media tunables. Media items carry their permanent
`messageUrl`; the viewer shows it with a **Copy link** button in the link popup and offers
**Open on YouTube** as a fallback for embeds that refuse to play. Backfill has
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
@@ -306,6 +309,9 @@ replaceable.
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
| POST | `/api/media/titles/start` | none | Start the title lane |
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
| GET | `/api/media/recover` | none | Recovery task state, remaining Discord-CDN preview failures, counts, and the task history |
| POST | `/api/media/recover/start` | none | Start the Discord preview recovery task |
| POST | `/api/media/recover/pause` | none | Pause the recovery task after the current row |
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |