Add message links, suspect retries, and Discord preview recovery
This commit is contained in:
@@ -210,6 +210,25 @@ a cached still show a labeled placeholder, and playback streams only when the us
|
||||
GIF hover plays the animated GIF cached in `thumbs/` (the original is kept alongside the static
|
||||
frame and referenced by `anim_hash`), never the provider, so scrolling cannot spam anyone.
|
||||
|
||||
## Failure policy and recovery
|
||||
|
||||
The archive never treats a transient HTTP answer as a dead link on the first try.
|
||||
|
||||
- A preview or download that fails with an HTTP status on Discord's CDN is retried once with a
|
||||
**refreshed URL** from the source message (the signed link expires in about a day). Only if the
|
||||
refreshed fetch fails is the row parked.
|
||||
- Parked rows become **suspects**: they get one delayed retry (about 50 s). The retry runs after
|
||||
the batch, not immediately. If a batch is two-thirds suspect failures, the cooldown doubles
|
||||
(floor 60 s) and the retries run before the next batch.
|
||||
- A suspect whose delayed retry fails again is left **unverified**: it is skipped by later passes,
|
||||
so a finished backfill never churns over genuinely unavailable media. The Bot tab shows suspects
|
||||
and unverified counts per member, and the progress bar fills blue for cached, gray for pending,
|
||||
and red for failed rows.
|
||||
- **Recover Discord previews** is a one-shot task in the Bot tab: it walks every scrape's
|
||||
Discord-CDN rows closed with HTTP errors, refreshes each URL through the message (slopbot REST or
|
||||
the puppet, whichever the scrape used), and fetches the preview again. It starts paused; when it
|
||||
finishes it moves to the task history. The code stays for reuse.
|
||||
|
||||
## Bot tab settings
|
||||
|
||||
The Bot tab owns the tunables; they persist in `preferences.json` and apply to live jobs at once.
|
||||
|
||||
@@ -101,6 +101,9 @@ media-scrape/db/
|
||||
- The grid never renders an embed's page URL as media. A direct video file without a preview shows
|
||||
a first-frame `<video>` only for playable containers (mp4, webm, m4v); anything else shows a
|
||||
labeled placeholder, and a failed image falls back to that placeholder.
|
||||
- Every item carries the permanent Discord message link (`messageUrl`) derived from
|
||||
`media_sources`; the viewer shows it (with a copy button) instead of the expiring CDN URL, and
|
||||
"View full" streams through the daemon so the source is refreshed automatically.
|
||||
- Every item records who posted it, when, and its dimensions: `author_id`, `author_name`,
|
||||
`posted_at`, `width`, `height`, and `aspect_ratio`. In `download` mode a missing resolution is
|
||||
filled in with `ffprobe` after the file lands.
|
||||
|
||||
+7
-1
@@ -166,7 +166,10 @@ box live in Chat, not Call, and the connection stats are not chips yet.
|
||||
file, then the CDN URL; videos and audio are link cards with the poster and posted time in the
|
||||
tooltip.
|
||||
- Bot shows the slopbot connection state, the backfill queue (previews), the **Titles (YouTube)**
|
||||
catch-up lane with its own Start/Pause, and the media tunables. Backfill has
|
||||
catch-up lane with its own Start/Pause, the one-shot **Recover Discord previews** task, the task
|
||||
history, the failure-policy hint, and the media tunables. Media items carry their permanent
|
||||
`messageUrl`; the viewer shows it with a **Copy link** button in the link popup and offers
|
||||
**Open on YouTube** as a fallback for embeds that refuse to play. Backfill has
|
||||
totals and an ETA with a Bot/Puppet split, then one job card per chat with its member scrapes,
|
||||
per-kind pending, a progress bar, an ETA, and Start/Pause; the header has **Start backfill** and
|
||||
**Pause all**. A red banner reads `API timed out, increase timers` during a Discord cooldown. The
|
||||
@@ -306,6 +309,9 @@ replaceable.
|
||||
| GET | `/api/media/titles` | none | Title catch-up lane: state, pending/resolved/failed, throughput, ETA, current scrape |
|
||||
| POST | `/api/media/titles/start` | none | Start the title lane |
|
||||
| POST | `/api/media/titles/pause` | none | Pause the title lane after the current batch |
|
||||
| GET | `/api/media/recover` | none | Recovery task state, remaining Discord-CDN preview failures, counts, and the task history |
|
||||
| POST | `/api/media/recover/start` | none | Start the Discord preview recovery task |
|
||||
| POST | `/api/media/recover/pause` | none | Pause the recovery task after the current row |
|
||||
| GET | `/api/media/authors` | none | Distinct posters across every scrape, with counts |
|
||||
| GET | `/api/media/stats` | same filters as `/api/media` | Counts per kind (`image`, `gif`, `video`, `audio`), each with `total`, `downloaded`, `cached`, `notDownloaded` |
|
||||
| GET | `/api/media/storage` | none | Usage against the limit, per-kind sizes, `nearLimit`; marks old scrapes when over |
|
||||
|
||||
Reference in New Issue
Block a user