From a5613b36ae59942124333543f592fa324939cc5a Mon Sep 17 00:00:00 2001 From: Luna Date: Sun, 7 Jun 2026 07:05:20 -0700 Subject: [PATCH] CLAUDE.md: document search/dedup subsystems + FTS migration caveat Bring the guide up to date with this session's work: the hang watchdog in poll(); the three (path,mtime)-keyed caches (info_cache, search index, fingerprints); the FTS5 drop+recreate migration exception to the ALTER-only rule; a Search & perceptual-dedup subsystem section (fingerprint.rs, vtt.rs, the shared rebuild_and_group pipeline, the background-job pattern); tests/api.rs integration tests + the opt-in real-ffmpeg test; the node --check tip for the embedded SPA; and sponsorblock_mode / fingerprint / vtt in the worked-example + module lists. Co-Authored-By: Claude Opus 4.8 --- CLAUDE.md | 71 +++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 13 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6661d56..e315bf2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,8 +14,9 @@ Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in ```bash cargo build --release # the binary (release profile is opt-level=3 + thin LTO, ~1-2 min) -cargo test --release # all unit tests (fast; no network) +cargo test --release # unit tests + tests/api.rs integration tests (no network) cargo test --release # single test by substring, e.g. `cargo test --release subs_disabled` +cargo test --release real_ffmpeg -- --ignored # opt-in: real-ffmpeg fingerprint accuracy/speed check ./target/release/yt-offline # desktop GUI mode (default) ./target/release/yt-offline --web 8080 # headless web server on a port @@ -34,6 +35,11 @@ scratch dir with a minimal `config.toml` (`[backup]\ndirectory = "..."`) and run `--web ` from inside it. `/api/*` endpoints require auth when a password is set in the target library's DB. +`tests/api.rs` automates exactly this: it spawns the real `--web` binary against a +throwaway dir and drives the HTTP API with `curl` (skipping if curl is absent; the +dedup test also needs ffmpeg and skips without it). That's the place to add +end-to-end coverage of a new endpoint. + ## Architecture ### Two front-ends, one engine @@ -48,12 +54,15 @@ add download behavior, it goes in `Downloader` and is automatically available to both UIs. `poll()` also drives the cross-cutting job machinery: **auto-retry** of transient -failures (rate-limit/network) with cooldown + adaptive throttle, and the -**post-download ffmpeg transcode** pass. These work by capturing specs onto the -`Job` at `start()` time (`RetrySpec`, `ConvertSpec`) and acting on them when the -job transitions state — the `pending_*_spec` fields on `Downloader` are stashed -by `start()` and consumed by `enqueue()`/`spawn_job()` so the four non-download -enqueue paths (repair/music/yt-dlp-update/pot-update) stay untouched. +failures (rate-limit/network) with cooldown + adaptive throttle, the +**post-download ffmpeg transcode** pass, and the **hang watchdog** (a job whose +`last_activity` is silent past `HANG_TIMEOUT` is SIGKILLed and re-queued — +classified `NetworkError` so the existing retry path re-issues it). The first two +work by capturing specs onto the `Job` at `start()` time (`RetrySpec`, +`ConvertSpec`) and acting on them when the job transitions state — the +`pending_*_spec` fields on `Downloader` are stashed by `start()` and consumed by +`enqueue()`/`spawn_job()` so the four non-download enqueue paths +(repair/music/yt-dlp-update/pot-update) stay untouched. ### Settings flow (the easy thing to get wrong) @@ -67,7 +76,8 @@ silently half-works: 5. Seed the `Downloader` field at construction **and** on settings-save, in **both** `app.rs` and `web.rs`. Grep an existing setting end-to-end before adding one — `subtitle_defaults`, -`youtube_player_clients`, and `convert_defaults` are complete worked examples. +`youtube_player_clients`, `sponsorblock_mode`, and `convert_defaults` are +complete worked examples. ### Filesystem layout invariant @@ -85,16 +95,30 @@ re-parsing unchanged `info.json` sidecars. `Clone` — the pool is an `Arc`, so the parallel scanner takes its own handle). Schema lives in `init_schema()`; new columns are added via idempotent `ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no migration -framework). The web UI holds library/notes snapshots in memory and mutating -endpoints mirror DB writes onto those caches + `bump_library_version()` (the -ETag) so `/api/library` stays consistent without a rescan. +framework). **Exception:** FTS5 virtual tables have no `ADD COLUMN`, so the +`video_search` index is migrated by dropping + recreating it and clearing +`search_meta` to force a one-time reindex (it's a derived cache — safe to rebuild; +see the `transcript`-column migration). The web UI holds library/notes snapshots +in memory and mutating endpoints mirror DB writes onto those caches + +`bump_library_version()` (the ETag) so `/api/library` stays consistent without a +rescan. + +Three `(path|video_id, mtime)`-keyed caches let expensive work happen once and be +skipped on unchanged files: `info_cache` (parsed `info.json` fields), +`search_meta` + the `video_search` FTS5 index (full-text title/channel/ +description/**transcript**), and `video_fingerprint` (perceptual dHashes). Each +has a `sync_*`/rebuild helper that diffs the current library against stored mtimes, +processes only the new/changed rows, and prunes vanished ones. ### Web UI is one embedded file `web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked into the binary at compile time — editing it requires a rebuild to take effect. It's served with `Cache-Control: no-store` so binary upgrades don't strand stale tabs. -Progress streams over `/ws/progress` (WebSocket) with HTTP-poll fallback. +Progress streams over `/ws/progress` (WebSocket) with HTTP-poll fallback. The JS +isn't unit-testable, so syntax-check it after edits by extracting the inline +`