Executes docs/superpowers/specs/2026-07-07-runtime-performance-pass-design.md:
- database.rs: every pooled connection now runs WAL + synchronous=NORMAL
+ busy_timeout=5000 + foreign_keys=ON via with_init (in-memory pools
get the latter two). Fixes the SQLITE_BUSY failure mode under the
parallel scanner and makes FK cascades work on every connection, not
just whichever one last ran the old one-shot pragma in delete_folder.
- database.rs: prepare_cached for the per-video info_cache queries; new
info_cache_put_many bulk upsert (single transaction) replaces the
per-row autocommitted info_cache_put, with a round-trip unit test.
- library.rs: enrich_with_cache collects cache misses and flushes them
once per channel; title sort uses sort_by_cached_key.
- app.rs: Title/ChannelAsc sorts use sort_by_cached_key instead of
allocating two lowercased strings per comparison.
- Cargo.toml: panic=abort in release (crash-hook still fires; Cargo
forces unwind for test harnesses). PACKAGING.md documents the local
RUSTFLAGS target-cpu=native opt-in; repo stays portable.
- .gitignore: catacomb.db-wal / catacomb.db-shm sidecars.
Verified: 146 tests pass (integration suite exercises the pragmas
end-to-end), x86_64-pc-windows-gnu check green, live smoke shows
journal_mode=wal + batched cache writes landing correct rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The startup channel scan used available_parallelism() (every core), which
could briefly peg the whole box on a cold/large library. Cap it at cores-1
so the machine stays responsive; the scan is disk-I/O-bound and runs as a
background task, so one fewer thread costs little. Mirrors the headroom the
fingerprint pass and thumbnail pool already leave.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 🔍 library search now matches spoken words, not just titles /
channels / descriptions.
- database.rs: add a `transcript` column to the `video_search` FTS5 index.
Fresh DBs get it directly; existing ones are migrated (FTS5 has no ADD
COLUMN, so the table is recreated and search_meta cleared to force a
one-time reindex). `sync_search_index` reads each video's first subtitle
(mtime-gated like the description), flattens it via the shared `vtt`
parser (`transcript_text`, capped at 256 KB), and indexes it. Search
snippets now use column -1 so the excerpt comes from whichever column
matched (description or transcript).
- library.rs: `build_search_entries` passes the first subtitle path.
- Both UIs: search copy updated to mention transcripts.
Tests: a unit test (a word only in the .vtt is found), a migration test
(seed an old 5-column index + stale meta, open, confirm the recreated
index indexes transcript text), and the search integration test now seeds
a subtitle and asserts a spoken-only word hits. Also bumped the test
server's startup budget so the ffmpeg-heavy dedup test can't starve a
sibling server into a flaky timeout. 120 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A new FTS5 index (`video_search`) over every video's title, channel, and
description — searchable across the whole library, not just the loaded
grid. `search_meta` tracks each video's mtime so a routine rescan only
re-reads a description sidecar when the video actually changed; vanished
videos are evicted. The index is refreshed after every scan in both
front-ends via the shared `library::build_search_entries`.
- database.rs: schema + `sync_search_index` (mtime-gated, one txn) +
`search_videos` (ranked, with a highlighted snippet) + a safe prefix
MATCH builder. Unit-tested (index/search/prefix/AND/evict/garbage).
- web.rs: `GET /api/search?q=&limit=`; index synced after the initial
scan, rescan, and maintenance-remove.
- Web UI: a 🔍 header button + `f` shortcut open a debounced search modal
with ranked results and highlighted description snippets; clicking a
result jumps to it.
- Desktop (egui): a 🔎 Search button opens a floating results window
querying the same index; clicking a result plays the video. Closes the
desktop/web parity gap for search.
Integration test seeds a real video + description, rescans, and asserts
title / prefix / description-only matches hit and unrelated misses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Four changes that compound — most users will notice these on first
launch of a large library.
## Release profile
- opt-level: 2 → 3 (more inlining + vectorization, ~5-15% on hot paths)
- lto = "thin" (cross-crate inlining; thin avoids the rust-lld + bundled
sqlite link breakage that full LTO previously caused, which is what
the !lto note was about)
- codegen-units = 1 (whole-crate inlining, ~5% runtime, +30s build)
- strip = "debuginfo" (drops ~30 MB from the binary)
## Thumbnail decode pool
The single decoder thread was the bottleneck on grid fill for libraries
with hundreds of channels. Replace with a worker pool sized at
clamp(2, cores/2, 8). Workers share the request channel behind a Mutex;
lock contention is negligible because each worker spends ~all its time
in image::open + resize.
## info.json parse cache
Library scans re-parse every video's info.json sidecar on every rescan —
serde_json::from_str is the dominant cost on a warm-filesystem scan
(file reads themselves are OS-cached). Add a `info_cache` SQLite table
keyed by (path, mtime); enrich_with_cache hits it first and only parses
on miss. mtime-keyed invalidation means yt-dlp re-writing an info.json
auto-invalidates without explicit eviction.
Each scan worker checks out its own r2d2 connection so lookups don't
serialize. Database is now Clone (the inner Pool is Arc, so cloning is
cheap) so we can hand handles to parallel workers.
## /api/library body cache
The ETag short-circuit catches clients with a matching cached version;
this catches clients without one (curl, freshly-opened tabs, mobile
WebViews that don't store ETags reliably). When the version matches
the cached entry's, we hand back the previously-serialized bytes
without re-walking the channel tree or re-running serde_json::to_string.
A post-build version check prevents poisoning the cache with stale
bytes when the library changes during serialization.
74 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Users can now organize channels into named folders independent of the
underlying platform groupings. v1 ships single-level (no nesting) since
that's where ~90% of the organizational value lives — nested folders +
drag-to-reparent + per-folder cascading options are deferred to v2 once
we have real-world feedback on what the workflow needs.
Data layer
- New `folders` SQLite table: id, name (UNIQUE), position, created_at.
- New `channel_assignments` table: (platform, handle) → folder_id with
ON DELETE CASCADE so deleting a folder cleanly unfiles its channels.
- Database methods: create_folder, rename_folder, delete_folder (with
PRAGMA foreign_keys ON for the cascade), list_folders,
set_channel_folder, get_all_channel_assignments.
- New `FolderRecord` struct (id/name/position) is `Serialize` so it
rides on the /api/library response.
Library
- `Channel` gains `folder_id: Option<i64>` (None = Unfiled).
- New `library::apply_channel_folders` mirror of `apply_channel_options`
populates it from the bulk DB map after every scan / rescan.
Web API
- `POST /api/folders` create with `{name}` → `{id}`.
- `POST /api/folders/:id/rename` rename with `{name}`.
- `DELETE /api/folders/:id` cascade-deletes assignments.
- `POST /api/channels/:platform/:handle/folder` upsert/clear via
`{folder_id: <i64|null>}`. All endpoints bump the library ETag.
- `/api/library` response now carries a `folders: [...]` array so the
client can render the sidebar grouping without a second round trip.
Web UI
- Sidebar gains a "📁 Folders" section above the platform groupings.
Each folder lists its member channels (with platform icon prefix);
channels with `folder_id` are pulled out of their platform section
to avoid duplication. Unfiled channels keep platform grouping.
- "manage" / "new" button next to the Folders heading opens a modal
with the full folder list (member counts inline) plus rename /
delete buttons and a Create input.
- New per-channel sub-action "📁 Move to folder…" opens a small
picker modal with one row per folder + "— Unfiled —". Current
assignment highlighted.
Desktop UI
- Sidebar refactored to build a `Vec<SidebarItem>` ordered list
(FolderHeader / FolderManageBtn / PlatformHeader / Channel) and
iterate it once, instead of the previous flat
`for i in 0..library.len()` loop. Folder section renders above
the platform sections; channels with `folder_id` are skipped from
the platform-grouped loop so they appear in exactly one place.
- New `folder_manager_window` + `move_to_folder_window` egui windows
cover create / rename (via the create-buffer field) / delete and
per-channel folder picking. Right-click context menu on a channel
gains "📁 Move to folder…".
- App struct gains: `folders`, `show_folder_manager`,
`folder_create_buffer`, `show_move_to_folder`, `move_to_folder_target`.
Tartube spec checklist
- Folder hierarchy box ticked (with the v2 deferrals noted in the
inline annotation).
55 unit tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the single largest gap with Tartube documented in docs/tartube-spec.md.
Users can now attach a small set of yt-dlp overrides to any individual
channel; the overrides apply automatically during scheduled re-checks
and the right-click "Check for new videos" action.
Data layer
- New `channel_options` SQLite table keyed on (platform, handle), with
`options_json TEXT` blob + updated_at. PK ensures one row per channel.
- Database gains get_channel_options / set_channel_options /
delete_channel_options / get_all_channel_options. The bulk getter is
used by the library scanner so each rescan hydrates options without
per-channel SQL round-trips.
- `DownloadQuality` derives Serialize/Deserialize so it can roundtrip
through the options blob.
New module: src/download_options.rs
- `DownloadOptions` struct with 9 fields:
quality (Option<DownloadQuality>) — per-channel quality cap
audio_only (bool) — force --extract-audio chain
limit_rate_kb (Option<u32>) — --limit-rate
min_filesize_mb / max_filesize_mb — yt-dlp filesize filters
date_after (Option<String>) — --dateafter YYYYMMDD
match_filter (Option<String>) — yt-dlp --match-filter passthrough
subtitle_langs (Vec<String>) — --sub-langs CSV
extra_args (Vec<String>) — raw yt-dlp passthrough (Tartube's
`extra_cmd_string`)
- `apply(&mut Command)` appends the right flags conditionally.
- `is_empty()` lets the API layer collapse no-op writes into DELETEs so
we don't keep useless rows around.
- `from_json()` falls back to defaults on malformed rows so a schema
drift doesn't take a channel offline.
- 8 new unit tests cover apply-emits-correct-flags, JSON roundtrip,
corrupt-fallback, and is_empty.
Library
- `Channel` gains `download_options: DownloadOptions` (default empty).
- New `library::apply_channel_options(channels, map)` helper that the
caller invokes after `scan_channels` to hydrate from the bulk DB load.
- All five `scan_channels` call sites updated.
Downloader
- `Downloader::start` gains a new `channel_options: Option<&DownloadOptions>`
parameter. Applied last so per-channel overrides win over global
defaults. The explicit "submit from URL bar" flow passes None
(channel unknown until yt-dlp resolves it). Scheduled re-checks +
right-click checks pass the channel's stored options. Channel-options'
`quality` field overrides the hard-coded DownloadQuality::Best for
re-checks.
Web API
- New routes on /api/channels/:platform/:handle/options:
GET — fetch (returns defaults when no row exists)
POST — upsert (empty body collapses to DELETE)
DELETE — clear overrides
- All three update the in-memory library snapshot immediately so the
next re-check sees the change without waiting for a rescan, and bump
the library_version ETag so polling clients pick up the new state.
Web UI
- Sidebar each channel's expanded section gains a "⚙ Channel options…"
entry next to "Check for new videos".
- openChannelOptions() fetches the current options + renders a modal
with one input per field. Save / Clear / Cancel buttons. The Clear
button confirms via window.confirm() because it's destructive.
- All form fields validate locally before POSTing; empty numeric
fields map to null.
Desktop UI
- New `channel_options_window` egui::Window opened from the existing
channel right-click context menu (new "⚙ Channel options…" item).
- Form mirrors the web side: ComboBox for quality, checkbox for
audio-only, TextEdit (singleline + multiline) for the other fields.
- Save / Clear / Cancel buttons; saves go straight to SQLite and the
live `App::library` so the next scheduled-check or right-click runs
with the new overrides.
Documentation
- `docs/tartube-spec.md` checklist: per-target download options ticked
with a note describing the v1 scope.
55 unit tests pass (47 + 8 new).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three roadmap items together since they touch overlapping code:
Live-stream recording
- New `live: bool` argument on Downloader::start. When true, attaches
`--live-from-start --wait-for-video 30`, suppresses
`--break-on-existing` (every recording is a unique file), and
prepends a UTC timestamp suffix to the output filename so
re-recordings of the same broadcaster don't collide.
- The job label gains a "🔴 LIVE" marker so a long-running record is
obviously different from a VOD pull.
- Both UIs gain a "🔴 Live stream" checkbox in the download panel,
hidden when Music mode is selected. Works for Twitch *and* YouTube
Live since yt-dlp recognizes both from the URL.
- Web `POST /api/download` accepts a `live: bool` body field; all
other callers (scheduled re-check, right-click "Check for new
videos") pass false.
Recent-additions feed
- `library::Video` gains `mtime_unix: Option<u64>` — populated from
the metadata we already read for `file_size`, so no extra fs hit.
- New `SidebarView::Recent` (desktop) + `showRecent` (web) sort the
whole library by mtime descending and cap at 100 entries.
- Sidebar entry "🕒 Recent additions" appears only when the library
has any dated content, so a fresh install doesn't show an empty view.
Per-platform impersonation
- New `Platform::impersonate_target()`:
- Twitch → None (skip impersonation; their OAuth/Helix dislikes
mismatched TLS fingerprints).
- TikTok → "Chrome-Android-131" (mobile profile matches their
first-party app surface).
- Everything else → "Chrome-146:Macos-26".
- `Downloader::apply_impersonation` now takes the platform and routes
through the override; `repair()` (always YouTube-implicit) and
`start_music()` (uses classify_url on the URL) wire through too.
- `/api/preview` does the same classification before dispatching to
yt-dlp.
- 3 new tests confirm Twitch → None, TikTok → mobile, YouTube → desktop.
44 unit tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yt-dlp already speaks ~1,800 sites; the changes here teach the rest of
the app to route them through platform-specific output folders and
remember where each creator was downloaded from.
New `platform` module
- `Platform` enum: YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud,
Odysee, Other.
- `classify_url` returns both the source platform and a `UrlKind`
(Channel / Playlist / Video / Unknown) — per-platform path parsing
for each supported host.
- `platform_root(channels_root, platform)` returns the on-disk folder
per platform. YouTube keeps the legacy `channels/` for backward
compat; the others live as siblings (`tiktok/`, `twitch/`, …).
- `write_source_url` / `read_source_url` persist the originating URL
in a `.source-url` sidecar at each creator folder so re-checks no
longer rely on the YouTube-only "UC + 22 chars" heuristic.
Downloader
- `Downloader::start` now takes `&UrlInfo` instead of `&UrlKind` and
routes the yt-dlp `-o` template into the right platform folder.
- For channel downloads it also writes `.source-url` so future
re-checks recover the exact URL.
- Output template uses `%(uploader,channel,creator|Unknown)s` so
non-YouTube sites that don't expose `channel` still get a sensible
creator folder.
- Per-platform `archive.txt` keeps cross-platform IDs (TikTok numeric
vs YouTube base64) from colliding.
- New `recheck_url(&Channel)` prefers `source_url` and falls back to
the legacy YouTube heuristic for libraries created before this
change.
Library scanner
- `scan_channels` now walks every platform's folder and tags each
`Channel` with its `platform` + `source_url`. Sort key puts the
platform first, then name, so the sidebar groups cleanly.
- `Channel` gains `platform: Platform` and `source_url: Option<String>`
fields, populated from the on-disk sidecar.
Web UI
- `WebState` gains `library_root` (= parent of channels_root). The
`/files/` ServeDir is mounted at library_root so non-YouTube videos
resolve at `/files/<platform>/<creator>/<file>`. Existing YouTube
URLs still work — they're now `/files/channels/...` instead of
`/files/...` and the server-rendered JSON updates accordingly.
- Maintenance scan + remove use `library_root` so non-YouTube content
is included.
- `ChannelInfo` JSON exposes `platform`, `platform_label`,
`platform_icon`, and `source_url`.
- Sidebar groups channels by platform with each platform's icon.
- "Check for new videos" now uses the stored `source_url` (with the
YouTube heuristic as a last-resort fallback).
- Download dialog preview shows the detected source + destination
folder.
Desktop UI
- App gains a `library_root` field mirroring the web side; maintenance
ops scan/remove against it.
- Sidebar channel labels are prefixed with the platform icon for
non-YouTube channels; tooltip carries the platform name.
- Download URL field surfaces both source platform and folder path
preview as you type.
Plex
- Non-YouTube creators get their show folder prefixed with the
platform name (e.g., `TikTok - cooluser`) so a YouTube channel and
a TikTok account with the same name don't collide.
Backward compatibility
- Existing YouTube libraries (`channels/<handle>/...`) are untouched.
New platform folders are created on first run as siblings.
- Channels without a `.source-url` (i.e., everything that pre-dates
this commit) fall through to the YouTube heuristic, preserving the
existing re-check behavior.
Tests
- 14 new `platform` tests cover URL classification per platform,
channel handle extraction, and `platform_root` layout. The old
`detect_url_kind` / `extract_after` tests move into platform.rs
since the canonical implementation lives there now.
- 41 unit tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bug fixes
- Desktop auto-rescan: snapshot was_running before check_notifications
updates prev_job_states, so the post-download rescan actually fires.
- web::run shutdown: replace `let _ = tx;` (which dropped the sender
immediately and short-circuited the shutdown select!) with
std::mem::forget(tx); add an idle fallback so the `-> !` is honored.
- /api/preview now resolves the bundled yt-dlp path and extends PATH
for deno discovery, matching the rest of the downloader.
- Scheduler interval clamped to >=1h on read so a manually-edited
config.toml with 0 hours can't trigger every tick.
- maintenance::is_within canonicalizes the parent + filename when the
target is missing; remove_files treats NotFound as success.
Security
- Session tokens stored with issued-at Instant and pruned past 30 day
TTL on every touch (was: unbounded HashSet).
- Login rate-limited per source IP: 5 failures → 60s lockout (429).
- Secure cookie flag added when X-Forwarded-Proto: https is present.
- 4 MiB body size cap via DefaultBodyLimit.
- Security headers middleware: CSP, X-Content-Type-Options,
X-Frame-Options, Referrer-Policy.
- JS safeUrl() defangs javascript:/vbscript:/data:text URLs before
interpolation into src= attributes.
- Bundled yt-dlp install verifies SHA-256 against the release's
SHA2-256SUMS; deno hash printed for visual inspection. Install also
reports byte counts every 3s so users see live progress.
- yt-offline.db chmod 0600 at open time on Unix.
New features
- Bundled yt-dlp + deno mode: settings toggle (System/Bundled) plus
one-click Install/Update from settings. Ensures executable bit on
every launch in case the install chmod was missed.
- Download queue with configurable max_concurrent (1–10); pending
jobs surfaced in both UIs.
- Music download mode: --extract-audio into music/<artist>/, separate
sidebar entry, /api/music + /music-files/ endpoints, inline <audio>.
- Download quality selector: Best / 1080p / 720p / 480p / 360p.
- Sort videos by Newest / Oldest (upload_date from info.json with
release_date fallback).
- Plex metadata: per-episode .nfo (title, season/episode, aired,
runtime, plot from .description), <stem>-thumb.jpg symlink, and
show-level tvshow.nfo.
- yt-dlp retry/throttle defaults: --retries 30 --fragment-retries 30
--retry-sleep linear=1:30:2 --sleep-requests 1 to ride out the
"Connection reset by peer" failures.
Performance
- password_required cached in AtomicBool; refreshed only on password
change. Eliminates a SQLite query per static-file fetch.
- Job::log → VecDeque for O(1) front-pop instead of O(n) drain.
- Library scan parallelized across cores via stdlib parallel_map.
- Web UI poll: 600ms while active, 5s when idle, 2s after errors.
- Music scan now recursive (Artist/Album/Track.opus).
- percent_encode_segment uses write! to avoid per-byte String allocs.
Code quality
- 27 unit tests covering parse_progress, parse_stem, extract_after,
detect_url_kind, srt_to_vtt, count_cookies, percent_encode_segment,
bind_mode_of, hash_password, lang_label, is_within, sanitize,
year_of, aired_date, xml_escape.
- Unified library::find_video + Channel::all_videos() iterator,
replacing three duplicated linear searches.
- Downloader::browser wired to --cookies-from-browser as a fallback
when no cookies.txt is present.
- mpv detection now matches the binary basename, not substring.
- Settings modal scroll fix; web settings expose all new fields.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Full-scan download toggle: omits --break-on-existing so all unarchived
videos are fetched regardless of order, filling gaps in partial archives
- Filter "Aborting remaining downloads" from job logs (break-on-existing noise)
- Password hash moved from config.toml to SQLite settings table; survives
git checkouts and rebuilds. config.toml and yt-offline.db added to .gitignore;
config auto-generated on first run if absent
- Channel re-check URLs derived from folder name (/@handle or /channel/UCxxx)
instead of info.json's canonical channel_url, preventing duplicate UCxxx folders
- SRT subtitles detected by library scanner and served via /api/sub-vtt/* endpoint
that converts to WebVTT on the fly; browser <track> element can now play them
- Cookie clear button in both desktop and web UI (DELETE /api/cookies)
- Per-job X close button on finished download jobs in desktop GUI
- Fix widget ID clash when multiple download jobs are shown (push_id per job)
- Watched videos excluded from Continue Watching list and badge count
- settings table added to SQLite schema (key/value store)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Features:
- Port all 7 desktop themes (Dracula, Trans, Emo variants, etc.) to the web UI
- Configurable bind interface (localhost/Tailscale/LAN/all) with detected IPs,
settable from both the web UI and desktop GUI
- Optional password that gates the whole UI and all API access via session
cookies (Argon2), with a login page, logout, and middleware; settable from
both UIs
- Library maintenance: scan for duplicate video IDs and missing assets
(thumbnail/info.json/description), remove duplicates (scan-and-confirm, with
out-of-library path guard), and re-fetch missing sidecars via yt-dlp.
Surfaced in both the web UI and a desktop GUI window.
Optimizations / cleanup:
- Cache has_chapters at scan time (library.rs) instead of re-reading and
parsing every info.json on each /api/library request
- Use the cached channel size in get_library
- Pool a single SQLite connection in WebState instead of opening one per request
- Remove a 1.2 GB stray nested src/yt-offline duplicate tree
Fixes:
- Rescan no longer clears all thumbnail textures (caused thumbnails to unload);
prune only removed entries and bump library_generation so the card grid
refreshes correctly
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- SQLite DB for watched/position persistence (database.rs)
- Mark watched toggle on cards and detail panel
- mpv IPC socket integration for resume position tracking (unix)
- Sort by title, duration, file size
- Thumbnail density slider in top bar
- Channel metadata banner (subscriber count, uploader, channel URL)
- Cookie browser selector dropdown in settings
- Auto-rescan library when a download job finishes
- Fix yt-dlp --no-progress-bar flag (does not exist; removed)
- Browser field wired through config → downloader
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- library.rs: scans channels/<name>/ into channels + videos by filename stem
- downloader.rs: runs yt-dlp in a background thread, streams progress to the UI
- app.rs / main.rs: channel sidebar, searchable thumbnail list, detail/description
panel, downloads panel; plays videos via mpv (falls back to xdg-open)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
