diff --git a/.forgejo/workflows/release.yml b/.forgejo/workflows/release.yml index 460d5a7..3277a5c 100644 --- a/.forgejo/workflows/release.yml +++ b/.forgejo/workflows/release.yml @@ -42,7 +42,7 @@ jobs: - name: Add Windows cross-compile target # package.sh's `all` only builds the Windows zip when this target + # mingw + zip are all present (win_available), so this step is what - # opts the release build into producing yt-offline-*-x86_64-windows.zip. + # opts the release build into producing catacomb-*-x86_64-windows.zip. run: rustup target add x86_64-pc-windows-gnu - name: Cache cargo registry + build @@ -87,7 +87,7 @@ jobs: - `.deb` — Debian / Ubuntu / Mint - `.rpm` — Fedora / RHEL / openSUSE - `.AppImage` — any Linux (chmod +x and run) - - `-x86_64-windows.zip` — Windows 10/11 (unzip, run yt-offline.exe) + - `-x86_64-windows.zip` — Windows 10/11 (unzip, run catacomb.exe) Runtime requirements: yt-dlp, ffmpeg, mpv on PATH (plus xdg-utils on Linux). The Windows zip's README lists the same. diff --git a/.gitignore b/.gitignore index 3f4c97a..ea76bf4 100644 --- a/.gitignore +++ b/.gitignore @@ -9,8 +9,8 @@ # makepkg build artifacts (finished .pkg.tar.zst is NOT ignored) /pkg/ -/src/yt-offline/ -/yt-offline/ +/src/catacomb/ +/catacomb/ # Rendered mdBook output (source lives in docs/src/) /docs/book/ @@ -27,4 +27,4 @@ cookies.txt config.toml # Runtime database (watched status, positions, password hash) — never commit -yt-offline.db +catacomb.db diff --git a/.vscode/launch.json b/.vscode/launch.json index dd8fe3f..8a86e68 100644 --- a/.vscode/launch.json +++ b/.vscode/launch.json @@ -5,25 +5,25 @@ "version": "0.2.0", "configurations": [ { - "name": "Debug executable 'yt-offline'", + "name": "Debug executable 'catacomb'", "type": "lldb", "request": "launch", "cargo": { "args": [ "run", - "--bin=yt-offline" + "--bin=catacomb" ] }, "args": [] }, { - "name": "Debug unit tests in executable 'yt-offline'", + "name": "Debug unit tests in executable 'catacomb'", "type": "lldb", "request": "launch", "cargo": { "args": [ "test", - "--bin=yt-offline" + "--bin=catacomb" ] } } diff --git a/AGENTS.md b/AGENTS.md index 6e5579f..d6ee9d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ This file provides guidance to Codex (Codex.ai/code) when working with code in t ## What this is -`yt-offline` — a single Rust binary that is **both** a desktop GUI (eframe/egui) +`catacomb` — a single Rust binary that is **both** a desktop GUI (eframe/egui) and a headless web server (axum), wrapping `yt-dlp` to archive YouTube/TikTok/ Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in [ROADMAP.md](ROADMAP.md); a structured analysis of the Tartube benchmark is in @@ -17,8 +17,8 @@ cargo build --release # the binary (release profile is opt-level=3 + t cargo test --release # unit tests + tests/api.rs integration tests (no network) cargo test --release # single test by substring, e.g. `cargo test --release subs_disabled` cargo test --release real_ffmpeg -- --ignored # opt-in: real-ffmpeg fingerprint accuracy/speed check -./target/release/yt-offline # desktop GUI mode (default) -./target/release/yt-offline --web 8080 # headless web server on a port +./target/release/catacomb # desktop GUI mode (default) +./target/release/catacomb --web 8080 # headless web server on a port scripts/package.sh [deb|rpm|appimage|all] # build distro packages → dist/ (see docs/PACKAGING.md) ``` @@ -103,6 +103,12 @@ in memory and mutating endpoints mirror DB writes onto those caches + `bump_library_version()` (the ETag) so `/api/library` stays consistent without a rescan. +Web login sessions are runtime-owned but restart-persistent: `web.rs` keeps the +live `HashMap` and mirrors tokens into the `sessions` table, then +rehydrates non-expired rows at startup. Logout and password changes clear those +rows. The DB is plain SQLite, not encrypted, so `/api/backup/db` exports a +sensitive file; at-rest DB encryption is deliberately deferred to Phase 4. + Three `(path|video_id, mtime)`-keyed caches let expensive work happen once and be skipped on unchanged files: `info_cache` (parsed `info.json` fields), `search_meta` + the `video_search` FTS5 index (full-text title/channel/ @@ -123,7 +129,7 @@ whole SPA — a `cargo build` won't catch it since the HTML is just a string). ### Bundled toolchain & anti-bot `ytdlp_bin.rs` manages an optional self-contained venv at -`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi` +`~/.local/share/catacomb/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi` for TLS impersonation + bundled `deno`). `pot_provider.rs` runs `bgutil-pot` (a loopback HTTP server) for YouTube Proof-of-Origin tokens; **its yt-dlp plugin must come from the same release as the server binary, not PyPI** (version skew @@ -154,14 +160,17 @@ both the transcript indexer and the transcript viewers. ## Conventions - **Never commit** `cookies.txt` (live session creds), `config.toml` (user- - specific), or `yt-offline.db` (contains the Argon2 password hash). All - gitignored. + specific), or `catacomb.db` (plain SQLite app state, including the Argon2 + password hash and restart-persistent session tokens). All gitignored. - Redact the absolute cookies path out of any log line surfaced to the UI/API (`redact_sensitive` in `downloader.rs`) — it leaks `$HOME`. - `app.rs` and `web.rs` are large (~3–4k lines) because each owns a full UI; new desktop code goes in `app.rs`, web handlers in `web.rs`, shared logic in the focused modules (`downloader`, `database`, `library`, `platform`, `fingerprint`, `vtt`, …). -- Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only/no-GTK by - design; keep that posture (it's why packaging avoids a GTK dep). Windows/macOS - are not yet first-class — the tray would need a per-OS backend. +- Tray (`ksni`) and the `rfd` xdg-portal dialog backend are Linux-only/no-GTK + by design (it's why packaging avoids a GTK dep). They're target-gated in + `Cargo.toml`; off Linux, `rfd` uses its native backend and `tray::start` is a + no-op. Windows now ships as a cross-compiled zip; macOS has a local osxcross + packaging path but is not in CI and still needs signing/notarization + a real + tray backend. diff --git a/CLAUDE.md b/CLAUDE.md index 40803ac..ecb299b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this is -`yt-offline` — a single Rust binary that is **both** a desktop GUI (eframe/egui) +`catacomb` — a single Rust binary that is **both** a desktop GUI (eframe/egui) and a headless web server (axum), wrapping `yt-dlp` to archive YouTube/TikTok/ Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in [ROADMAP.md](ROADMAP.md); a structured analysis of the Tartube benchmark is in @@ -17,8 +17,8 @@ cargo build --release # the binary (release profile is opt-level=3 + t cargo test --release # unit tests + tests/api.rs integration tests (no network) cargo test --release # single test by substring, e.g. `cargo test --release subs_disabled` cargo test --release real_ffmpeg -- --ignored # opt-in: real-ffmpeg fingerprint accuracy/speed check -./target/release/yt-offline # desktop GUI mode (default) -./target/release/yt-offline --web 8080 # headless web server on a port +./target/release/catacomb # desktop GUI mode (default) +./target/release/catacomb --web 8080 # headless web server on a port scripts/package.sh [deb|rpm|appimage|all] # build distro packages → dist/ (see docs/PACKAGING.md) ``` @@ -103,6 +103,12 @@ in memory and mutating endpoints mirror DB writes onto those caches + `bump_library_version()` (the ETag) so `/api/library` stays consistent without a rescan. +Web login sessions are runtime-owned but restart-persistent: `web.rs` keeps the +live `HashMap` and mirrors tokens into the `sessions` table, then +rehydrates non-expired rows at startup. Logout and password changes clear those +rows. The DB is plain SQLite, not encrypted, so `/api/backup/db` exports a +sensitive file; at-rest DB encryption is deliberately deferred to Phase 4. + Three `(path|video_id, mtime)`-keyed caches let expensive work happen once and be skipped on unchanged files: `info_cache` (parsed `info.json` fields), `search_meta` + the `video_search` FTS5 index (full-text title/channel/ @@ -123,7 +129,7 @@ whole SPA — a `cargo build` won't catch it since the HTML is just a string). ### Bundled toolchain & anti-bot `ytdlp_bin.rs` manages an optional self-contained venv at -`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi` +`~/.local/share/catacomb/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi` for TLS impersonation + bundled `deno`). `pot_provider.rs` runs `bgutil-pot` (a loopback HTTP server) for YouTube Proof-of-Origin tokens; **its yt-dlp plugin must come from the same release as the server binary, not PyPI** (version skew @@ -154,8 +160,8 @@ both the transcript indexer and the transcript viewers. ## Conventions - **Never commit** `cookies.txt` (live session creds), `config.toml` (user- - specific), or `yt-offline.db` (contains the Argon2 password hash). All - gitignored. + specific), or `catacomb.db` (plain SQLite app state, including the Argon2 + password hash and restart-persistent session tokens). All gitignored. - Redact the absolute cookies path out of any log line surfaced to the UI/API (`redact_sensitive` in `downloader.rs`) — it leaks `$HOME`. - `app.rs` and `web.rs` are large (~3–4k lines) because each owns a full UI; new @@ -172,6 +178,7 @@ both the transcript indexer and the transcript viewers. cross-compiles — `cargo check --target x86_64-pc-windows-gnu` is green — so keep new platform-specific code behind `cfg(unix)`/`cfg(windows)`/ `cfg(target_os = …)` with a fallback, the way `disk_space`, `plex`, the mpv - IPC, and the `chmod` guards already do. Windows/macOS still aren't *shipped* - (no tray backend, no link/CI yet), but don't reintroduce an unconditional - Linux-only dep. + IPC, and the `chmod` guards already do. Windows now ships as a cross-compiled + zip from CI; macOS has a local osxcross packaging path but is not in CI and + still needs signing/notarization + a real tray backend. Don't reintroduce an + unconditional Linux-only dep. diff --git a/Cargo.lock b/Cargo.lock index 7241da1..ec3af20 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -727,6 +727,33 @@ dependencies = [ "wayland-client", ] +[[package]] +name = "catacomb" +version = "0.1.0" +dependencies = [ + "argon2", + "axum", + "eframe", + "image", + "ksni", + "libc", + "notify-rust", + "r2d2", + "r2d2_sqlite", + "rand 0.8.6", + "reqwest", + "rfd", + "rusqlite", + "serde", + "serde_json", + "tokio", + "tokio-stream", + "tokio-util", + "toml", + "tower-http 0.5.2", + "windows-sys 0.59.0", +] + [[package]] name = "cc" version = "1.2.62" @@ -5408,33 +5435,6 @@ dependencies = [ "synstructure", ] -[[package]] -name = "yt-offline" -version = "0.1.0" -dependencies = [ - "argon2", - "axum", - "eframe", - "image", - "ksni", - "libc", - "notify-rust", - "r2d2", - "r2d2_sqlite", - "rand 0.8.6", - "reqwest", - "rfd", - "rusqlite", - "serde", - "serde_json", - "tokio", - "tokio-stream", - "tokio-util", - "toml", - "tower-http 0.5.2", - "windows-sys 0.59.0", -] - [[package]] name = "zbus" version = "4.4.0" diff --git a/Cargo.toml b/Cargo.toml index 3a13ca0..170cc75 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,5 @@ [package] -name = "yt-offline" +name = "catacomb" version = "0.1.0" edition = "2021" description = "Self-hosted archive for YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee and more. Desktop GUI + web UI, bundled yt-dlp with curl_cffi impersonation, Plex export, SQLite-backed resume tracking. AGPL-3.0." @@ -97,10 +97,10 @@ priority = "optional" depends = "libc6, libxcb1, yt-dlp, ffmpeg, mpv, xdg-utils" recommends = "libnotify4" assets = [ - ["target/release/yt-offline", "usr/bin/", "755"], - ["youtube-backup.desktop", "usr/share/applications/yt-offline.desktop", "644"], - ["icon.png", "usr/share/pixmaps/yt-offline.png", "644"], - ["README.md", "usr/share/doc/yt-offline/README.md", "644"], + ["target/release/catacomb", "usr/bin/", "755"], + ["catacomb.desktop", "usr/share/applications/catacomb.desktop", "644"], + ["icon.png", "usr/share/pixmaps/catacomb.png", "644"], + ["README.md", "usr/share/doc/catacomb/README.md", "644"], ] # ── Fedora / RHEL / openSUSE package (cargo-generate-rpm) ───────────────────── @@ -113,9 +113,9 @@ license = "AGPLv3" # Fedora; ffmpeg lives in RPM Fusion but we still declare the dep. requires = { yt-dlp = "*", ffmpeg = "*", mpv = "*", "xdg-utils" = "*" } assets = [ - { source = "target/release/yt-offline", dest = "/usr/bin/yt-offline", mode = "755" }, - { source = "youtube-backup.desktop", dest = "/usr/share/applications/yt-offline.desktop", mode = "644" }, - { source = "icon.png", dest = "/usr/share/pixmaps/yt-offline.png", mode = "644" }, - { source = "README.md", dest = "/usr/share/doc/yt-offline/README.md", mode = "644" }, - { source = "LICENSE", dest = "/usr/share/licenses/yt-offline/LICENSE", mode = "644" }, + { source = "target/release/catacomb", dest = "/usr/bin/catacomb", mode = "755" }, + { source = "catacomb.desktop", dest = "/usr/share/applications/catacomb.desktop", mode = "644" }, + { source = "icon.png", dest = "/usr/share/pixmaps/catacomb.png", mode = "644" }, + { source = "README.md", dest = "/usr/share/doc/catacomb/README.md", mode = "644" }, + { source = "LICENSE", dest = "/usr/share/licenses/catacomb/LICENSE", mode = "644" }, ] diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..4e4ab72 --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,163 @@ +# Session handoff — Catacomb + +> Working notes for continuing this work in another session/agent (e.g. Codex). +> Written 2026-06-19. Delete or update freely; this is not a tracked spec. + +## TL;DR — where things are + +- Project: `catacomb` — one Rust binary that is **both** an egui desktop GUI + and an axum web server wrapping `yt-dlp`. See [CLAUDE.md](CLAUDE.md) for the + authoritative architecture; [ROADMAP.md](ROADMAP.md) for the plan. +- Recent work has been almost entirely on the **web UI** (`src/web_ui/index.html`, + a single `include_str!`-baked SPA) plus a few `web.rs`/`database.rs` changes. +- Branch: `main`. Last commit: `2f95e7f` (unified cinematic video player). +- A dev web server is usually running on **:8081** against the user's real + library. It keeps getting reaped between turns — just restart it (see below). + +## How to build / run / test + +```bash +cargo build --release # ~1.5 min (opt-level=3 + thin LTO) +cargo test --release # 128 unit + 11 integration (tests/api.rs); no network +./target/release/catacomb --web 8081 # headless web server (what the user uses) +./target/release/catacomb # desktop GUI (default) +``` + +### CRITICAL gotchas (learned this session) + +1. **The web SPA is one big embedded file** — `src/web_ui/index.html`, baked in + at compile time via `include_str!`. **Editing it requires a `cargo build`** to + take effect. A JS syntax error there will NOT be caught by `cargo build` (the + HTML is just a string), so after every edit: + ```bash + awk '/