diff --git a/.forgejo/workflows/docs.yml b/.forgejo/workflows/docs.yml new file mode 100644 index 0000000..d87c77a --- /dev/null +++ b/.forgejo/workflows/docs.yml @@ -0,0 +1,44 @@ +# Build the mdBook docs and publish them to Codeberg Pages. +# +# Codeberg serves the `pages` branch of a repo (or a dedicated `pages` +# repo) at https://.codeberg.page//. This workflow builds the +# book on a push to main that touches docs/ and pushes the rendered HTML +# to the `pages` branch. +name: docs + +on: + push: + branches: [main] + paths: + - 'docs/**' + - '.forgejo/workflows/docs.yml' + workflow_dispatch: + +jobs: + build: + runs-on: docker + container: + image: rust:1.85-bookworm + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Install mdBook + run: cargo install mdbook --no-default-features --features search + + - name: Build the book + run: mdbook build docs + + - name: Publish to the pages branch + # forgejo-release isn't right here; push the built docs/book/ to a + # `pages` branch that Codeberg Pages serves. Uses the repo token. + env: + TOKEN: ${{ secrets.PAGES_TOKEN }} + run: | + cd docs/book + git init -q + git config user.name "ci" + git config user.email "ci@localhost" + git add -A + git commit -q -m "docs build ${{ github.sha }}" + git push -f "https://x:${TOKEN}@codeberg.org/${{ github.repository }}.git" HEAD:pages diff --git a/.gitignore b/.gitignore index c75be1f..3f4c97a 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,9 @@ /src/yt-offline/ /yt-offline/ +# Rendered mdBook output (source lives in docs/src/) +/docs/book/ + # Editor / OS cruft *.swp .DS_Store diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..6661d56 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,122 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +`yt-offline` — a single Rust binary that is **both** a desktop GUI (eframe/egui) +and a headless web server (axum), wrapping `yt-dlp` to archive YouTube/TikTok/ +Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in +[ROADMAP.md](ROADMAP.md); a structured analysis of the Tartube benchmark is in +[docs/tartube-spec.md](docs/tartube-spec.md). + +## Commands + +```bash +cargo build --release # the binary (release profile is opt-level=3 + thin LTO, ~1-2 min) +cargo test --release # all unit tests (fast; no network) +cargo test --release # single test by substring, e.g. `cargo test --release subs_disabled` +./target/release/yt-offline # desktop GUI mode (default) +./target/release/yt-offline --web 8080 # headless web server on a port + +scripts/package.sh [deb|rpm|appimage|all] # build distro packages → dist/ (see docs/PACKAGING.md) +``` + +There is no separate lint step; `cargo build` warnings are the lint. The egui +dependency emits ~39 `f32: From` fallback warnings on a clean build — those +are upstream, not from this code. + +### Running / verifying against a real library + +The app reads `config.toml` and `cookies.txt` **from the process working +directory**, not a fixed path. To smoke-test the web server in isolation, make a +scratch dir with a minimal `config.toml` (`[backup]\ndirectory = "..."`) and run +`--web ` from inside it. `/api/*` endpoints require auth when a password is +set in the target library's DB. + +## Architecture + +### Two front-ends, one engine + +`main.rs` dispatches: `--web` → `web::run()` (axum, blocks forever); otherwise +`app::App` (eframe). **Both share `downloader::Downloader`**, the single source +of truth for yt-dlp job lifecycle. `Downloader` is *not* async — it spawns an OS +thread per `yt-dlp` process, streams stdout/stderr back over an `mpsc` channel +into each `Job`'s log buffer, and the caller must pump `Downloader::poll()` +regularly (the egui frame loop and a web background task both do this). When you +add download behavior, it goes in `Downloader` and is automatically available to +both UIs. + +`poll()` also drives the cross-cutting job machinery: **auto-retry** of transient +failures (rate-limit/network) with cooldown + adaptive throttle, and the +**post-download ffmpeg transcode** pass. These work by capturing specs onto the +`Job` at `start()` time (`RetrySpec`, `ConvertSpec`) and acting on them when the +job transitions state — the `pending_*_spec` fields on `Downloader` are stashed +by `start()` and consumed by `enqueue()`/`spawn_job()` so the four non-download +enqueue paths (repair/music/yt-dlp-update/pot-update) stay untouched. + +### Settings flow (the easy thing to get wrong) + +Almost every feature has the same five-touchpoint shape — miss one and it +silently half-works: + +1. `config.rs` — a field/section + its `Default` and the `default_with_dir()` constructor. +2. `download_options.rs` — an `Option<…>` per-channel override (None = defer to global). +3. `downloader.rs` — a resolver that merges global config + per-channel override into yt-dlp/ffmpeg args, plus a `pub` field on `Downloader` holding the global default. +4. **Both** UIs render the global setting *and* the per-channel override: desktop in `app.rs` (egui Settings screen + the channel-options dialog), web in `web_ui/index.html` (Settings modal + channel-options dialog) **and** `web.rs`'s `SettingsPayload` struct (GET reads from config, POST writes config + pushes the value onto the live `Downloader`). +5. Seed the `Downloader` field at construction **and** on settings-save, in **both** `app.rs` and `web.rs`. + +Grep an existing setting end-to-end before adding one — `subtitle_defaults`, +`youtube_player_clients`, and `convert_defaults` are complete worked examples. + +### Filesystem layout invariant + +`platform::platform_root(channels_root, platform)` = `channels_root.join(dir_name)`. +**All** platforms (including YouTube, whose `dir_name` is `channels`) nest under +the one configured `backup.directory`. `library_root == channels_root` now (a +historical two-level split was removed). `.source-url` sidecars in each creator +folder let channel re-checks recover the exact URL. Library scanning +(`library.rs`) is parallel and consults a `(path, mtime)` SQLite cache to skip +re-parsing unchanged `info.json` sidecars. + +### Persistence + +`database.rs` wraps an r2d2 SQLite pool (file-backed; `Database` is cheaply +`Clone` — the pool is an `Arc`, so the parallel scanner takes its own handle). +Schema lives in `init_schema()`; new columns are added via idempotent +`ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no migration +framework). The web UI holds library/notes snapshots in memory and mutating +endpoints mirror DB writes onto those caches + `bump_library_version()` (the +ETag) so `/api/library` stays consistent without a rescan. + +### Web UI is one embedded file + +`web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked into +the binary at compile time — editing it requires a rebuild to take effect. It's +served with `Cache-Control: no-store` so binary upgrades don't strand stale tabs. +Progress streams over `/ws/progress` (WebSocket) with HTTP-poll fallback. + +### Bundled toolchain & anti-bot + +`ytdlp_bin.rs` manages an optional self-contained venv at +`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi` +for TLS impersonation + bundled `deno`). `pot_provider.rs` runs `bgutil-pot` (a +loopback HTTP server) for YouTube Proof-of-Origin tokens; **its yt-dlp plugin +must come from the same release as the server binary, not PyPI** (version skew +silently produces no tokens — see the module doc). `error_class.rs` pattern- +matches yt-dlp stderr into actionable classes (the captcha "Video unavailable" +wall is classified RateLimited, not NotFound — order matters in `classify()`). + +## Conventions + +- **Never commit** `cookies.txt` (live session creds), `config.toml` (user- + specific), or `yt-offline.db` (contains the Argon2 password hash). All + gitignored. +- Redact the absolute cookies path out of any log line surfaced to the UI/API + (`redact_sensitive` in `downloader.rs`) — it leaks `$HOME`. +- `app.rs` and `web.rs` are large (~3–4k lines) because each owns a full UI; new + desktop code goes in `app.rs`, web handlers in `web.rs`, shared logic in the + focused modules (`downloader`, `database`, `library`, `platform`, …). +- Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only/no-GTK by + design; keep that posture (it's why packaging avoids a GTK dep). Windows/macOS + are not yet first-class — the tray would need a per-OS backend. diff --git a/README.md b/README.md index b0451a0..224d63a 100644 --- a/README.md +++ b/README.md @@ -192,7 +192,7 @@ fields are also editable in Settings. ```toml [backup] -directory = "/path/to/library/channels" # YouTube root; siblings hold other platforms +directory = "/path/to/library" # library root; every platform nests under it (channels/, tiktok/, …) max_concurrent = 3 # parallel yt-dlp processes use_bundled_ytdlp = false # true = use the venv set up by the Install button diff --git a/docs/book.toml b/docs/book.toml new file mode 100644 index 0000000..36f5fa2 --- /dev/null +++ b/docs/book.toml @@ -0,0 +1,12 @@ +[book] +title = "yt-offline" +description = "Self-hosted archive for YouTube, TikTok, Twitch and more. Setup, usage, anti-bot, troubleshooting, and architecture." +authors = ["InannaBeloved"] +language = "en" +src = "src" + +[output.html] +default-theme = "navy" +preferred-dark-theme = "navy" +git-repository-url = "https://codeberg.org/anassaeneroi/yt-offline" +edit-url-template = "https://codeberg.org/anassaeneroi/yt-offline/_edit/main/docs/{path}" diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md new file mode 100644 index 0000000..dffdbf1 --- /dev/null +++ b/docs/src/SUMMARY.md @@ -0,0 +1,16 @@ +# Summary + +[Introduction](./introduction.md) + +# Using yt-offline + +- [Installation](./installation.md) +- [First run & configuration](./first-run.md) +- [Downloading](./downloading.md) +- [Staying ahead of YouTube's bot detection](./anti-bot.md) +- [Troubleshooting](./troubleshooting.md) + +# Contributing + +- [Architecture](./architecture.md) +- [Packaging](./packaging.md) diff --git a/docs/src/anti-bot.md b/docs/src/anti-bot.md new file mode 100644 index 0000000..7f810e2 --- /dev/null +++ b/docs/src/anti-bot.md @@ -0,0 +1,97 @@ +# Staying ahead of YouTube's bot detection + +YouTube increasingly fingerprints and rate-limits automated clients. +yt-offline ships a layered defense; understanding the layers makes the +difference between "everything downloads" and "constant captchas." + +In rough order of impact: + +## 1. Be logged in (cookies) + +This is the single biggest factor. **Anonymous requests get captcha- +walled the hardest.** Provide a `cookies.txt` exported from a browser +where you are **signed in to YouTube**, or point yt-offline at the +browser profile directly. + +A *valid* logged-in jar contains the auth cookies `SID`, `SAPISID`, +`__Secure-1PSID`, `__Secure-3PSID`, `LOGIN_INFO`, etc. A jar with only +`VISITOR_INFO1_LIVE`, `PREF`, `YSC` is **anonymous** — it's not signed in +and actually makes detection *worse* than no cookies. yt-offline's +Settings → Cookies panel warns you when your jar is anonymous or expired. + +Two ways to supply cookies: + +- **Export a `cookies.txt`** with a browser extension like *Get + cookies.txt LOCALLY*, then paste/upload it in Settings → Cookies. A + file in the working directory takes precedence over the browser option. +- **Read the browser profile live** by setting the cookie browser to a + yt-dlp `--cookies-from-browser` spec. Plain `firefox`/`chrome`/`brave` + works for default profiles; for a non-default profile (e.g. Brave's + beta channel) use the full form: + + ``` + brave:/home/you/.config/BraveSoftware/Brave-Origin-Beta + ``` + + The path is the **profile root** (yt-dlp appends the `Default` + subdirectory itself). The advantage: cookies are read fresh from the + live session each download, so they never go stale. + +> Cookies are session credentials — yt-offline never commits or transmits +> `cookies.txt` unprompted, and redacts the cookie path out of any log +> line shown in the UI. + +## 2. TLS impersonation (curl_cffi) + +yt-dlp's `--impersonate` makes requests carry a real browser's TLS +fingerprint (via `curl_cffi`), so the connection doesn't *look* like a +script. The bundled install sets this up automatically and yt-offline +picks an impersonation target per platform. + +If impersonation silently does nothing, it's almost always a +**yt-dlp ⇄ curl_cffi version mismatch** — which is exactly why the +bundled install uses **nightly** yt-dlp (it accepts current curl_cffi; +stable lags and disables all impersonate targets when a newer curl_cffi +is present). See +[Troubleshooting → impersonation](./troubleshooting.md#impersonate-targets-show-unavailable). + +## 3. POT tokens (Proof-of-Origin) + +YouTube increasingly binds a per-video **Proof-of-Origin token** to +playback; without one, format URLs come back empty. yt-offline can run +[bgutil-pot](https://github.com/jim60105/bgutil-ytdlp-pot-provider-rs), a +loopback HTTP server that mints these tokens, and point yt-dlp at it. + +Enable **Settings → Use POT token provider** (requires the bundled +yt-dlp; the matching plugin installs into its venv) and click **Install**. + +> **Version-skew footgun:** the yt-dlp plugin must come from the *same +> release* as the bgutil-pot server binary — **not** the PyPI package, +> which versions independently and silently produces no tokens on a +> mismatch. yt-offline's installer handles this by unpacking the +> version-matched plugin zip from the server's release. + +## 4. Player-client selection + +YouTube cracks down on different internal "player clients" over time — +the `web` client is currently the most captcha-prone, while `tv` and +`mweb` are the least. yt-offline no longer forces `web`; it lets yt-dlp +pick good defaults. If a specific channel keeps hitting captchas, set a +client override (global or per-channel): + +``` +tv,mweb +``` + +## 5. Throttling + +A burst of ~30 rapid requests is a classic trip-wire. yt-offline inserts +a small jittered pause between videos (a fixed cadence looks robotic; a +random one looks human) and, after any rate-limit hit, triples the +sleeps for the rest of the batch before recovering. + +--- + +**TL;DR for a clean setup:** bundled (nightly) yt-dlp + fresh +**logged-in** cookies + POT provider enabled. That combination resolves +the vast majority of captcha / "Video unavailable" failures. diff --git a/docs/src/architecture.md b/docs/src/architecture.md new file mode 100644 index 0000000..dc39578 --- /dev/null +++ b/docs/src/architecture.md @@ -0,0 +1,108 @@ +# Architecture + +For contributors. The repo's `CLAUDE.md` is the terse version of this; +read both. + +## Two front-ends, one engine + +`main.rs` dispatches: `--web` → `web::run()` (axum, blocks forever); +otherwise `app::App` (eframe/egui desktop GUI). **Both share +`downloader::Downloader`**, the single source of truth for the yt-dlp job +lifecycle. + +`Downloader` is **not** async — it spawns an OS thread per yt-dlp process, +streams stdout/stderr back over an `mpsc` channel into each `Job`'s log +buffer, and the caller pumps `Downloader::poll()` regularly (the egui +frame loop and a web background task both do). Anything you add to +`Downloader` is automatically available to both UIs. + +`poll()` also drives the cross-cutting job machinery: auto-retry of +transient failures (cooldown + adaptive throttle), the hang watchdog, and +the post-download ffmpeg transcode pass. These work by capturing specs +onto the `Job` at `start()` time (`RetrySpec`, `ConvertSpec`) and acting +on them when the job changes state. + +## The settings flow (the easy thing to get wrong) + +Almost every configurable feature has the same five-touchpoint shape — +miss one and it silently half-works: + +1. `config.rs` — a field/section + its `Default` + the `default_with_dir` + constructor. +2. `download_options.rs` — an `Option<…>` per-channel override (None = + defer to global). +3. `downloader.rs` — a resolver merging global config + per-channel + override into yt-dlp/ffmpeg args, plus a `pub` field on `Downloader` + holding the global default. +4. **Both** UIs render the global setting *and* the per-channel override: + desktop in `app.rs`, web in `web_ui/index.html` **and** `web.rs`'s + `SettingsPayload` (GET reads config, POST writes config + pushes onto + the live `Downloader`). +5. Seed the `Downloader` field at construction **and** on settings-save, + in **both** `app.rs` and `web.rs`. + +`subtitle_defaults`, `youtube_player_clients`, and `convert_defaults` are +complete worked examples — grep one end-to-end before adding a setting. + +## Filesystem layout + +`platform::platform_root(channels_root, platform)` = +`channels_root.join(dir_name)`. **All** platforms (including YouTube, +whose `dir_name` is `channels`) nest under the one configured +`backup.directory`. `.source-url` sidecars in each creator folder let +re-checks recover the exact URL. Library scanning (`library.rs`) is +parallel and consults a `(path, mtime)` SQLite cache to skip re-parsing +unchanged `info.json` sidecars. + +## Persistence + +`database.rs` wraps an r2d2 SQLite pool. `Database` is cheaply `Clone` +(the pool is an `Arc`), so the parallel scanner takes its own handle. +Schema lives in `init_schema()`; new columns are added via idempotent +`ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no +migration framework). The web UI keeps library/notes snapshots in memory; +mutating endpoints mirror DB writes onto those caches and bump a version +counter (the `/api/library` ETag) so reads stay consistent without a +rescan. + +The long-lived `WebState` mutexes are accessed via +`util::LockExt::lock_recover()`, which recovers a poisoned lock instead of +cascading one handler's panic into a dead server. + +## Web UI is one embedded file + +`web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked +into the binary at compile time — editing it requires a rebuild. Served +`Cache-Control: no-store` so binary upgrades don't strand stale tabs. +Progress streams over `/ws/progress` (WebSocket) with an HTTP-poll +fallback. + +## Anti-bot subsystems + +`ytdlp_bin.rs` manages the optional self-contained venv at +`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` + `curl_cffi` + +bundled `deno`). `pot_provider.rs` runs `bgutil-pot` for Proof-of-Origin +tokens — its yt-dlp plugin must come from the same release as the server +binary. `error_class.rs` pattern-matches yt-dlp stderr into actionable +classes (order matters in `classify()`: the captcha "Video unavailable" +wall is RateLimited, not NotFound). + +## Tests + +- Unit tests are inline `#[cfg(test)]` modules (parsers, resolvers, the + error classifier, DB merge logic). +- `tests/api.rs` spawns the **real** `--web` binary against a scratch dir + and drives the HTTP API with curl — genuine end-to-end coverage of the + axum + SQLite + config stack. + +`cargo test` runs both. `.forgejo/workflows/test.yml` runs them on every +push. + +## Platform support + +Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only / no-GTK +by design — that's why packaging avoids a GTK dependency. Windows/macOS +aren't first-class yet: the tray needs a per-OS backend before a clean +cross-build. The rest (eframe/wgpu, axum, rusqlite-bundled) already +compiles cross-platform, and `ytdlp_bin` already has `cfg!(windows)` +branches. diff --git a/docs/src/downloading.md b/docs/src/downloading.md new file mode 100644 index 0000000..342440d --- /dev/null +++ b/docs/src/downloading.md @@ -0,0 +1,74 @@ +# Downloading + +## Starting a download + +Paste any supported URL into the download bar (desktop) or the ⬇ +Downloads modal (web). yt-offline classifies the URL by platform, routes +it to the right folder, and starts yt-dlp. A channel/playlist URL pulls +the whole thing; a single-video URL pulls just that one. + +**Quality picker:** Best / 1080p / 720p / 480p / 360p, or **Music mode** +for audio-only extraction into `music//`. + +**Fast mode** stops at the first already-downloaded video (quick routine +re-checks). Turn it off for a full gap-filling scan. + +## Per-channel options + +Right-click a channel (or use the ⚙ on its sidebar row) for overrides +that apply to scheduled re-checks and the "Check for new videos" action: + +- Quality cap, audio-only, bandwidth cap, min/max file size, date cutoff. +- A free-form `--match-filter` (e.g. `duration > 60 & view_count > 100`). +- Subtitle overrides, YouTube player-client override, post-download + convert mode — each defaulting to the global setting. +- **Skip auth check** — silences yt-dlp's "playlists that require + authentication" warning for **public** channels (see + [Troubleshooting](./troubleshooting.md#the-youtubetab-authentication-warning)). + +Per-channel options ride along in library backup/restore. + +## Subtitles + +Global defaults (Settings → Subtitles) + per-channel overrides control: +download on/off, auto-generated captions, embedding into the container, +language filter, and format conversion (`srt` is the most +Plex/player-compatible). Subtitles are written as sidecar files and +optionally embedded. + +## Format conversion + +A post-download ffmpeg pass (Settings → Format conversion, or per +channel): + +- **Remux → mp4** — instant container change, no re-encode (device/Plex + compatibility). +- **Re-encode → H.264 mp4** — shrink large 4K files at a chosen CRF + + x264 preset. +- **Extract audio** — mp3 / m4a / opus / flac. + +It runs as a distinct transcode job after the download. **Keep original** +preserves `.original.` alongside the converted file; otherwise +the source is removed once the convert succeeds. + +## Resilience + +Downloads are hardened against YouTube's flakiness automatically: + +- **Retry + backoff** on connection resets (`--retries 30`, linear + retry-sleep). +- **Jittered throttle** between videos so a long channel scan doesn't + look robotic and trip the captcha wall. +- **Auto-retry** of transient (rate-limit / network / captcha) failures + after a cooldown, with adaptive slow-down for the rest of the batch. +- **Hang watchdog** kills a job that produces no output for 5 minutes (a + wedged request) and re-queues it. + +Failures are classified and shown with a one-line suggested fix — see +[Troubleshooting](./troubleshooting.md). + +## Scheduler + +Enable it (Settings → Auto-check channels) to re-check every channel for +new uploads on an interval. Each channel uses its own stored options. +There's also a per-folder "Check all" action. diff --git a/docs/src/first-run.md b/docs/src/first-run.md new file mode 100644 index 0000000..fd7840a --- /dev/null +++ b/docs/src/first-run.md @@ -0,0 +1,91 @@ +# First run & configuration + +## The config file + +yt-offline reads `config.toml` **from its working directory** (the +directory you launch it from), not a fixed path. The same goes for +`cookies.txt`. Everything in `config.toml` is also editable in Settings; +edits there are written back to the file. + +```toml +[backup] +directory = "/path/to/library" # the umbrella dir; all platforms nest under it +max_concurrent = 3 # parallel yt-dlp processes +use_bundled_ytdlp = false # true = use the managed venv (see below) +use_pot_provider = false # YouTube Proof-of-Origin tokens (see Anti-bot) +youtube_player_clients = "" # e.g. "tv,mweb" to route around captchas + +[player] +command = "mpv" # any executable taking a file path as its last arg +browser = "firefox" # cookie source when no cookies.txt is set (see Anti-bot) + +[ui] +theme = "dark" # dark | light | dracula | trans | emo-nocturnal | emo-coffin | emo-scene-queen +ui_scale = 1.0 # global zoom for the whole desktop UI + +[scheduler] +enabled = false +interval_hours = 24 # auto re-check every channel for new uploads + +[web] +port = 8080 +bind = "127.0.0.1" # 127.0.0.1 | 0.0.0.0 | a Tailscale/LAN address +transcode = false # MKV → MP4 on the fly for browsers that can't decode MKV + +[subtitles] +enabled = true +auto_generated = true # include machine captions +embed = false # also embed into the container +format = "" # "" = native; "srt" for Plex compatibility +langs = "" # "" = all; "en" or "en,ja" to filter + +[convert] +mode = "" # "" / "remux-mp4" / "h264-mp4" / "audio" +crf = 23 # for h264-mp4 (lower = bigger/better) +preset = "medium" +audio_format = "mp3" # for audio mode +keep_original = false # keep .original. after converting + +[plex] +library_path = "/path/to/plex/TV/youtube" # leave unset to disable +``` + +## The library layout + +Everything nests under the one `backup.directory`: + +```text +/ + channels/ ← YouTube creators + tiktok/ twitch/ vimeo/ bandcamp/ soundcloud/ odysee/ other/ + music/ ← audio-only "Music mode" downloads, by artist + archive.txt ← yt-dlp's global download archive + cookies.txt ← optional, if you set one + yt-offline.db ← watched/positions/flags/folders/notes/cache + password hash +``` + +Each creator folder gets a hidden `.source-url` sidecar so re-checks +always know the exact URL to refresh from. + +## Bundled vs system yt-dlp + +In **Settings → yt-dlp binary** you choose: + +- **System** — uses whatever `yt-dlp` is on your `PATH`. +- **Bundled** — click **Install** and yt-offline builds a self-contained + venv at `~/.local/share/yt-offline/`: nightly `yt-dlp[default]` + + `curl_cffi` (TLS impersonation) + a bundled `deno` (player-JS). The + same button updates it later. + +The bundled path is recommended — it installs **nightly** yt-dlp, which +keeps pace with YouTube's frequent anti-bot changes (stable lags). It's +also required for the [POT token provider](./anti-bot.md#3-pot-tokens-proof-of-origin). + +## The two front-ends + +- `yt-offline` — desktop GUI (eframe/egui). +- `yt-offline --web [PORT]` — headless web server. Bind to `127.0.0.1` + (default) for localhost-only, a Tailscale address for your tailnet, or + `0.0.0.0` for the LAN. **Set a password** (Settings) before exposing it + beyond localhost — the UI and all `/api` routes are then gated behind an + Argon2-hashed, rate-limited login. diff --git a/docs/src/installation.md b/docs/src/installation.md new file mode 100644 index 0000000..222c609 --- /dev/null +++ b/docs/src/installation.md @@ -0,0 +1,73 @@ +# Installation + +yt-offline is a single Rust binary. You can install a prebuilt package, +build from source, or grab the AppImage. + +## Runtime dependencies + +Whichever way you install, these are invoked as subprocesses at runtime: + +- **yt-dlp** — the download engine. You can use the system one *or* let + yt-offline manage a bundled copy (see [First run](./first-run.md)). +- **ffmpeg** — muxing, format conversion, on-the-fly transcode for the + web player. +- **mpv** — the default desktop player (any player taking a file path + works; set it in Settings). +- **xdg-utils** — `xdg-open` for "Show in file manager". + +## Prebuilt packages (Linux) + +Releases attach `.deb`, `.rpm`, and `.AppImage` artifacts. + +```sh +# Debian / Ubuntu / Mint +sudo apt install ./yt-offline_*_amd64.deb + +# Fedora / RHEL / openSUSE (ffmpeg via RPM Fusion) +sudo dnf install ./yt-offline-*.x86_64.rpm + +# Any Linux — AppImage +chmod +x yt-offline-*-x86_64.AppImage +./yt-offline-*-x86_64.AppImage +``` + +## Arch / CachyOS / Manjaro + +A `PKGBUILD` ships in the repo root. Build it from a **clean** directory: + +```sh +mkdir build && cd build +cp /path/to/repo/PKGBUILD . +makepkg -si +``` + +For repeated builds after pulling new commits, always pass `-C` +(cleanbuild) so makepkg re-checks out the latest source instead of +reusing a stale cached clone. + +## From source + +```sh +# Debian/Ubuntu build deps +sudo apt install build-essential pkg-config curl git python3-venv \ + libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev \ + libxkbcommon-dev libssl-dev +curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh +source "$HOME/.cargo/env" + +git clone https://codeberg.org/anassaeneroi/yt-offline +cd yt-offline +cargo build --release +./target/release/yt-offline # desktop GUI +./target/release/yt-offline --web 8080 # headless web server +``` + +`python3-venv` is only needed for the bundled-yt-dlp install path; skip +it if you'll always use system yt-dlp. + +## Windows / macOS + +Not first-class yet. The Linux-only system tray (`ksni`) and file dialog +(`rfd` xdg-portal) need per-OS backends before a clean cross-build; the +rest of the stack already compiles. See the +[architecture notes](./architecture.md#platform-support). diff --git a/docs/src/introduction.md b/docs/src/introduction.md new file mode 100644 index 0000000..3b942cc --- /dev/null +++ b/docs/src/introduction.md @@ -0,0 +1,40 @@ +# yt-offline + +A self-hosted media archive for YouTube and friends. Paste any URL, it +routes the download to the right folder by source, tracks what you've +watched, and plays everything back from a desktop GUI **or** a browser — +even offline, or after the source video is taken down. + +Built on [yt-dlp](https://github.com/yt-dlp/yt-dlp); written in Rust as a +single binary that is **both** a desktop app and a headless web server. + +## What it backs up + +| Platform | Channels | Playlists | Single videos | +|---|---|---|---| +| YouTube | ✅ | ✅ | ✅ | +| TikTok | ✅ | — | ✅ | +| Twitch (VODs + clips) | ✅ | — | ✅ | +| Vimeo | ✅ | ✅ | ✅ | +| Bandcamp | ✅ (artist) | ✅ (albums) | ✅ (tracks) | +| SoundCloud | ✅ | ✅ (sets) | ✅ | +| Odysee | ✅ | — | ✅ | +| Anything else yt-dlp accepts | `other/` | `other/` | `other/` | + +## Why it exists + +Tartube is the mature open-source yt-dlp GUI and the benchmark in this +space. yt-offline matches its feature set while adding things Tartube +doesn't have: a real web UI reachable from any device, a single-binary +distribution with a bundled toolchain, a modern security model +(password-gated UI, Argon2, rate-limited login), a built-in anti-bot +stack (TLS impersonation + Proof-of-Origin tokens), and one-click +post-download format conversion. + +## How to read these docs + +- New here? **[Installation](./installation.md)** → + **[First run](./first-run.md)** → **[Downloading](./downloading.md)**. +- Hitting captchas or "Video unavailable"? Go straight to + **[Anti-bot](./anti-bot.md)** and **[Troubleshooting](./troubleshooting.md)**. +- Hacking on it? **[Architecture](./architecture.md)**. diff --git a/docs/src/packaging.md b/docs/src/packaging.md new file mode 100644 index 0000000..6fd42eb --- /dev/null +++ b/docs/src/packaging.md @@ -0,0 +1,39 @@ +# Packaging + +Build distributable Linux packages with one script: + +```sh +scripts/package.sh all # .deb + .rpm + .AppImage → dist/ +scripts/package.sh deb # just the .deb +scripts/package.sh rpm +scripts/package.sh appimage +``` + +It builds the release binary once and reuses it for every format, +installing `cargo-deb` / `cargo-generate-rpm` on demand and downloading +`appimagetool` to `dist/tools/` on first AppImage build. Per-format +failures are isolated and summarized at the end. Output (gitignored) +lands in `dist/`. + +## Formats + +- **`.deb`** — built by [cargo-deb](https://github.com/kornelski/cargo-deb) + from `[package.metadata.deb]` in `Cargo.toml`. +- **`.rpm`** — built by + [cargo-generate-rpm](https://github.com/cat-in-136/cargo-generate-rpm) + from `[package.metadata.generate-rpm]`. (`ffmpeg` on Fedora is in + RPM Fusion.) +- **AppImage** — a hand-rolled AppDir + appimagetool. Bundles the GUI + binary's shared-library closure only; `yt-dlp`/`ffmpeg`/`mpv` stay host + PATH deps, same as the package declarations. +- **Arch** — use the repo's `PKGBUILD` (not this script); run `makepkg` + from a clean directory. + +## CI + +`.forgejo/workflows/release.yml` runs `scripts/package.sh all` on every +pushed `v*` tag and attaches the artifacts to the Codeberg release. +`.forgejo/workflows/test.yml` runs the full test suite on every push. + +The repo's [`docs/PACKAGING.md`](https://codeberg.org/anassaeneroi/yt-offline/src/branch/main/docs/PACKAGING.md) +has the per-distro install commands and the Windows/macOS status in full. diff --git a/docs/src/troubleshooting.md b/docs/src/troubleshooting.md new file mode 100644 index 0000000..df62613 --- /dev/null +++ b/docs/src/troubleshooting.md @@ -0,0 +1,113 @@ +# Troubleshooting + +yt-offline classifies failed downloads into one of nine classes and shows +a one-line suggested fix next to the failed job. This page expands on the +most common ones, plus a few non-download issues. + +## "Video unavailable. YouTube is requiring a captcha challenge" + +**Class:** rate-limited. **Not** a removed video — it's the bot-detection +wall. In order of effectiveness: + +1. **Use fresh, logged-in cookies.** Anonymous cookies are the usual + culprit — see [Anti-bot → cookies](./anti-bot.md#1-be-logged-in-cookies). + Settings → Cookies warns when your jar is anonymous or expired. +2. **Switch to bundled (nightly) yt-dlp** if you're on system stable. +3. **Enable the POT token provider.** +4. **Try a player-client override** of `tv,mweb` for that channel. +5. If it's a one-off, just wait — yt-offline auto-retries transient + rate-limits after a cooldown. + +## Impersonate targets show "(unavailable)" + +`yt-dlp --list-impersonate-targets` lists every target as `(unavailable)` +even though `curl_cffi` is installed. + +**Cause:** a yt-dlp ⇄ curl_cffi version gate. Stable yt-dlp caps the +curl_cffi version it accepts; a newer curl_cffi makes it disable *all* +impersonate targets. + +**Fix:** use the **bundled** yt-dlp (it installs nightly via `--pre`, +which accepts current curl_cffi), or pin curl_cffi to a compatible +version in your own environment. + +## POT provider produces no tokens + +You enabled the POT provider and installed it, but downloads still fail +as if no token was generated. yt-dlp logs a *"plugin and HTTP server +major versions are mismatched"* warning. + +**Cause:** the yt-dlp plugin came from PyPI (Brainicism's package, which +versions independently) instead of the jim60105 Rust server's release. + +**Fix:** re-run the POT **Install/Update** button — yt-offline installs +the version-matched plugin zip from the same release as the server +binary. Don't `pip install bgutil-ytdlp-pot-provider` yourself. + +## The youtubetab authentication warning + +``` +ERROR: [youtube:tab] @Channel: Playlists that require authentication may +not extract correctly without a successful webpage download... +``` + +Despite the `ERROR:` prefix this is a soft warning, usually a symptom of +the bot-detection issues above (YouTube served an incomplete page). It +does **not** change which videos are found. + +**Fix:** for **public** channels, enable **Skip auth check** in that +channel's options (adds `--extractor-args youtubetab:skip=authcheck`) to +silence it. Leave it **off** for members-only/private channels you +archive with cookies — there the warning is a real "your cookies may not +be working" signal. + +## "Sign in to confirm you're not a bot" + +Same family as the captcha wall. Fix with fresh logged-in cookies + POT; +see [Anti-bot](./anti-bot.md). + +## Downloads stall forever + +A job sits running with no progress. yt-offline's **hang watchdog** +auto-kills any job silent for 5 minutes and re-queues it, so this should +self-heal. If it recurs on a specific URL, it's usually a server-side +issue with that source; check the job log in the Downloads panel. + +## Disk fills up / downloads fail with ENOSPC + +yt-offline runs a **disk-full preflight** and refuses to start a download +when the target filesystem has less than ~500 MB free, surfacing it as a +clear "disk full" failure rather than a half-written file. Free space and +retry. + +## A whole platform folder shows up as one "channel" + +If you see `bandcamp`, `tiktok`, or `channels` listed as a single channel +in the sidebar, your library directory predates the current layout. All +platforms must **nest under** the one `backup.directory` +(`/channels/`, `/tiktok/`, …). Move stray creator folders into +their platform's subdir; see [First run → library layout](./first-run.md#the-library-layout). + +## The desktop window crashes on maximize + +Older builds crashed with a Glutin `EGL_BAD_ALLOC` on NVIDIA + Wayland +when maximized. Current builds use the **wgpu (Vulkan)** renderer, which +handles the resize cleanly. Make sure you have a working Vulkan driver +(`vulkan-icd-loader` + your GPU's Vulkan package), which any desktop with +working graphics already has. + +## The web UI looks like an old version after an upgrade + +The SPA is served `Cache-Control: no-store`, so a hard reload +(Ctrl+Shift+R) always picks up the new binary's UI. If you upgraded the +binary, also **restart the running `--web` process** — the HTML is baked +into the binary at compile time, so the old process keeps serving the old +UI until restarted. + +## Where to look next + +- **The job log** — every download/transcode job keeps its full yt-dlp / + ffmpeg output in the Downloads panel (expand the job). +- **`yt-offline.crash.log`** — next to your `yt-offline.db`. A panic in + any thread (UI, web worker, download) is appended here with a + timestamp, so it survives a GUI launched without a terminal.