diff --git a/Cargo.toml b/Cargo.toml index 7288d1f..0cfb1c7 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,7 +2,7 @@ name = "yt-offline" version = "0.1.0" edition = "2021" -description = "A desktop app for archiving YouTube channels with yt-dlp" +description = "Self-hosted archive for YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee and more. Desktop GUI + web UI, bundled yt-dlp with curl_cffi impersonation, Plex export, SQLite-backed resume tracking. AGPL-3.0." [dependencies] eframe = "0.29" diff --git a/README.md b/README.md index 51ac0f4..b0451a0 100644 --- a/README.md +++ b/README.md @@ -2,204 +2,319 @@ ![yt-offline icon](icon.png) -A desktop app for archiving YouTube channels using [yt-dlp](https://github.com/yt-dlp/yt-dlp). Browse your downloaded channels, manage downloads, and play videos — all from a native GUI. +A self-hosted media archive for YouTube and friends. Pastes any URL, routes it +to the right folder by source, tracks what you've watched, and lets you play +everything back from a desktop GUI or browser — even when you're offline or +the source video has been taken down. + +Built on [yt-dlp](https://github.com/yt-dlp/yt-dlp); written in Rust. + +## What it backs up + +| Platform | Channels | Playlists | Single videos | +| ---------------------------- | ----------- | ----------- | ------------- | +| YouTube | ✅ | ✅ | ✅ | +| TikTok | ✅ | — | ✅ | +| Twitch (VODs + clips) | ✅ | — | ✅ | +| Vimeo | ✅ | ✅ | ✅ | +| Bandcamp | ✅ (artist) | ✅ (albums) | ✅ (tracks) | +| SoundCloud | ✅ | ✅ (sets) | ✅ | +| Odysee | ✅ | — | ✅ | +| Anything else yt-dlp accepts | `other/` | `other/` | `other/` | + +Each source lands in its own sibling folder under your backup directory +(`channels/` for YouTube, `tiktok/`, `twitch/`, etc.). A `.source-url` +sidecar is dropped in every creator folder so re-checks always know the +exact URL to refresh from. ## Features -- **Channel Browser** — Browse downloaded channels and videos with thumbnails -- **Smart Download Routing** — Paste any YouTube URL; channels, playlists, and single videos are automatically routed to the right folder -- **Playlist View** — Channels with playlist subdirectories show them in the sidebar -- **Themes** — Dark, Light, Dracula, Trans, and three Emo/Scene themes -- **Settings GUI** — Configure everything from inside the app -- **Video Playback** — Launch videos in mpv, VLC, or any player -- **Search** — Filter across your entire library in real time -- **System Tray** — Minimize to tray +### Library + +- **Multi-platform sidebar** grouped by source, with platform icons. +- **Search** across your whole library, instant. +- **Sort by:** Newest / Oldest (upload date), Largest / Smallest, Longest / + Shortest, Title. +- **Watched tracking** + **resume positions** persisted in SQLite. +- **Continue watching** view of partially-played videos. +- **Subtitles** (auto + manual, SRT and WebVTT) + **chapters** + **SponsorBlock** + marks embedded into the video file. +- **Statistics** — totals, top channels by size/count, weekly downloads + histogram, upload-year histogram. +- **Maintenance** — finds duplicate video IDs across folders and missing + sidecars (thumbnail / info.json / description), with one-click repair. + +### Playback + +- **Desktop GUI** launches your configured player (mpv by default; mpv gets + resume-position tracking via JSON-IPC). +- **Web UI** plays videos inline in any browser, with subtitle tracks and a + chapters panel. Optional on-the-fly **ffmpeg transcoding** for browsers + that can't decode MKV. + +### Downloads + +- **Bundled yt-dlp + curl_cffi + deno** — one click in Settings sets up a + self-contained Python venv with `yt-dlp[default]` plus `curl_cffi` + (browser-impersonation for bot-detection bypass) and `deno` for player-JS + evaluation. No system installation required. +- Or use **system yt-dlp** — toggleable in Settings. +- **Concurrent-download limit** (default 3) with a visible queue. +- **Quality picker:** Best / 1080p / 720p / 480p / 360p, or **Music mode** + for audio-only extraction into `music//`. +- **Retry hardening:** `--retries 30 --fragment-retries 30 + --retry-sleep linear=1:30:2 --sleep-requests 1` so connection resets self-heal + instead of failing the job. +- **Scheduler** — periodically re-check every channel for new uploads. +- **Cookies** — paste / file-pick a Netscape `cookies.txt`, or fall back to + `--cookies-from-browser` for whatever browser you pick. + +### Web server + +- Single binary, no Node / Python / Docker required (Python only for the + optional bundled yt-dlp venv). +- **Bind interface picker** — localhost only (default), Tailscale, LAN, or + all interfaces. +- **Password-gated UI** when enabled — Argon2 hashed, 256-bit session + tokens, 30-day TTL with lazy pruning, per-IP rate-limit on `/api/login` + (5 failures → 60s lockout). +- **Security headers:** Content-Security-Policy, X-Frame-Options DENY, + X-Content-Type-Options nosniff, Referrer-Policy no-referrer. +- **`Secure` cookie flag** when `X-Forwarded-Proto: https` is present + (i.e. behind a reverse proxy doing TLS). +- 4 MiB request body limit on every route. + +### Plex integration + +- One click generates a symlink tree your Plex "TV Shows" library can ingest + directly: `//Season 2024/ - S2024E001 - title.mkv`. +- Writes `.nfo` sidecars per episode (title, season, episode, aired date, + runtime, plot) plus a show-level `tvshow.nfo`, so Plex's "Personal Media + (TV Shows)" agent picks up YouTube metadata correctly. +- Symlinks the thumbnail as `-thumb.jpg` so Plex shows episode art. +- Non-YouTube creators get ` - ` show folders to avoid + collisions across sources. + +### Themes + +Ten themes shared across desktop and web — Dark, Light, Dracula, Trans, +plus three Emo/Scene flavours (Nocturnal, Coffin, Scene Queen). + +### Misc + +- Native notifications when downloads finish (notify-rust / zbus, no GTK + build dep). +- `/api/stats`, `/api/ytdlp/update`, `/api/scheduler/run` — every UI button + has an equivalent JSON endpoint. + +## Quick start + +```bash +git clone https://codeberg.org/anassaeneroi/yt-offline +cd yt-offline +cargo build --release + +# Desktop GUI +./target/release/yt-offline + +# Or web server only (headless) +./target/release/yt-offline --web 8080 +``` + +On first run a `config.toml` is created in the working directory. Settings +can also be changed from inside either UI — no manual editing required. + +### Bundled yt-dlp setup + +1. Open **Settings**. +2. Under **yt-dlp binary**, choose **Bundled** and click **Install**. +3. The installer creates `~/.local/share/yt-offline/venv/` with + `yt-dlp[default]` + `curl_cffi`, plus `~/.local/share/yt-offline/bin/deno`. + Progress streams into a regular job entry. + +Updates are the same button. Switching to **System** uses whatever `yt-dlp` +is on PATH instead. ## Building -### Arch Linux / Manjaro - -A `PKGBUILD` is included: +### Arch / CachyOS / Manjaro ```bash +sudo pacman -S --needed rust mpv # python3 + python-pip already present git clone https://codeberg.org/anassaeneroi/yt-offline cd yt-offline -makepkg -si +makepkg -si # or: cargo build --release ``` -Or build manually: +A `PKGBUILD` is included. -```bash -sudo pacman -S --needed rust yt-dlp mpv -cargo build --release -sudo install -Dm755 target/release/yt-offline /usr/bin/yt-offline -``` - ---- - -### Debian / Ubuntu / Linux Mint - -Install build dependencies: +### Debian / Ubuntu ```bash sudo apt install \ - build-essential pkg-config curl git \ + build-essential pkg-config curl git python3-venv \ libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev \ - libxkbcommon-dev libssl-dev \ - libgtk-3-dev libayatana-appindicator3-dev -``` - -Install Rust (if not already installed): - -```bash + libxkbcommon-dev libssl-dev curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source "$HOME/.cargo/env" -``` - -Install runtime dependencies: - -```bash -sudo apt install yt-dlp mpv -``` - -Build and install: - -```bash -git clone https://codeberg.org/anassaeneroi/yt-offline -cd yt-offline cargo build --release -sudo install -Dm755 target/release/yt-offline /usr/bin/yt-offline -cp youtube-backup.desktop ~/.local/share/applications/yt-offline.desktop ``` ---- +`python3-venv` is required for the bundled yt-dlp install path (skipable +if you only ever use system yt-dlp). ### macOS -Install Xcode command line tools and Homebrew dependencies: - ```bash xcode-select --install -brew install yt-dlp mpv -``` - -Install Rust: - -```bash +brew install mpv curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source "$HOME/.cargo/env" +cargo build --release ``` -Build: +### Windows -```bash +Install [Rust](https://rustup.rs/) (MSVC toolchain), Visual Studio Build +Tools with the **Desktop development with C++** workload, then: + +```powershell +winget install mpv.net python.python.3.12 git clone https://codeberg.org/anassaeneroi/yt-offline cd yt-offline cargo build --release ``` -The binary is at `target/release/yt-offline`. Copy it wherever you like, or run it in place. - ---- - -### Windows - -1. Install [Rust](https://rustup.rs) — accept the default MSVC toolchain when prompted -2. Install [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) with the **Desktop development with C++** workload -3. Install yt-dlp and mpv: - - ```powershell - winget install yt-dlp.yt-dlp - winget install mpv.net - ``` - -4. Clone and build: - - ```powershell - git clone https://codeberg.org/anassaeneroi/yt-offline - cd yt-offline - cargo build --release - ``` - -The binary is at `target\release\yt-offline.exe`. Copy it wherever you like and run it. - -> **Note:** The first build takes a while — Rust compiles all dependencies from scratch. - ---- - ## Configuration -On first run, create a `config.toml` next to the binary (or edit it from the Settings button inside the app): +`config.toml` lives next to the binary (or in the working directory). All +fields are also editable in Settings. ```toml [backup] -directory = "/path/to/your/video/library" +directory = "/path/to/library/channels" # YouTube root; siblings hold other platforms +max_concurrent = 3 # parallel yt-dlp processes +use_bundled_ytdlp = false # true = use the venv set up by the Install button [player] -command = "mpv" +command = "mpv" # any executable that takes a file path as last arg +browser = "firefox" # used by --cookies-from-browser when no cookies.txt is set [ui] -theme = "dark" +theme = "dark" # dark | light | dracula | trans | emo-nocturnal | emo-coffin | emo-scene-queen + +[scheduler] +enabled = false +interval_hours = 24 + +[web] +port = 8080 +bind = "127.0.0.1" # 127.0.0.1 | 0.0.0.0 | a Tailscale/LAN address +transcode = false # MKV → MP4 on the fly for browsers (needs ffmpeg) +source_url = "https://codeberg.org/anassaeneroi/yt-offline" # AGPL §13 footer link + +[plex] +library_path = "/path/to/plex/TV/youtube" # leave unset to disable ``` -### Options - -| Setting | Default | Description | -| ------- | ------- | ----------- | -| `backup.directory` | `./channels` | Where downloaded videos are stored | -| `player.command` | `mpv` | Command used to launch the video player | -| `ui.theme` | `dark` | `dark`, `light`, `dracula`, `trans`, `emo-nocturnal`, `emo-coffin`, `emo-scene-queen` | - ---- - ## Usage -1. **Download a channel/video/playlist** — click **⬇ Downloads**, paste a YouTube URL. The app detects the URL type and routes it automatically: - - Channel URL → `channels//` - - Single video → `channels//` - - Playlist → `channels///` +### Desktop GUI -2. **Browse your library** — channels appear in the left sidebar. If a channel has playlist subdirectories they show as collapsible sub-items. +1. Click **⬇ Downloads**, paste any URL — channel, video, playlist, or + profile from any supported platform. The dialog shows the detected + source and target folder before you confirm. +2. Choose quality (or **🎵 Music** for audio-only). +3. Pick **Fast mode** to stop at the first already-archived video, or leave + it off for a full back-fill. -3. **Play a video** — click **▶ Play** on any video card, or double-click the thumbnail. +The sidebar groups channels by platform. Right-click any channel for +**Check for new videos** and **Open folder**. The download bar shows the +active job count plus what's queued. -4. **Change settings** — click **⚙ Settings** to change the backup directory, player, or theme without editing the file. +Other top-bar buttons: ---- +- **⟳ Rescan** — re-read the library directory. +- **📊 Stats** — totals, top channels, weekly download histogram. +- **🩺 Maintenance** — find and resolve duplicates / missing sidecars. +- **⚙ Settings** — everything in `config.toml`, plus cookies, password, + bundled-yt-dlp install/update, and starting the web server. + +### Web UI + +Start the server (`--web` CLI flag or **Start** button in Settings → +**Web server**), open `http://localhost:8080` (or your configured bind +address). The UI mirrors the desktop one and adds inline video playback. + +If a download password is set, you'll be redirected to a login page; +the session cookie is HttpOnly + SameSite=Strict, valid 30 days. + +## Library layout + +```text +/path/to/library/ +├── channels/ ← YouTube +│ ├── @creator-name/ +│ │ ├── .source-url ← original URL, for re-checks +│ │ ├── archive.txt ← yt-dlp's downloaded-ID record +│ │ ├── Video Title [abc123].mkv +│ │ ├── Video Title [abc123].webp +│ │ ├── Video Title [abc123].info.json +│ │ ├── Video Title [abc123].description +│ │ └── Video Title [abc123].en.vtt +│ └── @another/ +├── tiktok/ +├── twitch/ +├── vimeo/ +├── bandcamp/ +├── soundcloud/ +├── odysee/ +├── other/ +├── music/ +│ └── / +│ └── Track Title [xyz].opus +└── yt-offline.db ← watched + positions + password hash +``` ## Troubleshooting -**`yt-dlp` not found** -Make sure it's installed and on your `$PATH`. On Windows, restart your terminal after install. +**"Impersonate target ... is not available"** +You're on bundled mode but the venv install hasn't completed. Open +Settings → yt-dlp binary → click **Install** and watch the job for the +"impersonation targets available" line. -**Build fails on Debian/Ubuntu with missing headers** -Make sure you installed all packages in the build dependencies block above, especially `libayatana-appindicator3-dev` and `libgtk-3-dev`. +**`yt-dlp` not found (system mode)** +Install yt-dlp via your package manager or `pip install yt-dlp`, or +switch to **Bundled** in Settings. -**Build fails on Windows with linker errors** -Make sure Visual Studio Build Tools are installed with the C++ workload selected, not just the base tools. +**Web UI says "authentication required" but no password is set** +Sessions cleared after the password was changed. Refresh the page. -**Videos won't play** -Check `player.command` in `config.toml` (or Settings). The command must accept a file path as its last argument. +**Videos play but seeking jumps to the start** +That's the transcoding mode (`web.transcode = true`). MKV streams can't be +seeked while transcoding live; turn it off if your browser can play MKV +directly. -**Thumbnails not loading** -Supported formats: JPEG, PNG, WebP. Other formats are silently skipped. +**Connection-reset / "Recv failure" errors on download** +The defaults already retry 30× with linear backoff. If they're persistent, +your IP is being rate-limited — paste fresh cookies from a browser session +into Settings → Cookies. ---- +**Build fails on Debian/Ubuntu** +Make sure all packages from the build dependencies block are installed, +plus `python3-venv` if you want the bundled yt-dlp install path. -## Project Structure +## Security -```text -src/ - main.rs entry point - app.rs UI and main loop - downloader.rs yt-dlp integration and URL detection - library.rs channel/playlist/video scanner - config.rs config file loading and saving - theme.rs all colour themes - database.rs SQLite (reserved for future use) - tray.rs system tray -``` +See [SECURITY_AUDIT.md](SECURITY_AUDIT.md) — the codebase has been audited +end-to-end, covering command/SQL injection, path traversal, XSS, session +management, rate-limiting, supply-chain (bundled binary SHA verification), +and dependency status. ## License -See [LICENSE](LICENSE). +[AGPL-3.0-or-later](LICENSE). When you run this software as a network +service, the AGPL requires that you offer network users access to the +Corresponding Source — the running version, with any modifications. +Set `web.source_url` in config to a public URL where the source is +available; the web UI surfaces it in the footer and `GET /api/settings`.