Documentation site: mdBook user + contributor guide (2.2)
An mdBook under docs/ rendering eight pages: introduction, installation, first-run/config, downloading, an anti-bot guide that captures the hard-won cookies/curl_cffi/POT/player-client knowledge, troubleshooting (the nine error classes + non-download issues), architecture (the two-front-ends/one-engine design, settings flow, layout invariant, persistence), and packaging. - .forgejo/workflows/docs.yml builds the book and publishes docs/book/ to the `pages` branch for Codeberg Pages on docs/ changes. - docs/book/ is gitignored (rendered output). - README: fix the stale backup.directory comment — every platform nests under the one library root, not a channels/ sibling split. - Add CLAUDE.md (repo guidance for Claude Code). Builds clean with mdbook 0.5; intra-doc anchor links verified. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
7906d7d07f
commit
e06c2ef826
14 changed files with 833 additions and 1 deletions
12
docs/book.toml
Normal file
12
docs/book.toml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
[book]
|
||||
title = "yt-offline"
|
||||
description = "Self-hosted archive for YouTube, TikTok, Twitch and more. Setup, usage, anti-bot, troubleshooting, and architecture."
|
||||
authors = ["InannaBeloved"]
|
||||
language = "en"
|
||||
src = "src"
|
||||
|
||||
[output.html]
|
||||
default-theme = "navy"
|
||||
preferred-dark-theme = "navy"
|
||||
git-repository-url = "https://codeberg.org/anassaeneroi/yt-offline"
|
||||
edit-url-template = "https://codeberg.org/anassaeneroi/yt-offline/_edit/main/docs/{path}"
|
||||
16
docs/src/SUMMARY.md
Normal file
16
docs/src/SUMMARY.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# Summary
|
||||
|
||||
[Introduction](./introduction.md)
|
||||
|
||||
# Using yt-offline
|
||||
|
||||
- [Installation](./installation.md)
|
||||
- [First run & configuration](./first-run.md)
|
||||
- [Downloading](./downloading.md)
|
||||
- [Staying ahead of YouTube's bot detection](./anti-bot.md)
|
||||
- [Troubleshooting](./troubleshooting.md)
|
||||
|
||||
# Contributing
|
||||
|
||||
- [Architecture](./architecture.md)
|
||||
- [Packaging](./packaging.md)
|
||||
97
docs/src/anti-bot.md
Normal file
97
docs/src/anti-bot.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Staying ahead of YouTube's bot detection
|
||||
|
||||
YouTube increasingly fingerprints and rate-limits automated clients.
|
||||
yt-offline ships a layered defense; understanding the layers makes the
|
||||
difference between "everything downloads" and "constant captchas."
|
||||
|
||||
In rough order of impact:
|
||||
|
||||
## 1. Be logged in (cookies)
|
||||
|
||||
This is the single biggest factor. **Anonymous requests get captcha-
|
||||
walled the hardest.** Provide a `cookies.txt` exported from a browser
|
||||
where you are **signed in to YouTube**, or point yt-offline at the
|
||||
browser profile directly.
|
||||
|
||||
A *valid* logged-in jar contains the auth cookies `SID`, `SAPISID`,
|
||||
`__Secure-1PSID`, `__Secure-3PSID`, `LOGIN_INFO`, etc. A jar with only
|
||||
`VISITOR_INFO1_LIVE`, `PREF`, `YSC` is **anonymous** — it's not signed in
|
||||
and actually makes detection *worse* than no cookies. yt-offline's
|
||||
Settings → Cookies panel warns you when your jar is anonymous or expired.
|
||||
|
||||
Two ways to supply cookies:
|
||||
|
||||
- **Export a `cookies.txt`** with a browser extension like *Get
|
||||
cookies.txt LOCALLY*, then paste/upload it in Settings → Cookies. A
|
||||
file in the working directory takes precedence over the browser option.
|
||||
- **Read the browser profile live** by setting the cookie browser to a
|
||||
yt-dlp `--cookies-from-browser` spec. Plain `firefox`/`chrome`/`brave`
|
||||
works for default profiles; for a non-default profile (e.g. Brave's
|
||||
beta channel) use the full form:
|
||||
|
||||
```
|
||||
brave:/home/you/.config/BraveSoftware/Brave-Origin-Beta
|
||||
```
|
||||
|
||||
The path is the **profile root** (yt-dlp appends the `Default`
|
||||
subdirectory itself). The advantage: cookies are read fresh from the
|
||||
live session each download, so they never go stale.
|
||||
|
||||
> Cookies are session credentials — yt-offline never commits or transmits
|
||||
> `cookies.txt` unprompted, and redacts the cookie path out of any log
|
||||
> line shown in the UI.
|
||||
|
||||
## 2. TLS impersonation (curl_cffi)
|
||||
|
||||
yt-dlp's `--impersonate` makes requests carry a real browser's TLS
|
||||
fingerprint (via `curl_cffi`), so the connection doesn't *look* like a
|
||||
script. The bundled install sets this up automatically and yt-offline
|
||||
picks an impersonation target per platform.
|
||||
|
||||
If impersonation silently does nothing, it's almost always a
|
||||
**yt-dlp ⇄ curl_cffi version mismatch** — which is exactly why the
|
||||
bundled install uses **nightly** yt-dlp (it accepts current curl_cffi;
|
||||
stable lags and disables all impersonate targets when a newer curl_cffi
|
||||
is present). See
|
||||
[Troubleshooting → impersonation](./troubleshooting.md#impersonate-targets-show-unavailable).
|
||||
|
||||
## 3. POT tokens (Proof-of-Origin)
|
||||
|
||||
YouTube increasingly binds a per-video **Proof-of-Origin token** to
|
||||
playback; without one, format URLs come back empty. yt-offline can run
|
||||
[bgutil-pot](https://github.com/jim60105/bgutil-ytdlp-pot-provider-rs), a
|
||||
loopback HTTP server that mints these tokens, and point yt-dlp at it.
|
||||
|
||||
Enable **Settings → Use POT token provider** (requires the bundled
|
||||
yt-dlp; the matching plugin installs into its venv) and click **Install**.
|
||||
|
||||
> **Version-skew footgun:** the yt-dlp plugin must come from the *same
|
||||
> release* as the bgutil-pot server binary — **not** the PyPI package,
|
||||
> which versions independently and silently produces no tokens on a
|
||||
> mismatch. yt-offline's installer handles this by unpacking the
|
||||
> version-matched plugin zip from the server's release.
|
||||
|
||||
## 4. Player-client selection
|
||||
|
||||
YouTube cracks down on different internal "player clients" over time —
|
||||
the `web` client is currently the most captcha-prone, while `tv` and
|
||||
`mweb` are the least. yt-offline no longer forces `web`; it lets yt-dlp
|
||||
pick good defaults. If a specific channel keeps hitting captchas, set a
|
||||
client override (global or per-channel):
|
||||
|
||||
```
|
||||
tv,mweb
|
||||
```
|
||||
|
||||
## 5. Throttling
|
||||
|
||||
A burst of ~30 rapid requests is a classic trip-wire. yt-offline inserts
|
||||
a small jittered pause between videos (a fixed cadence looks robotic; a
|
||||
random one looks human) and, after any rate-limit hit, triples the
|
||||
sleeps for the rest of the batch before recovering.
|
||||
|
||||
---
|
||||
|
||||
**TL;DR for a clean setup:** bundled (nightly) yt-dlp + fresh
|
||||
**logged-in** cookies + POT provider enabled. That combination resolves
|
||||
the vast majority of captcha / "Video unavailable" failures.
|
||||
108
docs/src/architecture.md
Normal file
108
docs/src/architecture.md
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
# Architecture
|
||||
|
||||
For contributors. The repo's `CLAUDE.md` is the terse version of this;
|
||||
read both.
|
||||
|
||||
## Two front-ends, one engine
|
||||
|
||||
`main.rs` dispatches: `--web` → `web::run()` (axum, blocks forever);
|
||||
otherwise `app::App` (eframe/egui desktop GUI). **Both share
|
||||
`downloader::Downloader`**, the single source of truth for the yt-dlp job
|
||||
lifecycle.
|
||||
|
||||
`Downloader` is **not** async — it spawns an OS thread per yt-dlp process,
|
||||
streams stdout/stderr back over an `mpsc` channel into each `Job`'s log
|
||||
buffer, and the caller pumps `Downloader::poll()` regularly (the egui
|
||||
frame loop and a web background task both do). Anything you add to
|
||||
`Downloader` is automatically available to both UIs.
|
||||
|
||||
`poll()` also drives the cross-cutting job machinery: auto-retry of
|
||||
transient failures (cooldown + adaptive throttle), the hang watchdog, and
|
||||
the post-download ffmpeg transcode pass. These work by capturing specs
|
||||
onto the `Job` at `start()` time (`RetrySpec`, `ConvertSpec`) and acting
|
||||
on them when the job changes state.
|
||||
|
||||
## The settings flow (the easy thing to get wrong)
|
||||
|
||||
Almost every configurable feature has the same five-touchpoint shape —
|
||||
miss one and it silently half-works:
|
||||
|
||||
1. `config.rs` — a field/section + its `Default` + the `default_with_dir`
|
||||
constructor.
|
||||
2. `download_options.rs` — an `Option<…>` per-channel override (None =
|
||||
defer to global).
|
||||
3. `downloader.rs` — a resolver merging global config + per-channel
|
||||
override into yt-dlp/ffmpeg args, plus a `pub` field on `Downloader`
|
||||
holding the global default.
|
||||
4. **Both** UIs render the global setting *and* the per-channel override:
|
||||
desktop in `app.rs`, web in `web_ui/index.html` **and** `web.rs`'s
|
||||
`SettingsPayload` (GET reads config, POST writes config + pushes onto
|
||||
the live `Downloader`).
|
||||
5. Seed the `Downloader` field at construction **and** on settings-save,
|
||||
in **both** `app.rs` and `web.rs`.
|
||||
|
||||
`subtitle_defaults`, `youtube_player_clients`, and `convert_defaults` are
|
||||
complete worked examples — grep one end-to-end before adding a setting.
|
||||
|
||||
## Filesystem layout
|
||||
|
||||
`platform::platform_root(channels_root, platform)` =
|
||||
`channels_root.join(dir_name)`. **All** platforms (including YouTube,
|
||||
whose `dir_name` is `channels`) nest under the one configured
|
||||
`backup.directory`. `.source-url` sidecars in each creator folder let
|
||||
re-checks recover the exact URL. Library scanning (`library.rs`) is
|
||||
parallel and consults a `(path, mtime)` SQLite cache to skip re-parsing
|
||||
unchanged `info.json` sidecars.
|
||||
|
||||
## Persistence
|
||||
|
||||
`database.rs` wraps an r2d2 SQLite pool. `Database` is cheaply `Clone`
|
||||
(the pool is an `Arc`), so the parallel scanner takes its own handle.
|
||||
Schema lives in `init_schema()`; new columns are added via idempotent
|
||||
`ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no
|
||||
migration framework). The web UI keeps library/notes snapshots in memory;
|
||||
mutating endpoints mirror DB writes onto those caches and bump a version
|
||||
counter (the `/api/library` ETag) so reads stay consistent without a
|
||||
rescan.
|
||||
|
||||
The long-lived `WebState` mutexes are accessed via
|
||||
`util::LockExt::lock_recover()`, which recovers a poisoned lock instead of
|
||||
cascading one handler's panic into a dead server.
|
||||
|
||||
## Web UI is one embedded file
|
||||
|
||||
`web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked
|
||||
into the binary at compile time — editing it requires a rebuild. Served
|
||||
`Cache-Control: no-store` so binary upgrades don't strand stale tabs.
|
||||
Progress streams over `/ws/progress` (WebSocket) with an HTTP-poll
|
||||
fallback.
|
||||
|
||||
## Anti-bot subsystems
|
||||
|
||||
`ytdlp_bin.rs` manages the optional self-contained venv at
|
||||
`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` + `curl_cffi` +
|
||||
bundled `deno`). `pot_provider.rs` runs `bgutil-pot` for Proof-of-Origin
|
||||
tokens — its yt-dlp plugin must come from the same release as the server
|
||||
binary. `error_class.rs` pattern-matches yt-dlp stderr into actionable
|
||||
classes (order matters in `classify()`: the captcha "Video unavailable"
|
||||
wall is RateLimited, not NotFound).
|
||||
|
||||
## Tests
|
||||
|
||||
- Unit tests are inline `#[cfg(test)]` modules (parsers, resolvers, the
|
||||
error classifier, DB merge logic).
|
||||
- `tests/api.rs` spawns the **real** `--web` binary against a scratch dir
|
||||
and drives the HTTP API with curl — genuine end-to-end coverage of the
|
||||
axum + SQLite + config stack.
|
||||
|
||||
`cargo test` runs both. `.forgejo/workflows/test.yml` runs them on every
|
||||
push.
|
||||
|
||||
## Platform support
|
||||
|
||||
Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only / no-GTK
|
||||
by design — that's why packaging avoids a GTK dependency. Windows/macOS
|
||||
aren't first-class yet: the tray needs a per-OS backend before a clean
|
||||
cross-build. The rest (eframe/wgpu, axum, rusqlite-bundled) already
|
||||
compiles cross-platform, and `ytdlp_bin` already has `cfg!(windows)`
|
||||
branches.
|
||||
74
docs/src/downloading.md
Normal file
74
docs/src/downloading.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Downloading
|
||||
|
||||
## Starting a download
|
||||
|
||||
Paste any supported URL into the download bar (desktop) or the ⬇
|
||||
Downloads modal (web). yt-offline classifies the URL by platform, routes
|
||||
it to the right folder, and starts yt-dlp. A channel/playlist URL pulls
|
||||
the whole thing; a single-video URL pulls just that one.
|
||||
|
||||
**Quality picker:** Best / 1080p / 720p / 480p / 360p, or **Music mode**
|
||||
for audio-only extraction into `music/<artist>/`.
|
||||
|
||||
**Fast mode** stops at the first already-downloaded video (quick routine
|
||||
re-checks). Turn it off for a full gap-filling scan.
|
||||
|
||||
## Per-channel options
|
||||
|
||||
Right-click a channel (or use the ⚙ on its sidebar row) for overrides
|
||||
that apply to scheduled re-checks and the "Check for new videos" action:
|
||||
|
||||
- Quality cap, audio-only, bandwidth cap, min/max file size, date cutoff.
|
||||
- A free-form `--match-filter` (e.g. `duration > 60 & view_count > 100`).
|
||||
- Subtitle overrides, YouTube player-client override, post-download
|
||||
convert mode — each defaulting to the global setting.
|
||||
- **Skip auth check** — silences yt-dlp's "playlists that require
|
||||
authentication" warning for **public** channels (see
|
||||
[Troubleshooting](./troubleshooting.md#the-youtubetab-authentication-warning)).
|
||||
|
||||
Per-channel options ride along in library backup/restore.
|
||||
|
||||
## Subtitles
|
||||
|
||||
Global defaults (Settings → Subtitles) + per-channel overrides control:
|
||||
download on/off, auto-generated captions, embedding into the container,
|
||||
language filter, and format conversion (`srt` is the most
|
||||
Plex/player-compatible). Subtitles are written as sidecar files and
|
||||
optionally embedded.
|
||||
|
||||
## Format conversion
|
||||
|
||||
A post-download ffmpeg pass (Settings → Format conversion, or per
|
||||
channel):
|
||||
|
||||
- **Remux → mp4** — instant container change, no re-encode (device/Plex
|
||||
compatibility).
|
||||
- **Re-encode → H.264 mp4** — shrink large 4K files at a chosen CRF +
|
||||
x264 preset.
|
||||
- **Extract audio** — mp3 / m4a / opus / flac.
|
||||
|
||||
It runs as a distinct transcode job after the download. **Keep original**
|
||||
preserves `<name>.original.<ext>` alongside the converted file; otherwise
|
||||
the source is removed once the convert succeeds.
|
||||
|
||||
## Resilience
|
||||
|
||||
Downloads are hardened against YouTube's flakiness automatically:
|
||||
|
||||
- **Retry + backoff** on connection resets (`--retries 30`, linear
|
||||
retry-sleep).
|
||||
- **Jittered throttle** between videos so a long channel scan doesn't
|
||||
look robotic and trip the captcha wall.
|
||||
- **Auto-retry** of transient (rate-limit / network / captcha) failures
|
||||
after a cooldown, with adaptive slow-down for the rest of the batch.
|
||||
- **Hang watchdog** kills a job that produces no output for 5 minutes (a
|
||||
wedged request) and re-queues it.
|
||||
|
||||
Failures are classified and shown with a one-line suggested fix — see
|
||||
[Troubleshooting](./troubleshooting.md).
|
||||
|
||||
## Scheduler
|
||||
|
||||
Enable it (Settings → Auto-check channels) to re-check every channel for
|
||||
new uploads on an interval. Each channel uses its own stored options.
|
||||
There's also a per-folder "Check all" action.
|
||||
91
docs/src/first-run.md
Normal file
91
docs/src/first-run.md
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
# First run & configuration
|
||||
|
||||
## The config file
|
||||
|
||||
yt-offline reads `config.toml` **from its working directory** (the
|
||||
directory you launch it from), not a fixed path. The same goes for
|
||||
`cookies.txt`. Everything in `config.toml` is also editable in Settings;
|
||||
edits there are written back to the file.
|
||||
|
||||
```toml
|
||||
[backup]
|
||||
directory = "/path/to/library" # the umbrella dir; all platforms nest under it
|
||||
max_concurrent = 3 # parallel yt-dlp processes
|
||||
use_bundled_ytdlp = false # true = use the managed venv (see below)
|
||||
use_pot_provider = false # YouTube Proof-of-Origin tokens (see Anti-bot)
|
||||
youtube_player_clients = "" # e.g. "tv,mweb" to route around captchas
|
||||
|
||||
[player]
|
||||
command = "mpv" # any executable taking a file path as its last arg
|
||||
browser = "firefox" # cookie source when no cookies.txt is set (see Anti-bot)
|
||||
|
||||
[ui]
|
||||
theme = "dark" # dark | light | dracula | trans | emo-nocturnal | emo-coffin | emo-scene-queen
|
||||
ui_scale = 1.0 # global zoom for the whole desktop UI
|
||||
|
||||
[scheduler]
|
||||
enabled = false
|
||||
interval_hours = 24 # auto re-check every channel for new uploads
|
||||
|
||||
[web]
|
||||
port = 8080
|
||||
bind = "127.0.0.1" # 127.0.0.1 | 0.0.0.0 | a Tailscale/LAN address
|
||||
transcode = false # MKV → MP4 on the fly for browsers that can't decode MKV
|
||||
|
||||
[subtitles]
|
||||
enabled = true
|
||||
auto_generated = true # include machine captions
|
||||
embed = false # also embed into the container
|
||||
format = "" # "" = native; "srt" for Plex compatibility
|
||||
langs = "" # "" = all; "en" or "en,ja" to filter
|
||||
|
||||
[convert]
|
||||
mode = "" # "" / "remux-mp4" / "h264-mp4" / "audio"
|
||||
crf = 23 # for h264-mp4 (lower = bigger/better)
|
||||
preset = "medium"
|
||||
audio_format = "mp3" # for audio mode
|
||||
keep_original = false # keep <name>.original.<ext> after converting
|
||||
|
||||
[plex]
|
||||
library_path = "/path/to/plex/TV/youtube" # leave unset to disable
|
||||
```
|
||||
|
||||
## The library layout
|
||||
|
||||
Everything nests under the one `backup.directory`:
|
||||
|
||||
```text
|
||||
<backup.directory>/
|
||||
channels/ ← YouTube creators
|
||||
tiktok/ twitch/ vimeo/ bandcamp/ soundcloud/ odysee/ other/
|
||||
music/ ← audio-only "Music mode" downloads, by artist
|
||||
archive.txt ← yt-dlp's global download archive
|
||||
cookies.txt ← optional, if you set one
|
||||
yt-offline.db ← watched/positions/flags/folders/notes/cache + password hash
|
||||
```
|
||||
|
||||
Each creator folder gets a hidden `.source-url` sidecar so re-checks
|
||||
always know the exact URL to refresh from.
|
||||
|
||||
## Bundled vs system yt-dlp
|
||||
|
||||
In **Settings → yt-dlp binary** you choose:
|
||||
|
||||
- **System** — uses whatever `yt-dlp` is on your `PATH`.
|
||||
- **Bundled** — click **Install** and yt-offline builds a self-contained
|
||||
venv at `~/.local/share/yt-offline/`: nightly `yt-dlp[default]` +
|
||||
`curl_cffi` (TLS impersonation) + a bundled `deno` (player-JS). The
|
||||
same button updates it later.
|
||||
|
||||
The bundled path is recommended — it installs **nightly** yt-dlp, which
|
||||
keeps pace with YouTube's frequent anti-bot changes (stable lags). It's
|
||||
also required for the [POT token provider](./anti-bot.md#3-pot-tokens-proof-of-origin).
|
||||
|
||||
## The two front-ends
|
||||
|
||||
- `yt-offline` — desktop GUI (eframe/egui).
|
||||
- `yt-offline --web [PORT]` — headless web server. Bind to `127.0.0.1`
|
||||
(default) for localhost-only, a Tailscale address for your tailnet, or
|
||||
`0.0.0.0` for the LAN. **Set a password** (Settings) before exposing it
|
||||
beyond localhost — the UI and all `/api` routes are then gated behind an
|
||||
Argon2-hashed, rate-limited login.
|
||||
73
docs/src/installation.md
Normal file
73
docs/src/installation.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
# Installation
|
||||
|
||||
yt-offline is a single Rust binary. You can install a prebuilt package,
|
||||
build from source, or grab the AppImage.
|
||||
|
||||
## Runtime dependencies
|
||||
|
||||
Whichever way you install, these are invoked as subprocesses at runtime:
|
||||
|
||||
- **yt-dlp** — the download engine. You can use the system one *or* let
|
||||
yt-offline manage a bundled copy (see [First run](./first-run.md)).
|
||||
- **ffmpeg** — muxing, format conversion, on-the-fly transcode for the
|
||||
web player.
|
||||
- **mpv** — the default desktop player (any player taking a file path
|
||||
works; set it in Settings).
|
||||
- **xdg-utils** — `xdg-open` for "Show in file manager".
|
||||
|
||||
## Prebuilt packages (Linux)
|
||||
|
||||
Releases attach `.deb`, `.rpm`, and `.AppImage` artifacts.
|
||||
|
||||
```sh
|
||||
# Debian / Ubuntu / Mint
|
||||
sudo apt install ./yt-offline_*_amd64.deb
|
||||
|
||||
# Fedora / RHEL / openSUSE (ffmpeg via RPM Fusion)
|
||||
sudo dnf install ./yt-offline-*.x86_64.rpm
|
||||
|
||||
# Any Linux — AppImage
|
||||
chmod +x yt-offline-*-x86_64.AppImage
|
||||
./yt-offline-*-x86_64.AppImage
|
||||
```
|
||||
|
||||
## Arch / CachyOS / Manjaro
|
||||
|
||||
A `PKGBUILD` ships in the repo root. Build it from a **clean** directory:
|
||||
|
||||
```sh
|
||||
mkdir build && cd build
|
||||
cp /path/to/repo/PKGBUILD .
|
||||
makepkg -si
|
||||
```
|
||||
|
||||
For repeated builds after pulling new commits, always pass `-C`
|
||||
(cleanbuild) so makepkg re-checks out the latest source instead of
|
||||
reusing a stale cached clone.
|
||||
|
||||
## From source
|
||||
|
||||
```sh
|
||||
# Debian/Ubuntu build deps
|
||||
sudo apt install build-essential pkg-config curl git python3-venv \
|
||||
libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev \
|
||||
libxkbcommon-dev libssl-dev
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
source "$HOME/.cargo/env"
|
||||
|
||||
git clone https://codeberg.org/anassaeneroi/yt-offline
|
||||
cd yt-offline
|
||||
cargo build --release
|
||||
./target/release/yt-offline # desktop GUI
|
||||
./target/release/yt-offline --web 8080 # headless web server
|
||||
```
|
||||
|
||||
`python3-venv` is only needed for the bundled-yt-dlp install path; skip
|
||||
it if you'll always use system yt-dlp.
|
||||
|
||||
## Windows / macOS
|
||||
|
||||
Not first-class yet. The Linux-only system tray (`ksni`) and file dialog
|
||||
(`rfd` xdg-portal) need per-OS backends before a clean cross-build; the
|
||||
rest of the stack already compiles. See the
|
||||
[architecture notes](./architecture.md#platform-support).
|
||||
40
docs/src/introduction.md
Normal file
40
docs/src/introduction.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# yt-offline
|
||||
|
||||
A self-hosted media archive for YouTube and friends. Paste any URL, it
|
||||
routes the download to the right folder by source, tracks what you've
|
||||
watched, and plays everything back from a desktop GUI **or** a browser —
|
||||
even offline, or after the source video is taken down.
|
||||
|
||||
Built on [yt-dlp](https://github.com/yt-dlp/yt-dlp); written in Rust as a
|
||||
single binary that is **both** a desktop app and a headless web server.
|
||||
|
||||
## What it backs up
|
||||
|
||||
| Platform | Channels | Playlists | Single videos |
|
||||
|---|---|---|---|
|
||||
| YouTube | ✅ | ✅ | ✅ |
|
||||
| TikTok | ✅ | — | ✅ |
|
||||
| Twitch (VODs + clips) | ✅ | — | ✅ |
|
||||
| Vimeo | ✅ | ✅ | ✅ |
|
||||
| Bandcamp | ✅ (artist) | ✅ (albums) | ✅ (tracks) |
|
||||
| SoundCloud | ✅ | ✅ (sets) | ✅ |
|
||||
| Odysee | ✅ | — | ✅ |
|
||||
| Anything else yt-dlp accepts | `other/` | `other/` | `other/` |
|
||||
|
||||
## Why it exists
|
||||
|
||||
Tartube is the mature open-source yt-dlp GUI and the benchmark in this
|
||||
space. yt-offline matches its feature set while adding things Tartube
|
||||
doesn't have: a real web UI reachable from any device, a single-binary
|
||||
distribution with a bundled toolchain, a modern security model
|
||||
(password-gated UI, Argon2, rate-limited login), a built-in anti-bot
|
||||
stack (TLS impersonation + Proof-of-Origin tokens), and one-click
|
||||
post-download format conversion.
|
||||
|
||||
## How to read these docs
|
||||
|
||||
- New here? **[Installation](./installation.md)** →
|
||||
**[First run](./first-run.md)** → **[Downloading](./downloading.md)**.
|
||||
- Hitting captchas or "Video unavailable"? Go straight to
|
||||
**[Anti-bot](./anti-bot.md)** and **[Troubleshooting](./troubleshooting.md)**.
|
||||
- Hacking on it? **[Architecture](./architecture.md)**.
|
||||
39
docs/src/packaging.md
Normal file
39
docs/src/packaging.md
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
# Packaging
|
||||
|
||||
Build distributable Linux packages with one script:
|
||||
|
||||
```sh
|
||||
scripts/package.sh all # .deb + .rpm + .AppImage → dist/
|
||||
scripts/package.sh deb # just the .deb
|
||||
scripts/package.sh rpm
|
||||
scripts/package.sh appimage
|
||||
```
|
||||
|
||||
It builds the release binary once and reuses it for every format,
|
||||
installing `cargo-deb` / `cargo-generate-rpm` on demand and downloading
|
||||
`appimagetool` to `dist/tools/` on first AppImage build. Per-format
|
||||
failures are isolated and summarized at the end. Output (gitignored)
|
||||
lands in `dist/`.
|
||||
|
||||
## Formats
|
||||
|
||||
- **`.deb`** — built by [cargo-deb](https://github.com/kornelski/cargo-deb)
|
||||
from `[package.metadata.deb]` in `Cargo.toml`.
|
||||
- **`.rpm`** — built by
|
||||
[cargo-generate-rpm](https://github.com/cat-in-136/cargo-generate-rpm)
|
||||
from `[package.metadata.generate-rpm]`. (`ffmpeg` on Fedora is in
|
||||
RPM Fusion.)
|
||||
- **AppImage** — a hand-rolled AppDir + appimagetool. Bundles the GUI
|
||||
binary's shared-library closure only; `yt-dlp`/`ffmpeg`/`mpv` stay host
|
||||
PATH deps, same as the package declarations.
|
||||
- **Arch** — use the repo's `PKGBUILD` (not this script); run `makepkg`
|
||||
from a clean directory.
|
||||
|
||||
## CI
|
||||
|
||||
`.forgejo/workflows/release.yml` runs `scripts/package.sh all` on every
|
||||
pushed `v*` tag and attaches the artifacts to the Codeberg release.
|
||||
`.forgejo/workflows/test.yml` runs the full test suite on every push.
|
||||
|
||||
The repo's [`docs/PACKAGING.md`](https://codeberg.org/anassaeneroi/yt-offline/src/branch/main/docs/PACKAGING.md)
|
||||
has the per-distro install commands and the Windows/macOS status in full.
|
||||
113
docs/src/troubleshooting.md
Normal file
113
docs/src/troubleshooting.md
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
# Troubleshooting
|
||||
|
||||
yt-offline classifies failed downloads into one of nine classes and shows
|
||||
a one-line suggested fix next to the failed job. This page expands on the
|
||||
most common ones, plus a few non-download issues.
|
||||
|
||||
## "Video unavailable. YouTube is requiring a captcha challenge"
|
||||
|
||||
**Class:** rate-limited. **Not** a removed video — it's the bot-detection
|
||||
wall. In order of effectiveness:
|
||||
|
||||
1. **Use fresh, logged-in cookies.** Anonymous cookies are the usual
|
||||
culprit — see [Anti-bot → cookies](./anti-bot.md#1-be-logged-in-cookies).
|
||||
Settings → Cookies warns when your jar is anonymous or expired.
|
||||
2. **Switch to bundled (nightly) yt-dlp** if you're on system stable.
|
||||
3. **Enable the POT token provider.**
|
||||
4. **Try a player-client override** of `tv,mweb` for that channel.
|
||||
5. If it's a one-off, just wait — yt-offline auto-retries transient
|
||||
rate-limits after a cooldown.
|
||||
|
||||
## Impersonate targets show "(unavailable)"
|
||||
|
||||
`yt-dlp --list-impersonate-targets` lists every target as `(unavailable)`
|
||||
even though `curl_cffi` is installed.
|
||||
|
||||
**Cause:** a yt-dlp ⇄ curl_cffi version gate. Stable yt-dlp caps the
|
||||
curl_cffi version it accepts; a newer curl_cffi makes it disable *all*
|
||||
impersonate targets.
|
||||
|
||||
**Fix:** use the **bundled** yt-dlp (it installs nightly via `--pre`,
|
||||
which accepts current curl_cffi), or pin curl_cffi to a compatible
|
||||
version in your own environment.
|
||||
|
||||
## POT provider produces no tokens
|
||||
|
||||
You enabled the POT provider and installed it, but downloads still fail
|
||||
as if no token was generated. yt-dlp logs a *"plugin and HTTP server
|
||||
major versions are mismatched"* warning.
|
||||
|
||||
**Cause:** the yt-dlp plugin came from PyPI (Brainicism's package, which
|
||||
versions independently) instead of the jim60105 Rust server's release.
|
||||
|
||||
**Fix:** re-run the POT **Install/Update** button — yt-offline installs
|
||||
the version-matched plugin zip from the same release as the server
|
||||
binary. Don't `pip install bgutil-ytdlp-pot-provider` yourself.
|
||||
|
||||
## The youtubetab authentication warning
|
||||
|
||||
```
|
||||
ERROR: [youtube:tab] @Channel: Playlists that require authentication may
|
||||
not extract correctly without a successful webpage download...
|
||||
```
|
||||
|
||||
Despite the `ERROR:` prefix this is a soft warning, usually a symptom of
|
||||
the bot-detection issues above (YouTube served an incomplete page). It
|
||||
does **not** change which videos are found.
|
||||
|
||||
**Fix:** for **public** channels, enable **Skip auth check** in that
|
||||
channel's options (adds `--extractor-args youtubetab:skip=authcheck`) to
|
||||
silence it. Leave it **off** for members-only/private channels you
|
||||
archive with cookies — there the warning is a real "your cookies may not
|
||||
be working" signal.
|
||||
|
||||
## "Sign in to confirm you're not a bot"
|
||||
|
||||
Same family as the captcha wall. Fix with fresh logged-in cookies + POT;
|
||||
see [Anti-bot](./anti-bot.md).
|
||||
|
||||
## Downloads stall forever
|
||||
|
||||
A job sits running with no progress. yt-offline's **hang watchdog**
|
||||
auto-kills any job silent for 5 minutes and re-queues it, so this should
|
||||
self-heal. If it recurs on a specific URL, it's usually a server-side
|
||||
issue with that source; check the job log in the Downloads panel.
|
||||
|
||||
## Disk fills up / downloads fail with ENOSPC
|
||||
|
||||
yt-offline runs a **disk-full preflight** and refuses to start a download
|
||||
when the target filesystem has less than ~500 MB free, surfacing it as a
|
||||
clear "disk full" failure rather than a half-written file. Free space and
|
||||
retry.
|
||||
|
||||
## A whole platform folder shows up as one "channel"
|
||||
|
||||
If you see `bandcamp`, `tiktok`, or `channels` listed as a single channel
|
||||
in the sidebar, your library directory predates the current layout. All
|
||||
platforms must **nest under** the one `backup.directory`
|
||||
(`<dir>/channels/`, `<dir>/tiktok/`, …). Move stray creator folders into
|
||||
their platform's subdir; see [First run → library layout](./first-run.md#the-library-layout).
|
||||
|
||||
## The desktop window crashes on maximize
|
||||
|
||||
Older builds crashed with a Glutin `EGL_BAD_ALLOC` on NVIDIA + Wayland
|
||||
when maximized. Current builds use the **wgpu (Vulkan)** renderer, which
|
||||
handles the resize cleanly. Make sure you have a working Vulkan driver
|
||||
(`vulkan-icd-loader` + your GPU's Vulkan package), which any desktop with
|
||||
working graphics already has.
|
||||
|
||||
## The web UI looks like an old version after an upgrade
|
||||
|
||||
The SPA is served `Cache-Control: no-store`, so a hard reload
|
||||
(Ctrl+Shift+R) always picks up the new binary's UI. If you upgraded the
|
||||
binary, also **restart the running `--web` process** — the HTML is baked
|
||||
into the binary at compile time, so the old process keeps serving the old
|
||||
UI until restarted.
|
||||
|
||||
## Where to look next
|
||||
|
||||
- **The job log** — every download/transcode job keeps its full yt-dlp /
|
||||
ffmpeg output in the Downloads panel (expand the job).
|
||||
- **`yt-offline.crash.log`** — next to your `yt-offline.db`. A panic in
|
||||
any thread (UI, web worker, download) is appended here with a
|
||||
timestamp, so it survives a GUI launched without a terminal.
|
||||
Loading…
Add table
Add a link
Reference in a new issue