luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add bundled yt-dlp, music library, Plex metadata, sort by date; security + perf hardening

Browse source 
Bug fixes
- Desktop auto-rescan: snapshot was_running before check_notifications
  updates prev_job_states, so the post-download rescan actually fires.
- web::run shutdown: replace `let _ = tx;` (which dropped the sender
  immediately and short-circuited the shutdown select!) with
  std::mem::forget(tx); add an idle fallback so the `-> !` is honored.
- /api/preview now resolves the bundled yt-dlp path and extends PATH
  for deno discovery, matching the rest of the downloader.
- Scheduler interval clamped to >=1h on read so a manually-edited
  config.toml with 0 hours can't trigger every tick.
- maintenance::is_within canonicalizes the parent + filename when the
  target is missing; remove_files treats NotFound as success.

Security
- Session tokens stored with issued-at Instant and pruned past 30 day
  TTL on every touch (was: unbounded HashSet).
- Login rate-limited per source IP: 5 failures → 60s lockout (429).
- Secure cookie flag added when X-Forwarded-Proto: https is present.
- 4 MiB body size cap via DefaultBodyLimit.
- Security headers middleware: CSP, X-Content-Type-Options,
  X-Frame-Options, Referrer-Policy.
- JS safeUrl() defangs javascript:/vbscript:/data:text URLs before
  interpolation into src= attributes.
- Bundled yt-dlp install verifies SHA-256 against the release's
  SHA2-256SUMS; deno hash printed for visual inspection. Install also
  reports byte counts every 3s so users see live progress.
- yt-offline.db chmod 0600 at open time on Unix.

New features
- Bundled yt-dlp + deno mode: settings toggle (System/Bundled) plus
  one-click Install/Update from settings. Ensures executable bit on
  every launch in case the install chmod was missed.
- Download queue with configurable max_concurrent (1–10); pending
  jobs surfaced in both UIs.
- Music download mode: --extract-audio into music/<artist>/, separate
  sidebar entry, /api/music + /music-files/ endpoints, inline <audio>.
- Download quality selector: Best / 1080p / 720p / 480p / 360p.
- Sort videos by Newest / Oldest (upload_date from info.json with
  release_date fallback).
- Plex metadata: per-episode .nfo (title, season/episode, aired,
  runtime, plot from .description), <stem>-thumb.jpg symlink, and
  show-level tvshow.nfo.
- yt-dlp retry/throttle defaults: --retries 30 --fragment-retries 30
  --retry-sleep linear=1:30:2 --sleep-requests 1 to ride out the
  "Connection reset by peer" failures.

Performance
- password_required cached in AtomicBool; refreshed only on password
  change. Eliminates a SQLite query per static-file fetch.
- Job::log → VecDeque for O(1) front-pop instead of O(n) drain.
- Library scan parallelized across cores via stdlib parallel_map.
- Web UI poll: 600ms while active, 5s when idle, 2s after errors.
- Music scan now recursive (Artist/Album/Track.opus).
- percent_encode_segment uses write! to avoid per-byte String allocs.

Code quality
- 27 unit tests covering parse_progress, parse_stem, extract_after,
  detect_url_kind, srt_to_vtt, count_cookies, percent_encode_segment,
  bind_mode_of, hash_password, lang_label, is_within, sanitize,
  year_of, aired_date, xml_escape.
- Unified library::find_video + Channel::all_videos() iterator,
  replacing three duplicated linear searches.
- Downloader::browser wired to --cookies-from-browser as a fallback
  when no cookies.txt is present.
- mpv detection now matches the binary basename, not substring.
- Settings modal scroll fix; web settings expose all new fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-23 04:54:26 -07:00
parent 1d72069913
commit 59996735f3
10 changed files with 2300 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

310
src/downloader.rs
View file

@ -23,12 +23,53 @@
//! | `--break-on-existing` | Stop when the archive file records the video as already downloaded |
//! | `--download-archive archive.txt` | Record downloaded IDs to avoid re-downloading |
use std::collections::VecDeque;
use std::io::{BufRead, BufReader};
use std::path::PathBuf;
use std::process::{Command, Stdio};
use std::sync::mpsc::{channel, Receiver};
use std::thread;
use crate::ytdlp_bin;
/// Video quality level passed as a `-f` format selector to yt-dlp.
#[derive(Clone, Copy, PartialEq, Eq, Default)]
pub enum DownloadQuality {
/// No `-f` flag — yt-dlp picks the best available streams (default).
#[default]
Best,
Res1080,
Res720,
Res480,
Res360,
}
impl DownloadQuality {
pub fn format_spec(self) -> Option<&'static str> {
match self {
Self::Best => None,
Self::Res1080 => Some("bestvideo[height<=1080]+bestaudio/best[height<=1080]"),
Self::Res720 => Some("bestvideo[height<=720]+bestaudio/best[height<=720]"),
Self::Res480 => Some("bestvideo[height<=480]+bestaudio/best[height<=480]"),
Self::Res360 => Some("bestvideo[height<=360]+bestaudio/best[height<=360]"),
}
}
pub fn label(self) -> &'static str {
match self {
Self::Best => "Best",
Self::Res1080 => "1080p",
Self::Res720 => "720p",
Self::Res480 => "480p",
Self::Res360 => "360p",
}
}
pub fn all() -> &'static [DownloadQuality] {
&[Self::Best, Self::Res1080, Self::Res720, Self::Res480, Self::Res360]
}
}
/// Describes the kind of YouTube URL being downloaded, which determines the
/// output path template passed to yt-dlp.
pub enum UrlKind {
@ -96,6 +137,9 @@ enum Msg {
Finished(bool),
}
/// Maximum lines retained in [`Job::log`] before old lines are evicted.
const JOB_LOG_CAP: usize = 800;
/// A single yt-dlp invocation tracked by the downloader.
pub struct Job {
pub url: String,
@ -104,8 +148,8 @@ pub struct Job {
pub state: JobState,
/// Download progress as a fraction in `[0.0, 1.0]`.
pub progress: f32,
/// Rolling log buffer — capped at 800 lines to avoid unbounded growth.
pub log: Vec<String>,
/// Rolling log buffer — capped at [`JOB_LOG_CAP`] lines via O(1) front-pop.
pub log: VecDeque<String>,
rx: Receiver<Msg>,
}
@ -114,10 +158,9 @@ impl Job {
while let Ok(msg) = self.rx.try_recv() {
match msg {
Msg::Line(line) => {
self.log.push(line);
if self.log.len() > 800 {
let cut = self.log.len() - 800;
self.log.drain(0..cut);
self.log.push_back(line);
while self.log.len() > JOB_LOG_CAP {
self.log.pop_front();
}
}
Msg::Progress(p) => self.progress = p,
@ -129,18 +172,108 @@ impl Job {
}
}
/// Manages all active and recently completed yt-dlp download jobs.
/// A download waiting to start once a concurrency slot opens.
struct PendingJob {
cmd: Command,
url: String,
label: String,
}
/// Manages all active, queued, and recently completed yt-dlp download jobs.
pub struct Downloader {
pub jobs: Vec<Job>,
pending: VecDeque<PendingJob>,
pub channels_root: PathBuf,
/// Browser name passed to `--cookies-from-browser` (unused) — cookie file
/// is currently always `cookies.txt`.
/// Browser name passed to `--cookies-from-browser` *when no cookies.txt
/// exists in the working directory*. Set to `"none"` to skip the fallback
/// entirely. Pasted/imported cookies.txt always takes precedence.
pub browser: String,
/// Maximum number of simultaneous yt-dlp processes. 0 = unlimited.
pub max_concurrent: usize,
/// If true, invoke the bundled yt-dlp under [`ytdlp_bin::bundled_dir`]
/// instead of the system PATH yt-dlp.
pub use_bundled_ytdlp: bool,
}
impl Downloader {
pub fn new(channels_root: PathBuf, browser: String) -> Self {
Self { jobs: Vec::new(), channels_root, browser }
pub fn new(channels_root: PathBuf, browser: String, max_concurrent: usize, use_bundled_ytdlp: bool) -> Self {
Self {
jobs: Vec::new(),
pending: VecDeque::new(),
channels_root,
browser,
max_concurrent,
use_bundled_ytdlp,
}
}
/// Append the cookie-source flags. Prefers `cookies.txt` in the working
/// directory (set up via the cookies UI), falling back to
/// `--cookies-from-browser <browser>` when no cookies.txt exists and the
/// user has chosen a browser (anything other than `"none"`).
fn apply_cookie_flags(&self, cmd: &mut Command) {
let cookies_txt = std::path::Path::new("cookies.txt");
if cookies_txt.exists() {
cmd.arg("--cookies").arg("cookies.txt");
} else if !self.browser.is_empty() && self.browser != "none" {
cmd.arg("--cookies-from-browser").arg(&self.browser);
}
}
/// Append the retry and throttling flags applied to every yt-dlp invocation.
///
/// YouTube occasionally resets connections mid-transfer; with default
/// settings yt-dlp gives up after 10 quick retries. We bump the retry
/// count and add a linear backoff so transient resets self-heal. The
/// sleep flags throttle per-IP request rate slightly so we don't trip
/// YouTube's rate limiter when many channels are being checked at once.
fn apply_retry_flags(cmd: &mut Command) {
cmd.arg("--retries").arg("30")
.arg("--fragment-retries").arg("30")
.arg("--retry-sleep").arg("linear=1:30:2")
.arg("--sleep-requests").arg("1");
}
/// Build a fresh `Command` invoking the currently configured yt-dlp binary.
///
/// In bundled mode, defensively re-applies the executable bit on every
/// binary inside the bundled bin dir. Without this, downloads can fail
/// with EACCES if a previous install left the file un-executable (e.g.
/// because the chmod step of the install script never ran).
fn ytdlp_cmd(&self) -> Command {
let path = ytdlp_bin::ytdlp_invocation(self.use_bundled_ytdlp);
if self.use_bundled_ytdlp {
ytdlp_bin::ensure_bundled_executable();
}
Command::new(path)
}
/// Number of jobs waiting in the queue (not yet started).
pub fn pending_count(&self) -> usize {
self.pending.len()
}
/// Labels and URLs of all queued (not yet started) jobs, in queue order.
pub fn pending_snapshots(&self) -> Vec<(String, String)> {
self.pending.iter().map(|p| (p.label.clone(), p.url.clone())).collect()
}
/// Promote pending jobs into running slots while capacity allows.
fn promote_queued(&mut self) {
while !self.pending.is_empty() {
if self.max_concurrent > 0 {
let running = self.jobs.iter().filter(|j| j.state == JobState::Running).count();
if running >= self.max_concurrent { break; }
}
let p = self.pending.pop_front().unwrap();
self.spawn_job(p.cmd, p.url, p.label);
}
}
/// Push a command into the pending queue (or start immediately if a slot is free).
fn enqueue(&mut self, cmd: Command, url: String, label: String) {
self.pending.push_back(PendingJob { cmd, url, label });
self.promote_queued();
}
/// Spawn a yt-dlp process for `url` and track it as a new [`Job`].
@ -153,7 +286,7 @@ impl Downloader {
/// video — fast for routine channel checks. When `full_scan` is true the
/// flag is omitted so every video is checked individually against the
/// download archive; slower, but correctly fills gaps in the history.
pub fn start(&mut self, url: String, kind: &UrlKind, full_scan: bool) {
pub fn start(&mut self, url: String, kind: &UrlKind, full_scan: bool, quality: DownloadQuality) {
let archive_path = self.channels_root.join("archive.txt");
let (out_arg, label) = match kind {
@ -175,12 +308,10 @@ impl Downloader {
),
};
let mut cmd = Command::new("yt-dlp");
cmd.arg("--newline")
.arg("--no-color")
.arg("--cookies")
.arg("cookies.txt")
.arg("--write-subs")
let mut cmd = self.ytdlp_cmd();
cmd.arg("--newline").arg("--no-color");
self.apply_cookie_flags(&mut cmd);
cmd.arg("--write-subs")
.arg("--write-auto-subs")
.arg("--write-thumbnail")
.arg("--write-description")
@ -200,6 +331,9 @@ impl Downloader {
.arg("--extractor-args")
.arg("youtube:player_client=web")
.arg("--progress");
if let Some(fmt) = quality.format_spec() {
cmd.arg("-f").arg(fmt);
}
if !full_scan {
cmd.arg("--break-on-existing");
}
@ -210,8 +344,9 @@ impl Downloader {
.arg("-o")
.arg(&out_arg)
.arg(&url);
Self::apply_retry_flags(&mut cmd);
self.spawn_job(cmd, url, label);
self.enqueue(cmd, url, label);
}
/// Re-fetch missing sidecar assets (thumbnail, info.json, description,
@ -227,13 +362,10 @@ impl Downloader {
let url = format!("https://www.youtube.com/watch?v={video_id}");
let label = format!("repair {stem}");
let mut cmd = Command::new("yt-dlp");
cmd.arg("--newline")
.arg("--no-color")
.arg("--skip-download")
.arg("--cookies")
.arg("cookies.txt")
.arg("--write-thumbnail")
let mut cmd = self.ytdlp_cmd();
cmd.arg("--newline").arg("--no-color").arg("--skip-download");
self.apply_cookie_flags(&mut cmd);
cmd.arg("--write-thumbnail")
.arg("--write-info-json")
.arg("--write-description")
.arg("--write-subs")
@ -245,8 +377,62 @@ impl Downloader {
.arg("-o")
.arg(&out_arg)
.arg(&url);
Self::apply_retry_flags(&mut cmd);
self.spawn_job(cmd, url, label);
self.enqueue(cmd, url, label);
}
/// Path to the music download directory (sibling of `channels_root`).
pub fn music_root(&self) -> PathBuf {
self.channels_root.with_file_name("music")
}
/// Download `url` as audio-only, storing tracks in `music/<artist>/`.
pub fn start_music(&mut self, url: String) {
let music_root = self.music_root();
let _ = std::fs::create_dir_all(&music_root);
let archive_path = self.channels_root.join("archive.txt");
let out_arg = format!(
"{}/%(artist,channel|Unknown)s/%(title)s [%(id)s].%(ext)s",
music_root.display()
);
let label = "music".to_string();
let mut cmd = self.ytdlp_cmd();
cmd.arg("--newline").arg("--no-color");
self.apply_cookie_flags(&mut cmd);
cmd.arg("--extract-audio")
.arg("--audio-format")
.arg("best")
.arg("--audio-quality")
.arg("0")
.arg("--write-thumbnail")
.arg("--write-info-json")
.arg("--embed-metadata")
.arg("--xattrs")
.arg("--extractor-args")
.arg("youtube:player_client=web")
.arg("--impersonate")
.arg("Chrome-146:Macos-26")
.arg("--progress")
.arg("--download-archive")
.arg(archive_path.display().to_string())
.arg("-o")
.arg(&out_arg)
.arg(&url);
Self::apply_retry_flags(&mut cmd);
self.enqueue(cmd, url, label);
}
/// Enqueue a job that downloads (or updates) the bundled yt-dlp + deno
/// binaries into [`ytdlp_bin::bundled_dir`]. Streams the curl/unzip output
/// into a normal [`Job`] entry so the user sees progress in the UI.
pub fn start_ytdlp_update(&mut self) {
let cmd = ytdlp_bin::install_command();
let url = "https://github.com/yt-dlp/yt-dlp/releases/latest".to_string();
let label = "update bundled yt-dlp + deno".to_string();
self.enqueue(cmd, url, label);
}
/// Spawn `cmd` on a background thread, streaming its output into a new [`Job`].
@ -256,6 +442,19 @@ impl Downloader {
.stdout(Stdio::piped())
.stderr(Stdio::piped());
// Prepend the bundled bin dir to PATH so yt-dlp can locate the bundled
// `deno` for JavaScript signature deciphering. Harmless when the dir
// doesn't exist or bundled mode is disabled.
let bundled_dir = ytdlp_bin::bundled_dir();
if bundled_dir.exists() {
let sep = if cfg!(windows) { ";" } else { ":" };
let new_path = match std::env::var_os("PATH") {
Some(existing) => format!("{}{}{}", bundled_dir.display(), sep, existing.to_string_lossy()),
None => bundled_dir.display().to_string(),
};
cmd.env("PATH", new_path);
}
thread::spawn(move || {
let mut child = match cmd.spawn() {
Ok(child) => child,
@ -310,16 +509,18 @@ impl Downloader {
let _ = tx.send(Msg::Finished(ok));
});
self.jobs.push(Job { url, label, state: JobState::Running, progress: 0.0, log: Vec::new(), rx });
self.jobs.push(Job { url, label, state: JobState::Running, progress: 0.0, log: VecDeque::new(), rx });
}
/// Drain pending messages from all job threads into their log buffers.
/// Drain pending messages from all job threads and promote queued jobs.
///
/// Call this regularly from the UI event loop to pick up progress updates.
pub fn poll(&mut self) {
for job in &mut self.jobs {
job.drain();
}
// Re-check after draining: finished jobs free slots for queued ones.
self.promote_queued();
}
pub fn any_running(&self) -> bool {
@ -349,3 +550,54 @@ fn parse_progress(line: &str) -> Option<f32> {
let value: f32 = rest[..pct_end].trim().parse().ok()?;
Some((value / 100.0).clamp(0.0, 1.0))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_progress_typical() {
let p = parse_progress("[download] 42.7% of 100MiB at 5MiB/s ETA 00:10").unwrap();
assert!((p - 0.427).abs() < 1e-4);
}
#[test]
fn parse_progress_clamps_to_one() {
let p = parse_progress("[download] 150% of garbage").unwrap();
assert_eq!(p, 1.0);
}
#[test]
fn parse_progress_rejects_non_download_lines() {
assert!(parse_progress("[info] Writing thumbnail").is_none());
assert!(parse_progress("").is_none());
}
#[test]
fn extract_after_strips_at_separator() {
assert_eq!(extract_after("https://youtube.com/@handle/videos", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle?x=1", "/@"), Some("handle"));
assert_eq!(extract_after("nothing-here", "/@"), None);
}
#[test]
fn check_url_for_folder_picks_channel_form_for_ids() {
let url = check_url_for_folder("UC1234567890123456789012");
assert_eq!(url, "https://www.youtube.com/channel/UC1234567890123456789012");
}
#[test]
fn check_url_for_folder_picks_handle_form_otherwise() {
let url = check_url_for_folder("LinusTechTips");
assert_eq!(url, "https://www.youtube.com/@LinusTechTips");
}
#[test]
fn detect_url_kind_classifies_paths() {
assert!(matches!(detect_url_kind("https://www.youtube.com/playlist?list=PL1"), UrlKind::Playlist));
assert!(matches!(detect_url_kind("https://www.youtube.com/watch?v=abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://youtu.be/abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://www.youtube.com/@handle"), UrlKind::Channel { .. }));
}
}