luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Add bundled yt-dlp, music library, Plex metadata, sort by date; security + perf hardening

Browse source 
Bug fixes
- Desktop auto-rescan: snapshot was_running before check_notifications
  updates prev_job_states, so the post-download rescan actually fires.
- web::run shutdown: replace `let _ = tx;` (which dropped the sender
  immediately and short-circuited the shutdown select!) with
  std::mem::forget(tx); add an idle fallback so the `-> !` is honored.
- /api/preview now resolves the bundled yt-dlp path and extends PATH
  for deno discovery, matching the rest of the downloader.
- Scheduler interval clamped to >=1h on read so a manually-edited
  config.toml with 0 hours can't trigger every tick.
- maintenance::is_within canonicalizes the parent + filename when the
  target is missing; remove_files treats NotFound as success.

Security
- Session tokens stored with issued-at Instant and pruned past 30 day
  TTL on every touch (was: unbounded HashSet).
- Login rate-limited per source IP: 5 failures → 60s lockout (429).
- Secure cookie flag added when X-Forwarded-Proto: https is present.
- 4 MiB body size cap via DefaultBodyLimit.
- Security headers middleware: CSP, X-Content-Type-Options,
  X-Frame-Options, Referrer-Policy.
- JS safeUrl() defangs javascript:/vbscript:/data:text URLs before
  interpolation into src= attributes.
- Bundled yt-dlp install verifies SHA-256 against the release's
  SHA2-256SUMS; deno hash printed for visual inspection. Install also
  reports byte counts every 3s so users see live progress.
- yt-offline.db chmod 0600 at open time on Unix.

New features
- Bundled yt-dlp + deno mode: settings toggle (System/Bundled) plus
  one-click Install/Update from settings. Ensures executable bit on
  every launch in case the install chmod was missed.
- Download queue with configurable max_concurrent (1–10); pending
  jobs surfaced in both UIs.
- Music download mode: --extract-audio into music/<artist>/, separate
  sidebar entry, /api/music + /music-files/ endpoints, inline <audio>.
- Download quality selector: Best / 1080p / 720p / 480p / 360p.
- Sort videos by Newest / Oldest (upload_date from info.json with
  release_date fallback).
- Plex metadata: per-episode .nfo (title, season/episode, aired,
  runtime, plot from .description), <stem>-thumb.jpg symlink, and
  show-level tvshow.nfo.
- yt-dlp retry/throttle defaults: --retries 30 --fragment-retries 30
  --retry-sleep linear=1:30:2 --sleep-requests 1 to ride out the
  "Connection reset by peer" failures.

Performance
- password_required cached in AtomicBool; refreshed only on password
  change. Eliminates a SQLite query per static-file fetch.
- Job::log → VecDeque for O(1) front-pop instead of O(n) drain.
- Library scan parallelized across cores via stdlib parallel_map.
- Web UI poll: 600ms while active, 5s when idle, 2s after errors.
- Music scan now recursive (Artist/Album/Track.opus).
- percent_encode_segment uses write! to avoid per-byte String allocs.

Code quality
- 27 unit tests covering parse_progress, parse_stem, extract_after,
  detect_url_kind, srt_to_vtt, count_cookies, percent_encode_segment,
  bind_mode_of, hash_password, lang_label, is_within, sanitize,
  year_of, aired_date, xml_escape.
- Unified library::find_video + Channel::all_videos() iterator,
  replacing three duplicated linear searches.
- Downloader::browser wired to --cookies-from-browser as a fallback
  when no cookies.txt is present.
- mpv detection now matches the binary basename, not substring.
- Settings modal scroll fix; web settings expose all new fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-23 04:54:26 -07:00
parent 1d72069913
commit 59996735f3
10 changed files with 2300 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

272
src/library.rs
View file

@ -23,6 +23,7 @@ use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
const VIDEO_EXTS: &[&str] = &["mkv", "mp4", "webm", "m4v", "mov", "avi"];
const AUDIO_EXTS: &[&str] = &["mp3", "m4a", "opus", "flac", "ogg", "wav", "aac"];
const THUMB_EXTS: &[&str] = &["webp", "jpg", "jpeg", "png"];
/// A single WebVTT subtitle track discovered alongside a video file.
@ -33,6 +34,20 @@ pub struct Subtitle {
pub path: PathBuf,
}
/// An audio track in the music library.
#[derive(Clone, Debug)]
pub struct Track {
pub id: String,
pub title: String,
pub artist: String,
pub path: PathBuf,
pub thumb_path: Option<PathBuf>,
#[allow(dead_code)]
pub info_path: Option<PathBuf>,
pub duration_secs: Option<f64>,
pub file_size: Option<u64>,
}
/// A fully enriched video entry, ready to serve to the UI.
#[derive(Clone, Debug)]
pub struct Video {
@ -55,6 +70,9 @@ pub struct Video {
pub has_chapters: bool,
/// Size of the video file on disk; `None` if the video file is missing.
pub file_size: Option<u64>,
/// Upload date as `YYYYMMDD` (yt-dlp's native format from info.json).
/// `None` if the info.json sidecar is missing or lacks the field.
pub upload_date: Option<String>,
}
/// A sub-directory inside a channel that contains videos (treated as a playlist).
@ -94,22 +112,58 @@ impl Channel {
pub fn total_videos(&self) -> usize {
self.total_videos_cached
}
/// Iterate over every [`Video`] in this channel, including those nested
/// inside playlists. Used widely; previously open-coded at each call site.
pub fn all_videos(&self) -> impl Iterator<Item = &Video> {
self.videos
.iter()
.chain(self.playlists.iter().flat_map(|p| p.videos.iter()))
}
}
/// Find a video by ID across a slice of channels. Returns the matching
/// [`Video`] alongside the channel it belongs to.
pub fn find_video<'a>(channels: &'a [Channel], id: &str) -> Option<(&'a Video, &'a Channel)> {
for ch in channels {
if let Some(v) = ch.all_videos().find(|v| v.id == id) {
return Some((v, ch));
}
}
None
}
/// Scan `root` for channel directories and return them sorted alphabetically.
///
/// Skips hidden directories (names starting with `.`) and directories that
/// contain no recognisable video files.
///
/// Each channel's per-video info.json reads are parallelised across the
/// available CPUs because that's where ~all the time goes for large
/// libraries (one fs read + one JSON parse per video, multiplied by hundreds
/// or thousands).
pub fn scan_channels(root: &Path) -> Vec<Channel> {
let mut channels = Vec::new();
let Ok(entries) = std::fs::read_dir(root) else { return channels };
for entry in entries.flatten() {
let path = entry.path();
if !path.is_dir() { continue; }
let name = entry.file_name().to_string_lossy().into_owned();
if name.starts_with('.') { continue; }
let Ok(entries) = std::fs::read_dir(root) else { return Vec::new() };
let dirs: Vec<(String, PathBuf)> = entries
.flatten()
.filter_map(|e| {
let path = e.path();
if !path.is_dir() { return None; }
let name = e.file_name().to_string_lossy().into_owned();
if name.starts_with('.') { return None; }
Some((name, path))
})
.collect();
// Process channels in parallel. We size the worker pool to min(channels, CPUs)
// so a small library doesn't spin up needless threads.
let n_workers = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(4)
.min(dirs.len().max(1));
let mut channels = parallel_map(dirs, n_workers, |(name, path)| {
let (videos, playlists) = scan_channel_dir(&path);
if videos.is_empty() && playlists.is_empty() { continue; }
if videos.is_empty() && playlists.is_empty() { return None; }
let meta = load_channel_meta(&videos);
let total_videos_cached =
videos.len() + playlists.iter().map(|p| p.videos.len()).sum::<usize>();
@ -118,7 +172,7 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
.chain(playlists.iter().flat_map(|p| p.videos.iter()))
.filter_map(|v| v.file_size)
.sum();
channels.push(Channel {
Some(Channel {
name,
path,
videos,
@ -126,12 +180,67 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
meta,
total_videos_cached,
total_size_cached,
});
}
})
})
.into_iter()
.flatten()
.collect::<Vec<_>>();
channels.sort_by_key(|c| c.name.to_lowercase());
channels
}
/// Fan an `items` slice across `n_workers` threads, applying `f` to each item
/// and returning results in the original input order.
///
/// Stdlib-only mini work-stealer: an atomic index hands out the next slot to
/// any worker that's free. Used to parallelise channel-directory scans
/// without dragging in rayon.
fn parallel_map<I, O, F>(items: Vec<I>, n_workers: usize, f: F) -> Vec<O>
where
I: Send + 'static,
O: Send + 'static + Default,
F: Fn(I) -> O + Send + Sync + 'static,
{
let len = items.len();
if len == 0 { return Vec::new(); }
if n_workers <= 1 {
return items.into_iter().map(f).collect();
}
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
let items: Vec<Mutex<Option<I>>> = items.into_iter().map(|v| Mutex::new(Some(v))).collect();
let items = Arc::new(items);
let results: Vec<Mutex<O>> = (0..len).map(|_| Mutex::new(O::default())).collect();
let results = Arc::new(results);
let next = Arc::new(AtomicUsize::new(0));
let f = Arc::new(f);
let mut handles = Vec::with_capacity(n_workers);
for _ in 0..n_workers {
let items = items.clone();
let results = results.clone();
let next = next.clone();
let f = f.clone();
handles.push(std::thread::spawn(move || {
loop {
let i = next.fetch_add(1, Ordering::Relaxed);
if i >= len { break; }
let input = items[i].lock().unwrap().take().unwrap();
let out = f(input);
*results[i].lock().unwrap() = out;
}
}));
}
for h in handles { let _ = h.join(); }
Arc::try_unwrap(results)
.unwrap_or_else(|_| unreachable!("workers joined; refs released"))
.into_iter()
.map(|m| m.into_inner().unwrap())
.collect()
}
fn load_channel_meta(videos: &[Video]) -> Option<ChannelMeta> {
// Pull channel-level fields out of the first video's info.json
let info_path = videos.iter().find_map(|v| {
@ -261,7 +370,7 @@ fn collect_raw_videos(entries: impl Iterator<Item = std::fs::DirEntry>) -> Vec<R
fn enrich(raws: Vec<RawVideo>) -> Vec<Video> {
let mut videos: Vec<Video> = raws.into_iter().map(|raw| {
// Parse info.json once for both duration and chapter presence.
let (duration_secs, has_chapters) = raw.info_path.as_ref()
let (duration_secs, has_chapters, upload_date) = raw.info_path.as_ref()
.and_then(|p| std::fs::read_to_string(p).ok())
.and_then(|t| serde_json::from_str::<serde_json::Value>(&t).ok())
.map(|val| {
@ -270,9 +379,14 @@ fn enrich(raws: Vec<RawVideo>) -> Vec<Video> {
.and_then(|c| c.as_array())
.map(|a| !a.is_empty())
.unwrap_or(false);
(dur, chap)
// Prefer `upload_date`; fall back to `release_date` for premiere/live content.
let date = val.get("upload_date")
.and_then(|v| v.as_str())
.or_else(|| val.get("release_date").and_then(|v| v.as_str()))
.map(|s| s.to_string());
(dur, chap, date)
})
.unwrap_or((None, false));
.unwrap_or((None, false, None));
let file_size = raw.video_path.as_ref()
.and_then(|p| std::fs::metadata(p).ok())
.map(|m| m.len());
@ -289,6 +403,7 @@ fn enrich(raws: Vec<RawVideo>) -> Vec<Video> {
duration_secs,
has_chapters,
file_size,
upload_date,
}
}).collect();
videos.sort_by_key(|v| v.title.to_lowercase());
@ -328,3 +443,132 @@ fn scan_channel_dir(dir: &Path) -> (Vec<Video>, Vec<Playlist>) {
playlists.sort_by_key(|p| p.name.to_lowercase());
(videos, playlists)
}
// ── Music library ─────────────────────────────────────────────────────────────
/// Scan `root` (the `music/` directory) for audio tracks, recursively.
///
/// The top-level subdirectory name is used as the default artist (overridden
/// by info.json's `artist`/`creator`/`uploader` when present). Deeper levels
/// — e.g. `music/Artist/Album/song.opus` — are walked but the top-level name
/// remains the fallback artist so albums don't reset the attribution.
pub fn scan_music(root: &Path) -> Vec<Track> {
let mut tracks = Vec::new();
let Ok(entries) = std::fs::read_dir(root) else { return tracks };
for entry in entries.flatten() {
let path = entry.path();
let name = entry.file_name().to_string_lossy().into_owned();
if name.starts_with('.') { continue; }
if path.is_dir() {
scan_music_dir(&path, &name, &mut tracks);
} else if let Some(track) = track_from_path(&path, "") {
tracks.push(track);
}
}
tracks.sort_by_key(|t| (t.artist.to_lowercase(), t.title.to_lowercase()));
tracks
}
fn scan_music_dir(dir: &Path, folder_artist: &str, tracks: &mut Vec<Track>) {
let Ok(entries) = std::fs::read_dir(dir) else { return };
for entry in entries.flatten() {
let path = entry.path();
let name = entry.file_name().to_string_lossy().into_owned();
if name.starts_with('.') { continue; }
if path.is_file() {
if let Some(track) = track_from_path(&path, folder_artist) {
tracks.push(track);
}
} else if path.is_dir() {
// Recurse into albums/subfolders while preserving the top-level
// artist label.
scan_music_dir(&path, folder_artist, tracks);
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_stem_extracts_id_and_title() {
let (title, id) = parse_stem("My great video [abc123]").unwrap();
assert_eq!(title, "My great video");
assert_eq!(id, "abc123");
}
#[test]
fn parse_stem_trims_trailing_dash() {
let (title, _id) = parse_stem("Some video - [xyz]").unwrap();
assert_eq!(title, "Some video");
}
#[test]
fn parse_stem_rejects_missing_brackets() {
assert!(parse_stem("no brackets here").is_none());
}
#[test]
fn parse_stem_rejects_empty_id() {
assert!(parse_stem("foo []").is_none());
}
#[test]
fn parse_stem_handles_brackets_in_title() {
// The last [..] is the id; earlier ones are part of the title.
let (title, id) = parse_stem("[NSFW] Some title [vidid]").unwrap();
assert_eq!(id, "vidid");
assert!(title.contains("[NSFW]"));
}
}
fn track_from_path(path: &Path, folder_artist: &str) -> Option<Track> {
let ext = path.extension()?.to_str()?.to_lowercase();
if !AUDIO_EXTS.contains(&ext.as_str()) { return None; }
let stem = path.file_stem()?.to_string_lossy().into_owned();
let (title, id) = parse_stem(&stem)?;
let dir = path.parent()?;
let thumb_path = THUMB_EXTS.iter().find_map(|e| {
let p = dir.join(format!("{stem}.{e}"));
p.exists().then_some(p)
});
let info_path = {
let p = dir.join(format!("{stem}.info.json"));
p.exists().then_some(p)
};
let (duration_secs, resolved_artist) = info_path.as_ref()
.and_then(|p| std::fs::read_to_string(p).ok())
.and_then(|t| serde_json::from_str::<serde_json::Value>(&t).ok())
.map(|val| {
let dur = val.get("duration").and_then(|v| v.as_f64());
let art = val.get("artist")
.or_else(|| val.get("creator"))
.or_else(|| val.get("uploader"))
.and_then(|v| v.as_str())
.map(String::from)
.unwrap_or_else(|| folder_artist.to_string());
(dur, art)
})
.unwrap_or_else(|| (None, folder_artist.to_string()));
let artist = if resolved_artist.is_empty() {
"Unknown".to_string()
} else {
resolved_artist
};
Some(Track {
id,
title,
artist,
path: path.to_path_buf(),
thumb_path,
info_path,
duration_secs,
file_size: std::fs::metadata(path).ok().map(|m| m.len()),
})
}