luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add multi-platform support: TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee

Browse source 
yt-dlp already speaks ~1,800 sites; the changes here teach the rest of
the app to route them through platform-specific output folders and
remember where each creator was downloaded from.

New `platform` module
- `Platform` enum: YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud,
  Odysee, Other.
- `classify_url` returns both the source platform and a `UrlKind`
  (Channel / Playlist / Video / Unknown) — per-platform path parsing
  for each supported host.
- `platform_root(channels_root, platform)` returns the on-disk folder
  per platform. YouTube keeps the legacy `channels/` for backward
  compat; the others live as siblings (`tiktok/`, `twitch/`, …).
- `write_source_url` / `read_source_url` persist the originating URL
  in a `.source-url` sidecar at each creator folder so re-checks no
  longer rely on the YouTube-only "UC + 22 chars" heuristic.

Downloader
- `Downloader::start` now takes `&UrlInfo` instead of `&UrlKind` and
  routes the yt-dlp `-o` template into the right platform folder.
- For channel downloads it also writes `.source-url` so future
  re-checks recover the exact URL.
- Output template uses `%(uploader,channel,creator|Unknown)s` so
  non-YouTube sites that don't expose `channel` still get a sensible
  creator folder.
- Per-platform `archive.txt` keeps cross-platform IDs (TikTok numeric
  vs YouTube base64) from colliding.
- New `recheck_url(&Channel)` prefers `source_url` and falls back to
  the legacy YouTube heuristic for libraries created before this
  change.

Library scanner
- `scan_channels` now walks every platform's folder and tags each
  `Channel` with its `platform` + `source_url`. Sort key puts the
  platform first, then name, so the sidebar groups cleanly.
- `Channel` gains `platform: Platform` and `source_url: Option<String>`
  fields, populated from the on-disk sidecar.

Web UI
- `WebState` gains `library_root` (= parent of channels_root). The
  `/files/` ServeDir is mounted at library_root so non-YouTube videos
  resolve at `/files/<platform>/<creator>/<file>`. Existing YouTube
  URLs still work — they're now `/files/channels/...` instead of
  `/files/...` and the server-rendered JSON updates accordingly.
- Maintenance scan + remove use `library_root` so non-YouTube content
  is included.
- `ChannelInfo` JSON exposes `platform`, `platform_label`,
  `platform_icon`, and `source_url`.
- Sidebar groups channels by platform with each platform's icon.
- "Check for new videos" now uses the stored `source_url` (with the
  YouTube heuristic as a last-resort fallback).
- Download dialog preview shows the detected source + destination
  folder.

Desktop UI
- App gains a `library_root` field mirroring the web side; maintenance
  ops scan/remove against it.
- Sidebar channel labels are prefixed with the platform icon for
  non-YouTube channels; tooltip carries the platform name.
- Download URL field surfaces both source platform and folder path
  preview as you type.

Plex
- Non-YouTube creators get their show folder prefixed with the
  platform name (e.g., `TikTok - cooluser`) so a YouTube channel and
  a TikTok account with the same name don't collide.

Backward compatibility
- Existing YouTube libraries (`channels/<handle>/...`) are untouched.
  New platform folders are created on first run as siblings.
- Channels without a `.source-url` (i.e., everything that pre-dates
  this commit) fall through to the YouTube heuristic, preserving the
  existing re-check behavior.

Tests
- 14 new `platform` tests cover URL classification per platform,
  channel handle extraction, and `platform_root` layout. The old
  `detect_url_kind` / `extract_after` tests move into platform.rs
  since the canonical implementation lives there now.
- 41 unit tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-24 22:16:40 -07:00
parent 4d83999edd
commit 12c32f642b
7 changed files with 699 additions and 133 deletions
Download patch file Download diff file Expand all files Collapse all files

84
src/app.rs
View file

@ -14,7 +14,8 @@ use eframe::egui;
use crate::config::Config; use crate::config::Config;
use crate::database::Database; use crate::database::Database;
use crate::downloader::{detect_url_kind, DownloadQuality, Downloader, JobState, UrlKind}; use crate::downloader::{DownloadQuality, Downloader, JobState};
use crate::platform::{self, classify_url, Platform, UrlKind};
use crate::library::{self, Video}; use crate::library::{self, Video};
use crate::theme; use crate::theme;
@ -67,6 +68,10 @@ pub struct App {
config: Config, config: Config,
config_path: PathBuf, config_path: PathBuf,
channels_root: PathBuf, channels_root: PathBuf,
/// Parent of `channels_root`. Owns every platform's sibling directory
/// (`channels/`, `tiktok/`, …). Used as the maintenance scan root so
/// non-YouTube content is included.
library_root: PathBuf,
library: Vec<library::Channel>, library: Vec<library::Channel>,
sidebar_view: SidebarView, sidebar_view: SidebarView,
selected_video: Option<String>, selected_video: Option<String>,
@ -145,6 +150,16 @@ impl App {
let channels_root = config.backup.directory.clone(); let channels_root = config.backup.directory.clone();
let settings_dir = channels_root.display().to_string(); let settings_dir = channels_root.display().to_string();
let _ = std::fs::create_dir_all(&channels_root); let _ = std::fs::create_dir_all(&channels_root);
let library_root = channels_root
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| channels_root.clone());
let _ = std::fs::create_dir_all(&library_root);
// Pre-create every platform's folder so scans see them.
for &p in Platform::all() {
let dir = platform::platform_root(&channels_root, p);
let _ = std::fs::create_dir_all(&dir);
}
let library = library::scan_channels(&channels_root); let library = library::scan_channels(&channels_root);
let status = format!( let status = format!(
"{} channels, {} videos", "{} channels, {} videos",
@ -192,6 +207,7 @@ impl App {
config, config,
config_path, config_path,
channels_root: channels_root.clone(), channels_root: channels_root.clone(),
library_root,
library, library,
sidebar_view: SidebarView::All, sidebar_view: SidebarView::All,
selected_video: None, selected_video: None,
@ -547,11 +563,11 @@ impl App {
fn run_scheduled_check(&mut self) { fn run_scheduled_check(&mut self) {
let mut count = 0; let mut count = 0;
let urls: Vec<String> = self.library.iter() let urls: Vec<String> = self.library.iter()
.map(|ch| crate::downloader::check_url_for_folder(&ch.name)) .map(|ch| crate::downloader::recheck_url(ch))
.collect(); .collect();
for url in urls { for url in urls {
let kind = detect_url_kind(&url); let info = classify_url(&url);
self.downloader.start(url, &kind, true, DownloadQuality::Best); self.downloader.start(url, &info, true, DownloadQuality::Best);
count += 1; count += 1;
} }
self.status = format!("Scheduled check: started {} channel downloads", count); self.status = format!("Scheduled check: started {} channel downloads", count);
@ -636,7 +652,7 @@ impl App {
self.show_maintenance = !self.show_maintenance; self.show_maintenance = !self.show_maintenance;
if self.show_maintenance { if self.show_maintenance {
self.health_report = self.health_report =
Some(crate::maintenance::scan(&self.channels_root, &self.library)); Some(crate::maintenance::scan(&self.library_root, &self.library));
} }
} }
if ui.selectable_label(self.show_settings, "⚙ Settings").clicked() { if ui.selectable_label(self.show_settings, "⚙ Settings").clicked() {
@ -734,17 +750,19 @@ impl App {
let mut pending_ch_download: Option<(String, String)> = None; // (url, channel_name) let mut pending_ch_download: Option<(String, String)> = None; // (url, channel_name)
for i in 0..self.library.len() { for i in 0..self.library.len() {
let (name, total, has_playlists, size_bytes, channel_url) = { let (name, total, has_playlists, size_bytes, channel_url, platform) = {
let ch = &self.library[i]; let ch = &self.library[i];
// Always derive the check URL from the folder name so yt-dlp // Prefer the stored `.source-url` over folder-name guessing so
// writes to the existing folder, not a new UCxxx one. // re-checks work across platforms (and across legacy YouTube
let url = crate::downloader::check_url_for_folder(&ch.name); // libraries where the URL was never recorded).
let url = crate::downloader::recheck_url(ch);
( (
ch.name.clone(), ch.name.clone(),
ch.total_videos(), ch.total_videos(),
!ch.playlists.is_empty(), !ch.playlists.is_empty(),
Self::channel_total_size(ch), Self::channel_total_size(ch),
url, url,
ch.platform,
) )
}; };
@ -756,12 +774,24 @@ impl App {
} else { } else {
String::new() String::new()
}; };
let label = format!("{} ({}{})", name, total, size_str); // Show platform icon for non-YouTube channels so the sidebar
// reads as a unified library while still making the source
// obvious at a glance.
let prefix = if platform == Platform::YouTube {
String::new()
} else {
format!("{} ", platform.icon())
};
let label = format!("{prefix}{} ({}{})", name, total, size_str);
let ch_selected_no_pl = matches!(self.sidebar_view, SidebarView::Channel(ci) if ci == i); let ch_selected_no_pl = matches!(self.sidebar_view, SidebarView::Channel(ci) if ci == i);
let resp = ui let resp = ui
.selectable_label(ch_selected_no_pl, label) .selectable_label(ch_selected_no_pl, label)
.on_hover_text(self.library[i].path.display().to_string()); .on_hover_text(format!(
"{}\n{}",
self.library[i].path.display(),
platform.display_name(),
));
if resp.clicked() { if resp.clicked() {
self.sidebar_view = SidebarView::Channel(i); self.sidebar_view = SidebarView::Channel(i);
self.selected_video = None; self.selected_video = None;
@ -805,8 +835,8 @@ impl App {
// Process deferred right-click download action // Process deferred right-click download action
if let Some((url, ch_name)) = pending_ch_download { if let Some((url, ch_name)) = pending_ch_download {
let kind = detect_url_kind(&url); let info = classify_url(&url);
self.downloader.start(url, &kind, !self.dl_full_scan, DownloadQuality::Best); self.downloader.start(url, &info, !self.dl_full_scan, DownloadQuality::Best);
self.status = format!("Checking {} for new videos…", ch_name); self.status = format!("Checking {} for new videos…", ch_name);
} }
}); });
@ -827,18 +857,22 @@ impl App {
.desired_width(f32::INFINITY), .desired_width(f32::INFINITY),
); );
let kind = detect_url_kind(self.dl_url.trim()); let info = classify_url(self.dl_url.trim());
let (type_label, dest_preview) = match &kind { let plat_dir = info.platform.dir_name();
let (type_label, dest_preview) = match &info.kind {
UrlKind::Channel { handle } => { UrlKind::Channel { handle } => {
("Channel", format!("→ channels/{}/", handle)) ("Channel", format!("→ {plat_dir}/{handle}/"))
} }
UrlKind::Playlist => ("Playlist", "→ channels/<channel>/<playlist>/".to_string()), UrlKind::Playlist => ("Playlist", format!("→ {plat_dir}/<creator>/<playlist>/")),
UrlKind::Video => ("Video", "→ channels/<channel>/".to_string()), UrlKind::Video => ("Video", format!("→ {plat_dir}/<creator>/")),
UrlKind::Unknown => ("—", String::new()), UrlKind::Unknown => ("—", String::new()),
}; };
if !self.dl_url.trim().is_empty() { if !self.dl_url.trim().is_empty() {
ui.horizontal(|ui| { ui.horizontal(|ui| {
ui.label("Source:");
ui.strong(format!("{} {}", info.platform.icon(), info.platform.display_name()));
ui.separator();
ui.label("Type:"); ui.label("Type:");
ui.strong(type_label); ui.strong(type_label);
}); });
@ -877,7 +911,7 @@ impl App {
self.downloader.start_music(url); self.downloader.start_music(url);
self.status = "Downloading music…".to_string(); self.status = "Downloading music…".to_string();
} else { } else {
self.downloader.start(url, &kind, !self.dl_full_scan, self.dl_quality); self.downloader.start(url, &info, !self.dl_full_scan, self.dl_quality);
self.status = format!("Downloading: {dest}"); self.status = format!("Downloading: {dest}");
} }
} }
@ -1062,7 +1096,7 @@ impl App {
let mut changed = false; let mut changed = false;
if !to_remove.is_empty() { if !to_remove.is_empty() {
let (removed, errors) = let (removed, errors) =
crate::maintenance::remove_files(&self.channels_root, &to_remove); crate::maintenance::remove_files(&self.library_root, &to_remove);
self.status = if errors.is_empty() { self.status = if errors.is_empty() {
format!("Removed {removed} file(s)") format!("Removed {removed} file(s)")
} else { } else {
@ -1082,7 +1116,7 @@ impl App {
if changed || rescan_health { if changed || rescan_health {
self.rescan(); self.rescan();
self.health_report = self.health_report =
Some(crate::maintenance::scan(&self.channels_root, &self.library)); Some(crate::maintenance::scan(&self.library_root, &self.library));
} }
} }
@ -1527,8 +1561,16 @@ impl App {
self.downloader.use_bundled_ytdlp = self.config.backup.use_bundled_ytdlp; self.downloader.use_bundled_ytdlp = self.config.backup.use_bundled_ytdlp;
if dir_changed { if dir_changed {
self.channels_root = new_dir.clone(); self.channels_root = new_dir.clone();
self.library_root = new_dir
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| new_dir.clone());
self.downloader.channels_root = new_dir; self.downloader.channels_root = new_dir;
let _ = std::fs::create_dir_all(&self.channels_root); let _ = std::fs::create_dir_all(&self.channels_root);
let _ = std::fs::create_dir_all(&self.library_root);
for &p in Platform::all() {
let _ = std::fs::create_dir_all(platform::platform_root(&self.channels_root, p));
}
self.rescan(); self.rescan();
} }
// Re-bind a running server so the new interface takes effect now. // Re-bind a running server so the new interface takes effect now.

106
src/downloader.rs
View file

@ -30,6 +30,7 @@ use std::process::{Command, Stdio};
use std::sync::mpsc::{channel, Receiver}; use std::sync::mpsc::{channel, Receiver};
use std::thread; use std::thread;
use crate::platform::{self, UrlInfo, UrlKind};
use crate::ytdlp_bin; use crate::ytdlp_bin;
/// Video quality level passed as a `-f` format selector to yt-dlp. /// Video quality level passed as a `-f` format selector to yt-dlp.
@ -70,23 +71,14 @@ impl DownloadQuality {
} }
} }
/// Describes the kind of YouTube URL being downloaded, which determines the /// Build a YouTube URL from a legacy library folder name. Used as a fallback
/// output path template passed to yt-dlp. /// when a channel folder has no `.source-url` sidecar (i.e. it predates the
pub enum UrlKind { /// multi-platform changes).
/// A channel URL (`/@handle`, `/channel/ID`, or `/c/name`).
Channel { handle: String },
Playlist,
Video,
Unknown,
}
/// Build a YouTube URL from a library folder name that yt-dlp will resolve to
/// the same folder it already downloaded to.
/// ///
/// Folder names that look like a channel ID (`UC` + 22 chars) use the /// Folder names that look like a channel ID (`UC` + 22 chars) use the
/// `/channel/` form; everything else is treated as a handle and gets `/@`. /// `/channel/` form; everything else is treated as a handle and gets `/@`.
/// This avoids the mismatch where info.json's canonical `channel_url` field /// This avoids the mismatch where info.json's canonical `channel_url` field
/// points to `/channel/UCxxx` and yt-dlp then creates a second folder. /// points to `/channel/UCxxx` and yt-dlp creates a second folder.
pub fn check_url_for_folder(folder_name: &str) -> String { pub fn check_url_for_folder(folder_name: &str) -> String {
if folder_name.starts_with("UC") && folder_name.len() == 24 { if folder_name.starts_with("UC") && folder_name.len() == 24 {
format!("https://www.youtube.com/channel/{folder_name}") format!("https://www.youtube.com/channel/{folder_name}")
@ -95,32 +87,17 @@ pub fn check_url_for_folder(folder_name: &str) -> String {
} }
} }
/// Classify a YouTube URL into a [`UrlKind`] by inspecting its path. /// Re-check URL for a [`crate::library::Channel`]. Prefers the stored
pub fn detect_url_kind(url: &str) -> UrlKind { /// `.source-url` sidecar when present, falls back to the YouTube heuristic
if url.contains("playlist?list=") { /// for legacy folders.
return UrlKind::Playlist; pub fn recheck_url(ch: &crate::library::Channel) -> String {
if let Some(url) = ch.source_url.as_deref() {
return url.to_string();
} }
if let Some(h) = extract_after(url, "/@") { // Legacy YouTube libraries: rebuild from folder name.
return UrlKind::Channel { handle: h.to_string() }; check_url_for_folder(&ch.name)
}
if let Some(h) = extract_after(url, "/channel/") {
return UrlKind::Channel { handle: h.to_string() };
}
if let Some(h) = extract_after(url, "/c/") {
return UrlKind::Channel { handle: h.to_string() };
}
if url.contains("watch?v=") || url.contains("youtu.be/") {
return UrlKind::Video;
}
UrlKind::Unknown
} }
fn extract_after<'a>(url: &'a str, marker: &str) -> Option<&'a str> {
let start = url.find(marker)? + marker.len();
let rest = &url[start..];
let end = rest.find(|c| c == '/' || c == '?' || c == '&' || c == '#').unwrap_or(rest.len());
if end == 0 { None } else { Some(&rest[..end]) }
}
/// Lifecycle state of a download job. /// Lifecycle state of a download job.
#[derive(Clone, Copy, PartialEq, Eq)] #[derive(Clone, Copy, PartialEq, Eq)]
@ -278,33 +255,51 @@ impl Downloader {
/// Spawn a yt-dlp process for `url` and track it as a new [`Job`]. /// Spawn a yt-dlp process for `url` and track it as a new [`Job`].
/// ///
/// The output path template is derived from `kind` so that channels, /// Output path template is derived from `info.platform` + `info.kind` so
/// playlists, and individual videos land in the right sub-directories. /// each platform lands in its own sibling directory (`channels/` for
/// YouTube, `tiktok/`, `twitch/`, etc. for others).
///
/// For channel downloads we also drop a `.source-url` sidecar so future
/// re-checks recover the original URL without folder-name guessing.
/// ///
/// When `full_scan` is false (default / incremental mode) `--break-on-existing` /// When `full_scan` is false (default / incremental mode) `--break-on-existing`
/// is passed so yt-dlp stops as soon as it hits the first already-archived /// is passed so yt-dlp stops as soon as it hits the first already-archived
/// video — fast for routine channel checks. When `full_scan` is true the /// video — fast for routine channel checks. When `full_scan` is true the
/// flag is omitted so every video is checked individually against the /// flag is omitted so every video is checked individually against the
/// download archive; slower, but correctly fills gaps in the history. /// download archive; slower, but correctly fills gaps in the history.
pub fn start(&mut self, url: String, kind: &UrlKind, full_scan: bool, quality: DownloadQuality) { pub fn start(&mut self, url: String, info: &UrlInfo, full_scan: bool, quality: DownloadQuality) {
let archive_path = self.channels_root.join("archive.txt"); let platform_dir = platform::platform_root(&self.channels_root, info.platform);
// Per-platform download archive keeps cross-platform IDs from colliding
// (TikTok IDs are numeric, YouTube IDs are 11-char base64, etc.).
let _ = std::fs::create_dir_all(&platform_dir);
let archive_path = platform_dir.join("archive.txt");
let platform_label = info.platform.dir_name();
let (out_arg, label) = match kind { let (out_arg, label) = match &info.kind {
UrlKind::Channel { handle } => { UrlKind::Channel { handle } => {
let dir = self.channels_root.join(handle); let dir = platform_dir.join(handle);
let _ = std::fs::create_dir_all(&dir); let _ = std::fs::create_dir_all(&dir);
// Remember the originating URL so re-checks don't have to
// guess from the folder name.
platform::write_source_url(&dir, &url);
( (
format!("{}/%(title)s [%(id)s].%(ext)s", dir.display()), format!("{}/%(title)s [%(id)s].%(ext)s", dir.display()),
format!("channels/{}/", handle), format!("{}/{}/", platform_label, handle),
) )
} }
UrlKind::Playlist => ( UrlKind::Playlist => (
format!("{}/%(channel)s/%(playlist_title)s/%(title)s [%(id)s].%(ext)s", self.channels_root.display()), format!(
"channels/<channel>/<playlist>/".to_string(), "{}/%(uploader,channel,creator|Unknown)s/%(playlist_title)s/%(title)s [%(id)s].%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/<playlist>/", platform_label),
), ),
UrlKind::Video | UrlKind::Unknown => ( UrlKind::Video | UrlKind::Unknown => (
format!("{}/%(channel)s/%(title)s [%(id)s].%(ext)s", self.channels_root.display()), format!(
"channels/<channel>/".to_string(), "{}/%(uploader,channel,creator|Unknown)s/%(title)s [%(id)s].%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/", platform_label),
), ),
}; };
@ -573,14 +568,6 @@ mod tests {
assert!(parse_progress("").is_none()); assert!(parse_progress("").is_none());
} }
#[test]
fn extract_after_strips_at_separator() {
assert_eq!(extract_after("https://youtube.com/@handle/videos", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle?x=1", "/@"), Some("handle"));
assert_eq!(extract_after("nothing-here", "/@"), None);
}
#[test] #[test]
fn check_url_for_folder_picks_channel_form_for_ids() { fn check_url_for_folder_picks_channel_form_for_ids() {
let url = check_url_for_folder("UC1234567890123456789012"); let url = check_url_for_folder("UC1234567890123456789012");
@ -592,12 +579,5 @@ mod tests {
let url = check_url_for_folder("LinusTechTips"); let url = check_url_for_folder("LinusTechTips");
assert_eq!(url, "https://www.youtube.com/@LinusTechTips"); assert_eq!(url, "https://www.youtube.com/@LinusTechTips");
} }
// URL classification tests live in `platform` now — see its tests module.
#[test]
fn detect_url_kind_classifies_paths() {
assert!(matches!(detect_url_kind("https://www.youtube.com/playlist?list=PL1"), UrlKind::Playlist));
assert!(matches!(detect_url_kind("https://www.youtube.com/watch?v=abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://youtu.be/abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://www.youtube.com/@handle"), UrlKind::Channel { .. }));
}
} }

63
src/library.rs
View file

@ -22,6 +22,8 @@
use std::collections::BTreeMap; use std::collections::BTreeMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::platform::{self, Platform};
const VIDEO_EXTS: &[&str] = &["mkv", "mp4", "webm", "m4v", "mov", "avi"]; const VIDEO_EXTS: &[&str] = &["mkv", "mp4", "webm", "m4v", "mov", "avi"];
const AUDIO_EXTS: &[&str] = &["mp3", "m4a", "opus", "flac", "ogg", "wav", "aac"]; const AUDIO_EXTS: &[&str] = &["mp3", "m4a", "opus", "flac", "ogg", "wav", "aac"];
const THUMB_EXTS: &[&str] = &["webp", "jpg", "jpeg", "png"]; const THUMB_EXTS: &[&str] = &["webp", "jpg", "jpeg", "png"];
@ -97,6 +99,11 @@ pub struct ChannelMeta {
pub struct Channel { pub struct Channel {
pub name: String, pub name: String,
pub path: PathBuf, pub path: PathBuf,
/// Source platform — drives sidebar grouping and the re-check URL.
pub platform: Platform,
/// Originating URL read from the `.source-url` sidecar. Falls back to a
/// folder-name heuristic for legacy YouTube libraries that predate it.
pub source_url: Option<String>,
/// Videos stored directly inside the channel directory (not in a sub-folder). /// Videos stored directly inside the channel directory (not in a sub-folder).
pub videos: Vec<Video>, pub videos: Vec<Video>,
/// Sub-directories that contain at least one video. /// Sub-directories that contain at least one video.
@ -133,35 +140,37 @@ pub fn find_video<'a>(channels: &'a [Channel], id: &str) -> Option<(&'a Video, &
None None
} }
/// Scan `root` for channel directories and return them sorted alphabetically. /// Scan all platform directories rooted at (or sibling to) `youtube_root`
/// and return every channel found, tagged with its source platform.
/// ///
/// Skips hidden directories (names starting with `.`) and directories that /// The configured `youtube_root` is treated as `Platform::YouTube` for
/// contain no recognisable video files. /// backward compatibility with libraries created before the multi-platform
/// changes landed. Each other platform's folder lives as a sibling — see
/// [`crate::platform::platform_root`].
/// ///
/// Each channel's per-video info.json reads are parallelised across the /// Per-channel info.json reads are parallelised across the available CPUs
/// available CPUs because that's where ~all the time goes for large /// because that's where ~all the time goes for large libraries (one fs read
/// libraries (one fs read + one JSON parse per video, multiplied by hundreds /// + one JSON parse per video, multiplied by thousands).
/// or thousands). pub fn scan_channels(youtube_root: &Path) -> Vec<Channel> {
pub fn scan_channels(root: &Path) -> Vec<Channel> { // Gather (platform, channel-folder-name, full-path) across every platform.
let Ok(entries) = std::fs::read_dir(root) else { return Vec::new() }; let mut dirs: Vec<(Platform, String, PathBuf)> = Vec::new();
let dirs: Vec<(String, PathBuf)> = entries for &platform in Platform::all() {
.flatten() let root = platform::platform_root(youtube_root, platform);
.filter_map(|e| { let Ok(entries) = std::fs::read_dir(&root) else { continue };
let path = e.path(); for entry in entries.flatten() {
if !path.is_dir() { return None; } let path = entry.path();
let name = e.file_name().to_string_lossy().into_owned(); if !path.is_dir() { continue; }
if name.starts_with('.') { return None; } let name = entry.file_name().to_string_lossy().into_owned();
Some((name, path)) if name.starts_with('.') { continue; }
}) dirs.push((platform, name, path));
.collect(); }
}
// Process channels in parallel. We size the worker pool to min(channels, CPUs)
// so a small library doesn't spin up needless threads.
let n_workers = std::thread::available_parallelism() let n_workers = std::thread::available_parallelism()
.map(|n| n.get()) .map(|n| n.get())
.unwrap_or(4) .unwrap_or(4)
.min(dirs.len().max(1)); .min(dirs.len().max(1));
let mut channels = parallel_map(dirs, n_workers, |(name, path)| { let mut channels = parallel_map(dirs, n_workers, |(platform, name, path)| {
let (videos, playlists) = scan_channel_dir(&path); let (videos, playlists) = scan_channel_dir(&path);
if videos.is_empty() && playlists.is_empty() { return None; } if videos.is_empty() && playlists.is_empty() { return None; }
let meta = load_channel_meta(&videos); let meta = load_channel_meta(&videos);
@ -172,9 +181,12 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
.chain(playlists.iter().flat_map(|p| p.videos.iter())) .chain(playlists.iter().flat_map(|p| p.videos.iter()))
.filter_map(|v| v.file_size) .filter_map(|v| v.file_size)
.sum(); .sum();
let source_url = platform::read_source_url(&path);
Some(Channel { Some(Channel {
name, name,
path, path,
platform,
source_url,
videos, videos,
playlists, playlists,
meta, meta,
@ -185,7 +197,12 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
.into_iter() .into_iter()
.flatten() .flatten()
.collect::<Vec<_>>(); .collect::<Vec<_>>();
channels.sort_by_key(|c| c.name.to_lowercase()); // Stable order: platform first (so the sidebar groups cleanly), then name.
channels.sort_by(|a, b| {
let pa = a.platform as u8;
let pb = b.platform as u8;
pa.cmp(&pb).then_with(|| a.name.to_lowercase().cmp(&b.name.to_lowercase()))
});
channels channels
} }

1
src/main.rs
View file

@ -19,6 +19,7 @@ mod database;
mod downloader; mod downloader;
mod library; mod library;
mod maintenance; mod maintenance;
mod platform;
mod plex; mod plex;
mod stats; mod stats;
mod theme; mod theme;

463
src/platform.rs Normal file
View file

@ -0,0 +1,463 @@
//! Source-platform classification.
//!
//! yt-dlp can pull from ~1,800 sites; this module organises the supported
//! ones into a small enum with three concerns:
//!
//! 1. **URL → platform**: pattern-match host to a [`Platform`] variant.
//! 2. **Platform → directory**: each platform owns a sibling folder next to
//! `channels/` (which remains YouTube for backward compatibility).
//! 3. **Folder → display**: icon and human label for the UI.
//!
//! Anything yt-dlp accepts that doesn't match a known host lands in
//! [`Platform::Other`] and goes into `other/<creator>/`.
//!
//! Per-creator metadata (the original source URL, so re-checks don't have to
//! guess from a folder name) is written to a `.source-url` sidecar at the
//! root of each creator folder. The library scanner picks it up.
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
/// Where a download came from. Drives output paths and sidebar grouping.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
pub enum Platform {
YouTube,
TikTok,
Twitch,
Vimeo,
Bandcamp,
SoundCloud,
Odysee,
Other,
}
impl Default for Platform {
fn default() -> Self { Self::YouTube }
}
impl Platform {
/// Folder name used under the backup root. YouTube keeps the legacy
/// `channels/` for backward compatibility with existing libraries.
pub fn dir_name(self) -> &'static str {
match self {
Self::YouTube => "channels",
Self::TikTok => "tiktok",
Self::Twitch => "twitch",
Self::Vimeo => "vimeo",
Self::Bandcamp => "bandcamp",
Self::SoundCloud => "soundcloud",
Self::Odysee => "odysee",
Self::Other => "other",
}
}
/// Human-readable label for the UI (sidebar / tooltips).
pub fn display_name(self) -> &'static str {
match self {
Self::YouTube => "YouTube",
Self::TikTok => "TikTok",
Self::Twitch => "Twitch",
Self::Vimeo => "Vimeo",
Self::Bandcamp => "Bandcamp",
Self::SoundCloud => "SoundCloud",
Self::Odysee => "Odysee",
Self::Other => "Other",
}
}
/// Single-character icon shown in the sidebar. Falls back to a generic
/// camera glyph for `Other` so unknown sources still get a visual hook.
pub fn icon(self) -> &'static str {
match self {
Self::YouTube => "▶",
Self::TikTok => "♪",
Self::Twitch => "📺",
Self::Vimeo => "🎬",
Self::Bandcamp => "🎵",
Self::SoundCloud => "☁",
Self::Odysee => "◈",
Self::Other => "📼",
}
}
/// Inverse of [`Self::dir_name`]. Available for callers that need to
/// recover the platform from a stored folder name.
#[allow(dead_code)]
pub fn from_dir_name(name: &str) -> Option<Self> {
Some(match name {
"channels" => Self::YouTube,
"tiktok" => Self::TikTok,
"twitch" => Self::Twitch,
"vimeo" => Self::Vimeo,
"bandcamp" => Self::Bandcamp,
"soundcloud" => Self::SoundCloud,
"odysee" => Self::Odysee,
"other" => Self::Other,
_ => return None,
})
}
/// Pick the platform from a URL's host portion. Substring matching is
/// intentionally loose so subdomains (`m.youtube.com`, `*.bandcamp.com`)
/// still resolve correctly.
pub fn from_url(url: &str) -> Self {
let s = url.to_lowercase();
if s.contains("youtube.com") || s.contains("youtu.be") || s.contains("youtube-nocookie.com") {
Self::YouTube
} else if s.contains("tiktok.com") || s.contains("vm.tiktok.com") {
Self::TikTok
} else if s.contains("twitch.tv") {
Self::Twitch
} else if s.contains("vimeo.com") {
Self::Vimeo
} else if s.contains(".bandcamp.com") || s.contains("//bandcamp.com") {
Self::Bandcamp
} else if s.contains("soundcloud.com") {
Self::SoundCloud
} else if s.contains("odysee.com") {
Self::Odysee
} else {
Self::Other
}
}
/// Iterate over every variant. Used by the sidebar UI and the scanner.
pub fn all() -> &'static [Platform] {
&[
Platform::YouTube,
Platform::TikTok,
Platform::Twitch,
Platform::Vimeo,
Platform::Bandcamp,
Platform::SoundCloud,
Platform::Odysee,
Platform::Other,
]
}
/// Whether downloads from this platform are audio-only by default. The
/// download dialog can use this to flip the Music toggle automatically.
#[allow(dead_code)]
pub fn is_audio_first(self) -> bool {
matches!(self, Self::Bandcamp | Self::SoundCloud)
}
}
/// What kind of URL we're looking at — drives the yt-dlp `-o` template.
///
/// Reused across all platforms; the platform comes from [`Platform::from_url`].
#[derive(Debug, Clone)]
pub enum UrlKind {
/// A creator profile / channel URL. `handle` becomes the on-disk folder.
Channel { handle: String },
/// A playlist / album / collection.
Playlist,
/// A single video / track / VOD.
Video,
/// We can't tell from the URL alone — let yt-dlp figure it out.
Unknown,
}
/// Fully classified URL: where it's from and what it points at.
#[derive(Debug, Clone)]
pub struct UrlInfo {
pub platform: Platform,
pub kind: UrlKind,
}
/// Classify any URL the user pastes. Returns a platform + URL kind.
///
/// Detection is best-effort: enough to pick an output folder and a label.
/// yt-dlp handles the actual extraction, so we don't need precise matching.
pub fn classify_url(url: &str) -> UrlInfo {
let platform = Platform::from_url(url);
let kind = match platform {
Platform::YouTube => classify_youtube(url),
Platform::TikTok => classify_tiktok(url),
Platform::Twitch => classify_twitch(url),
Platform::Vimeo => classify_vimeo(url),
Platform::Bandcamp => classify_bandcamp(url),
Platform::SoundCloud => classify_soundcloud(url),
Platform::Odysee => classify_odysee(url),
Platform::Other => UrlKind::Unknown,
};
UrlInfo { platform, kind }
}
// ── Per-platform URL → UrlKind ────────────────────────────────────────────────
fn classify_youtube(url: &str) -> UrlKind {
if url.contains("playlist?list=") { return UrlKind::Playlist; }
if let Some(h) = extract_after(url, "/@") { return UrlKind::Channel { handle: h.to_string() }; }
if let Some(h) = extract_after(url, "/channel/") { return UrlKind::Channel { handle: h.to_string() }; }
if let Some(h) = extract_after(url, "/c/") { return UrlKind::Channel { handle: h.to_string() }; }
if url.contains("watch?v=") || url.contains("youtu.be/") { return UrlKind::Video; }
UrlKind::Unknown
}
fn classify_tiktok(url: &str) -> UrlKind {
// tiktok.com/@user/video/<id> → Video (specific clip)
// tiktok.com/@user → Channel (whole profile)
if let Some(rest) = url.split("/@").nth(1) {
let handle = rest.split('/').next().unwrap_or("");
if handle.is_empty() { return UrlKind::Unknown; }
if rest.contains("/video/") { return UrlKind::Video; }
return UrlKind::Channel { handle: handle.to_string() };
}
if url.contains("vm.tiktok.com") { return UrlKind::Video; }
UrlKind::Unknown
}
fn classify_twitch(url: &str) -> UrlKind {
// twitch.tv/<user> → Channel
// twitch.tv/videos/<id> → Video (VOD)
// twitch.tv/<user>/clip/<id> → Video (clip)
if let Some(rest) = url.split("twitch.tv/").nth(1) {
let first = rest.split('/').next().unwrap_or("").trim_end_matches('?');
if first.is_empty() { return UrlKind::Unknown; }
if first == "videos" { return UrlKind::Video; }
if rest.contains("/clip/") || rest.contains("/video/") {
return UrlKind::Video;
}
// No nested path means a bare channel page.
let nested = rest.trim_start_matches(first).trim_start_matches('/');
let nested = nested.split('?').next().unwrap_or("");
if nested.is_empty() || nested == "videos" || nested == "about" {
return UrlKind::Channel { handle: first.to_string() };
}
}
UrlKind::Unknown
}
fn classify_vimeo(url: &str) -> UrlKind {
// vimeo.com/<numeric_id> → Video
// vimeo.com/user<id> → Channel
// vimeo.com/channels/<name> → Channel
// vimeo.com/<word> → Channel (profile)
if let Some(rest) = url.split("vimeo.com/").nth(1) {
let first = rest.split(|c| c == '/' || c == '?' || c == '#').next().unwrap_or("");
if first.is_empty() { return UrlKind::Unknown; }
if first == "channels" {
if let Some(name) = rest.split('/').nth(1) {
if !name.is_empty() {
return UrlKind::Channel { handle: name.to_string() };
}
}
return UrlKind::Unknown;
}
// All digits → single video.
if first.chars().all(|c| c.is_ascii_digit()) {
return UrlKind::Video;
}
return UrlKind::Channel { handle: first.to_string() };
}
UrlKind::Unknown
}
fn classify_bandcamp(url: &str) -> UrlKind {
// <artist>.bandcamp.com → Channel (whole discography)
// <artist>.bandcamp.com/track/<x> → Video (track)
// <artist>.bandcamp.com/album/<x> → Playlist (album)
if let Some(host_start) = url.find("://") {
let host_and_path = &url[host_start + 3..];
if let Some(host_end) = host_and_path.find('/') {
let host = &host_and_path[..host_end];
let path = &host_and_path[host_end..];
if let Some(artist) = host.strip_suffix(".bandcamp.com") {
if path.starts_with("/track/") { return UrlKind::Video; }
if path.starts_with("/album/") { return UrlKind::Playlist; }
return UrlKind::Channel { handle: artist.to_string() };
}
} else if let Some(artist) = host_and_path.strip_suffix(".bandcamp.com") {
return UrlKind::Channel { handle: artist.to_string() };
}
}
UrlKind::Unknown
}
fn classify_soundcloud(url: &str) -> UrlKind {
// soundcloud.com/<user> → Channel
// soundcloud.com/<user>/<track> → Video
// soundcloud.com/<user>/sets/<playlist> → Playlist
if let Some(rest) = url.split("soundcloud.com/").nth(1) {
let first = rest.split(|c| c == '/' || c == '?' || c == '#').next().unwrap_or("");
if first.is_empty() { return UrlKind::Unknown; }
if rest.contains("/sets/") { return UrlKind::Playlist; }
let tail = rest.trim_start_matches(first).trim_start_matches('/');
let tail = tail.split(|c| c == '?' || c == '#').next().unwrap_or("");
if tail.is_empty() {
return UrlKind::Channel { handle: first.to_string() };
}
UrlKind::Video
} else {
UrlKind::Unknown
}
}
fn classify_odysee(url: &str) -> UrlKind {
// odysee.com/@channel:n → Channel
// odysee.com/@channel:n/title:m → Video
if let Some(rest) = url.split("/@").nth(1) {
let head = rest.split('/').next().unwrap_or("");
if head.is_empty() { return UrlKind::Unknown; }
// Strip ":<claimid>" so the on-disk folder is just the channel handle.
let handle = head.split(':').next().unwrap_or(head).to_string();
let tail = rest.trim_start_matches(head).trim_start_matches('/');
let tail = tail.split(|c| c == '?' || c == '#').next().unwrap_or("");
if tail.is_empty() {
return UrlKind::Channel { handle };
}
UrlKind::Video
} else {
UrlKind::Unknown
}
}
fn extract_after<'a>(url: &'a str, marker: &str) -> Option<&'a str> {
let start = url.find(marker)? + marker.len();
let rest = &url[start..];
let end = rest.find(|c| c == '/' || c == '?' || c == '&' || c == '#').unwrap_or(rest.len());
if end == 0 { None } else { Some(&rest[..end]) }
}
// ── Filesystem layout ─────────────────────────────────────────────────────────
/// Absolute path to a given platform's video folder, derived from the
/// configured YouTube `channels_root` (its parent is treated as the
/// implicit library root, with each platform as a sibling folder).
pub fn platform_root(channels_root: &Path, platform: Platform) -> PathBuf {
if platform == Platform::YouTube {
// YouTube keeps the legacy path verbatim.
return channels_root.to_path_buf();
}
channels_root.with_file_name(platform.dir_name())
}
/// Where the `.source-url` sidecar lives for a creator folder.
pub fn source_url_path(creator_dir: &Path) -> PathBuf {
creator_dir.join(".source-url")
}
/// Persist the originating URL alongside the downloaded files so a future
/// re-check can recover it without heuristic URL rebuilding.
pub fn write_source_url(creator_dir: &Path, url: &str) {
let _ = std::fs::write(source_url_path(creator_dir), url.trim());
}
/// Read back a previously-written `.source-url`. Returns `None` if the file
/// is missing or unreadable.
pub fn read_source_url(creator_dir: &Path) -> Option<String> {
let s = std::fs::read_to_string(source_url_path(creator_dir)).ok()?;
let t = s.trim();
if t.is_empty() { None } else { Some(t.to_string()) }
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn youtube_urls_route_to_youtube() {
assert_eq!(Platform::from_url("https://www.youtube.com/@x"), Platform::YouTube);
assert_eq!(Platform::from_url("https://youtu.be/abc"), Platform::YouTube);
assert_eq!(Platform::from_url("https://m.youtube.com/watch?v=x"), Platform::YouTube);
}
#[test]
fn other_platforms_route_correctly() {
assert_eq!(Platform::from_url("https://www.tiktok.com/@u"), Platform::TikTok);
assert_eq!(Platform::from_url("https://www.twitch.tv/x"), Platform::Twitch);
assert_eq!(Platform::from_url("https://vimeo.com/12345"), Platform::Vimeo);
assert_eq!(Platform::from_url("https://artist.bandcamp.com/album/x"), Platform::Bandcamp);
assert_eq!(Platform::from_url("https://soundcloud.com/x"), Platform::SoundCloud);
assert_eq!(Platform::from_url("https://odysee.com/@x"), Platform::Odysee);
assert_eq!(Platform::from_url("https://example.com/video.mp4"), Platform::Other);
}
#[test]
fn tiktok_handle_extracted() {
let info = classify_url("https://www.tiktok.com/@coolperson");
assert_eq!(info.platform, Platform::TikTok);
match info.kind {
UrlKind::Channel { handle } => assert_eq!(handle, "coolperson"),
_ => panic!("expected Channel"),
}
}
#[test]
fn tiktok_video_url_classified_as_video() {
let info = classify_url("https://www.tiktok.com/@coolperson/video/7123456789");
assert!(matches!(info.kind, UrlKind::Video));
}
#[test]
fn twitch_channel_url() {
let info = classify_url("https://www.twitch.tv/streamername");
assert!(matches!(info.kind, UrlKind::Channel { .. }));
}
#[test]
fn twitch_vod_url_is_video() {
let info = classify_url("https://www.twitch.tv/videos/123456789");
assert!(matches!(info.kind, UrlKind::Video));
}
#[test]
fn vimeo_numeric_is_video() {
let info = classify_url("https://vimeo.com/123456");
assert!(matches!(info.kind, UrlKind::Video));
}
#[test]
fn vimeo_user_is_channel() {
let info = classify_url("https://vimeo.com/staffpicks");
assert!(matches!(info.kind, UrlKind::Channel { .. }));
}
#[test]
fn bandcamp_subdomain_is_channel() {
let info = classify_url("https://amazingartist.bandcamp.com/");
assert!(matches!(info.kind, UrlKind::Channel { .. }));
}
#[test]
fn bandcamp_album_is_playlist() {
let info = classify_url("https://amazingartist.bandcamp.com/album/my-album");
assert!(matches!(info.kind, UrlKind::Playlist));
}
#[test]
fn soundcloud_user_is_channel() {
let info = classify_url("https://soundcloud.com/awesomeartist");
assert!(matches!(info.kind, UrlKind::Channel { .. }));
}
#[test]
fn soundcloud_set_is_playlist() {
let info = classify_url("https://soundcloud.com/awesomeartist/sets/myset");
assert!(matches!(info.kind, UrlKind::Playlist));
}
#[test]
fn odysee_handle_strips_claim_id() {
let info = classify_url("https://odysee.com/@somebody:abc");
match info.kind {
UrlKind::Channel { handle } => assert_eq!(handle, "somebody"),
_ => panic!("expected Channel"),
}
}
#[test]
fn platform_root_youtube_is_channels_root() {
let cr = Path::new("/foo/channels");
assert_eq!(platform_root(cr, Platform::YouTube), cr);
}
#[test]
fn platform_root_others_are_siblings() {
let cr = Path::new("/foo/channels");
assert_eq!(platform_root(cr, Platform::TikTok), Path::new("/foo/tiktok"));
assert_eq!(platform_root(cr, Platform::Twitch), Path::new("/foo/twitch"));
}
}

10
src/plex.rs
View file

@ -173,7 +173,15 @@ pub fn generate(channels: &[Channel], plex_root: &Path) -> GenerateResult {
if all_videos.is_empty() { continue; } if all_videos.is_empty() { continue; }
let show_name = sanitize(&ch.name); // Prefix the show folder with the platform name when it's not YouTube
// so multi-platform libraries don't collide on shared creator names
// (e.g. a YouTube and a TikTok account both called "MusicianName").
let base_name = sanitize(&ch.name);
let show_name = if ch.platform == crate::platform::Platform::YouTube {
base_name
} else {
format!("{} - {}", ch.platform.display_name(), base_name)
};
let show_dir = plex_root.join(&show_name); let show_dir = plex_root.join(&show_name);
if let Err(e) = std::fs::create_dir_all(&show_dir) { if let Err(e) = std::fs::create_dir_all(&show_dir) {

105
src/web.rs
View file

@ -44,7 +44,8 @@ use argon2::password_hash::SaltString;
use crate::config::Config; use crate::config::Config;
use crate::database::Database; use crate::database::Database;
use crate::downloader::{detect_url_kind, DownloadQuality, Downloader, JobState}; use crate::downloader::{DownloadQuality, Downloader, JobState};
use crate::platform::classify_url;
use crate::library; use crate::library;
use crate::maintenance; use crate::maintenance;
@ -73,7 +74,12 @@ pub struct WebState {
pub positions: Mutex<HashMap<String, f64>>, pub positions: Mutex<HashMap<String, f64>>,
/// Shared SQLite connection, opened once at startup instead of per request. /// Shared SQLite connection, opened once at startup instead of per request.
pub db: Mutex<Database>, pub db: Mutex<Database>,
/// YouTube channels directory (the legacy `channels/` folder, kept for
/// backward compat). Other platforms live as siblings under [`library_root`].
pub channels_root: PathBuf, pub channels_root: PathBuf,
/// Parent of `channels_root`. Backs the `/files/` static-file mount so
/// non-YouTube platforms are reachable at `/files/<platform>/<creator>/...`.
pub library_root: PathBuf,
pub config_path: PathBuf, pub config_path: PathBuf,
pub config: Mutex<Config>, pub config: Mutex<Config>,
/// Whether to transcode MKV→mp4 on the fly for playback (requires ffmpeg). /// Whether to transcode MKV→mp4 on the fly for playback (requires ffmpeg).
@ -137,6 +143,17 @@ struct LibraryResponse {
#[derive(Serialize)] #[derive(Serialize)]
struct ChannelInfo { struct ChannelInfo {
name: String, name: String,
/// dir_name() of the channel's source platform. Used by the UI to render
/// the platform icon and group entries.
platform: &'static str,
/// Human-readable platform name (e.g. "YouTube", "TikTok").
platform_label: &'static str,
/// Platform icon used in the sidebar.
platform_icon: &'static str,
/// Original URL the channel was downloaded from, if a `.source-url` sidecar
/// exists. Used by the UI's "Check for new videos" action to avoid relying
/// on a folder-name heuristic.
source_url: Option<String>,
total_videos: usize, total_videos: usize,
size_bytes: u64, size_bytes: u64,
subscriber_count: Option<u64>, subscriber_count: Option<u64>,
@ -280,8 +297,12 @@ pub struct BindOption {
} }
// Build a `/files/<rel>` URL from an absolute path, percent-encoding each segment. // Build a `/files/<rel>` URL from an absolute path, percent-encoding each segment.
fn file_url(channels_root: &StdPath, full: &StdPath) -> Option<String> { //
let rel = full.strip_prefix(channels_root).ok()?; // `library_root` is the parent of channels_root so that paths like
// `<root>/channels/handle/video.mkv` become `/files/channels/handle/video.mkv`
// and `<root>/tiktok/user/clip.mp4` becomes `/files/tiktok/user/clip.mp4`.
fn file_url(library_root: &StdPath, full: &StdPath) -> Option<String> {
let rel = full.strip_prefix(library_root).ok()?;
let mut parts: Vec<String> = Vec::new(); let mut parts: Vec<String> = Vec::new();
for c in rel.components() { for c in rel.components() {
if let std::path::Component::Normal(s) = c { if let std::path::Component::Normal(s) = c {
@ -781,7 +802,10 @@ async fn get_library(State(state): State<Arc<WebState>>) -> impl IntoResponse {
let lib = state.library.lock().unwrap(); let lib = state.library.lock().unwrap();
let watched = state.watched.lock().unwrap(); let watched = state.watched.lock().unwrap();
let positions = state.positions.lock().unwrap(); let positions = state.positions.lock().unwrap();
let root = state.channels_root.as_path(); // file_url() now resolves relative to library_root (= parent of
// channels_root) so non-YouTube platforms are reachable at
// `/files/<platform>/<creator>/<video>`.
let root = state.library_root.as_path();
let transcode = state.transcode.load(Ordering::Relaxed); let transcode = state.transcode.load(Ordering::Relaxed);
let to_info = |v: &library::Video, watched: &HashSet<String>| { let to_info = |v: &library::Video, watched: &HashSet<String>| {
@ -834,6 +858,10 @@ async fn get_library(State(state): State<Arc<WebState>>) -> impl IntoResponse {
.find_map(|v| v.thumb_path.as_deref().and_then(|p| file_url(root, p))); .find_map(|v| v.thumb_path.as_deref().and_then(|p| file_url(root, p)));
ChannelInfo { ChannelInfo {
name: ch.name.clone(), name: ch.name.clone(),
platform: ch.platform.dir_name(),
platform_label: ch.platform.display_name(),
platform_icon: ch.platform.icon(),
source_url: ch.source_url.clone(),
total_videos: ch.total_videos(), total_videos: ch.total_videos(),
size_bytes: ch.total_size_cached, size_bytes: ch.total_size_cached,
subscriber_count: ch.meta.as_ref().and_then(|m| m.subscriber_count), subscriber_count: ch.meta.as_ref().and_then(|m| m.subscriber_count),
@ -911,8 +939,8 @@ async fn post_download(
"360p" => DownloadQuality::Res360, "360p" => DownloadQuality::Res360,
_ => DownloadQuality::Best, _ => DownloadQuality::Best,
}; };
let kind = detect_url_kind(&url); let info = classify_url(&url);
dl.start(url, &kind, body.full_scan, quality); dl.start(url, &info, body.full_scan, quality);
} }
(StatusCode::ACCEPTED, "ok").into_response() (StatusCode::ACCEPTED, "ok").into_response()
} }
@ -1084,9 +1112,9 @@ async fn get_sub_vtt(
State(state): State<Arc<WebState>>, State(state): State<Arc<WebState>>,
Path(rel): Path<String>, Path(rel): Path<String>,
) -> Response { ) -> Response {
let path = state.channels_root.join(&rel); let path = state.library_root.join(&rel);
// Reject path traversal outside the library. // Reject path traversal outside the library.
let ok = match (state.channels_root.canonicalize(), path.canonicalize()) { let ok = match (state.library_root.canonicalize(), path.canonicalize()) {
(Ok(root), Ok(p)) => p.starts_with(root), (Ok(root), Ok(p)) => p.starts_with(root),
_ => false, _ => false,
}; };
@ -1369,7 +1397,7 @@ async fn post_plex_generate(State(state): State<Arc<WebState>>) -> impl IntoResp
/// `GET /api/maintenance/scan` — report duplicate videos and missing assets. /// `GET /api/maintenance/scan` — report duplicate videos and missing assets.
async fn get_maintenance_scan(State(state): State<Arc<WebState>>) -> impl IntoResponse { async fn get_maintenance_scan(State(state): State<Arc<WebState>>) -> impl IntoResponse {
let lib = state.library.lock().unwrap(); let lib = state.library.lock().unwrap();
let report = maintenance::scan(&state.channels_root, &lib); let report = maintenance::scan(&state.library_root, &lib);
Json(report) Json(report)
} }
@ -1384,7 +1412,7 @@ async fn post_maintenance_remove(
State(state): State<Arc<WebState>>, State(state): State<Arc<WebState>>,
Json(body): Json<RemoveRequest>, Json(body): Json<RemoveRequest>,
) -> impl IntoResponse { ) -> impl IntoResponse {
let (removed, errors) = maintenance::remove_files(&state.channels_root, &body.paths); let (removed, errors) = maintenance::remove_files(&state.library_root, &body.paths);
// Refresh the library so the removed copies disappear from the UI. // Refresh the library so the removed copies disappear from the UI.
let new_lib = library::scan_channels(&state.channels_root); let new_lib = library::scan_channels(&state.channels_root);
*state.library.lock().unwrap() = new_lib; *state.library.lock().unwrap() = new_lib;
@ -1414,7 +1442,7 @@ async fn post_scheduler_run(State(state): State<Arc<WebState>>) -> impl IntoResp
} }
let urls: Vec<String> = state.library.lock().unwrap() let urls: Vec<String> = state.library.lock().unwrap()
.iter() .iter()
.map(|ch| crate::downloader::check_url_for_folder(&ch.name)) .map(|ch| crate::downloader::recheck_url(ch))
.collect(); .collect();
if urls.is_empty() { if urls.is_empty() {
return (StatusCode::OK, "no channels to check").into_response(); return (StatusCode::OK, "no channels to check").into_response();
@ -1422,8 +1450,8 @@ async fn post_scheduler_run(State(state): State<Arc<WebState>>) -> impl IntoResp
let count = urls.len(); let count = urls.len();
let mut dl = state.downloader.lock().unwrap(); let mut dl = state.downloader.lock().unwrap();
for url in urls { for url in urls {
let kind = detect_url_kind(&url); let info = classify_url(&url);
dl.start(url, &kind, true, DownloadQuality::Best); dl.start(url, &info, true, DownloadQuality::Best);
} }
*state.last_scheduled_check.lock().unwrap() = Some(std::time::Instant::now()); *state.last_scheduled_check.lock().unwrap() = Some(std::time::Instant::now());
(StatusCode::ACCEPTED, format!("started {count} channel checks")).into_response() (StatusCode::ACCEPTED, format!("started {count} channel checks")).into_response()
@ -1513,6 +1541,20 @@ pub fn run_with_shutdown(config: Config) -> std::sync::mpsc::Sender<()> {
async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) { async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
let channels_root = config.backup.directory.clone(); let channels_root = config.backup.directory.clone();
let _ = std::fs::create_dir_all(&channels_root); let _ = std::fs::create_dir_all(&channels_root);
// library_root holds every platform folder side-by-side (channels/,
// tiktok/, twitch/, …). The implicit anchor is the parent of the
// user-configured channels dir.
let library_root = channels_root
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| channels_root.clone());
let _ = std::fs::create_dir_all(&library_root);
// Pre-create every platform's folder so the static-file mount can serve
// them without 404ing on first access.
for &p in crate::platform::Platform::all() {
let dir = crate::platform::platform_root(&channels_root, p);
let _ = std::fs::create_dir_all(&dir);
}
let db_path = channels_root.join("yt-offline.db"); let db_path = channels_root.join("yt-offline.db");
let config_path = std::env::current_dir() let config_path = std::env::current_dir()
.unwrap_or_else(|_| PathBuf::from(".")) .unwrap_or_else(|_| PathBuf::from("."))
@ -1544,6 +1586,7 @@ async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
positions: Mutex::new(positions), positions: Mutex::new(positions),
db: Mutex::new(db), db: Mutex::new(db),
channels_root: channels_root.clone(), channels_root: channels_root.clone(),
library_root: library_root.clone(),
config_path, config_path,
config: Mutex::new(config), config: Mutex::new(config),
transcode, transcode,
@ -1576,13 +1619,13 @@ async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
if !due { continue; } if !due { continue; }
let urls: Vec<String> = sched_state.library.lock().unwrap() let urls: Vec<String> = sched_state.library.lock().unwrap()
.iter() .iter()
.map(|ch| crate::downloader::check_url_for_folder(&ch.name)) .map(|ch| crate::downloader::recheck_url(ch))
.collect(); .collect();
if urls.is_empty() { continue; } if urls.is_empty() { continue; }
let mut dl = sched_state.downloader.lock().unwrap(); let mut dl = sched_state.downloader.lock().unwrap();
for url in urls { for url in urls {
let kind = detect_url_kind(&url); let info = classify_url(&url);
dl.start(url, &kind, true, DownloadQuality::Best); dl.start(url, &info, true, DownloadQuality::Best);
} }
*sched_state.last_scheduled_check.lock().unwrap() = Some(std::time::Instant::now()); *sched_state.last_scheduled_check.lock().unwrap() = Some(std::time::Instant::now());
} }
@ -1616,7 +1659,10 @@ async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
.route("/api/music", get(get_music)) .route("/api/music", get(get_music))
.route("/api/login", post(post_login)) .route("/api/login", post(post_login))
.route("/api/logout", post(post_logout)) .route("/api/logout", post(post_logout))
.nest_service("/files", ServeDir::new(&channels_root)) // Serve from library_root (parent of channels_root) so URLs become
// `/files/<platform>/<creator>/...` for every platform, not just
// YouTube. ServeDir rejects `..` and refuses symlink escapes.
.nest_service("/files", ServeDir::new(&library_root))
.nest_service("/music-files", ServeDir::new(&music_root)) .nest_service("/music-files", ServeDir::new(&music_root))
.layer(middleware::from_fn_with_state(state.clone(), auth_middleware)) .layer(middleware::from_fn_with_state(state.clone(), auth_middleware))
// Cap any uploaded body at 4 MiB. cookies.txt and POSTed JSON payloads // Cap any uploaded body at 4 MiB. cookies.txt and POSTed JSON payloads
@ -1901,12 +1947,18 @@ function renderSidebar(){
h+=`<div class="ch-item${showChannels?' active':''}" onclick="setChannels()">⊟ Channels (${library.length})</div>`; h+=`<div class="ch-item${showChannels?' active':''}" onclick="setChannels()">⊟ Channels (${library.length})</div>`;
h+=`<div class="ch-item${!showContinue&&!showChannels&&activeChannelIdx===null?' active':''}" onclick="setView(null,null)">⊞ All (${total})</div>`; h+=`<div class="ch-item${!showContinue&&!showChannels&&activeChannelIdx===null?' active':''}" onclick="setView(null,null)">⊞ All (${total})</div>`;
h+=`<div class="ch-item${showMusic?' active':''}" onclick="setMusic()">♫ Music (${musicTracks.length})</div>`; h+=`<div class="ch-item${showMusic?' active':''}" onclick="setMusic()">♫ Music (${musicTracks.length})</div>`;
h+=`<div class="sidebar-label" style="margin-top:8px">Channels</div>`; // Group sidebar entries by platform so a multi-platform library reads as
// distinct sections rather than one flat list.
let lastPlatform=null;
for(let i=0;i<library.length;i++){ for(let i=0;i<library.length;i++){
const ch=library[i]; const ch=library[i];
if(ch.platform!==lastPlatform){
h+=`<div class="sidebar-label" style="margin-top:8px">${esc(ch.platform_icon||'')} ${esc(ch.platform_label||'Channels')}</div>`;
lastPlatform=ch.platform;
}
const active=activeChannelIdx===i&&activePlaylistIdx===null&&!showContinue; const active=activeChannelIdx===i&&activePlaylistIdx===null&&!showContinue;
const size=ch.size_bytes>0?' · '+fmtSize(ch.size_bytes):''; const size=ch.size_bytes>0?' · '+fmtSize(ch.size_bytes):'';
h+=`<div class="ch-item${active?' active':''}" onclick="setView(${i},null)">${esc(ch.name)} (${ch.total_videos}${size})</div>`; h+=`<div class="ch-item${active?' active':''}" onclick="setView(${i},null)" title="${esc(ch.platform_label||'')}">${esc(ch.name)} (${ch.total_videos}${size})</div>`;
if(activeChannelIdx===i&&!showContinue){ if(activeChannelIdx===i&&!showContinue){
for(let pi=0;pi<ch.playlists.length;pi++){ for(let pi=0;pi<ch.playlists.length;pi++){
const pl=ch.playlists[pi]; const pl=ch.playlists[pi];
@ -2105,13 +2157,16 @@ async function confirmDownload(url,btn){
}catch(e){setStatus('Error: '+e.message)} }catch(e){setStatus('Error: '+e.message)}
} }
async function downloadChannel(url){try{await api('/api/download',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({url,full_scan:true,quality:'best'})});setStatus('Checking for new videos…')}catch(e){setStatus('Error: '+e.message)}} async function downloadChannel(url){try{await api('/api/download',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({url,full_scan:true,quality:'best'})});setStatus('Checking for new videos…')}catch(e){setStatus('Error: '+e.message)}}
function checkUrlForChannel(name){ function rebuildChannelUrl(ch){
// Mirror check_url_for_folder in downloader.rs: UC+22 chars = channel ID, else handle // Prefer the stored source URL (works for every platform). Fall back to
return(/^UC.{22}$/.test(name)) // the legacy YouTube heuristic only when the channel pre-dates the
?'https://www.youtube.com/channel/'+name // multi-platform changes and has no .source-url sidecar.
:'https://www.youtube.com/@'+name; if(ch.source_url)return ch.source_url;
return(/^UC.{22}$/.test(ch.name))
?'https://www.youtube.com/channel/'+ch.name
:'https://www.youtube.com/@'+ch.name;
} }
async function downloadChannelByIdx(i){await downloadChannel(checkUrlForChannel(library[i].name))} async function downloadChannelByIdx(i){await downloadChannel(rebuildChannelUrl(library[i]))}
/* ── Rescan ─────────────────────────────────────────────────────── */ /* ── Rescan ─────────────────────────────────────────────────────── */
async function rescan(){try{await api('/api/rescan',{method:'POST'});await loadLibrary()}catch(e){setStatus('Error: '+e.message)}} async function rescan(){try{await api('/api/rescan',{method:'POST'});await loadLibrary()}catch(e){setStatus('Error: '+e.message)}}