Add multi-platform support: TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee
yt-dlp already speaks ~1,800 sites; the changes here teach the rest of the app to route them through platform-specific output folders and remember where each creator was downloaded from. New `platform` module - `Platform` enum: YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee, Other. - `classify_url` returns both the source platform and a `UrlKind` (Channel / Playlist / Video / Unknown) — per-platform path parsing for each supported host. - `platform_root(channels_root, platform)` returns the on-disk folder per platform. YouTube keeps the legacy `channels/` for backward compat; the others live as siblings (`tiktok/`, `twitch/`, …). - `write_source_url` / `read_source_url` persist the originating URL in a `.source-url` sidecar at each creator folder so re-checks no longer rely on the YouTube-only "UC + 22 chars" heuristic. Downloader - `Downloader::start` now takes `&UrlInfo` instead of `&UrlKind` and routes the yt-dlp `-o` template into the right platform folder. - For channel downloads it also writes `.source-url` so future re-checks recover the exact URL. - Output template uses `%(uploader,channel,creator|Unknown)s` so non-YouTube sites that don't expose `channel` still get a sensible creator folder. - Per-platform `archive.txt` keeps cross-platform IDs (TikTok numeric vs YouTube base64) from colliding. - New `recheck_url(&Channel)` prefers `source_url` and falls back to the legacy YouTube heuristic for libraries created before this change. Library scanner - `scan_channels` now walks every platform's folder and tags each `Channel` with its `platform` + `source_url`. Sort key puts the platform first, then name, so the sidebar groups cleanly. - `Channel` gains `platform: Platform` and `source_url: Option<String>` fields, populated from the on-disk sidecar. Web UI - `WebState` gains `library_root` (= parent of channels_root). The `/files/` ServeDir is mounted at library_root so non-YouTube videos resolve at `/files/<platform>/<creator>/<file>`. Existing YouTube URLs still work — they're now `/files/channels/...` instead of `/files/...` and the server-rendered JSON updates accordingly. - Maintenance scan + remove use `library_root` so non-YouTube content is included. - `ChannelInfo` JSON exposes `platform`, `platform_label`, `platform_icon`, and `source_url`. - Sidebar groups channels by platform with each platform's icon. - "Check for new videos" now uses the stored `source_url` (with the YouTube heuristic as a last-resort fallback). - Download dialog preview shows the detected source + destination folder. Desktop UI - App gains a `library_root` field mirroring the web side; maintenance ops scan/remove against it. - Sidebar channel labels are prefixed with the platform icon for non-YouTube channels; tooltip carries the platform name. - Download URL field surfaces both source platform and folder path preview as you type. Plex - Non-YouTube creators get their show folder prefixed with the platform name (e.g., `TikTok - cooluser`) so a YouTube channel and a TikTok account with the same name don't collide. Backward compatibility - Existing YouTube libraries (`channels/<handle>/...`) are untouched. New platform folders are created on first run as siblings. - Channels without a `.source-url` (i.e., everything that pre-dates this commit) fall through to the YouTube heuristic, preserving the existing re-check behavior. Tests - 14 new `platform` tests cover URL classification per platform, channel handle extraction, and `platform_root` layout. The old `detect_url_kind` / `extract_after` tests move into platform.rs since the canonical implementation lives there now. - 41 unit tests pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
4d83999edd
commit
12c32f642b
7 changed files with 699 additions and 133 deletions
|
|
@ -22,6 +22,8 @@
|
|||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use crate::platform::{self, Platform};
|
||||
|
||||
const VIDEO_EXTS: &[&str] = &["mkv", "mp4", "webm", "m4v", "mov", "avi"];
|
||||
const AUDIO_EXTS: &[&str] = &["mp3", "m4a", "opus", "flac", "ogg", "wav", "aac"];
|
||||
const THUMB_EXTS: &[&str] = &["webp", "jpg", "jpeg", "png"];
|
||||
|
|
@ -97,6 +99,11 @@ pub struct ChannelMeta {
|
|||
pub struct Channel {
|
||||
pub name: String,
|
||||
pub path: PathBuf,
|
||||
/// Source platform — drives sidebar grouping and the re-check URL.
|
||||
pub platform: Platform,
|
||||
/// Originating URL read from the `.source-url` sidecar. Falls back to a
|
||||
/// folder-name heuristic for legacy YouTube libraries that predate it.
|
||||
pub source_url: Option<String>,
|
||||
/// Videos stored directly inside the channel directory (not in a sub-folder).
|
||||
pub videos: Vec<Video>,
|
||||
/// Sub-directories that contain at least one video.
|
||||
|
|
@ -133,35 +140,37 @@ pub fn find_video<'a>(channels: &'a [Channel], id: &str) -> Option<(&'a Video, &
|
|||
None
|
||||
}
|
||||
|
||||
/// Scan `root` for channel directories and return them sorted alphabetically.
|
||||
/// Scan all platform directories rooted at (or sibling to) `youtube_root`
|
||||
/// and return every channel found, tagged with its source platform.
|
||||
///
|
||||
/// Skips hidden directories (names starting with `.`) and directories that
|
||||
/// contain no recognisable video files.
|
||||
/// The configured `youtube_root` is treated as `Platform::YouTube` for
|
||||
/// backward compatibility with libraries created before the multi-platform
|
||||
/// changes landed. Each other platform's folder lives as a sibling — see
|
||||
/// [`crate::platform::platform_root`].
|
||||
///
|
||||
/// Each channel's per-video info.json reads are parallelised across the
|
||||
/// available CPUs because that's where ~all the time goes for large
|
||||
/// libraries (one fs read + one JSON parse per video, multiplied by hundreds
|
||||
/// or thousands).
|
||||
pub fn scan_channels(root: &Path) -> Vec<Channel> {
|
||||
let Ok(entries) = std::fs::read_dir(root) else { return Vec::new() };
|
||||
let dirs: Vec<(String, PathBuf)> = entries
|
||||
.flatten()
|
||||
.filter_map(|e| {
|
||||
let path = e.path();
|
||||
if !path.is_dir() { return None; }
|
||||
let name = e.file_name().to_string_lossy().into_owned();
|
||||
if name.starts_with('.') { return None; }
|
||||
Some((name, path))
|
||||
})
|
||||
.collect();
|
||||
/// Per-channel info.json reads are parallelised across the available CPUs
|
||||
/// because that's where ~all the time goes for large libraries (one fs read
|
||||
/// + one JSON parse per video, multiplied by thousands).
|
||||
pub fn scan_channels(youtube_root: &Path) -> Vec<Channel> {
|
||||
// Gather (platform, channel-folder-name, full-path) across every platform.
|
||||
let mut dirs: Vec<(Platform, String, PathBuf)> = Vec::new();
|
||||
for &platform in Platform::all() {
|
||||
let root = platform::platform_root(youtube_root, platform);
|
||||
let Ok(entries) = std::fs::read_dir(&root) else { continue };
|
||||
for entry in entries.flatten() {
|
||||
let path = entry.path();
|
||||
if !path.is_dir() { continue; }
|
||||
let name = entry.file_name().to_string_lossy().into_owned();
|
||||
if name.starts_with('.') { continue; }
|
||||
dirs.push((platform, name, path));
|
||||
}
|
||||
}
|
||||
|
||||
// Process channels in parallel. We size the worker pool to min(channels, CPUs)
|
||||
// so a small library doesn't spin up needless threads.
|
||||
let n_workers = std::thread::available_parallelism()
|
||||
.map(|n| n.get())
|
||||
.unwrap_or(4)
|
||||
.min(dirs.len().max(1));
|
||||
let mut channels = parallel_map(dirs, n_workers, |(name, path)| {
|
||||
let mut channels = parallel_map(dirs, n_workers, |(platform, name, path)| {
|
||||
let (videos, playlists) = scan_channel_dir(&path);
|
||||
if videos.is_empty() && playlists.is_empty() { return None; }
|
||||
let meta = load_channel_meta(&videos);
|
||||
|
|
@ -172,9 +181,12 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
|
|||
.chain(playlists.iter().flat_map(|p| p.videos.iter()))
|
||||
.filter_map(|v| v.file_size)
|
||||
.sum();
|
||||
let source_url = platform::read_source_url(&path);
|
||||
Some(Channel {
|
||||
name,
|
||||
path,
|
||||
platform,
|
||||
source_url,
|
||||
videos,
|
||||
playlists,
|
||||
meta,
|
||||
|
|
@ -185,7 +197,12 @@ pub fn scan_channels(root: &Path) -> Vec<Channel> {
|
|||
.into_iter()
|
||||
.flatten()
|
||||
.collect::<Vec<_>>();
|
||||
channels.sort_by_key(|c| c.name.to_lowercase());
|
||||
// Stable order: platform first (so the sidebar groups cleanly), then name.
|
||||
channels.sort_by(|a, b| {
|
||||
let pa = a.platform as u8;
|
||||
let pb = b.platform as u8;
|
||||
pa.cmp(&pb).then_with(|| a.name.to_lowercase().cmp(&b.name.to_lowercase()))
|
||||
});
|
||||
channels
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue