luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add multi-platform support: TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee

Browse source 
yt-dlp already speaks ~1,800 sites; the changes here teach the rest of
the app to route them through platform-specific output folders and
remember where each creator was downloaded from.

New `platform` module
- `Platform` enum: YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud,
  Odysee, Other.
- `classify_url` returns both the source platform and a `UrlKind`
  (Channel / Playlist / Video / Unknown) — per-platform path parsing
  for each supported host.
- `platform_root(channels_root, platform)` returns the on-disk folder
  per platform. YouTube keeps the legacy `channels/` for backward
  compat; the others live as siblings (`tiktok/`, `twitch/`, …).
- `write_source_url` / `read_source_url` persist the originating URL
  in a `.source-url` sidecar at each creator folder so re-checks no
  longer rely on the YouTube-only "UC + 22 chars" heuristic.

Downloader
- `Downloader::start` now takes `&UrlInfo` instead of `&UrlKind` and
  routes the yt-dlp `-o` template into the right platform folder.
- For channel downloads it also writes `.source-url` so future
  re-checks recover the exact URL.
- Output template uses `%(uploader,channel,creator|Unknown)s` so
  non-YouTube sites that don't expose `channel` still get a sensible
  creator folder.
- Per-platform `archive.txt` keeps cross-platform IDs (TikTok numeric
  vs YouTube base64) from colliding.
- New `recheck_url(&Channel)` prefers `source_url` and falls back to
  the legacy YouTube heuristic for libraries created before this
  change.

Library scanner
- `scan_channels` now walks every platform's folder and tags each
  `Channel` with its `platform` + `source_url`. Sort key puts the
  platform first, then name, so the sidebar groups cleanly.
- `Channel` gains `platform: Platform` and `source_url: Option<String>`
  fields, populated from the on-disk sidecar.

Web UI
- `WebState` gains `library_root` (= parent of channels_root). The
  `/files/` ServeDir is mounted at library_root so non-YouTube videos
  resolve at `/files/<platform>/<creator>/<file>`. Existing YouTube
  URLs still work — they're now `/files/channels/...` instead of
  `/files/...` and the server-rendered JSON updates accordingly.
- Maintenance scan + remove use `library_root` so non-YouTube content
  is included.
- `ChannelInfo` JSON exposes `platform`, `platform_label`,
  `platform_icon`, and `source_url`.
- Sidebar groups channels by platform with each platform's icon.
- "Check for new videos" now uses the stored `source_url` (with the
  YouTube heuristic as a last-resort fallback).
- Download dialog preview shows the detected source + destination
  folder.

Desktop UI
- App gains a `library_root` field mirroring the web side; maintenance
  ops scan/remove against it.
- Sidebar channel labels are prefixed with the platform icon for
  non-YouTube channels; tooltip carries the platform name.
- Download URL field surfaces both source platform and folder path
  preview as you type.

Plex
- Non-YouTube creators get their show folder prefixed with the
  platform name (e.g., `TikTok - cooluser`) so a YouTube channel and
  a TikTok account with the same name don't collide.

Backward compatibility
- Existing YouTube libraries (`channels/<handle>/...`) are untouched.
  New platform folders are created on first run as siblings.
- Channels without a `.source-url` (i.e., everything that pre-dates
  this commit) fall through to the YouTube heuristic, preserving the
  existing re-check behavior.

Tests
- 14 new `platform` tests cover URL classification per platform,
  channel handle extraction, and `platform_root` layout. The old
  `detect_url_kind` / `extract_after` tests move into platform.rs
  since the canonical implementation lives there now.
- 41 unit tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-24 22:16:40 -07:00
parent 4d83999edd
commit 12c32f642b
7 changed files with 699 additions and 133 deletions
Download patch file Download diff file Expand all files Collapse all files

106
src/downloader.rs
View file

@ -30,6 +30,7 @@ use std::process::{Command, Stdio};
use std::sync::mpsc::{channel, Receiver};
use std::thread;
use crate::platform::{self, UrlInfo, UrlKind};
use crate::ytdlp_bin;
/// Video quality level passed as a `-f` format selector to yt-dlp.
@ -70,23 +71,14 @@ impl DownloadQuality {
}
}
/// Describes the kind of YouTube URL being downloaded, which determines the
/// output path template passed to yt-dlp.
pub enum UrlKind {
/// A channel URL (`/@handle`, `/channel/ID`, or `/c/name`).
Channel { handle: String },
Playlist,
Video,
Unknown,
}
/// Build a YouTube URL from a library folder name that yt-dlp will resolve to
/// the same folder it already downloaded to.
/// Build a YouTube URL from a legacy library folder name. Used as a fallback
/// when a channel folder has no `.source-url` sidecar (i.e. it predates the
/// multi-platform changes).
///
/// Folder names that look like a channel ID (`UC` + 22 chars) use the
/// `/channel/` form; everything else is treated as a handle and gets `/@`.
/// This avoids the mismatch where info.json's canonical `channel_url` field
/// points to `/channel/UCxxx` and yt-dlp then creates a second folder.
/// points to `/channel/UCxxx` and yt-dlp creates a second folder.
pub fn check_url_for_folder(folder_name: &str) -> String {
if folder_name.starts_with("UC") && folder_name.len() == 24 {
format!("https://www.youtube.com/channel/{folder_name}")
@ -95,32 +87,17 @@ pub fn check_url_for_folder(folder_name: &str) -> String {
}
}
/// Classify a YouTube URL into a [`UrlKind`] by inspecting its path.
pub fn detect_url_kind(url: &str) -> UrlKind {
if url.contains("playlist?list=") {
return UrlKind::Playlist;
/// Re-check URL for a [`crate::library::Channel`]. Prefers the stored
/// `.source-url` sidecar when present, falls back to the YouTube heuristic
/// for legacy folders.
pub fn recheck_url(ch: &crate::library::Channel) -> String {
if let Some(url) = ch.source_url.as_deref() {
return url.to_string();
}
if let Some(h) = extract_after(url, "/@") {
return UrlKind::Channel { handle: h.to_string() };
}
if let Some(h) = extract_after(url, "/channel/") {
return UrlKind::Channel { handle: h.to_string() };
}
if let Some(h) = extract_after(url, "/c/") {
return UrlKind::Channel { handle: h.to_string() };
}
if url.contains("watch?v=") || url.contains("youtu.be/") {
return UrlKind::Video;
}
UrlKind::Unknown
// Legacy YouTube libraries: rebuild from folder name.
check_url_for_folder(&ch.name)
}
fn extract_after<'a>(url: &'a str, marker: &str) -> Option<&'a str> {
let start = url.find(marker)? + marker.len();
let rest = &url[start..];
let end = rest.find(|c| c == '/' || c == '?' || c == '&' || c == '#').unwrap_or(rest.len());
if end == 0 { None } else { Some(&rest[..end]) }
}
/// Lifecycle state of a download job.
#[derive(Clone, Copy, PartialEq, Eq)]
@ -278,33 +255,51 @@ impl Downloader {
/// Spawn a yt-dlp process for `url` and track it as a new [`Job`].
///
/// The output path template is derived from `kind` so that channels,
/// playlists, and individual videos land in the right sub-directories.
/// Output path template is derived from `info.platform` + `info.kind` so
/// each platform lands in its own sibling directory (`channels/` for
/// YouTube, `tiktok/`, `twitch/`, etc. for others).
///
/// For channel downloads we also drop a `.source-url` sidecar so future
/// re-checks recover the original URL without folder-name guessing.
///
/// When `full_scan` is false (default / incremental mode) `--break-on-existing`
/// is passed so yt-dlp stops as soon as it hits the first already-archived
/// video — fast for routine channel checks. When `full_scan` is true the
/// flag is omitted so every video is checked individually against the
/// download archive; slower, but correctly fills gaps in the history.
pub fn start(&mut self, url: String, kind: &UrlKind, full_scan: bool, quality: DownloadQuality) {
let archive_path = self.channels_root.join("archive.txt");
pub fn start(&mut self, url: String, info: &UrlInfo, full_scan: bool, quality: DownloadQuality) {
let platform_dir = platform::platform_root(&self.channels_root, info.platform);
// Per-platform download archive keeps cross-platform IDs from colliding
// (TikTok IDs are numeric, YouTube IDs are 11-char base64, etc.).
let _ = std::fs::create_dir_all(&platform_dir);
let archive_path = platform_dir.join("archive.txt");
let platform_label = info.platform.dir_name();
let (out_arg, label) = match kind {
let (out_arg, label) = match &info.kind {
UrlKind::Channel { handle } => {
let dir = self.channels_root.join(handle);
let dir = platform_dir.join(handle);
let _ = std::fs::create_dir_all(&dir);
// Remember the originating URL so re-checks don't have to
// guess from the folder name.
platform::write_source_url(&dir, &url);
(
format!("{}/%(title)s [%(id)s].%(ext)s", dir.display()),
format!("channels/{}/", handle),
format!("{}/{}/", platform_label, handle),
)
}
UrlKind::Playlist => (
format!("{}/%(channel)s/%(playlist_title)s/%(title)s [%(id)s].%(ext)s", self.channels_root.display()),
"channels/<channel>/<playlist>/".to_string(),
format!(
"{}/%(uploader,channel,creator|Unknown)s/%(playlist_title)s/%(title)s [%(id)s].%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/<playlist>/", platform_label),
),
UrlKind::Video | UrlKind::Unknown => (
format!("{}/%(channel)s/%(title)s [%(id)s].%(ext)s", self.channels_root.display()),
"channels/<channel>/".to_string(),
format!(
"{}/%(uploader,channel,creator|Unknown)s/%(title)s [%(id)s].%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/", platform_label),
),
};
@ -573,14 +568,6 @@ mod tests {
assert!(parse_progress("").is_none());
}
#[test]
fn extract_after_strips_at_separator() {
assert_eq!(extract_after("https://youtube.com/@handle/videos", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle", "/@"), Some("handle"));
assert_eq!(extract_after("https://youtube.com/@handle?x=1", "/@"), Some("handle"));
assert_eq!(extract_after("nothing-here", "/@"), None);
}
#[test]
fn check_url_for_folder_picks_channel_form_for_ids() {
let url = check_url_for_folder("UC1234567890123456789012");
@ -592,12 +579,5 @@ mod tests {
let url = check_url_for_folder("LinusTechTips");
assert_eq!(url, "https://www.youtube.com/@LinusTechTips");
}
#[test]
fn detect_url_kind_classifies_paths() {
assert!(matches!(detect_url_kind("https://www.youtube.com/playlist?list=PL1"), UrlKind::Playlist));
assert!(matches!(detect_url_kind("https://www.youtube.com/watch?v=abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://youtu.be/abc"), UrlKind::Video));
assert!(matches!(detect_url_kind("https://www.youtube.com/@handle"), UrlKind::Channel { .. }));
}
// URL classification tests live in `platform` now — see its tests module.
}