luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add multi-platform support: TikTok, Twitch, Vimeo, Bandcamp, SoundCloud, Odysee

Browse source 
yt-dlp already speaks ~1,800 sites; the changes here teach the rest of
the app to route them through platform-specific output folders and
remember where each creator was downloaded from.

New `platform` module
- `Platform` enum: YouTube, TikTok, Twitch, Vimeo, Bandcamp, SoundCloud,
  Odysee, Other.
- `classify_url` returns both the source platform and a `UrlKind`
  (Channel / Playlist / Video / Unknown) — per-platform path parsing
  for each supported host.
- `platform_root(channels_root, platform)` returns the on-disk folder
  per platform. YouTube keeps the legacy `channels/` for backward
  compat; the others live as siblings (`tiktok/`, `twitch/`, …).
- `write_source_url` / `read_source_url` persist the originating URL
  in a `.source-url` sidecar at each creator folder so re-checks no
  longer rely on the YouTube-only "UC + 22 chars" heuristic.

Downloader
- `Downloader::start` now takes `&UrlInfo` instead of `&UrlKind` and
  routes the yt-dlp `-o` template into the right platform folder.
- For channel downloads it also writes `.source-url` so future
  re-checks recover the exact URL.
- Output template uses `%(uploader,channel,creator|Unknown)s` so
  non-YouTube sites that don't expose `channel` still get a sensible
  creator folder.
- Per-platform `archive.txt` keeps cross-platform IDs (TikTok numeric
  vs YouTube base64) from colliding.
- New `recheck_url(&Channel)` prefers `source_url` and falls back to
  the legacy YouTube heuristic for libraries created before this
  change.

Library scanner
- `scan_channels` now walks every platform's folder and tags each
  `Channel` with its `platform` + `source_url`. Sort key puts the
  platform first, then name, so the sidebar groups cleanly.
- `Channel` gains `platform: Platform` and `source_url: Option<String>`
  fields, populated from the on-disk sidecar.

Web UI
- `WebState` gains `library_root` (= parent of channels_root). The
  `/files/` ServeDir is mounted at library_root so non-YouTube videos
  resolve at `/files/<platform>/<creator>/<file>`. Existing YouTube
  URLs still work — they're now `/files/channels/...` instead of
  `/files/...` and the server-rendered JSON updates accordingly.
- Maintenance scan + remove use `library_root` so non-YouTube content
  is included.
- `ChannelInfo` JSON exposes `platform`, `platform_label`,
  `platform_icon`, and `source_url`.
- Sidebar groups channels by platform with each platform's icon.
- "Check for new videos" now uses the stored `source_url` (with the
  YouTube heuristic as a last-resort fallback).
- Download dialog preview shows the detected source + destination
  folder.

Desktop UI
- App gains a `library_root` field mirroring the web side; maintenance
  ops scan/remove against it.
- Sidebar channel labels are prefixed with the platform icon for
  non-YouTube channels; tooltip carries the platform name.
- Download URL field surfaces both source platform and folder path
  preview as you type.

Plex
- Non-YouTube creators get their show folder prefixed with the
  platform name (e.g., `TikTok - cooluser`) so a YouTube channel and
  a TikTok account with the same name don't collide.

Backward compatibility
- Existing YouTube libraries (`channels/<handle>/...`) are untouched.
  New platform folders are created on first run as siblings.
- Channels without a `.source-url` (i.e., everything that pre-dates
  this commit) fall through to the YouTube heuristic, preserving the
  existing re-check behavior.

Tests
- 14 new `platform` tests cover URL classification per platform,
  channel handle extraction, and `platform_root` layout. The old
  `detect_url_kind` / `extract_after` tests move into platform.rs
  since the canonical implementation lives there now.
- 41 unit tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-24 22:16:40 -07:00
parent 4d83999edd
commit 12c32f642b
7 changed files with 699 additions and 133 deletions
Download patch file Download diff file Expand all files Collapse all files

84
src/app.rs
View file

@ -14,7 +14,8 @@ use eframe::egui;
use crate::config::Config;
use crate::database::Database;
use crate::downloader::{detect_url_kind, DownloadQuality, Downloader, JobState, UrlKind};
use crate::downloader::{DownloadQuality, Downloader, JobState};
use crate::platform::{self, classify_url, Platform, UrlKind};
use crate::library::{self, Video};
use crate::theme;
@ -67,6 +68,10 @@ pub struct App {
config: Config,
config_path: PathBuf,
channels_root: PathBuf,
/// Parent of `channels_root`. Owns every platform's sibling directory
/// (`channels/`, `tiktok/`, …). Used as the maintenance scan root so
/// non-YouTube content is included.
library_root: PathBuf,
library: Vec<library::Channel>,
sidebar_view: SidebarView,
selected_video: Option<String>,
@ -145,6 +150,16 @@ impl App {
let channels_root = config.backup.directory.clone();
let settings_dir = channels_root.display().to_string();
let _ = std::fs::create_dir_all(&channels_root);
let library_root = channels_root
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| channels_root.clone());
let _ = std::fs::create_dir_all(&library_root);
// Pre-create every platform's folder so scans see them.
for &p in Platform::all() {
let dir = platform::platform_root(&channels_root, p);
let _ = std::fs::create_dir_all(&dir);
}
let library = library::scan_channels(&channels_root);
let status = format!(
"{} channels, {} videos",
@ -192,6 +207,7 @@ impl App {
config,
config_path,
channels_root: channels_root.clone(),
library_root,
library,
sidebar_view: SidebarView::All,
selected_video: None,
@ -547,11 +563,11 @@ impl App {
fn run_scheduled_check(&mut self) {
let mut count = 0;
let urls: Vec<String> = self.library.iter()
.map(|ch| crate::downloader::check_url_for_folder(&ch.name))
.map(|ch| crate::downloader::recheck_url(ch))
.collect();
for url in urls {
let kind = detect_url_kind(&url);
self.downloader.start(url, &kind, true, DownloadQuality::Best);
let info = classify_url(&url);
self.downloader.start(url, &info, true, DownloadQuality::Best);
count += 1;
}
self.status = format!("Scheduled check: started {} channel downloads", count);
@ -636,7 +652,7 @@ impl App {
self.show_maintenance = !self.show_maintenance;
if self.show_maintenance {
self.health_report =
Some(crate::maintenance::scan(&self.channels_root, &self.library));
Some(crate::maintenance::scan(&self.library_root, &self.library));
}
}
if ui.selectable_label(self.show_settings, "⚙ Settings").clicked() {
@ -734,17 +750,19 @@ impl App {
let mut pending_ch_download: Option<(String, String)> = None; // (url, channel_name)
for i in 0..self.library.len() {
let (name, total, has_playlists, size_bytes, channel_url) = {
let (name, total, has_playlists, size_bytes, channel_url, platform) = {
let ch = &self.library[i];
// Always derive the check URL from the folder name so yt-dlp
// writes to the existing folder, not a new UCxxx one.
let url = crate::downloader::check_url_for_folder(&ch.name);
// Prefer the stored `.source-url` over folder-name guessing so
// re-checks work across platforms (and across legacy YouTube
// libraries where the URL was never recorded).
let url = crate::downloader::recheck_url(ch);
(
ch.name.clone(),
ch.total_videos(),
!ch.playlists.is_empty(),
Self::channel_total_size(ch),
url,
ch.platform,
)
};
@ -756,12 +774,24 @@ impl App {
} else {
String::new()
};
let label = format!("{} ({}{})", name, total, size_str);
// Show platform icon for non-YouTube channels so the sidebar
// reads as a unified library while still making the source
// obvious at a glance.
let prefix = if platform == Platform::YouTube {
String::new()
} else {
format!("{} ", platform.icon())
};
let label = format!("{prefix}{} ({}{})", name, total, size_str);
let ch_selected_no_pl = matches!(self.sidebar_view, SidebarView::Channel(ci) if ci == i);
let resp = ui
.selectable_label(ch_selected_no_pl, label)
.on_hover_text(self.library[i].path.display().to_string());
.on_hover_text(format!(
"{}\n{}",
self.library[i].path.display(),
platform.display_name(),
));
if resp.clicked() {
self.sidebar_view = SidebarView::Channel(i);
self.selected_video = None;
@ -805,8 +835,8 @@ impl App {
// Process deferred right-click download action
if let Some((url, ch_name)) = pending_ch_download {
let kind = detect_url_kind(&url);
self.downloader.start(url, &kind, !self.dl_full_scan, DownloadQuality::Best);
let info = classify_url(&url);
self.downloader.start(url, &info, !self.dl_full_scan, DownloadQuality::Best);
self.status = format!("Checking {} for new videos…", ch_name);
}
});
@ -827,18 +857,22 @@ impl App {
.desired_width(f32::INFINITY),
);
let kind = detect_url_kind(self.dl_url.trim());
let (type_label, dest_preview) = match &kind {
let info = classify_url(self.dl_url.trim());
let plat_dir = info.platform.dir_name();
let (type_label, dest_preview) = match &info.kind {
UrlKind::Channel { handle } => {
("Channel", format!("→ channels/{}/", handle))
("Channel", format!("→ {plat_dir}/{handle}/"))
}
UrlKind::Playlist => ("Playlist", "→ channels/<channel>/<playlist>/".to_string()),
UrlKind::Video => ("Video", "→ channels/<channel>/".to_string()),
UrlKind::Playlist => ("Playlist", format!("→ {plat_dir}/<creator>/<playlist>/")),
UrlKind::Video => ("Video", format!("→ {plat_dir}/<creator>/")),
UrlKind::Unknown => ("—", String::new()),
};
if !self.dl_url.trim().is_empty() {
ui.horizontal(|ui| {
ui.label("Source:");
ui.strong(format!("{} {}", info.platform.icon(), info.platform.display_name()));
ui.separator();
ui.label("Type:");
ui.strong(type_label);
});
@ -877,7 +911,7 @@ impl App {
self.downloader.start_music(url);
self.status = "Downloading music…".to_string();
} else {
self.downloader.start(url, &kind, !self.dl_full_scan, self.dl_quality);
self.downloader.start(url, &info, !self.dl_full_scan, self.dl_quality);
self.status = format!("Downloading: {dest}");
}
}
@ -1062,7 +1096,7 @@ impl App {
let mut changed = false;
if !to_remove.is_empty() {
let (removed, errors) =
crate::maintenance::remove_files(&self.channels_root, &to_remove);
crate::maintenance::remove_files(&self.library_root, &to_remove);
self.status = if errors.is_empty() {
format!("Removed {removed} file(s)")
} else {
@ -1082,7 +1116,7 @@ impl App {
if changed || rescan_health {
self.rescan();
self.health_report =
Some(crate::maintenance::scan(&self.channels_root, &self.library));
Some(crate::maintenance::scan(&self.library_root, &self.library));
}
}
@ -1527,8 +1561,16 @@ impl App {
self.downloader.use_bundled_ytdlp = self.config.backup.use_bundled_ytdlp;
if dir_changed {
self.channels_root = new_dir.clone();
self.library_root = new_dir
.parent()
.map(|p| p.to_path_buf())
.unwrap_or_else(|| new_dir.clone());
self.downloader.channels_root = new_dir;
let _ = std::fs::create_dir_all(&self.channels_root);
let _ = std::fs::create_dir_all(&self.library_root);
for &p in Platform::all() {
let _ = std::fs::create_dir_all(platform::platform_root(&self.channels_root, p));
}
self.rescan();
}
// Re-bind a running server so the new interface takes effect now.