luna/catacomb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Live-stream recording, recent-additions feed, per-platform impersonate profiles

Browse source 
Three roadmap items together since they touch overlapping code:

Live-stream recording
- New `live: bool` argument on Downloader::start. When true, attaches
  `--live-from-start --wait-for-video 30`, suppresses
  `--break-on-existing` (every recording is a unique file), and
  prepends a UTC timestamp suffix to the output filename so
  re-recordings of the same broadcaster don't collide.
- The job label gains a "🔴 LIVE" marker so a long-running record is
  obviously different from a VOD pull.
- Both UIs gain a "🔴 Live stream" checkbox in the download panel,
  hidden when Music mode is selected. Works for Twitch *and* YouTube
  Live since yt-dlp recognizes both from the URL.
- Web `POST /api/download` accepts a `live: bool` body field; all
  other callers (scheduled re-check, right-click "Check for new
  videos") pass false.

Recent-additions feed
- `library::Video` gains `mtime_unix: Option<u64>` — populated from
  the metadata we already read for `file_size`, so no extra fs hit.
- New `SidebarView::Recent` (desktop) + `showRecent` (web) sort the
  whole library by mtime descending and cap at 100 entries.
- Sidebar entry "🕒 Recent additions" appears only when the library
  has any dated content, so a fresh install doesn't show an empty view.

Per-platform impersonation
- New `Platform::impersonate_target()`:
  - Twitch → None (skip impersonation; their OAuth/Helix dislikes
    mismatched TLS fingerprints).
  - TikTok → "Chrome-Android-131" (mobile profile matches their
    first-party app surface).
  - Everything else → "Chrome-146:Macos-26".
- `Downloader::apply_impersonation` now takes the platform and routes
  through the override; `repair()` (always YouTube-implicit) and
  `start_music()` (uses classify_url on the URL) wire through too.
- `/api/preview` does the same classification before dispatching to
  yt-dlp.
- 3 new tests confirm Twitch → None, TikTok → mobile, YouTube → desktop.

44 unit tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Luna 2026-05-25 00:17:59 -07:00
parent c0be4bb533
commit b24ef4be67
5 changed files with 245 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

122
src/downloader.rs
View file

@ -19,7 +19,7 @@
//! | `--xattrs` | Store metadata in filesystem extended attributes |
//! | `--sponsorblock-mark all` | Mark (but don't remove) SponsorBlock segments |
//! | `--extractor-args youtube:player_client=web` | Use the web player API to avoid throttling |
//! | `--impersonate Chrome-146:Macos-26` | Impersonate a real browser for bot detection |
//! | `--impersonate <target>` | Browser TLS fingerprint per source platform (see [`crate::platform::Platform::impersonate_target`]) |
//! | `--break-on-existing` | Stop when the archive file records the video as already downloaded |
//! | `--download-archive archive.txt` | Record downloaded IDs to avoid re-downloading |
@ -30,7 +30,7 @@ use std::process::{Command, Stdio};
use std::sync::mpsc::{channel, Receiver};
use std::thread;
use crate::platform::{self, UrlInfo, UrlKind};
use crate::platform::{self, Platform, UrlInfo, UrlKind};
use crate::ytdlp_bin;
/// Video quality level passed as a `-f` format selector to yt-dlp.
@ -184,14 +184,15 @@ impl Downloader {
}
}
/// Append `--impersonate Chrome-146:Macos-26` to bypass YouTube's bot
/// detection. Both the bundled venv (which pip-installs `curl_cffi` via
/// [`ytdlp_bin::install_command`]) and a system yt-dlp with curl_cffi
/// support this. We leave it on unconditionally now; yt-dlp's
/// `--list-impersonate-targets` warning surfaces in the install log if
/// curl_cffi is unavailable, so the user has a clear path forward.
fn apply_impersonation(&self, cmd: &mut Command) {
cmd.arg("--impersonate").arg("Chrome-146:Macos-26");
/// Append `--impersonate <target>` chosen per source platform. Both the
/// bundled venv (which pip-installs `curl_cffi`) and a system yt-dlp
/// with curl_cffi can satisfy this. Platforms that prefer no
/// impersonation (e.g. Twitch's OAuth) return `None` from
/// [`Platform::impersonate_target`] and the flag is omitted.
fn apply_impersonation(&self, platform: Platform, cmd: &mut Command) {
if let Some(target) = platform.impersonate_target() {
cmd.arg("--impersonate").arg(target);
}
}
/// Append the cookie-source flags. Prefers `cookies.txt` in the working
@ -277,7 +278,13 @@ impl Downloader {
/// video — fast for routine channel checks. When `full_scan` is true the
/// flag is omitted so every video is checked individually against the
/// download archive; slower, but correctly fills gaps in the history.
pub fn start(&mut self, url: String, info: &UrlInfo, full_scan: bool, quality: DownloadQuality) {
///
/// When `live` is true, the invocation is configured to record a live
/// stream from the start: `--live-from-start --wait-for-video 30` is
/// added, `--break-on-existing` is suppressed (each recording is unique),
/// and the output filename gains a UTC timestamp suffix so re-recordings
/// of the same channel don't collide.
pub fn start(&mut self, url: String, info: &UrlInfo, full_scan: bool, quality: DownloadQuality, live: bool) {
let platform_dir = platform::platform_root(&self.channels_root, info.platform);
// Per-platform download archive keeps cross-platform IDs from colliding
// (TikTok IDs are numeric, YouTube IDs are 11-char base64, etc.).
@ -285,6 +292,16 @@ impl Downloader {
let archive_path = platform_dir.join("archive.txt");
let platform_label = info.platform.dir_name();
// Live recordings get a UTC timestamp suffix in the filename so a
// re-recording of the same stream doesn't overwrite the prior one.
// VOD downloads rely on yt-dlp's stable `%(id)s` for uniqueness.
let live_suffix = if live {
format!(" [{}]", format_compact_utc(now_unix()))
} else {
String::new()
};
let live_label = if live { " 🔴 LIVE" } else { "" };
let (out_arg, label) = match &info.kind {
UrlKind::Channel { handle } => {
let dir = platform_dir.join(handle);
@ -293,23 +310,23 @@ impl Downloader {
// guess from the folder name.
platform::write_source_url(&dir, &url);
(
format!("{}/%(title)s [%(id)s].%(ext)s", dir.display()),
format!("{}/{}/", platform_label, handle),
format!("{}/%(title)s [%(id)s]{live_suffix}.%(ext)s", dir.display()),
format!("{}/{}/{}", platform_label, handle, live_label),
)
}
UrlKind::Playlist => (
format!(
"{}/%(uploader,channel,creator|Unknown)s/%(playlist_title)s/%(title)s [%(id)s].%(ext)s",
"{}/%(uploader,channel,creator|Unknown)s/%(playlist_title)s/%(title)s [%(id)s]{live_suffix}.%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/<playlist>/", platform_label),
format!("{}/<creator>/<playlist>/{}", platform_label, live_label),
),
UrlKind::Video | UrlKind::Unknown => (
format!(
"{}/%(uploader,channel,creator|Unknown)s/%(title)s [%(id)s].%(ext)s",
"{}/%(uploader,channel,creator|Unknown)s/%(title)s [%(id)s]{live_suffix}.%(ext)s",
platform_dir.display()
),
format!("{}/<creator>/", platform_label),
format!("{}/<creator>/{}", platform_label, live_label),
),
};
@ -339,12 +356,21 @@ impl Downloader {
if let Some(fmt) = quality.format_spec() {
cmd.arg("-f").arg(fmt);
}
if !full_scan {
if live {
// Record the broadcast from the start instead of joining live.
// `--wait-for-video` polls the URL every 30 s until a stream
// is actually live, so scheduling a recording before the
// stream begins works naturally.
cmd.arg("--live-from-start").arg("--wait-for-video").arg("30");
} else if !full_scan {
// Live recordings should never short-circuit on existing archive
// entries — every recording is its own file. Only honor
// `--break-on-existing` for VOD/channel-check downloads.
cmd.arg("--break-on-existing");
}
cmd.arg("--download-archive")
.arg(archive_path.display().to_string());
self.apply_impersonation(&mut cmd);
self.apply_impersonation(info.platform, &mut cmd);
cmd.arg("-o").arg(&out_arg).arg(&url);
Self::apply_retry_flags(&mut cmd);
@ -374,7 +400,10 @@ impl Downloader {
.arg("--write-auto-subs")
.arg("--extractor-args")
.arg("youtube:player_client=web");
self.apply_impersonation(&mut cmd);
// `repair()` rebuilds a YouTube watch URL from a stored video ID, so
// the source platform is always YouTube here regardless of where the
// original video lives on disk.
self.apply_impersonation(Platform::YouTube, &mut cmd);
cmd.arg("-o").arg(&out_arg).arg(&url);
Self::apply_retry_flags(&mut cmd);
@ -411,7 +440,11 @@ impl Downloader {
.arg("--xattrs")
.arg("--extractor-args")
.arg("youtube:player_client=web");
self.apply_impersonation(&mut cmd);
// Music downloads can come from any audio-first platform — classify
// the URL once so SoundCloud/Bandcamp pulls get their appropriate
// (typically no-op) impersonation profile.
let platform = platform::classify_url(&url).platform;
self.apply_impersonation(platform, &mut cmd);
cmd.arg("--progress")
.arg("--download-archive")
.arg(archive_path.display().to_string())
@ -541,6 +574,53 @@ impl Downloader {
}
}
/// Current UNIX timestamp in seconds. Used to disambiguate live-recording
/// filenames at job-start time.
fn now_unix() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0)
}
/// Format a UNIX timestamp as `YYYYMMDD-HHMMSS` (UTC) for embedding in
/// filenames. No `chrono` dep — short manual calendar walk; good enough
/// for human-readable filename suffixes.
fn format_compact_utc(unix: u64) -> String {
let day = unix / 86_400;
let day_secs = unix % 86_400;
let hour = day_secs / 3600;
let minute = (day_secs % 3600) / 60;
let second = day_secs % 60;
let mut year = 1970u32;
let mut remaining_days = day;
loop {
let leap = is_leap(year);
let yd = if leap { 366 } else { 365 };
if remaining_days < yd as u64 { break; }
remaining_days -= yd as u64;
year += 1;
}
let months: [u8; 12] = if is_leap(year) {
[31,29,31,30,31,30,31,31,30,31,30,31]
} else {
[31,28,31,30,31,30,31,31,30,31,30,31]
};
let mut month = 0usize;
while month < 12 && remaining_days >= months[month] as u64 {
remaining_days -= months[month] as u64;
month += 1;
}
let day_of_month = remaining_days as u32 + 1;
format!(
"{:04}{:02}{:02}-{:02}{:02}{:02}",
year, month as u32 + 1, day_of_month, hour, minute, second
)
}
fn is_leap(y: u32) -> bool { (y % 4 == 0 && y % 100 != 0) || y % 400 == 0 }
/// Parse a yt-dlp `[download] 42.7% …` line into a `[0.0, 1.0]` fraction.
fn parse_progress(line: &str) -> Option<f32> {
let rest = line.trim_start().strip_prefix("[download]")?.trim_start();