Smart auto-tagging: suggest folder groups for unfiled channels (3.4)
New autotag.rs classifies each unfiled channel from already-scanned metadata — source platform + median video duration + upload cadence — into a suggested folder group (Music / Shorts / Long-form & Podcasts / Streams & VODs), with a confidence and a human-readable reason. Mid-length YouTube is left unsuggested rather than guessed at; channels already in a folder are skipped. Pure arithmetic over the in-memory library, computed on demand (no background job). Surfaced in both Maintenance views: - web: GET /api/autotag/suggest + POST /api/autotag/apply (create/reuse the named folder and assign), rendered with per-channel checkboxes and an "Apply -> group" button that re-analyzes after applying. - desktop: a section listing each group with a "Move all -> group" button. Apply mirrors post_assign_folder: DB write + in-memory library update + ETag bump. Reuses an existing folder of the same name (case-insensitive). Unit tests cover the classifier; verified the apply/revert round-trip live. ROADMAP 3.4 marked DONE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Luna
parent
aed577ea2f
commit
ec8cf6f934
7 changed files with 658 additions and 4 deletions
167
AGENTS.md
Normal file
167
AGENTS.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# AGENTS.md
|
||||
|
||||
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
||||
|
||||
## What this is
|
||||
|
||||
`yt-offline` — a single Rust binary that is **both** a desktop GUI (eframe/egui)
|
||||
and a headless web server (axum), wrapping `yt-dlp` to archive YouTube/TikTok/
|
||||
Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in
|
||||
[ROADMAP.md](ROADMAP.md); a structured analysis of the Tartube benchmark is in
|
||||
[docs/tartube-spec.md](docs/tartube-spec.md).
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
cargo build --release # the binary (release profile is opt-level=3 + thin LTO, ~1-2 min)
|
||||
cargo test --release # unit tests + tests/api.rs integration tests (no network)
|
||||
cargo test --release <name> # single test by substring, e.g. `cargo test --release subs_disabled`
|
||||
cargo test --release real_ffmpeg -- --ignored # opt-in: real-ffmpeg fingerprint accuracy/speed check
|
||||
./target/release/yt-offline # desktop GUI mode (default)
|
||||
./target/release/yt-offline --web 8080 # headless web server on a port
|
||||
|
||||
scripts/package.sh [deb|rpm|appimage|all] # build distro packages → dist/ (see docs/PACKAGING.md)
|
||||
```
|
||||
|
||||
There is no separate lint step; `cargo build` warnings are the lint. The egui
|
||||
dependency emits ~39 `f32: From<f64>` fallback warnings on a clean build — those
|
||||
are upstream, not from this code.
|
||||
|
||||
### Running / verifying against a real library
|
||||
|
||||
The app reads `config.toml` and `cookies.txt` **from the process working
|
||||
directory**, not a fixed path. To smoke-test the web server in isolation, make a
|
||||
scratch dir with a minimal `config.toml` (`[backup]\ndirectory = "..."`) and run
|
||||
`--web <port>` from inside it. `/api/*` endpoints require auth when a password is
|
||||
set in the target library's DB.
|
||||
|
||||
`tests/api.rs` automates exactly this: it spawns the real `--web` binary against a
|
||||
throwaway dir and drives the HTTP API with `curl` (skipping if curl is absent; the
|
||||
dedup test also needs ffmpeg and skips without it). That's the place to add
|
||||
end-to-end coverage of a new endpoint.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Two front-ends, one engine
|
||||
|
||||
`main.rs` dispatches: `--web` → `web::run()` (axum, blocks forever); otherwise
|
||||
`app::App` (eframe). **Both share `downloader::Downloader`**, the single source
|
||||
of truth for yt-dlp job lifecycle. `Downloader` is *not* async — it spawns an OS
|
||||
thread per `yt-dlp` process, streams stdout/stderr back over an `mpsc` channel
|
||||
into each `Job`'s log buffer, and the caller must pump `Downloader::poll()`
|
||||
regularly (the egui frame loop and a web background task both do this). When you
|
||||
add download behavior, it goes in `Downloader` and is automatically available to
|
||||
both UIs.
|
||||
|
||||
`poll()` also drives the cross-cutting job machinery: **auto-retry** of transient
|
||||
failures (rate-limit/network) with cooldown + adaptive throttle, the
|
||||
**post-download ffmpeg transcode** pass, and the **hang watchdog** (a job whose
|
||||
`last_activity` is silent past `HANG_TIMEOUT` is SIGKILLed and re-queued —
|
||||
classified `NetworkError` so the existing retry path re-issues it). The first two
|
||||
work by capturing specs onto the `Job` at `start()` time (`RetrySpec`,
|
||||
`ConvertSpec`) and acting on them when the job transitions state — the
|
||||
`pending_*_spec` fields on `Downloader` are stashed by `start()` and consumed by
|
||||
`enqueue()`/`spawn_job()` so the four non-download enqueue paths
|
||||
(repair/music/yt-dlp-update/pot-update) stay untouched.
|
||||
|
||||
### Settings flow (the easy thing to get wrong)
|
||||
|
||||
Almost every feature has the same five-touchpoint shape — miss one and it
|
||||
silently half-works:
|
||||
|
||||
1. `config.rs` — a field/section + its `Default` and the `default_with_dir()` constructor.
|
||||
2. `download_options.rs` — an `Option<…>` per-channel override (None = defer to global).
|
||||
3. `downloader.rs` — a resolver that merges global config + per-channel override into yt-dlp/ffmpeg args, plus a `pub` field on `Downloader` holding the global default.
|
||||
4. **Both** UIs render the global setting *and* the per-channel override: desktop in `app.rs` (egui Settings screen + the channel-options dialog), web in `web_ui/index.html` (Settings modal + channel-options dialog) **and** `web.rs`'s `SettingsPayload` struct (GET reads from config, POST writes config + pushes the value onto the live `Downloader`).
|
||||
5. Seed the `Downloader` field at construction **and** on settings-save, in **both** `app.rs` and `web.rs`.
|
||||
|
||||
Grep an existing setting end-to-end before adding one — `subtitle_defaults`,
|
||||
`youtube_player_clients`, `sponsorblock_mode`, and `convert_defaults` are
|
||||
complete worked examples.
|
||||
|
||||
### Filesystem layout invariant
|
||||
|
||||
`platform::platform_root(channels_root, platform)` = `channels_root.join(dir_name)`.
|
||||
**All** platforms (including YouTube, whose `dir_name` is `channels`) nest under
|
||||
the one configured `backup.directory`. `library_root == channels_root` now (a
|
||||
historical two-level split was removed). `.source-url` sidecars in each creator
|
||||
folder let channel re-checks recover the exact URL. Library scanning
|
||||
(`library.rs`) is parallel and consults a `(path, mtime)` SQLite cache to skip
|
||||
re-parsing unchanged `info.json` sidecars.
|
||||
|
||||
### Persistence
|
||||
|
||||
`database.rs` wraps an r2d2 SQLite pool (file-backed; `Database` is cheaply
|
||||
`Clone` — the pool is an `Arc`, so the parallel scanner takes its own handle).
|
||||
Schema lives in `init_schema()`; new columns are added via idempotent
|
||||
`ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no migration
|
||||
framework). **Exception:** FTS5 virtual tables have no `ADD COLUMN`, so the
|
||||
`video_search` index is migrated by dropping + recreating it and clearing
|
||||
`search_meta` to force a one-time reindex (it's a derived cache — safe to rebuild;
|
||||
see the `transcript`-column migration). The web UI holds library/notes snapshots
|
||||
in memory and mutating endpoints mirror DB writes onto those caches +
|
||||
`bump_library_version()` (the ETag) so `/api/library` stays consistent without a
|
||||
rescan.
|
||||
|
||||
Three `(path|video_id, mtime)`-keyed caches let expensive work happen once and be
|
||||
skipped on unchanged files: `info_cache` (parsed `info.json` fields),
|
||||
`search_meta` + the `video_search` FTS5 index (full-text title/channel/
|
||||
description/**transcript**), and `video_fingerprint` (perceptual dHashes). Each
|
||||
has a `sync_*`/rebuild helper that diffs the current library against stored mtimes,
|
||||
processes only the new/changed rows, and prunes vanished ones.
|
||||
|
||||
### Web UI is one embedded file
|
||||
|
||||
`web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked into
|
||||
the binary at compile time — editing it requires a rebuild to take effect. It's
|
||||
served with `Cache-Control: no-store` so binary upgrades don't strand stale tabs.
|
||||
Progress streams over `/ws/progress` (WebSocket) with HTTP-poll fallback. The JS
|
||||
isn't unit-testable, so syntax-check it after edits by extracting the inline
|
||||
`<script>` and running `node --check` (a syntax error there silently breaks the
|
||||
whole SPA — a `cargo build` won't catch it since the HTML is just a string).
|
||||
|
||||
### Bundled toolchain & anti-bot
|
||||
|
||||
`ytdlp_bin.rs` manages an optional self-contained venv at
|
||||
`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi`
|
||||
for TLS impersonation + bundled `deno`). `pot_provider.rs` runs `bgutil-pot` (a
|
||||
loopback HTTP server) for YouTube Proof-of-Origin tokens; **its yt-dlp plugin
|
||||
must come from the same release as the server binary, not PyPI** (version skew
|
||||
silently produces no tokens — see the module doc). `error_class.rs` pattern-
|
||||
matches yt-dlp stderr into actionable classes (the captcha "Video unavailable"
|
||||
wall is classified RateLimited, not NotFound — order matters in `classify()`).
|
||||
|
||||
### Search & perceptual dedup
|
||||
|
||||
`Database::sync_search_index` feeds the `video_search` FTS5 index from
|
||||
`library::build_search_entries`; it's refreshed after every scan (construction,
|
||||
rescan, maintenance-remove) in both UIs and reads each video's description +
|
||||
first subtitle (via `vtt::parse`) only when the mtime changed. `search_videos`
|
||||
returns ranked hits with a `snippet(…, -1, …)` excerpt (column -1 = best-matching
|
||||
column).
|
||||
|
||||
`fingerprint.rs` is the content-aware dedup engine: ffmpeg keyframe-seek samples
|
||||
6 frames/video → 9×8 grayscale → 64-bit dHash; `group_similar` clusters by
|
||||
Hamming distance but only within a duration-tolerance window (sorted sliding
|
||||
window + union-find) to stay near-linear. `rebuild_and_group` is the shared
|
||||
pipeline (mtime-gate → parallel `compute_batch` → upsert → prune → group) both
|
||||
UIs call. Because that pass is slow, it runs as a **background job**: web keeps a
|
||||
`DedupState` (atomics for progress + `Mutex` for the result) polled via
|
||||
`/api/maintenance/dedup/status`; desktop spawns a thread and delivers the result
|
||||
over an `mpsc` channel. `vtt.rs` is a small shared WebVTT/SRT cue parser used by
|
||||
both the transcript indexer and the transcript viewers.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Never commit** `cookies.txt` (live session creds), `config.toml` (user-
|
||||
specific), or `yt-offline.db` (contains the Argon2 password hash). All
|
||||
gitignored.
|
||||
- Redact the absolute cookies path out of any log line surfaced to the UI/API
|
||||
(`redact_sensitive` in `downloader.rs`) — it leaks `$HOME`.
|
||||
- `app.rs` and `web.rs` are large (~3–4k lines) because each owns a full UI; new
|
||||
desktop code goes in `app.rs`, web handlers in `web.rs`, shared logic in the
|
||||
focused modules (`downloader`, `database`, `library`, `platform`, `fingerprint`,
|
||||
`vtt`, …).
|
||||
- Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only/no-GTK by
|
||||
design; keep that posture (it's why packaging avoids a GTK dep). Windows/macOS
|
||||
are not yet first-class — the tray would need a per-OS backend.
|
||||
17
ROADMAP.md
17
ROADMAP.md
|
|
@ -140,11 +140,20 @@ Native client over the existing web API. Background download via
|
|||
WorkManager + JobScheduler. Push notifications via Tailscale-routed
|
||||
HTTPS or a userland push channel.
|
||||
|
||||
### 3.4 Smart auto-tagging
|
||||
### 3.4 Smart auto-tagging — DONE
|
||||
|
||||
Cluster channels by uploader frequency, content type, and metadata.
|
||||
Suggest groups ("looks like a music channel — move to Music?"). Builds
|
||||
on Phase 1.2's group system.
|
||||
`autotag.rs` classifies each *unfiled* channel from already-scanned
|
||||
metadata — source platform plus the median video duration and upload
|
||||
cadence — into a suggested folder group: "Music" (music platforms or
|
||||
audio-only channels), "Shorts" (median < 90 s / TikTok), "Long-form &
|
||||
Podcasts" (median > 25 min), or "Streams & VODs" (Twitch). Mid-length
|
||||
YouTube is deliberately left unsuggested rather than guessed at. Each
|
||||
suggestion carries a confidence and a human reason ("median length 76 s;
|
||||
~1/mo"). Surfaced in both Maintenance views: the web shows per-channel
|
||||
checkboxes + "Apply → group" (POST /api/autotag/apply creates/reuses the
|
||||
folder and assigns), desktop shows "Move all → group". Pure arithmetic
|
||||
over the in-memory library, so it's recomputed on demand with no job.
|
||||
Never moves anything until you apply.
|
||||
|
||||
### 3.5 Federation / multi-host
|
||||
|
||||
|
|
|
|||
75
src/app.rs
75
src/app.rs
|
|
@ -229,6 +229,9 @@ pub struct App {
|
|||
// Maintenance (library health) screen state. The flag is gone now —
|
||||
// Screen::Maintenance + this report's presence drive the render.
|
||||
health_report: Option<crate::maintenance::HealthReport>,
|
||||
/// Auto-tag grouping suggestions, recomputed when the Maintenance screen
|
||||
/// opens and after a group is applied. Empty when there's nothing to suggest.
|
||||
autotag_suggestions: Vec<crate::autotag::GroupSuggestion>,
|
||||
stats_report: Option<crate::stats::StatsReport>,
|
||||
// Per-channel download-options dialog state
|
||||
show_channel_options: bool,
|
||||
|
|
@ -573,6 +576,7 @@ impl App {
|
|||
backup_open_tx,
|
||||
backup_open_rx,
|
||||
health_report: None,
|
||||
autotag_suggestions: Vec::new(),
|
||||
stats_report: None,
|
||||
show_channel_options: false,
|
||||
channel_options_target: None,
|
||||
|
|
@ -1446,6 +1450,7 @@ impl App {
|
|||
} else {
|
||||
self.health_report =
|
||||
Some(crate::maintenance::scan(&self.library_root, &self.library));
|
||||
self.autotag_suggestions = crate::autotag::suggest(&self.library);
|
||||
self.current_screen = Screen::Maintenance;
|
||||
}
|
||||
}
|
||||
|
|
@ -2098,6 +2103,8 @@ impl App {
|
|||
let mut rescan_health = false;
|
||||
let mut start_dedup = false;
|
||||
let mut dedup_remove: Vec<PathBuf> = Vec::new();
|
||||
let mut apply_autotag_group: Option<usize> = None;
|
||||
let autotag_suggestions = self.autotag_suggestions.clone();
|
||||
let dedup_groups = self.dedup_groups.clone();
|
||||
let dedup_running = self.dedup_running;
|
||||
let dedup_started = self.dedup_started;
|
||||
|
|
@ -2234,9 +2241,39 @@ impl App {
|
|||
});
|
||||
}
|
||||
}
|
||||
|
||||
ui.add_space(12.0);
|
||||
ui.heading("🏷 Auto-tag suggestions");
|
||||
ui.label(egui::RichText::new(
|
||||
"Unfiled channels grouped by platform + typical video length. \
|
||||
Apply a group to move those channels into a folder.").weak().small());
|
||||
if autotag_suggestions.is_empty() {
|
||||
ui.label(egui::RichText::new(
|
||||
"No suggestions — every channel is already filed or too ambiguous to call.")
|
||||
.weak());
|
||||
}
|
||||
for (gi, g) in autotag_suggestions.iter().enumerate() {
|
||||
ui.group(|ui| {
|
||||
ui.horizontal(|ui| {
|
||||
ui.label(egui::RichText::new(
|
||||
format!("📁 {} ({})", g.group, g.channels.len())).strong());
|
||||
if ui.button(format!("Move all → {}", g.group)).clicked() {
|
||||
apply_autotag_group = Some(gi);
|
||||
}
|
||||
});
|
||||
for c in &g.channels {
|
||||
ui.label(format!(" {} · {} · {}",
|
||||
c.display_name, c.platform_label, c.reason));
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
if let Some(gi) = apply_autotag_group {
|
||||
self.apply_autotag_group(gi);
|
||||
}
|
||||
|
||||
// ── Apply collected actions ────────────────────────────────────────
|
||||
let mut changed = false;
|
||||
if !to_remove.is_empty() {
|
||||
|
|
@ -2279,6 +2316,44 @@ impl App {
|
|||
}
|
||||
}
|
||||
|
||||
/// Apply one auto-tag suggestion: create (or reuse) a folder with the
|
||||
/// suggested name and move all of that group's channels into it, mirroring
|
||||
/// the DB write onto the in-memory library. Recomputes suggestions so the
|
||||
/// now-filed channels drop out.
|
||||
fn apply_autotag_group(&mut self, gi: usize) {
|
||||
let Some(group) = self.autotag_suggestions.get(gi).cloned() else { return };
|
||||
let existing = self.folders.iter()
|
||||
.find(|f| f.name.eq_ignore_ascii_case(&group.group))
|
||||
.map(|f| f.id);
|
||||
let folder_id = match existing {
|
||||
Some(id) => id,
|
||||
None => match self.db.create_folder(&group.group) {
|
||||
Ok(id) => {
|
||||
self.folders = self.db.list_folders().unwrap_or_default();
|
||||
id
|
||||
}
|
||||
Err(e) => {
|
||||
self.status = format!("Create folder failed: {e}");
|
||||
return;
|
||||
}
|
||||
},
|
||||
};
|
||||
let mut moved = 0;
|
||||
for c in &group.channels {
|
||||
if self.db.set_channel_folder(&c.platform, &c.handle, Some(folder_id)).is_ok() {
|
||||
for ch in self.library.iter_mut() {
|
||||
if ch.platform.dir_name() == c.platform && ch.name == c.handle {
|
||||
ch.folder_id = Some(folder_id);
|
||||
break;
|
||||
}
|
||||
}
|
||||
moved += 1;
|
||||
}
|
||||
}
|
||||
self.status = format!("Moved {moved} channel(s) → {}", group.group);
|
||||
self.autotag_suggestions = crate::autotag::suggest(&self.library);
|
||||
}
|
||||
|
||||
fn stats_screen(&mut self, ctx: &egui::Context) {
|
||||
let report = match &self.stats_report {
|
||||
Some(r) => r.clone(),
|
||||
|
|
|
|||
266
src/autotag.rs
Normal file
266
src/autotag.rs
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
//! Smart auto-tagging — heuristic grouping suggestions for the library.
|
||||
//!
|
||||
//! Looks at each *unfiled* channel's source platform and the duration
|
||||
//! distribution of its videos and proposes a folder group: "Music",
|
||||
//! "Shorts", "Long-form & Podcasts", or "Streams & VODs". Suggestions are
|
||||
//! advisory only — the user applies them from the Maintenance view, which
|
||||
//! creates the folder (if it doesn't exist yet) and assigns the channels via
|
||||
//! the existing folder machinery. Channels already in a folder are left
|
||||
//! untouched. Roadmap 3.4.
|
||||
//!
|
||||
//! Everything here is pure arithmetic over already-scanned metadata
|
||||
//! (`Channel`/`Video`), so it's cheap enough to recompute on demand without a
|
||||
//! background job.
|
||||
|
||||
use serde::Serialize;
|
||||
|
||||
use crate::library::Channel;
|
||||
use crate::platform::Platform;
|
||||
|
||||
/// A single channel's suggested placement.
|
||||
#[derive(Serialize, Clone, Debug, PartialEq)]
|
||||
pub struct ChannelSuggestion {
|
||||
/// Platform dir-name (`channels`/`tiktok`/…) — half of the assignment key.
|
||||
pub platform: String,
|
||||
pub platform_label: String,
|
||||
/// Channel handle (its folder name) — the other half of the key.
|
||||
pub handle: String,
|
||||
/// Friendly name for display (uploader if known, else the handle).
|
||||
pub display_name: String,
|
||||
/// Human-readable justification, e.g. "median length 42 s; ~15/mo".
|
||||
pub reason: String,
|
||||
/// 0.0–1.0 signal strength; drives UI emphasis (strong vs tentative).
|
||||
pub confidence: f32,
|
||||
}
|
||||
|
||||
/// A proposed folder and the channels that look like they belong in it.
|
||||
#[derive(Serialize, Clone, Debug, PartialEq)]
|
||||
pub struct GroupSuggestion {
|
||||
/// Suggested folder name (created on apply if absent).
|
||||
pub group: String,
|
||||
pub channels: Vec<ChannelSuggestion>,
|
||||
}
|
||||
|
||||
// Duration thresholds (seconds).
|
||||
const SHORTS_MAX: f64 = 90.0;
|
||||
const LONGFORM_MIN: f64 = 25.0 * 60.0;
|
||||
// A channel needs at least this many videos before we trust the signal.
|
||||
const MIN_VIDEOS: usize = 3;
|
||||
|
||||
/// Compute grouping suggestions for every unfiled channel that has enough
|
||||
/// videos to form a signal. Channels already assigned to a folder are skipped.
|
||||
pub fn suggest(channels: &[Channel]) -> Vec<GroupSuggestion> {
|
||||
use std::collections::BTreeMap;
|
||||
let mut groups: BTreeMap<&'static str, Vec<ChannelSuggestion>> = BTreeMap::new();
|
||||
|
||||
for ch in channels {
|
||||
if ch.folder_id.is_some() || ch.total_videos() < MIN_VIDEOS {
|
||||
continue;
|
||||
}
|
||||
let durations: Vec<f64> = ch
|
||||
.all_videos()
|
||||
.filter_map(|v| v.duration_secs)
|
||||
.filter(|d| *d > 0.0)
|
||||
.collect();
|
||||
|
||||
let (Some(group), confidence) = classify(ch, &durations) else {
|
||||
continue;
|
||||
};
|
||||
|
||||
let mut reason = group_reason(ch, &durations);
|
||||
if let Some(per_month) = cadence_per_month(ch) {
|
||||
reason.push_str(&format!("; ~{per_month}/mo"));
|
||||
}
|
||||
|
||||
groups.entry(group).or_default().push(ChannelSuggestion {
|
||||
platform: ch.platform.dir_name().to_string(),
|
||||
platform_label: ch.platform.display_name().to_string(),
|
||||
handle: ch.name.clone(),
|
||||
display_name: ch
|
||||
.meta
|
||||
.as_ref()
|
||||
.and_then(|m| m.uploader.clone())
|
||||
.filter(|u| !u.is_empty())
|
||||
.unwrap_or_else(|| ch.name.clone()),
|
||||
reason,
|
||||
confidence,
|
||||
});
|
||||
}
|
||||
|
||||
groups
|
||||
.into_iter()
|
||||
.map(|(group, mut channels)| {
|
||||
// Strongest signal first within each group.
|
||||
channels.sort_by(|a, b| {
|
||||
b.confidence
|
||||
.partial_cmp(&a.confidence)
|
||||
.unwrap_or(std::cmp::Ordering::Equal)
|
||||
});
|
||||
GroupSuggestion {
|
||||
group: group.to_string(),
|
||||
channels,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Choose a group + confidence for a channel, or `(None, _)` when no signal is
|
||||
/// strong enough to suggest anything (the common mid-length YouTube case).
|
||||
fn classify(ch: &Channel, durations: &[f64]) -> (Option<&'static str>, f32) {
|
||||
match ch.platform {
|
||||
Platform::Bandcamp | Platform::SoundCloud => return (Some("Music"), 0.95),
|
||||
Platform::Twitch => return (Some("Streams & VODs"), 0.9),
|
||||
Platform::TikTok => return (Some("Shorts"), 0.9),
|
||||
_ => {}
|
||||
}
|
||||
// The user already downloads this channel as audio → treat as music.
|
||||
if ch.download_options.audio_only {
|
||||
return (Some("Music"), 0.85);
|
||||
}
|
||||
match median(durations) {
|
||||
Some(m) if m < SHORTS_MAX => (Some("Shorts"), 0.7),
|
||||
Some(m) if m >= LONGFORM_MIN => (Some("Long-form & Podcasts"), 0.7),
|
||||
_ => (None, 0.0),
|
||||
}
|
||||
}
|
||||
|
||||
fn group_reason(ch: &Channel, durations: &[f64]) -> String {
|
||||
match ch.platform {
|
||||
Platform::Bandcamp | Platform::SoundCloud => return "music platform".to_string(),
|
||||
Platform::Twitch => return "Twitch channel".to_string(),
|
||||
Platform::TikTok => return "TikTok channel".to_string(),
|
||||
_ => {}
|
||||
}
|
||||
if ch.download_options.audio_only {
|
||||
return "downloaded as audio-only".to_string();
|
||||
}
|
||||
match median(durations) {
|
||||
Some(m) => format!("median length {}", fmt_dur(m)),
|
||||
None => "no duration data".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn median(durations: &[f64]) -> Option<f64> {
|
||||
if durations.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut v = durations.to_vec();
|
||||
v.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||
let mid = v.len() / 2;
|
||||
Some(if v.len() % 2 == 0 {
|
||||
(v[mid - 1] + v[mid]) / 2.0
|
||||
} else {
|
||||
v[mid]
|
||||
})
|
||||
}
|
||||
|
||||
fn fmt_dur(secs: f64) -> String {
|
||||
let s = secs.round() as u64;
|
||||
if s < 90 {
|
||||
format!("{s} s")
|
||||
} else {
|
||||
format!("{} min", (s + 30) / 60)
|
||||
}
|
||||
}
|
||||
|
||||
/// Rough videos-per-month from the spread of upload dates. `None` if fewer than
|
||||
/// two videos carry a parseable `YYYYMMDD`.
|
||||
fn cadence_per_month(ch: &Channel) -> Option<u64> {
|
||||
let mut dates: Vec<u32> = ch
|
||||
.all_videos()
|
||||
.filter_map(|v| v.upload_date.as_deref())
|
||||
.filter_map(|d| d.parse::<u32>().ok())
|
||||
.filter(|d| *d > 0)
|
||||
.collect();
|
||||
if dates.len() < 2 {
|
||||
return None;
|
||||
}
|
||||
dates.sort_unstable();
|
||||
let months = month_span(*dates.first()?, *dates.last()?).max(1);
|
||||
Some((dates.len() as u64 / months).max(1))
|
||||
}
|
||||
|
||||
/// Whole-month span between two `YYYYMMDD` integers (clamped at 0).
|
||||
fn month_span(a: u32, b: u32) -> u64 {
|
||||
let (ay, am) = ((a / 10000) as i64, ((a / 100) % 100) as i64);
|
||||
let (by, bm) = ((b / 10000) as i64, ((b / 100) % 100) as i64);
|
||||
((by - ay) * 12 + (bm - am)).max(0) as u64
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::library::{Channel, Video};
|
||||
|
||||
fn vid(dur: f64, date: &str) -> Video {
|
||||
Video {
|
||||
id: "x".into(),
|
||||
title: "t".into(),
|
||||
stem: "t".into(),
|
||||
video_path: None,
|
||||
thumb_path: None,
|
||||
description_path: None,
|
||||
info_path: None,
|
||||
subtitles: Vec::new(),
|
||||
has_live_chat: false,
|
||||
duration_secs: Some(dur),
|
||||
has_chapters: false,
|
||||
file_size: None,
|
||||
mtime_unix: None,
|
||||
upload_date: Some(date.into()),
|
||||
}
|
||||
}
|
||||
|
||||
fn channel(platform: Platform, durations: &[f64]) -> Channel {
|
||||
let videos: Vec<Video> = durations.iter().map(|d| vid(*d, "20240101")).collect();
|
||||
let total = videos.len();
|
||||
Channel {
|
||||
name: "chan".into(),
|
||||
path: std::path::PathBuf::from("/tmp/chan"),
|
||||
platform,
|
||||
source_url: None,
|
||||
videos,
|
||||
playlists: Vec::new(),
|
||||
meta: None,
|
||||
total_videos_cached: total,
|
||||
total_size_cached: 0,
|
||||
download_options: Default::default(),
|
||||
folder_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shorts_detected_by_median_duration() {
|
||||
let groups = suggest(&[channel(Platform::YouTube, &[30.0, 45.0, 20.0])]);
|
||||
assert_eq!(groups.len(), 1);
|
||||
assert_eq!(groups[0].group, "Shorts");
|
||||
assert_eq!(groups[0].channels.len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn longform_detected_by_median_duration() {
|
||||
let groups = suggest(&[channel(Platform::YouTube, &[3600.0, 2700.0, 4000.0])]);
|
||||
assert_eq!(groups[0].group, "Long-form & Podcasts");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn music_platform_grouped_regardless_of_duration() {
|
||||
let groups = suggest(&[channel(Platform::Bandcamp, &[200.0, 240.0, 180.0])]);
|
||||
assert_eq!(groups[0].group, "Music");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mid_length_youtube_yields_no_suggestion() {
|
||||
// ~8 min median: ambiguous, deliberately not suggested.
|
||||
let groups = suggest(&[channel(Platform::YouTube, &[480.0, 500.0, 460.0])]);
|
||||
assert!(groups.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn filed_or_tiny_channels_are_skipped() {
|
||||
let mut filed = channel(Platform::YouTube, &[30.0, 30.0, 30.0]);
|
||||
filed.folder_id = Some(7);
|
||||
let tiny = channel(Platform::YouTube, &[30.0]); // < MIN_VIDEOS
|
||||
assert!(suggest(&[filed, tiny]).is_empty());
|
||||
}
|
||||
}
|
||||
|
|
@ -14,6 +14,7 @@
|
|||
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
|
||||
|
||||
mod app;
|
||||
mod autotag;
|
||||
mod config;
|
||||
mod crash;
|
||||
mod database;
|
||||
|
|
|
|||
81
src/web.rs
81
src/web.rs
|
|
@ -2112,6 +2112,85 @@ async fn get_maintenance_scan(State(state): State<Arc<WebState>>) -> impl IntoRe
|
|||
Json(report)
|
||||
}
|
||||
|
||||
/// `GET /api/autotag/suggest` — heuristic folder-grouping suggestions for
|
||||
/// unfiled channels (see [`crate::autotag`]). Pure arithmetic over the
|
||||
/// in-memory library, so it's computed on demand rather than as a job.
|
||||
async fn get_autotag_suggest(State(state): State<Arc<WebState>>) -> impl IntoResponse {
|
||||
let lib = state.library.lock_recover();
|
||||
Json(crate::autotag::suggest(&lib))
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct AutotagChannel {
|
||||
platform: String,
|
||||
handle: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct AutotagApplyBody {
|
||||
/// Target folder name; created if it doesn't already exist.
|
||||
group: String,
|
||||
channels: Vec<AutotagChannel>,
|
||||
}
|
||||
|
||||
/// `POST /api/autotag/apply` — accept a suggested group: create the folder
|
||||
/// (or reuse an existing one with the same name) and move the given channels
|
||||
/// into it. Mirrors the move onto the in-memory library + bumps the ETag, the
|
||||
/// same way `post_assign_folder` does for a single channel.
|
||||
async fn post_autotag_apply(
|
||||
State(state): State<Arc<WebState>>,
|
||||
Json(body): Json<AutotagApplyBody>,
|
||||
) -> impl IntoResponse {
|
||||
let name = body.group.trim();
|
||||
if name.is_empty() {
|
||||
return (StatusCode::BAD_REQUEST, "empty group name").into_response();
|
||||
}
|
||||
// Reuse a folder of the same name (case-insensitive) if one exists,
|
||||
// otherwise create it.
|
||||
let existing = match state.db.list_folders() {
|
||||
Ok(folders) => folders
|
||||
.iter()
|
||||
.find(|f| f.name.eq_ignore_ascii_case(name))
|
||||
.map(|f| f.id),
|
||||
Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, format!("db: {e}")).into_response(),
|
||||
};
|
||||
let folder_id = match existing {
|
||||
Some(id) => id,
|
||||
None => match state.db.create_folder(name) {
|
||||
Ok(id) => id,
|
||||
Err(e) => {
|
||||
return (StatusCode::INTERNAL_SERVER_ERROR, format!("create folder: {e}"))
|
||||
.into_response()
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
let mut applied = 0usize;
|
||||
for c in &body.channels {
|
||||
if state
|
||||
.db
|
||||
.set_channel_folder(&c.platform, &c.handle, Some(folder_id))
|
||||
.is_ok()
|
||||
{
|
||||
applied += 1;
|
||||
}
|
||||
}
|
||||
{
|
||||
let mut lib = state.library.lock_recover();
|
||||
for ch in lib.iter_mut() {
|
||||
if body
|
||||
.channels
|
||||
.iter()
|
||||
.any(|c| ch.platform.dir_name() == c.platform && ch.name == c.handle)
|
||||
{
|
||||
ch.folder_id = Some(folder_id);
|
||||
}
|
||||
}
|
||||
}
|
||||
bump_library_version(&state);
|
||||
Json(serde_json::json!({ "folder_id": folder_id, "applied": applied })).into_response()
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RemoveRequest {
|
||||
paths: Vec<PathBuf>,
|
||||
|
|
@ -3088,6 +3167,8 @@ async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
|
|||
.route("/api/plex/generate", post(post_plex_generate))
|
||||
.route("/api/maintenance/scan", get(get_maintenance_scan))
|
||||
.route("/api/maintenance/remove", post(post_maintenance_remove))
|
||||
.route("/api/autotag/suggest", get(get_autotag_suggest))
|
||||
.route("/api/autotag/apply", post(post_autotag_apply))
|
||||
.route("/api/maintenance/dedup/scan", post(post_dedup_scan))
|
||||
.route("/api/maintenance/dedup/status", get(get_dedup_status))
|
||||
.route("/api/maintenance/repair/:id", post(post_maintenance_repair))
|
||||
|
|
|
|||
|
|
@ -2200,8 +2200,63 @@ function renderMaintenance(r){
|
|||
h+=`<h3 style="font-size:13px;margin:16px 0 8px">Similar content (perceptual)</h3>
|
||||
<div class="settings-hint" style="margin-bottom:8px">Finds the same video re-uploaded under a different ID — mirrors, re-encodes, resolution changes — by comparing sampled frames. The first scan fingerprints your library (a few minutes); after that it's cached, so re-scans are quick.</div>
|
||||
<div id="dedup-area"></div>`;
|
||||
h+=`<h3 style="font-size:13px;margin:16px 0 8px">🏷 Auto-tag suggestions</h3>
|
||||
<div class="settings-hint" style="margin-bottom:8px">Groups your <b>unfiled</b> channels by source platform and typical video length. Review a group and apply it to drop those channels into a folder — nothing moves until you click Apply.</div>
|
||||
<div id="autotag-area"><em style="color:var(--muted);font-size:12px">Analyzing…</em></div>`;
|
||||
body.innerHTML=h;
|
||||
initDedupArea();
|
||||
initAutotagArea();
|
||||
}
|
||||
|
||||
/* ── Smart auto-tagging (suggested folder groups) ───────────────── */
|
||||
async function initAutotagArea(){
|
||||
const area=document.getElementById('autotag-area');if(!area)return;
|
||||
let groups;
|
||||
try{groups=await(await api('/api/autotag/suggest')).json();}
|
||||
catch(e){area.innerHTML=`<div style="color:#f87171;font-size:12px">Could not analyze: ${esc(e.message)}</div>`;return}
|
||||
renderAutotag(groups);
|
||||
}
|
||||
function renderAutotag(groups){
|
||||
const area=document.getElementById('autotag-area');if(!area)return;
|
||||
if(!groups||!groups.length){area.innerHTML='<div style="color:var(--muted);font-size:12px">No grouping suggestions — every channel is already filed or too ambiguous to call.</div>';return}
|
||||
// Confidence → dot colour: strong (green) ≥ .85, medium (amber) ≥ .7, else grey.
|
||||
const dot=c=>{const col=c>=0.85?'#4ade80':c>=0.7?'#facc15':'#9ca3af';return `<span title="confidence ${(c*100).toFixed(0)}%" style="flex-shrink:0;width:8px;height:8px;border-radius:50%;background:${col};display:inline-block"></span>`};
|
||||
let h='';
|
||||
groups.forEach((g,gi)=>{
|
||||
h+=`<div style="border:1px solid var(--border);border-radius:6px;padding:8px;margin-bottom:8px" data-grp="${gi}">
|
||||
<div style="display:flex;align-items:center;gap:8px;margin-bottom:6px">
|
||||
<span style="font-weight:600;font-size:12px;flex:1">📁 ${esc(g.group)} <span style="color:var(--muted)">(${g.channels.length})</span></span>
|
||||
<button class="primary" onclick="applyAutotag(${gi},this)">Apply → ${esc(g.group)}</button>
|
||||
</div>`;
|
||||
g.channels.forEach((c,ci)=>{
|
||||
h+=`<label style="display:flex;align-items:center;gap:8px;font-size:12px;padding:3px 0">
|
||||
<input type="checkbox" class="at-chk" data-grp="${gi}" data-ci="${ci}" checked>
|
||||
${dot(c.confidence)}
|
||||
<span style="flex:1;overflow:hidden;text-overflow:ellipsis;white-space:nowrap">${esc(c.display_name)} <span style="color:var(--muted)">· ${esc(c.platform_label)} · ${esc(c.reason)}</span></span>
|
||||
</label>`;
|
||||
});
|
||||
h+=`</div>`;
|
||||
});
|
||||
area.innerHTML=h;
|
||||
// Stash the data so applyAutotag can read the channel keys back.
|
||||
area._groups=groups;
|
||||
}
|
||||
async function applyAutotag(gi,btn){
|
||||
const area=document.getElementById('autotag-area');
|
||||
const groups=area&&area._groups;if(!groups||!groups[gi])return;
|
||||
const g=groups[gi];
|
||||
const picked=[...area.querySelectorAll(`.at-chk[data-grp="${gi}"]`)]
|
||||
.filter(cb=>cb.checked)
|
||||
.map(cb=>g.channels[+cb.dataset.ci])
|
||||
.map(c=>({platform:c.platform,handle:c.handle}));
|
||||
if(!picked.length){setStatus('No channels selected');return}
|
||||
btn.disabled=true;btn.textContent='Applying…';
|
||||
try{
|
||||
const r=await(await api('/api/autotag/apply',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({group:g.group,channels:picked})})).json();
|
||||
setStatus(`Moved ${r.applied} channel${r.applied===1?'':'s'} → ${g.group}`);
|
||||
await loadLibrary();renderSidebar();
|
||||
initAutotagArea(); // re-analyze: applied channels drop out (now filed)
|
||||
}catch(e){setStatus('Apply failed: '+e.message);btn.disabled=false;btn.textContent='Apply → '+g.group;}
|
||||
}
|
||||
|
||||
/* ── Perceptual dedup (similar content) ─────────────────────────── */
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue