Smart auto-tagging: suggest folder groups for unfiled channels (3.4)
New autotag.rs classifies each unfiled channel from already-scanned metadata — source platform + median video duration + upload cadence — into a suggested folder group (Music / Shorts / Long-form & Podcasts / Streams & VODs), with a confidence and a human-readable reason. Mid-length YouTube is left unsuggested rather than guessed at; channels already in a folder are skipped. Pure arithmetic over the in-memory library, computed on demand (no background job). Surfaced in both Maintenance views: - web: GET /api/autotag/suggest + POST /api/autotag/apply (create/reuse the named folder and assign), rendered with per-channel checkboxes and an "Apply -> group" button that re-analyzes after applying. - desktop: a section listing each group with a "Move all -> group" button. Apply mirrors post_assign_folder: DB write + in-memory library update + ETag bump. Reuses an existing folder of the same name (case-insensitive). Unit tests cover the classifier; verified the apply/revert round-trip live. ROADMAP 3.4 marked DONE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
aed577ea2f
commit
ec8cf6f934
7 changed files with 658 additions and 4 deletions
167
AGENTS.md
Normal file
167
AGENTS.md
Normal file
|
|
@ -0,0 +1,167 @@
|
||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
`yt-offline` — a single Rust binary that is **both** a desktop GUI (eframe/egui)
|
||||||
|
and a headless web server (axum), wrapping `yt-dlp` to archive YouTube/TikTok/
|
||||||
|
Twitch/etc. AGPL-3.0. North-star goal and feature-parity tracking live in
|
||||||
|
[ROADMAP.md](ROADMAP.md); a structured analysis of the Tartube benchmark is in
|
||||||
|
[docs/tartube-spec.md](docs/tartube-spec.md).
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo build --release # the binary (release profile is opt-level=3 + thin LTO, ~1-2 min)
|
||||||
|
cargo test --release # unit tests + tests/api.rs integration tests (no network)
|
||||||
|
cargo test --release <name> # single test by substring, e.g. `cargo test --release subs_disabled`
|
||||||
|
cargo test --release real_ffmpeg -- --ignored # opt-in: real-ffmpeg fingerprint accuracy/speed check
|
||||||
|
./target/release/yt-offline # desktop GUI mode (default)
|
||||||
|
./target/release/yt-offline --web 8080 # headless web server on a port
|
||||||
|
|
||||||
|
scripts/package.sh [deb|rpm|appimage|all] # build distro packages → dist/ (see docs/PACKAGING.md)
|
||||||
|
```
|
||||||
|
|
||||||
|
There is no separate lint step; `cargo build` warnings are the lint. The egui
|
||||||
|
dependency emits ~39 `f32: From<f64>` fallback warnings on a clean build — those
|
||||||
|
are upstream, not from this code.
|
||||||
|
|
||||||
|
### Running / verifying against a real library
|
||||||
|
|
||||||
|
The app reads `config.toml` and `cookies.txt` **from the process working
|
||||||
|
directory**, not a fixed path. To smoke-test the web server in isolation, make a
|
||||||
|
scratch dir with a minimal `config.toml` (`[backup]\ndirectory = "..."`) and run
|
||||||
|
`--web <port>` from inside it. `/api/*` endpoints require auth when a password is
|
||||||
|
set in the target library's DB.
|
||||||
|
|
||||||
|
`tests/api.rs` automates exactly this: it spawns the real `--web` binary against a
|
||||||
|
throwaway dir and drives the HTTP API with `curl` (skipping if curl is absent; the
|
||||||
|
dedup test also needs ffmpeg and skips without it). That's the place to add
|
||||||
|
end-to-end coverage of a new endpoint.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Two front-ends, one engine
|
||||||
|
|
||||||
|
`main.rs` dispatches: `--web` → `web::run()` (axum, blocks forever); otherwise
|
||||||
|
`app::App` (eframe). **Both share `downloader::Downloader`**, the single source
|
||||||
|
of truth for yt-dlp job lifecycle. `Downloader` is *not* async — it spawns an OS
|
||||||
|
thread per `yt-dlp` process, streams stdout/stderr back over an `mpsc` channel
|
||||||
|
into each `Job`'s log buffer, and the caller must pump `Downloader::poll()`
|
||||||
|
regularly (the egui frame loop and a web background task both do this). When you
|
||||||
|
add download behavior, it goes in `Downloader` and is automatically available to
|
||||||
|
both UIs.
|
||||||
|
|
||||||
|
`poll()` also drives the cross-cutting job machinery: **auto-retry** of transient
|
||||||
|
failures (rate-limit/network) with cooldown + adaptive throttle, the
|
||||||
|
**post-download ffmpeg transcode** pass, and the **hang watchdog** (a job whose
|
||||||
|
`last_activity` is silent past `HANG_TIMEOUT` is SIGKILLed and re-queued —
|
||||||
|
classified `NetworkError` so the existing retry path re-issues it). The first two
|
||||||
|
work by capturing specs onto the `Job` at `start()` time (`RetrySpec`,
|
||||||
|
`ConvertSpec`) and acting on them when the job transitions state — the
|
||||||
|
`pending_*_spec` fields on `Downloader` are stashed by `start()` and consumed by
|
||||||
|
`enqueue()`/`spawn_job()` so the four non-download enqueue paths
|
||||||
|
(repair/music/yt-dlp-update/pot-update) stay untouched.
|
||||||
|
|
||||||
|
### Settings flow (the easy thing to get wrong)
|
||||||
|
|
||||||
|
Almost every feature has the same five-touchpoint shape — miss one and it
|
||||||
|
silently half-works:
|
||||||
|
|
||||||
|
1. `config.rs` — a field/section + its `Default` and the `default_with_dir()` constructor.
|
||||||
|
2. `download_options.rs` — an `Option<…>` per-channel override (None = defer to global).
|
||||||
|
3. `downloader.rs` — a resolver that merges global config + per-channel override into yt-dlp/ffmpeg args, plus a `pub` field on `Downloader` holding the global default.
|
||||||
|
4. **Both** UIs render the global setting *and* the per-channel override: desktop in `app.rs` (egui Settings screen + the channel-options dialog), web in `web_ui/index.html` (Settings modal + channel-options dialog) **and** `web.rs`'s `SettingsPayload` struct (GET reads from config, POST writes config + pushes the value onto the live `Downloader`).
|
||||||
|
5. Seed the `Downloader` field at construction **and** on settings-save, in **both** `app.rs` and `web.rs`.
|
||||||
|
|
||||||
|
Grep an existing setting end-to-end before adding one — `subtitle_defaults`,
|
||||||
|
`youtube_player_clients`, `sponsorblock_mode`, and `convert_defaults` are
|
||||||
|
complete worked examples.
|
||||||
|
|
||||||
|
### Filesystem layout invariant
|
||||||
|
|
||||||
|
`platform::platform_root(channels_root, platform)` = `channels_root.join(dir_name)`.
|
||||||
|
**All** platforms (including YouTube, whose `dir_name` is `channels`) nest under
|
||||||
|
the one configured `backup.directory`. `library_root == channels_root` now (a
|
||||||
|
historical two-level split was removed). `.source-url` sidecars in each creator
|
||||||
|
folder let channel re-checks recover the exact URL. Library scanning
|
||||||
|
(`library.rs`) is parallel and consults a `(path, mtime)` SQLite cache to skip
|
||||||
|
re-parsing unchanged `info.json` sidecars.
|
||||||
|
|
||||||
|
### Persistence
|
||||||
|
|
||||||
|
`database.rs` wraps an r2d2 SQLite pool (file-backed; `Database` is cheaply
|
||||||
|
`Clone` — the pool is an `Arc`, so the parallel scanner takes its own handle).
|
||||||
|
Schema lives in `init_schema()`; new columns are added via idempotent
|
||||||
|
`ALTER TABLE … ADD COLUMN` that swallows the duplicate-column error (no migration
|
||||||
|
framework). **Exception:** FTS5 virtual tables have no `ADD COLUMN`, so the
|
||||||
|
`video_search` index is migrated by dropping + recreating it and clearing
|
||||||
|
`search_meta` to force a one-time reindex (it's a derived cache — safe to rebuild;
|
||||||
|
see the `transcript`-column migration). The web UI holds library/notes snapshots
|
||||||
|
in memory and mutating endpoints mirror DB writes onto those caches +
|
||||||
|
`bump_library_version()` (the ETag) so `/api/library` stays consistent without a
|
||||||
|
rescan.
|
||||||
|
|
||||||
|
Three `(path|video_id, mtime)`-keyed caches let expensive work happen once and be
|
||||||
|
skipped on unchanged files: `info_cache` (parsed `info.json` fields),
|
||||||
|
`search_meta` + the `video_search` FTS5 index (full-text title/channel/
|
||||||
|
description/**transcript**), and `video_fingerprint` (perceptual dHashes). Each
|
||||||
|
has a `sync_*`/rebuild helper that diffs the current library against stored mtimes,
|
||||||
|
processes only the new/changed rows, and prunes vanished ones.
|
||||||
|
|
||||||
|
### Web UI is one embedded file
|
||||||
|
|
||||||
|
`web_ui/index.html` is the entire SPA (HTML+CSS+JS), `include_str!`-baked into
|
||||||
|
the binary at compile time — editing it requires a rebuild to take effect. It's
|
||||||
|
served with `Cache-Control: no-store` so binary upgrades don't strand stale tabs.
|
||||||
|
Progress streams over `/ws/progress` (WebSocket) with HTTP-poll fallback. The JS
|
||||||
|
isn't unit-testable, so syntax-check it after edits by extracting the inline
|
||||||
|
`<script>` and running `node --check` (a syntax error there silently breaks the
|
||||||
|
whole SPA — a `cargo build` won't catch it since the HTML is just a string).
|
||||||
|
|
||||||
|
### Bundled toolchain & anti-bot
|
||||||
|
|
||||||
|
`ytdlp_bin.rs` manages an optional self-contained venv at
|
||||||
|
`~/.local/share/yt-offline/` (nightly `yt-dlp[default]` via `--pre` + `curl_cffi`
|
||||||
|
for TLS impersonation + bundled `deno`). `pot_provider.rs` runs `bgutil-pot` (a
|
||||||
|
loopback HTTP server) for YouTube Proof-of-Origin tokens; **its yt-dlp plugin
|
||||||
|
must come from the same release as the server binary, not PyPI** (version skew
|
||||||
|
silently produces no tokens — see the module doc). `error_class.rs` pattern-
|
||||||
|
matches yt-dlp stderr into actionable classes (the captcha "Video unavailable"
|
||||||
|
wall is classified RateLimited, not NotFound — order matters in `classify()`).
|
||||||
|
|
||||||
|
### Search & perceptual dedup
|
||||||
|
|
||||||
|
`Database::sync_search_index` feeds the `video_search` FTS5 index from
|
||||||
|
`library::build_search_entries`; it's refreshed after every scan (construction,
|
||||||
|
rescan, maintenance-remove) in both UIs and reads each video's description +
|
||||||
|
first subtitle (via `vtt::parse`) only when the mtime changed. `search_videos`
|
||||||
|
returns ranked hits with a `snippet(…, -1, …)` excerpt (column -1 = best-matching
|
||||||
|
column).
|
||||||
|
|
||||||
|
`fingerprint.rs` is the content-aware dedup engine: ffmpeg keyframe-seek samples
|
||||||
|
6 frames/video → 9×8 grayscale → 64-bit dHash; `group_similar` clusters by
|
||||||
|
Hamming distance but only within a duration-tolerance window (sorted sliding
|
||||||
|
window + union-find) to stay near-linear. `rebuild_and_group` is the shared
|
||||||
|
pipeline (mtime-gate → parallel `compute_batch` → upsert → prune → group) both
|
||||||
|
UIs call. Because that pass is slow, it runs as a **background job**: web keeps a
|
||||||
|
`DedupState` (atomics for progress + `Mutex` for the result) polled via
|
||||||
|
`/api/maintenance/dedup/status`; desktop spawns a thread and delivers the result
|
||||||
|
over an `mpsc` channel. `vtt.rs` is a small shared WebVTT/SRT cue parser used by
|
||||||
|
both the transcript indexer and the transcript viewers.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Never commit** `cookies.txt` (live session creds), `config.toml` (user-
|
||||||
|
specific), or `yt-offline.db` (contains the Argon2 password hash). All
|
||||||
|
gitignored.
|
||||||
|
- Redact the absolute cookies path out of any log line surfaced to the UI/API
|
||||||
|
(`redact_sensitive` in `downloader.rs`) — it leaks `$HOME`.
|
||||||
|
- `app.rs` and `web.rs` are large (~3–4k lines) because each owns a full UI; new
|
||||||
|
desktop code goes in `app.rs`, web handlers in `web.rs`, shared logic in the
|
||||||
|
focused modules (`downloader`, `database`, `library`, `platform`, `fingerprint`,
|
||||||
|
`vtt`, …).
|
||||||
|
- Tray (`ksni`) and file dialogs (`rfd` xdg-portal) are Linux-only/no-GTK by
|
||||||
|
design; keep that posture (it's why packaging avoids a GTK dep). Windows/macOS
|
||||||
|
are not yet first-class — the tray would need a per-OS backend.
|
||||||
17
ROADMAP.md
17
ROADMAP.md
|
|
@ -140,11 +140,20 @@ Native client over the existing web API. Background download via
|
||||||
WorkManager + JobScheduler. Push notifications via Tailscale-routed
|
WorkManager + JobScheduler. Push notifications via Tailscale-routed
|
||||||
HTTPS or a userland push channel.
|
HTTPS or a userland push channel.
|
||||||
|
|
||||||
### 3.4 Smart auto-tagging
|
### 3.4 Smart auto-tagging — DONE
|
||||||
|
|
||||||
Cluster channels by uploader frequency, content type, and metadata.
|
`autotag.rs` classifies each *unfiled* channel from already-scanned
|
||||||
Suggest groups ("looks like a music channel — move to Music?"). Builds
|
metadata — source platform plus the median video duration and upload
|
||||||
on Phase 1.2's group system.
|
cadence — into a suggested folder group: "Music" (music platforms or
|
||||||
|
audio-only channels), "Shorts" (median < 90 s / TikTok), "Long-form &
|
||||||
|
Podcasts" (median > 25 min), or "Streams & VODs" (Twitch). Mid-length
|
||||||
|
YouTube is deliberately left unsuggested rather than guessed at. Each
|
||||||
|
suggestion carries a confidence and a human reason ("median length 76 s;
|
||||||
|
~1/mo"). Surfaced in both Maintenance views: the web shows per-channel
|
||||||
|
checkboxes + "Apply → group" (POST /api/autotag/apply creates/reuses the
|
||||||
|
folder and assigns), desktop shows "Move all → group". Pure arithmetic
|
||||||
|
over the in-memory library, so it's recomputed on demand with no job.
|
||||||
|
Never moves anything until you apply.
|
||||||
|
|
||||||
### 3.5 Federation / multi-host
|
### 3.5 Federation / multi-host
|
||||||
|
|
||||||
|
|
|
||||||
75
src/app.rs
75
src/app.rs
|
|
@ -229,6 +229,9 @@ pub struct App {
|
||||||
// Maintenance (library health) screen state. The flag is gone now —
|
// Maintenance (library health) screen state. The flag is gone now —
|
||||||
// Screen::Maintenance + this report's presence drive the render.
|
// Screen::Maintenance + this report's presence drive the render.
|
||||||
health_report: Option<crate::maintenance::HealthReport>,
|
health_report: Option<crate::maintenance::HealthReport>,
|
||||||
|
/// Auto-tag grouping suggestions, recomputed when the Maintenance screen
|
||||||
|
/// opens and after a group is applied. Empty when there's nothing to suggest.
|
||||||
|
autotag_suggestions: Vec<crate::autotag::GroupSuggestion>,
|
||||||
stats_report: Option<crate::stats::StatsReport>,
|
stats_report: Option<crate::stats::StatsReport>,
|
||||||
// Per-channel download-options dialog state
|
// Per-channel download-options dialog state
|
||||||
show_channel_options: bool,
|
show_channel_options: bool,
|
||||||
|
|
@ -573,6 +576,7 @@ impl App {
|
||||||
backup_open_tx,
|
backup_open_tx,
|
||||||
backup_open_rx,
|
backup_open_rx,
|
||||||
health_report: None,
|
health_report: None,
|
||||||
|
autotag_suggestions: Vec::new(),
|
||||||
stats_report: None,
|
stats_report: None,
|
||||||
show_channel_options: false,
|
show_channel_options: false,
|
||||||
channel_options_target: None,
|
channel_options_target: None,
|
||||||
|
|
@ -1446,6 +1450,7 @@ impl App {
|
||||||
} else {
|
} else {
|
||||||
self.health_report =
|
self.health_report =
|
||||||
Some(crate::maintenance::scan(&self.library_root, &self.library));
|
Some(crate::maintenance::scan(&self.library_root, &self.library));
|
||||||
|
self.autotag_suggestions = crate::autotag::suggest(&self.library);
|
||||||
self.current_screen = Screen::Maintenance;
|
self.current_screen = Screen::Maintenance;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -2098,6 +2103,8 @@ impl App {
|
||||||
let mut rescan_health = false;
|
let mut rescan_health = false;
|
||||||
let mut start_dedup = false;
|
let mut start_dedup = false;
|
||||||
let mut dedup_remove: Vec<PathBuf> = Vec::new();
|
let mut dedup_remove: Vec<PathBuf> = Vec::new();
|
||||||
|
let mut apply_autotag_group: Option<usize> = None;
|
||||||
|
let autotag_suggestions = self.autotag_suggestions.clone();
|
||||||
let dedup_groups = self.dedup_groups.clone();
|
let dedup_groups = self.dedup_groups.clone();
|
||||||
let dedup_running = self.dedup_running;
|
let dedup_running = self.dedup_running;
|
||||||
let dedup_started = self.dedup_started;
|
let dedup_started = self.dedup_started;
|
||||||
|
|
@ -2234,9 +2241,39 @@ impl App {
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
ui.add_space(12.0);
|
||||||
|
ui.heading("🏷 Auto-tag suggestions");
|
||||||
|
ui.label(egui::RichText::new(
|
||||||
|
"Unfiled channels grouped by platform + typical video length. \
|
||||||
|
Apply a group to move those channels into a folder.").weak().small());
|
||||||
|
if autotag_suggestions.is_empty() {
|
||||||
|
ui.label(egui::RichText::new(
|
||||||
|
"No suggestions — every channel is already filed or too ambiguous to call.")
|
||||||
|
.weak());
|
||||||
|
}
|
||||||
|
for (gi, g) in autotag_suggestions.iter().enumerate() {
|
||||||
|
ui.group(|ui| {
|
||||||
|
ui.horizontal(|ui| {
|
||||||
|
ui.label(egui::RichText::new(
|
||||||
|
format!("📁 {} ({})", g.group, g.channels.len())).strong());
|
||||||
|
if ui.button(format!("Move all → {}", g.group)).clicked() {
|
||||||
|
apply_autotag_group = Some(gi);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
for c in &g.channels {
|
||||||
|
ui.label(format!(" {} · {} · {}",
|
||||||
|
c.display_name, c.platform_label, c.reason));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
if let Some(gi) = apply_autotag_group {
|
||||||
|
self.apply_autotag_group(gi);
|
||||||
|
}
|
||||||
|
|
||||||
// ── Apply collected actions ────────────────────────────────────────
|
// ── Apply collected actions ────────────────────────────────────────
|
||||||
let mut changed = false;
|
let mut changed = false;
|
||||||
if !to_remove.is_empty() {
|
if !to_remove.is_empty() {
|
||||||
|
|
@ -2279,6 +2316,44 @@ impl App {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Apply one auto-tag suggestion: create (or reuse) a folder with the
|
||||||
|
/// suggested name and move all of that group's channels into it, mirroring
|
||||||
|
/// the DB write onto the in-memory library. Recomputes suggestions so the
|
||||||
|
/// now-filed channels drop out.
|
||||||
|
fn apply_autotag_group(&mut self, gi: usize) {
|
||||||
|
let Some(group) = self.autotag_suggestions.get(gi).cloned() else { return };
|
||||||
|
let existing = self.folders.iter()
|
||||||
|
.find(|f| f.name.eq_ignore_ascii_case(&group.group))
|
||||||
|
.map(|f| f.id);
|
||||||
|
let folder_id = match existing {
|
||||||
|
Some(id) => id,
|
||||||
|
None => match self.db.create_folder(&group.group) {
|
||||||
|
Ok(id) => {
|
||||||
|
self.folders = self.db.list_folders().unwrap_or_default();
|
||||||
|
id
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
self.status = format!("Create folder failed: {e}");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let mut moved = 0;
|
||||||
|
for c in &group.channels {
|
||||||
|
if self.db.set_channel_folder(&c.platform, &c.handle, Some(folder_id)).is_ok() {
|
||||||
|
for ch in self.library.iter_mut() {
|
||||||
|
if ch.platform.dir_name() == c.platform && ch.name == c.handle {
|
||||||
|
ch.folder_id = Some(folder_id);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
moved += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.status = format!("Moved {moved} channel(s) → {}", group.group);
|
||||||
|
self.autotag_suggestions = crate::autotag::suggest(&self.library);
|
||||||
|
}
|
||||||
|
|
||||||
fn stats_screen(&mut self, ctx: &egui::Context) {
|
fn stats_screen(&mut self, ctx: &egui::Context) {
|
||||||
let report = match &self.stats_report {
|
let report = match &self.stats_report {
|
||||||
Some(r) => r.clone(),
|
Some(r) => r.clone(),
|
||||||
|
|
|
||||||
266
src/autotag.rs
Normal file
266
src/autotag.rs
Normal file
|
|
@ -0,0 +1,266 @@
|
||||||
|
//! Smart auto-tagging — heuristic grouping suggestions for the library.
|
||||||
|
//!
|
||||||
|
//! Looks at each *unfiled* channel's source platform and the duration
|
||||||
|
//! distribution of its videos and proposes a folder group: "Music",
|
||||||
|
//! "Shorts", "Long-form & Podcasts", or "Streams & VODs". Suggestions are
|
||||||
|
//! advisory only — the user applies them from the Maintenance view, which
|
||||||
|
//! creates the folder (if it doesn't exist yet) and assigns the channels via
|
||||||
|
//! the existing folder machinery. Channels already in a folder are left
|
||||||
|
//! untouched. Roadmap 3.4.
|
||||||
|
//!
|
||||||
|
//! Everything here is pure arithmetic over already-scanned metadata
|
||||||
|
//! (`Channel`/`Video`), so it's cheap enough to recompute on demand without a
|
||||||
|
//! background job.
|
||||||
|
|
||||||
|
use serde::Serialize;
|
||||||
|
|
||||||
|
use crate::library::Channel;
|
||||||
|
use crate::platform::Platform;
|
||||||
|
|
||||||
|
/// A single channel's suggested placement.
|
||||||
|
#[derive(Serialize, Clone, Debug, PartialEq)]
|
||||||
|
pub struct ChannelSuggestion {
|
||||||
|
/// Platform dir-name (`channels`/`tiktok`/…) — half of the assignment key.
|
||||||
|
pub platform: String,
|
||||||
|
pub platform_label: String,
|
||||||
|
/// Channel handle (its folder name) — the other half of the key.
|
||||||
|
pub handle: String,
|
||||||
|
/// Friendly name for display (uploader if known, else the handle).
|
||||||
|
pub display_name: String,
|
||||||
|
/// Human-readable justification, e.g. "median length 42 s; ~15/mo".
|
||||||
|
pub reason: String,
|
||||||
|
/// 0.0–1.0 signal strength; drives UI emphasis (strong vs tentative).
|
||||||
|
pub confidence: f32,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A proposed folder and the channels that look like they belong in it.
|
||||||
|
#[derive(Serialize, Clone, Debug, PartialEq)]
|
||||||
|
pub struct GroupSuggestion {
|
||||||
|
/// Suggested folder name (created on apply if absent).
|
||||||
|
pub group: String,
|
||||||
|
pub channels: Vec<ChannelSuggestion>,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Duration thresholds (seconds).
|
||||||
|
const SHORTS_MAX: f64 = 90.0;
|
||||||
|
const LONGFORM_MIN: f64 = 25.0 * 60.0;
|
||||||
|
// A channel needs at least this many videos before we trust the signal.
|
||||||
|
const MIN_VIDEOS: usize = 3;
|
||||||
|
|
||||||
|
/// Compute grouping suggestions for every unfiled channel that has enough
|
||||||
|
/// videos to form a signal. Channels already assigned to a folder are skipped.
|
||||||
|
pub fn suggest(channels: &[Channel]) -> Vec<GroupSuggestion> {
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
let mut groups: BTreeMap<&'static str, Vec<ChannelSuggestion>> = BTreeMap::new();
|
||||||
|
|
||||||
|
for ch in channels {
|
||||||
|
if ch.folder_id.is_some() || ch.total_videos() < MIN_VIDEOS {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let durations: Vec<f64> = ch
|
||||||
|
.all_videos()
|
||||||
|
.filter_map(|v| v.duration_secs)
|
||||||
|
.filter(|d| *d > 0.0)
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let (Some(group), confidence) = classify(ch, &durations) else {
|
||||||
|
continue;
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut reason = group_reason(ch, &durations);
|
||||||
|
if let Some(per_month) = cadence_per_month(ch) {
|
||||||
|
reason.push_str(&format!("; ~{per_month}/mo"));
|
||||||
|
}
|
||||||
|
|
||||||
|
groups.entry(group).or_default().push(ChannelSuggestion {
|
||||||
|
platform: ch.platform.dir_name().to_string(),
|
||||||
|
platform_label: ch.platform.display_name().to_string(),
|
||||||
|
handle: ch.name.clone(),
|
||||||
|
display_name: ch
|
||||||
|
.meta
|
||||||
|
.as_ref()
|
||||||
|
.and_then(|m| m.uploader.clone())
|
||||||
|
.filter(|u| !u.is_empty())
|
||||||
|
.unwrap_or_else(|| ch.name.clone()),
|
||||||
|
reason,
|
||||||
|
confidence,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
groups
|
||||||
|
.into_iter()
|
||||||
|
.map(|(group, mut channels)| {
|
||||||
|
// Strongest signal first within each group.
|
||||||
|
channels.sort_by(|a, b| {
|
||||||
|
b.confidence
|
||||||
|
.partial_cmp(&a.confidence)
|
||||||
|
.unwrap_or(std::cmp::Ordering::Equal)
|
||||||
|
});
|
||||||
|
GroupSuggestion {
|
||||||
|
group: group.to_string(),
|
||||||
|
channels,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Choose a group + confidence for a channel, or `(None, _)` when no signal is
|
||||||
|
/// strong enough to suggest anything (the common mid-length YouTube case).
|
||||||
|
fn classify(ch: &Channel, durations: &[f64]) -> (Option<&'static str>, f32) {
|
||||||
|
match ch.platform {
|
||||||
|
Platform::Bandcamp | Platform::SoundCloud => return (Some("Music"), 0.95),
|
||||||
|
Platform::Twitch => return (Some("Streams & VODs"), 0.9),
|
||||||
|
Platform::TikTok => return (Some("Shorts"), 0.9),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
// The user already downloads this channel as audio → treat as music.
|
||||||
|
if ch.download_options.audio_only {
|
||||||
|
return (Some("Music"), 0.85);
|
||||||
|
}
|
||||||
|
match median(durations) {
|
||||||
|
Some(m) if m < SHORTS_MAX => (Some("Shorts"), 0.7),
|
||||||
|
Some(m) if m >= LONGFORM_MIN => (Some("Long-form & Podcasts"), 0.7),
|
||||||
|
_ => (None, 0.0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn group_reason(ch: &Channel, durations: &[f64]) -> String {
|
||||||
|
match ch.platform {
|
||||||
|
Platform::Bandcamp | Platform::SoundCloud => return "music platform".to_string(),
|
||||||
|
Platform::Twitch => return "Twitch channel".to_string(),
|
||||||
|
Platform::TikTok => return "TikTok channel".to_string(),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
if ch.download_options.audio_only {
|
||||||
|
return "downloaded as audio-only".to_string();
|
||||||
|
}
|
||||||
|
match median(durations) {
|
||||||
|
Some(m) => format!("median length {}", fmt_dur(m)),
|
||||||
|
None => "no duration data".to_string(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn median(durations: &[f64]) -> Option<f64> {
|
||||||
|
if durations.is_empty() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let mut v = durations.to_vec();
|
||||||
|
v.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||||
|
let mid = v.len() / 2;
|
||||||
|
Some(if v.len() % 2 == 0 {
|
||||||
|
(v[mid - 1] + v[mid]) / 2.0
|
||||||
|
} else {
|
||||||
|
v[mid]
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fmt_dur(secs: f64) -> String {
|
||||||
|
let s = secs.round() as u64;
|
||||||
|
if s < 90 {
|
||||||
|
format!("{s} s")
|
||||||
|
} else {
|
||||||
|
format!("{} min", (s + 30) / 60)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Rough videos-per-month from the spread of upload dates. `None` if fewer than
|
||||||
|
/// two videos carry a parseable `YYYYMMDD`.
|
||||||
|
fn cadence_per_month(ch: &Channel) -> Option<u64> {
|
||||||
|
let mut dates: Vec<u32> = ch
|
||||||
|
.all_videos()
|
||||||
|
.filter_map(|v| v.upload_date.as_deref())
|
||||||
|
.filter_map(|d| d.parse::<u32>().ok())
|
||||||
|
.filter(|d| *d > 0)
|
||||||
|
.collect();
|
||||||
|
if dates.len() < 2 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
dates.sort_unstable();
|
||||||
|
let months = month_span(*dates.first()?, *dates.last()?).max(1);
|
||||||
|
Some((dates.len() as u64 / months).max(1))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whole-month span between two `YYYYMMDD` integers (clamped at 0).
|
||||||
|
fn month_span(a: u32, b: u32) -> u64 {
|
||||||
|
let (ay, am) = ((a / 10000) as i64, ((a / 100) % 100) as i64);
|
||||||
|
let (by, bm) = ((b / 10000) as i64, ((b / 100) % 100) as i64);
|
||||||
|
((by - ay) * 12 + (bm - am)).max(0) as u64
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use crate::library::{Channel, Video};
|
||||||
|
|
||||||
|
fn vid(dur: f64, date: &str) -> Video {
|
||||||
|
Video {
|
||||||
|
id: "x".into(),
|
||||||
|
title: "t".into(),
|
||||||
|
stem: "t".into(),
|
||||||
|
video_path: None,
|
||||||
|
thumb_path: None,
|
||||||
|
description_path: None,
|
||||||
|
info_path: None,
|
||||||
|
subtitles: Vec::new(),
|
||||||
|
has_live_chat: false,
|
||||||
|
duration_secs: Some(dur),
|
||||||
|
has_chapters: false,
|
||||||
|
file_size: None,
|
||||||
|
mtime_unix: None,
|
||||||
|
upload_date: Some(date.into()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn channel(platform: Platform, durations: &[f64]) -> Channel {
|
||||||
|
let videos: Vec<Video> = durations.iter().map(|d| vid(*d, "20240101")).collect();
|
||||||
|
let total = videos.len();
|
||||||
|
Channel {
|
||||||
|
name: "chan".into(),
|
||||||
|
path: std::path::PathBuf::from("/tmp/chan"),
|
||||||
|
platform,
|
||||||
|
source_url: None,
|
||||||
|
videos,
|
||||||
|
playlists: Vec::new(),
|
||||||
|
meta: None,
|
||||||
|
total_videos_cached: total,
|
||||||
|
total_size_cached: 0,
|
||||||
|
download_options: Default::default(),
|
||||||
|
folder_id: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn shorts_detected_by_median_duration() {
|
||||||
|
let groups = suggest(&[channel(Platform::YouTube, &[30.0, 45.0, 20.0])]);
|
||||||
|
assert_eq!(groups.len(), 1);
|
||||||
|
assert_eq!(groups[0].group, "Shorts");
|
||||||
|
assert_eq!(groups[0].channels.len(), 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn longform_detected_by_median_duration() {
|
||||||
|
let groups = suggest(&[channel(Platform::YouTube, &[3600.0, 2700.0, 4000.0])]);
|
||||||
|
assert_eq!(groups[0].group, "Long-form & Podcasts");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn music_platform_grouped_regardless_of_duration() {
|
||||||
|
let groups = suggest(&[channel(Platform::Bandcamp, &[200.0, 240.0, 180.0])]);
|
||||||
|
assert_eq!(groups[0].group, "Music");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mid_length_youtube_yields_no_suggestion() {
|
||||||
|
// ~8 min median: ambiguous, deliberately not suggested.
|
||||||
|
let groups = suggest(&[channel(Platform::YouTube, &[480.0, 500.0, 460.0])]);
|
||||||
|
assert!(groups.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn filed_or_tiny_channels_are_skipped() {
|
||||||
|
let mut filed = channel(Platform::YouTube, &[30.0, 30.0, 30.0]);
|
||||||
|
filed.folder_id = Some(7);
|
||||||
|
let tiny = channel(Platform::YouTube, &[30.0]); // < MIN_VIDEOS
|
||||||
|
assert!(suggest(&[filed, tiny]).is_empty());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -14,6 +14,7 @@
|
||||||
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
|
#![cfg_attr(not(debug_assertions), windows_subsystem = "windows")]
|
||||||
|
|
||||||
mod app;
|
mod app;
|
||||||
|
mod autotag;
|
||||||
mod config;
|
mod config;
|
||||||
mod crash;
|
mod crash;
|
||||||
mod database;
|
mod database;
|
||||||
|
|
|
||||||
81
src/web.rs
81
src/web.rs
|
|
@ -2112,6 +2112,85 @@ async fn get_maintenance_scan(State(state): State<Arc<WebState>>) -> impl IntoRe
|
||||||
Json(report)
|
Json(report)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `GET /api/autotag/suggest` — heuristic folder-grouping suggestions for
|
||||||
|
/// unfiled channels (see [`crate::autotag`]). Pure arithmetic over the
|
||||||
|
/// in-memory library, so it's computed on demand rather than as a job.
|
||||||
|
async fn get_autotag_suggest(State(state): State<Arc<WebState>>) -> impl IntoResponse {
|
||||||
|
let lib = state.library.lock_recover();
|
||||||
|
Json(crate::autotag::suggest(&lib))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct AutotagChannel {
|
||||||
|
platform: String,
|
||||||
|
handle: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct AutotagApplyBody {
|
||||||
|
/// Target folder name; created if it doesn't already exist.
|
||||||
|
group: String,
|
||||||
|
channels: Vec<AutotagChannel>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `POST /api/autotag/apply` — accept a suggested group: create the folder
|
||||||
|
/// (or reuse an existing one with the same name) and move the given channels
|
||||||
|
/// into it. Mirrors the move onto the in-memory library + bumps the ETag, the
|
||||||
|
/// same way `post_assign_folder` does for a single channel.
|
||||||
|
async fn post_autotag_apply(
|
||||||
|
State(state): State<Arc<WebState>>,
|
||||||
|
Json(body): Json<AutotagApplyBody>,
|
||||||
|
) -> impl IntoResponse {
|
||||||
|
let name = body.group.trim();
|
||||||
|
if name.is_empty() {
|
||||||
|
return (StatusCode::BAD_REQUEST, "empty group name").into_response();
|
||||||
|
}
|
||||||
|
// Reuse a folder of the same name (case-insensitive) if one exists,
|
||||||
|
// otherwise create it.
|
||||||
|
let existing = match state.db.list_folders() {
|
||||||
|
Ok(folders) => folders
|
||||||
|
.iter()
|
||||||
|
.find(|f| f.name.eq_ignore_ascii_case(name))
|
||||||
|
.map(|f| f.id),
|
||||||
|
Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, format!("db: {e}")).into_response(),
|
||||||
|
};
|
||||||
|
let folder_id = match existing {
|
||||||
|
Some(id) => id,
|
||||||
|
None => match state.db.create_folder(name) {
|
||||||
|
Ok(id) => id,
|
||||||
|
Err(e) => {
|
||||||
|
return (StatusCode::INTERNAL_SERVER_ERROR, format!("create folder: {e}"))
|
||||||
|
.into_response()
|
||||||
|
}
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut applied = 0usize;
|
||||||
|
for c in &body.channels {
|
||||||
|
if state
|
||||||
|
.db
|
||||||
|
.set_channel_folder(&c.platform, &c.handle, Some(folder_id))
|
||||||
|
.is_ok()
|
||||||
|
{
|
||||||
|
applied += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
{
|
||||||
|
let mut lib = state.library.lock_recover();
|
||||||
|
for ch in lib.iter_mut() {
|
||||||
|
if body
|
||||||
|
.channels
|
||||||
|
.iter()
|
||||||
|
.any(|c| ch.platform.dir_name() == c.platform && ch.name == c.handle)
|
||||||
|
{
|
||||||
|
ch.folder_id = Some(folder_id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
bump_library_version(&state);
|
||||||
|
Json(serde_json::json!({ "folder_id": folder_id, "applied": applied })).into_response()
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
struct RemoveRequest {
|
struct RemoveRequest {
|
||||||
paths: Vec<PathBuf>,
|
paths: Vec<PathBuf>,
|
||||||
|
|
@ -3088,6 +3167,8 @@ async fn serve(config: Config, shutdown_rx: std::sync::mpsc::Receiver<()>) {
|
||||||
.route("/api/plex/generate", post(post_plex_generate))
|
.route("/api/plex/generate", post(post_plex_generate))
|
||||||
.route("/api/maintenance/scan", get(get_maintenance_scan))
|
.route("/api/maintenance/scan", get(get_maintenance_scan))
|
||||||
.route("/api/maintenance/remove", post(post_maintenance_remove))
|
.route("/api/maintenance/remove", post(post_maintenance_remove))
|
||||||
|
.route("/api/autotag/suggest", get(get_autotag_suggest))
|
||||||
|
.route("/api/autotag/apply", post(post_autotag_apply))
|
||||||
.route("/api/maintenance/dedup/scan", post(post_dedup_scan))
|
.route("/api/maintenance/dedup/scan", post(post_dedup_scan))
|
||||||
.route("/api/maintenance/dedup/status", get(get_dedup_status))
|
.route("/api/maintenance/dedup/status", get(get_dedup_status))
|
||||||
.route("/api/maintenance/repair/:id", post(post_maintenance_repair))
|
.route("/api/maintenance/repair/:id", post(post_maintenance_repair))
|
||||||
|
|
|
||||||
|
|
@ -2200,8 +2200,63 @@ function renderMaintenance(r){
|
||||||
h+=`<h3 style="font-size:13px;margin:16px 0 8px">Similar content (perceptual)</h3>
|
h+=`<h3 style="font-size:13px;margin:16px 0 8px">Similar content (perceptual)</h3>
|
||||||
<div class="settings-hint" style="margin-bottom:8px">Finds the same video re-uploaded under a different ID — mirrors, re-encodes, resolution changes — by comparing sampled frames. The first scan fingerprints your library (a few minutes); after that it's cached, so re-scans are quick.</div>
|
<div class="settings-hint" style="margin-bottom:8px">Finds the same video re-uploaded under a different ID — mirrors, re-encodes, resolution changes — by comparing sampled frames. The first scan fingerprints your library (a few minutes); after that it's cached, so re-scans are quick.</div>
|
||||||
<div id="dedup-area"></div>`;
|
<div id="dedup-area"></div>`;
|
||||||
|
h+=`<h3 style="font-size:13px;margin:16px 0 8px">🏷 Auto-tag suggestions</h3>
|
||||||
|
<div class="settings-hint" style="margin-bottom:8px">Groups your <b>unfiled</b> channels by source platform and typical video length. Review a group and apply it to drop those channels into a folder — nothing moves until you click Apply.</div>
|
||||||
|
<div id="autotag-area"><em style="color:var(--muted);font-size:12px">Analyzing…</em></div>`;
|
||||||
body.innerHTML=h;
|
body.innerHTML=h;
|
||||||
initDedupArea();
|
initDedupArea();
|
||||||
|
initAutotagArea();
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Smart auto-tagging (suggested folder groups) ───────────────── */
|
||||||
|
async function initAutotagArea(){
|
||||||
|
const area=document.getElementById('autotag-area');if(!area)return;
|
||||||
|
let groups;
|
||||||
|
try{groups=await(await api('/api/autotag/suggest')).json();}
|
||||||
|
catch(e){area.innerHTML=`<div style="color:#f87171;font-size:12px">Could not analyze: ${esc(e.message)}</div>`;return}
|
||||||
|
renderAutotag(groups);
|
||||||
|
}
|
||||||
|
function renderAutotag(groups){
|
||||||
|
const area=document.getElementById('autotag-area');if(!area)return;
|
||||||
|
if(!groups||!groups.length){area.innerHTML='<div style="color:var(--muted);font-size:12px">No grouping suggestions — every channel is already filed or too ambiguous to call.</div>';return}
|
||||||
|
// Confidence → dot colour: strong (green) ≥ .85, medium (amber) ≥ .7, else grey.
|
||||||
|
const dot=c=>{const col=c>=0.85?'#4ade80':c>=0.7?'#facc15':'#9ca3af';return `<span title="confidence ${(c*100).toFixed(0)}%" style="flex-shrink:0;width:8px;height:8px;border-radius:50%;background:${col};display:inline-block"></span>`};
|
||||||
|
let h='';
|
||||||
|
groups.forEach((g,gi)=>{
|
||||||
|
h+=`<div style="border:1px solid var(--border);border-radius:6px;padding:8px;margin-bottom:8px" data-grp="${gi}">
|
||||||
|
<div style="display:flex;align-items:center;gap:8px;margin-bottom:6px">
|
||||||
|
<span style="font-weight:600;font-size:12px;flex:1">📁 ${esc(g.group)} <span style="color:var(--muted)">(${g.channels.length})</span></span>
|
||||||
|
<button class="primary" onclick="applyAutotag(${gi},this)">Apply → ${esc(g.group)}</button>
|
||||||
|
</div>`;
|
||||||
|
g.channels.forEach((c,ci)=>{
|
||||||
|
h+=`<label style="display:flex;align-items:center;gap:8px;font-size:12px;padding:3px 0">
|
||||||
|
<input type="checkbox" class="at-chk" data-grp="${gi}" data-ci="${ci}" checked>
|
||||||
|
${dot(c.confidence)}
|
||||||
|
<span style="flex:1;overflow:hidden;text-overflow:ellipsis;white-space:nowrap">${esc(c.display_name)} <span style="color:var(--muted)">· ${esc(c.platform_label)} · ${esc(c.reason)}</span></span>
|
||||||
|
</label>`;
|
||||||
|
});
|
||||||
|
h+=`</div>`;
|
||||||
|
});
|
||||||
|
area.innerHTML=h;
|
||||||
|
// Stash the data so applyAutotag can read the channel keys back.
|
||||||
|
area._groups=groups;
|
||||||
|
}
|
||||||
|
async function applyAutotag(gi,btn){
|
||||||
|
const area=document.getElementById('autotag-area');
|
||||||
|
const groups=area&&area._groups;if(!groups||!groups[gi])return;
|
||||||
|
const g=groups[gi];
|
||||||
|
const picked=[...area.querySelectorAll(`.at-chk[data-grp="${gi}"]`)]
|
||||||
|
.filter(cb=>cb.checked)
|
||||||
|
.map(cb=>g.channels[+cb.dataset.ci])
|
||||||
|
.map(c=>({platform:c.platform,handle:c.handle}));
|
||||||
|
if(!picked.length){setStatus('No channels selected');return}
|
||||||
|
btn.disabled=true;btn.textContent='Applying…';
|
||||||
|
try{
|
||||||
|
const r=await(await api('/api/autotag/apply',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({group:g.group,channels:picked})})).json();
|
||||||
|
setStatus(`Moved ${r.applied} channel${r.applied===1?'':'s'} → ${g.group}`);
|
||||||
|
await loadLibrary();renderSidebar();
|
||||||
|
initAutotagArea(); // re-analyze: applied channels drop out (now filed)
|
||||||
|
}catch(e){setStatus('Apply failed: '+e.message);btn.disabled=false;btn.textContent='Apply → '+g.group;}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ── Perceptual dedup (similar content) ─────────────────────────── */
|
/* ── Perceptual dedup (similar content) ─────────────────────────── */
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue